Update README.md

README.md

@@ -1,3 +1,99 @@
----
-license: apache-2.0
----
+---
+frameworks:
+- Pytorch
+
+tasks:
+- image-to-image-synthesis
+
+#model-type:
+##如 gpt、phi、llama、chatglm、baichuan 等
+#- gpt
+
+#domain:
+##如 nlp、cv、audio、multi-modal
+#- nlp
+
+#language:
+##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa
+#- cn
+
+#metrics:
+##如 CIDEr、Blue、ROUGE 等
+#- CIDEr
+
+#tags:
+##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他
+#- pretrained
+
+#tools:
+##如 vllm、fastchat、llamacpp、AdaSeq 等
+#- vllm
+base_model:
+  - Qwen/Qwen-Image-Edit
+base_model_relation: adapter
+---
+# Qwen-Image-Edit Low-Resolution Input Repair LoRA
+
+## Model Introduction
+
+[Qwen-Image-Edit](https://www.modelscope.cn/models/Qwen/Qwen-Image-Edit) is a powerful open-source image editing model. However, when the input resolution of the model is lower than the target resolution for image generation, the model's ability to maintain image details is poor. To address this, we made the following two modifications:
+
+1. Rope Interpolation: The position encoding of the input image in Qwen-Image DiT is changed to an interpolated sampling of the position encoding at the target resolution. This modification can take effect independently of modification 2.
+2. LoRA Fine-tuning: Quickly train a LoRA model to enhance the generalization of this interpolated encoding by DiT.
+
+With these two modifications, the model can produce consistent edited images even when given low-resolution input. Additionally, compared to high-resolution input, the inference time of the model is significantly reduced.
+
+## Effect Demonstration
+
+Image Editing Instruction: Change the skirt to pink.
+|Input Resolution|A100 Inference Time|Input Image|Original Model|Rope Interpolation|Rope Interpolation + LoRA Fine-tuning|
+|-|-|-|-|-|-|
+|256x256| 39 s |![](./assets/image1.jpg)|![](./assets/origin_256.jpg)|![](./assets/rope_256.jpg)|![](./assets/lora_256.jpg)|
+|512x512| 50 s |![](./assets/image1.jpg)|![](./assets/origin_512.jpg)|![](./assets/rope_512.jpg)|![](./assets/lora_512.jpg)|
+|768x768| 67 s |![](./assets/image1.jpg)|![](./assets/origin_768.jpg)|![](./assets/rope_768.jpg)|![](./assets/lora_768.jpg)|
+|1024x1024| 98 s|![](./assets/image1.jpg)|![](./assets/origin_1024.jpg)|![](./assets/origin_1024.jpg)|![](./assets/lora_1024.jpg)|
+
+## Limitations
+
+1. Using low-resolution input and generating high-resolution output will greatly reduce the inference time, but it may degrade the model's editing performance.
+2. The above analysis is only focused on the model's ability to maintain image detail.
+
+## Inference Code
+```
+git clone https://github.com/modelscope/DiffSynth-Studio.git  
+cd DiffSynth-Studio
+pip install -e .
+```
+
+```python
+from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig
+import torch
+from modelscope import snapshot_download
+
+pipe = QwenImagePipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="Qwen/Qwen-Image-Edit", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"),
+    ],
+    tokenizer_config=None,
+    processor_config=ModelConfig(model_id="Qwen/Qwen-Image-Edit", origin_file_pattern="processor/"),
+)
+snapshot_download("DiffSynth-Studio/Qwen-Image-Edit-Lowres-Fix", local_dir="models/DiffSynth-Studio/Qwen-Image-Edit-Lowres-Fix", allow_file_pattern="model.safetensors")
+pipe.load_lora(pipe.dit, "models/DiffSynth-Studio/Qwen-Image-Edit-Lowres-Fix/model.safetensors")
+
+prompt = "Exquisite portrait of an underwater girl with flowing blue dress and fluttering hair. Transparent light and shadow, surrounded by bubbles. Her face is serene, with exquisite details and dreamy beauty."
+image = pipe(prompt=prompt, seed=0, num_inference_steps=40, height=1024, width=768)
+image.save("image.jpg")
+
+prompt = "turn skirt pink"
+image = image.resize((512, 384))
+image = pipe(prompt, edit_image=image, seed=1, num_inference_steps=40, height=1024, width=768, edit_rope_interpolation=True, edit_image_auto_resize=False)
+image.save(f"image2.jpg")
+```
+
+---
+license: apache-2.0
+---