HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing

We thank the reviewer for the detailed comments and find that our HQ-Edit is of a high resolution and better editing compared to the existing widely used InstructPix2Pix, which is supported by human evaluation. We address your concerns as follows:

Incremental improvements compared to InstructPix2Pix

We apologize for any confusion regarding our contribution. As explained in the Common Concerns/Necessity of diptych generation, DALL-E 3 and Prompt2Prompt differs significantly in functionality as: 1. DALL-3 API can not be naively utilized to do image-to-image generation. 2. In InstructPix2Pix, image consistency is achieved via manipulating the attention matrix of the generation model, which is impossible with any close-sourced model. In contrast to InstructPix2Pix, the pipeline of HQ-Edit is universal and can utilize any image generation model.

These restrictions motivate us to take diptych generation as a workaround and develop a curated post-processing pipeline so that such a powerful model can be utilized effectively and at scale.

Please see Common Concerns/Necessity of diptych generation for more.

Usage of synthetic image pairs has become less important

Thank you for raising this concern. We would like to highlight that an important contribution of our paper is providing a high-quality and scalable pipeline to collect data from powerful (closed-source) text-to-image (T2I) models. While our specific target is DALL-E 3 in this paper, we clarify that our pipeline can be easily adapted to other T2I models, such as the recent FLUX or Midjourney, which can generate more photorealistic images.

Next, following your suggestion, we evaluated our approach on the EMU Edit benchmark to demonstrate that, even with DALL-E 3, our results remain competitive:

As shown in the table, HQ-Edit performs the best in terms of CLIP_dir, CLIP_img, and Reward Score (Jiazheng et al. [1]) and ranks second-best in terms of DINO and SSIM metrics. Some qualitative results are shown in the real_image_results.png in this anonymous repo. With these results, We conclude that the superb quality and large size of HQ-Edit can still enable models to handle real-life and photorealistic images reasonably well.

Lastly, compared to collecting real images, we highlight that using synthetic images can be more scalable and comprehensive. We believe that a stronger editing dataset could be a mixture of synthetic and real images, where synthetic images contribute to comprehensiveness and precision—properties that may be challenging to achieve with real images alone.

We will add these discussions and the results in the revision.

Please see Common Concerns/Performance on real images (EMU Edit) for more.

Expert models produce better training data

Thanks for presenting this idea. Using expert models such as image inpainting models can yield better results in certain areas. However, this would significantly limit the range of editing operations. As illustrated in Fig. 1 and Fig. 11, our HQ-Edit process requires a lot of long and complicated editing instructions, which are far beyond the scope of expert models that are designed for very certain kinds of tasks. These complicated editing instructions in the training set also contribute to our finetuned models' performance.

Results showing un-edited regions are changed

We appreciate you bringing up this concern. We acknowledge that altering unedited regions is a common and open issue in instruction-based image editing, regardless of the datasets used for training. We hope this issue can be mitigated in the future through improved instruction-editing frameworks, such as enhanced prompt design and architectural changes.

Thanks for providing new evidences. Although some of my opinions have been changed, the major concerns are still not be addressed.

1. Given the Dall-E 3 generation pipeline, we need discuss whether it is the optimal way for generated high quality image editing dataset ? From current rebuttals, it is clear that the answer is no. Since there is no guarantee the data will contain high quality image pairs, therefore, no matter how well the filtering function it is, there is no way to result a high quality dataset. As I mentioned, I have seen lots of background and identity shift from the examples. Especially the newly provided examples give lots evidences to support my opinion. The reason behind that is the dataset itself has some bias for identity preservation. For example, replacement, view point change. even remove, all the results suffer id lost. Thus it is more like a image generation rather than image editing. The Dall-E3 is a text 2image model, it dose not learn how to preserve the image content. This is the main reason.

2. How dose the proposed method dataset applied to real world image editing? It is still questionable. There is no strong evidence on it. Given the newly showing results, I can observe that most examples are not useful in real world application. For example, 1 the hood of the car is not visually plausible. 2. make it rainy but changes whole thing. 3 changed the dog is still a dog. 4 Hydrant background totally changed. 5 removed refrigerator but adding a new one. 6,7,8 the image is not a sketch .... Only last three results look OK.

I am open to change my mind but I could not find many promising results to support it.

Thanks for your responses. However, we respectfully disagree with your opinions and would like to emphasize why we believe this work is valuable:

Firstly, as generative models become increasingly powerful, many of them remain closed-source. We believe it is both interesting and important to leverage their APIs to advance research within the open-source community. Our HQ-Edit is an example of this approach, demonstrating how to create a pipeline for generating high-quality datasets for instruction-based image editing. Our extensive experiments confirm its effectiveness: for instance, using our own dataset, we observe consistent and substantial improvements over other methods; with EMU Edit, we achieve competitive performance on real images. This suggests that well-designed synthetic data generation pipelines can be a viable alternative to human-annotated datasets in advancing image editing capabilities.

Moreover, we would like to clarify that our work can be easily extended to models other than DALL-E 3. For example, we believe that by applying our framework to more recent text-to-image models like FLUX, which can generate highly photorealistic images, we can produce an even stronger dataset to support instruction-based editing.

With these points in mind, we believe our work is valuable, and our experiments using DALL-E 3 serve as a valid proof-of-concept supporting the effectiveness of our developed framework.

Next, we would like to point out that your judgment of our generated image quality may not fully consider the comparative context. The appropriate way to assess our generated images is by evaluating them among all candidate models. To this end, we present per-image metrics (i.e., Alignment, Coherence, and CLIP_dir) below. We observe that our approach consistently maintains the highest win rate across all metrics, suggesting the effectiveness of our model. To create images that better satisfy visual judgments, leveraging larger models, longer training times, and other enhancements could help. However, this is beyond the focus of this paper.

We are happy to answer if you have any other questions.

As a researcher in image editing for many years. I want to express my deeply concerns again. I have emphasized many times in the discussion, Image editing should consider $**identity preservation**$. This is KEY difference to image generation. No matter which method, Dall-E3 or Flux. is used, if we only leverage prompts to generate a pair of editing images, it will be extremely difficult. HQ-Edit has demonstrated that with carefully design of filtering and prompting, the trained model has little value to improve the editing performance as most of real world image editing results are not acceptable. Please consider the human perceptual results rather than the metrics. All of these metrics can not fully indicate the results. Please explain my concerns on the newly presented failures. Please explain why the HQ-Edit claimed many editing types but replacement, inpaiting, insertion, and camera move/view point change can't give stratifying results. I think the dataset quality is not good enough otherwise those cases should not be failed. Prompt 2 prompt can handle both local editing and global editing, even though the quality is not good enough, it still has lots of identity well preserved cases with good filtering metrics. In terms of HQ-Edit, we can clearly see the identity preservation is main problem, it can't be addressed as the data generation pipeline has a fundamental problem: Dalle3 can't generate identity preserved editing pairs on object replacement, inpatinting, object insertion and etc,. Moreover, through API, the results has very limited control and this is reason that I don't believe the dataset will be useful for future research. Leveraging prompt control for image generation methods to generated a high quality local editing pairs is very difficult. HQ-Edit dataset claimed editing coverage is way over it can actually handle.

Dear Reviewer x6Ut, We may not disclose the exact data strategy. However, based on this limited tech report that we can share [https://anonymous.4open.science/r/hqedit_re-8D01/SeedEdit_Align_Image_Re_Generation_to_Image_Editing.pdf ], we assert that a significant portion of the data used in the SeedEdit model is regenerated or sampled using an internal commercialized T2I model, following similar principles to [diptych generation]. This approach is also independently acknowledged in recent studies, such as those proposing in-context LoRA [ https://arxiv.org/pdf/2410.23775 ] and OminiControl [ https://arxiv.org/pdf/2411.15098. Check A.1. Generation pipeline]

Overall, we could foresee that HQEdit has the potential to serve as a foundational strategy and provide strong motivation for diverse data generation. From many perspectives, transferring the methodology from HQEdit to more advanced released models, such as FLUX, can significantly enhance data performance and narrow the performance gap for real image applications.

We thank the reviewer for the detailed comments and find that the HQ-Edit dataset poccess high quality and utility, the data collection pipeline is scalable and well-defined, and the final model's performance is impressive. We address your concerns as follows:

Distribution shifting with synthetic images

Thank you for raising this concern. Yes, this is one limitation of our work at this stage. To better understand how our model handles real images, we first evaluate on the test set of EMU Edit, which consists only of real-life images.

As shown in the table, HQ-Edit performs the best in terms of CLIP_dir, CLIP_img, and Reward Score (Jiazheng et al. [1]) and ranks second-best in terms of DINO and SSIM metrics. Some qualitative results are shown in the real_image_results.png in this anonymous repo. With these results, We conclude that the superb quality and large size of HQ-Edit can still enable models to handle real-life and photorealistic images reasonably well.

Moreover, we believe this domain issue will be addressed in the future. For example, one possible strategy is to design a (sophisticated) mixture training algorithm to mitigate this distribution bias. This issue may also be effectively alleviated by switching from DALL-E 3 to more recent T2I models like FLUX, which can generate very photorealistic images to construct an upgraded version of HQ-Edit.

We will add these discussions in the next version.

Please see Common Concerns/Performance on real images (EMU Edit) for more.

Necessity of diptych generation

Thank you for raising this concern. As detailedly explained in the Common Concerns, we use diptych generation mainly as a workaround so that DALL-E 3, whose API does not innately support image-to-image generation, can be used to generate image pairs that have high relevance. We will add these clarifications in the next version.

Furthermore, to accurately and effectively extract info from generated diptych, we proposed a series of post-processing methods, including:

1. Image-wise We further improved image alignment using Warping & Filtering. As validated empirically in Sec. 4.4 and visually demonstrated in Fig. 3, these designs can improve image quality and, consequently, the model's performance, enabling our proposed dataset to further enhance the image editing models' capabilities, such as InstructPix2Pix.
2. Text-wise Additionally, we leveraged Instruction Refinement, using GPT-4V to refine the original editing instructions further, ensuring all still-existing misalignments after image post-processing are encompassed in editing instructions. Arguably, this makes all the image pairs achieve almost precise alignment as they correspond to the rewritten instructions in detail.

Please see Common Concerns/Necessity of diptych generation for more.

More classic metrics in evaluation

Thanks. Following your suggestion, we first present evaluation results with more classic metrics, including the most commonly used CLIP directional similarity ($\text{CLIP}_\text{dir}$), CLIP image similarity ($\text{CLIP}_\text{img}$), DINO image similarity (DINO), and SSIM. The results are as follows:

We can observe that the model trained with HQ-Edit consistently yields the highest performance across all metrics.

Please see Common Concerns/Other evaluation metrics and human evaluation for more.

[1] Xu, J., et al. (2023). ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In NeurIPS 2023

We thank the reviewer for the detailed comments and find that our HQ-Edit is of high quality and constructed with very advanced foundation models, e.g., GPT-4V and DALL-E 3, and covers a variety of editing operations. We address the concerns as follows:

Gap between real-life images and synthetic images

Thank you for raising this concern. Yes, this is one limitation of our work at this stage. To better understand how our model handles real images, we first evaluate on the test set of EMU Edit, which consists only of real-life images.

As shown in the table, HQ-Edit performs the best in terms of CLIP_dir, CLIP_img, and Reward Score (Jiazheng et al. [1]) and ranks second-best in terms of DINO and SSIM metrics. Some qualitative results are shown in the real_image_results.png in this anonymous repo. With these results, We conclude that the superb quality and large size of HQ-Edit can still enable models to handle real-life and photorealistic images reasonably well.

Moreover, we believe this domain issue will be addressed in the future. For example, one possible strategy is to design a (sophisticated) mixture training algorithm to mitigate this distribution bias. This issue may also be effectively alleviated by switching from DALL-E 3 to more recent T2I models like FLUX, which can generate very photorealistic images to construct an upgraded version of HQ-Edit.

We will add these discussions in the next version.

Please see Common Concerns/Performance on real images (EMU Edit) for more.

Using more popular metrics

Thanks. Following your suggestion, we first present evaluation results with more classic metrics, including the most commonly used CLIP directional similarity ($\text{CLIP}_\text{dir}$), CLIP image similarity ($\text{CLIP}_\text{img}$), DINO image similarity (DINO), and SSIM. The results are as follows:

We can observe that the model trained with HQ-Edit consistently yields the highest performance across all metrics.

Please see Common Concerns/Other evaluation metrics and human evaluation for more.

Human evaluation for Coherence

Thanks for raising this concern. We conducted a human evaluation with 160 image pairs formed with output images from HQ-Edit. Human evaluators are tasked with determining which image in each pair is better. "Image 1 is better", "Tie" and "Image 2 is better" are encoded as -1, 0 and 1. We then compute the Pearson Correlation between human evaluation scores and deltas between the GPT-evaluated scores of the first and second images.

We also employ the ImageReward (Jiazheng et al. [1]), which developed a BLIP-based model for evaluating alignment between text-to-image pairs and human preference, as the reward model for evaluating the output image quality. The correlation between scores from ImageReward and human evaluation is computed in the same way. The results are as follows:

The results above suggest that the Coherence score we proposed is significantly more aligned with human preference than the newly proposed reward model, ImageReward.

Please see Common Concerns/Other evaluation metrics and human evaluation for more.

Thank you for your response~

We thank the reviewer for the detailed comments and agree that the empirical evidence supports HQ-Edit's effectiveness as a finetuning dataset, and the proposed GPT-based evaluation is superior to the currently popular metric, which is the CLIP score. We address the concerns as follows:

HQ-Edit only contains non-rigid pair data

Sorry for the confusion. We would like to clarify that our HQ-Edit actually includes rigid operations like changing object viewpoint and rotating objects, e.g., viewpoint alteration, as shown in Fig. 1(a); object replacement, as shown in Fig. 1(d) and Fig. 10(left); object addition, as shown in Fig. 9(rightmost image).

More examples are available in this anonymous repo. See rigid_data_examples.png for more samples for objection addition, object removal, object replacement, and viewpoint alteration.

More qualitative results with rigid operation

Thanks for the suggestion. We just add such examples, which include operations such as "add objects", "remove objects", "replace objects", available at the rigid_operation_results.png in this anonymous repo., to illustrate the corresponding visual comparison. It is shown that the model finetuned with HQ-Edit yields generally better results in terms of the instruction following and image quality compared to InstructPix2Pix, MagicBrush, and HIVE.

Reward model for evaluation

Thanks for the suggestion. We employ ImageReward (Jiazheng et al. [1]), which develops a BLIP-based reward model for accessing text-to-image pairs' alignment with human preference, as the reward model for measuring the quality of output images. Since scores from ImageReward are unbounded, we normalize reward scores to $[0, 1]$ with Sigmoid. The results are as follows:

One can see that the model trained with HQ-Edit yields the highest performance.

We also present the human evaluation results with Coherence and ImageReward scores. We conducted a human evaluation with 160 image pairs formed with output images from HQ-Edit. Human evaluators are tasked with determining which image in each pair is better. "Image 1 is better", "Tie" and "Image 2 is better" are encoded as -1, 0 and 1. We then compute the Pearson Correlation between human evaluation scores and deltas between the GPT-evaluated scores of the first and second images. The correlation between scores from ImageReward and human evaluation is computed in the same way.

The results above suggest that the Coherence score we proposed is significantly more aligned with human preference than the newly proposed reward model. As the development of reward models for image generation and image editing is still at a preliminary stage, we find, alternatively, using a mature and well-tested general-purpose MLLM that is GPT-4V is valid and reasonable (though maybe not the best) at this stage.

Please see Common Concerns/Other evaluation metrics and human evaluation for more.

Typo in Sec4.4

Thanks for pointing this out. We will fix this in the revision.

Finetuned architecture of InstructPix2Pix

Sorry for missing the details. The model we finetuned is the Base version. We expect the switch to XL will lead to further performance improvement because of the large number of high-quality training samples provided by HQ-Edit, but we are unable to empirically verify it during the rebuttal stage mainly due to computing resource limitation (e.g., compared to Base, training XL requires much larger GPU memory consumption and much longer training time).

[1] Xu, J., et al. (2023). ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In NeurIPS 2023

Performance on real images (EMU Edit)

We first systematically evaluate our model on the test set of EMU Edit, which consists only with real-life images.

As shown in the table, HQ-Edit performs the best in terms of CLIP_dir, CLIP_img, and Reward Score (Jiazheng et al. [1]) and ranks second-best in terms of DINO and SSIM metrics. Some qualitative results are provided in the real_image_results.png in this anonymous repo. With these results, we can conclude that the superb quality and large size of HQ-Edit can still enable models to reasonably handle real-life and photorealistic images despite the domain gap. We also believe that adapting our framework to collect data from the more recent T2I models like FLUX--which can generate more photorealistic images--will enable us to see stronger performance on EMU Edit.

Lastly, compared to collecting real images, we stress that using synthetic images can be more scalable and comprehensive. We believe that a stronger editing dataset could be a mixture of synthetic and real images, where synthetic images contribute to comprehensiveness and precision—properties that may be challenging to achieve with real images alone.

[1] Xu, J., et al. (2023). ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In NeurIPS 2023