Benchmarking Diffusion Based Text-Guided Image Editing Methods

We thank the reviewer for the constructive comments, especially that the problem is important and well-motivated given a lack of standardized editing benchmarks!

Below we provide responses to the reviewer's questions:

the proposed benchmark only includes real images from MSCOCO, a missing evaluation on common use case is editing on synthetic generated images, which I think should be added to the benchmark – the automatic evaluation pipeline cannot capture hallucinations, i.e. Figure 1 object addition,.... : We thank the reviewer for these ideas - they are interesting! The scope of our paper was to develop an evaluation pipeline to evaluate edits on real images. Although our pipeline could plausibly be extended to evaluating edits in synthetically generated images, we did not explicitly consider these in our work. We also note that expanding the size of the evaluation dataset can significantly increase the evaluation time, so our choice of dataset size was carefully considered. For this reason, we consider an extension to synthetically generated images as future work.

For the reviewer’s second point on understanding if the original content is preserved or not, we highlight that the automatic evaluation does contain computing the DINO scores in conjunction with the scores from OwL-ViT. These DINO scores are proxies for the third question in human-study which captures if the original image content is preserved well or not. We have updated Sec. (3.1) to better reflect this. Indeed, we find that it displays a strong correlation with human-study as well (see Sec. 4.3 and Sec. (D)). We have made these points more clear in the main paper and provided additional guidelines in the form of a checklist (see updated Sec. (5)) on how our framework can be used to evaluate text-guided editing methods.

what are your thoughts on the practical usage of the benchmark? does achieving a high score translate into a better editing method in practice or does it just mean it is relatively better than other methods? ; I do have concerns on the practical usefulness of the benchmark.. : Current text-guided image editing methods predominantly hinge on a singular metric, the CLIP-Score. Additionally, there is no single standardized dataset that covers a diverse array of edit operations (e.g. object-addition or changing the location of an object) which further compounds the evaluation challenge. Existing works often resort to human studies, yet the templates employed vary across papers, making fair comparison difficult. Our paper addresses these challenges by presenting a comprehensive standardized package, inclusive of a dataset and two evaluation schemes, designed to capture different facets of text-guided image editing methods. Our benchmark's evaluations offer richer insights than CLIP-Scores, probing methods across various dimensions of edits. Moreover, our human-study template can seamlessly adapt to future editing methods without the reliance on edited images from preceding methods. Our benchmark can be employed as a practical checklist or guide for evaluating text-guided editing methods. We expound on these points in the main paper (refer to Sec. 5) to clarify how EditVal can be practically utilized. In essence, we propose the following use of EditVal as a checklist: (i) Dataset Utilization: Incorporate the 648 edit-operations from our dataset along with a new image editing method. (ii) Editing Accuracy Evaluation: (a) Employ Owl-ViT rules in the automatic evaluation pipeline to obtain scores for the six edit-types. (b)Use our standardized human-study template to acquire scores for the seven edit-types. (iii) Context Preservation Assessment: Utilize DINO and FID to gauge the efficacy of the method in preserving image context. (iv) Compile a comprehensive report integrating information from Steps 1, 2, and 3 to comprehend the effectiveness of the editing method in fine-grained detail.

We contend that relying on a single score is inadequate, as exemplified by Fig. 5, where SDE-Edit outperforms Dreambooth in object-addition but lags behind in object-replacement. A singular score fails to unveil such nuanced details about a method, underscoring the importance of our proposed comprehensive benchmark report for a more thorough understanding of editing method efficacy. We want to highlight another important point in terms of practical usage : We have open-sourced our code at https://github.com/deep-ml-research/editval_code, will maintain a leaderboard for different edit-types at https://deep-ml-research.github.io/editval/ and our dataset is extremely easy to use with a COCO-dataloader used in conjunction with a json file (as shown in Sec. (B)).

We thank the reviewer for providing constructive comments, appreciating our extensive study and findings!

Below we provide responses to the reviewer's questions:

Only 6 out of 13 edits are supported. : The reviewer rightly points out that our OWL-ViT pipeline is used to evaluate only 6 out of the 13 edit types. This is because these are the edit types for which the pipeline produces scores that are strongly correlated with human scores. For the remaining 7 edit types, the OWL-ViT pipeline, similar to CLIP produces scores that are only weakly correlated with human scores (see updated Sec. (K)). We did not want to throw the baby out with the bathwater, so we proposed the OWL-based pipeline as a proxy for a human study for the first 6 edit types. We recommend our standardized human template to evaluate the remaining 7 edit types which does not require using edited images from earlier methods to evaluate a given editing method. We note that this standardized template is a contribution in and of itself, and can be used by future editing methods to compare to earlier methods. While this does not present a ‘unified’ evaluation pipeline, we note that the OWL-based pipeline offers a more fine-grained metric compared to CLIP Scores (see Sec. (J)). .

While we would have liked to also contribute an automated way of evaluating the remaining 7 edits types, we have left this for future work - perhaps a future extension of EditVal!

We have also updated Sec. (5) in the paper to provide guidelines/a checklist for how to evaluate an image editing method on EditVal and how to interpret the results.

The pipeline can only check the object presence, location and size. For example, it cannot recognize if a cup stands naturally on the bench for “add a cup on the bench”. These aspects make this evaluator quite vulnerable. : We recognize that our current pipeline has a limitation in determining whether a cup is naturally placed on a bench. We consider this issue as part of the larger challenge of assessing text-to-image generation methods across all conceivable scenarios. Our work represents a step forward in evaluating these methods within a specific subset of scenarios - For e.g.,(i) if a new object is added (object-addition), then our pipeline checks if both the old and the new object is present; (ii) If an object is altered (alter-parts), then the alteration is within that given object, leveraging predicted bounding boxes.

We consider our automated evaluation pipeline to be a significant step from CLIP Scores, which are the current standard for evaluating text-to-image models and provide little fine-grained understanding of where models are failing and succeeding. As the evaluation of generative models continues to evolve, we anticipate that criteria addressing issues such as the natural placement of objects can be smoothly incorporated into future versions of our framework.

It does not evaluate image fidelity and image-context preservation. One still needs FID and DINO/CLIP scores which are not specifically designed for editing evaluation. : The reviewer correctly notes that our pipeline uses FID and DINO scores to evaluate image fidelity and image-context preservation (which correspond to the second and third question in our human study). Even though these metrics are not specifically designed for editing evaluation, we find that they are strongly correlated with our human study scores (see Sec. (4.3) and Sec. (D) in the appendix). We have updated the text in Sec. (3) to emphasize that our automated pipeline uses FID/DINO metrics to check image fidelity and context preservation, so that there is no confusion.

We thank the reviewer for the constructive comments and appreciating the fact that we tackle a timely and important research topic!

Below we provide responses to the reviewer's questions:

One big limitation of the paper is its poor presentation. It is very crowded, with a lot of vspace adjustments, making the reading experience uncomfortable. Several facts are repeated numerous times, notably editval’s description and main contributions, while key elements are left for the reader to find in the appendix (e.g. the choice of the COCO dataset as source of images) : We apologize that the paper’s presentation felt crowded - we had many results to share! To address this, we have: (i) Updated Sec. (3) (especially 3.1) to be more readable with additional context to remove ambiguity. (ii) Removed the \vspace adjustments from all the sections to make the paper more presentable. (iii) Removed irrelevant information from 3.1 to the Appendix, to only highlight the main parts of the data-collection and annotation process. (iv) We have also clarified the definition of EditVal in the introduction and other parts of the paper (Sec. 3 in particular). Specifically, we consider EditVal as a benchmark for evaluating text-guided editing methods. EditVal, therefore, includes all the pieces needed to evaluate a method - the data and its annotations, the automated evaluation pipeline (for 6 edit-types) and the templates for the standardized human-study (for 7 edit-types). (v) In addition, we have added a new section (see Sec. (5)) with guidelines on how to use it, to remove any confusion.

We hope these changes collectively make the paper more readable and presentable!

The paper is presenting many contributions : building a benchmark, analyzing pre-existing methods through a human study, developing an automated evaluation metric and comparing this metric’s performance to user preferences. It is impossible to address all thoroughly and clearly, leading to a lack of in depth discussion and exploration. For example, the user study can provide .... : We thank the reviewer for raising this point. We did consider splitting our contributions into multiple papers, but felt that we could not present a new benchmark without thoroughly validating it (through the automated and human-based assessment of pre-existing methods). We therefore relied on the appendix, with deeper explorations of the models and findings included there. To remedy this, we have moved some of the important points of our work to the main paper such as : (i) Better describing the dataset and its filtering process in updated Sec. (3.1); (ii) Reorganizing the presentation of results in Sec. (4.2). (iii) Provide guidelines on how to use EditVal in a new Sec. 5.

In response to ‘overclaiming contribution that automated pipeline is an alternative to a human study’ - In almost all cases, an automated pipeline (e.g. using OWL-ViT) cannot be used as a full replacement for a human study. As the reviewer notes, these pipelines often only give simplified outputs (e.g. binary success/fail) and cannot be used to evaluate more complex edit types (e.g. image quality). These simplified outputs however, can still provide useful signals for understanding model performance. So rather than throwing the baby out with the bathwater, EditVal leverages these for 6 of the edit types, and complements them with other metrics (e.g. FID, DINO) and a human study to evaluate the remaining 7 edit types.

We consider these other metrics as part of the full automated pipeline, however, note that we have slightly conflated some terminology. We have updated the manuscript in Sec. (3.2.2) to refer to the OWL-ViT based pipeline as the ‘detector-based automated pipeline’ and the DINO/FID metrics as the context-preservation pipeline. These two parts in conjunction make the automated pipeline in our paper.

We show that our detector-based pipeline using OWL-ViT produces scores that are strongly correlated with human scores for the 6 edit types we use it to evaluate. OWL’s scores are poorly correlated with human scores for the remaining 7 edit types (similar to CLIP), which are more complex, hence we recommend the DINO metrics and human study templates for them. We have updated Sec. (K) for a discussion on this.

We believe that EditVal takes a significant step beyond CLIP Scores, which are the current evaluation standard, to provide a more fine-grained way to evaluate image-editing methods leveraging our dataset and its annotations on different edit-types.

We thank the reviewer for appreciating our work, especially the fact that we tackle a previously unaddressed issue in image editing evaluation and provide thorough evaluations!

Below we provide our responses to the reviewer's questions:

There is a need for more detailed information regarding the implementation of the image editing methods. For example, methods such as Null-Text and SINE, are not based on instructions. It is confusing that instruction-based editing evaluations are applied to them : We emphasize that, when utilizing Null-Text and SINE, we abstain from employing explicit instructions to generate edited images. We have provided details of our instruction-free prompts in Section I and Table 3-Appendix, and will aim to make these more prominent in the final manuscript. Specifically, the prompts are constructed by focusing on the object in the image (e.g., a bench) and its corresponding editing operation (e.g., object-addition). We ensure that the prompts share a similar nature across models, comprising the object and descriptive text indicating the editing operation (e.g., "A brown bench" when altering the bench color to 'brown'). However, there are instances where the prompt may be adjusted based on the specific editing method; for example, for Dreambooth, which necessitates a [V]* token, the curated prompt might be 'A brown [V]*bench'. In the case of instruction-based approaches like Instruct-Pix2Pix, we adhere to a generic instruction template. For instance, if the task involves adding a new object alongside an existing one, the prompt would be 'Add a <new-object> to the <object>'. These details are also updated in Sec. (I).

The evaluation appears to lack focus on image-editing methods that deal with complex editing operations. For instance, methods like Diffusion Self-Guidance for Controllable Image Generation, which is designed to handle the shape, location, and appearance of objects, could be evaluated to show the performance of spatial manipulation. : We thank the reviewer for bringing this paper to our attention. At the time of submission, this paper was still under review at NeurIPS ‘23, and so we did not consider it in our evaluation. During the rebuttal, we tried to evaluate their method on our EditVal benchmark. However, the authors have not yet released their code. Their project page (https://dave.ml/selfguidance/) and paper (https://arxiv.org/abs/2306.00986) does not contain any links for code which would enable us to evaluate this method on EditVal. We have added a note on this in the updated version of our paper (see Sec. 2). However, we note that our benchmark is flexible to be used with any text-guided image-editing method in the future. Our human-study template as well as automatic evaluation does not require comparing edited images from earlier methods and only utilizes the current method in question. We hope that when their code is publicly available, EditVal can be used to benchmark its performance in a standardized way against other image-editing methods.

In our paper, we only choose a few popular editing methods to provide a starting point for comparing editing methods on EditVal, but with the flexibility of EditVal, any new method can be added!