InstructBrush: Learning Attention-based Visual Instruction for Image Editing

Thank you very much for your constructive comments. We addressed you comments and questions and revised the paper. We summarize the answers below.

[W1] Input Constraint

We respect but disagree with your view regarding the proposed method's setup. Essentially, our method is not intended to enhance the effectiveness of general image editing tasks, but rather aims to accurately replicate the transformations between images pairs and apply them to new images. This setup has been thoroughly explored in the 'image analogy' task [1][2]. It uses pairs of images before-after transformation as conditions and provides an input image as a query to analogize to the result. Therefore, providing example image pairs to represent editing effects is necessary. Moreover, some existing works have explored representing transformations through example image pairs (image prompts) to guide image editing [3][4][5], thus the setup of this study has a solid theoretical background.

Moreover, because the example image pairs are responsible for demonstrating the transformation effects, they naturally contain similar parts. If the two images differ too much, it may be challenging to intuitively reflect the transformation concepts involved. Regarding the practicality of our method, we have tested various types of transformation effects. In addition to the editing covered in TOP-Bench, such as global editing, local editing, multi-object editing, shape editing, style editing, identity replacement, object replacement, and face attribute manipulation, we have also introduced image tone modification (Figure 7), hybrid editing (Figure 8), and real-world image editing (Figure 11). The results indicate that the method has strong generalization capabilities.

[W2] Redundancy of Exemplar Image Pairs

For the reviewer's concern about the necessity of using image pairs as input, we have already explained this in detail in [w1].

Furthermore, it is necessary for us to reiterate the implementation process of our method. The primary purpose of using image pairs as input is to serve as a supervisory signal, aiding the feature optimization process of the editing instructions. The role of using image pairs to acquire unique phrases is to introduce semantic information into our optimization process. Therefore, our method is essentially based on optimization to learn image pair-based editing, similar to the image analogy task [1][2]. This optimization approach can directly learn pixel-level or even mask-level transformations between image pairs, which are difficult for text-guided editing methods like GPT-4+IP2P to capture.

While it might be possible to introduce semantic priors into our optimization process through slight modifications to GPT-4, our unique phrase extraction method is more lightweight and efficient. In addition, it supports customization of the phrase dictionary and CLIP models to better suit downstream tasks.

[W3] Lack of Novelty

We fully understand the reviewer's emphasis on innovation. but as Reviewer 8Vpr stated in Strength 2, we emphasize that the innovation of our method does not lie in proposing a novel module. Instead, it delves into the extraction of editing concepts based on image prompts. We leverage the prior implementation of Instruction-based image editing models and innovatively introduce the KV token optimization in Attention for image prompts learning to represent the target editing instructions, enhancing the representational power of the concept. Furthermore, to further explore the impact of instructions on editing process, we introduce a time-aware method to optimize the instructions at different denoising times, comprehensively investigate the representation of editing instructions, and provide a clear solution for visual prompt editing.

[1] Hertzmann et al., Image analogies, SIGGRAPH 2001.

[2] Liao et al., Visual attribute transfer through deep image analogy, SIGGRAPH 2017.

[3] Nguyen et al., Visual instruction inversion: Image editing via image prompting, NIPS 2023.

[4] Yang et al., Imagebrush: Learning visual in-context instructions for exemplar-based image manipulation, NIPS 2023.

[5] Gu et al., Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion Model, SIGGRAPH 2024.

Thank you very much for your constructive comments. We addressed you comments and questions and revised the paper in brown color. We summarize the answers below.

[W1] The attention-based instruction module, seems to be a relatively simple technique, without too much novel insight.

We fully understand the reviewer's emphasis on innovation, but as Reviewer 8Vpr stated in Strength 2, we emphasize that the innovation of our method does not lie in proposing a novel module. Instead, it delves into the extraction of editing concepts based on image prompts. We leverage the prior implementation of Instruction-based image editing models and innovatively introduce the KV token optimization in Attention for image prompts learning to represent the target editing instructions, enhancing the representational power of the concept. Furthermore, to further explore the impact of instructions on editing process, we introduce a time-aware method to optimize the instructions at different denoising times, comprehensively investigate the representation of editing instructions, and provide a clear solution for visual prompt editing.

[W2] The explanation of when the time-aware instruction is effective or ineffective seems insufficient. Additional details on this aspect would enhance clarity

Thanks for the constructive suggestion. Time-aware optimization method emphasizes learning its focused edits at different denoising time steps. The edits obtained using the time-aware optimization approach are more stable and can be learned in more detail. We have described in detail in Section 6.2 that this module plays a key role in learning fine-grained edits, and visualize more results on the ablation of the time-aware module in Figure 12.

[W3] Adding more real-world image tests could improve the robustness of the results. Further comparative experiments with text-guided editing methods would also strengthen the analysis

Just as the reviewer is concerned about the application of our method in real image scenarios and its comparison with text-guided methods, we have not overlooked these aspects either. Figure 11 demonstrates the application of our method on real images, where both local and global edits can extract the editing effects from reference image pairs and apply them to new images, further highlighting the generalization capability of our method. Figure 13 shows a comparison of results between our method and text-based editing methods. Text-based editing methods have failed in extracting the editing effects between reference image pairs, further demonstrating the significance of our method's design.

[Q1] Ablation results regarding the degree of truncation would be insightful. To what extent does truncation impact the final results?

Thank you for your clarification. We conduct an ablation study on truncation to explore its contribution to the overall method. As shown in Figure 14, we visualized the initialized instructions and editing results with and without truncation (column 5-6). The results indicate that without truncation, incorrect semantic priors are introduced, hindering the learning of the desired editing effects, e.g. edit in the first line to get red glasses.

[Q2] In certain cases, IP2P+GPT-4 appears to deliver superior results compared to other methods. Further insights into this comparison would be valuable.

Thank you for the constructive suggestions. In some instances, the editing results of IP2P outperform Analogist as well as Visii in terms of image quality, and we analyze the reasons as follows, and add it in line 415 of the paper. IP2P+GPT-4 employs text instructions to guide image editing. Although this text-based editing approach cannot accurately extract the editing concepts between image pairs, the generalization ability of text and the model priors of IP2P ensure the quality of the generated images.

In contrast, Analogist leverages the priors of the inpainting diffusion model, and compared to IP2P, it has a lesser understanding of the editing instructions provided by GPT-4. Additionally, the extra structural constraints imposed on the attention further exacerbate its lower adherence to the instructions. For example, suboptimal results are observed in the local edits from row 1 to row 3 in Figure 3.

Visii optimizes text tokens to specifically learn the editing concepts between image pairs. However, its initialization method introduces excessive content information unrelated to the edits from the reference image, causing content leakage, such as the leaves in row 2 and the vehicles in the background of the row 3 in Figure 3.

In contrast, our method can more accurately extract the editing concepts between reference image pairs, as reflected in our results being more consistent with the ground truth.

[Q3] If unique phrase extraction is conducted effectively, is it possible to sample images based on these conditions? It would be interesting to see how images generated from unique phrases align with actual concepts. If these generated images closely resemble the intended concepts, it would enhance the persuasiveness of the paper’s claims.

Thank you for your thought-provoking questions. We believe that relying solely on textual conditions may not fully accomplish this task. For simple image transformations such as cat-to-dog or peach-to-apple, ideal unique phrases can capture the concept to the maximum extent. However, for pixel-level or even mask-level transformations, such as tone editing, different instances may require different transformations at the pixel level. For such tasks, it is challenging to guide editing through textual conditions, so employing subsequent optimization methods to learn these abstract changes is necessary. This also explains why we optimize instructions in the attention feature space rather than the token space of text input, as it can fully represent pixel-level information.

Additionally, the reviewer's suggestion to visualize the editing results of unique phrases is highly constructive. We have consequently shown several sets of transformation effects in Figure 14, along with the extracted unique phrases and the results of editing using these unique phrases, to verify the effectiveness of this module.

Dear reviewer 6UAg,

Given the discussion phase is quickly passing, we want to know if our response resolves your concerns. If you have any further questions, we are more than happy to discuss them. Thanks again for your valuable suggestions!

Best, All anonymous authors

[Q1] The notation "CAP_x" is used but not previously defined.

Thank you for the careful correction. Exactly as the reviewer's judgment, we have corrected the typo error of CAP_x in line 281 as P_x.

[Q2] The Time-aware Instruction method does not appear effective in the Peach-Apple example in Figure 5. Could an ablation study on this be provided?

Thank you for your positive feedback. We believe that the reason why time-aware module did not significantly improve the result of the peach-apple editing in the second row of Figure 5 is that this type of editing is relatively simple. Using other modules are sufficient to achieve this editing effect. The fine-grained face glasses editing in the first row of Figure 5 proves the significance of the time-aware module setting. In addition, we provide quantitative results of the ablation experiment in Table 2, the improvement of the metrics indicates the necessity of the module design. In addition, we provide more qualitative results on the ablation of the time-aware module in Figure 12 of the paper to visualize the effectiveness of this design.

[Q3] It would be beneficial to include an ablation study on the optimization of the initial m tokens compared to optimizing the full set of tokens.

Thank you for your constructive suggestions. We are indeed optimizing the initial m tokens. Because we believe that optimizing all tokens is not conducive to the generalization of the editing instructions to be extracted. We have ablated the impact of optimizing the first m tokens initialized in KV and the impact of optimizing all tokens in Table 4 of the paper. The results show that optimizing the first m tokens, which is our current setting, yields better effects.

Table 4: Extra Ablation Study. We ablate the impact of optimizing the first $m$ tokens initialized in K, V (Ours) and the impact of optimizing all tokens.

Thank you very much for your constructive comments. We addressed you comments and questions and revised the paper in blue color. We summarize the answers below.

[W1] [...] more logical to explore the direct relationship between image features and to encode these editing directions directly [...], the focus should be on translating image differences into descriptive language [...], Existing editing methods appear sufficient for such language-based tasks

1. Our introduction of image pairs to guide editing does not depend entirely on converting image pairs to text. We emphasize that the essence of our method is based on optimization. Our method compares the differences between images pairs through the diffusion process to optimize the editing direction in the instruction space. The introduction of unique phrases only adds semantic information to the optimization process (as shown in Table 2, where the initialization module's impact on the semantic score CLIP-D). Similar methods have also been implemented in [1] and [2]. The method of encoding the differences between image pairs as a signal guided diffusion model is essentially an encoder based method and is not within the scope of our research.

2. We would like to clarify that we indeed only convert the differences between images into natural language to introduce semantic priors related to editing for the optimization. To this end, we have designed the "Transofmation-oriented Instruction Initialization" method, which extracts unique phrases representing the differences between pre- and post-edited images as semantic priors to assist in subsequent instruction optimization.

3. We reiterate that what we aim to solve is not text-based editing tasks, but rather to precisely replicate the editing effects between image pairs. The conversion of natural text related to editing cannot achieve this function, but it can serve as semantic priors to aid in the subsequent optimization process.

[1] Visual Instruction Inversion: Image Editing via Visual Prompting, NIPS 2023

[2] Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion Model, Siggraph 2024

[W2] Some of the qualitative results highlighted in the paper remain suboptimal, especially when compared to IP2P+GPT-4o

We fully understand the concerns of the reviewer, and a good visual result is not only pleasing but also helps to understand the advantages of the method. Compared with IP2P+GPT-4o, our method achieves more consistent editing results with the reference image pairs. As we demonstrated in Figure 1 in the paper, in the first row, we change the entire tone of the input image to be consistent with the reference image. However, IP2P+GPT-4o cannot control the pixel level and even mask level transformations with reference examples. In the second row, the smile of our method is more consistent with the reference image pairs, and although IP2P+GPT-4o also captures this concept, there is a significant difference in the degree of representation compared to the reference image. There are still many similar examples in Figure 3, indicating that compared to IP2P+GPT-4o, our method is more capable of replicating the editing effects of example image pairs.

We fully appreciate the concerns of the reviewer and hope to be provided examples of what the reviewer consider to be suboptimal results, in order to update these results in the latest version and present better visual effects to readers.

Dear reviewer wgFX,

Given the discussion phase is quickly passing, we want to know if our response resolves your concerns. If you have any further questions, we are more than happy to discuss them. Thanks again for your valuable suggestions!

Best, All anonymous authors

Thank you for your responses. I have carefully reviewed all the comments and your replies. Some of my concerns have been addressed. However, as reviewers 6UAg and uD16 have also noted, I remain worried about the technical novelty and performance superiority of this paper. Therefore, I maintain my current score.

Thank you to the reviewer for the positive evaluation and valuable feedback on our work. Firstly, we are honored to have made some progress in the fine-grained editing for visual prompting, and we appreciate your recognition of our efforts.

At the same time, we take the issue of proper citation very seriously and are willing to detail in our response how we plan to improve this aspect. We have conducted a thorough review and verification of this part and found that there are indeed some formulas that were not properly attributed, particularly the formula mentioned on line 760. We have now cited these references to enhance the transparency and academic integrity of our work, which also helps readers better understand the origin and development of these formulas.

We fully understand the importance of citation accuracy in academic research and are committed to strictly adhering to this standard in our revised manuscript. Thank you once again for your review and suggestions, which are crucial for enhancing the quality of our work. If you have any further suggestions or need additional clarification, please feel free to let us know, as we are more than willing to discuss further.