Thank you for your careful review and constructive suggestions. We have modified our paper to rectify your concerns. The changes relevant to your concerns are marked in orange (color will be removed in the final version).

W1. Whether ICLR is a suitable venue

In our method design, we study the feature representation learned by the pre-trained diffusion model to design energy functions. During the editing process, we inject the gradient guidance generated by the energy function into the discrete latent representation of the pre-trained diffusion model, achieving a general image editing framework. Therefore, our method is also a study of the feature representation and latent representation of diffusion models.

In addition, we check the Subject Areas of ICLR, which include applications in vision, audio, speech, language, music, etc. Therefore, we believe that our work is suitable for ICLR.

W2. Writing

Thank you for these constructive suggestions. We have reviewed the relevant sections and made the necessary modifications.

Q1. Compare with self-guidance

Yes, comparing with self-guidance is necessary. Since self-guidance was not open-sourced before the ICLR submission, our original paper did not include a performance comparison with it. Our main difference lies in modeling editing operations. Self-guidance calculates the similarity between the text (corresponding to the object to be edited) and image features, selects regions with high similarity as editing areas, and then applies editing operations. Therefore, the attention in self-guidance is used to locate editing operations. However, this text-guided image editing is coarse. For example: (1) Text can only locate content at the object level, and more accurate local content cannot be expressed in text, making it impossible to be edited with self-guidance. (2) In multi-object scenes, a single text corresponds to multiple objects, making it impossible to use self-guidance to edit a specific object. (3) Text can not provide effective content consistency, leading to deviations between the editing results and the original image. In contrast, our method models editing operations based on the correspondence of intermediate features in the pre-trained diffusion model, which can achieve pixel-to-pixel correspondence, and the editing area is manually defined. Therefore, our method can achieve fine-grained image editing, such as content dragging, which is not supported by self-guidance. In our method, attention is used to inject content consistency guidance, resulting in better consistency between the edited results and the original image.

In Sec.4.1 of the updated version, we add the comparison between our method against Self-Guidance in object moving and appearance replacing. We add more comparisons in Appendix.8. These results demonstrate that our method has higher editing accuracy and content consistency.

Dear reviewer, thanks a lot for your previous constructive comments. We would like to know if our revisions have addressed your concerns? We welcome any discussions and suggestions that will help us further improve this paper.

Thank you for your careful review and constructive suggestions. We have modified our paper to rectify your concerns. The changes relevant to your concerns are marked in purple (color will be removed in the final version).

W1. The problems targeted in this paper.

• drag-style editing is the target task

In this paper, we want to build a general framework to complete drag-style image editing, e.g., object moving, resizing, pasting, content dragging, and appearance replacing. This has not been achieved before.

• feature matching builds the energy guidance as an editing tool without training

In this paper, we choose energy guidance in diffusion models to achieve image editing. Energy guidance is the second term in Eq.2 in our original paper, which combines the generation target y with the diffusion process as q(y|$x_t$). Therefore, the task becomes designing a suitable energy function [1] to measure the distance between the current state x_t and the editing target y. The first generation control design in energy guidance is to control the category of the generation result. It trains a classifier for $x_t$ to measure the distance between the current state $x_t$ and the target category y. However, this type of method requires additional training, and due to the influence of noise, the classification of $x_t$ is challenging. In this paper, we propose taking advantage of the strong semantics in the intermediate features of pre-trained stable diffusion, which transforms the editing target y into the feature matching between the editing starting point and the target point. Our approach eliminates training cost while achieving good editing results.

[1] Diffusion models beat GANs on image synthesis

• memory bank is a container to store information for visual cross-attention

Although feature matching can achieve editing operations, the final result can not maintain consistency with the original image, such as in unedited areas. To maintain consistency, we use the memory bank to collect intermediate information during the image inversion and inject it into the editing process through visual cross-attention. This allows the editing process to fully capture the original image information. In Fig.9 of the original paper, we demonstrate the important role of visual cross-attention in maintaining content consistency.

Therefore, these components are indispensable for our editing framework.

W2. FID compared with DragGAN

This is determined by the base model. DragGAN is designed based on the pre-trained StyleGAN, while our method is based on the pre-trained Stable Diffusion. The generative capability of stable diffusion is significantly better than that of StyleGAN. The results in Fig.8 in the original paper also show that the generative capability of GAN models is limited. For unaligned images, GAN models can produce severe degradation. Even in aligned images, GANs cannot generate a good background, as shown in DragGAN’s results in our paper. In the new version of the paper, we added more discussion on DragGAN's robustness in Appendix.7.

W3. Identity by visual cross-attention

We need to clarify that our method does not require training. Compared to the latest diffusion-based image editing method, DragDiffusion, our method also has performance advantages without the need for training LORA for each image like DragDiffusion. We will continue to improve content consistency in future work.

W4.&Q1. Ablation study of hyper-parameters and guidance time 

Thank you for this useful suggestion. We add an ablation study on the weights ($w_e$, $w_c$) of the image editing ($E_{edit}$) and content consistency ($E_{content}$) energy functions in Appendix 4. Results show that $E_{edit}$ can complete the editing operation with a small $w_e$, and the influence of $w_c$ on $E_{edit}$ is relatively small. Therefore, in our design, we apply a larger weight to $E_{content}$ to better maintain content consistency while achieving the editing goals.

The ablation study on when to introduce gradient guidance shows that it is important to introduce guidance in the early sampling stages. As the sampling iterations progress, the effect of the guidance information becomes weaker. Moreover, introducing guidance information in more time steps helps improve the texture details. Finally, we introduce guidance information in the first 30 steps to balance performance and complexity.

Dear reviewer, thanks a lot for your previous constructive comments. We would like to know if our revisions have addressed your concerns? We welcome any discussions and suggestions that will help us further improve this paper.

Thank you for approving our revision.

To address this issue, we explore the editing guidance of our method and DragDiffusion. Specifically, DragDiffusion selects a latent $z_t$ in the diffusion process as a learnable parameter and optimizes $z_t$ through multiple iterations. However, this optimization method only considers the editing target and ignores the impact on the diffusion process, i.e., it cannot guarantee that the optimized $z_t$ conforms to the current time step t. In contrast, our method is built on score-based gradient guidance to solve $p(z_{t-1}|z_{t}, y)$, where $y$ is the editing target. It has been proved that $p(z_{t-1}|z_{t}, y)\sim N(\mu+\eta g, \Sigma)$, where $\eta g$ is the gradient guidance produce from an energy function $p(y|z_t)$. Therefore, our guidance method is compatible with the diffusion process. The energy function (i.e., feature correspondence) in our method is also proven to be accurate and robust in measuring the feature correspondence. We believe that the difference in guidance methods is one of the reasons for the performance difference.

We update our paper and demonstrate a comparison of the two editing guidance methods in Appendix 13. We show the editing effects of these two guidance methods without their content consistency design (LORA and visual cross-attention). Retaining only DDIM inversion as the generation prior of the source image. The results demonstrate that the latent optimization in DragDiffusion has a deviation impact on diffusion sampling.

Thank you for your careful review and constructive suggestions. We have modified our paper to rectify your concerns. The changes relevant to your concerns are marked in green (color will be removed in the final version).

W1. The clarity of how the memory bank is meaningful

We need to clarify that the memory bank is a container we designed to store intermediate information ($z_t$, $K_t$, $V_t$) during the DDIM inversion process. These intermediate outputs will be reused to guide the subsequent editing process. In Fig.9 of the original paper, we demonstrated the importance of $K_t$ and $V_t$ (i.e., visual cross-attention) in the memory bank. We also showed the importance of generating content consistency gradients $E_{content}$ through the stored $z_t$. This intermediate information can also be generated in other ways, such as a separate generation branch. But this would result in additional computation cost. In Appendix.2 of the original paper, we conducted an ablation study on the efficiency of the memory bank. However, the connection between these two ablation parts is not tight enough. We added a brief description of Appendix.2 in the ablation study section to highlight the contribution of the memory bank.

W2. How energy design works

Energy guidance is the second term in Eq.2 in our original paper, which combines the generation target y with the diffusion process as q(y|$x_t$). It has been rigorously proven in [1] for generation control in the diffusion models. To make it work, we need to design an energy function to measure the distance between the current state $x_t$ and the editing target y. The first generation control design in energy guidance is to control the category of the generation result. It trains a classifier for $x_t$ to measure the distance between the current state $x_t$ and the target category y. However, this type of method requires additional training, and due to the influence of noise, the classification on $x_t$ is challenging. In this paper, we propose taking advantage of the strong semantics of the intermediate features in pre-trained stable diffusion to transform the editing target y into the feature similarity between the editing starting point and the target point. Our approach eliminates training cost while achieving good editing results. In Fig. 4, we investigate how to make the energy guidance work better. In Fig. 5, we demonstrate how to combine different energy functions to achieve editing goals. In addition, we update more ablation studies in Appendix.3 and Appendix.4, showing the impact of more components on the working of energy guidance.

To make the gradient guidance process clearer, we visualize the gradients generated by the content consistency ($E_{content}$) and editing ($E_{edit}$) energy functions at different time steps in Appendix.5. It presents a gradually converging process, and the gradient generated by each energy function mainly focuses on their respective target areas.

[1] Diffusion models beat GANs on image synthesis

W3. Incomparable with DragGAN

The higher inference complexity is determined by the base model. DragGAN is designed based on StyleGAN, while our method is based on Stable Diffusion. GAN models only require one inference to output results, while diffusion requires multiple iterations, resulting in a higher time complexity. However, the generative capability of the diffusion models is significantly better than that of the GANs. For example, in terms of robustness, DragGAN requires a specific trained model when editing different categories of images, and the input image cannot deviate from the distribution of GAN's training set. In Fig.8 of the original paper, we show that DragGAN's editing results have severe degradation without image alignment. In the new version of the paper, we added more discussion on DragGAN's robustness in Appendix 7. As can be seen, the GAN-based editing method fails in complex scenarios. In contrast, our method is based on the diffusion model with high robustness to edit general images. It is worth noting that the complexity of our method in the preparation stage is significantly lower than that of DragGAN (3.62s vs 52.40s in Tab.1).

Most importantly, our DragonDiffusion is a general image editing method that can accomplish a range of image editing tasks, e.g., object moving, resizing, pasting, content dragging, and appearance replacing. In contrast, DragGAN is limited to content dragging.

Dear reviewer, thanks a lot for your previous constructive comments. We would like to know if our revisions have addressed your concerns? We welcome any discussions and suggestions that will help us further improve this paper.

Dear Reviewer Fbvf,

Thank you again for your great efforts and valuable suggestions. We have carefully addressed your concerns. We hope you are satisfied with our response. As the discussion phase is about to close, we are looking forward to hearing from you about any further feedback. We will be happy to clarify further concerns (if any).

Best,

Authors

Thank you for your careful review and constructive suggestions. We have modified our paper to rectify your concerns. The changes relevant to your concerns are marked in blue (color will be removed in the final version).

W1. More challenging and diverse qualitative samples

Thanks for this useful suggestion. We add more content dragging results in complex scenarios in Appendix.7. These results further illustrate that

• The generalization ability of DragGAN is limited by the GANs. Each trained model can only edit a specific category of images, such as face, elephant, and car. Moreover, the input image cannot deviate (e.g., unaligned faces, multiple faces or cars) from the training distribution of the corresponding GAN model. Otherwise, obvious degradation will occur.
• Our method performs better than DragDiffusion without the need for training. To analyze the performance differences, we further discuss the differences in guidance methods between our approach (score-based guidance) and DragDiffusion (latent optimization) in Appendix.13. The results show that treating $z_t$ as a learnable variable in DragDiffusion can affect the original iteration distribution of the diffusion model.

The most important is that both DragGAN and DragDiffusion are designed specifically for content dragging. Our method is a general editing framework, not limited to content dragging. Due to the previous version of the paper focusing too much on content dragging in the experiment, narrowing down this contribution (as pointed out by R1). We have reformulated the experiment section, demonstrating the editing capabilities for various tasks, further highlighting that our approach is a general and fine-grained image editing framework.

W2. Ablation of $S_{global}$

In our method, $S_{global}$ controls the appearance of a certain area to be consistent with a reference area. It is used in the inpainting of the object moving task and the appearance control in the appearance replacing task. Indeed, the previous version lacked an experiment to show the effectiveness of $S_{global}$. We add an ablation study for $S_{global}$ in Appendix.3, demonstrating its appearance control in object moving and appearance replacing.

W3. Ablation of weights for different energy functions

Yes, there is a trade-off between different energy functions. The energy functions in our method mainly include $E_{edit}$ and $E_{content}$, designed for editing and content consistency, respectively. We add an ablation study for the weights ($w_e$, $w_c$) of these two functions in Appendix.4. Results show that $E_{edit}$ can complete the editing operation with a small w_e, and the influence of $w_c$ on $E_{edit}$ is relatively small. Therefore, in our design, we apply a larger weight to $E_{content}$ to better maintain content consistency while achieving the editing goals. In our demo video, we directly connect the results of different editing operations into a video, which may amplify some subtle jitter.

W4. Identity preservation 

Yes, some details may have some deviations. Our method does not require training, and maintaining high consistency is indeed a challenge. We will continue to improve in future work.

Dear reviewer, thanks a lot for your previous constructive comments. We would like to know if our revisions have addressed your concerns? We welcome any discussions and suggestions that will help us further improve this paper.

Thank you for your careful review and constructive suggestions. To rectify your concerns, we have modulated our paper. The changes relevant to your concerns are marked in red (color will be removed in the final version).

W1.&W2. More comparisons

Thank you for this suggestion to help us highlight the contributions of this paper. A minor clarification is that Self-Guidance was not open-sourced before the ICLR submission, so the initial version did not include a comparison with it. In our new version, we divide Section 4.1 into two parts: content dragging comparison and comparisons on other tasks. In the updated part, we demonstrate our method against Self-Guidance and Paint-by-example in object moving, appearance replacing, and object pasting tasks. We add more comparisons in Appendix.8.

Moreover, we further discuss the differences between our method and text-guided editing methods in Appendix.9 by comparing with Null-text inversion, InstructPix2Pix, and Self-Guidance. 

• Compare with Self-Guidance. Self-Guidance is a text-guided editing method. Due to the coarse-grained correspondence between text and image content, it can only achieve object-level editing and is difficult to achieve editing objectives in some complex scenes or multi-object scenarios. In addition, Self-Guidance lacks consistency constraints, which leads to deviations between the editing results and the original image. In comparison, the impact of text on our method is minimal, as shown in Appendix.4. It shows that the editing results produced by different texts (relevant or irrelevant) are close and meet the editing requirements.
• Compare with Paint-by-example. Paint-by-example is a learnable object pasting method. We must admit that such learnable methods can produce more natural pasting effects. However, encoding the object image into multiple tokens can disrupt the object's identity, leading to changes in the identity of the editing results. In contrast, our method adds guidance at each diffusion step, resulting in better identity consistency in the editing results.
• Compare with other text-guided methods (e.g., Null-text inversion, InstructPix2Pix). Although these methods can also achieve object modulation, the editing results have no reference and randomly dependent on text description. In addition, similar to Self-Guidance, text has difficulty achieving a good correspondence with images in complex scenes and multi-object scenarios, leading to editing failures. 

It is worth noting that most of these compared methods can only accomplish one task, while we can complete these tasks with a single framework. Through these comparisons, we want to demonstrate that our method is a general fine-grained image editing framework with promising performance.

Another related change is that we remove the original Application section and present the application visualization as the teaser on the first page to reduce redundant content.

W3. Typos

Thanks for the careful review. We have corrected Equation 2.

W4. Limitations

Thanks for this suggestion. We add a discussion of limitations in Appendix.12. It mainly includes that there is still room to further improve the inference speed, and there are some hyperparameters in our training-free pipeline that can affect the editing results.

Dear reviewer, thanks a lot for your previous constructive comments. We would like to know if our revisions have addressed your concerns? We welcome any discussions and suggestions that will help us further improve this paper.