Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models

W1, Q1

We have revised the justification regarding the assumption in Appendix A.1. In the revised version, we have argued after Proposition 2, that the “velocity field”, which the model is supposed to learn, is continuous in $z$ and $t>0$. It implies that, even though the equality $\epsilon_{\theta}(z_t,t,C)=\epsilon_{\theta}(z_{t-1},t-1,C)$ does not hold exactly, one can expect that it holds approximately once one assume $|\alpha_t - \alpha_{t-1}|$ is small.

The reviewer’s concern that the accuracy of the approximation will worsen as the sampling step width increases is reflected in the experimental results in figure 4 left, which show that taking a smaller sampling step width improves the quality of reconstruction.

W2

We appreciate the reviewer for the thoughtful proposal. As the reviewer assumed, we have shown the results of DDIM inversion ($w=1$) followed by DDIM sampling ($w=7.5$) in the manuscript. One can also consider a method of DDIM inversion ($w=7.5$) followed by DDIM sampling ($w=7.5$), but this led to very bad results.

Regarding the proposal to evaluate on the same benchmark as Mokady et al. (2023), we did not conduct that comparison since we could not identify the setting of editing and could not replicate the results of Table 2 in Mokady et al. (2023).

Q2

We appreciate the reviewer for the thoughtful question. As the reviewer suggested, we tried two experiments; One is to optimize a single embedding in null-text inversion, and the other is to initialize the optimized embedding by the prompt embedding $C$. In the former setting, we tried the same method as presented in section F of supplementary materials for Mokady et al. (2023), which is called global null-text inversion. Since we could see that this method could achieve the almost equal reconstruction quality to null-text inversion by taking about 20 times longer than the original, we calculated the cosine similarity of the optimized embedding and the prompt embedding in the same manner as in Appendix A.2. Since we could experiment with only some of the data due to time constraints, although we cannot exactly compare the results, the mean cosine similarity was $0.10$, and this result shows the optimized shared embedding was more similar to the prompt embedding than the optimized embedding in null-text inversion whose similarity have been shown in Appendix A.2, figure 6. We believe that longer optimization will make the optimized embedding more similar to the prompt embedding, however, getting too close to the prompt embedding will degrade the reconstruction quality. In the later setting, it yielded slight improvements in the reconstruction quality but did not beat null-text inversion. We are planning to continue analyzing why this setting cannot beat null-text inversion.

Q3

We appreciate the reviewer for this advice. We have added comparison of LPIPS when the computation time is limited to less than 30 seconds in Appendix C.1, figure 7. Considering realistic waiting time, our method outperformed null-text inversion in the reconstruction quality.

Q4

In text-guided image generation, if a negative-prompt is given, it is input to the model instead of the null-text in CFG calculation. As a result, the generated image diverges from the negative-prompt. Since our inversion takes a similar process in which uses a text prompt instead of the null-text, we named it “negative-prompt inversion”.

W1

We appreciate the reviewer's insightful comments regarding the comparability of our method to DDIM inversion without CFG, specifically in the context of reconstruction. We acknowledge the similarities highlighted; however, our approach distinctly focuses on editing by employing a source prompt instead of the null-text, which marks a significant difference from the standard DDIM framework. To elucidate this distinction more clearly, we have revised the itemization in Section 3.4 on page 6.

W2

We thank the reviewer for this comment. The term of computational cost represents the computational time, whose results have been shown in figure 4. In memory usage, our method uses half the memory of null-text inversion. We have added this description at the end of Section 4.2.

W3

We appropriate the reviewer for this comment. We have removed expressions that were considered redundant and clarified the general idea of our method in the third paragraph of the Introduction section.

W4

We have trimmed and shortened the explanations that we considered unnecessary in the Method section.

Q1

We have included our code in supplementary materials for the evaluation purpose by the reviewers, as described in the reproducibility statement of the manuscript. Additionally, we will publicly release our code if our paper is accepted.

Q2

We thank the reviewer for pointing this out. According to the author guide, we cannot change the abstract except in the camera-ready version, so we are planning to change the corresponding part, “changing the style” to “changing the subject and the style”, if our paper is accepted.

Q3

We thank the reviewer for this question. We have revised the description regarding parallelization in the conclusion section. We assumed parallelization to mean processing several frames simultaneously using multiple GPUs, which improves not the latency but the throughput, not parallelizing the reverse diffusion process which is iterative as the reviewer assumed.

Q4

In response to the reviewer’s comment, we have revisited the author guide for ICLR 2024, which explicitly states that the reproducibility statement should be placed at the end of the main text and before the reference list, and that it will not count toward the page limit. We thus believe that our manuscript does conform to the formatting instructions.

W1

We thank the reviewer for this comment. As the reviewer suggested, we have added results of Imagic for comparison, which shows that our method also outperformed Imagic. We also tried UniTune and prompt tuning inversion, but we failed to replicate the respective papers’ results since the official codes are not released.

W2

CLIP score calculates the similarity between an image and its prompt, and it does not regard how an edited image is similar to the original one. By summarizing the evaluation both via LPIPS and CLIP score, we believe that the editing performance of our method is not below DDIM with CFG.

W3

We thank the reviewer for this comment. We have revised the justification regarding the assumption in Appendix A.1. The assumption of equivalence of null-text inversion features and DDIM inversion trajectory is just for the mathematical induction: One has $z^{*}_T = \bar{z}_T$, and we argue that, assuming that the equivalence holds true at diffusion step $t$, it should hold true at diffusion step $t-1$.

The second hypothesis is related to Q2, so we have answered it in Q2.

Regarding the statement $\alpha_t \approx \alpha_{t-1} \Rightarrow z_t^* \approx z_{t-1}^*$,which the reviewer pointed out, it is justified by (3), since $z_t^*$ is calculated by DDIM inversion. Notice that in (3) one has $z_{t+1} \rightarrow z_t$ as $\alpha_{t+1} \rightarrow \alpha_t$.

Q1

As the reviewer pointed out, it was difficult to tell what the description of “DDIM Inv” represents. The description “DDIM Inv” represents DDIM inversion followed by DDIM + CFG. We have aligned the description in the manuscript. Regarding the disagreement with the CLIP score, we think it as follows: It is true that DDIM inversion fails to generate snow in the fourth row of figure 9, but the other descriptions for a Doberman are correct, which could be leading not to low the CLIP score. In fact, the CLIP score of the image is $27.25$, which is higher than the mean score.

Q2

We thank the reviewer for this question. We have extensively rewritten Appendix A.1. In the revised version, we have argued after Proposition 2, that, even when one considers the sample distribution rather than a single sample, the “velocity field”, which the model is supposed to learn, is continuous in $z$ and $t>0$. It implies that, even though the equality $\epsilon_\theta(z_t^*,t,C)=\epsilon_\theta(z_{t-1}^*,t-1,C)$ does not hold exactly, one can expect that it holds approximately once one assumes $|\alpha_t-\alpha_{t-1}|$ is small.

W1

We thank the reviewer for this comment. The assumption on the predicted noises at adjacent diffusion steps, which the reviewer pointed out, has been argued in Appendix A.1. The model is supposed to learn the “velocity field”, which is shown in Proposition 1 in the revised manuscript to be continuous in $z$ and $t$. So the assumption that the predictions at adjacent diffusion steps are approximately equal holds provided that one takes small diffusion steps. Additionally, we have revised the justification regarding the assumption in Appendix A.1.

W2

We appreciate the reviewer for this comment. We evaluated our method by using 100 images in the same manner as in the original paper of Null-text Inversion (Mokady et al. CVPR2023). Regarding human subjective evaluation, we did not conduct it since it was difficult for us to prepare it and to collect dozens of subjects during the short discussion period. We think that several editing images in the manuscript including those in appendices should be helpful for judging our editing quality.

W3

We thank the reviewer for this comment. We have provided some failure cases, including an instance of face reconstruction, in Appendix C.4, figure 12. Please refer to this section.