Lightning-Fast Image Inversion and Editing for Text-to-Image Diffusion Models

As the weaknesses were not addressed further in the questions, I have organized additional questions to help strengthen the manuscript. All the additional questions are related to the weaknesses I have written.

3. Norm Application: Could you explain more clearly how your idea of applying norms in the Newton-Raphson approach for diffusion model inversion is different from or improves on existing methods, in terms of novelty? This would help to appeal the novelty.

4. Comparison with Anderson Acceleration: Could you provide a detailed comparison between your guiding regularizer and Anderson acceleration (Pan et al., 2023), focusing on any key differences or improvements. Also, any extra analysis on the instability at $\lambda=0$ would be useful.

5. Memory Usage Analysis: Could you add a comparison of the memory usage of your method versus other approaches (Pan et al., 2023; Hong et al., 2024), with some examples across different GPU sizes?

If these points are addressed, I would consider adjusting the score. Thank you very much for your patience.

5. Typos and duplicate citations.
Thank you for helping us improve this paper. We fixed this in the revised manuscript.

6. There is a slight improvement over existing methods on rare concept generation.
Thank you for your comment. The goal of rare concept generation is to enhance tail performance without compromising performance on the head categories. The results indeed show this trend where our approach demonstrates a significant absolute 3.8% gain in the tail classes (from 85.3% to 89.1%) without hurting head class accuracy. Additionally, our method achieves a substantial reduction in generation time, 10 seconds faster (a 20% improvement) compared to the best baseline. This balance of improved performance in tail classes and enhanced efficiency highlights the strength of our approach.

7. More results for rare concept generation.
Thank you. Further results on rare concept generation can be found in Table 2 and Appendix E.

Thank you for your detailed and insightful feedback. We address all your concerns below.

1. Applying the norm seems standard.
We thank the reviewer for the comment. We do not claim that the novelty of the paper lies in applying the norm of the residual (Eq (7)), but in the following: (1) the novel application of Newton-Raphson for diffusion inversion and (2) the prior added to the objective based on the known distribution $q(z_t|z_{t-1})$ to guide optimization toward in-distribution solutions. Following the comment, we revised the text to clarify applying the norm has been done in previous work, and included the references provided by the reviewer.

2. The guiding regularizer … appears similar to Anderson acceleration, which also aims to prevent large changes from previous estimates.
Thank you for this comment. One key difference is that the guidance term introduced in Eq.(9) does not pull $z_t$ toward $z_{t-1}$ but towards the mean $\mu_t$. By that, it directs the Newton-Raphson method toward solutions having a high likelihood, using “side-information” about the noise. In contrast, Anderson acceleration relies on previous values of the implicit function $f(z_i)$ and residuals but lacks awareness of the latent distribution (see for example, line 13 in Algorithm 1 of Pan et al.).

In GNRI, keeping $z_t$ close to $\mu$, encourages solutions that are more “in-distribution” and aligned with the diffusion model. In other words, if $z_{t-1}$ starts far out-of-distribution, keeping $z_t$ close to $z_{t-1}$ is likely to yield worse solutions compared to guiding $z_t$ toward inputs that the model expects to receive. We illustrate this effect in the table below. We analyzed the inversion of 300 COCO images that the model would consider to be non-typical by selecting the samples with the lowest likelihood under $q(z_0)$ (Eq.(1)). The table shows that GNRI drives the latents toward significantly higher likelihood solutions (mean $q(z_t|z_{t-1})$), leading to substantially better reconstructions (higher PSNR) than Anderson acceleration.
More broadly, our guidance term could potentially be applied to other methods, such as fixed-point iteration approaches like those proposed by Pan et al. It would also be interesting to find methods that can include more domain knowledge into the inversion process.

3. There is a significant drop in performance when λ=0. Add analysis.
Thank you for pointing out this topic and allowing us to make it clearer. Setting $λ=0$ corresponds to the Newton-Raphson method without the guidance term (Section 7). In this case, we observed rapid convergence to solutions that lead to poor image reconstructions (Figure C2). We presume this is because the Newton-Raphson process selects solutions that solve the implicit equation, but are outside the distribution of latents observed when training the model. We provided analysis and details in Section 5.1 and in Appendix D. We’re happy to provide any specific analysis the review may still ask for.

4. Add a comparison of the memory usage of your method versus other approaches.
Following the reviewer’s comment we compared the memory usage of our approach, with AIDI [1] and ExactDPM [2] on an A100 GPU (40 GB VRAM) and an RTX 3090Ti GPU (24GB VRAM). Results are given in the table below. Our approach consumes about the same GPU memory as ExactDPM, however, it converges faster about x43 times than ExactDPM on an A100 GPU, and x25 faster on an RTX 3090 GPU.

[1] Pan et al. (2023) “Effective Real Image Editing with Accelerated Iterative Diffusion Inversion”.
[2] Hong et al. (2024) “On Exact Inversion of DPM-Solvers”.

Thank all the authors for addressing my concerns. I will raise my score 6->8.

Thank you for finding our approach efficient with a notable advancement over existing techniques. We address your comments below.

1. Missing comparison with DDPM Inversion for the inversion performance.
We thank the reviewer for the comment. The table below compares inversion performance with DDPM inversion (Edit Friendly [1]) using SDXL Turbo on the COCO validation set. Edit Friendly achieves the maximum PSNR possible given the VAE distortion, because it does a stochastic DDPM inversion, saving additional noise matrices at every step and ``overfitting” the inverted image. As a result, it requires extensive inversion time (more than 30 seconds), whereas our GNRI reaches a relatively high PSNR in under a second. Our approach solves deterministic inversion (DDIM or Euler) making it suitable for applications like interpolation and rare concept generation, where DDPM is not feasible.

2. Please include the DDPM-Inv result on PIE-Bench for image editing.
The table below compares our approach with EditFriendly [1] on the Pie-bench benchmark. Our method outperforms EditFriendly across all metrics, demonstrating superior editing performance and faster inversion and editing. We added these comparisons to the revised version.

3. Includes more editing results from the PIE-Benchmark.
We provide an additional qualitative comparison of the PIE benchmark in Figure E2 (Appendix F).

[1] Huberman-Spiegelglas et al. (2024) “An Edit Friendly DDPM Noise Space: Inversion and Manipulations”.

Thank you for finding our approach effective, efficient and clearly stated. We kindly address your comments below.

1. How does GNRI match DDIM's efficiency despite the added back-propagation in Equation (10)?
DDIM requires a single UNet pass taking 0.2 seconds per step in Flux.1-Schnell, while GNRI performs 1–2 iterations (≈1.5 on average), taking 0.4 seconds, nearly double DDIM’s time. Fig. 4 visually marginalized this difference due to the axis scale, influenced by slower methods like ExactDPM (15+ seconds). In addition, backpropagation is fast thanks to PyTorch's optimized derivative computation. We clarified in the text that GNRI is 2x slower than DDIM.

2. Could authors provide PSNR vs NFE?
In response to the reviewer’s comment, we provide PSNR vs. NFE results using SDXL Turbo on the COCO validation set. Our method requires 6 NFEs, calculated as 4 steps with an average of 1.5 NR iterations per step (some steps needing 1 iteration, others 2). As shown in the table below, our approach achieves the highest PSNR while using the fewest NFEs compared to other methods.

3. Editing multiple objects.
Thank you. While this is not our main focus, we conducted several experiments that show that our method is capable of editing multiple objects simultaneously. Quantitative examples can be found in Appendix J.

4. Linebreak in Figure 2.
Thank you. We have fixed this in the updated PDF.

[1] Garibi et al. (2024) “ReNoise: Real Image Inversion Through Iterative Noising”. [2] Hong et al. (2024) “On Exact Inversion of DPM-Solvers”.
[3] Wu et al. (2024) "TurboEdit: Instant text-based image editing"

Thank you for your feedback. We are encouraged that you find our approach novel, the methodological foundation to be rigorous and well-defined, with the potential for broad impact. We address your comments below.

1. The concept of FPI is not completely new.
Indeed. The reviewer is absolutely correct that solving the implicit inversion function using FPI approaches has been explored (AIDI [1], ReNoise [2]). We describe and compare these approaches in the paper. This paper is the first to introduce Newton-Raphson's (NR) solution to this problem and add a guidance term that drives solutions toward in-distribution latents. The experiments show that our approach offers significant advantages in terms of both speed and accuracy over these methods.

2. Which FPI baseline is used in Figure 3?
We apologize for the omission, it was plain fixed-point iterations, which is equivalent to AIDI [1] without Anderson Acceleration. We clarified this in the revised PDF.

3. Notational inconsistency in referencing Eq. 5 instead of Eq. 4.
Thank you for catching this! We fixed it in the revised PDF.

4. The method solves an approximation of the original inversion problem rather than the exact roots.
We thank the reviewer for this comment. There is no guarantee that $F(z_t)=0$ has an exact root. Since our objective $\mathcal{F}$ is non-negative, the Newton-Raphson (NR) method effectively tends to minimize the residual, for $\mathcal{F}$, leading to an approximate solution for Eq. (9). In practice, we observed that our Guided Newton Raphson converges to a solution where both $\hat{r}$ in Eq. (7) and $G$ in Eq. (8) reach low values, indicating that these constraints (residual and likelihood) are at the end best satisfied.

5. Why does GNRI, despite its additional regularization term, achieve a better residual than NRI?
Thank you for an insightful observation. Indeed, GNRI reaches a lower residual than NRI despite the addition of the guidance term. Our main hypothesis is that since $F(z)$ has many local minima (likely approximate solutions), running NRI may select a far-from-optimal minimum. Adding the guidance term, which is very smooth, may help optimization escape those local minima. This effect may become stronger when the diffusion model receives z's that are outside its training distribution, since the landscape there may be far less smooth. With the guidance, z is driven to a different part of the space and a distinct minimum (solution) with significantly lower residual values.

6. What is the residual level if you set zt=mu_t(G(zt)=0)?
Setting $z_t=\mu_t(G(zt)=0)$ results in a mean residual of $5 * 10^{-4}$ on COCO val set, with a PSNR of $21.2$. Compare this with applying 2 iterations of GNRI, which results in a residual of $4.2*10^{-5}$ and PSNR of $25.6$. This demonstrates that while $\mu_t$ perfectly satisfies the guidance term, it does not necessarily provide an optimal solution for the residual term and thus results in poor reconstructions. GNRI, however, effectively reaches low values for both the residual and guidance terms, resulting in accurate inversions and high-quality reconstructions.

[1] Pan et al. (2023) “Effective Real Image Editing with Accelerated Iterative Diffusion Inversion”.
[2] Garibi et al. (2024) “ReNoise: Real Image Inversion Through Iterative Noising”.