Precise Diffusion Inversion: Towards Novel Samples and Few-Step Models

We sincerely thank the reviewer SWAE for their thoughtful feedback. The reviewer’s comments helped us identify important issues and improve the clarity and rigor of the paper.

---

Q1: The proposed method should be described as a test-time algorithm rather than a learning framework. The manuscript should clarify this distinction and include pseudocode for the final algorithm.

A1: Thank you for pointing out the misrepresentation, and we apologize for any confusion. The proposed method is indeed a test-time optimization algorithm, not a conventional learning framework. No model parameters are trained; instead, we perform gradient-based optimization for each test instance. We will revise the abstract, introduction, and conclusion to remove references to a "general learning framework" and clarify that our approach is a test-time procedure. Additionally, we will try to move the pseudocode for the DDIM-based version from Appendix B.2 to follow Eq. (18) in the main text.

Q2: In the experiments there is a convergence threshold $\eta$, which is not discussed before. Please describe how the threshold is used during optimization.

A2: We apologize for the lack of clarity. The convergence threshold $\eta$ sets an upper bound on the local reconstruction error for each subproblem $P(t)$ during optimization. Specifically, when the reconstruction error falls below $\eta$, we terminate the optimization for that subproblem and move on to the next. We will clarify this in both the method and experimental sections of the revised manuscript.

Q3: Table 2 & 3: Why does smaller T lead to better results?

A3: Great observation! As explained in A2, each subproblem is optimized until the local reconstruction error reaches the threshold $\eta$. When T increases, local errors accumulate, leading to higher global error and lower reconstruction quality. However, as shown in the table below, the error grows sublinearly with T, due to the Laplacian continuity of the diffusion latent space. From an efficiency standpoint, smaller T reduces the total number of optimization problems, speeding up inference. This underscores the core advantage of our method: accurate inversion with fewer steps, enabling both high-quality and fast real image reconstruction.

Q4: Is there a trade-off of quality versus number of steps T?

A4: Thank you for the question. This trade-off is analyzed in Appendix C.2 (Lines 372–375). As shown in Table 5, with a fixed $\eta$, increasing T initially improves reconstruction quality and reduces inference time. However, beyond a certain point, both quality and efficiency decline. This reflects the trade-off: while larger T reduces initialization errors for each subproblem, it also increases the risk of error accumulation across steps.

Q5: In Bottom up computation it is stated that without loss of generality, we adopt the DDIM sampling strategy. Would it be possible to use the DDPM strategy (Eq 6), and would that be easier or more difficult due to the additional noise term?

A5: Thanks for the question! We have extended PreciseInv to support both the DDPM sampler and the Euler Discrete Sampler of flow matching. In the DDPM setting, the additional noise term in Eq. (6) is treated as a learnable variable and optimized during inference. Formal derivations for both extensions are provided in Appendix A.1, and we report quantitative results on a COCO 2017 subset in Table 4 of Appendix C.2. A direct comparison between DDIM and DDPM samplers using the SD v1.4 model is shown in the table below. The results demonstrate that our method converges reliably with both samplers and achieves comparable reconstruction quality. Interestingly, inference with DDPM is slightly faster, likely due to the added flexibility of the parameterized noise in fitting the generative trajectory. We will clarify this in the revised manuscript.

Q6: Table 2: What is the value for T?

A6: We apologize for the confusion. In Table 2, we used the optimal number of inference steps T for each method to ensure a fair comparison in reconstruction quality. As detailed in Appendix B.2: DDIM Inversion uses T=1000; ReNoise and BDIA use T=50; EasyInv and BELM use T=100; and both EDICT and our PreciseInv use T=2. We will clarify this setting in the main text in the revised version.

Q7: In Figure 2, there are two $x_t$ depicted. One of them should be $x_{t+1}$.

A7: Thank you for pointing this out. We will fix the notation error in Figure 2 in the revised version.

Q8: Please provide more details and insight on the optimization implementation: how is the threshold used, what is the impact of the value of $\eta$, how do the obtained latents look like, and is it really easy to use a different sampler?

A8: Sorry for the unclear. For details on the use and impact of the convergence threshold $\eta$, please refer to A2. In addition, our method is sampler-agnostic and can be easily adapted to DDPM and other samplers, as shown in A5.

Q9: I would like to get more insights in the optimization, how many steps (on average) are needed to reach the threshold? Is the resulting noisy image indeed a sample from a N(0,1) Gaussian? Is there a clear tradeoff between the used threshold, the number of steps T, and the image reconstruction quality.

A9: Thank you for the insightful question. We computed the KL divergence between the optimized $x_T$ from 100 randomly sampled test images and 100 samples from a standard Gaussian $N(0,1)$. The divergence was below $10^{−2}$, indicating that the optimized latents are effectively drawn from a Gaussian distribution. We will also include visualizations of the optimized latents in the appendix in our revised version. Please refer to A4 for the discussion on the tradeoff between parameters and reconstruction outcomes.

We sincerely thank reviewer 48AE for the critical and constructive feedback! Below, we provide point-by-point responses to the specific questions and concerns raised, and we hope these clarifications adequately address the issues.

---

Q1: The paper should avoid presenting the method as a learning-based inversion approach, as it primarily functions at inference time. In addition, the denoising backbone appears overly large, which may result in suboptimal efficiency for the inversion task.

A1: Thank you for pointing this out, and sorry for the confusion. Our method is indeed a test-time optimization approach, not a conventional learning-based method. We will revise the manuscript to avoid potentially misleading terms like "learning framework" and clarify that no offline training is involved.

Regarding the backbone, we agree it may be overparameterized for simple reconstructions. However, we intentionally retain the original denoiser (e.g., SDXL) to ensure faithful alignment with the model’s generative process. This design choice reflects the flexibility of our framework, it can readily accommodate lighter or customized backbones if efficiency is prioritized.

However, we also want to emphasize that our contributions remain despite the imprecise phrasing, i.e., (i) a general framework for diffusion inversion based on sequential subproblem solving; (ii) a dynamic programming-inspired strategy for efficient optimization; (iii) state-of-the-art reconstruction quality; and (iv) latent inference that preserves semantic continuity in the diffusion space.

Q2: The analysis and experimental results on image editing is insufficient.

A2: Thank you for the critical comments. While our method is designed as a general-purpose diffusion inversion framework (as noted in Lines 219–224) rather than a dedicated image editing algorithm, we agree that standardized evaluation on editing tasks is valuable.

To address this, we now include quantitative results on the PIE-Bench subset (as suggested by reviewer 4z49) using the SD v1.4 model. As shown in the table below, our method significantly outperforms baselines in structural consistency, background preservation, and CLIP score, consistent with the qualitative results in Figure 4.

Q3: According to l29-32, backward optimization methods improve reconstruction quality by following the reverse diffusion trajectory. However, l179-180 claim these methods are not designed for accurate reconstruction and are excluded from comparison. Isn't this contradictory?

A3: Thank you for pointing this out. We apologize for the confusion. Lines 179–180 aim to clarify that backward optimization methods do not reconstruct the original generative trajectory of the diffusion model. Instead, they rely on recording the forward noising process and then manually adjusting the denoising steps to match the target image, thus effectively bypassing the need for optimization.

While this enables accurate reconstructions in some cases, it deviates from the generative process and limits generalizability. Therefore, consistent with recent works focusing on forward optimization [a–d], we exclude these methods from direct comparison in our reconstruction experiments.

[a] ReNoise: Real image inversion through iterative noising. ECCV, 2024.

[b] EASYINV: Toward fast and better DDIM inversion. ArXiv, 2025.

[c] Lightning-fast image inversion and editing for text-to-image diffusion models. ICLR, 2025.

[d] Fixed-point inversion for text-to-image diffusion models. ArXiv, 2023.

Q4: What does the threshold $\eta$ in l186 represent?

A4: Apologies for the confusion. The threshold η defines the stopping criterion for each subproblem P(t): once the reconstruction error falls below η, optimization proceeds to the next step. It provides a flexible trade-off between accuracy and inference time. We will clarify this in the main text.

Q5: In Table 1 and Table 2, are the number of inversion steps for the proposed method and the baseline algorithms the same?

A5: Sorry for the unclear. As described in Appendix B.2 (Lines 333–339), we tuned the number of inference steps T and other hyperparameters for each baseline on 100 randomly sampled COCO images to ensure optimal reconstruction. The same settings were then applied across all datasets to avoid overfitting, ensuring a fair and reliable comparison.

Q6: Gradient-based optimization typically requires multiple iterations. Why, then, is it faster than numerical solvers?

A6: Good observation, thank you for the insightful feedback. As shown in Tables 1 and 2, our method achieves comparable reconstruction quality with significantly lower inference time. This efficiency comes from two key factors: (i) our approach uses far fewer diffusion steps (e.g., $T=2$) compared to the dozens or hundreds used by baseline methods; and (ii) gradient-based optimization leverages GPU acceleration through modern deep learning frameworks. To address your concern directly, we provide additional results showing that with $\eta=10^{-2}$ and $T=2$, the local error drops smoothly and meets the threshold within just 50 steps, requiring only 3.61 seconds in total. We will clarify this in the revised version.

Q7: In Sec. 3.3 on prompt-driven image editing, what is the rationale for denoising from the inverted latent using Eq. 21? Is there any theoretical justification for Eq. 21? Why is denoising not performing using ?

A7: Thank you for the thoughtful question. Following prior work [a], we adopt the standard inversion-then-generation paradigm without altering the editing pipeline. In Eq. (21), we reuse the inverted latent $x_T^*$, swapping the prompt roles—using the target as the positive prompt and the source as the negative, to steer generation away from the original semantics. This heuristic is widely used in existing methods. We also tested removing classifier-free guidance (CFG) during inversion (Eq. 19) and observed similar results, suggesting that CFG primarily influences sampling rather than inversion.

[a] ReNoise: Real image inversion through iterative noising. ECCV, 2024.

Q8: In Figure 5, what prompt was used for denoising the interpolated latent ? Were interpolation-aware prompts also used in the baseline methods for a fair comparison?

A8: Thank you for the question. In Figure 5, we used an empty string as the prompt, and all methods used the same prompt to ensure a fair comparison. To further validate this, we also tested with a category prompt like “a photo of a car” and observed consistent results, i.e., our method produces an $x_T$ that lies in a smoother semantic region of the latent space. Prior work [e] shows that diffusion latent spaces are semantically continuous. By accurately recovering the model’s native generative trajectory, our approach yields latent representations that better align with this structure.

[e] Unsupervised representation learning from pre-trained diffusion probabilistic models. NeurIPS, 2022.

I appreciate the author's detailed response. The rebuttal helped me clearly understand that the proposed method follows an optimization-based inversion approach. I believe that inversion is inherently tied to tasks such as image editing and manipulation. If the sole objective were to reconstruct a given image, there would be no need to go through the inversion and re-generation process, since the image is already available. From this perspective, I find it valuable contribution that the authors validated the effectiveness and superiority of their method on PIE-Bench, as suggested by Reviewer 4z49. Although the response could not include visual examples due to the policy constraints, it would be beneficial to include representative PIE-Bench examples in the manuscript.

Aside from the initial mischaracterization as a learning-based approach, I do recognize originality in the proposed method. However, as pointed by Reviewer 4z49, the absence of formal theoretical guarantees, such as a convergence proof or error bounds, seems to remain a significant limitation. Including such mathematical analysis could help the technical soundness of the work and align it more closely with the standards typically expected at NeurIPS.

We sincerely thank reviewer v5Ud for the insightful feedback! The suggestions have helped us further clarify key aspects of the work.

---

Q1: When adjusting the convergence threshold, the computation time seems to increase quite a lot, especially in the case of SDXL. The speed-quality tradeoff needs to be discussed.

A1: We thank the reviewer for the helpful suggestion. We, indeed, have analyzed the speed–quality tradeoff under different convergence thresholds $\eta$ in Table 5 (Appendix C.1). Empirically, smaller $\eta$ leads to better reconstruction quality (lower LPIPS, higher SSIM/PSNR) but increases inference time due to stricter convergence.

We also observe that this effect is more pronounced for large-scale models such as SDXL, as noted by the reviewer. This is expected, since SDXL has a deeper denoising UNet and a larger latent space, which increases the per-iteration computation cost. However, PreciseInv provides a flexible tradeoff mechanism: users can adjust $\eta$ based on available computational budget. For example, using a relatively loose threshold (e.g., $\eta = 10^{-2}$ with T=2), PreciseInv still achieves competitive reconstruction performance on SDXL while being significantly faster than optimization-based baselines.

Q2: The method seems to be quite slower for SD3, which may suggest that the computation time saving might not scale to larger models such as SD3 or FLUX, which can limit the applicability. A comparison against baseline methods for SD3 or FLUX might also be necessary.

A2: Thank you for pointing this out. We acknowledge that inference time on SD3 is longer than on SDv1.4 or SDXL due to its larger model size. However, accurately inverting the generative trajectory of such a large-scale model is inherently challenging, and to the best of our knowledge, our method is the first to achieve this effectively.

Since our method belongs to the forward optimization family, it is generally not compared with backward correction-based approaches like [a, b]. While method [c] also follows a forward approach, its implementation is only available for SDXL-Turbo, not SD3 or Flux.

Thus, we compare against FPI on SD3 with carefully tuned settings. As shown in the following table, although FPI is faster, our method significantly outperforms it in LPIPS, SSIM, and PSNR. Qualitative results further reveal that FPI reconstructions are often blurry or semantically inconsistent. We will include these results and discussions in the revised version.

[a] Stable Flow: Vital Layers for Training-Free Image Editing. CVPR, 2025.

[b] Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editing. CVPR, 2025.

[c] Lightning-fast image inversion and editing for text-to-image diffusion models. ICLR, 2025.

Q3: Can the method be used with shortcut models as well?

A3: Good spot! Yes, PreciseInv can be naturally extended to Shortcut models [d], as both SD3 and Shortcut adopt the flow-matching paradigm. Our method does not rely on any specific architecture; it only requires a differentiable and continuous mapping from $\mathbf{x} _ t$ to $\mathbf{x}_{t-\Delta}$, which remains valid in both SD3 and Shortcut. Although Shortcut introduces a step-size indicator $\Delta$ and reduces parameter count, it preserves key properties such as Lipschitz continuity of the velocity field. Since PreciseInv is already formally defined and empirically validated on SD3, it can be directly applied to Shortcut without modification. We plan to support this extension in our public code release.

[d] One Step Diffusion via Shortcut Models. ICLR, 2025.

Q4: It would be interesting to know a more detailed reasoning behind why the proposed method results in a smoother interpolation between latents.

Thank you for the insightful question. Prior work [e] has demonstrated that the latent space of diffusion models exhibits inherent semantic continuity. Our method does not introduce any explicit regularization or interpolation constraints. Instead, it accurately reconstructs the model’s own generative trajectory, allowing the inferred latent $\mathbf{x} _ T$to reside naturally within a semantically coherent region. This results in significantly smoother interpolation between latents.

We further conducted empirical analysis and observed that this continuity deteriorates as the reconstruction precision decreases. In particular, when the convergence threshold $\eta$ exceeds approximately 0.025, the interpolated results begin to show noticeable semantic discontinuities. This observation underscores the importance of precise inversion in preserving the smoothness and structure of the semantic latent space. However, the deeper mechanism behind this phenomenon remains an open question and is an exciting direction for future exploration.

[e] Unsupervised representation learning from pre-trained diffusion probabilistic models. NeurIPS, 2022.

Our sincere thanks to Reviewer 4z49 for their thoughtful and insightful comments. We've addressed each of the specific questions and concerns they brought up in our detailed, point-by-point responses below.

---

Q1: The theoretical analysis is missing. Although the proposed method is straightforward, the analysis of convergence or approximation error in the optimization should be provided.

A1: Thank you for the suggestion. While our method is primarily empirical, we provide convergence-related evidence through additional experiments shown in Table 1 and Table 2. Table 1 illustrates the local reconstruction error across iterations for each subproblem $P(t)$, showing smooth convergence. This behavior stems from the Lipschitz continuity of the diffusion latent space, which facilitates stable gradient-based optimization. Table 2 analyzes the global reconstruction error over T subproblems under varying upper bounds $\eta$ on local error. Notably, global error does not accumulate linearly with T. Moreover, fixing T and progressively reducing $\eta$ leads to a converged global error around 0.0019, which reflects the intrinsic reconstruction limit of the VAE in latent diffusion. These findings demonstrate that our method enjoys stable convergence and favorable approximation accuracy under varying inference steps T. We will incorporate these results and discussions into our revised version.

Table 1: Local reconstruction error at each optimization step in subproblem $P(t)$. Results are averaged over 32 images from the COCO 2017 validation set.

Table 2: Global accumulated reconstruction error over $T$ subproblems. The hyperparameter $\eta$ defines the error bound for each subproblem $P(t)$: if and only if the local reconstruction error drops below $\eta$, the optimization moves to the next subproblem $P(t+1)$.

Q2: Could the author provide a quantitative comparison in image editing? PIE-Benchmark is a good choice.

A2: Thank you for the construction comments. Following your suggestion, we provide additional quantitative results on PIE-Bench [a] in Table 3, where PreciseInv significantly outperforms baseline methods in terms of structural consistency, background preservation, and instruction fidelity. This improvement stems from our accurate reconstruction of the original generative trajectory, which retains more information from the source image $\mathbf{x}_0$ and produces a latent $\mathbf{x}_T$ situated in a smoother, semantically meaningful space. These quantitative results are also consistent with the qualitative comparisons shown in Fig. 4.

[a] PNP Inversion: Boosting diffusion-based editing with 3 lines of code. ICLR, 2024.

Table 3: Quantitative results on the PIE-Bech subset. For each category, we randomly sample 10 test instances. We adopt Stable Diffusion v1.4 as the base model for all methods.

Q3: How sensitive is the method to the choice of the convergence threshold $\eta$?

A3: Thank you for the question. In fact, we have thoroughly analyzed the sensitivity of our method to the convergence threshold $\eta$, as shown in Table 5 of Appendix C.3 in the original manuscript. We observe that lowering $\eta$ from $10^{-2}$ to $10^{-5}$ consistently improves reconstruction quality across metrics such as LPIPS, SSIM, and PSNR. However, further tightening (e.g., to $10^{-6}$ or $10^{-7}$) leads to diminishing returns, especially when the number of inference steps T is small.

In terms of efficiency, smaller $\eta$ values do increase inference time. Nevertheless, thanks to the smoothness of the latent space and our progressive optimization strategy, the runtime remains practical (e.g., 5.33s per image at $\eta = 10^{-5}$) and becomes very fast under relaxed settings like $\eta = 10^{-2}$.

Crucially, PreciseInv remains robust across a wide range of $\eta$ values, outperforming all baselines in both accuracy and speed even with loose thresholds. This indicates that the method does not require fine-grained tuning to deliver strong and reliable results.

Thank you for your constructive comments. After a deeper investigation of this issue, we have now developed a more formal framework for analyzing convergence behavior and bounding reconstruction error. We hope this provides a clearer theoretical perspective and better addresses your concern.

We will include the formal proof outlined below in our revised version.

---

Problem Setup

We formulate diffusion inversion as an optimization problem over a parameterized noise sequence $\\{\epsilon_ t\\}_{t=1}^T$, aiming to minimize the reconstruction error:

$\min _ {\\{\epsilon_t\\}} \mathcal{L} := \||x_0 - \hat{x} _ 0(\epsilon_{1:T})\||^2$

The image reconstruction $\hat{x}_0$ is obtained by applying the backward denoising steps of a pre-trained diffusion model:

$x_{t-1} = f_\theta(x_t, \epsilon_t), \quad t = T, \ldots, 1$

We further decompose this into local subproblems using a dynamic programming-style recursive strategy, optimizing $\epsilon_t$ sequentially to reduce local errors:

$\mathcal{L} _ t := \||x _ {t-1} - f_\theta (x_t, \epsilon_t)\||^2$

Assumptions

To enable convergence and error analysis, we introduce the following assumptions:

• (A1) Lipschitz Continuity of the Model: The denoising function $f_\theta(x_t, \epsilon_t)$ is Lipschitz continuous in $\epsilon_t$ with constant $L_m > 0$, i.e., $\||f_\theta(x_t, \epsilon) - f_\theta(x_t, \epsilon')\|| \leq L_m \cdot \||\epsilon - \epsilon'\|| \quad \forall \epsilon, \epsilon'$
• (A2) Lipschitz Smoothness of the Loss Function: The local reconstruction loss $\mathcal{L}_t(\epsilon_t)$ has $L$-Lipschitz continuous gradients with respect to $\epsilon_t$, i.e., $\||\nabla \mathcal{L}_t(\epsilon) - \nabla \mathcal{L}_t(\epsilon')\|| \leq L \cdot \||\epsilon - \epsilon'\|| \quad \forall \epsilon, \epsilon'$ This constant $L$ governs the smoothness of the loss landscape and is crucial for step-size selection in gradient-based optimization.
• (A3) Locally Bounded Gradients: Within a compact neighborhood $\mathcal{B}_r(\epsilon_t^{(0)})$ around the initialization of $\epsilon_t$, the gradient is uniformly bounded: $\||\nabla \mathcal{L}_t(\epsilon_t)\|| \leq G, \quad \forall \epsilon_t \in \mathcal{B}_r(\epsilon_t^{(0)})$ This assumption reflects empirical behavior in diffusion models, where network normalization (i.e., denoising backbones UNet or Transformer commonly use BatchNorm, GroupNorm or LayerNorm, which control the scale of intermediate activations), bounded input noise (i.e., inversion typically starts with $\epsilon_t \sim \mathcal{N}(0, I)$, and optimization trajectories remain within a moderate neighborhood), and the smoothness of the latent space (i.e., diffusion latent space exhibits Lipschitz continuity) ensure stable gradients during inference. These properties collectively ensure that gradient norms do not explode and remain bounded throughout optimization.

Convergence Analysis

Under the above assumptions (i.e., each denoising function $f_\theta$ is Lipschitz continuous in $\epsilon_t$ with constant $L_t$, and the loss functions $\mathcal{L}_t$ are differentiable with bounded gradients), standard results from non-convex optimization apply [a]. Specifically, using gradient descent with a fixed learning rate $\eta < \frac{2}{L}$, we can obtain:

• Each local optimization subproblem decreases its objective value monotonically.
• After $K$ gradient steps, the minimum gradient norm satisfies: $\min_{k \in [1, K]} \||\nabla \mathcal{L}_t^{(k)}\||^2 \leq \mathcal{O}\left(\frac{1}{K}\right)$ indicating convergence to a stationary point.
• Since each step optimizes a local reconstruction loss, and the overall optimization proceeds recursively, the global loss $\mathcal{L}$ is non-increasing across steps.

Error Bound Analysis

Let the estimation error at each step satisfy $\||\epsilon_t - \hat{\epsilon} _ t\|| \leq \delta_t$, and assume the mapping $x_{t-1} = f_\theta(x_t, \epsilon_t)$ is Lipschitz in $\epsilon_t$. Then the accumulated reconstruction error satisfies (following A1): $\||x _ 0 - \hat{x}_0\|| \leq \sum _ {t=1}^T \left( \prod _ {j=1}^{t-1} L_m^{(j)} \right) \cdot \delta_t$ If $L_m^{(j)} \leq L_m < 1$ (satisfied due to Lipschitz continuity in diffusion latent space, and empirical observations in Table 1 in our initial response) and $\delta_t \leq \delta$ for all $t$, we obtain a geometric upper bound:

$\||x_0 - \hat{x}_0\|| \leq \delta \cdot \sum _ {t=1}^T L_m^{t-1} \leq \frac{\delta}{1 - L_m}$

This aligns with our empirical results (see our initial response Table 2), where the reconstruction error saturates and does not grow linearly with $T$.

Reference

[a] Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization 2013.

Detailed proof of the geometric upper bound:

Assume that at each subproblem $t$, the local reconstruction error is bounded by $\delta_t \leq \delta$. Furthermore, suppose the error propagation factor at each stage is Lipschitz continuous with constant $L_m \in (0, 1)$, i.e., $\||\text{error at step } t\|| \leq L_m \cdot \||\text{error at step } t-1\|| + \delta$ Then, the global error accumulates as follows:

$e_1 = \delta$,

$e_2 \leq L_m \cdot e_1 + \delta = L_m \cdot \delta + \delta$

$e_3 \leq L_m \cdot e_2 + \delta = L_m^2 \cdot \delta + L_m \cdot \delta + \delta$

...

$e_T \leq \delta \cdot \sum_{k=0}^{T-1} L_m^k = \delta \cdot \frac{1 - L_m^T}{1 - L_m} \leq \frac{\delta}{1 - L_m}$

Hence, the final error between the estimated and true initial latent is bounded by $\||x_0 - \hat{x}_0\|| \leq \||e_T\|| \leq \frac{\delta}{1 - L_m}$

Thank you for your prompt and thoughtful questions. We are glad to continue this exchange and have provided detailed responses to each point below. We look forward to your thoughts on our clarifications.

---

1. Do the optimization trajectories remain within this neighborhood throughout inference?

Re1: Yes, the iterates do remain close to their initialization in our setting. Specifically:

• We initialize $\epsilon_t$ from the model’s typical starting guess ( $\epsilon_t \sim \mathcal{N}(0, I)$ ), so the initialization $\epsilon_t^{(0)}$ is already in a region the network was trained on (samples from $\mathcal{N}(0, I)$ or close variants).
• We perform only a small number of gradient steps per subproblem. Empirically, this yields minor changes per iteration, so the iterates stay within a moderate neighborhood of $\epsilon_t^{(0)}$. This follows naturally from the Lipschitz continuity of the model (Assumption A1).
• By setting the step size (learning rate) $\eta'$ to satisfy $\eta' < \frac{2}{L}$ (from the cited work [a]), we can further guarantee that the optimization trajectories remain within this neighborhood.

2. Each $\epsilon_t$ is optimized locally to reduce $\mathcal{L}_t$. How does it guarantee that the overall reconstruction loss $\mathcal{L}$ is minimized?

Re 2: According to the convergence analysis in our proof above, if $f_\theta$ is Lipschitz in $\epsilon_t$ (A1), then a local decrease in $\mathcal{L}_t$ reduces the downstream discrepancy $\\|x_0 - \hat{x}_0\\|$ by at most a Lipschitz factor. Formally, for a small local improvement $\Delta_t$, we have $\\|x_0 - \hat{x}_0\\| \leq C_t \cdot \Delta_t$ where $C_t$ depends on the product of downstream Lipschitz constants $L_m^{(j)}$ ($L_m^{(j)} \leq L_m < 1$ due to Lipschitz continuity in the diffusion latent space). Thus, local descent yields bounded global descent: $\\|x_0 - \hat{x} _ 0\\| \leq \delta \cdot \sum _ {t=1}^T L_m^{t-1} \leq \frac{\delta}{1 - L_m}$ Please refer to our Formal Theoretical Proof - part 2 for more details on this geometric upper bound. Moreover, empirical observations suggest that the overall $\mathcal{L}$ decreases monotonically as subproblems are solved (see Table 2 in our response to Q1).

3. Are these values ($\delta_t$,$L_m$) empirically estimated? Are they consistent across time steps?

Re 3: Yes. These constants can be empirically estimated and, if desired, allowed to vary with timesteps $t$. For convenience, we often adopt conservative uniform bounds (i.e., consistent values across time steps). Specifically, we can

• Estimate $\delta_t$ (local error) by running the local optimization on a validation set and record the reconstruction error $\Delta_t = \\|x_t - \hat{x}_t \\|$ for each $t$. A uniform bound can then be set as $\delta = \max_t \Delta_t$.

• Estimate $L_m^{(j)}$ (local Lipschitz constant) by approximation via finite differences on a validation set: L_m^{(j)} \approx \max_{i} \frac{\left\\| f_{\theta}\left(x_t^{(i)}, \, \epsilon^{(i)} + \Delta\right) - f_{\theta}\left(x_t^{(i)}, \, \epsilon^{(i)} \right) \right\\|} {\\|\Delta\\|} for small random $\Delta$. A conservative bound is then $L_m = \max_j L_m^{(j)}$.

In practice, we enforce a per-step accuracy threshold $\eta$ during subproblem optimization, i.e., we stop optimizing step $t$ once: $\Delta_t := \\|x_t - \hat{x}_t\\| \le \eta$ This simple rule yields several beneficial effects:

• Setting $\delta := \eta$ provides a uniform per-step upper bound, which directly yields the global bound $\\|x_0 - \hat{x}_0\\| \le \delta \sum _ {t=1}^T L_m^{t-1} \le \frac{\delta}{1-L_m}$ under standard Lipschitz assumptions.

• Thresholding empirically keeps iterates in a small neighborhood of the initialization, making the bounded-gradient assumption more realistic.

• As noted in our Q1 response in the initial rebuttal, $\eta$ introduces an explicit accuracy–efficiency trade-off: smaller $\eta$ improves final reconstruction at higher per-step cost, while larger $\eta$ speeds up inversion with a looser bound.