Latent Refinement via Flow Matching for Training-free Linear Inverse Problem Solving

We sincerely thank the reviewer for their feedback. We address all comments individually below.

1. What are Assumptions on Latent Prior $p(\mathbf{z}_0)$ ?

• (1). Log-concave Prior. Our assumption is first grounded in a bound on Jacobian of vector field $\nabla_{\mathbf{z}_t} \mathbf{v} _\theta(\mathbf{z}_t, t)$ (Revised Proposition 3.3, which holds for any strongly log-concave prior, including Smoothed Laplace, Logistic Distribution, Generalized Gaussian with hight $\beta$, and Sub-Gaussian priors with strong tail curvature, ensuring theoretical robustness even when the prior deviates mildly from exact Gaussianity. We further refer the reviewer to our complementary response to the third question of Reviewer xFYk regarding this point.

• (2). Gaussian prior. Using a Gaussian latent prior is standard practice in models such as Stable Diffusion and LDMs, enabling closed-form sampling and efficient inference. In VAEs, the latent codes are explicitly regularized towards a standard normal prior ($\mathcal{N}(0, \mathbf{I})$) during training, so the aggregate posterior $q(\mathbf{z})$ is encouraged to match this prior—which also serves as the base distribution for score-based generative modeling (e.g., Vahdat et al., 2021). Although the true posterior may be complex, this procedure results in an aggregate latent distribution that is approximately Gaussian, as established in prior work. Importantly, if the latent prior were highly non-Gaussian, reliable sampling from the autoencoder (and thus from any latent generative model) would become fundamentally challenging. Since standard practice in VAEs and latent diffusion models is to sample $\mathbf{z}_0 \sim \mathcal{N}(0, \mathbf{I})$ for generation, a significant mismatch would lead to unrealistic or out-of-distribution decodings—impacting all approaches relying on pretrained autoencoders, not just ours. Thus, our assumption is both practical and broadly consistent with the requirements of latent generative modeling. Furthermore, our assumption is significantly less restrictive than the Gaussian assumption made by $\Pi$GDM on the data itself (i.e., $p(\mathbf{x}_0) \sim \mathcal{N}(0, \boldsymbol{\Sigma}))$. This is shown in proof A.7.

• (3). Remark. In this remark (please see response 3 to xFYk), we show that the Gaussian prior is a tight instance of this bound (The Gaussian prior makes the inequality in the bound turn into an equality), justifying the closed-form posterior covariance in Eq. 17.

• Posterior Covariance in pixel space. Under Gaussian data assumption, an analogous posterior covariance can be derived in pixel space, making the Optimal Vector Field applicable there as well. However, the latent space formulation is much more natural and theoretically justified: the latent Gaussian prior is better supported (e.g., by VAE training), and the reduced dimensionality enables more stable and tractable posterior estimation.

2. Architectural Differences

We respectfully offer a different perspective and provide two complementary reasons:

• First, the generative process is evaluated based on image quality, not the architecture itself. What ultimately matters in evaluating a generative prior is the quality of the samples it produces, as measured by such metrics as LPIPS and FID—not the specific architecture used. Therefore, if the prior in our method does not outperform the baselines in unconditional generation (as we explicitly state), any observed improvement in inverse problem performance can be reasonably attributed to the inference strategy rather than architectural differences. The comparison remains valid and well-justified.

• Second, solving inverse problems inherently requires balancing prior fidelity and measurement consistency. While strong priors (e.g., Stable Diffusion 1.5) have advantages, they may compromise consistency and overall performance, as shown in baseline experiments. This reflects a general principle: even if a stronger prior is used, it does not necessarily yield better inverse problem results unless it aligns well with the measurement model.

3. Baselines and Fair Comparisons

Baseline Comparisons:

• All existing flow matching baselines (e.g., OT-ODE, C-$\Pi$GFM) operate in pixel space, whereas our method is designed for latent space. In this work, we primarily focused on latent-space baselines to enable a fair and meaningful comparison within the same modelling paradigm. That said, we acknowledge that OT-ODE can, in principle, be extended to the latent space. In fact, we already explored such a variant in our ablation study, using the OT-ODE posterior covariance (denoted Cov-$\Pi$GDM), as described in the Appendix (p. 25).

• We have now included additional pixel-space comparison on the CelebA-HQ dataset. For completeness, we also provide results for the C-$\Pi$GFM. The additional experimental results are shown in the following table. Corresponding qualitative results are available but cannot be included here due to policy restrictions.

• Results are now presented in two categories: latent-based methods (MPGD, Resample, PSLD, LatentDAPS on three datasets, and LatentDMPlug on FFHQ) and pixel-based methods ($\Pi$GDM, OT-ODE, and C-$\Pi$GFM on CelebA-HQ).

Table 3: Quantitative comparison of LFlow against other Pixel-based flow solvers across four inverse problems (BIP: Box InPainting, GDB: Gaussian DeBlurring, MDB: Motion DeBlurring, SR: Super-Resolution) on CelebA-HQ datasets.

Fair comparison:

• As noted in our paper, flow matching models offer greater efficiency during both training and inference by producing straighter sampling paths under modified noise schedules. This efficiency can be further improved when applied in the latent space. However, this does not necessarily mean that ODE-based sampling yields higher sample quality compared to SDE-based sampling. This is precisely why other flow-based methods still compare their performance against diffusion-based counterparts.

4. Restructuring the Method Section

Thank you for this helpful suggestion. We have revised the methodology section to improve clarity and better distinguish our novel contributions from prior work. We also updated and relocated the Related Work section to Section 2. We added a conceptual figure showing the flow of our method and its difference with OT-ODE in posterior covariance estimation. Regarding the specific examples you mentioned:

• Posterior sampling: While posterior sampling using $\underline{*unconditional* \text{ vector fields}}$ has been explored in pixel space for inverse problems, it has not been theoretically justified. In this work, we provide a formal proposition 3.1 establishing when and why posterior sampling is valid under our setting. Building on this foundation, we further extend the framework to handle conditional vector fields, enabling guided inference consistent with measurements. We further improved our methodology section (for example, for posterior covariance estimation, we refer the reviewer to our response to the third question of Reviewer xFYk).

• Initialization strategy: We have now moved this explanation to the Implementation section. While we acknowledge being "inspired by" prior works, our initialization strategy is notably distinct. Specifically, across all tasks, we encode the measurement directly using the encoder $\mathcal{E}$ to obtain the latent code—without applying the forward operator $\mathcal{A}$ or its pseudoinverse $\mathcal{A}^\dagger$, as commonly done in other latent diffusion or pixel-space flow-based methods (e.g., C-$\Pi$GFM).

We sincerely appreciate your follow-up comments and clarifications.

Although we still believe that our initial response regarding architecture is both theoretically sound and intuitively convincing, we are more than happy to provide further clarification on the following points.

1. On PSLD
We are glad that you agree with us regarding PSLD adopting a less fair comparison setting.

2. On OT‑ODE
As you noted, OT‑ODE considers two settings:
(1) reusing a pretrained VP‑SDE and converting the score function to a velocity field; and
(2) training a generic flow‑based model.

For our scenario, the second setting provides a more relevant comparison, as converting pretrained score functions into velocity fields is impractical for latent-based methods (we can elaborate further if needed). However, could you please clarify whether, in OT-ODE, the “generic flow model” is trained under the same training objective as its diffusion-based counterpart? Our understanding is that it is not.

3. On your last point regarding shared pretrained models
We agree that some recent works evaluate sampling strategies using a shared pretrained model to isolate the effect of the sampler. However, these works—such as RED‑Diff—are, first, all in pixel space and discrete-time, and second, are primarily situated in the variational perspective or variable splitting of posterior inference.

Training‑free inverse problem solvers leveraging diffusion or flow priors span multiple methodological classes:

1. Variable splitting methods decompose inference into alternating data‑fidelity and regularization steps~\cite{wang2024dmplug, song2024solving, zhu2023denoising}.
2. Variational Bayesian methods introduce a parameterized surrogate posterior, often Gaussian, and optimize it using a variational objective~\cite{Feng_2023_ICCV, feng2023efficient, mardani2024a}.
3. Asymptotically exact methods combine generative priors with classical samplers (MCMC, SMC, Gibbs) for provably correct posterior approximation~\cite{wu2023practical, trippe2023diffusion, cardoso2024monte, dou2024diffusion}.
4. Guidance‑based methods correct the generative trajectory with an approximate likelihood gradient to steer samples toward the posterior~\cite{jalal2021robust, chung2023diffusion, song2023pseudoinverseguided, boys2024tweedie}.

Our method explicitly aligns with the guidance-based category. Given the differing assumptions and inference paradigms among these methodological classes, we believe fairness criteria must be contextually assessed.

4. Essentially, comparisons with pixel‑based methods cannot be entirely fair, but this remains common practice among all latent‑based methods.

5. Lastly, we wish to explicitly address a crucial point: If someone intends to utilize a pretrained latent flow model for posterior sampling, it remains unclear which baselines would constitute fair and meaningful comparisons, and under precisely what conditions. Given your expertise, we would greatly appreciate specific guidance on this matter. We firmly believe this clarification is essential—not only to strengthen our current analysis but also to inform future methodological practices. If there are approaches or considerations that we have inadvertently overlooked, we kindly ask you to point them out.

Dear Reviewer,

We have made continuous efforts, collectively as all authors, to address the issues raised by the reviewer. We sincerely hope that the clarifications and additional experiments we have provided will assist the reviewer in reassessing their evaluation.

We would greatly appreciate it if the reviewer could let us know whether our clarifications have satisfactorily addressed their concerns. If not, we would be happy to know where and how our response may have fallen short, so that we can further clarify.

Best regards,

The Authors

We are truly grateful for your willingness to reconsider the score and for the constructive suggestions provided.

We agree that making the OT-ODE comparison more explicit in the main table can further clarify our results, and we will incorporate this change now. Regarding the backbones, we will take steps to meet your concern. We also agree that including additional perceptual metrics such as FID and KID would make the evaluation more comprehensive. We actually already reported FID scores for all datasets in Table 6 and included some discussion on this point. We also consider this score for other new baselines.

Once again, we sincerely thank the reviewer for the thoughtful engagement.

Dear Reviewer, Thank you so much for your engagement.

We did, actually — we really did. Both for the FFHQ dataset and the ImageNet dataset, we have complete results for OT-ODE in latent space. For at least two tasks from each dataset, these results are presented in Table 3. However, because part of our contribution is posterior covariance estimation, we preferred to treat this as an ablation study to highlight the significance of our work — not only in terms of metrics (Table 3), but also in terms of convergence (Table 4). Otherwise, it would have been no difference for us to place these results in Table 1 and Table 2.

We already have done many experiments with OT-ODE with pre-trained latent flow model and we can present the full results if you want.

We appreciate the reviewer’s comments, which helped us improve the clarity of the paper.

1. Flow vs. Diffusion: Two Sides of the Same Coin?

• Diffusion trajectories can be converted into flow paths by matching the signal-to-noise ratio, which allows one to derive the corresponding time and scale mappings. However, this holds only under certain assumptions—namely, that both processes share the same distributional path (e.g., Gaussian) and the diffusion model is trained in continuous time. For discrete-time models, which operate at fixed steps, this mapping becomes nontrivial. Although heuristic rounding methods exist [Karras 2022], they are often impractical—particularly for latent diffusion models or Stable Diffusion with non-linear noise schedulers—because they require solving a cubic equation to infer the flow time, making conversion extremely difficult, if not infeasible in practice.

• Flow-based models offer significant advantages in both training and inference efficiency, thanks to their straighter sampling trajectories. This led the researchers to apply it for inverse problems in pixel space, as demonstrated by methods such as OT-ODE [43] and D-Flow [44].

• Latent flow models, such as Stable Diffusion3 (Esser et al., 2024) and OpenSora-2, have shown great success in synthesis but remain underexplored for inverse problems. Our work takes a step in this direction by enabling efficient and accurate posterior sampling in the latent flow framework.

2. Missing Conceptual Comparison

• We have a conceptual figure in Appendix (page 25) that illustrates the differences between our method and prior methods such as OT-ODE and $\Pi$GDM (we moved this now to the method section). We have also clarified the distinctions in the revised Related Work section (now Section 2). Additionally, we have now included a conceptual figure that illustrates the overall flow of our proposed method.

3. Likelihood and Posterior Covariance Estimation

We should have included our related work section earlier, before the method section (this has now been revised), which would have made the differences more apparent. We approximate the likelihood as $\nabla _{\mathbf{x}_t} \log \mathbb{E} _{\mathbf{x}_0 \sim p(\mathbf{x}_0 | \mathbf{x}_t)}[p(\mathbf{y} | \mathbf{x}_0)]$, whereas previous methods approximate it as $\nabla _{\mathbf{z}_t} \log \mathbb{E} _{\mathbf{x}_0 \sim p(\mathbf{x}_0 | \mathbf{z}_t)}[p(\mathbf{y} | \mathbf{x}_0)]$. We believe that, roughly, all methods assume the reverse process $p(\mathbf{x}_0|\mathbf{x}_t)$ is Gaussian, characterized by a posterior mean and posterior covariance. The key difference lies in modeling posterior covariance: Most of methods, whether in pixel or latent spaces, such as DPS, DAPS, Resample, and so on, consider it zero. $\Pi$GDM assumes Gaussian data distribution and derives scaling from a forward process, which is prior-agnostic (proved in page 22 and 25), TMD applies a second-order Tweedie’s formula with a row-sum approximation. In contrast, we achieved as follows (revised version).

Latent Posterior Covariance.

To address this, we further analyze the structure of the posterior covariance and the Jacobian $\nabla _{\mathbf{z}_t}\mathbf{v} _{\boldsymbol{\theta}}(\mathbf{z}_t, t)$, under certain regularity assumptions on the latent prior $p(\mathbf{z}_0)$.

Assumption (Strong Log-Concavity of the Latent Prior \cite{cattiaux2014semi}) Let $p(\mathbf{z}_0) = \exp(-\Phi(\mathbf{z}_0))$ be the latent prior over $\mathbb{R}^d$, where $\Phi$ is a twice continuously differentiable potential function. We assume that $p(\mathbf{z}_0)$ is $\gamma$-strongly log-concave for some $\gamma > 0$; that is, $\nabla^2 \Phi(\mathbf{z}_0) \succeq \gamma \, \boldsymbol{I}_d, \text{for all } \mathbf{z}_0 \in \mathbb{R}^d.$

(1) Log-concave Prior: This assumption imposes a uniform lower bound on the Hessian of the prior's potential function, which translates to a lower and upper bound on the Jacobian of the vector field, as formalized below.

Proposition (Bound on the Jacobian of the Vector Field). Let $\mathbf{v}(\mathbf{z}_t, t)$ denote the velocity field of the interpolant $\mathbf{z}_t = \alpha(t)\mathbf{z}_0 + \sigma(t)\mathbf{z}_1$ between a standard Gaussian prior and a target distribution $p(\mathbf{z}_0)$, defined via coefficients $\alpha(t), \sigma(t) \in \mathbb{R}$. Under Assumption 1, the Jacobian of the vector field satisfies the following bound:

$\frac{\mathrm{d}}{\mathrm{d}t}( \tfrac{1}{2} \log(\alpha(t)^2 + \gamma \sigma(t)^2)) \cdot \boldsymbol{I}_{d_z} \preceq \nabla _{\mathbf{z}_t} \mathbf{v} _{\boldsymbol{\theta}}(\mathbf{z}_t, t) \prec \frac{\mathrm{d}}{\mathrm{d}t} \log \sigma(t) \cdot \boldsymbol{I} _{d_z} \quad \text{for all } t \in (0,1],\; \mathbf{z} \in \mathbb{R}^d$

The proof is provided in Appendix A. The resulting sandwich bound ensures the Jacobian remains well-behaved—the upper bound guarantees valid and stable posterior covariance, while the lower bound prevents overestimated uncertainty during posterior-guided inference.

(2) Gaussian Prior: We now consider a tractable special case where the latent prior is Gaussian, i.e., $\mathbf{z}_0 \sim \mathcal{N}(0, \sigma _{*latr*}^2 \boldsymbol{I})$, a choice that is both theoretically justified and widely used in practice—for instance, in variational autoencoders (VAEs), where the latent prior is regularized toward a standard Gaussian via KL divergence~\cite{vahdat2021scorebased}. This enables us to derive the optimal velocity field and its Jacobian in closed form, which is characterized in the next proposition, whose proof can be found in Appendix A.

Proposition (Optimal Vector Field). Let $\mathbf{z}_0 \sim \mathcal{N}(\mathbf{0}, \sigma _{\text{latr}}^2 \boldsymbol{I} _{d_z})$ and $\mathbf{z}_1 \sim \mathcal{N}(\mathbf{0}, \boldsymbol{I} _{d_z})$ be independent random variables. Define $\mathbf{z}_t = (1 - t) \mathbf{z}_0 + t \mathbf{z}_1$ for $t \in [0, 1]$. The optimal vector field $\mathbf{v}^{\star}(\mathbf{z}_t, t)$ that minimizes the expected squared error

$\arg \min_{\mathbf{v}} \, \mathbb{E}\left[ \left\| \mathbf{v}(\mathbf{z}_t, t) - (\mathbf{z}_0 - \mathbf{z}_1) \right\|^2 \right]$ is given by $\mathbf{v}^{\star}(\mathbf{z}_t, t) = \frac{(1 - t) \sigma _{*latr*}^2 - t}{(1 - t)^2 \sigma _{*latr*}^2 + t^2} \mathbf{z}_t.$

Remark (Tightness of Bound). When $\mathbf{z}_0 \sim \mathcal{N}(0, \sigma _{*latr*}^2 \boldsymbol{I})$, the optimal vector field $\mathbf{v}^\star(\mathbf{z}_t, t)$ achieves the Jacobian lower bound in Proposition 3. This follows from the potential function $\Phi(\mathbf{z}_0)=\frac{1}{2\sigma _{*latr*}^2} \|\mathbf{z}_0\|^2$, yielding $\gamma = \sigma _{*latr*}^{-2}$, and interpolation coefficients $\alpha(t) = 1 - t$, $\sigma(t) = t$. Substituting into the bound confirms it matches the exact Jacobian of $\mathbf{v}^\star$, making the bound tight.

Plugging the Jacobian of the optimal vector field into \eqref{} yields:

$\mathbb{C}\mathrm{ov}[\mathbf{z}_0 \mid \mathbf{z}_t] = r^2(t) \cdot \boldsymbol{I} _{d_z}, \quad \text{with} \quad r^2(t) = \frac{t^2 \left[(1 - t)(1 - 2t) + 2t^2 \right]}{(1 - t)\left[(1 - t)^2 + t^2\right]},$

where we assumed $\sigma_{\text{latr}} = 1$. As a result, the propagated covariance can also be simplified as

$\mathbb{C}\mathrm{ov}[\mathbf{x}_0 \mid \mathbf{z}_t] \approx J _{\mathcal{D}} \cdot \mathbb{C}\mathrm{ov}[\mathbf{z}_0 \mid \mathbf{z}_t] \cdot J _{\mathcal{D}}^\top \approx r^2(t) \cdot J _{\mathcal{D}} J _{\mathcal{D}}^\top.$

Computing the full Jacobian $J_{\mathcal{D}} \in \mathbb{R}^{d_x \times d_z}$ is often infeasible in practice. To simplify, we assume that the decoder acts approximately as a local isometry near $\bar{\mathbf{z}}_0$ \cite{dai2020usual}, such that $J _{\mathcal{D}} J _{\mathcal{D}}^\top \approx \cdot P$, where $P$ is the orthogonal projector onto the image of $J _{\mathcal{D}}$. For computational convenience, we approximate this behavior as isotropic in the full space, resulting in:

$\mathbb{C}\mathrm{ov}[\mathbf{x}_0 \mid \mathbf{z}_t] \approx r^2(t) \cdot \boldsymbol{I} _{d_x}.$

4. Baselines

We considered five baselines and provided both quantitative and qualitative results for four tasks of each. We notice that SITCOM was published at ICML 2025, with its latest arXiv version appearing after our NeurIPS submission. Nonetheless, we now cite it appropriately. For DMPlug, we have included additional results Table 2.

Table 2: Quantitative comparison of LFlow and LatentDMplug across four inverse problems (BIP: Box InPainting, GDB: Gaussian DeBlurring, MDB: Motion DeBlurring, SR: Super-Resolution) on the FFHQ dataset.

We respectfully note that several recent top-conference works have also focused exclusively on linear inverse problems, including but not limited to OT-ODE (TMLR 2024), PSLD (NeurIPS 2024), PnP-Flow (ICLR 2025), FPS-SMC (ICLR 2025), and C-$\Pi$GFM (NeurIPS 2024).

5. Questions

1. Details of the posterior covariance ablation are on page 25, but now explained in the right place.
2. We added DAPS result in Table 4.

We appreciate the reviewer’s engagement and address the issue as follows.

1. "flow model... be converted into diffusion models [1, 2], ... with minimal effort"...

We believe both references [1, 2] cited by the reviewer—one of which we already cite as [74]—only establish the theoretical connection between the vector field and the score function for a single model with one specific noise schedule.

For two models (flow and diffusion) with different noise schedules, we clarify that a practical conversion procedure was first introduced in [3] and later refined in [4], which provided an extended proof and a minor correction.

In the first version of [4], the authors stated: “Proposition 1 allows us to initialize the Reflow with pre-trained diffusion models such as EDM [21] or Stable Diffusion [38].” We had the opportunity to discuss with the authors of [4] some of the practical challenges, particularly in applying these methods to models such as SDs and LDMs, which operate in latent space. Notably, in the final version of [4], the reference to Stable Diffusion was removed. We interpret this as an indication that the authors came to realise that implementing such theoretical ideas in practice—especially for latent-space models—is a far cry from what the theory might suggest.

2. "Even for quadratic schedulers..., as ... cubic equation introduced by EDM can ...."

In the EDM reference [5], no cubic equation is introduced. We would like to explain further what we meant by "cubic equation". Based on [4], now the question is how we can find $s_{**vp**}$ and $\tau_{**vp**}$ for Latent Diffusion Models (LDM) or stable diffusions!. One practical solution is as follows: LDM uses a modification of the DDPM schedule. Both are variance preserving schedules, i.e., $\sigma'(t) = \sqrt{1 - \alpha'(t)^2}$, and define $\alpha'(t)$ for discrete timesteps in terms of diffusion coefficients $\mathbf{\beta_t}$ as $\alpha'(t) = \left(\prod_{s=0}^{\mathbf{t}} (1 - \mathbf{\beta_s})\right)^{\frac{1}{2}}$. For given boundary values $\mathbf{\beta_0}$ and $\mathbf{\beta_{T-1}}$, LDM uses $\mathbf{\beta_t} = \left(\sqrt{\mathbf{\beta_0}} + \frac{\mathbf{t}}{T-1}(\sqrt{\mathbf{\beta_{T-1}}} - \sqrt{\mathbf{\beta_0}})\right)^2$. If we use Eq. 29 in Song's SDE paper [6] to compute the perturbation kernel of SD or LDM's noise schedule, we can get the following cubic equation, from which we may obtain $(\tau_{\mathrm{vp}}, s_{\mathrm{vp}})$ given $\beta_{\mathrm{start}}$ and $\beta_{\mathrm{end}}$.

\frac{(\sqrt{\beta_{\text{end}}} - \sqrt{\beta_{\text{start}}})^2}{3} \tau^3 + (\sqrt{\beta_{\text{start}}} \(\sqrt{\beta_{\text{end}}} -\sqrt{\beta_{\text{start}}})) \tau^2 + \beta_{\text{start}} \tau + \ln (\frac{(1-t)^2}{(1-t)^2+ t^2})= 0

While, in theory, it is possible to recover $\tau$ from a given value of $t$, in practice the presence of the logarithmic term introduces significant numerical challenges. In particular, to cover any meaningful range of $\tau$ (e.g., $0$ to $999$), we must evaluate the roots of the above cubic equation for values of $t$ increasingly close to $1$. The table below shows that even for $t$ values extremely close to $1$, the roots of the equation fail to span the desired useful range of $\tau$. This indicates that the practical difficulty is not at all reflected in the theoretical formulation. This is yet another example where the complexities of practical implementation can hinder—and at times even render impossible— the realisation of what theoretical results predict.

3. "I believe that designing a stronger generative prior is not ...of this paper"...

We agree with the reviewer; however, we never made such a claim. On the contrary, our generative prior does not exceed that of LDMs or Stable Diffusion v1.5, as explicitly stated in both the main text and the appendix.

Most importantly, regardless of conversion, we must consider that our LFlow method is theoretically formulated for latent flow models whose autoencoders are trained with a VAE objective. This VAE‑based regularization is essential for our latent posterior covariance modeling. In contrast, other latent-based inverse models such as Resample and DAPS employ available LDMs with vector‑quantized regularization (i.e., LDM-VQ-4), which may not be suitable for our method, as they cannot provide the same theoretical justification.

[3]. Training-free linear image inverses via flows (pokle2023training)

[4]. Improving the training of rectified flows (lee2024improving)

[5]. Elucidating the design space of diffusion-based generative models (karras2022elucidating)

[6]. Score-based generative modeling through stochastic differential equations (song2021scorebased)

Dear Reviewer,

We have made continuous efforts, collectively as all authors, to address the issues raised by the reviewer. We sincerely hope that the clarifications and additional experiments we have provided will assist the reviewer in reassessing their evaluation.

We would greatly appreciate it if the reviewer could let us know whether our clarifications have satisfactorily addressed their concerns. If not, we would be happy to know where and how our response may have fallen short, so that we can further clarify.

Best regards,

The Authors

We are grateful for the reviewer’s insights and suggestions. We have carefully considered each point and provide our responses below.

1. Inference time

We appreciate the reviewer’s observation. While our inference time is already faster than that of prior methods (as shown in Table 4), we acknowledge that the current implementation can still require approximately 3 min 40 s to 10 min to solve inverse problems. Specifically, for deblurring and super-resolution, the average time ranges from about 2 min 30 s to 5 min 40s, whereas for inpainting it is around 3 min 40 s to 10 min. This is why we generally report a range of 4 to 10 minutes. We believe this runtime is primarily due to two factors:

• Use of adaptive solvers: Our inference relies on an adaptive ODE solver, which results in variable computation time across samples. While some samples complete in ~3 minutes, others—depending on task complexity—can take longer.

• Repeated decoding for data consistency: Similar to other latent-based inverse solvers, our approach enforces measurement consistency by repeatedly decoding intermediate latent states during inference. This decoding step, while necessary for accurate reconstruction, is computationally expensive unless the measurement operator is learned during training.

For context, D-Flow [1] reports sampling times ranging from 5 to 15 minutes on average. We believe our method offers a reasonable trade-off, and in future work we aim to further reduce inference time through solver optimization or by exploring alternative numerical methods like accelerated integration techniques [2].

2. PSNR values

We appreciate the reviewer’s observation. While our PSNR is competitive but not always the highest among all baselines, our method consistently outperforms all baselines in SSIM and LPIPS across all tasks, as shown in Table 1. These two metrics are more aligned with perceptual and structural fidelity, which is reflected in the qualitative results in Figure 1—our reconstructions preserve fine textures (e.g., skin, hair) and edges more faithfully. The slightly lower PSNR may be due to small misalignments or high-frequency texture recovery, which benefit perceptual quality but increase pixel-wise error. This trade-off has also been observed in prior perceptual-focused methods such as Resample. Please note that on CelebA-HQ, our method achieves the best PSNR in three tasks.

3. Noiseless setting

We thank the reviewer for pointing this out. All our baselines implemented in the latent space are evaluated only under the noisy setting, so we followed this setting for a fair and consistent comparison. We now included results for the completely noiseless setting on the CelebA-HQ dataset for super resolution and inpainting task.

4. Questions

1. For computational efficiency, we experimented with $K=1$ and $K=2$. As shown in the ablation results provided below (Table 1), using $K=2$ yields slightly better average performance across tasks while maintaining reasonable inference time. We also tested $K=>3$, but observed little noticeable improvement—but increased computational cost. In what follows, we present our ablation study of hyperparameter $K$ on FFHQ dataset.

Table 1. Ablation of parameter $K$ on FFHQ. Increasing $K$ offers marginal gains with higher cost; $K{=}2$ achieves the best trade-off between performance and efficiency.

2. While Eq. (8) assumes matched dimensions for notational simplicity, our method is fully general and does not require $\mathbf{z}_t \in \mathbb{R}^k$ and $\mathbf{y} \in \mathbb{R}^m$ to have the same dimension. The likelihood is defined in image space as $p(\mathbf{y} \mid \mathbf{x}_0)$, with $\mathbf{x}_0 = \mathcal{D} _\varphi(\bar{\mathbf{z}}_0)$ and $\bar{\mathbf{z}}_0 = \mathbb{E}[\mathbf{z}_0 \mid \mathbf{z}_t]$. The guidance gradient is computed using the chain rule: $\nabla _{\mathbf{z}_t} \log p(\mathbf{y} \mid \mathbf{z}_t) =(\nabla _{\mathbf{z}_t} \bar{\mathbf{z}}_0)^\top J _\mathcal{D}^\top \mathcal{A}^\top \nabla _{\mathbf{x}_0} \log p(\mathbf{y} \mid \mathbf{x}_0)$. For a Gaussian likelihood, this yields the explicit formula given in Eq. (12) of the paper. We have clarified this mapping and removed any ambiguity in the revised manuscript.

Later, posterior mean of $\mathbf{z}_t$ given $\mathbf{z}_t$ is passed through the decoder, and consistency is enforced in pixel space as following

• $\nabla_{\mathbf{z}_t} \log p(\mathbf{y} \mid \mathbf{z}_t) \approx \underbrace{(\nabla _{\mathbf{z}_t} \bar{\mathbf{z}}_0)^\top J _{\mathcal{D}}^\top} \underbrace{.... (\mathbf{y} - \mathcal{A} \mathcal{D} _\varphi(\bar{\mathbf{z}}_0))} _{\mathbf{v}}$

• $\underbrace{(\nabla _{\mathbf{z}_t} \bar{\mathbf{z}}_0)^\top J _{\mathcal{D}}^\top} = (J _{\mathcal{D}} \cdot \nabla _{\mathbf{z}_t} \bar{\mathbf{z}}_0)^\top = (\nabla _{\mathbf{z}_t} \mathcal{D}(\bar{\mathbf{z}}_0))^\top$

3. Please note that we use an adaptive solver, where the number of function evaluations (NFEs) naturally varies across samples. For this reason, instead of reporting a fixed NFE, we focus on inference time, which provides a more practical and consistent measure of efficiency across tasks and methods.

4. We thank the reviewer for the question. While our main focus is the noisy setting—in line with all baselines implemented in latent space—we included the results for the completely noiseless setting on the CelebA dataset for two tasks. We agree that $\Pi$GDM performs well in the noiseless setting; however, its performance drops significantly under noisy conditions.

[1]. D-flow: Differentiating through flows for controlled generation (ben2024d)

[2]. FireFlow: Fast Inversion of Rectified Flow for Image Semantic Editing (deng2024fireflow)

========================================================================================== ==========================================================================================

Note. Due to space constraints in the individual response sections, we provide a brief summary of our main revisions and clarifications here. Reviewer-specific details are included where space allowed.

Summary of Revisions:

We would like to provide all reviewers with a concise summary of the steps we have taken to address the comments and suggestions. For each major point, we have incorporated changes as recommended by the specific reviewer(s), as detailed below:

• 1. As recommended by Reviewer R8py, we revised and modified the Related Work section and moved it before the Methodology section to clearly highlight prior approaches and better position our own contributions.
• 2. As suggested by Reviewer H6V9, we improved the Methodology section—especially our contributions on latent likelihood approximation and posterior covariance estimation—and restructured it to better highlight our key contributions.
• 3. Following Reviewer R8py’s suggestion, we added a conceptual figure illustrating the flow of our proposed method. We also moved the original figure (previously on page 25) to appear alongside this new figure, clearly presenting our methodology and highlighting the differences from prior work.
• 4. In response to Reviewer xFYk and Reviewer H6V9, we added additional baselines. Results are now presented in two categories: latent-based methods (MPGD, Resample, PSLD, LatentDAPS on three datasets, and LatentDMPlug on FFHQ) and pixel-based methods ($\Pi$GDM, OT-ODE, and C-$\Pi$GFM on CelebA-HQ). Accordingly, we have revised and updated the Implementation Details section for improved readability and completeness.
• 5. Based on feedback from Reviewer kCk5, we included an ablation study on the hyperparameter $K$, which controls the number of gradient update steps. This analysis clarifies the impact of $K$ on performance.

We hope these targeted revisions directly address each reviewer's feedback and strengthen the overall clarity and quality of the paper.

I thank the authors for answering my questions about notation and for including additional ablations. One issue that I previously missed (and has been raised by another reviewer) is that in some of the results, the backbone model is not the same (as well as the checkpoint). I previously assumed that they were the same for fair comparison as has been done in previous works. While I understand that it is difficult to replicate all the results of all the previous methods, the paper should ideally include the performance of at least 1-2 other SOTA methods on the same model checkpoint. As a result, I would like to retain my score for now.

Thank you for your reply to my concerns raised above. I would like to further elucidate upon my concerns.

"As stated in the Implementation Details section of our paper and appendix, we have not modified any of the baseline methods. All baselines are adopted exactly as released by their authors, using the same official checkpoints and configurations from their respective GitHub repositories."

I understand that the authors have not modified any of the baseline methods and i agree that it is useful to include these metrics for comparison. However, 1-2 of these SOTA methods should be replicated with the same architecture and model checkpoint as the one used in LFlow.

"The SOTA latent-based methods (e.g., PSLD, LatentDMPlug, Resample, LatentDAPS) are based on discrete-time latent diffusion models. In contrast, our method leverages a continuous-time flow in latent space, which is novel in its own right, as no prior work has explored solving inverse problems using a latent-based continuous-time flow prior."

I also understand the point that this is a latent-based continuous time flow model. However, we can easily replicate inverse solvers for discrete time models with checkpoint for continuous time models. The vice versa is ofcourse not possible.

We thank the reviewer for their time and constructive comments.

1. On the Necessity and Scope of Our Assumptions

a. Proposition 3.1. This proposition is based on the reverse-time continuity equation (Liouville equation) for deterministic ODE flows, rather than the reverse-time Fokker-Planck equation, which describes stochastic diffusion processes (SDEs). Thus, our theoretical results apply directly to deterministic ODEs, not SDEs. This follows a standard setup used in prior works (e.g., Lipman et al., 2023) to justify sampling in continuous-time generative models. We revised the text to reflect this correction and avoid confusion.

b. Dimension relaxation. We thank the reviewer for this question. While Eq. (8) assumes matched dimensions for notational simplicity, our method is fully general and does not require $\mathbf{z}_t \in \mathbb{R}^k$ and $\mathbf{y} \in \mathbb{R}^m$ to have the same dimension. The likelihood is defined in image space as $p(\mathbf{y} \mid \mathbf{x}_0)$, with $\mathbf{x}_0 = \mathcal{D} _\varphi(\bar{\mathbf{z}}_0)$ and $\bar{\mathbf{z}}_0 = \mathbb{E}[\mathbf{z}_0 \mid \mathbf{z}_t]$. The guidance gradient is computed using the chain rule: $\nabla _{\mathbf{z}_t} \log p(\mathbf{y} \mid \mathbf{z}_t) =(\nabla _{\mathbf{z}_t} \bar{\mathbf{z}}_0)^\top J _\mathcal{D}^\top \mathcal{A}^\top \nabla _{\mathbf{x}_0} \log p(\mathbf{y} \mid \mathbf{x}_0)$. For a Gaussian likelihood, this yields the explicit formula given in Eq. (12) of the paper. We have clarified this mapping and removed any ambiguity in the revised manuscript.

c. Decoder linearity. We thank the reviewer for this point. Most latent-based inverse solvers ignore decoder nonlinearity during inference and effectively assume local linearity, but often without formal justification—see, e.g., MPGD, Resample, LatentDMplug, and LatentDAPS. While PSLD [31] attempts to explicitly address the decoder’s nonlinearity, recent work [1] has shown this approach to be ineffective and prone to artifacts. In our method, we use a first-order Taylor expansion around $\bar{\mathbf{z}}_0$, which provides a tractable and theoretically grounded way to propagate uncertainty. This linearization is empirically justified by the observed smoothness of pretrained decoders and the fact that posterior samples typically remain in high-density regions of the latent prior, where the decoder behaves approximately linearly.

d. Motivation for the Gaussian assumption on $p(\mathbf{z}_0 | \mathbf{z}_t)$. We thank the reviewer for this question. The Gaussian assumption for $p(\mathbf{x}_0 | \mathbf{x}_t)$ is widely used in training-free (zero-shot) guidance methods for pixel-space inverse solvers (see, e.g., [2]). This approximation is motivated by tractability: computing $\nabla{\mathbf{x}_t} \log \mathbb{E} _{\mathbf{x}_0 \sim p(\mathbf{x}_0 \mid \mathbf{x}_t)} [p(\mathbf{y} | \mathbf{x}_0)]$ is otherwise intractable, as it requires sampling from the full conditional at each step. In our case, the Gaussian assumption is further justified by the Gaussian latent prior and the smoother, more linear dynamics in latent space, which make the conditional distribution $p(\mathbf{z}_0 | \mathbf{z}_t)$ closely approximate Gaussianity—especially near high-density regions where the posterior concentrates. This is also supported by standard results for linear Gaussian systems in diffusion and flow models. We have clarified this motivation in the revised manuscript.

f. Latent Prior $p(\mathbf{z}_0)$. Our assumption is grounded in a bound on Jacobian of vector field $\nabla_{\mathbf{z}_t} \mathbf{v} _\theta(\mathbf{z}_t, t)$ (Revised Proposition 3.3, which holds for any strongly log-concave prior, including Smoothed Laplace, Logistic Distribution, Generalized Gaussian with hight $\beta$, and Sub-Gaussian priors with strong tail curvature, ensuring theoretical robustness even when the prior deviates mildly from exact Gaussianity. In Remark 3.4 (Please see our response to Reviewer xFYk), we show that the Gaussian prior is a tight instance of this bound, justifying the closed-form posterior covariance in Eq. 17.

g. Local isometry. While assuming $J_{\mathcal{D}}^\top J_{\mathcal{D}} \approx \mathbf{I}$ is indeed a simplification, we emphasize that: first, many decoder networks used in generative models (e.g., VAEs, diffusion decoders) are designed to approximately preserve local structure in the latent space, especially when trained with reconstruction loss and smooth latent priors. This implies that the decoder acts locally as a near-isometry around well-behaved regions of the latent distribution [3], [4]; second, our assumption is only invoked locally around the posterior mean $\bar{\mathbf{z}}_0$, not globally across the entire latent space. In this regime, the decoder Jacobian typically varies smoothly, and near-orthonormal behavior has been observed empirically in prior work (e.g., VAEs, latent diffusion); third, importantly, the assumption allows us to arrive at a tractable and interpretable approximation of the likelihood gradient, which facilitates training-free inference.

2. Baselines

We initially compared against five strong baselines. In response to all reviewer feedback, we have added additional baselines. Results are now presented in two categories: latent-based methods (MPGD, Resample, PSLD, LatentDAPS on three datasets, and LatentDMPlug on FFHQ) and pixel-based methods ($\Pi$GDM, OT-ODE, and C-$\Pi$GFM on CelebA-HQ). We kindly refer the reveiwer to Table 2 (Reviewer xFYk) and Table 3 (Reviewer H6V9).

3. Supervised baseline

Our focus was on training-free (zero-shot) methods, and none of the baselines we compare against—including recent state-of-the-art approaches—assume access to paired training data or require supervised fine-tuning. This aligns with the standard protocol adopted in many zero-shot studies. Supervised methods typically require task-specific training and do not generalize across different degradation operator settings.

4. Table 2: PSNR-LPIPS tradeoff and spurious features

We thank the reviewer for this observation. As shown in Table 2, LFlow achieves consistently superior perceptual similarity (lower LPIPS) and SSIM across all tasks, while maintaining PSNR values comparable to the strongest baselines. This is in line with widely observed trends in generative models: methods optimized for perceptual quality may slightly sacrifice pixel-wise metrics like PSNR to better capture semantic structure and fine details. Occasionally, this can result in enhanced textures or subtle features that, while visually plausible, do not perfectly align with the ground truth—referred to as “spurious features.” Importantly, our qualitative results (Fig. 2) show that LFlow faithfully reconstructs key structures without introducing unnatural or fabricated artifacts. We have added further discussion of this well-known tradeoff and its implications for reconstruction quality in the revised manuscript.

5. Figure 2: dog case

We thank the reviewer for the observation. Upon closer inspection, we note that LFlow’s reconstruction of the dog preserves the pose and structure consistent with the ground truth, while enhancing texture and fur detail. We believe the perceived "spurious features" may stem from higher-frequency detail restoration, not from hallucinated or incorrect content. In the revised manuscript, we have now annotated the visual examples in Figure 2 with their corresponding PSNR and LPIPS values for each method, including the dog example in the last row. This allows a direct quantitative comparison between perceptual and distortion-based metrics for the highlighted regions. We hope this clarifies how each method balances detail restoration and faithfulness to the reference image.

6. Questions

1. We would like to clarify that the phrase “training-free inference” appears only once in our paper (Line 28), and we are happy to rephrase it if the wording is unclear. First, we acknowledge that diffusion models have proved to be effective for both training-free inference and test-time adaptation, the latter of which can involve updating the model during inference. Second, we recognize that the term test-time adaptation is broad and applies to various tasks—including classification, object detection, and inverse problems—whenever one has access to observations from a target domain and a pre-trained model from a source domain. In our case, however, we do not adapt the model at inference time; thus, no additional training is involved.

2. We have now included a derivation for Equation 3.

3. It is fixed now. Thank you.

4. In practice, $\mathbf{y}$ is a degraded version of $\mathbf{x}_0$, and we resize or zero-fill it to match the input size of the encoder $\mathcal{E} _\phi$, ensuring dimensional compatibility. Also, as clarified in the experimental section, $\mathbf{y}$ indeed contains noise. Yes — we directly pass the measurement to the encoder rather than a reconstructed image ($\mathcal{A}^\dagger \mathbf{y}$).

The phrase closer to the posterior mode $\mathbf{z}_0 | \mathbf{y}$'' refers to initializing in a plausible latent region guided by $\mathbf{y}$, which improves the stability of reverse-time sampling.

5. We did this now.

6. We have now revised the Related Work section and moved it to Section 2.

7. We added a trajectory of image flow to demonstrate the generation process.

[1]. Prompt-tuning latent diffusion models for inverse problems (chung2023prompt)

[2]. A Survey on Diffusion Models for Inverse Problem (daras2024survey)

[3]. When is Unsupervised Disentanglement Possible? (horan2021unsupervised)

[4]. Variational Autoencoders Pursue PCA Directions (by Accident) (2019Rolınek)

We sincerely appreciate your positive feedback and are glad that you found our clarifications on the assumptions reasonable. Below, we provide further clarification on the remaining points. Due to space constraints, we address your points in two parts.

1. Clarifying the term Training-Free

We fully understand the reviewer’s concern, and we agree that the term "training-free" can sometimes be confusing. Here, we would like to clarify how this term is commonly used in the context of diffusion/flow-based inverse problem solvers. Most diffusion/flow based approaches to inverse problems fall into two distinct paradigms based on how they handle the measurement data during training:

• 1. Supervised (Task-Specific) Methods: These methods train a conditional diffusion model $p(\mathbf{x} | \mathbf{y})$ that is tailored to a specific degradation operator (e.g., super-resolution, deblurring). This setup requires paired datasets of measurements (degraded images) and clean images and is analogous to supervised learning. For example [1].

• 2. Zero-Shot (problem-agnostic) Methods: In contrast, zero-shot methods (also referred to as training-free [2, 3, 4] or plug-and-play [5]) leverage a pretrained unconditional diffusion model $\nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t)$ trained only once for generative modeling, without conditioning on specific measurements or inverse problems. At inference time, these methods combine the pretrained prior with any degradation operator (e.g., blur, mask, undersampling) to produce conditional samples. Crucially, no retraining or finetuning of the generative model is performed for each new inverse problem.

• Why Zero-shot [6] ? Zero-shot learning is one of the several learning paradigms aimed at out-of-distribution generalization, where the algorithm is trained to categorize objects or concepts that it has not been exposed to during training. For inverse problems, where the distribution of measurements $\mathbf{y}$ can change based on the undersampling pattern, the term zero-shot solver is sometimes applied. This usage is intended to reflect the solver’s ability to adapt to different undersampling patterns without retraining, similar to how zero-shot learning generalizes to new classes without prior exposure.

Therefore, when we refer to a method as training-free, we mean that the model is trained once on a dataset and can be directly applied for inference across various tasks and degradation operators without any retraining. Such zero-shot methods offer clear practical advantages, especially in scenarios where retraining for each task is infeasible. In this work, we specifically focus on zero-shot solvers for inverse problems.

2. Training-free methods would be ones that don’t need pre-trained models like DIP, etc.

We totally agree with the reviewer about DIP and its being training-free. But we also note that the term training-free is commonly used in the context of diffusion/flow model-based solvers. This is not something we introduced, but follows from prior works, for example [2 , 3, 4]. If the reviewer is still concerned, we are happy to adopt the term zero-shot or plug-and-play instead.

3. Test-Time Adaptation (TTA)

TTA refers to adapting (training) a model at inference time, without access to training data, using only the test sample(s) or their measurements, which is distributionally different from the training data. This allows the model to handle distribution shifts or unknown degradations not seen during training. In the context of zero-shot inverse problem solvers, TTA has been explicitly introduced in [7, 8], where the titles themselves indicate that the methods are designed to address or challenge this task. For example, in [8], the authors treat the already pretrained generative model as a DIP model and introduce additional LoRA parameters, which are then optimized during inference. OOD adaptation is beyond the scope of this work, and we believe it warrants dedicated investigation in a separate study, as we outlined in our discussion of future work.

[1]. SUD $\^2$: Supervision by Denoising Diffusion Models for Image Reconstruction (chan2023sud)

[2]. Training-free linear image inverses via flows (pokle2024training)

[3]. **Understanding... Training-free Loss-based Diffusion Guidance (shen2024understanding)

[4]. TFG-Flow: Training-free Guidance in Multimodal Generative Flow (lin2025tfg)

[5]. Loss-guided diffusion models for plug-and-play controllable generation (song2023loss)

[6]. Zero-shot image restoration using ... null-space model (wang2023zero)

[7]. Steerable Conditional Diffusion for Out-of-Distribution Adaptation in ...Reconstruction (barbano2025steerable)

[8]. Deep diffusion image prior for efficient ood adaptation in 3d inverse problems (chung2024deep)

Dear Reviewer,

We have made continuous efforts, collectively as all authors, to address the issues raised by the reviewer. We sincerely hope that the clarifications and additional experiments we have provided will assist the reviewer in reassessing their evaluation.

We would greatly appreciate it if the reviewer could let us know whether our clarifications and additional results have satisfactorily addressed their concerns. If not, we would be happy to know where and how our response may have fallen short, so that we can further clarify.

Best regards,

The Authors