Noise Conditional Variational Score Distillation

W1: ... additional datasets ...

Please refer to W7.

W2: ... visual comparison ...

Please refer to W8.

W3: For example ...

Please refer to W9.

W4: In L369 ... & W5: The authors say in L369 ...

We respectfully disagree. The tradeoff between PSNR and LPIPS has been extensively demonstrated in prior works. For instance:

• Figure 14 in DAPS and Figure 5 in RED-diff (Mardani et al., 2024) illustrate this tradeoff.
• Section C.6 in DPS explicitly discusses this phenomenon.

Generative image restoration methods often prioritize perceptual quality, which emphasizes preserving high-frequency details. This focus can lead to lower PSNR, as PSNR penalize deviations from the mean / MMSE solution. However, these deviations often result in better perceptual quality, as captured by metrics like LPIPS.

The claim that our method produces results "closer" to posterior samples refers to its tendency to avoid regressing to the mean solution that maximizes PSNR. To address the reviewer's concern, we will revise the statement as follows:

"Our method tends to generate results that retain more high-frequency details rather than approximate the mean of all possible solutions. This approach typically leads to higher MSE (lower PSNR) but aligns more closely with perceptual quality metrics such as LPIPS."

W6: I think that evaluating ...

To the best of our knowledge, related works such as CD, ECM, EDM2, and sCM primarily use FID as the main evaluation metric, while KID is not commonly reported. Additionally, FD is typically considered an auxiliary metric and is often included in the Appendix. To address reviewer's concern, we provide FD_DINOv2 below:

Table: FD_DINOv2 on ImageNet-512x512.

FD_DINOv2 exhibit similar trends to FID: scaling test-time compute enables our method matches performance of larger-sized sCM models.

W7: The method is evaluated ...

For image generation, we follow the EDM2, focusing on ImageNet-64x64 and ImageNet-512x512, and pretrained EDM2 models for other datasets are unavailable. Moreover, ImageNet-64x64 and ImageNet-512x512 serve as challenging benchmarks for image and latent domain, respectively. So we believe they sufficiently demonstrate our method's effectiveness and generalizability.

For inverse problem solving, pretraining an EDM2 model on ImageNet-256×256 is computationally costly. For example, training an S-size model at 64×64 resolution takes over 5 days on 32 A100 GPUs (Figure 11(a), EDM2). Additionally, the baseline PnP-DM focuses solely on the FFHQ dataset. While our current results already demonstrate the effectiveness of our method, we have also started running experiments on the ImageNet-256, and we expect to report preliminary results by the discussion stage.

W8: There are no visual ...

We thank the reviewer for the suggestion. For image generation, visual samples are in Appendix D, with the classes align to those used in sCM, enabling visual comparisons.

For inverse problems, we include additional visual comparisons (https://anonymous.4open.science/r/ncvsd-D396) in the revised version. These visual examples further demonstrate that our method produce much better perceptual quality and preserve more high frequency details compared to baseline.

W9: The quantitative results ...

We emphasize that achieving SOTA on inverse problems is not the primary goal of this paper, nor do we claim that LPIPS is more important than PSNR. Instead, inverse problem solving serves as a proof-of-concept to showcase the plug-and-play probabilistic inference capabilities of our method. Moreover, we already demonstrate competitive performance against well-established diffusion-based methods with significantly fewer NFEs. In contrast, prior works like CM (Song et al., 2023) on image editing provide only visual results without quantitative comparisons. The focus should be on the novelty and conceptual contributions of our approach rather than solely on performance metrics.

W10: Important quantitative ...

We standardize FID for comparing different methods, following EDM2. This metric is consistently reported in baseline methods, making it a convenient choice for benchmarking. In contrast, metrics like Precision or Recall are either not reported, e.g., ECM, or reported only for indicating the performance tradeoffs, e.g., EDM2 and sCM. To address the reviewer’s concern, we include additional Precision and Recall in the Table below. For additional metrics for inverse problems, please refer to our response to W3 from Reviewer 7Tng.

Table: Precision and Recall on ImageNet-64x64.

Precision is superior, reflecting better quality of generated samples, while the slightly lower recall score reflects the mode-seeking behaviour of reverse KL minimization.

W1: After reviewing this paper ...

Conceptually, the key distinction between diffusion models with iterative refinement and our approach lies in how clean data $x\_0$ is predicted from its noisy counterpart $y\_{\sigma} \sim \mathcal{N}(x\_0, \sigma^2 I)$. Diffusion models primarily focus on learning the MMSE prediction, whereas generative denoisers aim to model the full posterior distribution over $x\_0$.

Regarding the reviewer's concern about whether certain acceleration techniques for diffusion models could achieve similar goals as our proposed method, we argue that significant gaps remain where our method demonstrates clear advantages:

• Image Generation: While acceleration techniques for diffusion models, such as DDIM (Song et al., 2020) or advanced numerical integrators (Lu et al., 2022; Karras et al., 2022), can reduce the required NFEs from 1k (original DDPM) to 10–100, the generative quality degrades significantly when NFEs drop below 10. In contrast, our method achieves state-of-the-art results with just 1–4 NFEs, with performance at 4 NFEs even surpassing that of the teacher diffusion model. Moreover, generative denoisers can theoretically match the data distribution using only 1 NFE, whereas diffusion models require an infinite number of time steps to achieve the same.

• Inverse Problem Solving: Current diffusion-based methods either suffer from irreducible approximation errors by using Dirac (Chung et al., 2022) or Gaussian (Song et al., 2023; Peng et al., 2024) approximations for the denoising posterior, or achieve asymptotically exact sampling at the cost of expensive reverse diffusion simulations (Wu et al., 2024). The former lacks theoretical guarantees, while the latter requires a significant number of NFEs (e.g., 2483 for PnP-DM) and remains affected by discretization errors. In contrast, our method achieves superior results with 20x fewer NFEs, offering both computational efficiency and theoretical robustness by avoiding prior-step errors beyond those introduced by imperfect model training.

W2: Regarding the experimental ...

We evaluate our method on the subset (100 images) of FFHQ dataset, following the standard practices established in diffusion-based restoration methods, e.g., DiffPIR (Zhu et al., 2023), DAPS (Zhang et al., 2024), PnP-DM (Wu et al., 2024), among others. Morevover, test on full validation set is computationally costly for methods like DPS, DAPS, PnP-DM (over 1000 NFE). This alignment also allows us to make a fair a comparison to existing methods with standard evaluation protocols.

For super-resolution tasks, both our method and the baseline diffusion models are trained and evaluated on 256×256 resolution datasets such as FFHQ, following common practice in recent literature. Since existing approaches—including EDM2—have not been trained on 2K-resolution data, we do not include evaluations on high-resolution benchmarks like DIV2K or Flick2K. Nevertheless, our results are consistent with prior work and effectively demonstrate the strength of our approach. Extending the framework to higher-resolution settings remains a promising direction for future research.

W3: I think SSIM ...

We thank the reviewer for the suggestion. The SSIM performance is provided in the table below.

Table. SSIM performance of inverse problem solving on FFHQ dataset.

Similar to PSNR, our method demonstrates competitive performance, being only outperformed by DAPS. However, DAPS achieves this by employing significantly more NFEs and extensive hyperparameter tuning. In contrast, PnP-DM, the most relevant baseline, delivers inferior performance despite utilizing more NFEs than our approach. This table will be included in the revised version to provide a more comprehensive evaluation.

W1：Choice of adaptive ...

We thank the reviewer for pointing out the ambiguity in our claim. A function $f(\cdot)$ is called L-gradient Lipschitz if it satisfies:

$$\lVert \nabla f(x\_1) - \nabla f(x\_2) \rVert\_2 \leq L \lVert x\_1 - x\_2 \rVert\_2, \quad \forall x\_1, x\_2.$$

Provided that $\mathcal{E}$ is L-gradient Lipschitz, the gradient difference of $\tfrac{1}{\beta}\mathcal{E}(\cdot) + \tfrac{1}{2 \sigma^2} \lVert \cdot - u \rVert\_2^2$

$$= \lVert \beta^{-1} (\nabla \mathcal{E}(x\_1) - \nabla \mathcal{E}(x\_2)) + \sigma^{-2} (x\_1 - x\_2) \rVert\_2$$ $$\leq \beta^{-1} \lVert \nabla \mathcal{E}(x\_1) - \nabla \mathcal{E}(x\_2) \rVert\_2 + \sigma^{-2} \lVert x\_1 - x\_2 \rVert\_2$$ $$\leq \beta^{-1} L \lVert x\_1 - x\_2 \rVert\_2 + \sigma^{-2} \lVert x\_1 - x\_2 \rVert\_2$$

Therefore, the postential function is $(\beta^{-1} L + \sigma^{-2})$-gradient Lipschitz.

W2: Optimality of Existing ...

The claim holds in the non-parametric limit. This limitation arises because the loss functions of existing methods do not ensure that $q(x\_0|y)$ is the unique minimizer. For instance, the objective in (Lee et al., 2025) is defined as:

$$\min\_{\theta} -\mathbb{E}\_{\mu\_{\theta}(x\_0|y)} [q(y|x\_0)] + \int w(t) D\_{KL}(p\_{\theta}(x\_t|y) || q(x\_t)) dt,$$

which can only be interpreted as a regularized optimization problem, with no guarantee that $\mu\_{\theta}(x\_0|y) = q(x\_0|y)$ at convergence. Similarily, (Mammadov et. al., 2024) employ ELBO for the prior term (Eq.(14) in the paper) that is also no guarantee. In contrast, NCVSD achieves the optimal iff $\mu\_{\theta}(x\_0|y) = q(x\_0|y)$ (Luo et al., 2023). This property allows marginal-preserving multi-step sampling (Sec. 3.3), and enables the application of the SGS (Sec. 4).

W3: Authors have cited ...

We thank the reviewer for bringing this relevant work to our attention. We are look forward to contributing further to the field of posterior sampling.

W4: The main weakness lies ... & Existing methods that this work ...

We thank the reviewer valuable suggestions. Please refer to W6,W8 for comparisons with PnP-DM and W6,W9 for one-step posterior sampler.

W5: All loss functions ...

We thank the reviewer for the valuable suggestion. We will explicitly include the parameters to be optimized in all loss function to further enhance clarity.

W6: The authors should ...

We appreciate the reviewer’s insightful suggestion. Below, we provide a dedicated background on posterior sampling, which will be included in the revised version by extending Section 2.1:

“Existing posterior sampling methods often involve trade-offs among flexibility, posterior exactness, and computational efficiency. Supervised approaches (e.g., Saharia et al., 2022) lack flexibility as they require retraining for each specific task. Zero-shot methods provide greater adaptability but introduce irreducible errors by approximating the denoising posterior with Dirac (Chung et al., 2022) or Gaussian distributions (Song et al., 2023). Asymptotically exact methods, such as PnP-DM (Wu et al., 2024), ensure asymptotically exact posterior sampling but are computationally intensive, relying on reverse diffusion simulations that demand a large number of NFEs.”

W7: The impact statement ...

We thank the reviewer for valuable suggestion. We will enhance the impact statement to emphasize the advancements in efficient generative modeling and inverse problem solving, and ethical considerations.

W8: What are the main ...

We appreciate the reviewer’s valuable suggestion. The primary distinction between the proposed method and PnP-DM lies exactly in how the prior step is approximated. To address this, we will include the following clarification after L277 in the revised version:

"The proposed method and PnP-DM are both built upon the foundation of SGS. The primary distinction lies in how the prior step is approximated. In PnP-DM, simulating the reverse diffusion process is required, which is not only computationally inefficient but also prone to irreducible discretization errors. In contrast, our approach significantly improves computational efficiency by requiring only one or a few NFEs for the prior step, while being free from any errors beyond those introduced by imperfect model training."

W9: The two prior works ...

We appreciate the reviewer’s valuable suggestion. High-quality sample generation is indeed a key strength of our method compared to prior works. Additionally, our approach offers two significant advantages: accurate posterior sampling (W2), and the flexibility to address a wide range of problems (W6).

In summary, we will incorporate the reviewers' insightful suggestions to further enhance the contributions of our paper. Specifically, we will:

• Highlight the fast sampling (W8).
• Include a dedicated background on posterior sampling (W6).
• Elaborate novelty (W2,W6,W9).
• Emphasize computational advantages (W8).
• Improve impact statement (W7).