Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs

We thank the reviewer for the useful feedback. We address the points that were raised below.

Presentation and clarity. We agree that a more precise wording at the outset of the paper is useful to avoid confusion. In the abstract, we now refer to 'training-free guided generation' instead of the more generic 'conditional generation'. This class of methods is, in principle, applicable to other types of conditions, like text-conditional generation [1], although we focus on the linear inverse problem setting.

Typographical errors. Thank you for raising our attention to these. We have gone through the text to make sure the typography is correct and removed typos and inconsistencies in notation.

Theoretical bound. While a complete theoretical error analysis is beyond our current scope, the question prompted us to add several important insights to the paper: In Appendix G, we analyse the convergence of the error in covariance estimation with increasing step count, showing that stochastic solvers indeed converge towards the true covariance. In contrast, for an ODE solver the curvature of the ODE sets an upper bound on the amount of covariance information attainable. We also show that there is a clear progression of increasing accuracy going from:

1. An identity covariance $Cov[x_0|x_t]=\frac{\sigma(t)^2}{1+\sigma(t)^2}I$
2. Our method, but using only the time updates
3. Our method with time+space updates.

Another potential theoretical approach to analyse the method would be to focus on the simplified case of no space updates and derive bounds on the errors introduced. This could be an interesting direction for future research.

We hope that this answers the concerns of the reviewer, and are open to any further questions and discussion.

References:
[1] Song et al. "Loss-guided diffusion models for plug-and-play controllable generation."

We thank the reviewer for their constructive feedback. We address the points that were raised below.

Choice of comparison methods. Our focus on reconstruction guidance-based methods was motivated by several factors:

1. Direct evaluation of covariance estimation: These methods differ primarily in their covariance approximation strategies, allowing us to isolate the impact of our contribution
2. Theoretical connection: Our method builds on and extends the reconstruction guidance framework
3. Broad applicability: The insights from our covariance estimation technique could potentially benefit other method classes as well, and direct comparisons to them might not be as insightful.

To provide broader context, we have now added comprehensive comparisons with DDNM+ [1] and DiffPIR [2] in Appendix H. We include results with extended step counts: 15, 30, 50, and 100 steps with both Heun and Euler solvers. Breakdown of the new results:

• Our method outperforms the others with low step counts, and remains the best or second-best in LPIPS scores even at 100 steps, and competitive for the PSNR and SSIM metrics.
• Note that DDNM+ is designed to enforce consistency with the measurement in cases where the measurement operator has a clearly defined nullspace, potentially driving the good performance in inpainting and super-resolution. In contrast, the reconstruction guidance methods perform consistently better on deblurring.

Lack of experiment on matching true covariance. We have added a detailed quantitative analysis of the error of different covariance approximations for a 2D example in Appendix F. A highlight is that with a stochastic sampler, the accuracy improves with larger step counts. This is because low-rank updates are done in all directions due to noise in the sample path. In contrast, a deterministic solver has a minimum level of error it can reach.

Other applications of covariances. We thank the reviewer for bringing these very relevant application areas to our attention. We have now added discussion to them in the Related Works section.

In Figure 6, the setting of y is what? Inpainting transform or deblurring? The task we tested on was Gaussian blurring. We optimized the hyperparameters of the methods with hyperparameters on Gaussian blurring. We have updated the paper to explain this.

We hope that this answers the main concerns, and we are open to further discussion.

References:
[1] Wang et al. "Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model."

[2] Zhu et al. "Denoising Diffusion Models for Plug-and-Play Image Restoration."

We thank the reviewer for the positive comments and constructive feedback. They have prompted us to improve the writing provide more empirical evidence for the method. Below are our answers to the concerns and questions.

Theoretical justification for algorithmic choices: There are specific reasons for our choices, and we have now elaborated more on them in Appendix H, with brief summaries in the main text. The reasons for our choices are:

• The DCT basis has a firm grounding in the image processing literature. In the paper, we justified it with some empirical and theoretical results showing that image covariances are approximately diagonal in frequency bases [1]. It has also advantages over the DFT basis: It is purely real-valued, does not assume periodic boundaries, and in practice needs less coefficients to efficiently represent natural images (which is one of the reasons for its use in the JPEG compression standard).
• The BFGS update has the attractive property of preserving positive-semidefiniteness (as opposed to, e.g., the symmetric rank-1 update). This combines well with performing the updates in the denoiser covariance, which is positive-definite (as opposed to the Hessian). Compared to Davidon-Fletcher-Powell (DFP), the difference is that the BFGS update minimizes a weighted Frobenius norm for the size of the update in inverse covariance [2], instead of the covariance directly. In the update formula in Eq.24., if we use A=I and the obseration noise is low, the inverse term is simply the inverse covariance. Thus, it could stabilise the updates across iterations, but this is more speculative, and DFP could work in practice as well.

Inconsistencies in notation and writing. We went through the paper to unify the notation. Indeed, $p(x,t)$, $p(x_t)$, and $p_t(x_t)$ referred to the same thing. We replaced all uses of $p_t(x_t)$ with $p(x_t)$. We still use the $p(x_t)$ vs. $p(x,t)$ notation in the particular case of deriving the time update. Here, it is important to separate out the functional dependence on $t$ and $x$. For the rest of the paper, separating out $x_0$ and $x_t$ is needed instead. We now state this clearly in the text. Having $\mu_{0\mid t}(x_t), \mu_{0\mid t}(x)$, and $\sigma(t), \sigma_t$ were confusing as well, we now use $\mu_{0\mid t}(x_t)$ and $\sigma(t)$ consistently.

Additional experiments. We have now conducted the following additional experiments, extending to the many-step regime (although still in the broad setting of linear inverse problems).

• Results with the Heun solver for an extended number of steps: 15, 30, 50, and 100. Our method outperforms the others with low step counts, and remains the best or second-best in LPIPS scores even at 100 steps, and competitive for the PSNR and SSIM metrics.
• Results with the Euler solver for 15, 30, 50, and 100 steps. The same conclusion holds here as well.
• Both result tables include two new models for comparison: DDNM+ [4] (noise-resilient variant of DDNM) and DiffPIR [5].
  • Especially on the Gaussian deblurring task, our method clearly outperforms DDNM+. In the low-step regime, it outperforms it across almost all tasks, and in the high-step regime, our model still outperforms the others in terms of LPIPS while remaining competitive in the other metrics.
  • Note that DDNM+ is designed to enforce consistency with the measurement in cases where the measurement operator has a clearly defined nullspace. In contrast, the reconstruction guidance-based approaches do not explicitly consider this, explaining the performance gap in inpainting and super-resolution.
• In response to reviewers SLRN and d5jC, we also added a quantitative analysis of the error of different covariance approximations in Appendix F.
• In principle, the method could be used for nonlinear inverse problems in the style of [3]. This would involve developing a method to sample from our low-rank Gaussian posterior approximation for estimating the integral in Eq.23. We consider this a promising direction for future work.

Have you considered training a model to directly estimate the denoiser covariance? The paper focuses on training-free methods, which was motivated by two main considerations:

1. Architectural. Current neural net architectures do not process data in frequency spaces, and as such it is unclear what would be the best way to make them output frequency-diagonal covariances. In contrast, our method is applicable to any basis in a straightforward manner.
2. Ease of use. Our method is applicable to all standard diffusion models, without need for additional training.

The work of [6] does consider trainable reverse covariances, and trains a model using a wavelet basis for the FFHQ dataset.

Comparison to DDNM. See our earlier answer in "Additional experiments".

We thank the reviewer for their positive comments and the constructive suggestions to improve the writing. We address the questions and comments below.

Inconsistent notation, typos, variable definitions. We went through the manuscript to address all the concerns raised, and make notation as consistent as possible. We have highlighted the more substantial changes in the text in green. We made a few exceptions:

• Although we have done significant cleaning up, we still use the $p(x,t)$ notation when deriving the time update, because it is important to separate out the functional dependence on $t$ and $x$. For the rest of the paper, we use $p(x_t)$ since explictly defining the random variables $x_0$ and $x_t$ is needed for defining the posterior $p(x_0\mid x_t)$. We now added a paragraph about this in the beginning of the methods section. We are open to suggestions on alternative notation.
• While we understand the point of minimising numbered equations, we believe that erasing the numbering in all unreferenced equations can be problematic as well. In particular, it may make it more difficult for future work to reference particular parts of the paper. As such, we only removed the labels for equations that clearly serve no other purpose than to lead to another equation, or labeled lines that were not proper equations (an equation split into two lines). We are open to more discussion.

Whether online updates always improve performance. There are a few possible reasons for why they do not always result in improved performance:

• The approximate nature of the Gaussian approximation means that we do not have a strict guarantee that a more accurate covariance always leads to better results in PSNR, SSIM or LPIPS scores. While our conclusion is that large improvements in the accuracy clearly improve performance (comparing the identity base covariance to DCT base covariance), it may also be the case that more modest improvements in covariance estimation only improve results on average across tasks, in contrast to improving every instance.
• The method itself is approximate, and inaccuracies in the finite-difference updates to the covariance could sometimes cause inaccuracies in the covariance estimate.

FH+Online result visualisation. We originally left FH+Online out of the image due to space reasons, but agree that it helps to have it as well. The results are not visually significantly different, although FH+Online does perform better on average. For the 30-step case, the generated images have slightly less details than with FH, possibly explaining the good PSNR and SSIM performance, but slightly worse LPIPS performance of FH+Online.

We thank the reviewer for the positive feedback, but also for the suggestions regarding additional experiments. This motivated us to improve the experiments section. We respond to each of the raised points below.

New experiments. We agreed that the experiments section would be stronger with a more thorough analysis. We have now added the following results in Appendix G:

• Results with the Heun solver for an extended number of steps: 15, 30, 50, and 100. Our method generally outperforms the others with low step counts, and remains the best or second-best in LPIPS scores even at 100 steps, and competitive for the PSNR and SSIM metrics. This suggests tha that while the importance of accurate covariance estimates diminishes with increasing step counts, it does not disappear entirely.
• Results with the Euler solver for 15, 30, 50, and 100 steps. The same conclusion holds here as well.
• Both result tables include two new models: DDNM+ [1] and DiffPIR [2].
  • Note that DDNM+ is designed to enforce consistency with the measurement in cases where the measurement operator has a clearly defined nullspace, and as such performs well in inpainting and super-resolution where this is the case. For deblurring, however, the other methods are more robust.
• In response to reviewers SLRN and d5jC, we also added a quantitative analysis of the error of different covariance approximations in Appendix F. We also did analysis with stochastic solvers in this context, and we show that they have the advantage of converging to the correct covariance with large step counts. For the high-dimensional image experiments, our focus was on fast solvers instead, which is why we focused on deterministic methods that are better at low step counts [9].

Whether linear-Gaussian observation models are realistic. The linear-Gaussian observation model, while seemingly restrictive, has wide applications that have been the subject of much study in inverse problems in different disciplines of science and engineering [3]. Examples include:

• Medical imaging problems, like Computed Tomography [4] and MRI reconstruction [5]
• The field of compressed sensing [6]
• Seismic inversion [7], i.e., determining the physical properties under the surface of the Earth with indirect measurements
• Radio interferometry [8]
• As considered in this paper, image deblurring, inpainting, and super-resolution.

Please, let us know if you have further concerns.

References:

[1] Wang et al. "Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model"

[2] Zhu et al. "Denoising Diffusion Models for Plug-and-Play Image Restoration."

[3] Tarantola, Albert. "Inverse problem theory and methods for model parameter estimation."

[4] Hsieh, Jiang. "Computed tomography: principles, design, artifacts, and recent advances."

[5] Liang, Zhi-Pei, and Paul C. Lauterbur. "Principles of magnetic resonance imaging."

[6] Duarte, Marco F., and Yonina C. Eldar. "Structured compressed sensing: From theory to applications."

[7] Cooke, D. A.; Schneider W. A. "Generalized linear inversion of reflection seismic data."

[8] Wiaux et al. "Compressed sensing imaging techniques for radio interferometry."

[9] Karras et al., "Elucidating the Design Space of Diffusion-Based Generative Models"