Consistency Model is an Effective Posterior Sample Approximation for Diffusion Inverse Solvers

Thank you for your detailed review. We are pleased to provide answers to your questions:

W1 The manuscript could benefit from improved clarity and organization

Thanks a lot for the detailed comments and suggestions. We have fixed the typos in the revised version of the paper.

• The $\zeta_t\Delta(f(\mathbb{E}[X_0|X_t]), y)$ should be the gradient of that distance: We changed it into $\zeta_t\nabla\Delta(f(\mathbb{E}[X_0|X_t]), y)$.
• The authors should think about introducing a clear distinction between the $p(X_0|X_t)$ and $p(X|y)$: We have fixed it by calling the $p(X_0|X_t)$ posterior and $p(X|y)$ conditional distribution.
• The abbreviation DPS is used but never introduced: we defined it in page 6 upon its first appearance.

Q1 why $p_{\theta}(y|X_0)$ contains $\theta$?

• Thanks for pointing it out. $p(y|X_0)$ is independent of $\theta$, and hence there should be no subscript. We have fixed it in the revised version.

Thanks for your detailed comments. It appears that there have been some misunderstandings, and we have provided our responses to your questions below.

W1 consistency models since these are quite difficult to train and pre-trained CMs are not widely available

CM is one of the most widely adopted approach to distill diffusion models. Training large scaled CM has been studied extensively (see e.g., [Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models] [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference]), and currently there are several high quality CMs that are easily accessible online:

• CM for ImageNet 64, LSUN Cat and Bedroom 256: https://github.com/openai/consistency_models
• CM for Stable Diffusion 1.5, SD-XL and PixArt alpha: https://github.com/luosiallen/latent-consistency-model

W2 the justification for the method is rather weak

In DPS, we only care about whether the approximated sample follows $p(X_0|X_t)$ or not. As long as the samples follows $p(X_0|X_t)$, DPS estimate the conditional score perfectly (Eq 5), and thus achieve sampling from $p(X_0|y)$. The $p(y|X_0)$ part in $p(y|X_t)$ will be handled by gradient ascent of DPS. Even if $p(y|X_0)=0$ for some initial $t$ and $X_0$, the gradient ascent will make it large for subsequent $t$.

Q1 The presentation of Proposition 3.3 is a bit misleading

Thanks for the comments. We have included $\sigma < \frac{1}{\sqrt{4 \pi e}}$ in the assumption.

Thank you for your detailed review. We are pleased to provide answers to your questions. We hope that our response can clarify the misunderstanding and we kindly ask you to reassess our contribution.

W1 Using CM over the existing inverse problem solving adds a lot of computational burden

We would like to clarify that the additional time and space complexity introduced by using CM is fairly small and should not be considered a significant burden. In Table 4 we have shown that using CM increase time complexity by 30% and increase RAM usage by 1GB. In Table 3 and Figure 5, it is shown that using CM improve DPS a lot, both quantatively and qualitatively.

The mentioned paper [DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models] is a concurrent work, which will not be published formally until Dec 2024. We were unable to use the insights from a paper published after ICLR deadline (and the citation of it is not required by ICLR policy).

W2 CM itself can be used as a good prior for solving inverse problems

[CoSIGN: Few-Step Guidance of ConSIstency Model to Solve General INverse Problems] is a concurrent paper published in Oct 2024. So we could not compare with it before the ICLR deadline. Besides, CoSIGN requires training, while our approach, DPS and other methods that we compare to, are zero-shot. In our opinion, CoSIGN and our approach attempt to solve different problems under different setting, hence should not be compared directly. However, its insight or idea may be helpful in zero-shot setting and futher exploration is an interesting future direction.

W3 More DIS baselines are desired such as DDNM:

Approaches such as DDNM [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model] require $f(.)$ to be linear. While in this paper, we consider more general and complex scenarios where $f(.)$ may be a neural network. To the best of our knowledge, DDNM cannot be used to solve non-linear problems. We have already included six strong baselines such as DPS, FreDOM, MPGD, UGD, LGD, STSL and clearly demonstrated the advantage of our method over the baselines when $f(.)$ is a neural network.

Thanks for your detailed comments. It appears that there have been some misunderstandings, and we have provided our responses to your questions below. We hope that our response can clarify the misunderstanding and you can reassess our contribution.

W1 why in (1) the authors formulate the Markov chain following a VE formulation?

The result of this paper is heavily based on stochastic differential equation (SDE) and probability flow ordinary equation (PF-ODE). The classic literature that introduces SDE and PF-ODE adopts VE notation [Score-Based Generative Modeling through Stochastic Differential Equations] and we follow the tradition in this paper. Besides, the VE and VP formulations are essentially equivalent and can be made interchangable by linear scaling. In popular projects such as Huggingface/diffusers, VP and VE diffusions are implementated in a unified way: VE diffusion solvers (Euler) can be applied to solve a VP diffusion (Stable Diffusion 2.0), and vice versa.

W2 the paper in Section 3.4 discusses that the CM is overfitting to f(.) and proposes an approach to make the framework robust:

We would like to emphasize here that No CM is trained for $f(.)$. Our approach works for the zero-shot setting, which is the main motivation of the paper.

Q1: Is CM trained with knowledge of f(.), or if this is a misunderstanding?

This question is related to the W2, and we would like to clarify that NO new CM is trained with the knowledge of f(.). We use existing unconditional CMs which do not know f(.).

Q2: If the CM is trained with f(.), please explain and justify comparing it to methods like DPS that don't use this information?

Again, CM is not trained with the knowledge $f(.)$.

W3 Lack of thorough literature

Thanks for the pointer. We will cite the reference [Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors] and discuss its relation with our work.

However, we would like to note that reference provided is a concurrent work: it will not be formally published until Dec 2024, which is after the deadline of ICLR 2025. Hence, we think not discussing this paper should not be considered as lack of thorough literature.

W4 The paper needs improvement in presentation

Thanks a lot for the comments and suggestion. We have polished the presentation as follows in the revised version:

• We have removed the notations in the abstract.
• We have added the definition of consistency models in introduction.
• We have included the implementation details of Table 1.
• "neural network operators" means the measurement operators $f(.)$. We have rephrased it as "neural network measurement operators".