Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior

Thank you for your detailed review. We are pleased to provide answers to your questions:

W1 The paper has several grammatical flaws

Thank you for pointing out the grammatical flaws. We have corrected them in the updated paper.

• Captions should be on top: We have moved them to the top.
• Add an arrow to indicate that lower FID is better: We have included an arrow.
• There are many undefined acronyms: We have moved their definition to where they first appear.
• adding the takeaway from each table and image: We have included takeaways so readers can understand the content without having to refer to the explanations in the paper.
• the notation for different math symbols is redundant: We have removed redundant notations.
• The claim is quite extreme: We have modified it into: With Observation I & II, one can conclude that DPS is not an effective conditional score estimator for image restoration.
• "we hypothesis" should be corrected to "we hypothesize.": We have fixed it.
• "(See Appendix C.1)" should not be placed after a sentence.: We have fixed it.
• The first quotation marks throughout the paper are incorrect: We have corrected them.
• citation style: The citation command \citet{} of Natbib, which produces "Chung et al. (2022a); Song et al. (2023c)", is also used in the ICLR 2025 template (See https://github.com/ICLR/Master-Template/blob/05833d63fe48bbf250b144741ea77691018bb328/iclr2025/iclr2025_conference.tex#L163). Additionally, the way we create multiple citations using \citet{} aligns with the Natbib reference sheet (See Page 1 of https://gking.harvard.edu/files/natnotes2.pdf). Also, this citation style appears in outstanding papers of ICLR 2024 [Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors] and [Towards a statistical theory of data selection under weak supervision ]. Therefore, we believe this citation style is appropriate for ICLR.

W2 and Q2: results for inpainting

Thank you for your suggestions. Below we provide the results for the inpainting operator on 1000 ImageNet images. We used a fixed box inpainting mask: a 128x128 rectangle centered at a 512x512 image. We have demonstrated that for the inpainting operator, our two proposed improvements (DMAP & DPS + CSE) also significantly boost the performance of DPS.

Thank you for your detailed review. We are pleased to provide answers to your questions:

W1 the generalizability of the MAP hypothesis to other inverse problems is not explored

Thanks for your suggestions. Below we provide the results for bedroom segmentation. we use 100 LSUN bedroom images and we are running 1000 images experiments. The segmentation model recognizes the layout of bedroom, as described in [Indoor Scene Layout Estimation from a Single Image]. To evaluate the segmentation result, we adopt a different set of metrics from image restoration, including MIoU and FID.

In the table below, we have shown that for segmentation operator, our two proposed improvements (DMAP & DPS + CSE) also boost the performance of DPS significantly.

W2 its performance still lags behind well-trained conditional diffusion models like StableSR

The objective of DPS and this paper is zero-shot (no training data) and few-shot (few training data) inverse problem solving. Despite the performance of DPS and this paper is worse than StableSR, the training cost of DPS and this paper is much smaller than StableSR. More specifically, DPS and DMAP (Improvement I in this paper) requires no training. DPS + CSE (Improvement II in this paper) requires 100-1000 images and 8 GPU hours training. On the other hand, StableSR requires 20,000 images and several days of training. In that sense, StableSR does not dominate DPS or this paper.

The study of zero-shot and few-shot approaches is very important to diffusion model. We believe effective zero-shot and few-shot approaches can greatly broaden the application of pre-trained diffusion models.

W3 why Adam helps DPS is not sufficiently substantiated

Thanks for the suggestion, we will provide a more detailed explaination:

• He et al. (2023) find that Adam improves the performance of DPS. This statement is not reasonable if DPS is a conditional score estimator, because Adam is an adaptive learning rate optimizer. In other words, Adam scales the score $\nabla_{X_{t}}\log p(y|X_t)$ unevenly in different dimensions. This scaling, theoretically, makes the estimation of $\nabla_{X_{t}}\log p(y|X_t)$ inaccurate, and should deteriorate the performance of DPS.
• However, if we assume DPS is solving the MAP in Eq. 16, then Adam, as a general method of gradient ascent, may not harm the performance of DPS. And this explains the phonemena that Adams improve DPS, observed by He et al. (2023).

Q1 experiment with toy distributions

Sure, we provide a toy example below.

• The source distribution $p(X_0)$ is 2D 2-GMM (Gaussian Mixture Model). The centers are $(-1,-1),(+1,+1)$ and the diagonal $\sigma_0=0.3$.
• The operator f is inpainting. More specifically, the second dimension of the random variable is set to 0. The measurement $y=(0.5,0)$
• The diffusion is 100 steps karras diffusion with $\sigma_{max} = 4$ and $\rho=7$.
• The diffusion scheduler is Euler Ancestral, and the DPS $\zeta_t=0.05$. The source distribution, true posterior and the DPS result is shown in this anonymous link: https://ibb.co/T2VSn32 It is shown that the true posterior is a two mode distribution. However, the samples of DPS cover only one mode of true posterior. In that example, the behaviour of DPS is closer to MAP.

Despite DPS+DGS being already MAP under Assumption 5, our method remains to be MAP without it. Therefore, DSG alone can not replace DMAP. And practically, its performance is inferior to DMAP as shown in Table 4.

With Assumption 5.1, the original DPS is equivalent to classifier-based guidance (Dhariwal 2021). However, what is classifier-based guidance on earth theoretically is not yet well explored. Therefore, we leave the strict theoretical relationship between original DPS and MAP to future works. Once this problem is solved, the nature of classifier-based guidance can be revealed. We believe this problem is beyond the scope of a paper focused on DPS.

W2 ...clarifying the reason why it works in MAP estimate...

We can think DPS as a simplification of DMAP, with one step gradient ascent and without spherical projection. As long as the step size of diffusion model is small, the $\log p(y|X_{t-1})$ can be locally linear (Assumption 5.2). In that case, one step of gradient ascent is enough. As long as the DPS guidance step is much smaller than the DDPM step, the distance from $X_{t-1}$ to the sphere surface $\mathcal{S}(\mathbb{E}[X_{t-1}|X_t],\sqrt{n}\sigma_t)$ is small. In that case, spherical projection is no longer necessary. And when those conditions are approximately satisfied, DPS also works well.

Q1 ...MAP estimate is the main reason for low-diversity generation..

Yes. The MAP estimate always go to the highest mode. When the distribution is multi-modal, other smaller modes will be neglected by MAP. And therefore, MAP causes low-diversity generation. This phenomena is also shown in the toy example.

Thank you for your detailed review. We are pleased to provide answers to your questions:

W1 ...theoretical insights will be strengthen their claim and more naturally lead to their MAP hypothesis...

Currently, we have not figured out how to directly derive MAP from the original DPS (Chung et al., 2022a). To compensate for this, we additionally provide a toy example, and derive MAP from a variant of DPS.

To better connect the original DPS and MAP, we provide a toy example below:

• The source distribution $p(X_0)$ is 2D 2-GMM (Gaussian Mixture Model). The centers are $(-1,-1),(+1,+1)$ with the diagonal $\sigma_0=0.3$.
• The operator $f(.)$ is inpainting. More specifically, the second dimension of the random variable is set to 0. And the measurement $y=(0.5,0)$
• The diffusion process uses 100 steps of Karras diffusion with $\sigma_{max} = 4$ and $\rho=7$.
• The diffusion scheduler is Euler Ancestral, and the original DPS $\zeta_t=0.05$.
• The source distribution, true posterior and the original DPS result is shown in this anonymous link: https://ibb.co/T2VSn32
• It is shown that the true posterior is a bimodal distribution. However, the samples from the original DPS cover only one mode of the true posterior. In this example, the behavior of the original DPS is closer to that of MAP.

On the other hand, with additional assumptions, it is possible to derive MAP directly from a variant of DPS: DSG (Yang et al., 2024). More specifically, DSG improves DPS with a spherical projection. The update of DSG is as follows:

Assumption 5. Additionally, we need some assumptions:

• 1.The posterior mean $\mathbb{E}[X_0|X_t]$ is a perfect approximation to posterior samples from $p(X_0|X_t)$. In that case, the approximation error in Eq. (7)(8) diminishes, i.e., $$p_{\theta}(y|X_t) = p(y|X_0=\mathbb{E}[X_0|X_t]) = \exp{-\zeta_t||f(\mathbb{E}[X_0|X_t]) - y||}.$$
• 2.The $\sigma_t^2$ in Eq. (5) is so small that $\log p(y|X_{t-1})$ is locally linear in range of $\sqrt{n}\sigma_t$. The $X_t, X_{t-1}$ is so close that $\nabla_{X_{t-1}}\log p(y|X_{t-1}) \approx \nabla_{X_t}\log p(y|X_{t})$.

Under those assumptions, we can show that DSG is also a MAP: Proposition 6. The DSG solves the MAP problem $X_{t-1} \leftarrow \arg\max p_{\theta}(X_{t-1}|X_t,y)$ in Hypothesis 2 when dimension $n$ is high.

• proof: We first consider a gradient ascent with learning rate $\alpha_t$: $$X_{t-1} = \mathbb{E}[X_{t-1}|X_t] + \alpha_t \nabla_{X_t}\log p(y|X_t).$$ As we have assumed in Assumption 5.2, the $\log p(y|X_t)$ is locally linear and $\nabla_{X_{t-1}}\log p(y|X_{t-1}) \approx \nabla_{X_t}\log p(y|X_{t})$. As gradient is the direct of steepest descent, when $\alpha_t$ changes, the $X_{t-1} = \mathbb{E}[X_{t-1}|X_t] + \alpha_t \nabla_{X_t}\log p(y|X_t)$ is the maximizer of $\log p(y|X_{t-1})$, given the distance from the point $X_{t-1}$ to center $\mathbb{E}[X_{t-1}|X_t]$ is fixed. On the other hand, the DSG can be written as the above gradient ascent $$X_{t-1} = \mathbb{E}[X_{t-1}|X_t] - \sqrt{n}\sigma_t\frac{\nabla_{X_t}||f(\mathbb{E}[X_0|X_t]) - y||}{||\nabla_{X_t}||f(\mathbb{E}[X_0|X_t]) - y||||} \\ \overset{a}{=} \mathbb{E}[X_{t-1}|X_t] + \frac{\sqrt{n}\sigma_t}{||\nabla_{X_t}f(\mathbb{E}[X_0|X_t]) - y||||}\nabla_{X_t}\log p(y|X_{t}).$$ where (a) holds due to Assumption 5.1. We further notice that DSG is equivalent to the above gradient ascent. Further, from the update procedure of DSG, the distance of $X_{t-1}$ to $\mathbb{E}[X_0|X_t]$ is $\sqrt{n}\sigma_t$. Therefore, the result of DSG is the minimizer of $\nabla_{X_{t-1}}p(y|X_{t-1})$, on the sphere surface centered at $\mathbb{E}[X_{t-1}|X_t]$ with radius $\sqrt{n}\sigma_t$: $$X_{t-1} \leftarrow \arg\max \log p(y|X_{t-1}), X_{t-1} \in \mathcal{S}(\mathbb{E}[X_{t-1}|X_t],\sqrt{n}\sigma_t),$$ where $\mathcal{S}(\mathbb{E}[X_{t-1}|X_t],\sqrt{n}\sigma_t)$ is the sphere surface centered at $\mathbb{E}[X_{t-1}|X_t]$ with radius $\sqrt{n}\sigma_t$. Besides, from (Yang et al., 2024), we know that as dimension $n\rightarrow \infty$, $X_{t-1}\sim p(X_{t-1}|X_t)$ is equivalent to $X_{t-1}\in \mathcal{S}(\mathbb{E}[X_{t-1}|X_t],\sqrt{n}\sigma_t)$. Then as dimension $n\rightarrow \infty$, DSG+DPS is eqvalient to: $$X_{t-1} \leftarrow \arg\max \log p(y|X_{t-1}), X_{t-1}~p(X_{t-1}|X_t),$$ which is again equivalent to Hypothesis 2 by Proposition 3. $\square$