A Mixture-Based Framework for Guiding Diffusion Models

Thank you for your thorough review of our paper. We address your main weak points/questions below. The additional tables we discuss below can be found here: https://anonymous.4open.science/r/rebuttal-F9B0/rebuttal_tables.pdf

[...] at least one example with a non-Gaussian likelihood [..].

In the initial version, we prioritized extensive comparisons against multiple methods across two modalities. Following your suggestion, we have conducted preliminary experiments with Poisson noise, comparing our method against DPS due to their similar experimental setup. For MGDM, we directly employed the Poisson likelihood without resorting to the Gaussian approximation and maintained the original hyperparameters. In contrast, we used the Gaussian approximation for DPS as outlined in Eqn. (19)[arXiv version], since the Poisson likelihood proved challenging to implement effectively and the original DPS paper lacks guidelines for this scenario. The results are detailed in Table 6, clearly showing that MGDM outperforms DPS across all considered metrics. Additionally, to address potential concerns regarding MGDM’s performance under higher noise conditions, we have included an extra benchmark with a noise standard deviation of 0.3, presented in Table 5.

Report runtime [...] Add FID as a metric

We have implemented all of your recommendations. Please see Fig. 1 and Tables 1, 2, and 3 in the attached document. We refer to our response to R. E9Pa for comments on our runtime and that of the competitors. Our method consistently achieves competitive FID scores across all three evaluated models.

better hyperparameter choices

PGDM has no official implementation, and directly implementing Algorithm 1 yielded suboptimal results. To ensure a fair comparison, we instead utilized the authors' official RedDiff implementation, where the guidance term is scaled by $\\alpha_{t-1}\\alpha_t$ rather than solely by $\alpha_t$ (lines 70 and 73 in https://github.com/NVlabs/RED-diff/blob/master/algos/pgdm.py, with grad_term_weight=1). This modification improved PGDM's performance across most tasks, except for JPEG2, where the original formulation was superior (line 71 in our pgdm.py). Supporting this observation, Boys et al. (2023, Table 2) and Rozet et al. (2024, Table 4) also reported that PGDM generally underperforms DPS. For a fair comparison, we carefully tuned PGDM at 1000 steps to match compute budgets, despite Rozet et al.'s suggestion that fewer steps (around 100) might yield better results. Even under these conditions, PGDM at best matches DPS performance, reinforcing our conclusion that MGDM consistently outperforms both methods. We stress that we promote the reproduction of all benchmarks (as detailed in Appendix B.4 and our openly accessible codebase), to ease verification by the community.

[...] number of Gibbs sampling steps [...]

In Fig. 3 of the paper, we demonstrate that performance consistently improves as the number of Gibbs steps increases for the audio separation task, surpassing all training-free posterior sampling methods. Furthermore, increasing the number of gradient steps allows our approach to exceed Demucs' performance. Across all experiments, we've consistently observed benefits from increasing Gibbs steps, whereas additional gradient steps yield diminishing returns beyond a certain threshold. Specifically, we selected $R=1$ for the image experiments as it provides the optimal balance between computational efficiency and competitive performance relative to other baselines.

effect of $M$, the number of DDPM steps [...]

We selected $M=20$ as it offered a trade-off between computational cost and image quality. While increasing $M$ enhances image quality, it does not significantly improve posterior exploration, in contrast to Gibbs and gradient steps. However, in response to your recommendation, we conducted additional experiments to investigate how varying $M$ and the number of diffusion steps affect performance; results can be found in Figure 2. Furthermore, regarding the choice of the weight sequence, we already provide in Apdx B.1 an empirical analysis of the strategy of sampling $s$ close to $0$ at every step.

• MC estimate of the KL: We also found this result surprising, given the performance gains observed. Specifically, we discovered that estimating the KL divergence using the same noise as for the likelihood expectation effectively reduces the variance of the gradient estimator, leading to improved results.
• Comparison to MGPS We have prioritized comparing well-established methods and recent contenders. Nevertheless, we commit to comparing with MGPS in the revised version of the paper.
• non-diagonal covariance We have already tested adding a low-rank perturbation to the diagonal covariance matrix but found that it doesn’t significantly improve the result. The most significant improvement comes from using a diagonal instead of a scalar matrix.

Thank you for your thorough review. We address the main weak points/questions below. The additional tables and figures mentioned below can be found here: https://anonymous.4open.science/r/rebuttal-F9B0/rebuttal_tables.pdf

comparison of runtime[...]

Although initially not mentioned (now corrected), we used the maximum diffusion steps (1000) for DPS, PGDM, DDNM, DiffPIR, and PSLD. For methods like RedDiff, DAPS, ReSample, and PnP-DM, we tuned the compute time by increasing Langevin/denoising/optimization steps until performance plateaued, ensuring optimal competitor performance. Hence, we are confident that we have pushed each competitor to their optimal performance limit. Please see Figure 1 for a figure summarizing the runtime and memory costs of each method. This figure will also be included in the revised version of our paper. Note that our method has similar memory demands as DPS and PGDM on pixel space, while in latent space its memory usage aligns with the other methods. Notably, in the latent-diffusion setting—which is particularly relevant given the prevalence of latent-space models—our method outperforms all others in speed while maintaining consistently strong performance across all benchmarks. In pixel space, our method is slower, but this comes with the benefit of consistent and strong performance. Please see our comment to Reviewer E9Pa regarding the runtime of the competitors.

evaluation of scaling the inference-time [..]

We thank the reviewer for the valuable suggestion. We conducted additional experiments analyzing performance evolution with hyperparameters $K, M, G$—representing regular diffusion steps, Gibbs repetitions, denoising steps in Gibbs for $p(x_0|x_s)$, and gradient steps for approximating $q(x_s|x_0,x_t) p(y|g(D(x_s))$, respectively. Specifically, we extended Figure 3 to explore increased compute along different axes for the phase retrieval task (see Figure 2). We concluded that scaling diffusion, denoising, or gradient steps have less impact for this task than increasing Gibbs steps.

model description [...]

Thank you for your suggestion. We acknowledge some parts of our presentation could be more apparent. To improve accessibility, we've reorganized Section 3, which begins with a concise conceptual overview before detailing the algorithm step-by-step. We have also added the main intuition behind latent variables $x_0, x_s$, and $x_t$ and their iterative evolution through denoising, noising, and variational updates.

Why do we introduce the intermediate s-timestep [...]

Our method introduces two latent variables $x_s, x_0$, enabling updates without relying on unrealistic approximations of the denoising distributions. This contrasts with approaches like DAPS, which approximate the posterior distribution $x_0 \mid x_t$ using a Gaussian distribution parameterized by a tunable covariance. Furthermore, our method leverages likelihood approximations derived from earlier diffusion steps $s < t$, a perspective that initially motivated our algorithm. Additionally, framing our approach through the lens of Gibbs sampling provides a novel dimension for performance enhancement. These three aspects—accurate latent variable integration, leveraging historical likelihood approximations, and exploiting Gibbs sampling insights—are currently underexplored and could significantly benefit the research community.

Why is sampling s close to 0 is a source of instabilities?” + Why do we use a mixture of timesteps s instead of a fixed s schedule?

As we explain in Appendix B.1, sampling $s$ close to 0 leads to an accurate likelihood approximation, which is particularly important for latent diffusion. This allows us to fit the observation quickly; however, the unobserved part of the state evolves very slowly, leading to a poor reconstruction of the unobserved part of the state; see Figure 4. This occurs because the prior $q_{s|0, t}(\cdot|x_0, x_t)$ is concentrated around $x_0$ when s is close to 0. Conversely, when $s$ is far from 0, the likelihood approximation is less accurate but the prior becomes less constrained. Using a uniform mixture balances these opposing behaviors effectively, with minimal hyperparameters.

[...]Boys et al. do use the network Jacobian, and as such the covariance is not constant w.r.t. $x_t$

In Boys et al., the likelihood approximation (Eqn. (11), arXiv version) explicitly assumes that the covariance—and consequently the Jacobian of the denoiser—is constant w.r.t $x_t$. Without this assumption, differentiating Eqn. (10) would not lead directly to Eqn. (11). Indeed, Boys et al. explicitly mention: ‘For this reason, we treat the matrix $C_{0|t}$ as constant w.r.t. $x_t$ when computing the gradient.’ Similarly, [2] employs the same assumption (see Eqn. (23), OpenReview version, and the corresponding commentary).

Again, we thank the reviewer for the valuable feedback. Please let us now if you have any further questions.

Thank you for your feedback. We appreciate your acknowledgment of the paper's clarity and the promising nature of our method. Below, we directly address your key points and questions, supported by additional results. The supplementary tables and figures mentioned can be accessed here: https://anonymous.4open.science/r/rebuttal-F9B0/rebuttal_tables.pdf.

Drawbacks in evaluation: 1) Missing evaluation metric FID, and 2) Heavy reliance on a single metric (LPIPS).

First, we refer the reviewer to Appendix B.6, where we show that LPIPS is particularly suitable for our tasks, where reconstructions naturally deviate significantly from the reference images. On the other hand, pixel-based metrics (such as PSNR or SSIM) are inadequate for capturing meaningful perceptual differences in such scenarios, potentially leading to misleading conclusions. For instance, Figure 8 in the original manuscript clearly illustrates this limitation. Although reconstructions from methods like DAPS and DiffPIR appear overly smooth and lack coherence, they nevertheless achieve superior PSNR and SSIM scores, highlighting the inadequacy of these pixel-wise metrics for evaluating our specific use case.

Nevertheless, we acknowledge the importance of evaluating our method with diverse metrics, and therefore we have included FID scores in the revised evaluation. Please refer to Tables 1, 2, and 3 in the provided PDF link for detailed results. These additional results confirm that our method remains highly competitive in terms of FID scores across all three models evaluated.

Could authors provide an analysis on efficiency of the proposed method compared with baselines? For example, runtime and memory cost.

Please see Figure 1 in the linked PDF, which summarizes both runtime and memory consumption across our method and the relevant baselines. This figure will be integrated into the revised manuscript.

Our method has memory requirements similar to DPS and PGDM in pixel space and aligns closely with other methods in latent space. Importantly, in latent diffusion—a highly relevant scenario given the prevalence of latent-space models—our method is notably faster than all competitors while consistently achieving strong performance across benchmarks. Conversely, when operating directly in pixel space, our method exhibits somewhat slower runtimes compared to some alternatives. However, this increase in computational overhead is consistently balanced by improved and stable reconstruction quality across all considered tasks. Thus, we position our method as offering a beneficial trade-off, especially in scenarios where quality and consistency of results are paramount.

We highlight several important points regarding the competitors’ runtime:

• For latent diffusion, DAPS and PnP-DM perform a significant amount of Langevin steps using the gradient of the likelihood. Since the latter involves a vector jacobian product of the decoder, the runtime increases significantly. More generally, when the likelihood function is expensive to evaluate, DAPS and PnP-DM are expected to be much slower than DPS, PGDM and MGDM.

• For PnP-DM on FFHQ, we have implemented the likelihood step exactly on the linear tasks. On ImageNet however, using Langevin steps provided better results and this explains the significant increase in runtime.

Thank you very much for your feedback. Please let us if you have any further question. We kindly refer you to our responses to Reviewers awaZ and SCZR, where we’ve provided additional experiments and metrics that further support our claims.

We thank the reviewer for their constructive comments and helpful suggestions. Below, we directly address the points raised:

1. Gibbs sampling can have convergence issues, even if you run more iterations. How do you address it?

We acknowledge that Gibbs sampling can indeed exhibit convergence challenges, especially in high-dimensional or strongly correlated distributions. In fact, in Appendix B.1 we show that for one specific choice of weight sequence, convergence issues arise. More specifically, when the weight sequence is designed to sample the index $s$ close to $0$ at all iterations, MGDM mixes very slowly due to the high correlation between $x_0$ and $x_s$, since $s \approx 0$. This motivates our approach of using intermediate uniform mixture posteriors, which bypasses these correlation issues and reduce dependence on the precise convergence of each Gibbs iteration, thereby enhancing overall robustness.

2. Is there diversity in generated samples?

Our empirical results confirm that MGDM generates diverse samples. This is qualitatively evident in the visual samples provided in our supplementary materials; see, for example, Figure 2 and Figures 7–12.

3. How to address the scaling issues of the algorithm?

We interpret the reviewer’s question as referring to memory scaling. Our algorithm’s memory footprint is comparable to DPS and PGDM, but higher than methods like DiffPIR and DDNM. This increased memory usage enables superior reconstruction quality, particularly for inpainting and outpainting (Figures 5, 6, and 8 of the original manuscript). Achieving similar quality with lower memory requirements remains an open research question. Importantly, on latent diffusion, our memory footprint is the same as all the other methods.

Thank you very much for your feedback. We kindly refer you to our responses to Reviewers awaZ and SCZR, where we provide additional experiments and metrics that further support our claims.