Improving Denoising Diffusion with Efficient Conditional Entropy Reduction

Dear Reviewer,

Thank you for your meticulous feedback. Below are our specific responses:

1. Theoretical Complexity and Readability

We've added intuitive explanations to improve manuscript readability.

Subtitles for Clarity:

• In Section 3.2, we have added subtitles at the beginning of key subsections to summarize their content succinctly. We have also elaborated on the relationships between different concepts and provided explanations of the assumptions, including their purposes and limitations, to help readers better contextualize our work. We have highlighted the main revisions using color coding.
• Relocating Proofs to the Appendix: To make our manuscript more accessible, we have moved the detailed proofs in Section 3.2 to the appendix. References to the proofs are provided at key points in the text for readers who wish to examine them.
• Enhancing Intuitive Understanding: We have included more intuitive explanations of the propositions where appropriate to lower the barrier to understanding.
• Improved Organization of Key Content: To aid readers in better grasping the content of Propositions 3.1 and 3.2, we have introduced a key statement of Variance-Driven Conditional Entropy Reduction in Gradient-Based Iterations. This title provides a clear structure, helping readers quickly locate and comprehend the foundation of these propositions.
• Explanations of Core Concepts: We have supplemented the main text with brief explanations of key concepts such as conditional entropy and gradient estimation, making the paper more accessible to readers with diverse backgrounds.

2.Correlation between conditional entropy and denoising quality

From an information-theoretic perspective, conditional entropy provides a fundamental measure of denoising performance. Low conditional entropy signifies effective noise removal, as it indicates a more concentrated probability distribution of the output image when a denoising model successfully reconstructs the original image from a noisy input. Specifically, an efficient denoising algorithm produces an output with low conditional entropy, meaning minimal uncertainty given the noisy input. Conversely, when the algorithm fails to effectively remove noise, the output image exhibits higher conditional entropy, reflecting greater uncertainty and lower image recovery accuracy. In diffusion models, each reverse conditional transition distribution represents the probability $p(x_t | x_{t+1}, x_0)$. Under the Gaussian assumption, the conditional entropy is fundamentally determined by the conditional variance. Theoretically, a smaller conditional variance implies that $x_t$ is closer to both $x_{t+1}$ and $x_0$, which directly correlates with improved denoising quality at each iterative step.

Regarding evaluation metrics, for fairness, we follow the standard FID and NFE metrics used by previous works to measure sampling performance, more specialized metrics will be worth exploring in the future. We have detailed these metrics in Appendix D4.

3. Diversity of Experimental Validation and Adaptability of the Method Thank you for pointing out the limitations in our experimental validation. Although our original manuscript already included comparative experiments on CelebA, in response to your suggestion, we have expanded our method's comparative experiments on Stable Diffusion and provided comparisons with the most advanced training-free sampling methods in text-to-image generation in the main text, with more detailed comparisons available in Appendix E. Furthermore, we conducted extensive ablation studies on ImageNet-256. Our research demonstrates that the variance-driven conditional entropy reduction iteration exhibits robustness across parameters and validates our method's consistent superiority over the most advanced sampling methods under various conditions. This proves that our proposed approach not only has theoretical potential for achieving superior performance but also attains SOTA sampling performance in practical comparisons, consistently improving training-free sampling methods for diffusion models. Additionally, our paper verifies that the variance-driven conditional entropy reduction iteration consistently improves sampling performance from different parameterization perspectives, such as noise-based, data-based, and DPM-Solver-v3-based approaches.

4. Computational Cost and Efficiency Analysis

The iterative approach based on RE-based sampling we propose is a purely training-free method, meaning that no additional high-cost optimization or training overhead is introduced. The RE algorithm is a standard iterative method, where the parameter introduced to reduce conditional variance is obtained in a single step, without incurring any additional optimization costs. Overall, the computational efficiency of our proposed method is on par with the DPM-Solver++ benchmark.

Once again, we sincerely appreciate your valuable feedback.

Dear Reviewer,

We sincerely thank you for your constructive feedback. Below are our detailed responses to the points you raised:

1. Improving clarity

We have enhanced the clarity of Sec. 3.2. Specifically, we added subtitles to summarize each major topic, clarified several definitions and relationships, and utilized the freed-up space to improve the completeness of the experimental descriptions, with more detailed provided in Appendix (App.) D. Following your suggestion, we moved a proof from Sec. 3.2 to App. B1 and used the additional space to include pseudocode for variance-driven conditional entropy reduction iterations, as shown in Algorithm 1 on page 8. Further, we added the conclusion to better highlight our contributions.

2. Conditional entropy

To introduce conditional entropy more clearly, we have refined the explanation of its connection with denoising iterations in as much detail as possible. Under the rules of denoising iterations, the reduction of conditional entropy aligns with the goal of minimizing reconstruction error. Specifically, under the denoising framework, the direction of efficient conditional entropy reduction corresponds to the direction of minimizing reconstruction error.

3.  Method implementation

We present two approaches:

Prior-Based Approach. By leveraging the known noise levels of the model at each time step, we utilize the prior variance of these noise levels to enhance traditional gradient-based iterations. As demonstrated in our work, the DPM-Solver iterations distinguish themselves from conventional methods by employing logSNR to balance the conditional variance during iterations. Our approach focuses primarily on reducing or balancing the conditional variance of the current iteration mechanism, while being largely indifferent to the specific space in which the Taylor approximation is performed. In page 26, we provide two alternative forms for reducing conditional variance (Eq. (D.13) and Eq. (D.14)), and our empirical findings show these methods outperform traditional gradient-based iterations. Additionally, we offer an implementation of this prior-based approach in App. D6.

Non-Prior Approach (Core Contribution). Our second approach represents a more substantial improvement over prior-based methods and constitutes the core practical contribution of our work. We overcame the challenge of formulating an optimization objective that liberates the mechanism from prior assumption cases. Specifically, we identified and captured the key parameters that control the reduction of conditional variance. Then, we developed an optimization objective for these critical parameters aimed at minimizing conditional variance. As detailed in Section 4.3, we improved these key parameters by: Minimizing the difference in states; Balancing differential approximations of gradient. We determined the parameter $\zeta_i$ by minimizing the difference between the approximated states $x_{t1}$ and $x_{t2}$ (obtained from forward and reverse processes) and the true state at time t. Fundamentally, this minimization reduces the conditional entropy relative to the true state and clean data. Simultaneously, minimizing the difference between the estimated and true gradients essentially reduces the error in gradient estimation. More detailed content can be found in App. D7 and Sec. 4.3.
Given that conventional optimization algorithms typically require multiple iterative refinements, which can burden sampling efficiency, we transform the closed-form solution available in special cases to directly substitute the final parameters, thereby avoiding additional optimization overhead. Our experimental results validate the consistent improvements of this approach. Compared to baseline methods like DPM-Solver++ and other advanced solvers, our variance-driven, conditional entropy reduction iteration achieves SOTA performance across various scenarios on high-dimensional datasets such as ImageNet256.

4-5. Visual comparisons and deterministic nature.

We sincerely thank the reviewer for their encouraging feedback, which has motivated us to further refine this work. We have added more visual comparisons in Figure 1 and App. E to address the reviewer's concern. Regarding the observation that our method may differ from DDIM, we acknowledge that such differences are expected, as our method introduces variance-driven conditional entropy reduction iterations. While these differences may lead to structural variations, the generated images remain deterministic and high-quality, as shown by the quantitative results in Figs. 9 and 10. We view this methodological diversity as a potential advantage, similar to how different solvers may deviate from DDIM in various ways. Perhaps the key consideration is not just the diversity these solvers introduce, but their stability moving forward.

We hope these clarifications address your concerns. Thank you once again.

Dear Reviewer,

Thank you for your comments. Below are our detailed responses to the points you raised:

Presentation and Clarity

In our revised manuscript, we have made the following efforts:

• We have revised the manuscript for better readability and clarity. Such as, we provided a subtitle at the beginning of some core content in Sec. 3.2 to appropriately summarize its content and highlighted our revisions using color coding for easy identification.
• We have moved some proofs to Appendix. With the space saved and in line with ICLR's new 10-page limit, we have expanded the discussion in the experimental section, with more detailed content provided in the added Appendix D. We added the paper's conclusion to improve the clarity of our research contribution.

Gaussian Assumption

• We have clarified that $x_{t_i}$ is conditioned on both $x_{t_{i+1}}$ and $x_0$, and explained that $x_{t_i}|x_{t_{i+1}}$ is shorthand for this conditional relationship, as seen in page 4. In fact, our earlier version was driven by this assumption, as used in our proof in the appendix.

• Based on Eq. (3.10), we have modified the description of "conditional entropy reduction" to "variance-driven conditional entropy reduction" in main text to better clarify our contribution.

The reversed flow ODE and Discretization

We agree with your observation that analyzing the reversed flow using continuous ODE properties in a discretized setting is not rigorous. Clearly, the primary focus of this paper is not on the properties of the ODE in its continuous form. Instead, we focus on the denoising iterative errors introduced by the discretization process. We only state that ODEs provide a principled mechanism. Our analysis centers on the gradient-based denoising process. Our motivation is that gradient-based iterations suffer from errors. In fact, our viewpoint is consistent. Our core contribution is to reduce these iterative errors by lowering the conditional variance, which improves the denoising process. Our experiments validate the proposed solution.

Independence Assumption in the Reversed Flow

Thank you for pointing out the limitations of the noise independence assumption. In our revisions, we have clarified that the neural network in the model shares parameters across different timesteps, which may introduce correlations in the predicted noise. However, relevant works, such as Song et al. [1], analyze the theoretical gap between models that assume independent noise and those that account for parameter-sharing-induced correlations. Their work justifies the use of the independence assumption as a reasonable approximation, showing that while there may be some performance difference, the impact of this simplification is generally small. This suggests that the independence assumption can still be a useful surrogate for theoretical analysis, especially given its alignment with training objectives like the mean squared error loss. To prevent any potential misunderstandings, we have explicitly stated the limitations of this assumption in the revised manuscript.

We are truly grateful for your comments, which have contributed to improving the clarity and rigor of our work. We believe the revisions, particularly regarding the noise independence assumption and the theoretical considerations, have strengthened the manuscript and addressed the concerns you raised. We sincerely hope that the changes we have made are now clearer and more convincing, and we deeply appreciate the time and effort you have invested in reviewing our paper.

[1]. Song et al., Denoising diffusion implicit models, ICLR 2021.

Dear Reviewer,

We would like to sincerely thank you for your constructive feedback and positive support for our work, which has been truly inspiring to us. Below are our detailed responses to the points you raised:

1.The motivation for reducing entropy

Under the Gaussian assumption framework, reducing conditional entropy is equivalent to reducing conditional variance. In diffusion models, since the conditional distribution is conditioned on the clean data, reducing the conditional variance implies reducing the distance from the clean data. This suggests that reducing conditional variance has the potential to improve the sampling process. Moreover, based on our analysis and observations, the DPM-Solver iteration, which differs from traditional iterations based on Taylor series truncation, essentially represents an iteration that reduces conditional variance. This is why DPM-Solver demonstrates superior performance compared to traditional gradient-based iterations. Motivated by these theoretical insights and empirical observations, our research is driven by the question of whether there exists a more efficient way to reduce conditional entropy during the iterative process. As observed in our experiments, our variance-driven conditional entropy reduction iteration differs from other iterative methods. Interestingly, it (the RE iteration) appears to be dedicated to ensuring the completeness, standardization, and clarity of objectives at each NFE stage, as illustrated in Figure 9 of the appendix.

2. This paper needs to compare with other distillation methods such as CM [1] or LCM [2]

We greatly appreciate your concern, which provides us an opportunity to clarify our work. In response to your comment, we have made an effort to briefly summarize the development of diffusion models and their efficient variants in Appendix A (under the "Related Work" section). While this may not be exhaustive, it is sufficient to clarify the positioning of our work. In general, CM and LCM are efficient models built on top of diffusion models, relying on training or re-training processes. Typically, such training-based efficient models require prior knowledge, often provided by the original diffusion model acting as a "teacher." This prior knowledge is then obtained by sampling using a training-free sampling method based on diffusion models. In contrast, our work focuses specifically on the training-free sampling methods in diffusion models, as stated in the abstract. Given the significant differences in cost and mechanisms between these two types, a direct comparison of these sampling methods would not be fair. Future research might explore applying our insights to knowledge distillation.

3.More results with different CFG should be desirable.

We truly value your professional suggestion. In Appendix, Tables 5 and 6, we have conducted ablation experiments under different CFG settings. Compared to the most advanced methods in the same category, our approach consistently shows improvements across various scenarios. We also performed ablation on the displacement parameters within our iterative method to validate its robustness. Additionally, we list the best results obtained on our server for each method when $s=2$ in the CFG as follows:

4. Can this method be applied to image editing tasks with inversion?

Honestly, we are not deeply familiar with this specific area. However, our method is compatible with Stable Diffusion. Perhaps techniques from Stable Diffusion could be adapted to image editing tasks involving inversion.

We are once again grateful for your encouragement. It gave us a sense of hope, as if we caught a glimpse of light at a particular moment, and we are truly thankful. We hope these clarifications address your concerns.