Inverse Engineering Diffusion: Deriving Variance Schedules with Rationale

We sincerely thank the reviewer for their thoughtful feedback, which highlights areas for improvement while acknowledging the strengths of our work. Below, we provide detailed responses to the reviewers comments and questions:

(i) Weakness 2:

• Our main contribution is the unification of variance scheduling, variance distribution, and loss weighting into a single framework via the concept of rationales. This eliminates the need for loss weighting, as demonstrated in Equations (22-24), and offers theoretical insights into curated loss weighting schemes from prior works [1, 2, 3].
• While these individual steps may appear intuitive, the comprehensive integration of these elements into a rationale-based framework is, to the best of our knowledge, novel and has not been previously explored.
• We also want to state that the generalized inverse of the CDF can be used to define variance schedules that would have been unattainable through conventional methods. To the best of our knowledge, this is a novelty in the context of diffusion models. This allows us to explore rationales such as the squared L2-norm, which yields faster convergence across both datasets.

We have made changes to the introductory section of the paper to better clarify these novelties, see changes on lines 065-067, page 2.

[1] Karras et al. "Elucidating the Design Space of Diffusion-Based Generative Models", NeurIPS, 2022.
[2] Song, Yang et al. "Consistency Models" ICML, 2023.
[3] Song, Yang et al. "Improved Techniques for Training Consistency Models" ICLR, 2023.

(ii) Weakness 3 and Question 1:

• To address the reviewers request to consider more schedules from prior literature, we expanded our evaluation to EDM schedules from Karras et al. [1] (see also Appendix C.4 for respective rationale) with two common hyperparameter-selections $\rho=5$ and $\rho=7$. In the following table, we display the results. For detailed plots, please consider Figures 6 & 7 in the updated document.

CIFAR-10:

ImageNet-32:

Results confirmed the superior sampling properties of our proposed rationales. These results are now included in Appendix D.3 and D.4 (see pages 17 & 18).

• We agree that the differences in IS scores on CIFAR-10 are within the margins of their respective standard deviations. However, on ImageNet-32, the improvements are statistically robust. For clarity, we bolded the highest mean values, and results with overlapping standard deviations were marked explicitly with an asterisk in each table. We are open to suggestions for alternative highlighting methods.
• We improved the motivation in sections 3.3.2 and 3.3.3 (pages 5 & 6) respectively, highlighting why these two rationales are of particular interest to driftless diffusion processes:

The squared L2-norm rationale is of particular interest for driftless processes. With this approach, we can ascribe high density to points in time of the diffusion process where distributions of different originating means (i.e., $x(0)$) should yield high separability. Oversampling these regions could lead to more detailed samples.

The Gaussian rationale is of particular interest for a parameterized variance schedule that can be tuned to different data-domains. The normal distribution is commonly used to model probabilistic processes due to its good generalization properties in most use-cases.

(iii) Weakness 1:

• We thank the reviewer for acknowledging the practicality of our focus on unnormalized density-based variance scheduling. We aimed to thoroughly introduce the concept of the rationale in section 3.1, featuring all steps from the rationale to its respective PDF & CDF and the respective variance schedule (see also Figure 1 for an overview of the relations between steps). We also provided examples for the derivation of variance schedules from rationales in 3.3.2, 3.3.3 and in Appendix C. If any aspects of the rationale’s presentation remain unclear, we would appreciate additional clarification to address these points more effectively.

Lastly, we would like to point out that the computational expense of our method is negligible. The required values can be precomputed, and sampling during training can be efficiently implemented using optimized algorithms, such as np.random.choice. During inference, sigma values are precomputed once and reused, resulting in no significant computational cost.

We hope these clarifications sufficiently address the reviewer's concerns and highlight the robustness and novelty of our work. We sincerely thank the reviewer again for the valuable feedback and kindly ask the reviewer to reconsider their initial assessment.

We sincerely thank the reviewer for their thoughtful feedback, which highlights the novelty and unifying perspective our work. The reviewer raises questions regarding the sensitivity of hyperparamters in rationales. Below, we provide detailed responses to the reviewers comments and questions:

(i) Weakness 1 and Question 1:

• We appreciate the reviewer highlighting the typos, which have been corrected and we thank the reviewer for helping improve the clarity of our work.

(ii) Weakness 2, 3 and Question 1:

• Parameter tuning for the normal distribution rationale was performed through sensitivity analysis regarding the hyper-parameter $\sigma_{\mathcal{N}}$ (see Appendix D.1), where we found an optimal $\sigma_\mathcal{N} = 5$ on CIFAR-10. For consistency, $\sigma_\text{max}$ was set to $3\sigma_\mathcal{N}$, encompassing 99.73% of the density.
• The squared L2-Norm rationale has minimal parameterization, apart from $\sigma_\text{max}$, which we matched to the VE rationale for direct comparison. As shown in Figures (3,4,6,7), despite similar density patterns (see Figure 2), training with the squared L2-Norm rationale demonstrated superior convergence on ImageNet in particular, showcasing its practical effectiveness. The parameterization of the VE schedule follows the prior work of Song et al. [1], which achieved the best overall FID results in their experiments.

[1] Song, Yang et al. "Score-Based Generative Modeling through Stochastic Differential Equations" ICLR, 2021.

(iii) Question 2:

• Our approach does not introduce significant computational overhead. Equation (9) provides the general method for sampling variance distributions, in practice, we can rely on efficient implementations such as np.random.choice, which samples from discrete probabilities (normalized discretization within the considered interval) in negligible time. Training-time complexity remains unaffected, as sampling during training is computationally negligible relative to the overall training time. For inference, precomputed sigma values eliminate additional costs entirely.

(iv) Question 3:

• We recommend a grid-search search for hyperparameter tuning of rationales that rely on hyperparameters, as demonstrated in Appendix D.1. This ensures that model performance can be optimized across diverse settings.

We hope these clarifications sufficiently address the reviewer's concerns about our work. We sincerely thank the reviewer again for the valuable feedback and kindly ask the reviewer to reconsider their initial assessment.

We hope these clarifications sufficiently address the reviewer's concerns about our work. We sincerely thank the reviewer again for the valuable feedback and kindly ask the reviewer to reconsider their initial assessment.

We sincerely thank the reviewer for their thoughtful feedback, which focuses on the statistical significance of the experiments and computational cost. Below, we provide detailed responses to the reviewers comments and questions:

(i) Weakness 1 and Question 1:

• Our framework requires only a probability density function and its corresponding cumulative distribution function for variance scheduling, which can be implemented straightforward using Equations (5) and (6). The rationale-based approach is designed to be intuitive while providing greater control over the resulting variance schedule.
• Even when the inverse of Equation (6) is not available in closed-form, equation (9) ensures efficient approximation of the true inverse function via discretization. Sampling during training can be implemented with highly optimized algorithms, such as np.random.choice, which evaluates sampling in a negligible amount of time relative to the GPU-days needed for training. For inference, sigma values are pre-computed once and reused, incurring no significant computational cost.

(ii) Weakness 2:

• Overfitting is a common challenge in generative models. A faster-converging training, achieved via the proposed squared L2-Norm rationale, may lead to earlier overfitting. However, this behavior is offset by its efficiency in achieving high-quality results within fewer optimization steps.
• We emphasize that the squared L2-Norm rationale performed consistently well across datasets and should be regarded as an efficient option rather than a limitation, delivering higher image quality within fewer optimization steps.

(iii) Weakness 3 and Question 2:

• Due to time constraints and the overall computational cost of training diffusion models, we restricted our experiments to CIFAR-10 and ImageNet-32, which are standard benchmarks in diffusion models. These datasets capture a range of complexities and serve as strong baselines.
• To address the reviewers concern about statistical significance, we conducted additional experiments with repeated, distinct samplings and report the average FID, as well as standard deviations of FID scores. These results (see updated Figures 6 & 7 and Tables 3 & 4 in Appendix D.3 & D.4) confirm the statistical robustness of our findings w.r.t. FID. We also extended the evaluation of prior work to feature samplings with EDM schedules [1], highlighting the efficacy of our methods.

CIFAR-10:

ImageNet-32:

We further changed the presentation in all tables to improve the interpretability of results (see also Tables 1 & 2). Results with overlapping standard deviations were marked explicitly with an asterisk. Hightlithing these overlaps also emphasizes that the squared L2-Norm rationale consistently led to faster convergence for ImageNet-32 in particular, as the only Inception Scores with overlaps originated from using the squared L2-norm rationale during ImageNet training (see Table 2).

[1] Karras et al. "Elucidating the Design Space of Diffusion-Based Generative Models", NeurIPS, 2022.

(iv) Weakness 4:

• We thank the reviewer for noting the typos. We have corrected the errors on lines 132 and 478, along with other minor issues throughout the text to improve clarity and presentation.