Constant Rate Schedule: Constant-Rate Distributional Change for Efficient Training and Sampling in Diffusion Models

1. The algorithmic design of this work exhibits several drawbacks. It is overly simplistic, relying solely on the FID as the distance metric for computing $\tilde{D}(\alpha,\alpha')$ without exploring other potentially more suitable options. Additionally, the authors fail to present a clear and comprehensive pipeline for their algorithm, leaving readers with an incomplete understanding of its implementation details.

2. The experimental results are subpar. In the current landscape of diffusion models, numerous advanced sampling methods have emerged. For instance, UniPC (https://arxiv.org/abs/2302.04867) and DPM-Solver-v3 (https://arxiv.org/abs/2310.13268) can achieve high-quality generation in as few as 10 sampling steps, and even DMD (https://arxiv.org/abs/2311.18828) and DMD2 (https://arxiv.org/abs/2405.14867) can generate satisfactory results in a single step. However, this work predominantly reports results for cases where the number of function evaluations (NFE) is larger than or equal 50. This focus is of limited relevance when the aim is to enhance the sampling speed of diffusion models. Moreover, in the case of using 30 NFEs for ImageNet-256 (as shown in Tables 19 and 20), the obtained FID is considerably higher than that achieved by DPM-Solver-v3 with only 10 NFEs (as evident from their Table 6).

3. There are issues with the mathematical derivations. In Line 179, the authors state that the derivation of the reverse SDE and ODE of the variance-preserving process can be found in Appendix C. However, Appendix C merely replicates content from other papers (ScoreSDE and EDM) without any additional explanation or modification. It is unclear why the authors removed the dependence on $t$ in Eqs. (8) and (9) (and Eq. (10) is a null equation).

In light of the aforementioned weaknesses, I have the impression that this submission clearly falls short of the standards expected for ICLR.

• One of the main limitations of the paper is the lack of details on the computation of the distance and its derivative which is absolutely central to the paper. I had a look at Appendix B but could not find satisfactory answers to many of my questions. For example, what is the overhead of computing all those distances? One has to compute the InceptionV3 features at many noise levels which might be quite expensive as there is no easy way to compute the InceptionV3 features. I am also wondering what is the significance of computing FID scores between noisy images. Sure FID score captures some information about the distributions and their misalignment but InceptionV3 was trained on natural images so I am questioning what really is measured when the noise level is positive. To summarize I think the authors need to provide more details around their main contributions, how to compute it and how to motivate it.

• I am a bit worried about the dependency with the hyperparameter $\xi$. This dependency is introduced and then never discussed but it changes values (see Table 12). This defeats the purpose of the paper as one of the goals is to derive adaptive schedules and reduce the number of hyperparameters but a new hyperparameter is introduced and not discussed. Changing this hyperparameter from the value of $1$ also changes the motivation of the work since we no longer have a constant rate schedule.

• -quality, clarity
  • But I have some concerns related to quality and clarity as summarized below.

Despite the paper's sound methodological contributions and promising experimental results, its presentation can benefit from substantial refinement for clarity and reproducibility. Specific issues and recommendations are as follows.

1. Methodology presentation

Section 4 would benefit from a clearer, more structured presentation of the CRS procedure, potentially in a pseudocode format. The lack of an explicit procedural outline with well-defined inputs, outputs, and step-by-step operations makes the method challenging for the readers to comprehend and reproduce. Clarifying this detail in the main text, rather than relying on Appendix B (indeed, I found some aspects remaining not very clear even after referring to the appendix), would provide greater transparency. The presentation could also improve by reorganizing Section 4 to enhance logical flow and readability. For instance, the authors can augment the procedure described in Lines 265 -- 269 into a concise pseudocode format and include it in Section 4.2, and maybe the authors may want to merge Sections 4.3 & 4.4 into "Additional considerations" etc.

2. Presentation of Experimental Results

In Section 5, the paper could reduce redundancy and make room for more critical content by moving some tables to the Appendix and bringing essential content, such as Algorithm 1 (which can be potentially converted into the pseudocode to include in Section 4), into the main text. Consolidating Tables 10 and 11 in the main text, while moving repetitive tables to the Appendix, would help readers focus on the primary results. Additionally, defining key metrics like FID, sFID, precision, and recall would support readers unfamiliar with diffusion models.

• Regarding latent-space diffusion models, a brief summary or introduction to these models would be beneficial for accessibility.
• In Section 5.2, a conjecture or discussion on why CRS showed limited improvement in pixel-space models as a training schedule could provide valuable insights.
• The rationale for selecting the hyperparameter $\xi$ may need to be explained in more detail. For instance, why did the authors choose \xi = 1.2 in Line 465, while the informal argument in Section 4.2 suggest \xi = 1? Also, the authors may want to give a better, more detailed justification in Line 251, why they introduce a hyperparameter \xi (not necessarily 1) for a more flexibile modeling/training.

3. Clarity & Logical Flow

• Summary of contributions: The stated contributions in Lines 90–103 could be revised for clarity and cohesion. In particular, Item 1’s reference to the “type” of diffusion model could be clarified (e.g., by specifying architecture or training objective). Additionally, Item 2 appears to be a detail that complements Item 1 rather than an independent contribution, which could enhance the readability of this section.
• Discussion and Conclusion: The Discussion and Conclusion sections could be consolidated to provide a cohesive summary of findings, limitations, and future directions. Alternatively, if the authors intended Section 6 for experimental analysis, integrating it into Section 5 would streamline the paper’s structure.

4. Minor suggestions

• Acronyms, such as CRS (Line 47) and AYS (Line 135), should be defined with their full names upon first usage. Additionally, vague notation (e.g., $\alpha$ in Lines 53-54, which is not defined or explained) may require further definition and context. Also, readers may find the distinction among $\alpha$ thresholds (e.g., 0.6 and 0.97) ambiguous and unclear whether they are universal or example-specific -- further, the distinction between the "second" and the "third" is not very clear to me.
• In Lines 78–79, the text would benefit from mentioning earlier that the CRS depends on the choice of a probability metric or divergence and is universally applicable to orient the reader.
• There are minor clarity issues, such as ambiguous conjunctions (e.g., "However" in Line 255 and "though" in Line 485). These can create logical dissonance, and revising these phrases for clarity would improve readability.
• It would be helpful to add definitions for FID, sFID, precision, and recall metrics, perhaps in the Appendix, to assist readers unfamiliar with these evaluation tools.