A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training

1. The rationale behind the motivation is not clearly stated and verified. The author states that the time steps can be empirically divided into 3 states. However, there is no empirical result to support the claim. All figures in the method part (i.e., Figure 1 and Figure 2) are pseudo figures. The value in real experiments should be provided.

2. It is not practical to decide the boundary of each state in real application. Also, it is not very clear how to decide the boundary of each state. If the depends on the convergent speed, the states could vary significantly based on the learning rate, model framework, and the quality of training data. Such a strategy is not practical. In fact, the analysis is not comprehensive. Is it possible that an adaptive learning rate will address the issue? The author should exclude other factors to verify it is the sampling quality, not another factor that results in the difference between the 3 stages.

3. The whole paper assumes that the diffusion model is DDPM. However, there are so many papers[1] that have already addressed the quality of sampling such as DDIM. The author should address the problem with a SOTA framework regarding efficiency.

4. The quality of the experiment is low. While the motivation is to speed up the training process, is it intuitive to report the real training time? The majority of the experiment is report FID with baselines. FID is not the metric to verify the efficiency. Figures 5 and 6 are used to report the convergent speed. However, it looks Log(FID) is not convergent yet. Also, what is the learning rate for each baseline in Figure 5? Why there are 3 sub-figures in Figure 5? Should it be a single figure including a comparison with all baselines?

5. Ablation study is missing. The author should remove the sampling strategy for each stage and vary the boundary. The presentation in the experiment could be improved.

6. Important baselines are missing including [2,3,4, 5] and many others.

[1] Shivam Gupta, Ajil Jalal, Aditya Parulekar, Eric Price, Zhiyang Xun: Diffusion Posterior Sampling is Computationally Intractable. [2] Tae Hong Moon, Moonseok Choi, EungGu Yun, Jongmin Yoon, Gayoung Lee, Jaewoong Cho, Juho Lee: A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models. ICML 2024 [3] Zhiwei Tang, Jiasheng Tang, Hao Luo, Fan Wang, Tsung-Hui Chang: Accelerating Parallel Sampling of Diffusion Models. ICML 2024 [4] Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon [5] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, Anima Anandkumar: Fast Sampling of Diffusion Models via Operator Learning. ICML 2023: 42390-42402 [6] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, Nima Anari: Parallel Sampling of Diffusion Models. NeurIPS 2023

Overall, there are major drawbacks in the proposed method and the quality of the experiment can be significantly improved.

1. Although the paper is overall well-presented and well-motivated, the presentation of some sections can be made clearer. For example, could the authors elaborate on the rescaling scheme in section 2.6?
2. The authors provide ablation studies showcasing the transferability of SpeeD into other types of schedule. However, it seems that the improvement regarding training speed for the quadratic and cosine schedulers are not as strong compared to linear scheduler. As the theoretical analysis is solely based on linear scheduler, can similar results be provided in the quadratic or cosine scheduler case?
3. Contrary to previous works, the authors argues one should put higher probability in sampling earlier timesteps (up to a threshold $\tau$) rather than the middle timesteps. Could the authors provide some empirical ablations on the choice of the threshold $\tau$?

1. Unclear motivation: The paper introduces the analysis of $\delta_t = x_{t+1} - x_t$ as a representation of each timestep, yet the motivation behind this choice could be clearer. Furthermore, the authors use $\partial_t \hat{\Psi}_t$ for timestep categorization. While $\delta_t$, representing the amount of change, seems more intuitive, the motivation for using $\partial_t \hat{\Psi}_t$ lacks explanation. Additional insights into this choice would enhance the understanding of the approach.

2. Limited applicability to general schedules: The method relies on a linear schedule for discrete-timestep diffusion models. Although general cases are mentioned, extending SpeeD to other scenarios, such as EDM, and supporting this with experiments would strengthen the paper’s applicability.

I have to admit that I found this paper hard to follow.

The paper is redundant and lacks proper organisation. For instance, on the last page of the article, just before the conclusions, the authors revisit what a diffusion model is (while the entire paper is about DMs) and mention classifier guidance (which is not referred to in the paper whatsoever).

The diagrams in the first pages (Figs 1,2 & 3) do not really help to clarify the contribution. They are cluttered, and it is difficult to understand what they are trying to illustrate. It is unclear whether these are based on data or just illustrations. In any case, it is unclear what the authors mean when the paper refers to these figures stating that they help to "visualize a loss curve"

The detailed presentation of the proposed method is also unclear. The first 4 pages of the paper introduce the motivation and context, and back up the observations leading to the proposed method. However, in Sec 2.4 (?), the method is finally presented but without the necessary clarity. Is eq (3) the main definition of the procedure?

Sec 2 is 3.5 pages long and covers the basics of DM and the brief presentation (see above) of the method. Then, the paper jumps directly to the experiments.

Overall, I think that there is certainly some practical value in this article, but there a fair amount of work needed in term of presentation and organisation to take this paper at the acceptance bar at ICLR.