Debias the Training of Diffusion Models

Thanks for acknowledging the meaning of our work as well as your careful and constructive suggestions.

1. Experiments. We follow the experimental setting and datasets selection of the baseline work P2 [1]. We also conduct more experiments on CelebA-HQ [2] dataset after submission, and the results are shown in the following table. | Steps T | Constant | P2 | Min-SNR | Ours | | :------ | :------: | :------: | :------: | :------: | | 1000 | 9.3739 | 7.2575 | 6.3218 | 5.9795 | | 500 | 10.2355 | 7.7183 | 6.9225 | 6.5717 | | 250 | 11.0974 | 8.4334 | 8.0162 | 7.6038 | | 100 | 12.0064 | 9.2972 | 9.3851 | 8.8362 |

   The employed FFHQ, CelebA-HQ, and AFHQ are all common, representative and high-quality benchmark datasets. The CIFAR-10 dataset is somewhat outdated. While, we will also complete the experiments on these two datasets as you suggested. While, need to emphasize that existing experimental datasets are already diverse and representative, as well as following the common practice of baseline works [1].

   Besides, there is no need to worry about the reproducibility and scalability of this work. The derivation and analyses are general for constant weighting strategy in diffusion models. We do not impose any assumptions and constrains on the generality and scalability. Additionally, our method only modifies the training weight strategy of the ADM codebase, and thus is of high reproducibility. We will also soon release the code and pretrained models.

2. The combination of theory, analyses and experimental results. This paper is not a purely theoretical work. Indeed, the derivation in the beginning is only the start of this work. We hope to provide researchers with comprehensive and in-depth insight and understanding on the bias problem. Thus, we conduct substantial analyses and experiments. Besides, the consistency between theory and experiments is also more convincing.

3. Stability of the training. Yes, we achieve consistent and significant performance improvement on all the performed experiments in this paper, regardless of the datasets, total sampling steps, and samplers. These are all clearly shown in the experimental part and the supplementary material. Besides, there is no need to worry about the reproducibility of this work. Our method only modifies the training weight strategy of the ADM codebase, and thus is of high reproducibility. We will also soon release the code and pretrained models.

4. Comparison with baseline work [1]. In this paper, we move the comparison with related works into experimental part section 5.2. This may cause the confusion of you and the other reviewers. Actually, we employ [1] as our baseline. The main difference between [1] and our work is that [1] designed their weighting strategy intuitively and fail to give the insight and analyses of the bias problem in the previous constant weighting strategy.

[1] Choi J, Lee J, Shin C, et al. Perception prioritized training of diffusion models. CVPR. 2022.

[2] Karras T, Aila T, Laine S, et al. Progressive growing of gans for improved quality, stability, and variation. ICLR. 2018.

We appreciate your responsible reviewing and the insightful comments.

1. Yes, as you observed, allocating lower weight for small t will slightly influence the optimization of the network at small steps. While, the optimization difficulty and importance of the denoising network are inherently different across step t, with small step less important and relatively redundant. This is also consistent with the experiments, where our weighting strategy achieves highest performance.
2. The example in Fig. 5 is randomly selected, while we can still see obvious difference, especially in the small steps. The randomness but superiority indeed effectively validate the generality and robustness of our method.
3. The comparison between different training targets is shown in the supplementary material section D. Our method significantly elevates the performance of the base $\epsilon$-prediction mode, and also largely surpasses the performance of other training targets.

Thanks for your valuable suggestions.

1. First, we introduce the differences between the work you mentioned and our method. The SNR relationship between $\epsilon$-prediction and $x_0$-prediction is not hard to derive and is not the main contribution of our work. Actually several previous methods get this conclusion [1][2] (including the work you mentioned). The difference is that these works usually stop by this conclusion and apply it to other use cases instead of training diffusion models for image generation. For example, [2] employed this relationship to scale the loss value to the same level between the two loss terms, in the use case of employing pretrained diffusion models as loss function to train the inverse network. Significant differently, this derivation only stands as the beginning of our paper. Our work focuses on the bias problem of constant weighting strategy in the basic and critical $\epsilon$-prediction mode for training the diffusion model from scratch for image generation. The SNR relationship is only employed as weight designing guidance. For example, (1) we propose the concept of “estimated $x_0$” part and “amplified error” part and in section 3. (2) We conduct a comprehensive and systematic exploration to dissect the inherent bias problem deriving from constant weighting loss from the perspectives of its existence, impact and reasons in section 4. (3) We validate our design choice with substantial experiments to train diffusion models from scratch in section 5.
2. SNR weighting for the sampling. There may exist misunderstanding of you for the work [2]. The “sampling” process of [2] is indeed training the inverse network with frozen pretrained diffusion model as loss function network. In our work, we employ SNR weighting as a training strategy for training diffusion model from scratch. Thus, our sampling process is certainly free of training strategy.
3. As you view this related work as the only weakness of our method, we hope our explanation can solve your misunderstanding and confusion.

[1] Rombach R, Blattmann A, Lorenz D, et al. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR. 2022.

[2] Mardani M, Song J, Kautz J, Vahdat A. A Variational Perspective on Solving Inverse Problems with Diffusion Models. arXiv. 2023.

It is unprofessional and irresponsible to leave two related works as the main weakness of this work with only few lines of words, even without carefully reviewing this work. It is even more ridiculous to use entry-level blog to help reviewing and understanding our work.

1. We show clear discussions and results comparison with related works in section 5.2 and supplementary material section D. Improvements over the \epsilon-prediction mode can be categorized into two types: weighting strategy and different training targets. We give comprehensive comparison between different weighting strategies in section 5.2. We also present the discussions and experimental results of different training targets in supplementary material, including v-prediction.
2. The recovered relationship between $\epsilon$-prediction and $x_0$-prediction only stands as the beginning part of this paper. While, you treat it as the main contribution of our work with prejudice. Our work focuses on comprehensive analyses and experiments to provide clear insight and understanding of the bias problem concealed within the constant weighting in $\epsilon$-prediction. For example, (1) we propose the concept of “estimated x_0” part and “amplified error” part and in section 3. (2) We conduct a comprehensive and systematic exploration to dissect the inherent bias problem deriving from constant weighting loss from the perspectives of its existence, impact and reasons in section 4. (3) We validate our design choice with substantial experiments to train diffusion models from scratch in section 5.

I think the authors have a limited understanding of diffusion models and their techniques (e.g. $x_0$ prediction), which have been extensively discussed and are even covered in entry-level blogs. I recommend that the authors take the time to carefully read such resources to acquire a basic understanding.

As I mentioned in my previous review, the "new objective" (Eq. 11) proposed, analyzed, and experimented on by the authors is essentially equal to $x_0$ prediction loss, a well-established and widely used technique in practice. (Indeed, people nowadays use v-prediction and EDM-type preconditions instead of the conventional $x_0$ prediction)

I kindly request that the authors refrain from making unwarranted accusations against the reviewer. I went through your paper carefully, and I believe that the primary issue is quite evident, making it unnecessary for me to point out other specific issues in my previous review. Indeed, it appears that there are several instances that show the authors' lack of basic understanding regarding diffusion models. For example, the difference between $\epsilon$ prediction and $\epsilon$ is not an error but a feature of diffusion models. The optimal $\hat{x}\_0$ should not be $x_0$ itself but rather the mean of the posterior $E_{p_{0|t}(x_0|x_t)}[x]$. In addition, the experiments are on simple single-mode face datasets, rather than more realistic multi-mode datasets like CIFAR-10, ImageNet, or LAION. The FID score reported in the experiments falls far outside the reasonable range, both for the baseline and the proposed method, making the experimental results unreliable.

Based on the author's response, I have decided to lower my rating. I strongly recommend that the authors take time to acquire relevant knowledge to gain a basic understanding of diffusion models and their associated techniques.