Slight Corruption in Pre-training Data Makes Better Diffusion Models

We thank the reviewer's efforts on this paper and his constructional suggestions on the ablation study of CEP. We now address the concerns as the follows.

I believe that the authors should emphasize...

Thanks for this suggestion. We will make condition perturbation more clear in our title and main paper. We will also include these relevant works into discussion in our revised paper.

The theoretical model is not very relevant...more related to the proposed algorithm in Section 5.

In the theoretical part, we considered a more general setting [1,2], where the label embedding is perturbed by Gaussian noise. We proved that slight corruption in pre-training data can lead to more diverse and higher quality generation in diffusion models. Additionally, the results from the theoretical part also inspire and justify the algorithm proposed in Section 5.

The theoretical framework in Section 4 can also be applied to other types of noise. For example, in Section 3, label flipping can be regarded as a special corrupted label embeddings, i.e., $\mathbf{c}(y^c) = \mathbf{c}(y)+ \boldsymbol{\xi}$, where the noise $\boldsymbol{\xi}$ are vectors with entries 0, 1, and -1. Assuming there are $K$ classes, and each class has an equal sample size (approximately 1300 of each class in the experiment), each label has a probability $p$ of being flipped.Then the noise $\boldsymbol{\xi}$ follows a distribution satisfies $\mathcal{F}(p\frac{K-2}{K}\mathbf{1},(p-p^2(\frac{K-2}{K})^2)\mathbf{I})$. After obtaining the mean and variance information of the noise $\boldsymbol{\xi}$ , the corresponding optimal linear denoising network and the generative distribution can still be solved using the techniques in Section 4.

[1] Hu W, et al. Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee. ICLR.

[2] Tang Y H, et al. Detecting label noise via leave-one-out cross-validation. arXiv preprint 2021.

Also, the existence of multiple centers in Section 4 does not seem relevant, unless I am missing something.

The multiple centers in Section 4 are derived from the assumption that the data follows a mixture of Gaussian distributions, which is a common assumption in the theoretical analysis of diffusion models [1-4]. The multiple centers correspond to the multiple classes of the real data. We consider modeling each center as a simplification of the real problem, and this simplification is sufficient to illustrate how noisy labels affect the training process of the denoising function (Section A.2), thereby influencing the final generative distribution (Section A.3), which shows improvements in both quality and diversity compared to the distribution generated with clean labels (Section A.4).

Although the theoretical part considers a simplified single-center version, no other theoretical work simultaneously consider both the training and generative processes of diffusion models, and their interaction, while precisely solving the generative distribution. Our theoretical contributions should not be overlooked. We also agree that employing a more practical theoretical model could improve the theoretical part, which is left for our future study direction.

[1] Shah, Kulin, et al.. "Learning mixtures of gaussians using the DDPM objective." NeurIPS 2023.

[2] Chen, Sitan, et al. "Learning general gaussian mixtures with efficient score matching." *arXiv preprint.

[3] Li, Puheng, et al. "On the generalization properties of diffusion models." NeurIPS 2024.

[4] Li, Yangming, et al. "Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models." ICLR.

For the reasons mentioned above...finding.

Thanks for pointing out this point. We also think CEP can be viewed as a regularization method for diffusion training. By adding small noise during the training process, CEP prevents the trained model from overly collapsing onto the training data, thereby improving the diversity and quality of the generation distribution.

Through ablation study of other methods, we showed that CEP is superior to other methods (dropout on conditional embedding and label smoothing of class labels for computing conditional embedding) and may function beyond regularization.

While the regularization methods like Dropout and Label Smoothing can also improve the performance of diffusion, CEP is significantly more effective than these methods. We will include this ablation study into the Appendix of our paper.

Could the authors please clarify whether the corruption in Sections 3 and Sections 5 happens...

In section 3, for our empirical study, we adopt fixed corruption, i.e., take a dataset, introduce corruption, and train with the corrupted dataset. In section 5, the proposed CEP is introduced randomly at each training iteration. However, for results shown in Figure 8 with noisy dataset, we still used the fixed corrupted dataset.

Here we compare these two types of corruption. We showed that CEP works the best among all corruption methods. Also fixed CEP is more effective than adding data corruption (fixed and random). Random data corruption can be viewed as a CEP-variant with embeddings from flipping label instead of adding noise, and thus is also more effective than fixed data corruption. We will include this analysis in our Appendix.

If you find our revise and response helpful, please consider raising the score for better support of our work. We are open to discuss more if any question still hold.

We thank the reviewer's acknowledgment on this paper. We now address the weakness raised as follows.

Notion of significance: The paper claims that slight pre-training corruption yield significantly better performance in terms of FID and IS metrics. Nevertheless, this notion of significance is never formally tested. For this at least a experiment with multiple repetitions should be performed to determine statistical significance.

Thanks for this great suggestion. Please understand that we will not be able to perform pre-training experiments for multiple repetitions, simply due to the un-affordable cost for that. This is also common in most study of generative models where only a single run is conducted in pre-training. Although we are not able to run multiple rounds of pre-training, for IN-1K models, we conducted an experiment for generating images with guidance scale of 2.25 of 5 random seeds, and reported average FID and IS with their standard deviation. Also, we conducted a t-test between the results from the clean model and noisy models and reported the p-value here. The results demonstrate that the improvement is indeed significant (extremely significant in terms of the p-value).

Due to the limited discussion time, we will try to include more significant test of other models and downstream applications for the final version of our paper.

Typos:, Page 8, lines 257 and 259 rrecision -> precision, Page 9, lines 299 dataset -> data

Thanks for pointing out the typos! We have fixed all of these in the paper.

Link between theoretical analysis and metrics should be made explicit. I.e. pointing out that the FID is essentially a Gaussian approximation based estimated for the 2-Wasserstein distance of inception features (preferably at page 7, line 223).

Thank you for your suggestion. We will emphasize on page 7, line 223 of the manuscript that the FID metric used in the experiments is the 2-Wasserstein distance under the assumption that the inception features follow a multivariate Gaussian distribution, to highlight their connection.

If you find the above response help resolve your concerns, please consider raising score for better support of our work. Thanks!

Thanks for the reviewer's constructional feedback on this paper. We now address the weaknesses and questions as follows.

While authors showed extensively that adding noise can help, they missed to show "when does it help"? For example, if I'm a practitioner who wants to train a large model from scratch...That insight is missing in this paper.

Thanks for mentioning this important question. However, we would like to highlight that core insight from our empirical and theoretical study is not to find the "optimal" noise ratio in the pre-training dataset that can help. Instead, we aim to present insights that, to pre-train diffusion models (and self-supervised models), it is important to ensure the diversity in the pre-training and slight corruption in the pre-training dataset may benefit diversity. No over-filtering on corruption is needed for pre-training data.

Insights for practitioners: the proposed CEP can be simply adapted into pre-training of diffusion models to improve performance, no matter the datasets and models. In Figure 8 (b), we also presented that, on noisy datasets (of different levels) as in practical settings, CEP can also facilitate both the pre-training and downstream transferring performance.

Does the optimal noise perturbation level transfer between the models? If 10% is the best noise level for LDM on one dataset, would it be the best noise for the same-size LDM on another?

This is indeed an intriguing question. While our empirical study shows that slight corruption universally improves the performance across models and datasets, we would like to mention that it also depends on the pre-training dataset size. As the dataset size scales larger, the slight corruption ratio may also decrease, since we may not expect 10% of our pre-training data of size 2B is corruption.

How do metrics look like throughout training for models trained with different levels of noise?

Thanks for mentioning this important question. We use the pre-trained LDM-4 IN-1K clean and noise $2.5$ to generate images at different checkpoints along training, using a guidance scale of $2.5$. We present the FID and IS results as follows:

FID:

IS:

From the results, we can observe that, with slight corruption, the model performs slightly worse at the beginning of the training (before 50K iter.), while becomes outperforming the clean one towards the end of the training.

We will include these results in our Appendix.

CC3M corruption details are not clear. I went to Appendix B1, you mentioned 5 levels of corruption, but it is still vague! What are these 5 levels?

Sorry for the confusion caused. For CC3M, to introduce corruptions, we randomly sample two image-text pairs, i.e. $(I_1, T_1)$ and $(I_2, T_2)$, and swap the text of these two pairs to make it $(I_1, T_2)$ and $(I_2, T_1)$. For corruption levels, we use the same levels, i.e. {2.5, 5, 7.5, 10, 15, 20}%, as in ImageNet. These are shown in the legend of CC3M figures and mentioned at line 112. We will add more details in Appendix B1 and highlight in the main text to make this more clear.

How is the number of training iterations determined for the models?...On what basis are these numbers decided?

The number of training iterations mainly follows settings in the previous papers. For example, in LDM paper [1], the main settings of LDM-4 on ImageNet and CC3M is 178K and 390K steps. To replicate this, we used a total batch size of 512 in our experiments, resulting in roughly 2.5K and 5.4k training steps per epoch. Thus we use 71 and 70 epochs of IN-1K (1.28M samples) and CC3M (2.8M samples due to expired url) to make the total training iteration as 178K and 390K. This is similar for DiT and LCM, where we follow their paper to setup the training steps.

"this again ties back to the question, when is "adding noise" helpful? Are we in an overtraining regime? Is that...and hence the metrics improve?"

From the above results, one can see that, adding noise starts being helpful once the model converges to generate reasonable figures and consistently improves along training after that. From our paper and ablation results for Reviewer XQzf, you can also see that CEP performs better than other regularization methods, including input perturbation (augmentation), dropout on embedding, label smoothing, etc. This demonstrates the better effectiveness of CEP than traditional regularization.

A few relevant citations are missing...last year's Neurips paper [1].

Thanks for mentioning this relevant work. We will add this work into discussion in our revised paper. Theie differences are:

This work mainly focuses on memorization and copying of diffusion models. The authors found adding perturbations to condition or condition embedding help relieve the copying issue (but FID remains similar as clean, possibly due to the small-scale dataset and focus on fine-tuning instead of pre-training), which aligns also with our findings as shown in Figure 11 (i) and Figure 13 (i), where large L2 distance of generated images and ground truth are present with pre-training corruption.

We hope the above response can resolve your concern. If you find our revise and response helpful, please consider raising the score to better support of our work. Also please let us know if there is any more question.

Thanks for your time reviewing this paper. We now address your questions as follows.

From my understanding, in Theorem 2...It would be beneficial to address this explicitly in the main text.

Thank you for your suggestion. We have already mentioned that the order of $\gamma$ is $O(\frac{1}{\sqrt{\max_k n_k}})$ in Theorem 2 and we will further emphasize in the main text (page 7, lines 219-220) that the noise level needs to satisfy $\gamma = O(\frac{1}{\sqrt{\max_k n_k}})$.

It would be beneficial if the method that determines the noise level could be linked to the theory like using the [...]

In the experimental section of Section 5.2, the (Gaussian) noise we used is approximately $0.04\epsilon$, where $\epsilon \sim N(0,I)$, while the theoretical part in Section 4 proves that the noise level is approximately $0.03 \epsilon$. The noise levels in the experiment and the theory are very close. We will further emphasize their link in the main text (page 8, line 247).

They briefly mentioned the various assumptions in their theory as limitations. It would be helpful to provide a detailed explanation of these assumptions and the scenarios in which they hold.

Our theoretical part includes the main setup that the data distribution follows a mixture of Gaussian distributions, which is a common assumption in the theoretical analysis of diffusion models [1,2,3,4,5]. Follow the previous work [2,6], we accordingly considered a piece-wise linear function as the denoising neural network, which has been proven sufficient to handle the Gaussian mixtures. Based on these previous theoretical works, our theoretical framework, although seems simple, provides a paltform for understanding the improvements in quality and diversity introduced by noised label embeddings.

[1] Shah, Kulin, Sitan Chen, and Adam Klivans. "Learning mixtures of gaussians using the DDPM objective." Advances in Neural Information Processing Systems 36 (2023): 19636-19649.

[2] Chen, Sitan, Vasilis Kontonis, and Kulin Shah. "Learning general gaussian mixtures with efficient score matching." arXiv preprint arXiv:2404.18893 (2024).

[3] Li, Puheng, et al. "On the generalization properties of diffusion models." Advances in Neural Information Processing Systems 36 (2024).

[4] Cui, H., Krzakala, F., Vanden-Eijnden, E., & Zdeborová, L. (2023). Analysis of learning a flow-based generative model from limited sample complexity. arXiv preprint arXiv:2310.03575.

[5] Li, Yangming, Boris van Breugel, and Mihaela van der Schaar. "Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models." The Twelfth International Conference on Learning Representations.

[6] Gatmiry K, Kelner J, Lee H. Learning mixtures of gaussians using diffusion models[J]. arXiv preprint arXiv:2404.18869, 2024.

If you find our response helpful, please consider raising the score for better supporting our work.