Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training

Please see our top-level comments for clarification on FID scores, where we show that we consistently produce SOTA results across a wide variety of settings on this metric.

We address the two weaknesses pointed out by the reviewer below.

1. Training cost The numerical integration in our loss is done using importance sampling. Details of this standard approach are discussed in many papers, e.g., in Kingma et al, Variational Diffusion Models, NeurIPS 2021. As pointed out by Kingma et al, the standard diffusion loss can also be interpreted as an evaluation of a weighted integral evaluated with importance sampling, so this is not unique to our method.

However, it is true that the form of our loss does incur additional training cost, and we pointed this out explicitly as a limitation.

• Trading off extra computation at training time for faster / better quality sampling at inference time (as we do) is considered advantageous for most applications. For instance, the entire area of score distillation methods train two models, rather than one, but are nonetheless popular in practice.
• The reviewer guidelines state “authors should be rewarded rather than punished for being up front about the limitations of their work”. We have tried to honor this guideline in our own reviews, and ask the reviewer to consider taking this guideline into consideration when deciding on a final score.

2. Related work

• The references [2,3,4] from XUix’s review also have “diffusion” and “classification” in the titles. However, these methods all consider classifying objects in images (e.g. “cat”/”dog”). They rely on the longstanding observation that conditional generative models can be used as classifiers via Bayes rule. In contrast, we introduce an interpretation of diffusion models as noise classifiers - a classifier that can distinguish the amount of noise added to an image. The derivation of these methods is nontrivial and totally different as it goes through I-MMSE results rather than Bayes rule. While the task of classifying noise may be unfamiliar to some, it comes from a classic line of work started by Gutmann & Hyvarinen on noise contrastive estimation (JMLR 2012) as an alternative foundation for machine learning (see response to Y79y for more detail). We will add a discussion of these papers and the distinction between noise classification and traditional classification with diffusion to related work.
• Reference [1], like most attempts to accelerate diffusion, relies on changes to the sampler. Our approach, on the other hand, changes the training procedure and can be combined with any sampler. We tested our approach with a representative sample of deterministic and stochastic samplers, and sequential and parallel samplers. We will add this reference to the list of other recent sampling approaches that our method could be combined with.

Can you experimentally show/quantify the improvement of the denoiser for out-of-distribution evaluation when trained using contrastive diffusion loss (CDL)?

Yes, a direct metric is to look at the score matching loss (L2 distance between true and estimated score) for OOD points, we will add this to the supplementary material. This requires access to a ground truth denoiser, so we can only show it for simple cases like the one shown in Fig. 1.

Thank you so much for taking the time to read and review our paper. Please see our top-level comments for clarification on FID scores, where we show that we consistently produce SOTA results across a wide variety of settings on this metric.

We address the two weaknesses pointed out by the reviewer below.

1. FID Score Discrepancy

The original Table 2 reported FID scores using 5k samples, as per the settings of our baseline paper [4]. FID scores depend on the number of samples, so that our previous results were not directly comparable to other results reported using 50k samples. In response to your feedback, we have updated Table 2 to include FID scores calculated with 50k samples, results are in the top-level rebuttal response.

The updated results show that our results are actually very good and in line with SOTAs. With the same parallel sampler, CDL-finetuned models consistently outperforms the models trained by the original diffusion loss (DDPM, VP, and VE losses). For example, compared to all other baseline losses under FID calculated with 50k samples, cdl achieves an FID improvement of ~0.7 on unconditional CIFAR10, ~0.5 on conditional CIFAR10.

2. Minimizing CDL and Meaningful Generation

Are there any insights or proofs provided on why minimizing the CDL would also lead to meaningful generation?

Our method is based on Density Ratio Estimation (DRE) rather than maximum log-likelihood methods. DRE techniques transform the task of learning data distributions into learning to classify between data samples and samples from a reference distribution [1,2,3]. Our approach bridges diffusion-based generative methods with density ratio estimation. Therefore, minimizing CDL effectively learns the underlying data distribution, which leads to meaningful generation.

For more detailed information about DRE, we refer to [2] section 2.1 Density Estimation by Comparison.

[1] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012

[2] Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2), 2012

[3] Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 33:4905–4916, 2020.

[4] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.

Thank you so much for taking the time to read and review our paper. We’re glad that you are positive about our paper. Please see our top-level comments for clarification on FID scores, where we show that we consistently produce SOTA results across a wide variety of settings on this metric.

We address the two weaknesses and two questions pointed out by the reviewer below.

1. Clarity on Notation

The clarity of the paper could be further enhanced

To clarify the distinction between the log-SNR used in the integral ($\alpha$ in Eq. 3) and the density of the noisy data distribution, which is a mixture of data and noise, we used $\zeta$ to represent the amount of noise in the noisy distribution. We acknowledge that $\alpha$ was used in Appendix A.2 due to a typographical error. This will be corrected in the revised manuscript.

2. Empirical Results FID scores

The difference in FID scores compared to EDM's default settings is because the original Table 2 reports FID scores using 5k samples, as per the baseline paper, whereas SOTA methods typically use 50k samples.

In the parallel sampling example, the method shows good performance, but even with the performance improvement, the results are still far from optimal compared to EDM with the default settings.

Table 2 reports FID scores for samples generated using parallel samplers from EDM-pretrained checkpoints, while EDM's default FID scores are for samples from sequential EDM samplers. This difference in sampling methods explains the difference in FID scores.

We have updated Table 2 to include FID scores calculated with 50k samples. For example, compared to all other baseline losses, cdl achieves an FID of 2.38 (0.62 better than the best baselines with an FID of 3.00) for unconditional CIFAR10. These updates are included in the top-level comments.

3. Why CDL Helps with Stable FID Scores Across Hyperparameters?

According to EDM [1], samplers introduce local discretization errors at each step, which accumulate as global errors. Although sequential samplers are designed to stay close to the forward / training paths $(z_\alpha, \alpha)$, local errors can deviate $z_\alpha$ from these paths. CDL, by training on asynchronous data pairs $(z_\alpha, \beta)$, allows the model to see deviations (e.g., $z_\alpha'$) during training and thus corrects for these local errors. This "correction" capability contributes to stability across hyperparameters in both deterministic and stochastic sampling settings. More generally, our approach can be viewed as noise contrastive density ratio estimation, an alternative approach to maximum likelihood for density estimation (Gutmann & Hyvarinen, 2012) [2]. By combining maximum likelihood and density ratio based estimates, we improve over either one individually.

4. Why Fine-Tuning Instead of Pretraining?

Fine-tuning on pretrained models is preferred because training diffusion models from scratch is computationally expensive. It is common practice to fine-tune pretrained models to balance efficiency and performance. Results on toy datasets were trained from scratch, with excellent results.

[1] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.

[2] Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2), 2012