Training Diffusion Classifiers with Denoising Assistance

We thank the reviewer for their insightful and helpful comments for improving the paper.

1. Presentation: Thank you for the feedback. We will fix the formatting in Eqs. (5,6), and we gladly welcome any other suggestions.

2. Motivation for using single-step denoised examples. We use denoised example $\hat{\bf x_t} = {\bf x_t}+\sigma^{2}(t) s_\theta({\bf x_t},t)$ as an additional input to the classifier network because (a) learning to classify denoised image is potentially easier than learning to classify noisy examples and (b) we have access to a pretrained score network $s_\theta({\bf x_t},t)$ that can help us estimate the additive gaussian noise contained in ${\bf x}_t$. Beyond just simplifying the learning task, we observe that this simple change also helps address the following deficiencies of vanilla classifier guidance: 1) Generalisation in fully-supervised and semi-supervised settings, 2) Perceptual-alignment of Classifier-gradients, and 3) Image Generation Performance. We support these findings both theoretically and empirically. Additionally, we also show significant advantages of using denoised-images in semi-supervised training as opposed to simple gaussian-noise augmented images. We will clarify this motivation in the revised paper.

3. Multi-step denoising. As described in Eq. (4), the unconditional score function estimated by $s_\theta({\bf x_t},t)$ is adjusted with the gradient of log-likelihood predicted by the classifier in classifier-guided diffusion sampling, as $s_\Theta({\bf x_t}, t|y) = \nabla_{{\bf x_t}} \log p_\phi(y|{\bf x_t},t) + s_\theta({\bf x_t},t)$. Here, $\theta$ refers to parameters of the unconditional score function, $\phi$ refers to parameters of the classifier function and $\Theta = \\{\theta, \phi\\}$. The suggestion of using the sample at $t-2$ or $t=0$ for estimating the score-function $s_\Theta({\bf x_t}, t|y)$ is indeed very interesting but requires separate theoretical and empirical investigation; in particular, it may require making appropriate modifications to the sampling algorithm. In this study, we consider using denoised example obtained at time $t$ as an additional input to the classifier for estimating $s_\Theta({\bf x_t}, t|y)$ and this does not require any modification to the sampling algorithm.

4. Uncertainty measures for pseudo-label estimation. To train the diffusion classifier on partially-labelled data, we need to estimate the pseudo-labels of the unlabeled examples in each batch. As described in the paper, we follow FixMatch and use confidence-thresholding, a traditional measure of uncertainty, to estimate the pseudo-labels. In order to estimate the pseudo-labels, we use two types of data samples from forward diffusion to estimate the pseudo-labels — samples diffused to 1) $\tau=0.01$ (i.e., with negligible gaussian noise) and 2) uniformly sampled time $s$. Since the samples diffused to time $\tau=0.01$ have negligible gaussian noise, that is similar to estimating pseudo-labels from raw data using traditional uncertainty measures. In addition, we also use pseudo-labels from confident predictions on samples diffused to random uniform time $s$ as they could be more informative than raw data in the following cases:

   • Additive gaussian noise in samples diffused to time $s$ could potentially help the noisy-classifier/DA-classifier better estimate the uncertainty in labels as compared to samples containing negligible gaussian noise.
   • Denoised images obtained from unlabeled examples with additive gaussian noise could potentially help classification, even when the DA-classifier is not able to confidently classify the original raw image.

   At each training step, we estimate pseudo-labels from confident predictions on samples diffused to both $\tau$ and $s$ and use them for computing the cross-entropy loss. We do not resolve disagreements between these two pseudo-labels because we expect that the confident predictions on diffused samples should become consistent over training. We thank the reviewer for this comment and we will clarify this discussion in the revised text.

We look forward to addressing any outstanding concerns in the discussion period.

We thank the reviewer for their comments. We will incorporate the minor suggestions & typo corrections in our revision. We address the remaining points as follows:

What is the motivation to develop DA classifier?

We systematically expose several deficiencies of vanilla classifier guidance,

1. Generalization in fully-supervised and semi-supervised settings,
2. Perceptual-alignment of Classifier-gradients,
3. Image Generation Performance

and we demonstrate how simply using denoised images as additional input can address all these deficiencies. We call these denoising-assisted (DA) classifiers.

What is the advantages of DA classifier compared to existed classifiers including conventional deep models (VGG, ResNet, etc.) or large models (CLIP, etc.)?

We believe that this question is asking for some overall context, so here is an overview: For classifier-guided diffusion, we train classifiers on images sampled from the forward diffusion, which generally contain additive gaussian noise. As described in the paper, a DA-Classifier refers to a classifier that takes both noisy and denoised images as simultaneous inputs, in contrast to a noisy-classifier that only receives noisy images as inputs. Noisy Classifier and DA-Classifier can be constructed with any backbone neural architecture: as described in the paper, we use WideResNet-28-2 for all CIFAR10, MNIST and SVHN experiments (following Song et al., 2021) while we use the downsampling-half of the UNET for ImageNet experiments (following Dhariwal and Nichol (2021b) ). The vanilla diffusion classifier and the corresponding denoising-assisted diffusion classifier have the same network architecture after the first layer and differ only in terms of the first convolution module: for example, if the first input-convolution of vanilla diffusion classifier takes an input volume of $(N, C_i, H_i, W_i)$ and gives an output volume of $(N, C_o, H_o, W_o)$, the DA-Classifier takes an input volume of $(N, 2C_i, H_i, W_i)$ and gives the same size output $(N, C_o, H_o, W_o)$.

“the authors should provide more detail about how to get the denoised images and analysing the influence of different types of denoised images”

As described below Eq. 6, we obtain denoised images $\hat{\bf x}$ using the pretrained score-network $s_\theta$ as $\hat{\bf x} = {\bf x}+\sigma^{2}(t) s_\theta({\bf x},t)$. We would be glad to provide additional details about this in an appendix, but we do not understand exactly what additional details would be helpful for this reader. We are also not sure what the reviewer means by “different types of denoised images”; any clarification on this would also be welcome. As explained above, we are denoising images that have been corrupted with additive gaussian noise.

“Why Theorem 1 explain the improvements of DA classifier?”

As written in Section 3, Theorem-1 “explains the improved perceptual alignment in Fig (3) and Fig (2), since multiplying a vector by $\text{Cov}[{\bf \bar{\bf x}}_t|{\bf x}]$ stretches the vector along the principal directions of the conditional distribution $p({\bf \bar{\bf x}}_t|{\bf x})$.” Intuitively, since the conditional distribution $p({\bf \bar{\bf x}}_t|{\bf x})$ corresponds to the distribution of candidate denoised images, the principal directions of variation are perceptually aligned and hence stretching the gradient along these directions will yield perceptually aligned gradients. We attribute the quantitative and qualitative improvements on the image generation task to 1) perceptually aligned classifier gradients and 2) improved generalisation to unseen examples.

We look forward to addressing any outstanding concerns in the discussion period.

We thank the reviewer for their insightful comments and summary. We address the mentioned weaknesses and answer the questions as follows:

1. Numerical results and consistency between metrics in Table 2 and Table 4: Thank you for asking about this. The Precision, Recall, Density and Coverage metrics compare between the manifold of generated distribution and manifold of the source distribution in terms of nearest-neighbours and can be computed conditionally (i.e., classwise) or unconditionally [1]. Table 2 shows the Precision and Recall metrics computed unconditionally while Table 4 shows the Precision, Recall, Density and Coverage metrics computed classwise. For CIFAR10, we do indeed observe significant improvements in terms of classwise Precision, Recall, Density and Coverage metrics (see Table 4). We also observe that using DA-Classifier instead of vanilla classifier improves FID/IS on both ImageNet and CIFAR-10 at similar precision and recall values – this suggests that both the classifiers have similar manifolds in the nearest-neighbours sense but the generated images with DA-Classifier are more realistic than vanilla-classifiers. In Fig. 5, we qualitatively observe that DA-Classifiers improve over Vanilla-classifiers in terms of overall coherence and since these improvements are generally subtle, we do observe slight but meaningful improvements of FID/IS over the baseline. We will incorporate this discussion into the revised text. Overall, we believe that the numbers are indeed consistent with strong performance, but if there is any remaining result or comparison that the reviewer sees as not being good, then if we failed to provide a clear explanation or interpretation of it, we would be grateful to have the opportunity to correct that. (Thank you!)

   Also, thank you for appreciating the strength of our visualizations! While visualizations can be hard to quantify, we are very excited by these results and agree that they demonstrate a significant contribution of this work!

2. Semi-supervised training: To clarify: the semi-supervised training is exactly analogous to the fully-supervised training in that the classifier is trained to classify samples from the forward diffusion (as described in section 2.2 and Eq. 5); the key difference between the two is that we use pseudo-labels for training over unlabeled examples. Table 5 summarises the image-generation performance of the semi-supervised Noisy-classifier and semi-supervised DA-Classifier trained on the same labeled and unlabeled portions of CIFAR-10: we observe that the semi-supervised DA-Classifier significantly outperforms semi-supervised vanilla-classifier in terms of the classwise Precision, Recall, Density and Coverage metrics. The FID/IS scores in the semi-supervised setting are similar to the fully-supervised case.

We look forward to addressing any outstanding concerns in the discussion period.

[1] Naeem, M.F., Oh, S.J., Uh, Y., Choi, Y. and Yoo, J. Reliable fidelity and diversity metrics for generative models. ICML, 2020.

5. Ablations

Thank you for suggesting these ablations. As described in the paper, we evaluated the relative importance of noisy and denoised examples in DA-classifiers by zeroing out one of the input images and measuring its effect upon classification accuracies; we found that while removing either one of the inputs causes the accuracy to drop below the noisy classifier, the drop is higher when the denoised image is masked out. This demonstrates that the DA-classifier makes use of both noisy and denoised examples in the classification. In fact, the intention behind using both noisy and denoised images as input is to allow the model to use parts of both images as needed. Upon acceptance, we will also include an ablation study considering the recommended ablations in the appendix.

We look forward to addressing any outstanding concerns in the discussion period.

[1] Bahjat Kawar, Roy Ganz, Michael Elad. Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance. TMLR, 2023. [2] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I. and Chen, M. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. ICML, 2022.

3. Provide empirical evidence that classifier-free guidance in a semi-supervised setting is hard:

We provide a two-part rebuttal to this comment: first, we argue that comparing between classifier-guidance and classifier-free guidance in fully-supervised settings is not straightforward to begin with. Then, we elaborate upon the challenges in implementation of classifier-free guidance in semi-supervised settings that further complicates comparison between the two methods in semi-supervised settings.

(a) classifier-free guidance and classifier-guidance represent two main categories of guidance methods for diffusion models; classifier-guidance is modular, theoretically motivated and allows for flexible conditioning without requiring to retrain the unconditional score model while classifier-free guidance is a heuristic method that bakes the conditioning into the score-model and requires careful hyperparameter tuning [1]. Comparing between classifier-guidance and classifier-free guidance properly is challenging: for example, previous studies such as GLIDE [2] do this very well, but they rely upon extensive (and expensive) human evaluation of generated samples to show that classifier-guidance often qualitatively underperforms classifier-free guidance; this result is in contrast to the quantitative metrics they computed such as CLIP-scores that usually favoured classifier-guided samples over classifier-free samples. We therefore argue that establishing fair comparison criteria between the two methods would require dedicated research efforts that would be beyond the scope of this paper. Here, we focus on addressing the following deficiencies of vanilla classifier guidance by using denoised images as additional input: 1) Generalization in fully-supervised and semi-supervised settings, 2) Perceptual-alignment of Classifier-gradients, 3) Image Generation Performance.

(b) Training a classifier-free diffusion model with partially labelled dataset is harder because the training relies heavily on labels, as described in our text: “…Ho and Salimans (2022) recommend training with class-label conditioning for 80-90% of each batch and null-token conditioning (i.e., no class label) for the remaining examples whereas we assume that the class-labels are available for less than 10% of the complete training dataset.” If we directly follow the suggestion in that paper, it is easy to see that we will overfit to the labeled examples as we recycle examples from the labeled pool more often than the examples in the unlabeled pool:

In the example of CIFAR10, we assume that 4K examples are labelled out of a total 50k samples. If we consider a batch-size of 512 and select the percentage of labeled examples in a batch to be 80%, we will need 410 labeled examples and 112 unlabeled examples for each gradient step. Overall, this implies that 1 epoch over the unlabeled examples corresponds to 41 epochs over the labeled examples (since, 10 gradient steps correspond to 1 epoch over the labeled data).

Thus, avoiding overfitting to the labelled pool would require separate research efforts–again outside the scope of this paper–-to identify reliable pseudo-labelling strategies that enable a feasible way to follow the recommendation in Ho and Salismans (2022) — in order to get good generative performance.

[1] Bahjat Kawar, Roy Ganz, Michael Elad. Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance. TMLR, 2023.

[2] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I. and Chen, M. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. ICML, 2022.

4. Sampling procedure

Thank you for raising this point: this gives us an opportunity to highlight the fact that the implementation of sampling procedure with DA-Classifiers is indeed as straightforward as with vanilla-classifiers: the gradient of the log-probability estimated by the classifier is used as additional guidance in both cases. In our implementation, we simply concatenate the noisy and denoised example and use this as input to the classifier. In terms of the model architecture, we simply double the in_channels of the input convolution for DA-Classifiers and the remaining architecture is left unchanged: in fact, for the Imagenet experiments in the paper, we expanded the in-channels of the first convolution layer and continued training the pretrained model for 50k steps. We will incorporate this into the paper.

We thank you for your detailed reviews and insightful questions. We will incorporate the minor corrections and suggestions in our revision along with elements of our response below. We address your concerns as follows:

1. Baselines

Even though DLSM-loss and DA-Classifier represent orthogonal improvements over vanilla classifier, as recognized by the reviewer, we directly compare between the two methods on CIFAR10 (trained on VE-SDE) based on the reviewer’s request:

While the FID and IS scores are comparable (especially in relation to the typical FID/IS scores observed in the literature), we note that our class-wise Precision, Recall, Density and Coverage metrics are either comparable or demonstrate a significant improvement. Although we are very interested to analyse the fusion of each methodology, this is out of scope of this current paper: furthermore, a careful analysis of fusion in terms of generalisation, classifier-gradients and image-generation performance also requires specific experiments for finding optimal hyperparameters for each fusion -- for example, the hyper parameters controlling the adversarial attack in Robust-Guidance, the softmax temperature in Entropy-driven Training/Sampling, and so on. We are excited to present these kinds of studies in a thorough and systematic future work. But in the present paper, our focus is, rather, to comprehensively analyse the benefits of using denoised examples as an additional input to the classifier both empirically and theoretically while only using the cross-entropy loss objective. The suggested baseline methods such as DLSM, Robust-Guidance and Entropy-Driven Training/Sampling propose novel loss-objectives additional to the usual cross-entropy loss and are hence orthogonal to our method in which we propose using denoised images as additional inputs to the classifier; furthermore, while the baselines are focused on improving classifier-guidance in fully-supervised settings, we also consider semi-supervised settings. In our initial exploration, we tested to see if the DLSM loss objective can help in semi-supervised settings but did not observe any improvements over just using the cross-entropy loss. We agree that these comparisons strengthen the paper and we will add these results in our revision.

2. Semi-supervised generation results

We thank the reviewer for mentioning this and giving us the opportunity to provide additional numerical evaluations. As mentioned in the paper, the FID and IS scores of images generated with semi-supervised classifiers are very similar to the FID/IS scores of images generated with fully-supervised classifiers: specifically, the FID/IS scores of semi-supervised vanilla classifier and semi-supervised DA-classifier are 2.82/9.61 and 2.35/9.86 respectively. We will include these in the revised version of the paper. The class-wise P/R/D/C values are, however, different for semi-supervised classifiers as compared to fully-supervised classifiers and we do include these in Table 5 (in Appendix B) – where, we can see that the DA-Classifier outperforms noisy-classifier in semi-supervised settings as well.

Thank you for your response. However, I still have the following concerns.

1. Baselines

The authors only compare to the reported values of DLSM among the mentioned baselines. For example, for Table 1 in ED, it appears that they compare the ImageNet experiment with the same settings in the paper, and it seems to perform better than the DA classifier. A comparison and analysis with ED is also necessary.

Additionally, the noisy classifier performance as reported in the manuscript also seems to be the same as the performance reported in the ED paper, so it seems to be extracted from that paper. If the values are reported in the previous work, it would be nice to clearly indicate that.

Also, the authors claim that the proposed method is orthogonal to existing methods, which needs to be supported. This is because it is possible, for example, that the existing methods simply reflect the effect that can be achieved with the Denoising Assistance in a different way. To avoid this concern, the fusion experiments mentioned by the authors are not out of scope, but would be very important to show the orthogonality of the proposed method.

2. Semi-supervised generation results

As far as I know, a revised version can be posted during the discussion period, so I would like to recommend that the discussion mentioned in the author response be added to the revised version.

3. Classifier-free guidance

The author claims that the classifier-free diffusion model could easily overfit in partially labeled datasets. As far as I know, there is no experimental evidence for this, and it would be very important to show this experimentally to support the author's opinion. As the author states, it is beyond the scope of this study to find and compare suitable techniques that do not overfit classifier-free guidance. However, I think it is necessary to show experimentally that the current intuitive application of classifier-free guidance leads to overfitting.

4. Sampling procedure

I understand that the practical implementation can be made not to differ much from the existing classifier guidelines, as the authors mentioned. However, the part I pointed out is that in the DA classifier, there is a noisy sample $x$ and a denoised sample $\hat{x}$, and the denoised sample $\hat{x}$ is computed from $x$. Therefore, I think the derivative for the DA classifier is expected to require a total derivative as I expressed above. I asked for a discussion of how the effect of a change in $x$ is reflected in the denoised sample, and if not, why it shouldn't be.