Noise Matters: Optimizing Matching Noise for Diffusion Classifiers

Thanks for your thoughtful feedback. We sincerely appreciate your recognition that our experiment is thorough and the paper is well-written.

We hope to address all your concerns below.

Concern 1: "Technical novelty or deeper theoretical insight of simple method"

We agree that our method is simple. However, technical novelty is not the contribution that our paper intends to produce, and simplicity $\neq$ no insight.

Firstly, we clarify that the main contribution of our paper is that we analyze the role of noise in the current diffusion classifier (DC). That is, the noise destroys some parts of the image, and DC tries to find the category that can best guide the diffusion model to reconstruct the destroyed parts. And this analysis of the noise role can exactly explain why there is a Noise Instability problem in DC. Then, based on the role analysis, we proposed the two principles (Frequency Matching and Spatial Matching) to guide the noise to destroy the image. The above analysis of the noise role, the noise instability problem, and guidance for noise optimization constitutes the insights of our paper and derives the design of our method.

Secondly, although the framework of our method is simple, the results show its effectiveness with consistent improvements. Simple but effective, it exactly indicates that we catch the key research gap that exists in the Diffusion Classifier.

Thirdly, there are a lot of advantages of the simple method. For example, simple and efficient algorithms are easier to implement in large-scale industries and for subsequent research expansion.

Similarly, [1, 2, 3, 4, 5] are some related and representative works with simple and effective methods.

Concern 2: Formulation and justification for the two proposed principles (Frequency and Spatial Matching)

Thanks for the reviewer's insightful feedback.

Firstly, by analyzing the role of noise in the Diffusion Classifier, i.e., destroying the category-related signals, it is natural to target the specific range of frequency space and spatial space. For the frequency space, there are some works [6, 7] that have discussed the category-related signals are located in the dataset-specific range. The Frequency Bias Theory [6, 7] can be a justification of our proposed Frequency-Matching principle (we will supplement it in the future version). For the Spatial Matching, it's intuitive that even for images within one dataset, the category-related signals should be located in different areas. Thus, we proposed another Spatial Matching principle. These two principles guide us to design the optimization framework with a dataset-specific learnable noise and a Meta-Network to produce the image-specific noise bias.

Secondly, as the reviewer mentioned that "both qualitative and quantitative validations of the proposed principles (frequency and spatial matching) is commendable." These results can verify that the optimization can actually align with these two principles and provide some empirical justifications.

Finally, we agree with the reviewer that a stricter formulation or justifications are required to prove that these two principles are optimal for classification stability and generalization. However, we do not overclaim that our proposed principles are optimal because the optimal principles for classification are a general and open question in this field. Instead, we claim that they are two typical and intuitive principles that can guide an effective optimization framework design.

Concern 3: Difference and contribution of our method compared with the current optimizing noise in the generation field

We apologize for any confusion.

As we discussed in Lines 99-107 of our paper, the noise optimization and selection have been widely explored in the generation scope [4, 5, 8, 9, 10, 11, 12, 13]. However, in the classification domain, the motivation and optimization guidelines are significantly different from the generation domain. We clarify the differences and contributions of our work below.

Different motivations

To the best of our knowledge, we are the first paper to focus on the noise optimization for the discriminative task. The noise instability is a very severe problem that decreases the inference stability and efficiency in Diffusion Classifier and even more general diffusion-based perception scenarios. Our motivation is to reveal the research gap, and one intuitive and effective method to mitigate this problem is noise optimization.

Different optimization guidelines

For the generation domain, the guideline or the final goal of noise optimization is to control the generation process and get the desired synthetic images. Based on this guideline, the statistics-based, attention-based, and VLM-based criteria were proposed to guide the optimization and find a good starting point for the desired generation. In contrast, the guideline for our method is derived from the role of noise played in the Diffusion Classifier, i.e., destroying category-related signals of the image. We argue that a good noise for the Diffusion Classifier can destroy the target part of the images and then fully unleash the diffusion model's reconstruction difference under different category conditions. Based on this guideline, we propose two principles (Frequency Matching and Spatial Matching) to guide the corresponding noise optimization framework design.

Technical gaps

There are technical gaps for noise optimization between generation and classification. For example, leveraging Z-normalization to scale the logits difference and maintain the gradient during training (c.r.f., Lines 166-173).

Our contribution

Our contribution is analysing the role of noise in the Diffusion Classifier and revealing the Noise Instability problem. To mitigate this problem, we proposed two principles to guide the noise optimization and derive the design an effective method.

Concern 4: Comparison with methods like noise-control or augmentation strategies in generative models

The noise-control or augmentation strategies like golden noise [5] and rectified flows with learned priors [14] mainly focus on the optimization objective for generation. However, there is an objective gap between generation and classification.

For the generation task, the objective can be maximizing the preference score or minimizing the RFDS (Rectified Flow Distillation Sampling) loss [14]. However, the optimization objective of classification is to maximize the reconstruction difference under different category conditions.

With different optimization objectives, they can not be comparable. Moreover, apart from the optimization objectives, the noise optimization frameworks of [5, 14] are too simple (directly optimizing an NPNet or parameterized noise) to be compared as baselines because their main significant contributions are to find effective optimization objectives instead of optimization method.

References:

[1] Zhou, Kaiyang, et al. "Learning to prompt for vision-language models." International Journal of Computer Vision 130.9 (2022): 2337-2348.

[2] Zhou, Kaiyang, et al. "Conditional prompt learning for vision-language models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Ma, Nanye, et al. "Inference-time scaling for diffusion models beyond scaling denoising steps." arXiv preprint arXiv:2501.09732 (2025).

[4] Li, Shuangqi, et al. "Enhancing compositional text-to-image generation with reliable random seeds." The Thirteenth International Conference on Learning Representations. 2024.

[5] Zhou, Zikai, et al. "Golden noise for diffusion models: A learning framework." arXiv preprint arXiv:2411.09502 (2024).

[6] Lin, Zhiyu, Yifei Gao, and Jitao Sang. "Investigating and explaining the frequency bias in image classification." arXiv preprint arXiv:2205.03154 (2022).

[7] Wang, Shunxin, et al. "What do neural networks learn in image classification? a frequency shortcut perspective." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[8] Guo, Xiefan, et al. "Initno: Boosting text-to-image diffusion models via initial noise optimization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[9] Wang, Ruoyu, et al. "The silent prompt: Initial noise as implicit guidance for goal-driven image generation." arXiv e-prints (2024): arXiv-2412.

[10] Chen, Changgu, et al. "Find: Fine-tuning initial noise distribution with policy optimization for diffusion models." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

[11] Miao, Boming, et al. "Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[12] Qi, Zipeng, et al. "Not all noises are created equally: Diffusion noise selection and optimization." arXiv preprint arXiv:2407.14041 (2024).

[13] Samuel, Dvir, et al. "Generating images of rare concepts using pre-trained diffusion models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.

[14] Yang, Xiaofeng, et al. "Text-to-image rectified flow as plug-and-play priors." arXiv preprint arXiv:2406.03293 (2024).

We thank the reviewer for the constructive feedback! We hope to address concerns below.

Concern 1: "Theoretical analysis to guarantee that this deterministic approach approximates the expectation of classification under noise sampling"

We apologize for any confusion.

Here we clarify that the Ensembling (expectation of classification under noise sampling) and our Noise Optimization improve the classification performance with different mechanisms.

Why does Ensembling work?

Let the $m$ base predictors be

$\hat{h}_i(x)=f(x)+\varepsilon_i(x), \qquad i=1,\dots,m$,

where $f(x)$ is the true function and the noises $\varepsilon_i(x)$ are i.i.d. such that

$\mathbb{E}[\varepsilon_i(x)] = 0, \qquad \operatorname{Var}[\varepsilon_i(x)] = \sigma^{2}.$

We define their expectation (simple average)

$\bar{h}(x)=\frac{1}{m}\sum_{i=1}^{m}\hat{h}_i(x).$

Then we compare the mean squared error (MSE) of a single model with the ensemble model.

For bias,

$\mathbb{E}[\hat{h}_i(x)] = f(x) \quad\text{and}\quad \mathbb{E}[\bar{h}(x)] = f(x).$

Both the single model and the ensemble are unbiased.

For variance,

\operatorname{Var}\\bigl[\bar{h}(x)\bigr] = \operatorname{Var}\\left( \frac{1}{m}\sum_{i=1}^{m}\varepsilon_i(x) \right) = \frac{1}{m^{2}}\sum_{i=1}^{m}\operatorname{Var}[\varepsilon_i(x)] = \frac{\sigma^{2}}{m}.

With squared‑error loss, $\operatorname{MSE}=\text{Bias}^2+\text{Variance}$. Because the biases are identical, the ensemble’s smaller variance yields

$\operatorname{MSE}\bigl[\bar h(x)\bigr] = \frac{\sigma^{2}}{m} < \sigma^{2} = \operatorname{MSE}\bigl[\hat h_i(x)\bigr], m \ge 2.$

Hence, averaging on multiple sampled noises strictly decreases expected risk.

Why does Noise Optimization work?

To answer this question, we recall the mechanism of the Diffusion Classifier (DC).

Given an inference image, the DC firstly adds some noise to destroy some of the signals that are related to the category. Then, using different categories as the prompts to reconstruct the noisy image. The final prediction is the category that can reconstruct it the best.

In this pipeline, a very crucial part is adding noise because it decides which signals are removed by adding noise. The best situation is trying to remove the category-related signals and maintain other category-independent signals. In that case, the reconstruction difference of diffusion models under different prompts can be fully unleashed.

Thus, our method can learn an optimized noise to target the category-related signals. This kind of optimized noise is usually in the low-probability areas of the standard Gaussian distribution, and thus is very difficult to sample. In comparison, the randomly sampled noises are usually in the high-probability areas of the standard Gaussian distribution but lack the target-destroying capacity.

Additional evaluation to justify why Noise Optimization works

Here we compared the random noise and the optimized noise and reported their mean, variance, and standard Gaussian distribution probability density (pdf).

We can see that, compared with the random sampled noise, the optimized noise's mean is farther away from $0$, variance is farther away from $1$, and the probability density is lower. This demonstrated that the optimized noise is usually in the low-probability areas of the standard Gaussian distribution to target the category-related signals. And this kind of noise is difficult to sample. Thus, we obtain it by optimization.

**Thus, the optimized noise does not try to approximate the expectation of multiple randomly sampled noises. Instead, it improves the classification performance through a different mechanism. That is, it targets and destroys the category-related signals. **

Concern 2: Performance difference from baseline [1]

We apologize for any confusion.

Here, we explain why our reported performance is different from baseline [1].

We take CIFAR-10 with Stable Diffusion-v1.4 as an example. In our paper, we reported that the accuracy is 67.59% (c.r.f., the blue dashed line in Figure 4, CIFAR-10). However, in Table 1 of [1], the CIFAR-10 accuracy is 72.1%.

The reason for this inconsistency is that we only ensemble with five different noises while [1] uses more ensemble noises (such as 20+). The specific number of ensemble scale for different datasets is dynamically decided by its pruning operation (c.r.f., Algorithm 1 in [1]). More noises are used for ensemble, resulting in higher inference cost and better performance.

For our paper, we focus on noise optimization. Thus, for the expensive ensembling method, we only use five noises for ensembling.

Concern 3: Evaluation for cross-dataset generalization

We thank the reviewer for valuable questions.

In response to the question of whether our method can achieve similar cross-dataset generalization to vision-language models (VLMs), we evaluated our method for cross-dataset generalization settings.

Specifically, we optimized the noise and the Meta-Network on 4-shot ImageNet (source dataset) and tested it on the source dataset and eight target datasets. We compared our method with the ensembling methods (5 times computation), and Δ denotes our method's gain over ensembling.

As we can see, the noise and Meta-Network learned on ImageNet can also be used in the eight target datasets. Specifically, six of eight can outperform the ensembling methods, and our method can improve 2.86% accuracy on average.

This demonstrated that the noise optimization can learn some generalized knowledge that is beneficial to the classification, i.e., how to destroy the target part of the image and better utilize the reconstruction capacity to achieve classification. To this end, this result indicates that although NoOp was trained on a particular dataset and at a single timestep, it can also be generalized to other datasets, showing remarkable generalization ability.

Concern 4: Sacrificing the stochastic diversity of diffusion models

We agree with the reviewer's perspective, that is, the fixed and optimized noise will sacrifice the stochastic diversity of diffusion models.

However, we clarify that diversity is not always a positive property across tasks.

Diversity in generation task

For the generation task, sampling different noises as the initial point can lead to diverse generation results. This satisfied the requirements of humans regarding to generation application.

Diversity in classification task

For the classification task, we use the Diffusion Classifier as an example. The stochastic diversity of different noise will lead to the Noise Instability problem (c.r.f., Figure 1 (b) in our paper). In that case, the stochastic diversity is negative for classifying an image.

Ensembling the optimized noises across timesteps

Although our method optimizes to a fixed noise for a given timestep, it can still leverage the diverse knowledge across timesteps. To be specific, for different timesteps, we can optimize different noises. This enables our method to leverage knowledge of the diffusion model conditioned on different timesteps.

We also conducted experiments for ensembling our NoOp across different timesteps in the Section. A.3.3 of the appendix. The results indicated that NoOp can optimize different noises for different timesteps and ensemble the prediction to gain a further performance improvement.

References:

[1] Clark, Kevin, and Priyank Jaini. "Text-to-image diffusion models are zero shot classifiers." Advances in Neural Information Processing Systems 36 (2023): 58921-58937.

We sincerely appreciate your constructive feedback, and thanks for finding that our method is simple and the paper is well-written.

We hope to address all your concerns below.

Concern 1: The detailed architecture design of the Meta-Network and the ablation study of the Meta-Network with various architectures

We apologize for any confusion.

The details of Meta-Network

The Meta-Network is a U-Net architecture with 3 up-sampling layers and 3 down-sampling layers. Each sampling layer consists of 2-D convolutional layers with ReLu activation and BatchNorm (c.f., more details can be found in Lines 108-110 of the appendix and the source code). As the basic module of the U-Net is the convolutional layer, our Meta-Network can be considered as a simple CNN network.

The reason why we use a U-Net architecture is that the Meta-Network's input (image) and output (noise offset) share the same shape. For this kind of scenario, U-Net is a common architecture to maintain the shape.

Additional Meta-Network Ablation

To respond to your question about the ablation of various architectures, parameter count, and FLOPs. We compared different Meta-Network architectures (CNN and ViT) and reported the parameter count, FLOPs, and classification accuracy. For the CNN-based Meta-Network, we used our original Meta-Network in the paper since it consists of convolutional layers. The ViT-based Meta-Network consists of six ViTBlocks with layernorm and residual connection. We conducted the experiments on the 16-shot OxfordPet with Stable Diffusion-v2.0. Other settings are the same as Section. 4.1.

As we can see, both CNN-based and ViT-based Meta-Networks can gain remarkable performance (6-7% accuracy improvement compared with random noise) with only 6-8 M parameters and less than 3 G FLOPs. This indicates that a light Meta-Network with common architectures can accurately learn the noise offset, demonstrating our method is robust and efficient.

Concern 2: Motivation behind the Frequency Matching noise for each dataset

We apologize for any confusion and clarify our motivation for Frequency Matching noise below.

Leveraging the prior that images within one dataset share a common frequency bias.

We agree with the reviewer's perspective: "Noise bias can be directly absorbed into the Meta Network as a final learnable element-wise bias parameter." However, we already have a prior that category-related signals of images within one dataset usually share a frequency range. Based on this prior, we can derive that there is a common noise bias for all images within this dataset. Then, we parametrized this common noise bias as the learnable noise to reduce the learning difficulty of the Meta-Network. In that case, the training can be more stable and converge to a better point compared with only training the Meta Network to learn element-wise bias.

A high-level perspective

In theory, the Meta-Network can satisfy both Frequency Matching and Spatial Matching. But it's more difficult to learn the dataset-specific bias and image-specific bias at the same time. If we add another component to learn the dataset-specific bias, the task will be easier for the Meta-Network (i.e., only focus on learning image-specific bias), making the training more stable.

The experimental perspective

We conducted the ablation study in Figure 6. We can see that jointly training a learnable noise (Frequency Matching) and a Meta-Network (Spatial Matching) is more stable and can achieve a better performance compared with only training a Meta-Network. This result can support the significance of optimizing a shared noise for the Frequency Matching.

Concern 3: Extending to open-set scenarios

Thanks for your insightful question.

To respond to the reviewer's question about how to apply our method to open-set classification, we tested our method in the cross-dataset transfer setting. To be specific, we optimized the noise and the Meta-Network on 4-shot ImageNet (source dataset) and tested on eight target datasets. This can be considered as an open-set problem since a lot of categories in the target datasets did not appear in the source dataset. We compared our method with the ensembling methods (5 times computation), and Δ denotes our method's gain over ensembling.

From the results, the noise and Meta-Network learned on ImageNet can also be used in the eight target datasets. Specifically, six of eight can outperform the ensembling methods, and our method can improve 2.86% accuracy on average.

This demonstrated that the noise optimization can learn some generalized bias that is beneficial to the classification on one general source dataset and then generalize to different target datasets. To this end, this result indicates that our method is robust and can be applied in open-set generalization scenarios.

Thanks for your valuable feedback on our work. We sincerely appreciate your recognition of our method as both novel and effective, as well as the comprehensive evaluations.

We hope to address all your concerns below.

Concern 1: Evaluation for open-world generalization

Thank you for your valuable suggestions.

In response to your feedback, we evaluated our method in the cross-dataset transfer setting to demonstrate its generalization ability on open-set recognition. We optimized the noise and the Meta-Network on 4-shot ImageNet (source dataset) and tested it on the source dataset and eight target datasets. We compared our method with the ensembling methods (5 times computation), and Δ denotes our method's gain over ensembling.

As we can see, the noise and Meta-Network learned on ImageNet can also be used in most of the target datasets. Specifically, six of eight can outperform the ensembling methods, and our method can improve 2.86% accuracy on average.

This demonstrated that the noise optimization can learn some generalized knowledge that is beneficial to the classification, i.e., how to destroy the target part of the image and better utilize the reconstruction capacity to achieve classification. To this end, this result indicates that our method can be further leveraged in open-world scenarios.

Concern 2: Scalability to scenarios with a large number of classes

Thank you for your insightful question.

In the original paper, we conducted experiments on eight datasets. However, the dataset with the largest class number is Flowers102 (102 classes). The reviewer suggested we verify whether our method can be scaled to scenarios with a large number of classes. Thus, we experimented on ImageNet (1000 classes, nearly 10 times more than Flowers102). We used the same setting as Section. 4.1 with Stable Diffusion-v2.0 and only replace the dataset with ImageNet.

This result indicated that our method can be applied to scenarios with a large number of classes, even with only an optimized noise and a small Meta-Network. Our method can outperform the ensemble baseline with only four shots, and the accuracy improves as the number of shots increases.

Concern 3: Computational cost analysis and comparisons

As the reviewer mentioned, the original Diffusion Classifier suffers from high computational cost and slow inference speed, and whether our method can improve or worsen the efficiency. It is worth mentioning that our method can significantly improve the overall efficiency. For this concern, we split into two parts (formal cost analysis and experimental comparisons) to respond.

Formal cost analysis

Given a dataset $\mathcal{D}$ with $K$ categories. We assume that the number of training samples for each category (i.e., the shot number) is $N$. The size of the test set is $M$ (i.e., $M$ test samples).

In that case,

Training Time $\propto KN$,

Test Time $\propto KM$.

Usually, $N=1,2,4,8,16$ for few-shot learning is significantly smaller than $M$. Thus, the training time of our method can be ignored compared with the test time. However, after this low-cost training, the sampling times of noises can be significantly decreased to improve the efficiency of the original Diffusion Classifier.

Experimental comparisons

We used ImageNet as an example to demonstrate how our method can improve the efficiency of the original Diffusion Classifier. The unit of time cost is one hour with 32 NVIDIA V100 GPUs. For the ensemble method, we randomly sample five different noises.

As we can see, our training time is quite smaller than the inference time. Even for the 16-shot training, the total time is only around 24% while the accuracy gains 3.19% compared with the 5-times ensembling method.