Is Noise Conditioning Necessary for Denoising Generative Models?

Thanks a lot for the insightful feedback and the supportive comments to our work!

1.Definition & Benefits of removing noise conditioning

Reviewer FoCJ: Without the noise conditioning, how are the models trained? … is it that we still train models with Eq. 2, but only replace $NN_{\theta}(\mathbf{z}\mid t)$with$NN_{\theta}(\mathbf{z})$?

Regarding the question: Correct. Without noise conditioning, the neural network is still trained with Eq.2 but only with $NN_{\theta}(\mathbf{z}|t)$ replaced with $NN_{\theta}(\mathbf{z})$.

Reviewer FoCJ: … we actually still use the noise conditioning. That is, the t is implicitly used since $z=a(t)x+b(t)\epsilon$.

This is the main message we wanted to convey: the noisy image contains information of the noise level $t$, and this is why explicit noise conditioning in the neural net is not needed.

Reviewer FoCJ: I am curious about the objective of removing the noise conditioning.

Our motivation actually roots from the curiosity on the necessity of noise conditioning, which is the “common wisdom” in most of the previous works on denoising generative models. In our opinion, the value of removing noise conditioning lies in challenging common sense and also lies in its theoretical meaning. Finally, as a direct result, some models (such as DiT+FM, as we share with Reviewer gvd2 and 7y3v) can enjoy improvement in removal of noise conditioning.

2. Low-Dimensional behavior

Reviewer FoCJ suggests to investigate model behavior in the senario where dimension $d$ is low, since our theory assumes a large enough $d$ value. This suggestion is very insightful.

Inspired by that, we explored low-dimensional cases, where the approximation $d\gg 1$ does not hold and our theoretical analysis is ineffective. Specifically, we did experiments on the toy “two moons” dataset (which has $d=2$) with Flow Matching (i.e. Flowing from standard Gaussian to the two-moons dataset) models, and visualize the generated samples in the noise-conditional and unconditional settings. The results are shown in the figure below:

reb — Postimages

By visualizing the generated samples, one can see that removing noise conditioning leads to a significant performance drop in low-dimensional cases, preventing proper modeling of the distribution.

Thank for the constructive feedback and supportive comments. Here, we address the concerns regarding classifier-free guidance (CFG), experimental random variance, model-wise analysis, and explanation on performance of 1-RF.

1. Latent Diffusion (DiT) & Classifier-free Guidance

Reviewer 7y3v commented, One issue is that diffusion sampling often involves classifier-free guidance, which introduces instability through extrapolation in $x_t$. Noise-unconditioning may exacerbate these instabilities.

This comment on the risk of unstability of generation with classifier-free guidance is very valuable. Also, we believe that incorporating larger-scale experiments such as Latent Diffusion (DiT) will definitely make our work more solid.

To address the concern, we conducted experiments DiT + FM (i.e. SiT) on the ImageNet 256x256 dataset with CFG. Here, all experiments use the same configuration as the original paper, using Euler sampler with 250 steps. The results demonstrate that our findings extend successfully to larger scales:

Notably, removing noise conditioning improves performance at optimal CFG scales. Also, for many different CFG scales, removing noise conditioning improves performance. We can see that in the setting of FM on large-scale dataset, the behavior of removing noise conditioning remains consistent with experiments in our paper.

These results show that it’s feasible to extend our conclusions to large-scale diffusion models, and our observations are robust with respect to classifier-free guidance.

2. Statistical Robustness of the Results

Reviewer 7y3v: In Figure 7, (a) shows 1-RF achieving an FID of 3.01, yet (b) achieves 2.58 and (d) achieves 2.61. Does this suggest that the baseline result of 3.01 was an outlier due to variance in training or sampling?

To address this concern, we report the variance of FID on 5 trials with different random seeds (the top row), as well as using three different training checkpoints (the bottom row). Results are as follows. These numbers correspond to the last row in Figure 7(a).

Note that here a, b, c, d are different settings.

These results confirm that the performance enhancement is not due to a statistical anomaly. The generation quality measured by FID-50k is robust enough for the evaluation.

3. Over simplification?

Reviewer 7y3v: In Statement 1 and 2, the focus is on FM , while Section 5 is heavily EDM-oriented… oversimplifies the issue.

We clarify that our analysis is valid for general cases. While we used FM formulation as our primary example, our analysis extends to all listed diffusion models. We chose FM for its notational simplicity and clarity, avoiding unnecessary complexity while maintaining theoretical rigor.

4. Explanations on 1-RF: Curvature of the Path

We greatly appreciate your insightful perspective on how 1-RF inherently aligns with the diffusion ODE trajectory.

Our theory mainly focuses on the bounding the error of removing noise conditioning, demonstrating that most models can work with the removal of noise-conditioning. We have to modestly admit that our theory is incapable of explaining why 1-RF becomes better when the noise-conditioning is removed.

Nevertheless, we believe that using the curvature of trajectory to build connection between the noise-unconditional model’s performance to specific diffusion formulations is a very promising future direction, and we would like to explore in more depth on this topic in follow-up works.

We sincerely appreciate the thoughtful feedback and recognition of our theoretical contributions. Below, we address the concerns regarding experimental scope and practical applicability to large-scale diffusion models.

1. Large-scale experiment

To address reviewer gvd2’s concerns on experimental scale, we conducted additional large-scale experiments with DiT + FM (i.e. SiT) [1, 2] on ImageNet 256x256 dataset. All experiments use the same configuration as the original paper, using Euler sampler with 250 steps. The results demonstrate that our findings extend successfully to larger scales:

Notably, removing noise conditioning improves performance at optimal CFG scales. Also, for many different CFG scales, removing noise conditioning improves performance. We can see that in the setting of FM on large-scale dataset, the behavior of removing noise conditioning remains consistent with experiments in our paper.

2. Performance on Other Types of Samplers

Reviewer gvd2: “It would be nice to include DPM-Solver, DPM-Solver++-type samplers as well, since these are still well-used samplers in practice, and also these samplers use multiple tabbed filtering. The performance for these type of samplers might be closely related to DDIM's failure as well.”

This is a very interesting extension to our current work, generalizing our theory and experiment to a wider range of commonly used sampling methods.

Regarding the DPM solver equation:

$$x_{t_i}=\frac{\alpha({t_i})}{\alpha(t_{i-1})}x_{t_{i-1}}-\sigma({t_i})(e^{h_i}-1)\epsilon_{\theta}(x_{t_{i-1}},t_{i-1})$$

(Eq. 3.7 in [3]), which, in our notation (Eq. 4 in our paper), becomes

$$\kappa_i = \frac{\alpha_{i+1}}{\alpha_{i}},\quad \eta_i = -\sigma_{i+1}(e^{h_{i+1}}-1),\quad \zeta_i=0$$

This $\kappa_i$ for DPM solver behaves very similar with DDIM (which has $\kappa_i=\sqrt{\frac{\alpha_{i+1}}{\alpha_i}}$), which leads to large error bound value (the term $\prod \kappa_i$ dominates the error bound, and $\prod \kappa_i$ approaches infinity due to division by 0). Thus, we can expect a bad performance similar with DDIM in the noise-unconditional scenario by our theory.

To further confirm this intuition, we perform numerical calculations of bound values as well as FID evaluations for both DPM-Solver and DPM-Solver++ (which has a similar formulation as DPM-Solver but uses $x$-prediction). The bound value of both of them are on the order of $10^6$ (which is of the same order with DDIM), and using DPM solver fails to generate good images (FID $>50$).

These results shows that our theoretical error bound (in Section 4.4) can qualitatively predict the performance accurately, demonstrating that our theory generalizes on even these carefully designed samplers.

3. Other Suggestions

We appreciate the feedback regarding figure clarity and will incorporate the suggestions to enhance the presentation of our results.

References

1. Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2023.
2. Ma, Nanye, et al. "Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.
3. Lu, Cheng, et al. "Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps." Advances in Neural Information Processing Systems 35 (2022): 5775-5787.
4. Lu, Cheng, et al. "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models." arXiv preprint arXiv:2211.01095 (2022).

Thanks for the thoughtful review and positive feedback regarding our work’s novelty and impact!

We agree that incorporating the suggested reference Noise2Noise: Learning image restoration without clean data will provide valuable context for the derivation in section 4.1. We will revise the manuscript to include this reference.