Elucidating the Exposure Bias in Diffusion Models

Thank you very much for the insightful and helpful suggestions. We answer each question in detail below:

Q1. The proposed method of Epsilon scaling is not tuning-free as it is sensitive to the choice of hyper parameter of the schedule $\lambda_t = kt+b$ and thus requires extensive hyper parameter tuning. The paper is also missing relevant sensitivity analysis for hyperparameters k and b, and it would be useful to include it (both over large strides and small strides).

A1. Thank you for pointing this out, Reviewer gcKq also asked a similar question. Overall, the parameters k,b are insensitive to the FID result, please refer to our answer A2 to Reviewer gcKq for details. Since the linear schedule $\lambda_t = kt+b$ is determined from its optimal uniform schedule $\lambda_t=b^*$, b of the linear schedule is actually computed rather than brute-force searched. Thus in the linear schedule case, only k is searched with several trials. The sensitivity analysis is of great interest and helpful for the reader to understand the easiness of our method, for example, we present a curve that shows the FID w.r.t the full range of $b$. We decided to include it in the paper later on.

Q2. The process of finding optimal hyperparameters for Epsilon scaling seems to be slow and can be computationally intensive. Usually, generating 10K sampling to calculate FID takes several minutes for small sampling steps and several hours for large sampling steps. Tuning hyper-parameters for each dataset and each choice of sampling length T, can thus require lot of GPU compute and time. While the method itself is simple, tuning this method for optimal performance can be tricky.

A2. As we mentioned above, the determination of $\lambda_t$ is practically easy and not expensive. For each dataset and network, we find the optimal $\lambda_t$ by always using a single A100 GPU, costing from several minutes to three hours. We hope this tackles your concerns about the computational cost of our method.

Q3. In Table 3, for CIFAR-10, values of EDM for VP and VE should be reported for NFE=35 which gets FID = 1.97 (VP) and 1.98 (VE) on CIFAR-10, respectively. Underreporting FID values of EDM is a bit misleading

A3. Thank you for the question, to make sure we measure the true improvement of our method, we re-run the sampling and FID measurement for EDM baseline in the same machine and environment. We report FID=1.97 for EDM (VP, uncond) and FID=1.82 for EDM (VP, cond) in the paper, the latter is different from FID=1.79 reported in EDM, which is possibly due to the machine difference. To avoid misunderstandings, we will add a sidenote in the table to mention the result is reported by re-running the baseline.

Q4. In Figure 5, Figure 9b and Figure 10b, what is the reason for smaller values of $|\epsilon|$ of ADM-ES compared to its values at training time towards end of sampling (time steps 18-21)?

A4. Thank you for the question, timestep 18-21 in these three figures means the beginning of the sampling, which we should clarify in the figure (will update it). The smaller $\epsilon$ norm corresponds to the effect that we push the sampling trajectory away from the original trajectory.

Q5. Could the authors include the number of sampling steps in qualitative results in Figures 11-15?

A5. That is a good suggestion, we will include it in the next version paper.

Q6. Could the authors indicate the amount of compute (number of GPUs as well as time) needed to find optimal hyperparameters?

A6. Finding an optimal uniform $\lambda_t$ takes 6 to 10 trails of searching, which corresponds to several minutes to 3 hours in a single A100 GPU, depending on NFE and dataset resolution. We will include the details in the appendix.

Thank you for your reviews and constructive suggestions. Our relies to each question is listed below.

Q1. Results for the proposed epsilon-scaling mechanism with DDIM solver aren't impressive. There's only a marginal change in the FID score, compared to the DDIM sampler which is used as a popular framework for many diffusion models. Here, the gains are marginal even for cases with reasonable timesteps (~50 or more)

A1. Although the FID gain of Epsilon Scaling in DDIM is less than the gain in DDPM, our method is still useful in DDIM sampler as a simple plug-in sampling solution. Except for the outlier in CelebA 20-step sampling, the FID improvement is also non-trivial and can be significant from 4.06 to 3.38 in CIFAR-10. In the case that DDIM-ES gains little benefit, one can easily get a better sample quality by increasing the value of $\eta$ from 0 to 0.2, 0.5, etc.

Q2. There is little information provided regarding how the scaling factor $k$ and $b$ are selected, and how many hyperparameters were searched for optimal $k$ and $b$. It would also be good to quantify sensitivity to these hyperparameters.

A2. Thank you for the suggestions. First, the FID gain can be achieved in a wide range of k,b, which indicates the insensitivity of the hyperparameters. For example, in Tables 11, 12, we present the FID and parameter b across ADM and EDM baselines. We are going to add a more complete sensitivity graph and analysis in the paper.

Table 11. FID on ADM baseline and CIFAR-10 under different parameters b (b=1 represents the baseline)

Table 12. FID on EDM baseline and CIFAR-10 under different parameters b (b=1 represents the baseline)

Thanks to the insensitivity of parameters $k$, $b$, the parameter searching is effortless, searching for a constant $b$ takes no more than 10 trials for a specific network, dataset and sampling step. We usually take 2 rounds of searching, first doing the coarse search with a stride of 0.005, then doing a fine search with a stride of 0.001. When using linear schedule $\lambda_t=kt+b$, we first find its optimal uniform schedule $\lambda_t=b^*$, then search for the optimal slope $k$ while maintaining $\sum\lambda_t$ unchanged (i.e. $\sum\lambda_t=NFE \times b^*$), in this way, $b$ of the linear schedule is calculated, we got only $k$ is searched. There are already descriptions of the selection of $k,b$ in Appendix A.8, but we will accordingly add more details to the appendix.

Q3. Does the method perform worse than Heun solvers with suboptimal $k$ and $b$ selection?

A3. Due to the wide range of near-optimal solutions, our method still works well under suboptimal k and b, see Table 12 where we use Heun solver.

Thank you very much for the interesting and insightful comments, our answers to each question are shown below:

Q1. Several prior works have observed and identified the "exposure bias" problem studied in the current paper. It would be helpful to discuss them in the paper: (1) Section 4 (practical considerations) and Fig 13 in EDM [1] points out that the neural network tends to remove slightly too much noise. Hence they use an inflated noise to counteract it. I think adding $S_{noise}$ into ODE or SDE samplers is a valid baseline for the current paper. (2) Section 4.2 in PFGM [2] / Section 5, Fig 4.b in PFGM++ [3] further dig into the exposure bias problem and show that the strong norm-t relation in the diffusion model is the cause.

A1. We think our Epsilon Scaling is consistent with EDM’s statement: the neural network tends to remove slightly too much noise, because our solution scales down the predicted noise, thus avoiding removing too much noise. We will update the SDE sampler experiments once we get the results. As for PFGM++ discussion, please refer to our answers A2 and A3.

Q2. The current paper demonstrates the discrepancy of the prediction norms (Fig 2) without further examining the reason. [2] seems to offer a plausible explanation for this phenomenon. If their argument is true, this "exposure bias" mainly occurs in large $t$ (norm-t is strongly correlated). I guess one simple experiment to validate the hypothesis is to use true score up to a certain time during sampling and study the effects (like current Fig.2, but the green and red curve would be the same in [t,T]).

A2. First of all, we reckon the strong norm-t opinion does not contradict our claim that the prediction error of $x_0$ is the cause of exposure bias. We are convinced that PFGM++ is robust to prediction error compared with diffusion models. So they are two concepts and can be concurrent. Also, we argue that exposure bias exists in the whole sampling chain of diffusion models. The experiment in paper [1] supports our opinion, where they measure the FID by leaking the ground truth $x_t$, $t \sim(1, T)$ and $T=1000$, to the network at the beginning of sampling, then the sampling starts at $x_t$. We replicate their results on ADM and ImageNet 32 in Table 6. We see that the FID is still far from near-zero even though the sampling starts at $x_{300}$ and $x_{100}$ which jump over the large t area (norm-t is strongly correlated).

Table 6. FID on ImageNet 32

[1] Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. ICML, 2023

Q3. It's encouraging to see that the method improves over EDM, especially in the small NFE regime. Could the authors try to apply the method on PFGM++, which is claimed to be more robust to the "exposure bias"? It could also verify if or not the exposure bias is due to the strong norm-t correlation pertaining to diffusion models.

A3. We think PFGM++ is an interesting baseline to apply our method. We present the results on PFGM++ (D=128) and PFGM++ (D=2048) under unconditional sampling (see Table 7, 8). We found that Epsilon Scaling (ES) is compatible with PFGM++ and even enjoys the extra benefit: under very small NEF regimes, the FID gain of ES is still significant. We believe the robustness to prediction error of PFGM++ is the reason. We would like to include the full experiments on PFGM++ in the main paper and discuss the strong norm-t relation in the paper as well.

Table 7. FID on PFGM++ baseline (D=128) and CIFAR-10

Table 8. FID on PFGM++ baseline (D=2048) and CIFAR-10

Regarding the small NFE regime on EDM baseline, we present the results of using the Euler solver below, since we found that, when NFE is smaller than 10 in EDM, the Heun solver performs much worse than the Euler solver mainly due to the twice-larger stepsize and severe truncation error.

Table 9. FID on EDM baseline under small NFE regimes (CIFAR-10 and unconditional)

Q4. I don't immediately see why the quotient of $\Delta N(t)=|\epsilon_s| / |\epsilon_t|$ can be translated into the scaling factor at $\epsilon (x_t, t)$ during sampling. Because $\Delta N(t)=|\epsilon_s| / |\epsilon_t|$ and $\Delta N(t) = \int_{t}^{T} \lambda_t$ only implies that $\frac{d|\epsilon^s (x_t, t)| / |\epsilon^t (x_t, t)|}{dt} = \lambda_t$.

A4. Updating soon

Dear reviewer, we summarize our points of view below:

1. Our Epsilon Scaling is consistent with EDM’s statement: the neural network tends to remove slightly too much noise. Our solution scales down the predicted noise thus avoiding removing too much noise.
2. The reasoning about the exposure bias is fully analysed throughout Section 3 in our paper.
3. We have shown that exposure bias exists in the whole chain of sampling, rather than mainly in the large t area where norm-t is strongly correlated.
4. We agree that PFGM++ framework is robust to prediction error, but our argument "the prediction error of $x_0$ is the cause of exposure bias" does not contradict PFGM's opinion. Also, our experiments have shown the existence of exposure bias in PFGM++ and our method can consistently improve it. Therefore, according to your hypothesis, we have verified that the exposure bias is not due to the strong norm-t correlation.
5. The $\Delta N(t)$ question is unclear to us for two reasons. First, the solution of your reverse ODE is $x_t = \frac{T}{t}x_T$, this leads to $x_0 = \infty$ which is contradictory with your defined forward diffusion $x_t=-t^2\epsilon$ because $x_0=0$ in this case. Second, the derivation of $\Delta N(t)=\int \lambda_t/t dt$ looks obscure to us and we had tried to get your clarification but failed.

Overall, the questions you raised were very interesting and insightful, but we deeply regret that your zero engagement in the discussion provided us with no opportunity for effective communication. Finally, we are always glad and open to having future discussions after the end of this discussion period. Thanks!

Thank you very much for your helpful comments and we now answer each of your questions.

Q1. The main concern lies in the assumption at the start of the derivation. In part 3.2, the authors assume that the reconstructed image x_{\theta}^t at the sampling process follows the Gaussian distribution, where the mean is the GT image x_0, and the variance is the Gaussian noise. This conflicts with some intuitive observations. For example, the reconstructed image x_{\theta}^t is often a degraded version of the GT image, and the mean of x_{\theta}^t is different from x_0.

A1. The generated sample is the result of multi-step sampling, where the Gaussian approximation of $x_0$ prediction error holds in a single step (please see our answer A1 to reviewer nv6a). For multi-step sampling, this Gaussian might be loose and exhibit the mean drift.

Q2. Some formulations, notations and explanations in part 3.1 and 3.1 are redundant. It is highly recommended to improve the organization and the writing of this section.

A2. Thanks for your suggestions, we will simplify section 3.1 since the DDPM formulation is already well-known in the field.

Q3. It is necessary to discuss the further meaning and defects of exposure bias correction. For instance, the necessity of matching the forward and sampling process. Reducing such bias forces the generated images to be more similar to the training images in distribution, i.e, more data repetition in the generated images. The authors are encouraged to give more discussions.

A3. From the sampling trajectory perspective, exposure bias correction indeed tries to match the training and sampling trace. The main benefit of this match is to avoid the intermediate sample $\hat{x_t}$ falling into the problematic vector field that has never been trained in the training phase. From the sample quality perspective, the exposure bias correction corresponds to the correction of overexposure, underexposure, and detail defects.

We empirically observed that the generated image samples before and after applying Epsilon Scaling share the same semantic mode. Thus, the exposure bias correction does not affect the diversity of the synthesised samples. Our experiments of recall further show that the correction of exposure bias results in even slightly better sample diversity (see Table 10 below). Thus, reducing exposure bias does not lead to data repetition. We plan to add this discussion in section 5.6

Table 10. Recall of ADM and ADM-ES using 100-step sampling

Thank you very much for the reviews and helpful comments, we now reply to your questions:

Q1. The assumption that the model value $x_{\theta}^t$ is averaged to the true $x_0$ should be more verified.

A1. The Gaussian assumption of the single step $x_0$ prediction error was first proposed at Analytic-DPM [1] and later on verified at [2] in their section 5.3 and appendix A.5, where they use Shapiro–Wilk statistical test to verify that $x_{\theta}^t-x_0$ is nearly an isotropic Gaussian, we will also mention this in the paper.

[1] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. ICLR, 2022b

[2] Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. ICML, 2023

Q2. Analytic-DPM is also a simulation-free (only calculates the optimal variance over the timesteps) This paper proposed that the limit of the Analytic-DPM method is that this is not advantageous in the low-NFE regime, but this paper reproduces well even in low-NFE regime (NFE=10), if the variance clipping in high-SNR (last or second last sampling step) timesteps is held. The fair comparison in various NFE regime should be done, which can affect the scoring of this paper..

A2. Our observation for Analytic-DPM is that their optimal sampling schedule $\beta_{t}^*$ underperforms the DDPM baseline in the large NFE regime (400~1000), and $\beta_{t}^*$ indeed performs well in the mid and low NFE range. We appreciate your suggestions of verifying our method in various NFE regimes and we add the results in A3

Q3. What about the case that the NFE is less than 20 or greater than 100?

A3. Thanks for mentioning this, our method also works well when NFE<20 and NFE>100. For example, in the ADM baseline, our ADM-ES consistently gains FID improvements on different datasets using NFE=10 and constant Epsilon Scaling (see Table 1). We will roll it out to all baselines we used in our paper after the short discussion period.

Table1. FID under NFE=10 on ADM baseline

As for large NFE regions, we show the full results on Analytic-DDPM baseline (Table 2) and ADM baseline (Table 3), where we use constant $\lambda_t$ for Epsilon Scaling (ES). However, we highlight that it is unnecessary to use very high NFE in practice since NFE=100 achieves the near-optimal FID in most cases while being computationally efficient. Many works in the literature have also noticed this phenomenon [3][4][5].

Table 2. FID on CIFAR-10 using Analytic-DDPM baseline (linear noise schedule)

Table 3. FID on CIFAR-10 using ADM baseline

[3] Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In ICML, 2021

[5] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., & Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS, 2022

[4] Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. NeurIPS, 2023

Q4. At least for small NFE, I recommend adding some ablation studies on fully-searched $\lambda_t$ with respect to all timesteps, rather than fixing this to the fixed value $b$ or taking a linear approximation

A4. Fully searching $\lambda_t$ for each timestep is an expensive ablation study even for small NFE as we need either multi-round simulations for each timestep or exponential effort of brute-force search. Due to the short discussion time, we found it difficult to present the ablation results now, but we will add these to the paper in the future.

Q5. It will more support the method, if the bias of the expected noise $\epsilon_{\theta}^s$ or $\epsilon_{\theta}^t$. This helps the reader to understand that by simply scaling down the expected noise calibrates the sampling steps, without taking bias (i.e. translating) the noise

It will be also helpful also to be mentioned if the Analytic-DPM requires the heavy clipping of the noise variance $\beta_t$ in the near-signal part of the sampling, which causes

A5. It looks like these two questions are sort of incomplete, would you mind re-stating your questions?

Q6. In the line below Equation 7, does $q(x_t | x_{t+1}, x_{\theta}^{t+1})$ have the same distribution as $q(x_{t+1} | x_t, x_{\theta}^t)$ ?

A6. Thank you for pointing this out, we intended to say that $q(\hat{x_t}| x_{t+1}, x_{\theta}^{t+1})$ and $q(x_{t-1}| x_{t}, x_{\theta}^{t})$ are in the same function format and they are just the difference in notation. Clear clarification would be made, such as, 'we now compute compute $q(\hat{x_t}| x_{t+1}, x_{\theta}^{t+1})$, which share the same function format as $q(x_{t-1}| x_{t}, x_{\theta}^{t})$, by substituting with the index $t+1$ and using $\hat{x_t}$ to highlight that it is generated in the sampling stage'

Q7. The exposure bias term $\delta_t$ should be more precisely denoted in the main section. This proposed metric is only introduced in detail in the appendix section.

A7. Thanks for the suggestion, we also realized that the exposure bias $\delta_t$ is important for the readers to understand the metric and we will move it to the main paper.

Q8. It will be helpful if the Analytic-DPM and the existing exposure bias methods are also considered as the benchmark, even though this does not use the baseline reverse diffusion process by modifying.

A8. We tested the performance of our Epsilon Scaling on Analytic-DPM, PFGM++ even though they are not the initial target baselines of our method (see Table 2 for the results of Analytic-DPM). Regarding the exposure bias solutions, we also directly compare Epsilon Scaling with Input Perturbation [2] (a training regularization to reduce exposure bias) on the baseline ADM. The results shown in Table 4,5 demonstrate the effectiveness and generalization of Epsilon Scaling. We use constant $\lambda_t$ for Epsilon Scaling (ES) below.

Table 4. FID on CIFAR-10 (2nd Heun solver and D=128)

Table 5. FID on CIFAR-10