Data Prediction Denoising Models: The Pupil Outdoes the Master

Thank you for liking our work, we will incorporate your wonderful suggestions in the revision. We will address your concerns one by one in the following paragraphs.

Q1. training DPDMs requires an auxiliary diffusion model, a retrained DM, and a multi-step denoiser model which may bring additional memory costs and training time requirements.

A1. We appreciate your keen intuition. We acknowledge that the additional diffusion model ($\boldsymbol{s}_ \phi$ in Algorithm 1) brings additional memory cost. However, this additional memory cost is limited because the memory bottleneck of training lies in the computational graph of the backpropagation, instead of only saving one more model. In DPDM's training, the auxiliary model $\boldsymbol{s}_ \phi$ and the model $\boldsymbol{d}_ \theta$ are updated alternatively, which means that the other model's parameters are fixed and do not participate in back-propagation when one model is being updated. So the memory cost for back-propagating through the computational graph is almost the same as one model. To quantitatively measure how much additional computational costs are brought in, we compare DPDM's training with another multi-step model, the consistency model (consistency distillation, CD) in terms of memory and computational costs in Table 6 in the Appendix of the submission.

Table 6 shows that DPDM brings in minor additional memory costs than CD (10.40 over 9.55). This is because the DPDM only needs additional GPU memory to save the auxiliary model. But $\boldsymbol{s}_\phi$ and $\boldsymbol{d}_ \theta$ are updated alternatively, so their computational graph does not interact. As a result, the memory bottleneck caused by computational graphs and back-propagation does not bring more costs to DPDM.

As for the wall-clock time for 1K iterations, we see that DPDM costs 0.0728 seconds, while the CD costs 0.0489 seconds. This is because each iteration of DPDM consists of two alternate steps as we show in Algorithm 1. Overall, the DPDM costs almost the same GPU and CPU memory as the baseline CD, but about 1.5 times wall-clock time than the CD for each iteration.

Q2. it does not explore or provide results for higher-resolution image generation or other types of data generation tasks. Can the authors demonstrate the results for generating higher resolution images such as 256x256 or higher and how DPDMs compare with DMs?

A2. We appreciate your suggestions on larger-scale experiments such as high-res image modeling or text2img generation. However, these experiments demand substantial computational resources beyond our current reach. For example, training a Stable Diffusion t2i model consumes a cloud cluster with 256 Nvidia A100 GPUs for about 150,000 hours, costing a total of about 600,000 USD, which is far beyond our financial affordance. However, in this work, we have demonstrated strong performances of DPDM on standard benchmarks, and we are glad to try to explore the possibility of scaling DPDM to larger-scale tasks such as high-res data generation and t2i generative models within our financial permissions.

Q3. Can the authors comment on the training and inference times and compute requirements for DPDMs and comparable DMs?

A3. The training of DPDM starts with an initialization of a pre-trained DM. We take the CIFAR10 dataset as a demonstration. Empirically, a DM on the CIFAR10 dataset is supposed to converge on 4 Nvidia-V100 GPUs within 2 wall-clock days. Starting from a pre-trained DM, we empirically find that a DPDM converges within 12 hours. Therefore, if we already have a DM, the total training time of DPDM is about 1/8 of a DM. If we do not have a pre-trained DM, therefore, we are supposed to first pre-train a DM and then train a DPDM. The total time for training DPDM from scratch costs about 9/8 of diffusion models.

As for the inference. We implement the DPDM with the same architecture as the diffusion model (i.e. a weight-sharing network $\boldsymbol{d}_\theta(x_t,t)$ to parametrize multiple denoisers), therefore, the inference cost per step of DPDM is the same as the diffusion model, so NFE is a fair metric for evaluating the inference efficiency when comparing DPDM and the DM. Table 4 in the submission has a relatively clear comparison of the inference efficiency of DPDM and DM. In Table 4, the FID value of DPDM converges below 3.0 (which is regarded as a line of good performance) with 6 NFEs, on the contrary, the baseline diffusion model's FID is still larger than 3.0 with even 18 NFEs. Since both models have the same cost for each NFE, we conclude that the DPDM is about 3 times more efficient than diffusion models for few-step generations.

We really thank you for your useful suggestions on notations and clarifications which we will incorporate in the revision. We hope our answers can resolve your concerns.

Thank you for your reviews. We will address your concerns one by one in the following paragraphs. Before that, we first give a summary of the main contributions of our work.

In this work, we propose a novel stand-alone generative model termed the Data-prediction denoising model (DPDM), which achieves significantly better generative performance than baseline diffusion models under the few-step sampling settings. We characterize the training process of DPDM through a novel smoothed KL divergence minimization formulation which results in significantly stronger data-recovery ability than the baseline diffusion model.

Next, we address your concerns one by one.

Q1. However, training this separate denoiser is the same as the training loss of the original diffusion model. From this aspect, it seems like this idea is more like a fine-tuning (or just training it longer) method. I cannot follow why such a fine-tuning idea is effective when having a small sampling step. Could the author clarify this point?

A1. First, we would like to emphasize that our baseline EDM diffusion model is obtained from official checkpoints which are trained till convergence. Second, we would like to emphasize that the training objective of DPDM is totally different from the diffusion model's training. Therefore, the training of DPDM has no relation to fine-tuning with diffusion model loss function.

More precisely, the training objective of the diffusion model is Equation 3.1, which encounters an L2 minimization to train weak denoisers. On the contrary, the DPDM is based on minimizing a novel smoothed KL divergence between the denoiser distribution and the ground truth data distribution. To minimize the smoothed KL for multi-level denoisers, we derive an explicit gradient formula of the denoisers' parameter $\theta$ with respect to a combination of the smoothed KL divergences, i.e. Equation 3.4. Overall, we propose a practical algorithm 1, which implements such a divergence minimization formulation to obtain stronger denoisers.

Q2. The sampling method considers the clean image as the mean of the next step distribution, which contradicts the theoretical analysis of DDPM. Could the author also justify this point?

A2. We do not clearly understand what you mean by contradicting the theoretical analysis of DDPM. As far as we guess, we assume you have mistaken our sampling algorithm 2 as the same as DDPM's sampling algorithm. We would like to emphasize that the DPDM is a stand-alone generative model that is totally different from diffusion models (so is the DDPM). And its model training and sampling are also different from the diffusion model (DDPM). More precisely, each sampling step of DDPM gradually predicts the mean of the next-step denoised distribution. On the contrary, each step of DPDM's sampling step directly predicts clean data and then adds a Gaussian noise to obtain samples from the less-noisy distribution. Provided that the DPDM is a different generative model, we do not see the necessity of its sampling algorithm to match the DPDM's sampling algorithm.

Q3. This paper claims it provides a solid mathematical foundation for the proposed method, which is unclear means?

A3. In section 3.2, we elaborately establish the mathematical formulation of enhancing the data-recovery ability of denoisers as a smoothed KL divergence minimization problem. This part highlights the math foundation of DPDM (through probability divergence minimization) and helps to derive a practical training algorithm of DPDM (Algorithm 1).

We hope our answers have resolved your concerns, and if you still have any concerns, please let us know, and we will be glad to further address them.

Dear reviewer:

Thank you for your reviews! The author-reviewer rebuttal period is coming to a close soon. We sincerely hope that our responses have adequately addressed the concerns you have raised.

In this work, we introduce a new class of multi-step generative models, named Data-prediction Denoising Models, which show significantly stronger performance than diffusion models under the few-step setting. We also explore the weak data distribution recovery ability of the diffusion model's denoisers and practical solutions for enhancing them. We believe that our findings and solutions may benefit future researchers in either improving diffusion models or proposing new generative models.

We hope our rebuttal has resolved your questions. If you still have any unresolved concerns or additional questions, please do let us know! We would be very glad to provide more clarification and address any remaining issues.

Best wishes,

Authors of the submission #5134.

Thank you for your helpful suggestions, we will incorporate them in the revision. We will address your concerns one by one in the following paragraphs.

In this paper, we propose a stand-alone generative model, the DPDM, which draws inspiration from diffusion models from a denoiser perspective. The DPDM shows strong performance under few-step generative settings.

Q1. The experiments would benefit from demonstrations with high-resolution datasets to illustrate the generalizability of the method.

A1. We appreciate your suggestions on larger-scale experiments such as high-res image modeling or text2img generation. However, these experiments demand substantial computational resources beyond our current reach. For example, training a Stable Diffusion t2i model consumes a cloud cluster with 256 Nvidia A100 GPUs for about 150,000 hours, costing a total of about 600,000 USD, which is far beyond our financial affordance. However, in this work, we have demonstrated strong performances of DPDM on standard benchmarks, and we are glad to try to explore the possibility of scaling DPDM to larger-scale tasks such as high-res data generation and t2i generative models within our financial permissions.

Q2. Metrics such as inference memory, seconds per iteration at inference, and overall inference time until convergence should be provided for a valid comparison. How do inference memory and time compare with state-of-the-art baselines?

A2. We appreciate your suggestions. First, we would like to say that when implementing DPDM, we use a weight-sharing multi-level denoiser network $\mathbf{d}_\theta(x_t,t)$ to parametrize multiple denoisers in a single network. This network follows the same architecture as the baseline EDM diffusion model. Therefore, the inference costs of DPDM per step are exactly the same as the EDM diffusion model. From this view, the NFE is a fair metric for comparing the sampling efficiency of the DPDM and EDM diffusion models. As for the inference cost for convergence, Table 4 in the submission has a relatively clear demonstration. In Table 4, the FID value of DPDM converges below 3.0 (which is regarded as a line of good performance) with 6 NFEs, on the contrary, the baseline diffusion model's FID is still larger than 3.0 with even 18 NFEs. Since both models have the same cost for each NFE, we conclude that the DPDM is about 3 times more efficient than diffusion models for few-step generations.

However, as we pointed out in Section 6, the performance of DPDM with sufficient NFEs (FID of 2.68 with 8 NFEs) is still worse than the diffusion model (FID of 1.97 with 35 NFEs). This is currently a limitation of DPDM and we are very glad to further explore the performance ceiling of DPDM through advanced training configurations, tricks, and other aspects.

Q3. What does $\tilde{x}$ stand for in Equation 1? I don't think it has been defined anywhere.

A3. We appreciate your careful reading and We feel sorry for the confusion. There is a typo in Equation 1 in Algorithm 1. Here the $\tilde{x}_t$ is a mistake for $\hat{x}_t$ in the expectation operation. Thank you for pointing out the typo.

We will incorporate your suggestions on other minor issues in the revision. We hope our answers have resolved your concerns. If you still have concerns, please let us know.

Thank you for your useful feedback, we will incorporate them in the revision. We will address your concerns one by one in the following paragraphs. Before that, we first give a summary of the main contributions of our work.

In this work, we propose a novel stand-alone generative model termed the Data-prediction denoising model (DPDM), which achieves significantly better generative performance than diffusion models under the few-step sampling settings.

Q1. DPDM shares similarities with Diff-Instruct [1], which also performs well with few sampling steps. Could you give some more discussions?

A1. We highly appreciate your keen intuition. In this work, when defining the DPDM, we draw inspiration from two amazing works: the diffusion models (i.e., the EDM work [2]) and the Diff-Instruct [1] (which achieves strong performance for single-step generation). The DPDM combines the advantages of both worlds, resulting in strong few-step generative performances. However, such a combination of both works is non-trivial, because:

(1). Though the Diff-Instruct achieves state-of-the-art single-step diffusion distillation performance during its publishing, it does not support multi-step sampling, because the Diff-Instruct can only train a one-step student model to directly transport a Gaussian latent vector to obtain generated samples. The most amazing part of Diff-Instruct is its mathematical foundations (i.e. the formulation of divergence minimization when deriving the algorithm). However, the possibility (with improved performances) of incorporating such a math formulation for stronger few-step generative modeling has not been yet explored. To our knowledge, we are the first work that explored and successfully achieved a strong few-step generative performance with an FID of 3.66 on ImageNet$32\times 32$ data with 8 sampling steps, which is stronger than Diff-Instruct (which only supports one-step sampling) and baseline diffusion models.

(2). Though diffusion models can generate promising data samples when the sampling steps are sufficient, they perform poorly under a few-step setting (even with the best solvers). In this work, we analyze and locate the reason for such a performance drop under a few-step setting as the weak distribution recovery ability of diffusion models' denoiser (Section 5.1 in the submission). Such an observation of weak denoisers not only motivates us to propose DPDM with improved denoisers but also sheds light on other future work which potentially aims to address the weakness of diffusion models.

(3). We incorporate all enhanced denoisers through a single $\mathbf{d}_\theta(x_t,t)$ network with weight sharing. This overcomes the heavy computational and memory cost when storing multiple denoisers.

Q2. Put more discussions on Eq.3.4.

A2. We feel sorry for the confusion. Equation 3.4 is a generalization of Equation 3.3 which only considers a single noise level. For a single noise level, we only consider a single denoiser. When generalizing to multiple denoisers, we use a weight-sharing multi-level denoiser network $\mathbf{d}_ \theta(x_t,t)$ to parametrize multiple denoisers in a single network $\mathbf{d}_\theta$.

Therefore, the gradient formula to update $\theta$ of $\mathbf{d}_\theta(x_t,t)$ is a weighted combination of the single-step gradient Equation 3.3 through a weighting function $w(t)$ over diffusion time t (or over different noise levels). We really thank you for your careful reading and we acknowledge that there is a small typo in Equation 3.4.:

the denoiser should write $x_t = \mathbf{d}_ \theta(x_0 + \sigma(t)\epsilon, t) + \tilde{\epsilon}$ instead of $x_t = \mathbf{d}_ \theta(x_0 + \epsilon) + \tilde{\epsilon}$, and the Equation 3.4 should be modified to

$\operatorname{Grad}(\theta)= \int_ 0^ T w(t)\mathbb{E}_ { \boldsymbol{x}_ t=\boldsymbol{d}_ \theta(\boldsymbol{x}_ 0 + \sigma(t)\epsilon, t)+\tilde{\epsilon} \atop \boldsymbol{x}_ 0 \sim p_ 0, \epsilon,\tilde{\epsilon} \sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{I})} \big[\boldsymbol{s}_ {q_t}( \boldsymbol{x}_ t, t) - \boldsymbol{s}_ { p_ t}( \boldsymbol{x}_ t, t) \big] \frac{\partial \boldsymbol{x}_ t}{ \partial \theta} \mathrm{d}t.$

We hope fixing this typo will make our presentation more clear and resolve your concerns.

Dear reviewer:

Thank you for your reviews! The author-reviewer rebuttal period is coming to a close soon. We sincerely hope that our responses have adequately addressed the concerns you have raised.

In this work, we introduce a new class of multi-step generative models, named Data-prediction Denoising Models, which show significantly stronger performance than diffusion models under the few-step setting. We also explore the weak data distribution recovery ability of the diffusion model's denoisers and practical solutions for enhancing them. We believe that our findings and solutions may benefit future researchers in either improving diffusion models or proposing new generative models.

We hope our rebuttal has resolved your questions. If you still have any unresolved concerns or additional questions, please do let us know! We would be very glad to provide more clarification and address any remaining issues.

Best wishes,

Authors of the submission #5134.