UDPM: Upsampling Diffusion Probabilistic Models

We would like to thank the reviewer for the valuable comments. Our response is detailed below.

Weaknesses:

1. Will be fixed in the revised version. We thank the reviewer for the effort.
2. Although the three datasets we examined UDPM on are diverse, with CIFAR10 and AFHQv2 being multi-class datasets; we additionally trained UDPM on the LSUN horses dataset at 128x128 resolution as can be seen in the attached rebuttal PDF file.
3. The methods shown in the paper are state-of-the-art accelerated diffusion-based generative models. We will be glad to compare our methods to other works we might have missed. There is a recent class of generative models derived from diffusion named consistency models [1], which can generate images with a single denoising step with an FID score of 8.70 on CIFAR10, which is still inferior to UDPM in both generation quality and efficiency.
   Note that we do not compare UDPM to distillation-based approaches and specifically to the distillation version of the consistency models, as distillation is complementary to our technique. Thus, we compare only to direct training. Note also that even if distillation-based approaches are taken into account, they require at least a single denoising step, leading to much longer runtimes ($\sim300\\%$) than UDPM.
4. Following the reviewer's comment, we have the following inception score comparison:
   Denoising student: 8.36
   TDPM: 8.65
   UDPM: 9.01
   Which shows that UDPM outperforms current SOTA efficient diffusion models also in the inception score measure. In the revised version of the paper we will make sure to add this comparison to Table 1.
5. We will make sure to polish the paper in the revised version.

Questions:

1. Indeed in ColdDiffusion they propose multiple approaches for defining the forward diffusion process. However, ColdDiffusion does not address the existence of the reverse diffusion process in their formulation. While in UDPM, Lemma 1 allows explicit access to the reverse process defined in equations (10-12).
2. UDPM needs a network architecture that upsamples its input while being aligned to other DDPM works to allow direct comparison. Therefore we use the popular SongUNet [2] used in many diffusion works while increasing the number of output channels from $3$ to $3\times \gamma^2$, followed by a depth-to-space layer that upsamples the output by rearranging the pixels (https://pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html). This architecture makes minimal changes to the network, assuring minimal additional latency over the baseline. This is reflected by the run time that we measure which indeed shows that.
3. The practical objective function we use is built from three different terms:
   $\ell_1$: Promotes high fidelity and agreement between the diffusion steps.
   $\ell_{per}$: Guides the network to more perceptual pleasing estimations.
   $\ell_{adv}$: Complementary term to the $\ell_{per}$, that makes sure the reconstructed variable statistics matches the true one.
   We found the combination of the three terms above very crucial for getting sharp and detailed generations. For instance, on using only $\ell_1$ when training UDPM on CIFAR10 we got an FID score of $\sim 60$, and when we added the $\ell_{per}$ it reached $\sim 30$. In the attached PDF file we show the effect of each loss term on the generation results. We will make sure this ablation study is presented in the revised version of the paper.
4. For the completeness of the comparison, below we report the average runtimes on CIFAR10 of UDPM with 3 steps, and DDPM with a single denoising step when a similar network is used. Benchmarked on a single NVIDIA RTX A6000 GPU and averaged on 100K image generations for eliminating any unwanted overhead:
   UDPM: 2008.21 FPS
   DDPM: 735.68 FPS
   Speedup = 2.73x
   The runtimes will be added to Table 1 in the revised version.
5. We use the popular SongUNet used in many diffusion works, particularly the implementation of EDM [3] (https://github.com/NVlabs/edm/). The specific hyperparameters used are detailed Table 4 in the supplementary.
6. The number of steps is determined by the smallest noise resolution you want to start with. Then everything is fixed. From our experiments, we found out that beyond the 3 diffusion steps, the diffusion steps did not benefit the generation quality, hence we use 3 diffusion steps. We will make sure to clarify this is the revised version.

[1] Consistency models by Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.
[2] Score-Based Generative Modeling through Stochastic Differential Equations
[3] Elucidating the Design Space of Diffusion-Based Generative Models (EDM)

Given the substantial improvements made in response to your feedback, we kindly request you to reconsider the score you initially assigned to our submission. We believe that the revised version of our paper now better aligns with the high standards of the NeurIPS conference.

We appreciate the reviewer’s valuable points. Our response to the reviewer’s comments are given below.

Weaknesses:

1. The number of steps is determined by the smallest noise resolution you want to start with. Then everything is fixed. From our experiments, we found out that beyond the 3 diffusion steps, the diffusion steps did not benefit the generation quality, hence we use 3 diffusion steps. We will make sure to clarify this is the revised version.

2. Because the size of the diffusion variables changes throughout the diffusion process in UDPM, it is not possible to directly compare it to the denoising steps of DDPM. Therefore, we compare the total computations required by each algorithm; from which we obtain that UDPM uses $\sim30\\%$ of the computations used in a single denoising step with the same network, or equivalently $\sim0.3$ the time of a single diffusion denoising step.
   Yet, for the completeness of the comparison, below we report the average runtimes on CIFAR10, of UDPM with 3 steps and DDPM with a single denoising step when a similar network is used. Benchmarked on a single NVIDIA RTX A6000 GPU and averaged on 100K image generations for eliminating any unwanted overhead:
   UDPM: 2008.21 FPS
   DDPM: 735.68 FPS
   $\Rightarrow$ Speedup = 2.73x
   The runtimes will be added to Table 1 in the revised version.

3. Simple Diffusion: This work investigates the noise scheduling and the network architecture, and their relation to the image's resolution. However, this approach complements our method, since it does not modify the diffusion structure itself and only optimizes the empirical setup.
   LEGO diffusion: This approach proposes a LEGO bricks architecture for improving the training, efficiency, and resolution generalization of diffusion models. This approach is also complementary to UDPM since it can be used alongside our approach for further improvements.
   Matryoshka diffusion: This method proposes to train a diffusion model on different resolutions simultaneously. The effectiveness of this approach shines when the desired generation resolution is relatively high, where it can generate images with resolutions similar to what Stable-Diffusion can produce but with no need for a separate image encoder. This approach however is very different from UDPM, since it does not modify the formulation of the diffusion model itself.
   All the missing references will be added to the revised version and discussed accordingly. We thank the reviewer for the effort.

Given the substantial improvements made in response to your feedback, we kindly request you to reconsider the score you initially assigned to our submission. We believe that the revised version of our paper now better aligns with the high standards of the NeurIPS conference.

We are grateful for the reviewer's insightful comments. Our response is provided below.

Weaknesses:

1. It is well known that diffusion models severely suffer from heavy computations to produce pleasing-looking images due to two aspects: (i) The large number of diffusion steps and (ii) the large dimensions of the latent variables at each step. Compared to GANs, where a single inference step is used, with latent space much smaller than the generated image size; DDPMs have significant drawbacks. As a result, in this paper, we narrow this gap significantly, reducing the computations using the same network by $\sim 65\\%$, while outperforming the fastest SOTA diffusion-based models.
   Additionally, it has been shown in many GAN papers that the latent space of the generative model is interpolatable and interpretable, which was not the case with DDPMs, where even a single step DDPM has latent space with the same dimensions as the input image; making the latent space very redundant. In UDPM however, the whole latent space is much smaller than the generated image, making the latent space smooth for interpolation (as shown in Figures 5, 7, and 8) and interpretable (as shown in Figures 6, 9, 10). For further understanding of the interpretability of the model, we present an ablation study where we generate an image, then fix two of its three noise maps, and perturb/rerandomize the third noise map 128 times to produce 128 different images. We then take these images and analyze them by examining the principal components of their covariance matrix to understand how each diffusion step affects the generated image (check the rebuttal PDF file). As can be seen in Figures 1 and 2 in the attached PDF file, the noisiest latent variable controls the semantics of the image (i.e. class, pose, background, etc.), while the initial and the middle noise level controls the fine details of the generation. As a result, one may get similar behavior to what has been done in StyleGAN simply by modifying the last diffusion variable; as demonstrated in Figure 3.

2. Complexity: The reverse process in UDPM requires super-resolving the latent variable from the previous step, which necessitates the use of a sophisticated loss term, as shown by previous super-resolution literature [1, 2, 3], which we find crucial in our case for obtaining sharp and detailed results.
   Stability: In contrast to pure adversarial loss used in GANs, using it as a regularization term for guiding the network to sharp solutions is in fact very stable. The only part that needs tuning is the weights of each term, which from our experiments remained fixed to the reported values in the paper for all datasets. To demonstrate the effect of each loss term, in Figure 4 in the attached rebuttal PDF, we show how each term contributes to the generation results. This ablation will be added and discussed in the revised version.

3. As mentioned in the limitation section, training diffusion models require heavy computational resources, particularly when increasing the images resolution; therefore, due to our limited resources we leave such exploration to future research. Yet, for completeness, we ran our approach on LSUN-horses dataset with resolution 128x128 and reported the results in the attached rebuttal PDF file.

Questions:

1. Because the size of the diffusion variables changes throughout the diffusion process in UDPM, it is not possible to directly compare it to the denoising steps of DDPM. Therefore, we compare the total computations required by each algorithm; from which we obtain that UDPM uses $\sim30\\%$ of the computations used in a single denoising step with the same network, or equivalently $\sim0.3$ the time of a single diffusion denoising step.

2. UDPM needs a network architecture that upsamples its input while being aligned to other DDPM works to allow direct comparison. Therefore, we use the popular SongUNet [4] used in many diffusion works while increasing the number of output channels from $3$ to $3\times \gamma^2$, followed by a depth-to-space layer that upsamples the output by rearranging the pixels (https://pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html). This architecture makes minimal changes to the network, assuring minimal additional latency over the baseline. This is reflected by the run time that we measure which indeed shows that.

3. UDPM generalizes traditional denoising diffusion schemes, therefore one can omit $\mathcal{H}$ in part of the diffusion steps (e.g. the last ones), such that some diffusion steps become denoising without upsampling. This may enable increasing the diffusion steps to arbitrary choice, similar to traditional diffusion. However, we did not examine such an approach as we focused on efficiency, and we leave it for further research.

[1] Photo-realistic single image super-resolution using a generative adversarial network
[2] Esrgan: Enhanced super-resolution generative adversarial networks
[3] Real-esrgan: Training real-world blind super-resolution with pure synthetic data
[4] Score-Based Generative Modeling through Stochastic Differential Equations

Given the substantial improvements made in response to your feedback, we kindly request you to reconsider the score you initially assigned to our submission. We believe that the revised version of our paper now better aligns with the high standards of the NeurIPS conference.