Pixel-Aware Accelerated Reverse Diffusion Modeling

We thank Reviewer for their time and comments. We have carefully addressed all questions and concerns below. We will happily provide further clarification if needed.

Q1 Lack of Recent GDM Studies:

Reply: We believe there is some misunderstanding on the part of Reviewer about our contributions. As noted in our response to other reviewers; we maintain that our approach fundamental changes in the forward diffusion stage which hasn't been addressed by earlier works. Earlier models have only sought to reduce the reverse trajectory by employing subsampling or fast ODE solver based strategies. We provided an overview of the closely related topics to our work, which we believe will subsequently benefit these backward diffusion solvers to make them more efficient. That being said, we also included a comparison to DPM solver and showed that our processing time in backward diffusion alone is comparable to algorithms including DMP-Solver. Considering that our forward diffusion trajectory has been shortened, our reverse trajectory is also shortened, and its parallelization by our introduction of of the image scale $x_{\delta}$ provides significant run-time reduction.

Q2 The term "dual" is used, which typically refers to the dual form of a convex optimization problem, making it an inappropriate choice here.

Reply: Point well taken. The ‘dual’ word has various contexts and is used here to refer to dual function space. We have corrected the typo the review pointed out in the revision and added explanation. We respectfully request the reviewer to further clarify their comment on: Significant figures in Tables 1 and 2 are inconsistent for us to address the issue.

Q3 It is unclear why the SNR is defined as Equation (11). It does not align with the general definition of SNR.

Reply: Noise is added in each step of the forward diffusion process (along with attenuation of the original information content), the SNR does decrease with increasing time. In other words, $SNR(t_2) < SNR(t_1), \forall t_2 > t_1$. This is true for any diffusion model, including ours, and we use the standard notion of SNR used in Information Sciences.

Q4 In Figure 2, why are the starting points at t=0 different? I believe that they should start with the same value.

Reply: The starting points are different, because conventional DDPM based models start with clean image, $x_0$ with pixel values normalized in the range [-1,1]. Our model on the other hand, requires the pixel values of $x_0$ to be positive to ensure that all the elements of $\alpha_i$’s are less than 1 (Please refer Eqns-7 and 8). In any case, equality of the starting point of the forward diffusion process is not critical. The aim is to completely remove any useful information at the end of the process (signified by 0 SNR in the end). As is evident in Fig.-2, both conventional as well as our design do achieve 0 SNR (implied by 0 mean and non-zero unit variance) in the end by following different trajectories. In our case, the trajectory is accelerated to achieve 0 SNR (signified by a steeper slope).

Thank you once again for the rebuttal and for addressing my concerns.

I have reviewed your response. Regarding Q2, Q3, and Q4, thank you for the clarification and for updating the manuscript.

For Q2, I realize there was a misunderstanding on my part regarding the comment on the 'significant figures in Tables 1 and 2 being inconsistent.' Please disregard this point.

However, I still think that the following point has not been adequately addressed. Regarding Q1, there is insufficient consideration of existing related research [1, 2]. These studies discuss not only adjustments in backward diffusion, such as with DPM solvers, but also approaches to handling noise scheduling in forward diffusion.

[1] Chen, Ting. "On the importance of noise scheduling for diffusion models." arXiv preprint arXiv:2301.10972 (2023).

[2] Chen, Yuzhu, et al. "Adaptive Time-Stepping Schedules for Diffusion Models." The 40th Conference on Uncertainty in Artificial Intelligence.

Thank you for agreeing with our clarification. The noise schedules presented in [1] and [2] remain constant and uniform across all the pixels. The variance changes occur across time-steps. We note and would like to emphasize that our work has been focused on applying unequal diffusion rates across different pixels as well. i.e., the spatial (across data) non-uniform diffusion impacts the dynamics of the underlying diffusion and hence of the associated dynamical equations. This is a novel approach that has not been proposed, let alone explored, in earlier papers. The noise schedules of [1] and [2], are distinct from our approach as they are discussing modulating diffusion rates across time, not pixels. We have included the references on account of their interesting discussion on scheduling across time-steps.

We thank Reviewer for their time and very valuable comments, as well as thankfully agree that the proposed approach is interesting and worthy of further study. We have carefully addressed all questions and concerns below. We will happily provide further clarification if needed.

Q1 I am not sure how Water Pouring Algorithm (WPA) benefits visual media such as image generation.

Reply: The aim of any forward diffusion process is a destruction of information in an image by progressive “ablation” through the SNR reduction of each pixel to 0 (i.e., incremental addition of noise). Pixels with higher starting value should require more noise at each iterative step to globally reach 0 SNR together with lower value pixels. Unequally valued pixels undergo unequal noise addition. This is akin to WPA, except with an inverted objective. In WPA, we add higher amplification (i.e. signal power) to a channel with more noise to maximize total SNR. In our case, we add more noise to a channel (i.e. pixel) with higher value to minimize the SNR. Conventional diffusion models simply add the same noise power to each pixel. This lengthens the process with increasingly long diffusion steps for reaching 0 SNR at convergence. Our design achieves the convergence state at a faster rate than conventional models.

Q2 How the new forward and backward processes look like?

Reply: We will add images of diffusion steps to show the progression our forward and reverse processes.

Q3 Typo corrections.

Reply: Points well taken. We have made corrections and will upload an updated submission shortly. Thank you for the suggestions.

Q4 In appendix D line 5, it seems to introduce unnecessary symbols like the L for reverse diffusion noise predictions.

Reply: L is the stylized Z in the LHS of Eqn.-14. We have changed the notation of noise predictions to remove any ambiguity.

Q5 In table 1 and 2, why all settings (e.g., number of parameters and steps) are different? I can see the network architecture is modified in Section 3.3, but it is not clear why it is around 2x larger.

Reply: Our network is larger on account of 1) Our inclusion of the parameters of the scale estimator VAE, when counting the parameters. 2) Simultaneous generation of noise prediction outputs (Eq. 14) for all time steps in comparison to the conventional models’ single-step output noise prediction.

Q6 The comparisons do not show how DDIM works with the same number of sampling steps, is there any reason for this?

Result: Using the DDIM results mentioned in the original DDIM paper, it is understood that the generation quality is proportional to the number of steps in the diffusion process. The same is true with the reduced FID values in DDIM case, per shown results with 10, 100 and 1000 steps. The FID for a 500 step-model would be somewhere between those for 100 and 1000 step-model. Tables-1 and 2 show that the FID of our model is lower than the FID for 1000 steps conventional DDIM based model. Hence, our FID would also be lower than a 500 steps DDIM based conventional model. We have noted this in the third paragraph of our results section in our original submission.

Q7 Why it is worth mentioning the continuous-time version in eq. 10, is it used later?

Reply: Please refer to our reply to Reviewer TBgv (Q2)

Q8 Which part of the architecture is different from VAE and U-Net that are widely used in other diffusion models?

Reply: The fusion of information hidden in $x_{\delta}$ and the parallel prediction of noise outputs for all the steps (per Eqn-14) underlines the differences with a conventional UNET based design. So, for an RGB image, while conventional U-Net models have a 3-channel output, our model has a 3 x 500 channels output for a 500-time steps trajectory. Additionally, just like fusion of time step embedding in conventional models, image structure information in the form of $x_{\delta}$ is also fused inside the U-Net.

Q9 In appendix B, the proof of convergence seems interesting. It discussed that we can carefully choose $\gamma$ to ensure effective pixel-wise diffusion, but is there ablation study to support that?

Reply: In the case of a single pixel, we have an exponential function that we can illustrate using the pixel value. We have also compared the conventional diffusion models with our design by conducting experiments with toy examples, where noise added in the forward direction is completely known beforehand. Specifically, we conducted reverse trajectory generation experiments where the entire forward direction trajectory (i.e., the noise samples Z in Step-5 of the Sampling algorithm-1 (Page-15) ) was known beforehand. In Sampling algorithm-1 the scale estimate \hat{x}$$\delta was replaced with actual scale value $x_{\delta}$ and we checked the MSE between generated reverse direction image, and the real image $x_0$. Appropriate value of $\gamma$ was chosen based on these experiments.

Thank you for taking time to review our paper and provide valuable comments. We have carefully addressed your questions and concerns in our following response. We will be glad to provide further clarification if you have any further concerns.

Q1 Is it possible to extend this method to latent space? The SNR value in latent space may be different from the pixel space, since the latent is trained with KL loss to be Gaussian.

Reply: We agree there are multiple design additions similar to those employed with conventional models, which can be applied to our model to further improve the performance. However, our focus in this submission was more on presenting a novel diffusion process than presenting a full-fledged diffusion process competitor.

As Eqns.-8 and 9, as well as Steps-2 and 3 of Algorithm-1 show, we do require the value of \hat{x}$$\delta for calculating/generating $\alpha_i$’s, and eventually the complete reverse time trajectory. Using a latent space representation of \hat{x}$$\delta (instead of pixel-space value) would require us to change our sampling algorithm. We would also have to add some parameterized learnable transformation of the latent space representation of \hat{x}$$\delta to our present model and modify the training objective accordingly. We welcome this idea of reviewer. But to keep the trajectory generation step easier to analyze, in our initial design, we have constrained all the training over pixel-space.

Q2 Will the optimization difficulty become harder as the resolution getting larger. The uneven pixel noise schedule may bring high uncertainty with higher resolution.

Reply: Indeed, with increased resolution, the model size would increase for all algorithms. However, in our case, with the help of our pixel-aware forward diffusion, the parameter increase factor is not significant. As is evident from the parameters count in Tables 1 and 2, moving from 32 x 32 to 128 x 128 resolution, only resulted in approximately 2x increase in the number of trainable parameters, which is not a significant an increase in comparison to conventional diffusion model. We expect a similar manageable increase in the case of larger resolutions.

We thank the reviewer for their time and valuable comments. We have carefully addressed all questions and concerns in what follows. We will happily provide further clarification for any further concerns.

Q1 Current literature includes other models that achieve high performance with lower sampling speeds. DDIM was the SOTA algorithm 3-4 years ago, and many improvements have been made since then.

Reply: Point well taken. Please note that our work firstly at fundamentally reducing the forward diffusion path. While recent improvements are mostly focusing on speeding up the reverse diffusion, improvements on forward diffusion are limited. The novelty of our algorithm is in the proposed variable strength of the pixelwise diffusion, which effectively achieves faster diffusion. For a better perspective, we feel that it is good to update our comparison to include DPM solver, as stated in our reply to reviewer aRbR--please see our detailed comments on the SOTA algorithms, the table only shows the backward diffusion time. Considering that our forward diffusion trajectory has been shortened, reverse trajectory is also consequently shortened, and its parallelization by our introduction of the image scale $x_{\delta}$ provides significant run-time reduction, while maintaining reasonable low FID. Comparison with DPM Solver applied to conventional DDPM for CelebA dataset:

Q2 It is unclear why the authors use the discrete formalism of DDPM in the background section, then apply their solution in a continuous scenario (Appendix B).

Reply: Keeping with the conventional development of evolution of diffusion models, we have first developed a discrete time model. This paper only discusses the discrete time scenario. The continuous time version is a work in progress. Our invoking of the continuous case, serves to explain our algorithm and helps in clarifying its advantages. Specifically, we use in Appendix B, a continuous time scenario to exploit the well-developed SDE theory to show time evolution of the mean of forward trajectory (Please see Sarkka and Solin (2019) in the references). As a result, we have shown that we can achieve faster convergence in the forward direction in comparison to conventional models. This resulted in reducing the trajectory length in reverse direction as well, which helps in reducing the parameters of our parallelized reverse diffusion model. We will add a bridging sentence to highlight the rationale.

Q3 Where does Equation (8) come from? Is it based on an assumption?

Reply: Eqn-8 is a definition of parameter $\alpha_i$. It is used to design our forward schedule. As stated in our original submission, line 215, we are able to achieve pixel-aware accelerated diffusion based on this forward schedule. We will further clarify in our revision.

Q4 A challenging part of the method is obtaining (\overline{\alpha} = \prod \alpha_t). How is this achieved in Equation (9)? I consider it tricky.

Reply: Point well taken. The derivation of Eqn-9 is similar to the one followed in conventional designs. Please refer to the derivations in the ‘Forward diffusion process’ section in this link: https://lilianweng.github.io/posts/2021-07-11-diffusion-models/

Q1 I would like to thank the authors for their effort regarding expanding the benchmarks with the frameworks that I pointed out.

Q4 I don't think that the derivation is similar to the basic one since the noise becomes dependent on the signal while the base derivation described in lilianweng's blogpost refers to the DDPM paper and in that case the noise is independent. I would like to have some proofs of that

Perhaps, the main problem of this paper is the whole presentation. Even thought the idea is interesting, the presentation has several issues. I still will keep my score 3 (rejected) because the figures are not for an high-quality standard publication. I listed in the previous answer all my concerns about it.

We thank the reviewer for their time and very valuable comments. We have carefully addressed all the questions and concerns below. We will happily provide further clarification if needed.

Q1 On Typos and figure clarity:

Reply: Point well taken. We have fixed them in the revision and the updated file will be up shortly.

Q2 Comparison with SOTA models.

Reply: Our updated table here shows a comparison with DPM Solver applied to conventional DDPM for CelebA dataset. Considering that our forward diffusion trajectory has been shortened, reverse trajectory is also consequently shortened, and its parallelization by our introduction of the image scale $x_{\delta}$ provides significant run-time reduction, while maintaining reasonable low FID. This further supports the reviewer comment, which we agree with,: “the goal of accelerating diffusion is valuable, and this paper makes some progress to that end”. Current models have only focused on reducing the reverse trajectory by employing sub-sampling or fast ODE solver based strategies (also applicable to our model) without trying to reduce the forward trajectory.

CelebA performance

Q3 Does the proposed approach generalize to latent diffusion models?

Reply: Point well taken. The latent space model, with some special care given to the image scale, should be able to swap the conventional diffusion with our new model. We will include this comment in the updated resubmission to clarify our contribution. As noted earlier, our focus is more on a novel diffusion model.

Q4 Is there a systematic approach to picking the values for T and $\gamma$ ?

Reply: Our pixel aware forward diffusion keeps T smaller than that in conventional DM. Larger $\gamma$ results in faster convergence to standard normal distribution in the forward direction and provides image scale without detailed features for reconstruction. Too large a value could break the Markovian characteristic of the forward trajectory, making the task of denoising in reverse direction unnecessarily difficult. Note that, we opted to keep a 1:10 ratio between $\gamma$ and T. While heuristically chosen, they followed a thorough experimental validation over toy examples. We have some discussion on the choice of the parameters in Section 5/pg. 8 of the original submission. Additionally, due to lower information content in lower resolution images, as in CIFAR10 dataset, the forward diffusion tunes in to noise at a much faster rate in comparison to when using higher resolution CelebA images. Additional clarification on the hyperparameter selection will be given in the resubmission.

Q5 I would be curious to see some elaboration on how well the VAE is able to approximate $x_{\delta}$ (as opposed to $x_0$)

Reply: The image scale, $x_{\delta}$ and $\gamma$ are independent of time-step (while $\overline{\alpha}$ ’s are dependent). Our scale approximator VAE learns to sample from the distribution of $x_{\delta}$ to estimate it. We will include an example contrasting a clean image, $x_0$ and $x_{\delta}$ to provide a clearer comparison between the two.

We have also trained an autoencoder sampling from the distribution of $x_0$ with the same number of parameters and architecture as our scale approximator VAE, to show that it is easier to learn to sample from the distribution of $x_{\delta}$ than from the distribution of $x_0$. As will be evident from example outputs from this VAE, its outputs have various unnecessary artefacts. We will update our resubmission with this information.

As $x_{\delta}$ works as an informational prior in our reverse generative model, without unnecessary spurious information thus reducing their complexity. $x_{\delta}$ with only coarse level information about the image structure, is a better-informed prior.