Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy

Q1: Method Explanation

• Assumption Validity: We follow the assumptions proposed in DPM-solver $^{[1]}$, which are commonly adopted for the high-order approximation in ODE solvers.

• Algorithm Pseudocode: We have provided the pseudocodes of AdaptiveDiffusion and the greed search algorithms in Appendix A.2.3.

Q2: Experimental Discussion

• Diversity of Models: We further explore the application of our method on unconditional image generation tasks. The results are shown in the following table. Specifically, following Deepcache, we perform unconditional image generation on CIFAR10 and LSUN-Bedrooms. Shown in the table below, our method still achieves a larger speedup ratio and higher image quality than Deepcache on both benchmarks.

• Hyperparameter Sensitivity: We have provided the sensitivity analysis of hyperparameters in Table 4.

• Memory Usage: We have listed the memory usage of different models in Table 1, 2, and 3.

Q3: Limitation and Future Work

Currently, our work mainly focuses on the acceleration of diffusion with ODE solvers. In future work, we should explore the effectiveness of our method on more kinds of solvers and models. For example, the acceleration of SDE solvers should consider the interference of randomly-generated noise in the high-order estimation of the skipping strategy. For consistency models, the acceleration should consider the impact of distillation on the trajectory of image generation, which would change the continuity of noise prediction.

Reference:

[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. NeurIPS 2022.

Dear Reviewer,

We respectfully disagree with the comment that our work lacks self-containment. We would like to clarify that Eq. (1) is the general formulation of the ODE solver, which can be referred to Eq. (3.7) of the DPM-solver$^{[1]}$ and Algorithm 2 of the Euler Sampler$^{[2]}$. Since we would like to formulate a unified and general derivation of the ODE solvers, the coefficients of $x_i$ and $\epsilon_\theta$ are symbolized as $f(i)$ and $g(i)$, respectively. According to the formulations in DPM-solver and Euler Sampler, we can easily get the properties of $f(i)$ and $g(i)$ as mentioned in the rebuttal.

We would like to emphasize that Eq. (1) is not a novel contribution of our work but rather a summary of existing formulations of ODE solvers. We have provided the necessary citations in the original manuscript to support this (See Line 105 of the manuscript). If there are any remaining misunderstandings, we are open to further discussion and would greatly appreciate the opportunity to clarify them.

Reference:

[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. NeurIPS 2022.

[2] Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2022.

Q1: Discussion of Single-step Sampling Works

Thank the reviewer for the valuable suggestion. We will provide a detailed discussion in the revised manuscript. Here is a brief discussion of single-step sampling works.

In addition to the acceleration paradigms mentioned in Sec 2.2., a recent training-based acceleration paradigm is increasingly receiving attentions from the community. Different from previous acceleration works reducing sampling steps or model size during inference time, this paradigm adopts the idea of flow-matching optimal transport to directly achieve single-step sampling during training, whose trajectory is approximately a straight line $^{[1,2]}$. Compared with this new trend, our method highlights the training-free acceleration of multi-step sampling models, with no need to train a new efficient diffusion model.

Q2: Effectiveness on SDE solvers

Compared with the ODE solver, the SDE solver includes an additional noise item for the latent update, which is unpredictable by previous randomly-generated noises. When the magnitude of random noise is not ignorable, the third-order derivative of the neighboring latents cannot accurately evaluate the difference between the neighboring noise predictions. Therefore, to apply our method to SDE solvers, we should design an additional indicator that decides whether the randomly-generated noise is minor enough or stably changed to trigger the third-order estimator. In this case, we design an additional third-order estimator for the scaled randomly-generated noise. When the third-order derivatives of both the latent and the scaled randomly-generated noises are under the respective thresholds, the noise prediction can be skipped by reusing the cached model output.

To validate the effectiveness of our improved method, we conduct experiments for SDXL with the SDE-DPM solver on COCO2017. The results are shown in the following table. Compared with Deepcache, our method can achieve higher image quality with a comparable speedup ratio, indicating the effectiveness of AdaptiveDiffusion on SDE solvers.

Reference:

[1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ICLR 2023.

[2] Flow matching for generative modeling. ICLR 2023.

Q1: Analysis of Improvements.

We describe the advantages of our method in two aspects.

• Novel Method Design: Endorsed by three other reviewers, AdaptiveDiffusion is the pioneering framework that accelerates the diffusion process adaptively for diverse prompts. Unlike the SOTA method Deepcache, which caches features of multiple blocks uniformly across all stages and imposes a static caching rule, our approach introduces adaptive acceleration with theoretical underpinnings and memory efficiency via a single cache unit for predicted latent.
• Comprehensive Performance Improvements: As noted by the other reviewers, AdaptiveDiffusion exhibits strong generalization across various models and tasks. Versus SOTA methods like DPM-solver and Deepcache, our method achieves a comparable or higher speedup ratio while maintaining superior image quality in both static image and video generation tasks. As shown in Tab. 2, due to the large sampling step number and limited prompt diversity, the conditional image generation task on ImageNet 256x256 is relatively easy for acceleration, with both methods achieving roughly 6x speedup with negligible quality loss (~0.09 LPIPS). Notably, for specific categories, e.g., 607th and 854th, etc., AdaptiveDiffusion reduces the NFEs to approximately 35 from 250, yielding ~7x speedup. When the generation diversity and complexity increases, e.g., video generation, the superiority of AdpativeDiffusion is clearly demonstrated in Tab. 3.

Q2: Theorem 1 with A Large Step Size.

We would like to clarify that the step size is not mentioned as a condition for Theorem 1. If understood correctly, the large step size mentioned by the reviewer may refer to the large skip step of noise prediction. In this case, as the one- and two-step skipping schemes have been explored in Appendix A.2.1 and A.2.2 respectively, we further derive the error estimation of an arbitrary skipping scheme.

Taking $i$-th step to perform $k$-step ($k\geq 2$) skipping of noise prediction, we obtain the following update formulations.

$\quad x_i=f(i)\ x_{i+1}-g(i)\ \epsilon_\theta(x_{i+1},t_{i+1})$;

$x_{i-1}=f(i-1)\ x_i-g(i-1)\ \epsilon_\theta(x_{i+1},t_{i+1})$;

$x_{i-2}=f(i-2)\ x_{i-1}-g(i-2)\ \epsilon_\theta(x_{i+1},t_{i+1})$;

$\vdots$

$x_{i-k}=f(i-k)\ x_{i-k+1}-g(i-k)\ \epsilon_\theta(x_{i+1},t_{i+1})$.

$\Rightarrow\varepsilon_{i-k}=\|x_{i-k}-x_{i-k}^{ori}\|$

$\quad\quad\quad \ =\| f(i-k)(x_{i-k+1}-x_{i-k+1}^{ori})-g(i-k)[\epsilon_\theta(x_{i+1},t_{i+1})-\epsilon_\theta(x_{i-k+1},t_{i-k+1})]\|$

$\quad\quad\quad \ \leqslant f(i-k)\varepsilon_{i-k+1}+g(i-k)\|\epsilon_\theta(x_{i+1},t_{i+1}) -\epsilon_\theta(x_{i-k+1},t_{i-k+1})\|$

$\quad\quad\quad \ \leqslant\sum_{m=1}^{k-1}{\|h^{k-m+1}(i-m)\cdot\mathcal{O}(t_{i-m+1}-t_{i-m+2})\|}+\sum_{m=1}^{k-1}{\|h^{k-m+1}(i-m)\cdot\mathcal{O}(x_{i-m+1}-x_{i-m+2})\|}$

$\quad\quad\quad\quad\ +\|g(i-k)\cdot\mathcal{O}(x_{i-k+1}-x_{i-k+2})\|+\|g(i-k)\cdot\mathcal{O} (t_{i-k+1}-t_{i-k+2})\|$

$\quad\quad\quad \ =\sum_{m=1}^k{\mathcal{O}(t_{i-m+1}-t_{i-m+2})+\mathcal{O}(x_{i-m+1}-x_{i-m+2})}$.

The derivation utilizes the property that $\|\epsilon_\theta(x_i,t_{i+1})-\epsilon_\theta(x_{i+1},t_{i+1})\|$ and $\|\epsilon_\theta(x_i,t_i)-\epsilon_\theta(x_i,t_{i+1})\|$ are upper-bounded by $\mathcal{O}(x_{i}-x_{i+1})$ and $\mathcal{O}(t_i-t_{i+1})$ respectively according to the Lipschitz continuity. Here, $h^{k-m+1}(i-m)\coloneqq g(i-m)\prod\nolimits_{j=1}^{k-m}{f(i-m-j)}$.

It is observed that the error of $k$-step skipping is upper-bounded by the accumulation of previous latent differences. Thus, if the skipping step of noise prediction is large, the upper bound of the error will naturally increase, as also empirically demonstrated by Fig. 5(b).

Q3: Novelty of AdaptiveDiffusion.

We first clarify our acceleration mechanism. Then, we will clarify our method's novelty from two aspects.

1. Mechanism:

Generally, the diffusion process comprises two stages at each step: noise prediction and latent update. Given the preset number of inference steps, the main purpose of AdaptiveDiffusion is to reduce the number of function evaluations (NFEs) while keeping the latent update number (sampling steps) unchanged. By reducing NFEs, we can significantly accelerate the diffusion process.

2. Novelty:

• Motivation-level Novelty: Prior studies like EDM $^{[1]}$ and DPM-solver leveraged high-order approximations between sampling steps to enhance image quality. In contrast, AdaptiveDiffusion employs these approximations to adaptively reduce NFEs. That is, while earlier high-order methods aimed at refining generation quality given a fixed sampling step number, our work prioritizes adaptive efficiency without compromising image quality.
• Algorithm-level Novelty: Our 3rd-order estimator is distinct in its adaptive acceleration tailored to various prompts. Unlike other high-order methods that flexibly choose solver orders for subsequent high-quality generation, our estimator is both empirically and theoretically constrained to 3rd-order approximations to decide whether to skip noise prediction, as shown in Sec. 3.3 and the global response. To our knowledge, this insight is the first in the field of diffusion model acceleration.

Briefly, AdaptiveDiffusion targets a different motivation and is a novel approach to the acceleration community.

Q4: Experiments on Pure Image Generation

Following Deepcache, we perform image generation using DDPMs on CIFAR10 and LSUN-Bedrooms and build our method upon 100-step DDIM for DDPM. As shown in the table below, our method still achieves a larger speedup ratio and higher image quality than Deepcache on both benchmarks.

Reference:

[1] Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2022.

Dear Reviewer XP4x:

Thank you for your precious time on the review.

As the deadline for the discussion period is approaching (Author-reviewer discussion will be closed on Aug 13 11:59pm AoE), we sincerely hope that our response can address your concerns and we are looking forward to further discussion on any other issues regarding our manuscript.

Best regards,

Authors of Paper 230

Dear Reviewer,

Thank you for your valuable time to review our work and for your recognition! Your valuable and positive feedback is greatly appreciated. We noticed that the score has not been updated. There seems to be no final rating box this time, so the score may have to be adjusted in the original rating box. We would greatly appreciate it if you could make adjustments.

Best regards,

The Authors

Q1: Theoretical Analysis of the Relationship between the Third-order Estimator and the Skipping Strategy.

To explore the theoretical relationship between the third-order estimator and the skipping strategy, we need to formulate the difference between the neighboring noise predictions. According to Eq.(1), we can get the following first-order differential equations regarding the latent $x$.

$\quad \varDelta x_i=x_i-x_{i+1}=[1-f(i)]x_{i+1}-g(i)\cdot\epsilon_{\theta}(x_{i+1},t_{i+1})$;

$\varDelta x_{i-1}= x_{i-1}-x_i=[1-f(i-1)]x_i-g(i-1)\cdot\epsilon_{\theta}(x_i,t_i)$.

Now, let $u(i)\coloneqq 1-f(i-1)$, and we further derive the second-order differential equations based on the above equations.

$\varDelta x_{i-1}-\varDelta x_i = u( i ) x_i-u( i+1 ) x_{i+1}+g( i ) \cdot \epsilon_{\theta}( x_{i+1}, t_{i+1} ) -g( i-1 ) \cdot \epsilon_{\theta}( x_i, t_i )$

$\quad\quad\quad\quad\quad\ \ \ =u( i ) ( x_i-x_{i-1} ) +u( i ) x_{i-1}-u( i+1 ) ( x_{i+1}-x_i ) -u( i+1 ) x_i+g( i ) \cdot \epsilon_{\theta}( x_{i+1}, t_{i+1} ) -g( i-1 ) \cdot \epsilon_{\theta}( x_i, t_i )$

$\quad\quad\quad\quad\quad\ \ \ =u( i ) \varDelta x_{i-1}-u( i+1 ) \varDelta x_i+\varDelta [ u( i ) x_{i-1} ] +g( i ) \cdot \epsilon_{\theta}( x_{i+1}, t_{i+1} ) -g( i-1 ) \cdot \epsilon_{\theta}( x_i, t_i )$

$\quad\quad\quad\quad\quad\ \ \ =u( i ) \varDelta x_{i-1}-u( i+1 ) \varDelta x_i+\varDelta [ u( i ) x_{i-1} ] +g( i ) [ \epsilon_{\theta}( x_{i+1}, t_{i+1} ) -\epsilon_{\theta}( x_i, t_i ) ] +[ g( i ) -g( i-1 ) ] \epsilon_{\theta}( x_i, t_i )$

$\quad\quad\quad\quad\quad\ \ \ =u( i ) \varDelta x_{i-1}-u( i+1 ) \varDelta x_i+\varDelta [ u( i ) x_{i-1} ] -g( i ) \varDelta \epsilon_{\theta}^{i}-\varDelta g( i ) \cdot \epsilon_{\theta}( x_i, t_i )$.

After simplification of the above equation, we can get the following formulation:

$f( i-1 ) \varDelta x_{i-1}-f( i ) \varDelta x_i=\varDelta [ u( i ) x_{i-1} ] -g( i ) \varDelta \epsilon_{\theta}^{i}-\varDelta g( i ) \cdot \epsilon_{\theta}( x_i, t_i )$.

From the above equation, we can observe that the difference between noise predictions $\varDelta \epsilon_{\theta}^{i}$ is related to the first- and second-order derivatives of $x_i$, as well as the noise prediction $\epsilon_{\theta}( x_i, t_i )$. Therefore, it would be difficult to estimate the difference without $\epsilon_{\theta}( x_i, t_i )$. Now we consider the third-order differential equation. From the above equation, we further obtain the following formulation.

$f( i ) \varDelta x_i-f( i+1 ) \varDelta x_{i+1}=\varDelta [ u( i+1 ) x_i ] -g( i+1 ) \varDelta \epsilon_{\theta}^{i+1}-\varDelta g( i+1 ) \cdot \epsilon_{\theta}( x_{i+1}, t_{i+1} )$.

$\Rightarrow \varDelta [ f( i-1 ) \varDelta x_{i-1} ] -\varDelta [ f( i ) \varDelta x_i ] =\varDelta ^{( 2 )}[ u( i ) x_{i-1} ] -\varDelta [ g( i ) \varDelta \epsilon_{\theta}^{i} ] -\varDelta [ \varDelta g( i ) \cdot \epsilon_{\theta}( x_i, t_i ) ]$.

$\Rightarrow \varDelta [ \varDelta g( i ) \cdot \epsilon_{\theta}( x_i, t_i ) ] =-\varDelta ^{( 2 )}[ f( i-1 ) \varDelta x_{i-1} ] +\varDelta ^{( 2 )}[ u( i ) x_{i-1} ] -\varDelta [ g( i ) \varDelta \epsilon_{\theta}^{i} ]$.

From the above equation, it can be observed that the difference of the neighboring noise predictions is explicitly related to the third- and second-order derivatives of $x_i$, as well as the second-order derivative of $\epsilon_\theta^{i}$. Since $\lim_{i\rightarrow 0} f( i ) =1,\lim_{i\rightarrow 0} u( i ) =0,\lim_{i\rightarrow 0} g( i ) =0$, we can finally get the conclusion that $\varDelta \epsilon_{\theta}^{i} \| \_{i \rightarrow 0} \approx \mathcal{O} ( \varDelta ^{( 3 )}x_{i-1} )$.