SADA: Stability-guided Adaptive Diffusion Acceleration

We sincerely thank the reviewer for the insightful and constructive comments.

Q1: Can its speedup be extended to $2 \times$ or beyond?

A1: Yes, it can. To apply an faster configuration, we leverage the inherent stability of the per‐step data reconstruction $x_0^t$. When the $x_0^t$ trajectory demonstrates high stability (e.g., the second half of Fig.2), we could employ larger step sizes compensated by higher-order approximations. Building on this insight, we implement a uniform step-wise pruning strategy after the stability of the denoising process with Lagrange interpolation for correction.

For example, consider a 50‐step process. To achieve a step-wise pruning interval of 4 after stable (i.e., compute every 4th step fully and interpolate the skipped steps via Lagrange), we store $\hat{x}_0^t$ every 4 steps before stabilization. Their indices define the fixed-size set $I$, which is updated dynamically to limit memory usage. For any skipped $t$:

Under this setting, we yield a $\geq 1.8 \times$ speedup regardless of models or solvers. The acceleration would be even more aggressive if further increasing the step size. To balance the degradation, we raise the Adams-Moulton approximation from second to third order, allowing $x_0^t$ to leverage information from the previous three steps (instead of two), thereby improving numerical accuracy and robustness. Our updated result in Table 1 demonstrates the effectiveness of the above improvements. Notably, we achieve a $2.02 \times$ speedup on the most powerful Flux.1 model with impressive $0.06$ LPIPS and $1.95$ FID.

Table 1: Quantitative results on MS-COCO 2017

Q2: Ablations on DPM-Solver++

We appreciate the reviewers for highlighting the practical importance of DPM++ and its ability to achieve high quality with fewer steps.

a. SADA performs worse than AdaptiveDiffusion?

To counterbalance the aggressive configuration, we increased the order of the Adams-Moulton approximation from second to third order. This enhancement incorporates additional information from the previous denoising trajectory, which in turn improves both accuracy and stability. As shown in Table 1, SADA now significantly outperforms AdaptiveDiffusion when used with DPM++.

b. Set the sampling step to 20?

Table 2 presents a comprehensive ablation study across various sampling steps. Our method achieves a 1.5× acceleration in the 25-step scenario with negletable difference. Furthermore, we observe that as the number of inference steps increases, the images generated by DPM++ initially change dramatically before converging when the base step is set to 25. An illustration, along with additional generation examples and comparisons, is available at the following link:

https://drive.google.com/file/d/168ovZu9fxcfY5PfE8F5AgkN4la6dvH9f/view?usp=sharing

Table 2: Ablation studyon sampling steps

Q3. SADA for Flow-matching & DiT Architecture

Under the flow matching objective, the model directly predicts the transportation vector field $dx/dt$ between noise and data distributions. Since the denoising trajectory is ODE-based, our criterion effectively measures its stability. Table 3 on Flux (DiT) shows that our method significantly outperforms the most recent work suggested by reviewer DBkM.

Table 3: Quantitative results on MS-COCO 2017

We sincerely appreciate the reviewer's thoughtful feedback and kind support for our work.

Based on the suggestions from Reviewer 4E6r and DBkM, we implement an aggressive version of SADA:

1. Implementing uniform step-wise pruning when the $x_0^t$ trajectory is stable, using Lagrange interpolation.

2. Mitigating degradation by upgrading the Adam-Moulton Approximation from second to third order.

Detailed motivation and formulation are provided in our response to Reviewer 4E6r, and updated results are shown in Table 1.

Q1: Comparison between $x_0$ and $x_t$-based criterion

A1: We compare our $x_0$ driven paradigm with AdaptiveDiffusion, which leverages the third-order difference of $x_t$ as acceleration criterion. As shown in Table 1, our method consistently delivers superior generation quality—achieving higher PSNR, lower LPIPS and FID—while maintaining a stable speed-up ratio of $\geq 1.8 \times$ regardless of model and scheduler.

The $x_0$ representation is naturally aligned with the final output, capturing essential semantic structures and enabling a more robust criterion than $x_t$.

Table 1: Comparison between $x_0$ based and $x_t$ based criterion

Q2 Theoretical claims

A2-1 (CLT): The independence assumption holds for $x_t$ as $\epsilon_t$ is sampled i.i.d. from Gaussian. For $\hat{x}_t=\sqrt{\bar{\alpha}_t}\hat{x}_0^t+\sqrt{1-\bar{\alpha}_t}\hat{\epsilon}_t$, we write $\hat{\epsilon}_t=\epsilon_t+(\hat{\epsilon}_t-\epsilon_t)$. The first term is i.i.d. Gaussian with zero mean by Law of Large Number (LLN). For the second, the training objective $E| \epsilon-\hat{\epsilon}_t|^2$ implies $\hat{\epsilon}_t\to E[\epsilon\mid x_t,t]$, so $E[\hat{\epsilon}_t-\epsilon_t]\to 0$, and by LLN, the sample mean $\overline{\hat{\epsilon}_t-\epsilon_t}\to 0$.

A2-2 (Lipschiz): The Lipschiz continuity of $\epsilon_\theta$ is widely assumed by preliminary works such as Adaptive Diffusion and DPM-Solver.

Q3 Computational overhead

a. Memory: For step-wise pruning with third-order Adam-Moulton, after reformulate we only need to store 1 previous $x_0^t$ and 2 previous $dx/dt$ in the cache. For token-wise pruning, we store 1 previous representation $\mathbf{x}^l_t$ for only transformers with the highest resolution. For example, we observe only a neglectable increase in memory usage in the SD-XL model (from 14981 MB to 15127 MB).

b. Complexity: All computation in the SADA framework is addition and scaling, $O(N)$. Note that SADA does not include any quadratic complexity computation (e.g. cosine similarity, matrix calculation) as in previous works.

Q4. Qualatative examples & Fewer step generation

We provide the following link for more generation samples and comparison with previous strategy. In Addition, the CLIP score for generation quality is provided.

https://drive.google.com/file/d/168ovZu9fxcfY5PfE8F5AgkN4la6dvH9f/view?usp=sharing

We appreciate for pointing out the better similarity when decreasing sampling steps. We believe the extent accumulation of error decrease when reducing sampling steps. This trend could be clearly found in our ablation table.

Q5. Comparison with other token-wise sparisty strategies

Table 2 shows that our method significantly outperforms ToMeSD. DiTFastAttention is limited to traditional Diffusion Transformer because its windowed attention cannot handle mixed-modality inputs (e.g., MM-DiT modules in SD-3 and Flux). In contrast, our approach easily adapts to these architectures, as demonstrated later.

Table 2: Quantitative results on MS-COCO 2017

Q6. SADA for Flow-matching & DiT Architecture

Under the flow matching objective, the model directly predicts the transportation vector field $dx/dt$ between noise and data distributions. Since the denoising trajectory is ODE-based, our criterion effectively measures its stability. Table 3 on Flux (DiT) shows that our method significantly outperforms the most recent work suggested by reviewer DBkM.

Table 3: Quantitative results on MS-COCO 2017

We thank the reviewer for comprehensive comments.

Q1: Aggressive configuration of SADA

A1: To implement an aggressive version, we leverage the inherent robustness of the per‐step data reconstruction $x_0^t$. When the $x_0^t$ trajectory demonstrates high stability (e.g., the second half of Fig.2), we could employ larger step sizes compensated by higher-order approximations. Building on this insight, we implement a uniform step-wise pruning strategy after the stability of the denoising process with Lagrange interpolation for correction.

For example, consider a 50‐step process. To achieve a step-wise pruning interval of 4 after stable (i.e., compute every 4th step fully and interpolate the skipped steps via Lagrange), we store $\hat{x}_0^t$ every 4 steps before stabilization. Their indices define the fixed-size set $I$, updated dynamically to limit memory usage. For any skipped $t$:

Under this setting, we yield a $\geq 1.8\times$ speedup regardless of models or solvers. To balance the degradation, we raise the Adams-Moulton approximation from second to third order, allowing $x_0^t$ to leverage information from the previous three steps (instead of two), thereby improving numerical accuracy and robustness. Our updated result in Table 1 demonstrates the effectiveness of the above improvements.

Table 1: Quantitative results on MS-COCO 2017

Q2: Comparison between $x_0$ and $x_t$-based criterion

A2: To our best knowledge, we are the first work that considers $x_0$ as acceleration criterion and approximation objective. We compare our paradigm with AdaptiveDiffusion, which leverages the third-order difference of $x_t$, as demonstrated in Table 1.

Residing in an image representation space, $x_0$ demonstrates structural alignment with the final output while evolving in a more robust trajectory (as shown in Fig. 2). It captures semantics and thus yields a more consistent sparsity allocation decision.

Q3: Comparisons to recent works

A3: We thank the reviewers for listing the four recent works, and we will cite and discuss them in the camera-ready version. The four works explore different Caching mechanisms within diffusion architectures that accelerate sampling. However, we believe our work fundamentally differs from the four works listed. Note that [1] is published after submission, which is impossible be addressed at the moment.

a. Methodology: The four works mentioned above accelerate diffusion in a fixed configuration (e.g., fixed caching interval and pruning ratio), while SADA is adaptive to different prompts. In addition, to our best knowledge, SADA is the first work that unifies token- and step-wise sparsity by a single criterion from the perspective of the ODE-solver process, achieving a multi-granularity adaptive acceleration strategy. The novelty is strongly supported by Reviewer km7j.

b. Motivation: SADA formulates the acceleration of the ODE-based generative modeling (e.g., Diffusion, Flow-matching) as a stability measure of the denoising trajectory, while the four works focus only on the redundancy of the denoising architecture with relatively weak theoretical justification.

c. Experiment: We believe the objective of post-training acceleration is to preserve the similarity (faithfulness) between original generated and accelerated samples while maximizing speed. Therefore, we evaluate using LPIPS and FID computed between these samples—unlike previous work, which only compares the FID of accelerated samples against the dataset. As shown in Table 2 of our response to Reviewer km7j, our method significantly outperforms [4] on FLUX.1 in terms of faithfulness at the same speed-up ratio.

Q4: CLIP/Pick Score

A4: Our objective is to preserve the original generation quality through our sparsity framework—metrics like these do not reflect that goal. For completeness, we have provided the requested metrics, along with generation samples and comparisons, via the link below:

https://drive.google.com/file/d/168ovZu9fxcfY5PfE8F5AgkN4la6dvH9f/view?usp=sharing

Q5: Computational overhead analysis & validation of Theoretical claim

Please refer to Q2, Q3 in our response to Reviewer km7j.