Adaptive Stochastic Coefficients for Accelerating Diffusion Sampling

We sincerely thank the reviewer for their positive assessment and valuable, constructive feedback. Your suggestions have provided a clear path for us to substantially improve our manuscript.

$\textcolor{red}{\text{Weakness1:}}$ On Comparison with Other Solvers

$\textcolor{blue}{\text{Response} :}$ Thank you for the valuable suggestion to include more baselines. In response to your feedback, we have added comparisons against DC-Solver [1] and DPM-Solver-v3 [2], based on the results reported in their respective papers. The preliminary results below show that our method remains highly competitive.

We wish to clarify that a direct comparison on the LSUN-Bedroom dataset was not feasible, as the base models for DC-Solver [1] and DPM-Solver-v3 [2] in their original publications differ from the one used in our study. To address this, we will incorporate comparisons with these solvers on relevant datasets in a future version of our work. Under standard experimental settings for the LSUN-Bedroom dataset, our method significantly outperforms current solvers. For example, at just 3 NFE, AdaSDE achieves a 69.1% improvement in FID score over the state-of-the-art AMED-Solver.

FID result of FFHQ 64 × 64

FID result of CIFAR10 32 × 32

FID result of MSCOCO 512 × 512 on Stable Diffusion v1.5

$\textcolor{red}{\text{Weakness2:}}$ On Training Efficiency and Comparison to Distillation

$\textcolor{blue}{\text{Response} :}$Thank you for these important questions. We will add a detailed analysis to the appendix covering convergence times and a cost-performance comparison against traditional distillation.

In short, our training time is significantly shorter than existing lightweight-training methods because we optimize only a few parameters instead of an auxiliary network. On CIFAR-10, our method converges in just 3-10 minutes with high stability.

This efficiency stands in stark contrast to traditional distillation methods, which require massive computational resources (e.g., 1000+ A100 hours for Consistency Models [3] and 100+ A100 hours for Consistency Trajectory Models [4] on CIFAR-10). Furthermore, those methods often suffer from training instability and can degrade the base model's performance due to altered parameters, leading to a limited quality ceiling where more inference steps do not yield better results.

$\textcolor{red}{\text{Weakness3:}}$ On Evaluation with Larger-Scale Models

$\textcolor{blue}{\text{Response} :}$Thank you for encouraging us to evaluate on larger-scale models; we agree this is crucial for demonstrating practical impact.

We have conducted new experiments on the Flux.1-dev, comparing our method against DPM-Solver-2. After just 4 hours of training on a single H800 GPU, our solver significantly outperforms the baseline second-order sampler. We will include qualitative visualizations from these experiments in the revised paper and also add results validating our plugin approach on the FLUX model.

Evaluation on MSCOCO 512x512 (Flux.1-dev Model)

Table 1: FID Score (Lower is Better)

Table 2: CLIP Score (Higher is Better)

Thank you once again for your time and thoughtful feedback. We are very encouraged by your positive assessment of our work's potential and believe your suggestions will help us substantially strengthen our manuscript.

[1] Zhao, Wenliang, et al. "Dc-solver: Improving predictor-corrector diffusion sampler via dynamic compensation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[2] Zheng, Kaiwen, et al. "Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics." Advances in Neural Information Processing Systems 36 (2023): 55502-55542.

[3] Song, Y., Dhariwal, P., Chen, M. and Sutskever, I., 2023. Consistency Models. arXiv preprint arXiv:2303.01469.

[4] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.

We are very grateful for your comprehensive review, which provided both an encouraging assessment of our method's strengths and valuable suggestions for improvement. Your valuable suggestions on improving the paper's structure, related work, and focus on large-scale experiments provide a clear path for our revision. Below, we address each of the points you raised.

$\textcolor{red}{\text{Weakness 1:}}$ On Paper Structure and Presentation

$\textcolor{blue}{\text{Response} :}$ Thank you for the suggestion on improving the paper's readability. We agree with your assessment. In our revision, we will significantly restructure the paper by moving a large portion of the theoretical analysis from the main text to the appendix. This will allow us to introduce our core methodology much earlier and expand upon its details, which will substantially enhance the paper's clarity and flow.

$\textcolor{red}{\text{Weakness 2:}}$ On Missing Related Work

$\textcolor{blue}{\text{Response} :}$ Thank you for pointing out these important references; we acknowledge this was an oversight in our work. We have now added the suggested papers [1-5] to our related work section. Furthermore, we have conducted a systematic survey of related methods and have comprehensively updated this section to better position our work within the current manuscript.

Thank you for this valuable feedback. We agree that demonstrating performance on large-scale models is crucial, and we have conducted new experiments on the Flux model to address this.

$\textcolor{red}{\text{Weakness 3:}}$ Performance On Large-Scale Models

$\textcolor{blue}{\text{Response} :}$ Thank you for this valuable feedback. We agree that demonstrating performance on large-scale models is crucial, and we have conducted new experiments on the Flux model to address this.

Before we discuss the performance of our sampler in the ultra-low NFE regime (e.g., 5 steps), it is important to first explain our NFE counting convention. In line with a stricter accounting practice, we count each conditional and unconditional forward pass for CFG separately. Consequently, one complete integration step of our method equates to 4 NFE. This differs from some other conventions and means that at 8 NFE, our method performs 2 full steps. To ensure a fair comparison in our experiments on the Flux model, we benchmark our method against DPM-Solver-2, which is also a second-order solver, reporting results at corresponding sampling steps (1 step = 4 NFE).

Evaluation on MSCOCO 512x512 (Flux.1-dev Model)

Table 1: FID Score (Lower is Better)

Table 2: CLIP Score (Higher is Better)

Thank you once again for your time and for providing such constructive feedback. We hope our responses have clarified our position and would welcome the opportunity for any further discussion concerning the paper or related research areas.

[1] Denoising Diffusion Implicit Models, Song et al.

[2] Efficient Integrators for Diffusion Generative Models, Pandey et al.

[3] Bespoke Solvers for Generative Flow Models, Shaul et al.

[4] SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models, Xue et al.

[5] SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models.

We sincerely thank the reviewer for their positive assessment and valuable, constructive feedback. We are encouraged that you see the value in our method's universality and have provided excellent suggestions to strengthen our paper. We address your points below.

$\textcolor{red}{\text{Suggestion 1:}}$ On Comparison to Distillation and Large-Scale Model Evaluation

$\textcolor{blue}{\text{Response} :}$ Thank you for the suggestion to compare our method against distillation on a large-scale model. We agree that this is a crucial test for real-world viability and share your view that distilled models often "struggle to perform well in the wild."

We acknowledge that Flux is a widely-used and powerful generative model, and its distilled version, Flux-Schnell, serves as a popular baseline. However, we believe a direct comparison between distillation model and our lightweight method is unfair, especially when considering adaptability. When faced with a novel model architecture (e.g. unify model), distillation requires a costly redesign and massive compute resources (1000+ H100 hours), whereas our model-agnostic method is significantly faster and cheaper to adapt because it focuses on sampling dynamics rather than internal architecture. Under standard experimental settings, our method significantly outperforms the baseline AMED-Solver.

• Advantages over Distillation: We believe the phenomenon that distilled models "struggle to perform well in the wild" is due to a fundamental drawback of the distillation process. By altering the model's original parameters, distillation can reduce generative diversity and impose a quality ceiling, meaning more inference steps often do not lead to better results. Furthermore, the training cost is enormous (e.g., 1000+ A100 hours for Consistency Models [1] and 100+ A100 hours for Consistency Trajectory Models [2] on CIFAR-10). Our method avoids these issues by preserving the original model and requiring minimal training (3-10 minutes on CIFAR-10). We see them as distinct pipelines, where our method offers a highly efficient and effective alternative.
• New Experiments on Flux.1-dev: Following your recommendation, we have conducted new experiments on the Flux.1-dev model. After just 3 hours of training on a single H800 GPU, our solver significantly outperforms the baseline second-order sampler, DPM-Solver-2, demonstrating our method's effectiveness and efficiency.

Evaluation on MSCOCO 512x512 (Flux.1-dev Model)

Table 1: FID Score (Lower is Better)

Table 2: CLIP Score (Higher is Better)

$\textcolor{red}{\text{Suggestion 2:}}$ On Specific Experimental Questions

$\textcolor{blue}{\text{Response} :}$ We are happy to answer your specific questions regarding the experiments:

1. Regarding Performance at 8 NFE: Thank you for this observation. The performance difference at 8 NFE is due to a discrepancy in the number of integration steps between our second-order method and multi-step solvers like DPM-Solver++(2M). At 8 NFE, our method completes 2 full steps, whereas DPM-Solver++ completes 4 steps. This creates a performance bottleneck for our method at this extremely low step count.

However, we have conducted preliminary visualization tests with our plugin version, and the qualitative results show that it significantly outperforms DPM-Solver++ even at this low NFE.

Moreover, our method has a much lower truncation error per step. This allows it to quickly overcome the initial bottleneck and surpass the baselines at 12+ NFE. More importantly, as shown in Appendix Figure 6, traditional solvers can still produce poor or unstable results even at 20 NFE, due to the accumulation of truncation error from their early steps. Traditional solvers often require 50+ NFE on Stable Diffusion 1.5 to achieve good generation quality. In contrast, our method achieves high-quality results at just 12 NFE.

2. Regarding Teacher Model Performance: Thank you for this recommendation. We confirm that in all our experiments, our student method (AdaSDE) significantly outperforms its teacher (DPM-Solver-2) when compared at the same NFE. To make this clear, we will add the teacher's results to every table in the final version of the paper.

3. Regarding NFE for SD1.5 vs. ImageNet: To clarify this point, we must first explain our NFE counting rule. While it is common to count the parallel cond and uncond passes in CFG as 1 NFE, we count them separately as 2 NFE. This is the primary reason for the apparent difference: for a given NFE, the SD 1.5 model (with CFG) completes only half the number of sampling steps as the ImageNet EDM model (conditional, no CFG).

More importantly, as we analyze in our paper, the total sampling error is a combination of sampler error (from discretization) and intrinsic model error (the discrepancy between the score network $s_\theta$ and gradient of log probability $\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t | \mathbf{x}_0)$). The pre-trained ImageNet model targets a simpler, "toy" distribution with low intrinsic model error. In contrast, SD 1.5 is a far more complex model for real-world scenes and thus has much higher intrinsic model error.

Our method's adaptive stochasticity is particularly effective at mitigating this high intrinsic model error. This is why AdaSDE achieves strong results on SD 1.5 at only ~12 NFE, a task where traditional solvers (e.g. DPM-Solver++) often require 50+ NFE to produce stable, high-quality images.

Thank you again for your time and insightful feedback, which will be invaluable in improving our manuscript. We hope our responses have clarified our position and would welcome the opportunity for any further discussion concerning the paper or related research areas.

[1] Song, Y., Dhariwal, P., Chen, M. and Sutskever, I., 2023. Consistency Models. arXiv preprint arXiv:2303.01469.

[2] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.

We sincerely thank you for your thorough evaluation and encouraging assessment of our work. We are particularly grateful that you highlighted our paper's clarity, the importance of the problem we address, and the robustness of our method's performance. We address your points below.

$\textcolor{red}{\text{Weakness 1: }}$ The paper could benefit from a more comprehensive review of existing methods, particularly alternative stochastic approaches and hybrid solvers.

$\textcolor{blue}{\text{Response} :}$ Thank you for this valuable suggestion. Following your feedback, we have conducted an extensive new survey and have updated our related work section to include a discussion of alternative stochastic and hybrid solvers, including the methods in [1-7].

$\textcolor{red}{\text{Weakness 2: }}$ On the Clarity and Intuition of the Method

$\textcolor{blue}{\text{Response} :}$ Thank you for this valuable feedback. We agree that making our method more accessible and intuitive for practitioners is very important. The theoretical exposition in the main text is not sufficiently streamlined for clarity. Intuitively, while SDE solvers correct for this using noise injection, our method, AdaSDE, enhances this principle by adaptively controlling the intensity of the noise at each step to improve generation quality. In our revised manuscript, we will restructure the presentation by moving some of the dense theoretical derivations to the appendix. We will use this space to add more intuitive explanations and visualizations of the adaptive noise injection mechanism to better illustrate how it works in practice.

$\textcolor{red}{\text{Questions 1 and Weakness 3:}}$ On Robustness and Broader Applicability

$\textcolor{blue}{\text{Response} :}$ Thank you for the question. In specialized domains like medical imaging, large-scale, standardized benchmarks are often unavailable due to the significant challenges in data collection. For this reason, general-purpose datasets like MS-COCO are widely regarded as the standard benchmark for text-to-image tasks, serving as the closest and most reliable proxy for diverse, real-world generation scenarios in academia. On this benchmark, as shown in Table 2 of our paper, AdaSDE consistently outperforms existing advanced solvers when paired with Stable Diffusion v1.5.

Following your recommendation, we conducted an experiment between our method and DPM-Solver-2 on the Flux.1-dev model.

Evaluation on MSCOCO 512x512 (Flux.1-dev)

Table 1: FID Score (Lower is Better)

Table 2: CLIP Score (Higher is Better)

$\textcolor{red}{\text{Questions 2 :}}$ On Computational Overhead and Training Efficiency

$\textcolor{blue}{\text{Response} :}$ In short, a key advantage of our method over other light-weight approaches like AMED-Solver lies in our optimization strategy. While they often require training an auxiliary network—a process that is both slower and prone to instability—we optimize only a small set of 8-40 parameters.

This parameter-based tuning is fundamentally more stable and significantly faster, requiring just 3-10 minutes on a single RTX 4090 for CIFAR-10. In contrast, Distillation-based methods demand massive computational resources.(e.g. 1000+ A100 hours for Consistency Models [8] and 100+ A100 hours for Consistency Trajectory Models [9] on CIFAR-10. Crucially, this tuning is a one-time cost for a reusable solver. At inference, it adds zero computational overhead—for the same NFE, our sampling time is identical to standard solvers, while our output quality is vastly superior. We thank you for raising these important points about the practical implementation of our algorithm. Following your suggestion, we will expand our paper to include a detailed comparison of our method's time consumption, convergence speed, and computational resource requirements against other lightweight-training approaches and training-bsaed distillation methods.

Thank you again for your time and valuable feedback. We hope this addresses your main points and welcome any further discussion on our work or related topics in this area.

[1] Dc-solver: Improving predictor-corrector diffusion sampler via dynamic compensation, Zhao et al.

[2] Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics, Zheng et al.

[3] Efficient Integrators for Diffusion Generative Models, Pandey et al.

[4] Bespoke Solvers for Generative Flow Models, Shaul et al.

[5] SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models, Xue et al.

[6] SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models. Gonzalez et al.

[7] Bespoke non-stationary solvers for fast sampling of diffusion and flow models. Shaul et al.

[8] Song, Y., Dhariwal, P., Chen, M. and Sutskever, I., 2023. Consistency Models. arXiv preprint arXiv:2303.01469.

[9] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.