Thanks for the valuable feedback

Thank you for taking the time to provide your valuable feedback. We apologize sincerely for the typos and sudden jumps in the text. We apologize any inconvenience they may have caused. We will thoroughly review every detail and submit a revised version that meets the ICLR standards.

Q1. Computational complexity compared to other methods.

For sampling When performing sampling over $n$ time steps, our solver caches all pre-sampled predictions, resulting in a memory complexity of $\mathcal{O}(n)$. The model function evaluation also has a complexity of $\mathcal{O}(n)$ ($\mathcal{O}(2 \times n)$ for CFG enabled). It is important to note that the memory required for caching predictions is negligible compared to that used by model weights and activations. Besides classic methods, we have also included a comparison with the latest Flowturbo published on NeurIPS24.

For Searching Solver-based algorithms, limited by their searchable parameter sizes, demonstrate significantly lower performance in few-step settings compared to distillation-based algorithms(5/6steps), making direct comparisons inappropriate. Consequently, we selected algorithms that are both acceleratable on ImageNet and comparable in performance, including popular methods such as DPM-Solver++, UniPC(reported in main paper Tab1 and Tab.2), and classic Adams-like linear multi-step methods. Since our experiments primarily utilize SiT, DiT, and FlowDCN that trained on the ImageNet dataset. We also provide fair comparisons by incorporating the latest acceleration method, FlowTurbo. Additionally, we have included results from the Henu method as reported in FlowTurbo.

We can achieve better or comparable performance with much fewer NFE and parameters compared to FlowTurbo.

Reference

[1]. Zhao, Wenliang, et al. "FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner." arXiv preprint arXiv:2409.18128 (2024)

Presentation issues

Thank you for your insightful feedback. We have taken your suggestions and made significant revisions to the article. To maintain clarity, we eliminated most of the redundant formulas and introduced theorems. The proof of theorems is in the appendix.

As the discussion period ends on November 26, we want to ensure that all your questions have been addressed. Your feedback is invaluable to us, and we would be deeply grateful if you could take a moment to provide a final rating and share your thoughts.

Thanks for valuable feedback

We appreciate the time you took to share your valuable feedback with us. We offer our sincerest apologies for the typos and any inconvenience they may have caused. We will conduct a thorough review of every detail and submit a revised version that meets the highest standards of the ICLR.

Q.1 Why do we need to prove that the error bound is related to timesteps and coefficients?

Our primary objective is to design a compact search space that enables the identification of a solver that achieves near-optimal performance. To accomplish this, we must first establish the constituent components of the search space for the optimal solution. Notably, if the error bound is independent of the number of steps, our search can be limited to the coefficients alone. In fact, it can be proved that the error bound is dictated by the time selection and the coefficients.

Q.2 What is $\eta$ in Section 4.3?

$\eta$ is a constant scalar. We will add more explanation of notations in the finial version.

Assumption.2 As shown below, the pre-trained velocity model $v_\theta$ is not perfect and the error between ${v}_\theta$ and ideal velocity field $\hat{v}$ is bounded, where $\eta$ is a constant scalar.

$||\hat{v}-v_\theta || \leq \eta \ll ||\hat{v}||$

Q.3 Richardson's extrapolation for solving ODE

Yes, the Adams-like linear multi-step method employs Lagrange interpolation to determine its coefficients, which makes it feasible to substitute Lagrange interpolation with alternative interpolation (or extrapolation) techniques[1], such as Richardson's method. Nevertheless, Richardson functions also solely rely on the variable $t$, without considering $x$.

Reference

[1] Fekete, Imre, and Lajos Lóczi. "Linear multistep methods and global Richardson extrapolation." Applied Mathematics Letters 133 (2022): 108267.

Q7.1. Solver Across different variance schedules

Since our solvers are searched on a specific noise scheduler and its corresponding pre-trained models, applying the searched coefficients and timesteps to other noise schedulers yields meaningless results. We have tried applied searched solver on SiT(Rectified flow) and DiT(DDPM with $\beta_{min}=0.1, \beta_{max}=20$) to SD1.5(DDPM with $\beta_{min}=0.085, \beta_{max}=12$), but the results were inconclusive. Notably, despite sharing the DDPM name, DiT and SD1.5 employ distinct $\beta_{min}, \beta_{max}$ values, thereby featuring different noise schedulers. A more in-depth discussion of these experiments can be found in Section(Extend to DDPM/VP).

Q7.2. Solver for different variance schedules

Since every discrete Denoising Diffusion Probabilistic Model (DDPM) has a corresponding continuous Variance Preserving (VP) scheduler, we can transform the discrete DDPM into a continuous VP, thereby successfully finding a better solver compared to traditional DPM-Solvers.

To put it simply, under the empowerment of our high-order solver, the performance of DDPM and Rectified flow does not differ significantly (8, 9, 10 steps), which contradicts the common belief that Rectified flow is stronger at limited sampling steps.

Thanks for the valuable feedback

Thank you for taking the time to provide your valuable feedback. We apologize sincerely for the typos and any inconvenience they may have caused. We will thoroughly review every detail and submit a revised version that meets the ICLR standards.

Q1. More Metrics of Searched Solver

We adhere to the evaluation guidelines provided by DM-nonuniform, reporting only the FID as the standard metric in the current main paper.

To clarify, we do not report selective results; we will provide sFID, IS, PR, and Recall metrics for SiT-XL(R256), FlowDCN-XL/2(R256), and FlowDCN-B/2(R256) in a new revision pdf (cause we can not directly submit figs on openreview). Our solver searched on FlowDCN-B/2, consistently outperforms the handcrafted solvers across FID, sFID, IS, and Recall metrics.

Q2. Computational complexity compared to other methods.

For sampling When performing sampling over $n$ time steps, our solver caches all pre-sampled predictions, resulting in a memory complexity of $\mathcal{O}(n)$. The model function evaluation also has a complexity of $\mathcal{O}(n)$ ($\mathcal{O}(2 \times n)$ for CFG enabled). It is important to note that the memory required for caching predictions is negligible compared to that used by model weights and activations. Besides classic methods, we have also included a comparison with the latest Flowturbo published on NeurIPS24.

For Searching Solver-based algorithms, limited by their searchable parameter sizes, demonstrate significantly lower performance in few-step settings compared to distillation-based algorithms(5/6steps), making direct comparisons inappropriate. Consequently, we selected algorithms that are both acceleratable on ImageNet and comparable in performance, including popular methods such as DPM-Solver++, UniPC(reported in main paper Tab1 and Tab.2), and classic Adams-like linear multi-step methods. Since our experiments primarily utilize SiT, DiT, and FlowDCN that trained on the ImageNet dataset. We also provide fair comparisons by incorporating the latest acceleration method, FlowTurbo. Additionally, we have included results from the Heun method as reported in FlowTurbo.

We can achieve better or comparable performance with much fewer NFE and parameters compared to FlowTurbo.

Reference

[1]. Zhao, Wenliang, et al. "FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner." arXiv preprint arXiv:2409.18128 (2024)

[2]. Xue, Shuchen, et al. "DM-nonuniform: Accelerating Diffusion Sampling with Optimized Time Steps." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q3. Ablation on Search Samples We ablate the number of search samples on the 10-step and 8-step solver settings. Samples means the total training samples the searched solver has seen. Unique Samples means the total distinct samples the searched solver has seen. Our searched solver converges fast and gets saturated near 30000 samples.

Q4. Stopped evaluation at 5 Steps.

Since DM-nonuniform introduced the most effective online optimization solver before our search-based approach, we leveraged their results for comparison on DDPM models. We followed the evaluation pipeline established by DM-nonuniform to report performance within 5 and 10 optimization steps. In general, solver-based methods tend to exhibit inferior results under extremely limited numbers of function evaluations (NFE), such as 5 or 6 steps.

Q5. comparison with distillation methods

We provide a comparison with FlowTurbo in Q.2

Under the given NFE (Number of Function Evaluations) condition, Adams-like linear multistep methods are the strongest manually designed solvers, with performance far superior to Heun and RK4, and relevant test results can be found in Q2. So we used the linear multi-step method as a comparison object.

As the solving difficulty increases and the number of searchable parameters decreases (e.g., only 10 searchable parameters for 4 steps and 6 searchable parameters for 3 steps), the performance of solver-based methods falls significantly behind that of distillation methods when limited to fewer than 5 steps. Notably, it is unlikely for solver-based methods to achieve performance comparable to or exceeding that of distillation methods, such as CM, given that their number of learnable parameters is tens of thousands of times larger than our searchable parameters.

Furthermore, integrating denoiser distillation with solver search holds significant promise for achieving even greater performance enhancements.

Q6. 10-step solver outperforming 50 Euler steps.

Linear multistep-based high-order solvers can significantly boost performance in simulations with a limited number of time steps. By leveraging the reference trajectory from the Euler solver with 100 steps, it is possible to outperform the Euler solver with 50 steps. As illustrated in all metrics, our solver enables SiT-XL/2-R256 and FlowDCN-XL/2-R256 to achieve better Recall scores than the Euler solver with 50 steps. Notably, FlowDCN-XL/2-R512 with our solver surpasses its Euler counterpart in terms of sFID, Precision, and Recall, demonstrating its exceptional performance.

Q4. Stopped evaluation at 5 Steps.

Since DM-nonuniform introduced the most effective online optimization solver before our search-based approach, we leveraged their results for comparison on DDPM models. We followed the evaluation pipeline established by DM-nonuniform to report performance within 5 and 10 optimization steps. In general, solver-based methods tend to exhibit inferior results under extremely limited numbers of function evaluations (NFE), such as 5 or 6 steps.

We provide the experiments below 5 steps. Note that when reduced to a single step, our algorithm essentially has no parameters and will have exactly the same performance of the Euler solver with 1 step.

So, theoretically speaking, our algorithm will converge to the same result as the Euler method and Adam-like methods in 1 step and huge sampling steps(eg. 500 steps or 1000 steps), and will significantly outperform these algorithms at intermediate step numbers.

Q.8 Text to Image Metrics Result

We take PixArt-XL-2-256x256[1] as the text-to-image model. We follow the evaluation pipeline of ADM and take COCO17-Val as the reference batch. We generate 5k images using DPM-Solver++, UniPC, and our solver(searched on DiT-XL/2-R256).

Our method consistently achieves better FID metrics results.

[1]. Chen, Junsong, et al. "Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis." arXiv preprint arXiv:2310.00426 (2023).

As the discussion period ends on November 26, we want to ensure that all your questions have been thoroughly addressed. Your feedback is instrumental to us, and we would be grateful if you could spare a moment to provide a final rating and share your thoughts. Your input will greatly inform our future improvements.