PFDiff: Training-Free Acceleration of Diffusion Models Combining Past and Future Scores

W7: The comparison of FID between PFDiff and AMED-Solver [R1].

A: In response to the reviewer's suggestion, we have included additional FID comparison results between PFDiff and AMED-Solver [R1]. Notably, [R1] is a TRAINING-BASED method, while PFDiff is a TRAINING-FREE method. Using the same pre-trained EDM model [R2], we calculated the FID on the ImageNet 64$\times$64 dataset, as shown in the following table:

ImageNet 64$\times$64, FID$\downarrow$

The results show that the proposed PFDiff, which incurs no training cost, can even outperform AMED-Solver [R1], one of the current strongly competitive training-based methods, under most NFE. This further demonstrates the effectiveness of the proposed PFDiff.

Others: Correct the reference years and move Figure 1 to the appendix.

A: Thank you for the reviewer's comment. We have moved Figure 1 to Appendix D.9 as suggested and have corrected the reference year in line 52.

Q1: Have the authors also tried adding the future gradient step to higher-order ODE solvers?

A: Yes, we have. However, adding the future gradient into higher-order ODE solvers is highly sensitive to the choices of hyperparameters $k$ and $h$, leading to unstable results. We believe this instability may arise from the fact that higher-order solvers do not converge with a few NFE (<10) [R3].

[R1] Zhou, Zhenyu, et al. "Fast ode-based sampling for diffusion models in around 5 steps." CVPR 2024.

[R2] Tero Karras et al., "Elucidating the design space of diffusion-based generative models." NeurIPS 2022.

[R3] Cheng Lu et al., "DPM-solver: a fast ODE solver for diffusion probabilistic model sampling in around 10 steps." NeurIPS 2022.

We sincerely appreciate the reviewer for carefully reviewing our work and providing valuable comments. Below are our responses to all questions. We would greatly appreciate it if you could consider increasing the score if you are satisfied with our response.

W1.1: The proposed method lacks theoretical support and is mainly motivated by the output similarity observation shown in Figure 2a. How reliable is the similarity observed in Figure 2a, and how much does it vary across different diffusion frameworks?

A: It is important to emphasize that our method consists of two core components: the use of future gradients and the use of past gradients. In Equation (15) and Proposition 3.1, we employ a Taylor expansion and further derivations to theoretically demonstrate that the use of future gradients can reduce discretization errors. Furthermore, the use of past gradients, based on the similarity observation in Figure 1a (original Figure 2a), is highly reliable. We have supplemented additional experimental results in Figure 5a and further analysis in Appendix D.8. The results demonstrate that this similarity exhibits strong consistency across three different diffusion frameworks: DDPM, ScoreSDE [R1], and EDM [R2], which fully validates the reliability of the similarity observation.

W1.2: Please provide more details of experiments in Figures 2a and 2b.

A: We have provided more details of the experiments regarding Figures 1a (original Figure 2a) and 1b (original Figure 2b) in Appendix D.8. Additionally, we added Figure 5 in Appendix D.8 to further expand the experimental content in this part. Figure 5a demonstrates that the similarity observation is consistent across different diffusion frameworks, while Figure 5b further validates that future gradient is more reliable than "springboard".

W1.3: Please explore the connection between the proposed method and the curvature of the sampling trajectory.

A1: Thank you for your valuable suggestions. We have carefully reviewed the literature on sampling trajectories [R3], [R4], [R5] and have added relevant analysis to Section 3.5. In summary, [R4] indicates that diffusion model sampling trajectories lie in a low-dimensional subspace and are close to a straight line. This supports the rationale behind the use of past gradients. [R5] notes that the trajectories exhibit a "boomerang" shape, meaning the curvature of the sampling trajectory starts small, then increases, and finally decreases. Based on this observation, we further analyze that PFDiff utilizes future gradients to predict the future update direction in the large curvature regions of the trajectory, thereby correcting discretization errors introduced by sampling along the tangent direction. Additionally, the PFDiff correction process shown in Figure 1c (original Figure 2c) is consistent with this analysis.

W2: Regarding writing issues.

A: We sincerely thank the reviewer for highlighting the writing issues. While we respectfully acknowledge the concerns raised, we have a differing perspective on some points and will address each one individually.

• The definition of $Q$ is not clear. We corrected the definition of $Q$ on lines 183 to 184. Specifically, when $n=0$, $\hat{t}\_{0} = t\_{i-1}$.
• Why not put Proposition 3.1 at the beginning of section 3.3? We made this arrangement to ensure that the logical description of the PFDiff update process in Sections 3.2 and 3.3 is more coherent.
• Function $s()$ is not defined in eq 7 and eq 8. In fact, the function $s()$ is already defined in eq (7). Specifically, $s(\epsilon_\theta(x_t,t),x_t,t) :=f(t)x_t+\frac{g^2(t)}{2\sigma_t}\epsilon_\theta(x_t,t)$.
• Function $h()$ is not defined in eq 9. In fact, the function $h()$ is already defined in lines 185 to 189.
• Line 348, notations $l$ and $h$ are undefined. We added the definition of $h$ in lines 327 to 328. Notably, the "1" is the digit "1" (not the letter $l$).
• Using 1000 NFE as the ground truth is not rigorous. We removed the expression of "ground truth" in Figure 1b (original Figure 2b).

[R1] Yang Song et al., "Score-based generative modeling through stochastic differential equations." ICLR 2021.

[R2] Tero Karras et al., "Elucidating the design space of diffusion-based generative models." NeurIPS 2022.

[R3] Sabour, Amirmojtaba, Sanja Fidler, and Karsten Kreis. "Align your steps: Optimizing sampling schedules in diffusion models." ICML 2024.

[R4] Zhou, Zhenyu, et al. "Fast ode-based sampling for diffusion models in around 5 steps." CVPR 2024.

[R5] Defang Chen, et al. "On the Trajectory Regularity of ODE-based Diffusion Sampling." ICML 2024.

W3: Additional FID results of PFDiff for NFE > 20.

We supplemented the FID experimental results for NFE > 20 on the CIFAR10 and CelebA 64$\times$64 datasets, as shown in the table below:

CIFAR10, FID$\downarrow$

CelebA 64$\times$64, FID$\downarrow$

PFDiff is designed for diffusion model acceleration, which is the reason we focus on less NFE. Still, as can be seen from the table, under the condition of NFE > 20, PFDiff consistently improves the sampling quality of DDIM and achieves faster convergence.

W4: Some FID results of PFDiff are missing.

A: In fact, the missing FID results for PFDiff were originally provided in Tables 2-6 of Appendix D. These results were deliberately omitted to make the comparisons in the figures clearer. For example, when NFE=12 in Figure 3 (original Figure 4) and NFE=10 in Figure 4 (original Figure 5), PFDiff already outperforms the comparison methods with 20 NFE. Therefore, omitting the subsequent PFDiff results helps to highlight the effectiveness of our method.

W5: Why PFDiff is worse than the baseline Analytic-DDIM when NFE=6?

A: This phenomenon may be due to the introduction of new errors in the estimation of the optimal variance by Analytic-DDIM [R1], which are amplified by PFDiff when NFE is small, leading to some unpredictable performance degradation. Specifically, although Analytic-DDIM derives the optimal variance for DDIM sampling from a theoretical perspective, the optimal variance contains an unknown term: $\mathbb{E}\_{q\_{n}\left(\boldsymbol{x}\_{n}\right)} \frac{\| \| \nabla\_{\boldsymbol{x}\_{n}} \log q\_{n}\left(\boldsymbol{x}\_{n}\right)\| \|^{2}}{d}$, which is approximated on different datasets [R1]. This introduces new errors.

W6: In Figure 5, the results of DPM-solver+PFDiff are missing.

A: We did not include the experimental results for the following reasons: First, DDIM and DPM-Solver-1 are equivalent [R2]. Second, with fewer NFE, higher-order DPM-Solvers perform worse than DDIM on Stable-Diffusion [R3]. Finally, DDIM+PFDiff has outperformed other previously best-performing solvers. Additionally, we also provide experimental results applying PFDiff to DPM-Solver in Stable Diffusion with a guidance scale of 7.5, as shown in the table below:

Stable Diffusion, FID$\downarrow$

As shown in the table above, PFDiff further improves the sampling quality of DPM-Solver-1 and DPM-Solver-2. Notably, the results in the table come from the original DPM-Solver implementation, while the DPM-Solver results in the paper utilized additional tricks from [R3] to achieve better performance.

[R1] Fan Bao, et al. "Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models.", ICLR 2022.

[R2] Cheng Lu et al., “DPM-solver: a fast ODE solver for diffusion probabilistic model sampling in around 10 steps.” NeurIPS 2022.

[R3] Cheng Lu et al., "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models." arXiv:2211.01095.

Thank you for your response and for raising the rating! We are so glad to hear that some of your doubts have been resolved!

Regarding the new issue about the search for parameters $k$ and $h$, we emphasize that the sample quality is only slightly affected by the choice of $k$ and $h$. By fixing $k=2$ and $h=1$, PFDiff achieves fairly satisfactory results across different NFE and datasets, as shown in Table 7. Furthermore, in Appendix D.6.1, we have proposed that by sampling only 5k samples to compute the FID, the optimal values of $k$ and $h$ can be easily determined. Notably, the search can also be easily generalized to other evaluation metrics. As suggested by the reviewer, we further analyzed the trajectory shape of the diffusion process and proposed a new Automatic Search strategy with almost no additional cost based on truncation errors for selecting $k$ and $h$. The details are as follows:

For the trajectory shape analysis, as discussed in Section 3.5, PFDiff uses future scores to correct the discretization errors of the baseline solver in regions of large curvature along the trajectory. This indicates that PFDiff accumulates less truncation error compared to the baseline solver. To validate this conclusion, we added Figure 5 in Appendix D.6.1, which shows a comparison of the truncation errors of PFDiff with and without DDIM. Figure 5 clearly shows that PFDiff significantly reduces truncation errors.

Based on the correlation between PFDiff's improvement in sample quality and its reduction in truncation errors, we propose the automatic search strategy: we "warm up" only 256 samples for different $k$ and $h$ respectively, compute the average truncation error using MSE for each, and use this to further determine the values of $k$ and $h$. To validate the effectiveness of this strategy, we conducted experiments on CIFAR10, varying the values of NFE, $k$, and $h$. For each combination, we sampled 256 samples and computed the average truncation error relative to DDIM with 1000 NFE. The results are shown in Table A1 below (for clearer comparison, we also provide the FID results with 50k samples in Table A2 (Table 7 in the paper)):

Table A1: Truncation error (MSE$\downarrow$), only 256 samples

Table A2: FID$\downarrow$, 50k samples

Based on Table A1, we selected the $k$ and $h$ corresponding to the minimal truncation error, and further obtained the corresponding FID values {22.38, 9.84, 7.01, 4.95, 4.63/4.39, 4.25, 4.14} with NFE $\in${4, 6, 8, 10, 12, 15, 20} in Table A2. This set of FID values is comparable to the optimal FID, indicating that determining $k$ and $h$ based on truncation error is reasonable. Notably, in Table A1, we "warmed up" only 256 samples, so the additional cost introduced by the automatic search strategy is negligible. The 256 samples are sufficient to capture the dataset's statistical properties because the shapes of sampling trajectories are highly similar [R1]. The above-mentioned relevant results and analyses have been added to lines 477-479 of Section 4.3 and Appendix D.6.1.

[R1] Defang Chen, et al. "On the Trajectory Regularity of ODE-based Diffusion Sampling." ICML 2024.

Thank you again for your response and insightful suggestions!

Per the reviewer’s suggestion, we have extended the automatic search strategy to additional datasets, including CelebA 64$\times$64, LSUN Church 256$\times$256, and LSUN Bedroom 256$\times$256. Specifically, we "warm up" only 256 samples for different values of $k$, $h$, and NFE, respectively, and computed the average truncation error using MSE relative to DDIM with 1000 NFE. The results are presented in Tables A1, A3, and A5 below. For clearer comparison, we also provide the FID results in Tables A2, A4, and A6. Based on Tables A1, A3, and A5, we selected the values of $k$ and $h$ corresponding to the minimal truncation error, and further obtained the corresponding FID values {13.29, 8.38, 5.88, 5.41, 5.24, 5.18, 5.19}, {37.90, 18.30, 14.34, 13.41, 12.90, 12.61, 11.76}, and {78.57, 18.72, 12.35, 9.55, 8.82, 8.34, 7.70} from Tables A2, A4, and A6, with NFE $\in$ {4/5, 6, 8, 10, 12, 15, 20}. Consistent with CIFAR-10, this set of FID values is comparable to the optimal FID, indicating that the automatic search strategy is robust and generalizable.

Table A1: CelebA 64$\times$64, Truncation error (MSE$\downarrow$), only 256 samples

Table A2: CelebA 64$\times$64, FID$\downarrow$, 50k samples

Thank you for recognizing the automatic search strategy and for increasing the rating!

We also greatly appreciate the reviewer’s valuable and insightful feedback during the review process, which has greatly contributed to the improvement of our manuscript!

We sincerely appreciate the reviewer's valuable review of the manuscript and the recognition of our work of the work presented in the paper. Below are our responses to all questions. We kindly hope you could consider increasing the score if you are satisfied.

W1: How will PFDiff maintain quality across different types of diffusion models?

A: All diffusion models can be unified under the perspective of SDE/ODE modeling [R1], and the sampling process of PFDiff is built upon the theoretical foundation of ODE. Therefore, theoretically, PFDiff can be applied to all types of diffusion models. Additionally, beyond the DDPM and ScoreSDE [R1] framework models mentioned in the paper, we have also included experimental results from a more advanced diffusion model within the EDM [R2] framework, as shown in the table below:

ImageNet 64$\times$64, FID$\downarrow$

As can be seen from the table, PFDiff also effectively improves the sampling quality of DDIM within the EDM framework, further validating the broad adaptability of PFDiff across different types of diffusion models.

W2.1: How does the algorithm handle scenarios where gradients exhibit dispersion?

A: In fact, the primary design goal of PFDiff is to address the gradient dispersion issue observed in baseline fast solvers (e.g., DDIM). Gradients represent the direction of sampling updates, and when the NFE is small, dispersion arises from misalignment between the intermediate states that need to be updated and the current direction of the sampling update (i.e., neural network output). PFDiff mitigates this issue by introducing the foresight update mechanism of Nesterov momentum, which predicts the future sampling update direction. This effectively alleviates the misalignment (i.e., gradient dispersion) between the gradients and the intermediate states that need to be updated in baseline fast solvers, as theoretically guaranteed in Proposition 3.1.

W2.2: Is there a possibility of accumulating errors in the proposed approach?

A: The error accumulation in PFDiff arises from the baseline fast solvers (e.g., DDIM). Notably, PFDiff accelerates sampling by reducing the discretization error of the existing fast solvers. Therefore, the error accumulation in PFDiff will be smaller than that in the baseline fast solvers.

W3: Can the value of $k$ be further increased, and how would PFDiff perform in that case?

A: Thank you for the reviewer’s suggestion. We have increased the value of the parameter $k$ to 4 on the CIFAR10 dataset and conducted further performance analysis, as shown in the following table:

CIFAR10, FID$\downarrow$

From the table, it can be seen that when the value of $k$ is increased to 4, PFDiff improves the sampling quality of DDIM at certain NFE. However, increasing $k$ leads to the problem of over-searching. Therefore, $k \le 3$ might be a more suitable choice, as it provides a better balance between sampling quality and the cost of parameter search.

[R1] Yang Song et al., "Score-based generative modeling through stochastic differential equations." ICLR 2021.

[R2] Tero Karras et al., "Elucidating the design space of diffusion-based generative models." NeurIPS 2022.

Dear Reviewer BQ25,

Thank you once again for your valuable review of the manuscript and for your recognition of our work! As the discussion period draws to a close, we would appreciate it if you could let us know whether your concerns have been addressed. We would be very pleased to continue the discussion if you have any further questions!

Best, Authors

We are so glad to hear that your concerns have been addressed! Thank you once again for recognizing our work!

We sincerely appreciate the reviewer's meticulous review and insightful comments, which have helped improve our paper. Below are our responses to all the questions. Please consider increasing the score if you are satisfied.

W1.1: The multi-step solver undermines the orthogonality.

A: It is important to emphasize that, in theory, higher-order (not multi-step) solvers and future gradients are orthogonal, as shown in Equation (14). However, in the practical experiments, we found that introducing future gradients into higher-order solvers is highly sensitive to the choices of hyperparameters $k$ and $h$, leading to unstable results. Removing the future gradients and only adding past gradients (which is also the core component of our method) to the higher-order solver yields more stable performance, which significantly mitigates the convergence issues, as shown in Figure 3(c) (originally Figure 4(c)).

W1.2 and Q1: Why are there no experimental results applying PFDiff to DPM-Solver in Stable Diffusion?

A: We did not include the experimental results for the following reasons: First, DDIM and DPM-Solver-1 are equivalent [R1]. Second, with fewer NFE, higher-order DPM-Solvers perform worse than DDIM on Stable-Diffusion [R2]. Finally, DDIM+PFDiff has outperformed other previously best-performing solvers. Additionally, we also provide experimental results applying PFDiff to DPM-Solver in Stable Diffusion with a guidance scale of 7.5, as shown in the table below:

Stable Diffusion, FID$\downarrow$

As shown in the table above, PFDiff further improves the sampling quality of DPM-Solver-1 and DPM-Solver-2. Notably, the results in the table come from the original DPM-Solver implementation, while the DPM-Solver results in the paper utilized additional tricks from [R2] to achieve better performance.

W2: The methodology can be classified as a standalone ODE solver.

A: We strongly agree with the insightful classification: the methodology can be classified as a standalone ODE solver, depending on the perspective. We have incorporated this classification in Section 3.4 (line 337 to line 339).

W3: The use of "gradient guidance" in the title and text is potentially misleading.

A: We appreciate the reviewer for pointing out the potential misunderstanding in the title. As revised, the title now reads: "PFDiff: Training-Free Acceleration of Diffusion Models Combining Past and Future Scores." Additionally, we have made corresponding changes in the paper, replacing "gradient" with "score" and removing the expression of "gradient guidance".

W4 and Q3: Are the MSE scales of the future gradient and the springboard directly comparable?

A: Thank you for raising this very valuable question. Based on the suggestion in Q3, we have added a fairer comparison in the paper, showing the MSE of the status separately updated using the "springboard" and future gradient. The corresponding results are presented in Figure 6(b) in Appendix D.8. Notably, the trends in Figure 6(b) and Figure 1(b) (original Figure 2(b)) are consistent, which validates the effectiveness of using the future gradient.

Q2: In Equation 14, is it correct to plug the $n$ points obtained from the $\Delta$ interval ODE solver into the $2\Delta$ interval ODE solver?

A: Yes, you are nearly correct. Equation 14 represents plugging the $p$ (not $n$) points obtained from the $\Delta$ interval ODE solver into the $2\Delta$ interval ODE solver. However, for first-order ODE solvers (e.g., DDIM), $p=1$. This means that we use the gradient corresponding to the endpoint $t_{i+1}$ of the interval $\Delta$ as a replacement for the gradient corresponding to the endpoint $t_{i}$ of the interval $2\Delta$. This replacement effectively reduces the discretization error of the first-order ODE solver, and a detailed theoretical analysis can be found in Proposition 3.1.

[R1] Cheng Lu et al., "DPM-solver: a fast ODE solver for diffusion probabilistic model sampling in around 10 steps." NeurIPS 2022.

[R2] Cheng Lu et al., "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models." arXiv:2211.01095.

Q4: Is there a theoretical analysis of Nesterov momentum on convergence speed?

A: In this work, we only discuss Nesterov momentum as the motivation for using future gradients. Nevertheless, in Equation (15) and Proposition 3.1, we employ a Taylor expansion and further derivations to theoretically demonstrate that the use of future gradients can reduce discretization errors, thereby accelerating the convergence of the sampling process. This theoretical result indirectly suggests that the foresight update mechanism of Nesterov momentum can effectively accelerate convergence.

Dear Reviewer sz2Y,

Thank you once again for your insightful feedback! These suggestions have greatly enhanced the quality of our manuscript. As the discussion period draws to a close, we would appreciate it if you could let us know whether your concerns have been addressed. We would be very pleased to continue the discussion if you have any further questions!

Best, Authors

Thank you very much for your response and for increasing the rating! We are so glad to hear that most of your concerns have been addressed!

In response to the reviewer’s suggestions, we have revised the relevant content and uploaded the revised paper. Specifically, we added Table 1 (i.e., the W1.2 table) to Section 4.2 to demonstrate the orthogonality of PFDiff. Meanwhile, we replaced Figure 1(b) with Figure 6(b) to provide a more intuitive comparison within the same image domain. Furthermore, we also added and modified the corresponding descriptions in Section 4.2 and Appendix D.8.

Thank you once again for your valuable and insightful feedback, which has greatly improved the quality of our manuscript.