$\Delta$-DiT: Accelerating Diffusion Transformers without training via Denoising Property Alignment

Concern 2: Marginal Performance Improvements and Practical Implications

"As I compared the results in Table 1 and Table 3, the performance improvement (speedup and latency) is marginal. (Ours (b=12), Faster Diffusion, PIXART, and TGATE have very close speedup). Yes, it is right that the proposed model has slightly better speed and performance, but it is not feasible. The authors are encouraged to discuss the practical implications of these improvements to better contextualize their importance in real-world applications."

Response:

1. Clarification of Experimental Setup:

   The reviewer notes that in Tables 1 and 3, the acceleration ratios of Ours (b=12), Faster Diffusion, and TGATE are close. This is because we aligned the computational costs of each model for fair comparison. Such a setup ensures that throughput is comparable, allowing us to focus on the generative performance of the models—a common practice in the literature, as seen the Table 4 in DeepCache[1].

   In Table 1, our $\Delta$-Cache and Δ-DiT methods outperform TGATE and Faster Diffusion in terms of FID, IS, and CLIP metrics. In Table 3, Faster Diffusion completely fails to produce usable results, and TGATE exhibits significant degradation in generative quality even at a modest speedup ratio of 1.12×.

2. Experiments with Different Speedups:

   Table 1 reports results under a 1.6× acceleration ratio. To evaluate model performance across various speedup levels, we provide a Pareto curve of generative quality versus computational cost in Appendix C, Figure 9. At 43 TMACs, our model achieves better generative quality compared to TGATE (61 TMACs) and Faster Diffusion (64 TMACs), demonstrating that the improvements are far from marginal.

   For LCM (more challenging image generation scenarios), we further tested our method under higher speedup ratios. At a 1.12$\times$ speedup ratio, our method achieves superior generative results compared to existing methods. Even at a higher speedup ratio of 1.4×\times×, our approach maintains its advantage in metrics like FID and IS, surpassing TGATE’s performance at 1.12$\times$.

   Method	MACs ↓	Speedup ↑	Latency ↓	FID ↓	IS ↑	CLIP ↑
   PIXART-$\alpha$-LCM	8.565T	1.00×	415.255	40.433	30.447	29.989
   Faster Diffusion ($I=4$)	7.953T	1.08×	401.137	468.772	1.146	-1.738
   Faster Diffusion ($I=6$)	7.647T	1.12×	391.081	468.471	1.146	-1.746
   TGATE (Gate=2)	7.936T	1.08×	400.256	42.038	29.683	29.908
   TGATE (Gate=1)	7.623T	1.12×	398.124	44.198	27.865	29.074
   Ours ($b=2$, $N_c=4$)	7.953T	1.08×	400.132	39.967	29.667	29.751
   Ours ($b=2$, $N_c=6$)	7.647T	1.12×	393.469	40.118	29.177	29.332
   Ours ($b=2$, $N_c=11$)	6.883T	1.24×	350.539	42.653	29.810	29.689
   Ours ($b=2$, $N_c=16$)	6.118T	1.40×	306.334	44.043	29.303	29.268

   It is worth noting:**

   • Faster Diffusion fails to generate properly even at a modest 1.12× speedup.

   • TGATE achieves only a maximum speedup ratio of 1.17× before significant quality degradation occurs.

3. Our method delivers a meaningful speedup, particularly for challenging tasks like LCM, demonstrating its robustness and effectiveness in both moderate and high-acceleration scenarios. Our method is a training-free approach that enables seamless acceleration of existing DiT models. This characteristic makes it particularly suitable for real-world generative scenarios, where it can deliver substantial speedups with minimal integration effort.

[1] DeepCache: Accelerating Diffusion Models for Free

Concern 3: Ambiguity in Terminology

"In page 6, line 302, what do you mean by 'imaginary parts of a complex number'?"

Response:

Re(·) and Im(·) represent the real and imaginary parts of a complex number, respectively. This is the exact mathematical expression for a complex number. You can refer to the article we cited in the manuscript (page 6, line 283). The imaginary parts of a complex number are mentioned on page 18 of that reference. Thank you for your feedback.

Concern 1: Clarity of Presentation

"The presentation of the paper is not clear. It is hard to understand the proposed model. For example, ∆-Cache and the DiT blocks are introduced with insufficient context. Even in the abstract, jumps between ideas such as the role of shallow and deep blocks without clear transitions complicate the flow. In the proposed method section, there are repetitions on the performance of the blocks. These parts can be compact to improve the presentation. Several parts of the paper are difficult to understand, for example, page 2, line 86–94, page 5, line 237–240."

Response:

1. Insufficient Context for ∆-Cache and DiT Block

   Regarding the $\Delta$-Cache, the technical details of its application in DiT are clearly presented in Figure 4b and Equation 3, which you can refer to. For details on the DiT Block, you can refer to page 183-187, where we have formalized the role of DiT Blocks.

2. Redundancies in the Performance Discussion of Blocks

   In Section 4, we have divided the content into two parts: Section 4.1 introduces $\Delta$-Cache and provides a qualitative and quantitative analysis of the role of DiT Blocks, while Section 4.2 proposes an alignment-based acceleration method based on the properties discovered in Section 4.1. In Section 4.1, our analysis of the blocks' performance is comprehensive, covering both qualitative and quantitative aspects, which may seem repetitive. However, this thorough analysis is essential to the conclusions on which our method relies. So we think this part is necessary.

3. Some Parts of the Paper Are Difficult to Understand

   The potential present improvements you pointed out have been revised in blue text in the manuscript. Please refer to the pdf.

Concern II:

1. Training-Free Acceleration:

We want to highlight that "training-free acceleration" is an inherent property of our proposed method. Its novelty lies in the design elements that confer this training-free characteristic, allowing for rapid adaptation without the need for costly training, which can be valuable in various real-life deployment scenarios. In particular, the novelty lies in three key aspects:

• Cache for DiT: We introduce a caching mechanism tailored specifically for transformer-based architectures like DiT, achieving an effective balance between computational efficiency and generative quality. This innovation broadens the utility of caching, positioning it as a powerful tool for accelerating diverse tasks in diffusion models.
• Differentiating Shallow and Deep Blocks: This paper qualitatively and quantitatively explores the effects of DiT blocks on the generated image. By distinguishing the roles of shallow and deep transformer blocks within diffusion models, our approach leverages their unique characteristics to devise more efficient alignment and caching strategies. This differentiation optimizes performance while preserving generative fidelity, a critical requirement in diffusion model acceleration.
• Denoising Property Alignment: Our method aligns seamlessly with the inherent characteristics of diffusion processes, ensuring that the caching mechanism preserves essential generative dynamics while achieving measurable speedups. 
• We respectfully draw the reviewer's attention to the fact that the corresponding methodological design and observations are novel, and they are also carefully detailed in our list of contributions.

2. Experimental Results:

The experimental results substantiate the effectiveness and robustness of our approach:

• In Table 1, our method surpasses existing approaches like TGATE and Faster Diffusion in key metrics such as FID, IS, and CLIP score, demonstrating superior quality and acceleration.
• In Table 3, we highlight the robustness of our method under challenging settings. Competing methods often fail to maintain generation quality or achieve meaningful speedup, whereas our approach performs reliably.
• In Appendix C, a Pareto curve analysis illustrates that our method consistently achieves superior performance across various computational budgets. This analysis reinforces the practicality and scalability of our approach in diverse scenarios.

3. Comprehensive Validation and Reviewer Endorsements:

As noted by other reviewers (Reviewers uETh, D8UN, 81e2), the training-free nature of our method makes it easy to adopt and widely applicable. We have rigorously validated our approach across a broad range of benchmarks and tasks, underscoring its versatility:

• Our method has been tested on multiple transformer-based diffusion models, including DiT, PixArt, and SD3 (Appendix B).
• We demonstrate compatibility with complementary acceleration techniques, such as quantization, further boosting performance (Appendix E).
• The wide-ranging validation across models and benchmarks highlights the adaptability and robustness of our approach, establishing it as a generalizable solution for real-world applications.

Concern III:

• In our manuscript, we have included this discussion in detail on Page 5, lines 257-259 of the main context. Given the high similarity of the $\Delta$ values between steps (as shown in Figure 4c), the impact of a single block on the final generation is minimal, and such minor changes are difficult to capture effectively. To enhance their contribution to the final output, we grouped several consecutive blocks into sections. As illustrated in Figure 2, the effects of these block groups on the generated output are clearly visible, especially in terms of image details and contours. Additionally, the varying proportions of high and low frequencies in the spectrum of Figure 5 further highlight the distinct effects of these blocks on the image details and contours.
• We have also included more fine-grained experiments in Appendix H, where we split the blocks into smaller groups (e.g., 1-7, 7-14, 14-21, and 21-28). The qualitative and quantitative results presented in Figure 12 support the conclusions drawn in the main text, reinforcing the findings and confirming the consistency of our approach.

We believe we have addressed these new concerns that the reviewer has raised, if the reviewer has further questions or concerns, please feel free to let us know.

Concern 3: Counter-Intuitive Performance Gains from Feature Caching

"The proposed delta-cache is training-free and can be used without training data. The main experimental results in the paper show that caching some feature changes can even improve the generation performance, which seems counter-intuitive. Could the authors provide a deeper analysis of this?"

Response:

The observed decrease in FID scores with caching can be attributed to the following factors:

1. Robustness of Diffusion Models with Reduced Timesteps: In the original training of diffusion models, the timestep is typically set to 1000 steps. However, fast samplers like DDIM demonstrate that these models can maintain similar performance in as few as 50 steps, showcasing robustness to reduced computational effort while preserving generative quality.
2. Alignment with Trends in Model Acceleration: Across various acceleration methods—such as pruning [1], token fusion [2], and our caching approach—the primary objective is to approximate the original model efficiently while reducing computational requirements. It is common for these methods to achieve improved FID scores compared to baselines. For example:
   • Pruning: Table 1 in [1] shows that the pruned model surpasses the baseline in generation quality on the CelebA-HQ dataset.
   • Token Fusion: Table 4 in [2] demonstrates better FID scores for their method when the compression rate is below 50%. Our caching approach may reduce noise in intermediate representations, acting as an implicit regularizer, occasionally leading to better FID scores.
3. Feature Prioritization: Caching $\Delta$ enhances the focus on salient feature changes, indirectly improving the overall output quality by prioritizing meaningful transformations.

[1] Structural Pruning for Diffusion Models. NeurIPS2023. [2] Token Merging for Fast Stable Diffusion. CVPR2023.

Furthermore, we provide additional metrics to facilitate a more comprehensive comparison with other methods, as detailed in Appendix A, Table 6. These results further substantiate the superiority of our proposed method, highlighting its consistent performance across various evaluation criteria. By including these additional metrics, we aim to provide a clearer and more robust demonstration of the advantages offered by $\Delta$-DiT over existing approaches.

Concern 2: Achieving More Significant Acceleration

"The method achieves 1.6x acceleration in 20-step generation, which seems insignificant compared to other compression methods such as post-training quantization and time-step distillation. Is it possible to achieve more than 2x acceleration without sacrificing performance? Could the authors clarify the advantage of cache-based methods on DiTs?"

Response:

We acknowledge the reviewer’s concern regarding the acceleration gains. While a 1.6x acceleration may appear modest compared to heavily training-dependent methods such as post-training distillation, it is important to note that our approach is orthogonal to other acceleration techniques. By combining Δ-DiT with methods like quantization or consistency model, we can achieve a speedup of over 2x in the condition of 20 steps generation without sacrificing quality. To conclude, Δ-DiT offers unique advantages that complement existing approaches:

1. Training-Free Deployment:

   Unlike methods such as time-step distillation or post-training quantization, Δ-DiT is entirely training-free. It does not require fine-tuning or access to additional training data, making it particularly suitable for scenarios with limited computational resources or restricted dataset availability.

2. Quality Preservation:

   While approaches like post-training quantization or lcm often degrade quality in high-fidelity tasks, Δ-Cache preserves generative quality alongside measurable acceleration. For example, in low computation mode (LCM) settings, our method maintains image quality where other approaches experience significant drops, such as a large decrease in CLIP score. In Appendix C, Table 8, we further demonstrate that even when the speedup ratio is pushed to 1.4x, Δ-DiT still sustains high-quality image generation.

   Method	MACs ↓	Speedup ↑	Latency ↓	FID ↓	IS ↑	CLIP ↑
   PIXART-$\\alpha$-LCM	8.565T	1.00×	415.255	40.433	30.447	29.989
   Faster Diffusion ($I=4$)	7.953T	1.08×	401.137	468.772	1.146	-1.738
   Faster Diffusion ($I=6$)	7.647T	1.12×	391.081	468.471	1.146	-1.746
   TGATE (Gate=2)	7.936T	1.08×	400.256	42.038	29.683	29.908
   TGATE (Gate=1)	7.623T	1.12×	398.124	44.198	27.865	29.074
   Ours ($b=2$, $N_c=4$)	7.953T	1.08×	400.132	39.967	29.667	29.751
   Ours ($b=2$, $N_c=6$)	7.647T	1.12×	393.469	40.118	29.177	29.332
   Ours ($b=2$, $N_c=11$)	6.883T	1.24×	350.539	42.653	29.810	29.689
   Ours ($b=2$, $N_c=16$)	6.118T	1.40×	306.334	44.043	29.303	29.268

3. Compatibility with Existing Models:

   Δ-Cache integrates seamlessly into pre-trained models like DiTs without requiring architectural modifications, making it highly practical for real-world deployment. Additionally, to achieve higher acceleration (e.g., >2x), our method can be combined with complementary techniques. For instance, in Appendix E, we show that combining Δ-DiT with quantization achieves over 2.06x speedup while further improving the FID score.

   Method	Latency ↓	Speedup ↑	FID ↓	IS ↑	CLIP ↑
   Pixart-$\\alpha$	2290.668	1.00×	39.002	31.385	30.417
   Pixart-$\\alpha$+Quantization	1609.016	1.42×	39.044	31.482	30.418
   Ours	1534.551	1.60×	35.882	32.222	30.403
   Ours+Quantization	1114.004	2.06×	35.855	32.305	30.394

By clarifying these advantages in our revision and presenting supporting analyses, we hope to address the reviewer’s concerns comprehensively.

Concern 1: Optimization of $b$ and $N_c$

"The choice of bound $b$ and the number of cached blocks $N_c$ is empirical. Figures 6 and 7 show that the results are sensitive to the values of $b$ and $N_c$. Is there a better optimization method for delta-cache?"

Response:

We appreciate the reviewer’s observation regarding the sensitivity of results to $b$ (Bound) and $N_c$ (number of cached blocks). These hyperparameters are currently empirically tuned to balance performance and computational efficiency, as demonstrated in Table 2. Our approach emphasizes simplicity and effectiveness, avoiding excessive search costs while still outperforming related works.

To address this concern further, we explored optimization techniques to automate the selection of $b$ and $N_c$, reducing sensitivity to manual choices. Specifically:

1. Hyperparameter Optimization:

   We conducted a search over $b$ and $N_c$, treating them as optimization parameters. Using a FlashEval setup with a CLIP-based evaluation metric and a 50-prompt benchmark, we ran the search across six GPUs for several hours. This systematic approach enabled us to identify the optimal settings for $b$ and $N_c$, achieving the best CLIP scores.

2. Performance Gains vs. Search Cost:

   While this search process yielded measurable improvements in performance (notably in CLIP scores), it also incurred a significant computational cost. In the updated Appendix F, Table 10, we present a comparison that highlights the tradeoffs between the modest performance gains achieved through search and the additional resource requirements.

This analysis underscores our current preference for simpler configurations, which deliver strong results without the overhead of extensive hyperparameter tuning. Nonetheless, the use of automated optimization techniques represents a promising direction for future work, particularly for scenarios where computational resources are less constrained.

Concern 6: Exact Timestep and Uniformity of Caching Computations

"At which exact timestep do you perform computations for caching? Are these computations performed at uniform intervals?"

Response:

Caching computations are performed at uniform intervals, determined by an empirically chosen bound $b$. Uniform intervals simplify implementation and provide a consistent reduction in computational overhead across timesteps.

In the 20-step generation setting, choosing intervel N=2 (as stated on page 7, line 349) is a reasonable configuration. If the intervals become too large, the similarity between cached and new features diminishes, reducing the effectiveness of caching. Our current choice prioritizes simplicity and avoids introducing additional search costs for hyperparameter tuning.

Concern 7: Unit of the DiT Block in Delta Computations

"What is the unit of the DiT Block when calculating Delta?"

Response:

The unit of the DiT block refers to the output features generated by each transformer layer. $\Delta$-Cache operates on the changes ($\Delta$s) between the output feature maps of consecutive layers, effectively capturing the incremental transformations within the model. This design ensures compatibility with transformer-based architectures while minimizing memory and computational overhead.

As illustrated in Figure 4(b), $\Delta$-Cache specifically caches the incremental changes between $N_c$ consecutive blocks, rather than storing all feature maps individually. This approach not only reduces the storage requirements but also focuses on preserving the most meaningful transformations, enabling efficient acceleration without compromising generation quality.

Concern 8: Memory Overhead of Delta-Diffusion

"What is the memory overhead required to run Delta-Diffusion?"

Response:

The memory overhead of $\Delta$-Cache primarily only depends on the size of the feature maps. The additional memory required is equivalent to storing one extra block’s feature map.

If finer-grained caching were applied—storing feature maps for multiple consecutive blocks—the memory requirements would increase significantly due to the larger number of cached feature maps. However, in our approach, we limit caching to a single feature map across consecutive blocks, effectively balancing memory usage with computational efficiency. This trade-off is justified by the notable performance and acceleration gains achieved while keeping the memory overhead manageable, as demonstrated in Tables 2 and 3 of the paper.

Concern 5: Why Does FID Sometimes Decrease with Caching?

"Despite applying caching, there seem to be many cases where the FID score actually decreases. What might be the reason for this result? Could this outcome imply an experimental error or a lack of accuracy in the FID metric? Are there experimental results using more recent metrics, such as CMMD [1]?"

Response:

The observed decrease in FID scores with caching can be attributed to the following factors:

1. Robustness of Diffusion Models with Reduced Timesteps:

   In the original training of diffusion models, the timestep is typically set to 1000 steps. However, fast samplers like DDIM demonstrate that these models can maintain similar performance in as few as 50 steps, showcasing robustness to reduced computational effort while preserving generative quality.

2. Alignment with Trends in Model Acceleration:

   Across various acceleration methods—such as pruning [1], token fusion [2], and our caching approach—the primary objective is to approximate the original model efficiently while reducing computational requirements. It is common for these methods to achieve improved FID scores compared to baselines. For example:

   • Pruning: Table 1 in [1] shows that on the CelebA-HQ dataset, the pruned model surpasses the baseline in generation quality.
   • Token Fusion: Table 4 in [2] demonstrates better FID scores for their method when the compression rate is below 50%.Our caching approach may reduce noise in intermediate representations, acting as an implicit regularizer, which can occasionally lead to better FID scores.

3. Feature Prioritization:

   Caching $\Delta$ enhances the focus on salient feature changes, indirectly improving the overall output quality by prioritizing meaningful transformations.

   [1] Structural Pruning for Diffusion Models. NeurIPS2023.

   [2] Token Merging for Fast Stable Diffusion. CVPR2023.

We acknowledge, however, that FID may not fully capture perceptual quality in all cases. To address this issue, we introduced the CMMD metric along with two other human preference metrics (ImageReward and HPSv2) as additional evaluation indicators. The results, presented in Appendix A, Table 6, demonstrate that $\Delta$-DiT improves performance across multiple metrics, providing a more comprehensive assessment of its advantages.

Concern 4: Marginal Acceleration Results in Table 3

"The acceleration results in Table 3 are too small."

Response:

While the acceleration gains in Table 3 may seem modest, it is important to emphasize the unique advantages of $\Delta$-DiT in this specific setting. Notably, $\Delta$-DiT is a training-free method that is fully compatible with pre-trained models, requiring no additional resources or fine-tuning. This makes it particularly valuable in scenarios where retraining is infeasible.

Additionally, more results on the tough scenario of LCM are shown in Appendix C Table 8. Under the same speedup ratio (1.12$\times$), our method achieves better generation results compared to existing methods. At a higher speedup ratio (1.4$\times$), the proposed method still maintains an advantage in generation metrics, outperforming TGATE at a 1.12$\times$ speedup. It is worth noting that Faster Diffusion fails to generate properly at a 1.12$\times$ speedup, and TGATE's maximum speedup is only 1.17$\times$. Our speedup ratio is groundbreaking, especially for challenging tasks like LCM.

Concern 1: Lack of Comprehensive Performance Comparisons

"There is a lack of experimental results demonstrating that Delta-DiT is actually faster than existing methods like DPM-Solver and Faster Diffusion. To compare performance accurately, it should be evaluated on a Pareto curve with latency and FID as axes. The paper only provides a single point of comparison in a table, which may be cherry-picked."

Response: (Refer to Appendix C, Figure 9)

We appreciate the reviewer’s suggestion to use Pareto curves for a more comprehensive comparison. While our current results in Tables 1 provides direct comparisons of CLIP-Score and latency with state-of-the-art methods, we agree that a Pareto analysis offers valuable insights into trade-offs.

In response, we have conducted additional experiments to plot Pareto curves using TMACs and CLIP-Score as axes. These curves, included in the updated Appendix C, Figure 9, illustrate that $\Delta$-DiT achieves a favorable balance compared to original models with fewer timesteps and other training-free acceleration methods such as Faster Diffusion and TGATE. This analysis highlights the practical efficiency and robustness of $\Delta$-DiT in balancing computational cost and generative quality.

Concern 2: Lack of Results with U-Net

"There are no experimental results using U-Net. While the paper primarily targets DiT, U-Net is still widely used in diffusion architecture, making a significant limitation to this work."

Response:

We acknowledge the importance of U-Net in diffusion architectures and recognize that the absence of U-Net results represents a limitation of our current experiments. However, it is important to note that existing caching strategies designed for U-Net architectures may not directly translate effectively to DiT due to key structural differences:

1. Model Architecture: U-Net’s encoder-decoder structure generates explicit intermediate feature maps at multiple resolutions, making it well-suited for feature map caching. In contrast, DiT’s pure transformer-based design requires a more generalizable and flexible caching approach, such as $\Delta$-Cache, to handle its distinct computational characteristics.
2. Cacheable Positions: U-Net provides a limited set of meaningful caching positions due to its predefined hierarchical structure, multi-resolution processing, and long skip connections. In comparison, $\Delta$-Cache takes advantage of DiT’s transformer-based architecture to enable caching at multiple transformer blocks, offering finer-grained control and achieving higher efficiency gains.

Our method is specifically tailored for transformer-based architectures, making it particularly effective for DiT and similar models. While methods like Faster Diffusion focus on accelerating U-Net-style architectures, $\Delta$-Cache addresses the less-explored but equally critical challenge of optimizing transformer-based diffusion models.

To address this limitation, we have adapted $\Delta$-Cache for U-Net architectures. The results, presented in Appendix G, Table 11, show that our method outperforms DeepCache, achieving a lower FID score and a higher IS score (32.6 → 33.0) under the same computational settings. These findings further highlight the versatility and effectiveness of $\Delta$-Cache across diverse architectures.

Concern 3: Error in Related Works

"In the case of So et al., they are also targeting DiT."

Response:

We appreciate the reviewer pointing out this oversight. The related works section will be revised to accurately reflect that So et al. also target DiT-based architectures. However, it is still the paradigm of cache feature maps, and our method caches feature deviations at two positions.

Dear Reviewer D8UN:

Thank you for your time and effort in reviewing our paper.

We greatly appreciate your valuable feedback, which has helped us improve our work. We hope that our responses and revisions have effectively addressed your concerns. If you have any additional questions or require further clarification, we would be happy to discuss them. If possible, we would be grateful if you could consider revisiting your score in light of the revised version of our paper.

Thank you again for your thoughtful review and support.

Best regards,

The Authors

Dear Reviewer D8UN,

We greatly appreciate your valuable feedback, which has helped us improve our work. We hope that our responses and revisions have effectively addressed your concerns. If you have any additional questions or require further clarification, we would be happy to discuss them. 

As the deadline for the extended discussion period ends today, we would greatly appreciate your response soon. If our rebuttal adequately addresses your questions, we kindly request an update to your evaluations. Thank you once again for your support.

Best regards,

The Authors

Concern 2: Generalization to Recent Models (e.g., Stable Diffusion 3)

"The authors conduct experiments on PixelArt-and DiT-XL. Could the authors also provide results on more recent models such as SD3, which is also transformer-based?"

Response: (Refer to Appendix B, Table 7)

Thank you for the constructive feedback. We initially chose PixArt-$\alpha$ and DiT-XL as our primary models due to their relevance in evaluating lightweight diffusion models and their widespread use in related works.

To address the reviewer’s concern, we have conducted additional experiments on Stable Diffusion 3 (SD3), a more recent and transformer-based model. The results, presented in the updated Appendix B, Table 7, demonstrate that $\Delta$-DiT retains its effectiveness and efficiency on SD3. Notably, $\Delta$-DiT achieves a 1.6x acceleration while maintaining a lower FID compared to methods using fewer timesteps. Furthermore, it consistently achieves comparable or superior performance to $\Delta$-Cache across three key metrics, confirming the generalizability of our method to state-of-the-art transformer-based diffusion models.

Concern 1: Effectiveness of $\Delta$-DiT compared to $\Delta$-Cache and alternative placements of $\Delta$-Cache

"It seems to me that the alignment ($\Delta$-DiT) does not stand out against the baseline method ($\Delta$-Cache) from Table 1, which makes me concerned about the effectiveness of the alignment. I'm curious if the authors have also tried some other ways to apply the $\Delta$-Cache in different blocks/steps."

Response:

We appreciate the reviewer highlighting this important comparison. While $\Delta$-DiT does not significantly outperform $\Delta$-Cache in certain metrics from Table 1, it offers several notable advantages:

1. Unified Contribution: Both $\Delta$-DiT and $\Delta$-Cache are contributions from our work, with $\Delta$-DiT specifically designed to enhance robustness and generalizability in transformer-based architectures.

2. Robustness Across Tasks and Metrics: $\Delta$-DiT demonstrates consistently competitive performance across diverse tasks compared to $\Delta$-Cache. For example, as shown in Table 2, $\Delta$-DiT outperforms $\Delta$-Cache on both FID and IS metrics. Furthermore, even when performances appear comparable, $\Delta$-DiT often exhibits significantly better robustness and reliability under more challenging configurations.

   Additional validation using advanced metrics, such as CMMD [1], HPSv2 [2], and ImageReward [3], further supports these findings. These evaluations confirm the advantages of $\Delta$-DiT in both quantitative performance and qualitative measures, reinforcing its efficacy as a robust acceleration method.

[1] Rethinking FID: Towards a Better Evaluation Metric for Image Generation.

[2] Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis.

[3] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.

3. Improved Structure and Detail Preservation (Refer to Appendix D, Figure 10): Evaluations using human preference metrics reveal that $\Delta$-DiT achieves superior generation quality compared to $\Delta$-Cache, particularly in preserving structural details. As shown in Appendix D Figure 10, $\Delta$-DiT demonstrates clearer and more consistent preservation of structure and detail, further enhancing its suitability for high-quality generation tasks.

We conducted experiments to explore the effects of applying $\Delta$-Cache at different transformer layers. Our findings, presented in Figures 6 and 7, reveal that the choice of placement significantly impacts the tradeoff between performance and computational overhead. By strategically selecting specific layers for caching, $\Delta$-DiT achieves a better balance between generative quality and efficiency.

Hyperparamter Search (Refer to Appendix F, Table 10):

To fine-tune the hyperparameters $N_c$ (number of cached blocks) and $b$ (bound), we treated them as optimization option and conducted a search over this hyperparameters. Using a FlashEval setup with a CLIP-based evaluation metric over a 50-prompt benchmark, we performed this search across six GPUs for several hours. This approach allowed us to identify the $N_c$ and $b$ settings that yielded the best CLIP scores.

While the search process brought measurable improvements in performance for CLIP-score, it is also important to consider the computational cost of such searches. In the updated table (Appendix F, Table 10), we highlight the tradeoffs between the modest gains achieved through search and the additional resource requirements. This underscores our current focus on simpler configurations, which deliver strong results without incurring the overhead of extensive hyperparameter tuning.