BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformers

1. The proposed method accelerates DiT-based models by only about 30% while incurring obvious performance degradation, making its practical application questionable especially compared to other techniques like quantization and distillation. Even the BlockDance-Ada version in Table 4 does not show impressive results.

2. BlockDance-Ada leverages reinforcement learning. What's the training cost?

• Novelty Issue: Both the proposed feature similarity and caching methods have already been introduced in previous works [1], [2]. The main contribution of this paper appears to be the proposal of BlockDance-Ada. However, the experimental results provided are limited, and the performance improvement is marginal. For instance, in Table 4, BlockDance-Ada shows a 0.15s latency reduction compared to BlockDance (N=2), but with a 0.02 increase in FID. This minor improvement may simply reflect a trade-off between latency and FID.
• Lack of Comparative Experimental Results: The paper does not provide sufficient evidence that BlockDance is genuinely faster than existing methods. To accurately compare performance, an evaluation on a Pareto curve with latency and FID as the axes would be more informative.
• Lack of Persuasiveness in Fig. 5: Figure 5 is unconvincing. At first glance, it suggests that the diffusion model maintains a high level of similarity across nearly all timesteps.
• Missing Comparison with Recent Methods: The paper lacks a comparison with recent caching-based acceleration methods, such as [2] and [3].

[1] Xu, Mengwei, et al. "Deepcache: Principled cache for mobile deep vision." Proceedings of the 24th annual international conference on mobile computing and networking. 2018.

[2] So, Junhyuk, Jungwon Lee, and Eunhyeok Park. "FRDiff: Feature Reuse for Universal Training-free Acceleration of Diffusion Models." arXiv preprint arXiv:2312.03517 (2023).

[3] Li, Senmao, et al. "Faster diffusion: Rethinking the role of unet encoder in diffusion models." arXiv e-prints (2023): arXiv-2312.

Although the block-wise optimization approach differs from DeepCache, i.e., the authors arguing that DeepCache does not specifically aim at highly similar features for reuse, the core concept of caching intermediate features for reuse is to some extent similar. The implementation and experimental impact of reusing similar features in this work appear somewhat marginal. See comments below for more details.

Experimental Results: While the proposed method shows slight improvements over DeepCache, the advantage remains limited, as seen in Tables 1, 2, and 4.

As summarized in the Related Works section, many existing approaches aim to reduce the number of sampling steps. Although this study involves dropping specific diffusion blocks within the network, it bears substantial similarity to step-reduction strategies. Comparisons with previous works (e.g., [1]) appear insufficiently comprehensive. Please note [1] is just one of many possible relevant comparisons.

The research first introduces a manually designed caching strategy, BlockDance, which I would like to consider as a vanilla baseline, and subsequently an instance-wise learnable strategy, BlockDance-Ada, for adaptively reusing features. While this adaptive reuse approach seems reasonable and to be a highlight of the paper, its demonstrated improvement seems limited (Table 4).

Minor Comments:

Claimed Conclusion: “Unlike .., BlockDance prioritizes the identification of the most structurally similar features, referred to as Structurally Similar Spatio-Temporal (STSS) features” Since MSE is used to quantify similarity, it is unclear why these features are referred to as the most ‘structurally’ similar. Could the authors clarify how these features represent 'structural' characteristics?

Claimed Demonstration: “while the deeper blocks shift their focus towards generating more complex high-frequency texture information, such as clouds and crowds within depth of field.” This difference is not immediately apparent to me. Could the authors indicate these distinctions with arrows or highlights?

Claimed Summary: “Several studies (Ma et al., 2024b; Li et al., 2023) have unearthed the existence of redundant features in U-Net-based diffusion models, but their coarse-grained feature reuse strategies include those low-similarity features, leading to structural distortions and text-image misalignment.” Could experimental evidence be provided to support this conclusion?

Figure 5: The text font is too small, which could be enhanced for readability.

[1] Li L, Li H, Zheng X, et al. Autodiffusion: Training-free optimization of time steps and architectures for automated diffusion model acceleration, ICCV, 2023.

1. The paper does not adequately address prior research efforts such as FORA [1], Delta DiT [2], and PAB [3]. The motivation highlighted in Figure 2, which describes the reduction of redundant computations, has been extensively covered in these works. Additionally, these relevant studies are not referenced or compared in the experimental section, limiting the contextual understanding of how BlockDance differentiates or builds on these methodologies.

2. The first contribution appears to have a significant overlap with existing work, suggesting that the novelty is relatively incremental. Without a deeper analysis or clearer distinction from prior caching strategies, the innovation might seem limited compared to the foundational ideas already established.

3. The practical impact of BlockDance seems constrained. As seen in Table 4, the adaptive strategy BlockDance-Ada only achieves about a 5% further reduction in computation compared to the standard BlockDance with 𝑁=2, indicating that the improvements, while present, may not justify the additional complexity introduced.

[1] FORA: Fast-Forward Caching in Diffusion Transformer Acceleration

[2] $\Delta$-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers

[3] Real-Time Video Generation with Pyramid Attention Broadcast