To Reviewer rhnM

We are pleased that you found our paper well-organized and appreciated the comprehensive experimental evaluation across various generation tasks. Your recognition of the effectiveness of our proposed modules is greatly appreciated. We are committed to addressing any remaining issues and questions you have.

Weakness 1: Lack of baselines

Thank you for emphasizing the importance of comparing our method with strong baseline techniques.

1. Compare with TokenCache [3]. | Method | Base Model | FID Score | Speed Up | | :---:|:---:| :---:| :---:| |TokenCache|DiT-XL/2 | 2.37 (+0.13) | x 1.32 | |Ours| DiT-XL/2 |2.38 (+0.11)|x 1.87|

We adopt the standard DiT baseline 2.27, while TokenCache applies a baseline 2.24. TokenCache achieves a 32% speed gain with a slight FID increase. Our approach attains an 87% speed gain with a smaller FID increase, demonstrating superior efficiency and performance.

2. Compare with Ditfastattn [4]. | Method | Base Model | FID Score | IS | Speed Up | | :---:| :---: | :---:|:---:| :---:| | Ditfastattn | DiT-XL/2 (512x512) | 4.52 (+1.36) | 180.34 | x 1.98| | Ours | DiT-XL/2 (512\x512) | 2.96 (-0.08) |242.4 | x 2.45 |

We adopt the standard DIT baseline 3.04, while TokenCache applies a baseline 3.16. Ditfastattn shows a substantial FID increase for speed gain, while our method results in better speed and less FID.

3. Compare with DeepCache [2] and BlcokCaching [1]

DeepCache and BlockCaching share similar features across sampling steps, relying on the redundancy present in multiple steps. In contrast, our approach targets reducing redundancy within the DiT structure at a single sampling step. Our method and their techniques improve diffusion model efficiency along orthogonal dimensions and should be considered complementary rather than directly comparable. For a 1000-step DiT model, DeepCache can achieve substantial reductions in FLOPs—up to 90%—by leveraging massive redundant features across steps. However, obtaining a 90% reduction through model structure compression alone is impractical. Conversely, in a single-step DiT model, our approach can achieve a 50% reduction in FLOPs, while DeepCache and BlockCaching cannot reduce FLOPs, as there are no features to share across steps. Therefore, our method complements DeepCache and BlockCaching by optimizing different aspects of the model's efficiency. Other state-of-the-art methods like DyDiT, DiffCR, and Ditfastattn similarly do not compare against DeepCache and BlockCaching directly.

Instead of comparing solely, we have integrated our approach with DeepCache, demonstrating enhanced performance as follows:

These results show the effectiveness of combining SparseDiT with DeepCache.

Weakness 2: Potential mischaracterization of flow-based methods.

Thank you for identifying conceptual inaccuracies in our paper. We agree with your points and will revise the flow-based approach descriptions, particularly those related to Rectified Flow, to better reflect their roles and frameworks. We experiment 10-step DPM in Table 5. DiT FID: 12.01; Our FID: 12.30.

Weakness 3: Lack of analysis protocol.

(1) SDTM Module Configuration: The number of SDTM modules is empirically determined based on the baseline model's layers. Typically comprising six layers with three sparse transformers, which can achieve a good performance-efficiency trade-off, a DiT-XL houses 28 layers, necessitating four SDTM modules in SparseDiT-XL. Please refer to my response to Reviewer kysk Q2 for more details.

(2) We calculate attention variance to assess the uniformity of attention value distribution. A variance close to zero indicates that attention values are nearly identical, allowing self-attention to function as a global pooling mechanism. Conversely, high variance suggests that each token focuses on a limited set of other tokens. Unlike understanding tasks, such as classification, where global tokens might exist, generation tasks involve attention patterns that tend to capture local information, visible as diagonal lines on attention maps (approximate full-rank matrix). High variance indicates that all tokens are necessary for feature integration. Therefore, a higher variance reflects a greater need for localized information capture. In understanding tasks, high variance can still signify global information capture, but this scenario does not apply to generation tasks.

Weakness 4: Missing isolated analysis of SDTM vs DiT block.

It seems there may be a misunderstanding regarding our method. We do not incorporate additional cross-attention layers; rather, the cross-attention layers in SDTM are derived from the existing self-attention layers. The total number of attention layers in SparseDiT, encompassing both self-attention and cross-attention, remains identical to that of DiT. The parameters of the cross-attention layers match those of the initial self-attention layers, hence no additional parameters are introduced via the cross-attention layers. We acknowledge that two additional linear layers, $W_1$ and $W_2$, introduce extra parameters. However, our poolingformers counterbalance this by utilizing parameter-free full-one matrices to replace self-attention layers, thereby reducing overall parameters. Consequently, the parameter count of SparseDiT remains nearly unchanged. Notably, one SDTM comprises six transformer layers. Below, we provide a detailed parameters and FLOPs for these six transformer layers ($r\in$ [0.61, 0.86]):

Each SDTM consists of six transformers, and should be compared with six DiT transformer blocks in terms of parameters, FLOPs, and throughput:

Owing to the efficiency of poolingformers, the difference in parameters between SparseDiT and DiT is minimal (680M vs. 680M). The efficiency gains are truly from token sparsity.

Weakness 5: Confound metrics from finetuning.

We acknowledge that continued training can enhance the performance of base models, as demonstrated in the table above. However, the primary objective of our method is to achieve a superior performance-efficiency trade-off, rather than improving performance. Additionally, the benefits gained from continued training apply equally to all post-training methods, such as DyDiT, DiffCR [10], TokenCache [3], and Ditfastattn [4]. Consequently, the comparative advantages of our method remain consistent.

Q1: High norm tokens

We did not specifically observe the norm of tokens in our analysis. Our approach focuses on reducing the number of tokens without directly discarding them. Tokens are merged via pooling layers, which represents a form of soft pruning, allowing both high and low norm tokens to be retained without information loss.

Q2: Upsampling and floating point numbers of $r$

Our method utilizes trilinear upsampling, implemented through torch.nn.Upsample. The computation of $r$ is detailed in line 166, where it is adjusted by altering the number of sparse tokens, set in configurations like $4\times 4$ or $8\times 8$. Notably, $r$ remains a discrete value rather than a continuous value.

Q3: l266ff

We have observed that our method generates more stable and precise images across different seeds. The seed used in Figure 4 is the default seed 1, chosen to facilitate easy reproduction by readers. We plan to provide additional visualizations in the next version to further support this observation.

Minors

We appreciate your detailed suggestions regarding the minors. We will modify these problems in the next version.

Limitation

Due to constraints on text space, I have not included a detailed discussion of the solution. Regarding the number of PoolingFormer layers, Figure 1 illustrates that the attention maps from the first two transformer layers exhibit a uniform distribution. Therefore, one or two PoolingFormer layers are adequate, as confirmed by Table 4. In the SDTM architecture, there are three fixed transformers: one for generating sparse tokens, one for recovering dense tokens, and one dense transformer. The number of sparse transformers can be varied, with more sparse transformers enhancing model efficiency. Performance remains consistent as long as the number of sparse tokens does not exceed three. Concerning the pruning rate $r$, our method is not particularly sensitive except for the rate in the final stage. As mentioned in lines 333-336, the effectiveness of the dynamic pruning strategy is closely related to the token count in the later phases of denoising. For practical applications, we suggest using a $10\times 10$ token count in the final stage for optimal performance and a $4\times 4$ token count in the initial stage for efficiency.

Reference

[10] You, H., Barnes, C., Zhou, Y., Kang, Y., Du, Z., Zhou, W., ... & Lin, Y. C. (2025). Layer-and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers. CVPR (pp. 18072-18082).

Thank you very much for your continued engagement and thoughtful feedback！We sincerely appreciate the time and effort you have dedicated to evaluating our rebuttal and the constructive questions you have raised. Your insights are invaluable to us and play a crucial role in enhancing the clarity and rigor of our research. We are committed to addressing your concerns comprehensively and are eager to ensure our paper reaches its highest potential.

Q2: We apologize that our explanation caused any confusion. The variable $M$ represents the number of sparse tokens, which are obtained through pooling operations. Those tokens have discrete values, such as $4\times 4$, $5\times 5$, or $6\times 6$. The $N$ is a fixed, so the equation $r=1-M/N$ results in discrete values. The number of $r$ is limited rather than spanning continuously between 0 and 1. Due to our dynamic pruning rate strategy, the pruning rate $r$ increases as the denoising process progresses. For example, $r\in [0.61, 0.86]$ signifies the the maximum token count is $10\times 10$l yielding $r=1-10\times 10/256=0.61$, while the minimum token count is $6\times 6$, resulting in $r = 1-6\times 6/256=0.86$. During intermediate timesteps, token counts are $7\times 7$ and $8\times 8$, with corresponding $r$ of $0.81$ and $0.75$, respectively. Thus, $r$ belongs to the discrete set {0.61, 0.75, 0.81, 0.86}, which we summarized as $r\in [0.61, 0.86]$ for convince. We apologize again for any misunderstanding and will offer a more detailed explanation in future revisions.

Q3: We think "more precise structure" stems from enhanced global information capture within our model. Sparse tokens encapsulate global information and, due to their limited quantity, integrate this information more effectively than the original DiT. With every token accessing global information, generated structures appear more precise and coherent. Previous research also suggests that global information capture can augment the structural integrity of generated content. For instance, LayoutDiffusion [1] demonstrates the benefits of incorporating global conditions into image features.

You said I have addressed some of your questions, especially weakness 1 and weakness 4. How about weakness 2, weakness 3, and weakness 5? Especially weakness 5, We believe it might be a significant concert for you. Please let us know if you have any further questions. We are eager to engage in further discussions and identify any aspects for improvement.

Reference

[1] Zheng, G., Zhou, X., Li, X., Qi, Z., Shan, Y., & Li, X. (2023). Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22490-22499).

To Reviewer gHMa

Thank you for recognizing the strengths in our paper, including the well-motivated token sparsification strategy and our impressive performance across multiple benchmarks. We're glad the paper was clear and well-written. We are eager to address your queries and improve our work based on your valuable feedback.

Q1: The practical value of this technique.

Training-free methods indeed garner significant interest in the community. These methods often target redundant sampling steps in diffusion models. For instance, advanced sampling strategies like DDIM and DPM can achieve similar results in fewer steps compared to DDPM, while DeepCache shares features between adjacent steps. Our approach, however, emphasizes a different dimension—simplifying the model structure, which typically necessitates additional fine-tuning. Specifically, we prune the number of tokens in a single sampling step, and thereby cannot circumvent finetuning. As diffusion models evolve, the number of sampling steps decreases, reducing the scope for optimization within sampling steps. We think that the significance of efficient model structures will increase.

Fine-tuning is widely recognized as an essential method for achieving good performance-efficiency diffusion models. Many common methods often require additional fine-tuning. For example, standard RF [1], consistency models [2], and other sampling step distillation [3] techniques all necessitate this process. Leading open-source T2I model Flux [4], and T2V model HuanyuanVideo [5] both employ additional distillation for efficiency gains. Consequently, we argue that minimal fine-tuning for enhancing efficiency is justifiable in practical applications.

Moreover, the overhead associated with fine-tuning in our approach is reasonable. We require only 6% of the training iterations. For instance, DiT-XL/2 necessitates just 400K iterations for fine-tuning (about 1.5 dayes on 8 A100 GPUs), compared to 7,000K iterations for full training, achieving comparable efficacy to recent works [6, 7, 8].

Q2: Lack deeper analysis of sparse/dense tokens.

Thank you for pointing out this crucial aspect, which warrants further exploration.

The differentiation between global and local tokens arises from observations documented in Figure 1, where some tokens correspond to larger areas of images, while others focus on smaller regions. Our method harnesses and amplifies this phenomenon by directing certain layers to generate global tokens, while others preserve their original representation. To substantiate the effectiveness of our approach, we extract the attention map with dimensions N$\times$ N, where N is the number of tokens of an H$\times$W image. By reshaping the attention map to $N\times H \times W$, each $H\times W$ heatmap is the receptive field of the corresponding token. It is evident that global token heatmaps emphasize larger regions, presenting more precise and meaningful semantic information, whereas local token heatmaps concentrate on smaller areas. I apologize for OpenReview's current constraints with image uploads, preventing us from displaying these visualizations. Instead, I provide quantitative results, calculating normalized variances of heatmaps using the method from Figure 1b, detailed in the table below:

Note: Values are normalized to a 0-1 scale, aligned with Figure 1b. Higher variance indicates a local property, while a lower variance suggests a global property. This table demonstrates that global and local tokens effectively capture global and local information, respectively. We intend to include visualization results in the next version of our paper.

Reference

[1] Liu, X., Gong, C., & Liu, Q. (2022). Flow straight and fast: Learning to generate and transfer data with rectified flow.

[2] Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). Consistency models.

[3] Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024). One-step diffusion with distribution matching distillation. computer vision and pattern recognition (pp. 6613-6623).

[4] FLUX.1: Redefining Text-to-Image AI with Superior Visual Fidelity.

[5] Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., ... & Zhong, C. (2024). Hunyuanvideo: A systematic framework for large video generative models.

[6] You, H., Barnes, C., Zhou, Y., Kang, Y., Du, Z., Zhou, W., ... & Lin, Y. C. (2025). Layer-and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers. Computer Vision and Pattern Recognition Conference (pp. 18072-18082).

[7] Zhao, W., Han, Y., Tang, J., Wang, K., Song, Y., Huang, G., ... & You, Y. (2024). Dynamic diffusion transformer. ICLR.

[8] Yuan, Z., Zhang, H., Pu, L., Ning, X., Zhang, L., Zhao, T., ... & Wang, Y. (2024). Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems, 37, 1196-1219.

To Reviewer pfYM

Thank you for your thoughtful and encouraging feedback. We are delighted that you found value in our empirical and conceptual analysis.

Q1: Temporal sparsification design.

Thank you for this insightful question. We will address it from three perspectives:

1. Theoretical support to timestep-wise pruning rate. The rationale behind the pruning schedule across different timesteps is detailed in lines 56-58 of our paper. Figure 1b shows that attention variance increases as the denoising process progresses, highlighting a growing emphasis on local information at lower-noise stages. As the denoising proceeds, the model increasingly focuses on local details, dynamically adapting to heightened demand for detail in the later stages. Consequently, we employ larger pruning rates ($r$) in the early stages and smaller rates in the later stages.

2. The choice of T/4. This decision is primarily informed by our experimental findings. We discuss pruning rates in Table 7, and lines 333-334 highlight that the performance of the dynamic pruning strategy is particularly impacted by the token count in the later stages of denoising. Our experiments reveal that performance correlates most strongly with the last T/4 timesteps. Choosing T/3 would entail higher computational costs for tiny gains, and choosing 5/T would result in a noticeable performance decline. T/4 provides the optimal performance-efficiency trade-off.

3. Linear interpolation chosen. Our ablation study, referenced in lines 328-336, provides insights into this decision. Experiments indicate that the performance of the dynamic pruning strategy is particularly sensitive to the token count in the later stages of denoising. Our method is less sensitive to $r$ during early stages. Thus, more complex interpolation and scheduling strategies did not result in significant improvements. We opted for simple linear interpolation to maintain a gradual increase in $r$.

We acknowledge that our paper may not fully convey the rationale behind various choices for our timestep-wise pruning rate strategy, and we will enrich the relevant sections in future revisions.

Q2: Limited baseline comparisons.

Thank you for pointing out the need for more comprehensive baseline comparisons. We have focused on comparing our method with ToMeSD and DyDiT because they provide a standardized baseline using DiT. Additionally, DyDiT is recognized as a state-of-the-art technique in this domain. Other methods utilize different base models, and even when employing DiT, they may not adhere to the standard and optimal results from the DiT paper (e.g., FID score 2.27 on $256\times 256$ ImageNet and FID score 3.04 on $512\times 512$ ImageNet), which complicates achieving a perfectly fair comparison. Below, I outline comparisons with nearly all relevant baselines:

1. Compare with TokenCache [1]. | Method |Base Model|FID Score|Speed Up| |:---:|:---:|:---:|:---:| | TokenCache | DiT-XL/2 | 2.37 (+0.13) | x 1.32 | | Ours | DiT-XL/2 | 2.38 (+0.11) | x 1.87 |

We adopt the standard DiT baseline 2.27, while TokenCache applies a baseline 2.24. TokenCache achieves a 32% speed gain with a slight FID increase. Our approach attains an 87% speed gain with a smaller FID increase, demonstrating superior efficiency and performance.

2. Compare with Ditfastattn [2]. | Method | Base Model | FID Score | IS | Speed Up | | :---:| :---: | :---:|:---:| :---:| | Ditfastattn | DiT-XL/2 (512x512) | 4.52 (+1.36) | 180.34 | x 1.98 | | Ours | DiT-XL/2 (512\x512) | 2.96 (-0.08) | 242.4 | x 2.45 |

We adopt the standard DiT baseline 3.04, while TokenCache applies a baseline 3.16. Ditfastattn shows a substantial FID increase for speed gain, while our method results in better speed and less FID.

3. Compare with DiffCR [3]. | Method | Base Model | FID Score | Speed Up | |:---:| :---:|:---:|:---:| | DiffCR | PixArt-sigma | 10.68 (-1.27) | x 1.26 | | Ours | PixArt-alpha | 4.29 (-0.24) | x 1.69 |

While DiffCR achieves a greater performance improvement (1.27 vs. 0.24), our results are based on a more challenging baseline of 4.53 compared to their 11.95. The marginal effect implies that enhancing performance from 4.53 is more challenging than from 11.95. Additionally, our method provides a significantly higher speed improvement than theirs.

4. Compare with ToMeSD and DyDiT.

(1) Quantitative comparisons

We include quantitative comparisons on class-conditional image generation on ImageNet in Appendix. ToMeSD, being a workshop paper, does not demonstrate high effectiveness, whereas DyDiT delivers substantial results. Therefore, I mainly compare our results with DyDiT. For the text-to-image task, we both employ PixArt-$alpha$ as our base model. DyDiT uses a smaller dataset, COCO, while we utilize a more complex dataset, SAM. The results are:

Our SparseDiT demonstrates better performance improvement and speed up in the text-to-image task.

(1) Quantitative comparisons

Due to OpenReview's limitations on image uploads, we assess visual quality using ChatGPT's evaluation comparing our SparseDiT with ToMeSD and DyDiT:

ToMeSD (Model A) v.s. SparseDiT (Model B)

ChatGPT assessment:

Model B is a far more powerful, capable, and useful generative model than Model A.

Profile of Model B (Bottom Set):

Technically Superior: It consistently produces high-resolution, sharp, and richly detailed images.

Aesthetically Driven: It demonstrates a sophisticated understanding of composition, lighting, and color theory to create beautiful, engaging results.

Idealized Generation: It excels at capturing the core essence of a concept (e.g., the eruption of a geyser, the majesty of a mountain) and presenting it in its most ideal and visually striking form.

Profile of Model A (Top Set):

Resembles a "Raw Database": Its output mirrors the varied quality and style of a large, unfiltered image dataset scraped from the web, containing both good and bad examples.

Occasional Serendipity: Due to its less guided nature, it can sometimes produce unique and narratively rich images (like the recumbent parrot) that a more polished, aesthetically-driven model might not. For any practical application requiring high-quality, visually appealing, and reliable image generation, Model B is the clear and definitive winner.

DyDiT (Model A) v.s. SparseDiT (Model B)

Both our Appendix and DyDiT's provide numerous visualizations, with our images presented at a resolution of $512\times 512$ in Figures 4-15 and DyDiT's at $256\times 256$ in Figures 9-22. Despite the difference in resolution, the comparison remains valid as both utilize the same training dataset. While both sets of results exhibit high generation quality, SparseDiT images display a more precise structure in certain cases compared to DyDiT.

ChatGPT assessment:

Model A (left) is a model that strives for ultimate aesthetics. It will create the most stunning, idealized, and dramatic images, sometimes at the cost of coherence. Its "ceiling" is very high, capable of producing breathtaking results. However, its "floor" is low; the artifact in the parrot image proves that it can fail in certain cases and cannot be fully trusted.

Model B (right) is a model that prioritizes realism and robustness. Its output is more like a faithful reproduction of high-quality photographs from the real world. Its aesthetic "ceiling" might not be as stunning as Model A's best shots, but its "floor" is very high—it rarely, if ever, produces obvious errors, making its output highly reliable and consistent.

Q3 Related work inclusion

Several optimization strategies mentioned in our related work. We have listed some comparisons in Q2. Some methods, like flow-based methods and timestep distillation methods focus on reducing or optimizing timesteps to enhance efficiency, whereas our method centers on pruning model structure and reducing FLOPs within a single timestep. Our method and their techniques improve diffusion model efficiency along orthogonal dimensions and should be considered complementary rather than directly comparable. In Table 5, we have shown that our method can be integrated with advanced sampling methods utilizing fewer timesteps. Additionally, we present another experiment to showcase its combination with DeepCache:

If there are specific papers or methods you believe warrant further comparison, I would appreciate you point out them and we will do further comparisons.

Limitations

(1) For hyperparameter selection details, please refer to my response to Reviewer kysk Q2.

(2) I provide an analysis of ToMeSD in our Appendix. Unlike ToMeSD, which directly adapts motivation and techniques from vision transformers, our approach involves a detailed observation of DiT's behavior in generation tasks and a comprehensive redesign of all modules accordingly.

Reference

[1] Lou, J., Luo, W., Liu, Y., Li, B., Ding, X., Hu, W., ... & Ma, C. (2024). Token Caching for Diffusion Transformer Acceleration. arXiv preprint arXiv:2409.18523.

[2] Yuan, Z., Zhang, H., Pu, L., Ning, X., Zhang, L., Zhao, T., ... & Wang, Y. (2024). Ditfastattn: Attention compression for diffusion transformer models. NeurIPS, 37, 1196-1219.

[3] You, H., Barnes, C., Zhou, Y., Kang, Y., Du, Z., Zhou, W., ... & Lin, Y. C. (2025). Layer-and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers. CVPR (pp. 18072-18082).

[4] G. Fang, X. Ma, and X. Wang. Structural pruning for diffusion models. NeurIPS, 2023.

To Reviewer kysk

Thank you for your valuable feedback and for highlighting the strengths of our work. We are delighted that you found our observations regarding early-layer attention maps in DiT insightful and supportive of the use of average pooling. We appreciate your recognition of the comprehensive validation of SparseDiT across various benchmarks and your acknowledgment of the experiments as solid. Furthermore, your observation of consistent performance gains across different model sizes indicates the scalability potential of our approach.

We are committed to addressing each of your inquiries and suggestions thoroughly in the upcoming revisions. Your feedback is instrumental in enhancing the quality of our research.

Q1: Training from scratch.

Thank you for your insightful observation. Our method indeed can be trained from scratch; however, leveraging a well-pretrained model offers significant advantages, aligning with our research goal of enhancing the efficiency of well-trained diffusion models by pruning tokens. Unlike specialized architectural design methods such as Pixart-$\alpha$ [1], our aim is not to develop a new model architecture. We just argue that our findings can inspire future designs. We opted for fine-tuning rather than training from scratch for two primary reasons:

1. Computational Cost: Training diffusion models from scratch, such as our baseline model, DiT-XL/2, requires 7,000K iterations, approximately two months on A100 GPUs. While our training speed surpasses the original DiT, it still demands about a month of processing, which is not feasible for us. Especially considering more complex T2I models, their computational cost will be more tremendous.

2. Practical Application: From a practical standpoint, an algorithm that requires training from scratch holds less value compared to one that can be fine-tuned. Fine-tuning techniques are increasingly common and effective. Leading open-source foundation generation models such as WanX2.1, HunyuanVideo, and Flux are built on traditional DiT architectures and derive their efficient variants through fine-tuning. Comparable works [2, 3, 4] also employ fine-tuning or training-free strategies without training from scratch.

I understand the importance of assessing our model's capability and the dependency of pre-trained weights. We have trained our model from scratch, although due to time constraints, only reaching 400K training iterations during the rebuttal period. A complete training cycle like DiT requires 7,000K steps, roughly equivalent to one month of computing. We compare our result with original DiT paper in Table 4. The results (cfg=1.0) are as follows:

These results indicate our method can be trained from scratch, achieving strong performance without dependence on pre-trained dense model weights. Nevertheless, we believe our method’s fine-tuning capability possesses more practical significance.

Regarding comparisons with distillation techniques, diffusion models commonly use timestep distillation, with some methods claiming single-step inference [5, 6, 7]. Unlike these, our approach focuses on model pruning to reduce computational costs in one step. Our method and distillation techniques improve diffusion model efficiency along orthogonal dimensions and should be considered complementary rather than directly comparable. For a one-step distillation method, they can reduce the timestep from 100 to 1, which saves 99% FLOPs, whereas achieving a 99% reduction in FLOPs through model structure compression alone is nearly impossible. To our knowledge, related works also do not compare their results with distillation methods.

Q2: The selection of the model's hyperparameters.

1. We have conducted ablation studies, as shown in Tables 4 and 6, to demonstrate the selection of several key hyperparameters. Regarding the number of PoolingFormer layers, Figure 1 indicates that the attention maps from the first two transformer layers display a uniform distribution. Consequently, one or two PoolingFormer layers suffice, as corroborated by Table 4. In the SDTM architecture, one transformer is designated for generating sparse tokens, another for recovering dense tokens, and one is dense transformer. These three transformers remain fixed. We provide an additional ablation study in the below table to verify it. The number of sparse transformers can be adjusted, with more sparse transformers yielding more efficient models. Performance remains stable as long as the number of sparse tokens does not exceed three. The final dense block should contain no more than three transformers to ensure efficiency, with additional transformers having no significant impact on performance. | No. of sparse transformer | FID | FLOPs | | :---: | :---: | :---: | | 3 | 2.38 | 68.05G | | 4 | 2.76 | 69.41G | | 7 | 3.89 | 65.67G |
2. The rationale for the pruning schedule across different timesteps is detailed in lines 56-58. In Figure 1 b, we find that attention variance increases as the denoising process advances, indicating a growing emphasis on local information at lower-noise stages. As denoising progresses, the model increasingly prioritizes local details, dynamically adapting to heightened demands for detail at later stages. Therefore, we adopt larger pruning rate at early stages while smaller pruning rate at later stages. We discuss how to set the pruning rates in Table 7. Our method is not sensitive for the pruning rates beside the pruning rate of the last stage. As said in line 333-336, the performance of the dynamic pruning strategy is particularly relative with the token count of the later stages of denoising. For instance, the configuration with token counts ranging from $4\times 4$ to $8\times 8$ only shows a 0.02 FID score increase compared to the configuration with a constant $8\times 8$ token count. In the practical application, we recommend a $10\times 10$ token count in the final stage for optimal performance, and a $4\times 4$ token count in the first stage for efficiency. The intermediate stages can be set to $6\times 6$ and $8\times 8$, with the method displaying robustness to these settings.

I will reorganize the content above and include it in our supplementary material to clearly assist readers in selecting the hyperparameters.

Q3: Training stability.

Despite our model's non-uniform architecture, it demonstrates remarkable training stability. PoolingFormer structures are inherently stable, as evidenced by [8]. Our SDTM architecture alternates between sparse and dense structures, with dense transformers ensuring stability. The final top block comprises standard transformers, which naturally exhibit stability. Throughout extensive experiments, we have encountered minimal issues concerning training stability. The sole instance of instability arises when no PoolingFormer layers are used, causing a sudden shift to sparse tokens in the initial layer. In training DiT-XL/2 from scratch, we employed a singular global learning rate, achieving rapid convergence. The training loss decreased from 0.0308 at 100 iterations to 0.0120 at 10,000 iterations, affirming the model's stability.

Reference

[1] Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., ... & Li, Z. (2023). Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis.

[2] You, H., Barnes, C., Zhou, Y., Kang, Y., Du, Z., Zhou, W., ... & Lin, Y. C. (2025). Layer-and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers.

[3] Zhao, W., Han, Y., Tang, J., Wang, K., Song, Y., Huang, G., ... & You, Y. (2024). Dynamic diffusion transformer.

[4] Bolya, D., & Hoffman, J. (2023). Token merging for fast stable diffusion.

[5] Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024). One-step diffusion with distribution matching distillation.

[6] Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). Consistency models.

[7] Xie, S., Xiao, Z., Kingma, D., Hou, T., Wu, Y. N., Murphy, K. P., ... & Gao, R. (2024). Em distillation for one-step diffusion models.

[8] W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng, and S. Yan. Metaformer is actually what you need for vision.

Dear Reviewers gHMa, pfYM, and rhnM,

As we're approaching the end of author-reviewer discussion period, please read the rebuttal and start discussion with the authors as soon as possible. If all your concerns have been addressed, please do tell them so. Please note that submitting mandatory acknowledgement without posting a single sentence to authors in discussions is not permitted. Please also note that non-participating reviewers will receive possible penalties of this year's responsible reviewing initiative and future reviewing invitations.

Thanks,

AC