We deeply thank Reviewer BEVg for the valuable comments and are glad that the reviewer finds our analysis thorough, evaluations comprehensive, and methods effective. We would like to address the concerns as below.

1. (W1A: Whether all the methods are evaluated on distilled models from FLUX) In Table 2, are the different efficient attention methods also evaluated based on distilled models from FLUX-1.dev with their attention layers replaced accordingly?

Thanks for the good question. Yes. All the attention methods are implemented and compared on FLUX-1.dev. We will clarify this in the revision.

2. (W1B: Approximation based on Swin Transformer) Additionally, since both Swin attention and CLEAR adopt local attention, differing mainly in window shape, is it theoretically possible to adjust Swin’s window size or shape to approximate CLEAR’s performance?

Thanks for the insightful question. In our initial exploration, we have indeed tried to adjust the window size of Swin Transformer. Unfortunately, we find that it still results in the "grid" artifact, similar to Fig. 3. We speculate that it is because of the non-overlapped window partition for each attention layer.

Nevertheless, we indeed find an approximated form between CLEAR and Swin Transformer. The key point here is to apply overlapped window partition instead. Specifically, CLEAR can be viewed as a form of local window attention with stride=1. We show in the following study that stride=2 achieves comparable performance.

This property could be useful for hardware optimization. Thanks again to the reviewer for introducing a promising direction.

3. (W2A: The source of the high-resolution capacity) Is the model’s ability to generate 8K-resolution images primarily distilled from FLUX-1.dev, or is it achieved through additional data-specific fine-tuning?

Thanks for the valuable question. The model’s ability to generate 8K-resolution images comes from the ability of training-free resolution extrapolation of FLUX. In fact, it requires two steps to achieve efficient high-resolution generation. The first one is to adapt the original DiT to make it effective at higher scales. The second is to make it more efficient. CLEAR mainly addresses the second step.

4. (W2B: The common issue of repeated content or artifacts) Moreover, how does the method address the common issue of repeated content or artifacts in high-resolution image generation?

Thanks for the insightful question. In fact, as shown in Line 311, we apply the SDEdit algorithm for high-resolution generation. Specifically, we first generate an image at the native resolution scale of the diffusion model. Then, we resize it to a larger size and add a certain noise to it. A noise scale of 0.7 is adopted empirically in our experiments. Starting from this point, the model conducts the remaining denoising steps. In this way, the original image structures are preserved and low-level details are refined. In other words, the issue of repeated content is suppressed by skipping the initial denoising steps corresponding to the overall image layouts. We include the following study to verify its effectiveness.

Moreover, as shown in Appendix E and Fig. 13, it is convenient to build CLEAR upon more advanced methods of training-free high-resolution generation, since our method is orthogonal to them. We will enhance the clarity of this part in the revision.

5. (W3A: Ablation studies on the individual contributions of the two additional loss terms) The ablation study appears insufficient, as the individual contributions of the two additional loss terms introduced (prediction loss and attention loss) are not separately verified.

Thanks for pointing this out. In fact, as shown in Lines 254~255, the two loss terms and the corresponding loss weights adopt the default setups in [30] and [37], which also focus on architectural distillation of diffusion models. Nevertheless, we are willing to add the corresponding studies below.

We find that the performance is overall insensitive to the hyper parameters. Too small values may result in insufficient effects of these loss terms, while too large values would not benefit performance, either.

6. (W3B: Generalizability across different 𝑟) Also, does each choice of attention radius 𝑟 require training a separate model, or can one model generalize across different domain sizes?

Thanks for pointing out an interesting experiment. Qualitatively, we find that the model trained with a smaller 𝑟 can be used for larger 𝑟. However, the opposite case does not work. We speculate that it is because the information provided for the model is complete for inference with additional clues when 𝑟 is larger, while the information would be insufficient if 𝑟 becomes smaller.

We supplement the quantitative results of GPT evaluation below.

Dear Reviewer BEVg,

We would like to genuinely thank the reviewer for the time and attention in reviewing and acknowledging our rebuttal.

We sincerely hope to know if there are any further questions or points requiring clarification, and we would be more than happy to provide any additional information or engage in further discussion. The time and effort the reviewer has dedicated to our submission are greatly appreciated.

Best regards, Authors of Submission 3487

We would like to thank Reviewer BEVg sincerely for the active engagement in the author-reviewer discussion and the further questions. We have the following opinions regarding them:

• W3A. While the authors provided a grid search over the loss hyperparameters, the individual contributions of each loss term remain unclear. Specifically, it would be helpful to see a more detailed analysis regarding which loss is more critical and whether the two losses are complementary or potentially redundant. In addition, the authors adopt the default hyperparameters from references [30] and [37], but it is not discussed whether these values are well-suited to the current task.

  Thanks for the valuable follow-up questions. In general, both loss terms contribute to the consistency between the distilled and original models. Nevertheless, we indeed find they are complementary:

  • $\alpha$ controls the weight of $L_{pred}$, regulating the consistency between the final output results, which, according to our experiments shown in the grid search, appears to be relatively more effective than $L_{attn}$. For example, the cases of $\alpha>0,\beta=0$ generally yield better FiD scores than those of $\alpha=0,\beta>0$.

  • $\beta$ controls the weight of $L_{attn}$, regulating the consistency between the intermediate attention results, which, according to our experiments, serves as an auxiliary measure to facilitate the training convergence. To verify this, we present the values of the flow matching loss $L_{fm}$ after moving average at various steps in the training progress:

    Training Steps	10	100	1000	2000	10000
    w/o $\mathcal{L}_{attn}$	0.512	0.494	0.487	0.438	0.358
    w $\mathcal{L}_{attn}$	0.396	0.371	0.362	0.351	0.347

    Results indicate that $L_{attn}$ can facilitate training by offering informative intermediate supervision signals, beyond the relatively late supervision provided by $L_{pred}$ at the final outputs. We will include an illustrative presentation in the revision.

  Overall, applying both terms achieves the best performance, and the performance is insensitive to the specific values of the hyperparameters according to the grid search. There can be some subtle trade-off between the FiD and CLIP-T metrics, where the default values achieve a good balance.

• W3B. It also remains unclear whether, in practical deployment, the model needs to be retrained for each target resolution, or if a single trained model can generalize effectively across different resolutions. Further clarification on this point would strengthen the applicability claims of the proposed method.

  Thanks for the good question! The model does not need to be retrained for each resolution. We only conduct training on the native resolution of FLUX, i.e., 1024x1024. As mentioned in our response to W2A, the cross-resolution generalization of FLUX is achieved by existing techniques of training-free resolution extrapolation. Our models, as the corresponding distilled models, inherit this capacity to be deployed for various resolutions.

We will definitely enhance the clarity regarding these questions in the revision. Sincere hope is that our response would alleviate the reviewer's concerns. Look forward to any further feedback.

Thank you for the authors’ response, which has addressed my initial concerns. However, I remain cautious about the claimed novelty and practical effectiveness of the CLEAR local attention mechanism.

As such, I will retain my current rating. Good luck.

We would like to thank Reviewer BEVg for the follow-up comments. We are glad to know that our previous response has fully addressed the initial concerns.

Here, we would like to respectfully reiterate the novelty and practical effectiveness of our work:

1. First to linearize the complexity of pre-trained DiT. To the best of our knowledge, we are the first to explicitly focus on and tackle the problem of linearizing the complexity of already pre-trained Diffusion Transformers (DiTs).
2. Systematic study of efficient attention. We conduct a systematic survey of various efficient attention mechanisms, and—through extensive experimentation—identify four necessary conditions that a feasible linearization method should satisfy.
3. Non-trivial properties of linearized DiT. We further explore practically useful benefits of the linearized DiT, including:
   • acceleration in high-resolution image generation,
   • cross-model and cross-component generalization, and
   • enhanced parallelization efficiency.

While our final design shares certain similarities with Neighborhood Attention, it emerges from a long process of trial-and-error exploration over numerous efficient attention variants. This search process leads us to distill useful guidelines that can be informative for future works. We sincerely hope this aspect could be taken into account. Moreover, as detailed in both the main paper and rebuttal, we have compared our method with Neighborhood Attention, showing that ours preserves performance while delivering higher efficiency and the additional benefit of rotation robustness.

Regarding practical effectiveness, we would like to clarify that the showcases in the paper are not cherry-picked. Instead, the test cases are randomly generated by GPT. Our method is indeed effective in practice and delivers substantial acceleration in high-resolution scenarios—this is quantitatively demonstrated across all experimental tables in the paper. If there are any specific concerns about practical effectiveness, we would be more than happy to address them during the remaining author–reviewer discussion phase, as we remain fully committed to resolving any such issues.

We truly appreciate the time and consideration of Reviewer BEVg, and we look forward to the feedback.

We appreciate Reviewer a9dc's thoughtful comments and are glad that the efficiency of our work for large images and the analysis of the 4 features, except for locality, are recognized. We would like to address the concerns as below.

1. (W1: Local attention may not be effective in dealing with long dependencies in images) The analysis of the importance of locality is questionable. Perturbing an image of "A small blue plane sitting on top of a field" may not get the conclusion that local features are more important, because this specific prompt and image itself does not have long dependency. An good example to demonstrate locality should be the one in Figure 4. This image shows strong long dependency. If locality is also more important here, we can thus conclude. In particular, in Figure 4, local attention seems not to make any better sense than the global attention.

Thanks for the insightful review. We would like to rephrase the reviewer's concern as "local attention may not be effective in dealing with long dependencies in images". Regarding this, we have the following opinions:

• No matter whether there is long dependency or not, we find that locality is a necessary condition. For the case of no long dependency, we provide examples in Figs. 3 and 5. For the case of long dependency, we supplement the GPT and GenEval scores below, which focus on evaluating the capability of handling the holistic layouts in generated images.

  	Aesthetic	Prompt Alignment	Overall	Win Rate vs Other	GenEval
  Strided Attn.	67.20	78.85	68.81	0.98	0.433
  CLEAR Attn.	89.62	92.13	88.52	-	0.674

​ The results suggest the importance of locality in the case of long dependency.

• Since images are locally continuous by nature, local attention is crucial to preserve this property. For long dependency, the overall receptive field of the network can be enlarged with a multi-layer structure of the DiT, a feature similar to CNN.

• We fully agree with the reviewer that using an image with a long dependency is helpful for the conclusion. We will use another sample to reflect this in the revision.

2. (W2: Circular window vs square window) The improvement compared with Neighborhood Attention seems questionable. Although the GFLOPS decrease by 12.5%, according to Figure 2, there is only a very limited improvement in real wall-time speed.

Thanks for the good point. As mentioned in Lines 321~325, the standard Neighborhood Attention and the proposed method with circular windows yield comparable performance. Given that the FLOPS can be reduced largely, we apply the latter form. Furthermore, as shown in the following study, it is indeed more efficient in terms of wall-clock time, especially at higher resolutions.

3. (W3: Improvement over Swin Transformer) The proposed method seems underperform Swin Transformer given the same GFLOPS.

Thanks for the comment. In fact, as mentioned in Fig. 3, the Swin Transformer's strategy, which lacks high-rank attention maps, results in "grid" artifacts and fails to generate visually coherent results. Also, as shown in Tab. 2, our method outperforms Swin Transformer for most metrics with less GFLOPS. We further validate the superiority of the proposed solution over Swin Transformer via the following study on the GPT scores and GenEval benchmark:

4. (Q1: Advantage to neighborhood attention) Compared with Neighborhood Attention, what is the wall-time speed improvement? 2., Except for the GFLOPS, is there any other advantages over Neighborhood Attention? Can Neighborhood Attention also be used for DiT Plugins, and Multi-GPU Parallel Inference?

Thanks for the good questions. We kindly refer the reviewer to our response to W1 for the wall-time speed. Indeed, Neighborhood Attention in principle equips with the features explored in the manuscripts. We would like to emphasize that the main contribution of the manuscript is to identify the four key factors for successful linearization of pre-trained DiTs, listed in Tab. 1, instead of introducing a brand-new attention strategy. The initial motivation of circular windows is that they can reduce the computational overhead without hurting the performance, which is a free-launch benefit.

Moreover, we indeed find that circular windows can offer an additional advantage: they are potentially more robust to image rotation, given that circles are rational invariance, which is not the case for square windows. To validate this property, we send original images to the network and get the outputs. Then, we rotate the original images by 60 degrees, send them to the network, and rotate the outputs back. The average SSIM between the two outputs are shown below.

5. (Q2: FID against real images in Table 2) In Table 2, why increasing radius significantly improves the FiD against original FLUX-1.dev, while has no almost no impact against real images? What are the vital changes between them?

Thanks for the insightful question. The reason behind this is that the model is distilled from the original FLUX. Increasing the radius can result in a smaller architectural gap with the original model, which is reflected in the FiD scores against original FLUX-1.dev. For the scores against real images, as shown in Tab. 2, the FLUX-1.dev itself has a relatively large FiD, indicating that images generated by the teacher model do not strictly follow the distribution of COCO images, which also affects the distilled models.

We would like to thank Reviewer a9dc again for the in-depth reviews. We would definitely love to further interact with the reviewer if there are any further questions.

We sincerely thank Reviewer JWRk for the valuable feedback on the manuscript and are very excited that the reviewer finds the comparative analysis valuable, the experiments comprehensive and convincing, and the method highly reproducible and potentially practical. The questions are addressed below.

1. (W1: Circular window vs square window) The proposed circular local attention mask offers limited novelty compared to existing square-shaped attention masks. Empirically, the observed improvements are marginal, which raises questions about the necessity of the circular design.
2. (Q1: Circular window vs square window) Could the authors further elaborate on the advantages of the proposed circular attention mask? While the design introduces a novel geometric locality, its empirical benefits over conventional square masks (e.g., those used in Neighborhood Attention) appear marginal based on current results. A clearer justification—whether theoretical (e.g., isotropy, boundary continuity) or task-specific—would help clarify the necessity and value of this design.

Thanks for the good points. We would like to clarify that the main contribution of the manuscript is to identify the four key factors for successful linearization of pre-trained DiTs, listed in Tab. 1, instead of introducing a brand-new attention strategy. Local attention based on circular windows is one feasible strategy following the four factors, and it reduces the computational overhead of square windows without hurting the performance, which is like a free lunch. That is the reason we adopt this form.

Moreover, we indeed find that circular windows can offer an additional advantage: they are potentially more robust to image rotation, given that circles are rational invariance, which is not the case for square windows. To validate this property, we send original images to the network and get the outputs. Then, we rotate the original images by 60 degrees, send them to the network, and rotate the outputs back. The average SSIM between the two outputs are shown below.

3. (W2: Technical details of high-resolution generation) While the paper discusses extending the method to ultra-high-resolution image generation (e.g., 2K and 4K) using SDEdit, NTK rotary embeddings, and disabling resolution-aware dynamic shifting, the technical details of this pipeline are insufficiently explained. Moreover, the paper lacks ablation studies isolating the impact of these components. Given that CLEAR's claimed speedup at ultra-high resolutions is one of its major selling points, the underexplored treatment of this aspect undermines the strength of the argument.

4. (Q2: Technical details of high-resolution generation) Can the authors provide more implementation details and design considerations regarding the application of their method to ultra-high-resolution generation? The paper mentions several techniques (e.g., SDEdit, NTK rotary embedding, disabling resolution-aware dynamic shifting) used to enable high-resolution generation, but these are not described in sufficient depth. A more detailed technical discussion would improve the clarity and reproducibility of the proposed pipeline.

Thanks for the insightful comments. In fact, it requires two steps to achieve efficient high-resolution generation. The first one is to adapt the original DiT to make it effective at higher scales. The second is to make it more efficient. CLEAR mainly addresses the second step. For the first step, the techniques are mainly adapted from previous works, e.g., SDEdit and NTK rotary embedding. That is why we do not include so many details about them.

Nevertheless, we fully agree with the reviewer that it would be helpful to supplement the related details. We include some key information here and will further elaborate on it in the revision.

1. SDEdit is a simple yet effective training-free method for image editing based solely on a pre-trained text-to-image diffusion model, which can be used for high-resolution generation in a coarse-to-fine manner. Specifically, we first generate an image at the native resolution scale of the diffusion model. Then, we resize it to a larger size and add a certain noise to it. A noise scale of 0.7 is adopted empirically in our experiments. Starting from this point, the model conducts the remaining denoising steps. In this way, the original image structures are preserved and low-level details are refined.
2. NTK rotary embedding applies a scaling factor $s$ to the rotary base $b$ used for rotary positional embedding, i.e., $b'=bs$, where $s=\sqrt{\frac{L_{cur}}{L_{native}}}$, and $L_{cur}$ is the current sequence length while $L_{native}$ is the native length seen during the training time.
3. Resolution-aware dynamic shifting applies a factor $s$ to the projection function of the current denoising time step $t$ for the flow matching scheduler: $t'=\frac{st}{1+(s-1)t}$, where $s$ is positively related to the image resolution. When this is enabled, at a high resolution, the projection function would appear too "skew", i.e., the number of denoising steps allocated to the stage of high noise level is insufficient.

5. (Q3: Ablation studies on high-resolution generation settings) Could the authors include ablation studies specifically focused on ultra-high-resolution settings? Since CLEAR’s improved efficiency at 2K–4K resolutions is one of its key claims, it would be valuable to see component-wise evaluations (e.g., NTK rotary embedding, disabling resolution-aware dynamic shifting) at these resolutions to understand which factors contribute most to quality and speed in this context.

Thanks for the good questions. Regarding the three factors demonstrated in our response to the previous question, we conduct the following GPT evaluation.

Through the results, we can observe that SDEdit mainly helps the robustness of the overall contents, and NTK rotary embedding disabling dynamic shifting mainly benefits details and aesthetics, especially for 4K images. We will include qualitative comparisons in the revision.

We would like to express our sincere gratitude to Reviewer SrwR for the constructive comments and are happy that the reviewer finds the four key factors for successful linearization of pre-trained DiTs and the introduced convolution-like local attention strategy effective. We would like to address the concerns and questions reflected in the review below.

1. (W1: Circular window vs square window) The convolution-like local attention strategy employs circular windows instead of a square that seems not indispensable. Although the FLOPS of a circular neighborhood is ∼ π/4× that of a square neighborhood, but more data operations are required, and they achieve comparable performance.

Thanks for the valuable point. Indeed, circular or square windows do not fundamentally affect the performance and the linearization properties shown in the manuscript. The main conclusion of the manuscript is on the four key factors for successful linearization of pre-trained DiTs, listed in Tab. 1. The standard neighborhood attention, which follows these four principles, is definitely effective, and so is the introduced method based on circular windows. Indeed, they yield comparable performance as mentioned in Lines 321~325. Given that the FLOPS can be reduced largely while preserving the performance, which is a free-lunch benefit, we apply the latter form. As shown in the following study, it is indeed more efficient in terms of wall-clock time, especially at higher resolutions.

Meanwhile, we find that circular windows can potentially be more robust to image rotation, given that circles are rational invariance, which is not the case for square windows. To validate this property, we send original images to the network and get the outputs. Then, we rotate the original images by 60 degrees, send them to the network, and rotate the outputs back. The average SSIM between the two outputs are shown below.

2. (W2: The high-rank attention maps and feature integrity) Among the four key elements, the high-rank attention maps and feature integrity seem not notable in the convolution-like local attention strategy.

Thanks for the comment. In fact, the proposed convolution-like local attention strategy are adhere to these two principles:

• On the one hand, each query token has a different set of key and value tokens, which makes the overall attention masks exhibit a high-rank pattern.
• On the other hand, the proposed method does not compress or recompute key and value features in any forms, which ensures their integrity.

These two factors contribute significantly to the image quality, as shown in Fig. 3:

• The Swin Transformer's strategy, which lacks high-rank attention maps, results in "grid" artifacts.
• If we zoom the images, we can find that the PixArt-Sigma's strategy, which lacks feature integrity, yields distorted local details.

As a result, they fail to produce high-quality images. We further introduce the following GPT and GenEval scores to verify the conclusions:

3. (W3: Performance vs window size) The experiment on relation of window size and performance is not presented.

Thanks for the valuable point, indicating an important study. The related experiments can be found in Tab. 2, Tab. 3, and Fig. 8, with various $r$. We will further highlight them in the revision.

4. (Q1: Definition of "convolution-like") The meaning of 'convolution-like' is not clear. What parts are convolution-like?

Thanks for the good question. For each query token, only several local key and value tokens are taken for attention interaction, which is similar to the behavior of a convolution with a fixed-size kernel. We provide an illustration of this paradigm in Fig. 6.

5. (Q2: Optimal window size) What size of the window is optimal? Isn't a larger window size always better?

Thanks for the good questions. The reviewer's insight that a larger window size yields better performance is correct. However, it requires more computational overhead, as mentioned in Fig. 2 and Tab. 2. We empirically find that $r=16$ yields the best trade-off between performance and efficiency and thus adopt this setup by default. We will enhance the clarity of this part in the revision.

We would like to thank Reviewer SrwR again for the valuable feedback. Hope our responses alleviate the reviewer's concerns, and we are happy to answer additional questions if there are.

Dear Reviewer SrwR,

We would like to thank the reviewer again for the thoughtful review of our submission.

Please feel free to let us know if there are any remaining questions or concerns about the manuscript or our rebuttal. We would be glad to provide any further clarification if helpful.

Best regards, Authors of Submission 3487