We thank the reviewer for their positive feedback and are pleased that they appreciate our comprehensive comparison of positional encoding methods for extrapolation in DiTs, the strong empirical results, and the thorough ablation studies, the design of LEDiT framework.

[W1]: The omission of LookHere in baselines and discussions is critical.

Thank you for your valuable comments and constructive feedback. We agree there are some conceptual similarities between LEDiT and LookHere. Both LEDiT and LookHere explore causal attention to enhance length extrapolation. LookHere carefully designs various combinations of causal masks to provide directional inductive biases. It introduces AliBi to penalize attention scores and demonstrates extrapolation improvement. LookHere conduct extensive experiments to demonstrate the extrapolation capabilities in classification tasks.

However, we clarify that there are key differences. Notably, directly adapting LookHere to image generation tasks does not yield effective length extrapolation. We have provided a comprehensive comparison between LEDiT and LookHere in both methodology and performance.

The key differences between LEDiT and LookHere are summarized in the following table:

• Theoretical Contribution: We provide a rigorous theoretical framework that explains why causal attention is capable of encoding positional information and enabling length extrapolation. This not only facilitates effective image generation with robust extrapolation capabilities, but also offers valuable insights into the underlying mechanisms of length extrapolation. In contrast, LookHere does not investigate the reasons behind causal attention's ability to encode positional information; it merely states that attention with 2D masks can "limit the distribution shift that attention heads face when extrapolating" (Page 2).

• Complementary Strategy to Causal Attention & Receptive Field: LEDiT not modify the causal attention mechanism itself. Instead, we introduce multi-dilation convolution as a complementary module to causal attention, ensuring that the global receptive field of causal attention remains unaffected. LookHere incorporates AliBi into causal attention. AliBi can be interpreted as a form of soft window attention, where an attention score penalty increases with distance, inherently restricts the receptive field of causal attention—even when LookHere explores different combinations of causal masks to cover the entire image.

• Is Suitable for Extrapolation in Image Generation: we conduct detailed experiments and observe that LookHere fails to achieve effective length extrapolation in image generation tasks— whereas LEDiT demonstrates robust and reliable extrapolation capabilities.

Detailed Experimental Comparison:

We evaluate all LookHere variants (LH-180, LH-90, LH-45) and compare their performance with LEDiT. We fine-tune all models for 100K steps on ImageNet-256x256 by loading the DiT-XL/2 pretrained weights. Since AliBi is a key component of LookHere, we also report results for AliBi to facilitate a comprehensive discussion. As shown in the table below, LEDiT outperforms both LookHere and AliBi at the training resolution as well as at higher resolutions. We observe that, when extrapolated to higher resolutions, samples generated by LookHere preserve fine image details but suffer from severe object duplication. Since we are unable to provide visualization of generated images, we focus on quantitative comparisons using FID and precision metrics, which primarily reflect the structural accuracy and quality of the generated images. LookHere achieves lower FID and precision scores compared to LEDiT.

Table 1: Out-of-Distribution (512x512) and In-distribution (256x256) comparison between LEDiT and other methods. we report FID-10K for 512x512 and report FID-50K for 256x256

This raises an interesting question: why LookHere demonstrated on image recognition tasks but fails on image generation tasks? We present some insights:

• Consistent Higher Training Loss for LookHere: We report the mean loss over the last 1,000 training steps, as shown in the table below. On the training set, LookHere shows a higher loss compared to LEDiT and Sin/Cos PE, indicating that LookHere is relatively underfitting. Increasing the training steps to 150K does not further reduce the training loss, suggesting that the model has already converged. The persistently high training loss suggests that the LookHere architecture may not be directly applicable to the image generation task.

• Image generation requires more precise positional information than image classification. In image classification, the input is a clear image, and the model can rely on strong image content priors to make predictions. In contrast, image generation starts from pure noise, providing no prior information. Therefore, stronger positional encoding is essential for generation tasks.

• LookHere uses causal attention and AliBi, but these mechanisms are unable to encode precise positional information. As discussed in our main paper, causal attention cannot accurately encode positional information. We also observed that AliBi performs poorly on in-distribution settings, indicating that it does not provide accurate positional information. We believe that while the combination of causal attention and AliBi works for image classification, their ability to capture positional information is limited, which makes LookHere unsuitable for image generation.

• The limited local receptive field of LookHere restricts its extrapolation capability. A global receptive field is crucial for enabling the network to capture global context, which is essential for robust length extrapolation. The causal attention in LEDiT provides access to the entire global context, resulting in strong extrapolation ability. Although LookHere uses directional attention to cover the whole image, it penalizes long-range attention scores, which impairs its ability to model long-distance dependencies. Consequently, LookHere tends to generate repeated objects and exhibits inferior performance during extrapolation.

In contrast, LEDiT combines simple causal attention with multi-dilation convolutions, achieving favorable performance in training resolution settings and demonstrating superior extrapolation ability. Compared to LookHere, LEDiT exhibits better positional encoding capabilities in image generation tasks. Thanks for your insightful comment. We will add the discussion and performance comparison with LookHere in the revised version.

[W2]: Slightly worse than prior methods in-distribution.

Thank you for your insightful comment. We have found that, by appropriately balancing causal attention and self-attention, our method achieves comparable in-distribution performance to RoPE while significantly outperforming existing extrapolation methods. We conducted controlled ablation studies by varying the number of causal attention layers in LEDiT. With only one causal layer, LEDiT significantly outperforms all previous methods at higher resolutions. At the training resolution, LEDiT achieves performance comparable to the widely used RoPE, demonstrating strong in-distribution performance while exhibiting superior extrapolation capabilities. Due to space limitations, we respectfully refer you to our response to Reviewer xXsT's first question for further details.

We thank the reviewer for the thoughtful feedback and are pleased that the reviewer appreciates the theoretical rigor in implicit positional encoding, and superior performance over state-of-the-art methods.

However, we respectfully clarify that the concern raised is based on a misunderstanding of our paper. We have addressed and clarified all relevant points in the following response.

[W1]: The contribution and significance of this paper are rather vague. LEDiT has not achieved SOTA performance.

Let us first clarify the significance of our work: Our main goal is to improve the generative performance of DiT at resolutions higher than their training resolutions, addressing a challenging and impactful out-of-distribution scenario relevant to many real-world applications, such as high-resolution medical imaging and high-definition film production. Current powerful DiT models, such as SD3, FLUX still struggle to generate images beyond the training resolution.

To address this limitation. We introduce a novel Diffusion Transformer without positional encoding (LEDiT) for higher-resolution image generation, departing from the commonly used RoPE-based methods. We provide both theoretical and empirical evidence that causal attention—the core component of LEDiT—implicitly encodes positional information, an that facilitates length extrapolation.

We conduct extensive experiments on both class-conditional and text-to-image generation tasks, demonstrating that LEDiT achieves superior extrapolation performance and outperforms state-of-the-art methods. For example, when extrapolating to 512×512 resolution, LEDiT reduces the previous best FID from 49.86 to 33.25. At 1024×1024 resolution, LEDiT surpasses VisionYaRN (the previous best method) by 17 FID points. We believe our method achieves state-of-the-art extrapolation performance.

It is worth mentioning that the significance of our findings has also been acknowledged by other reviewers, who recognized that our work provide insightful theoritial analysis (Reviewer xXsT, JebD ) and shows compelling improvements in image fidelity (Reviewer xXsT , JebD, f3XA). It was also highlighted by reviewer 3ixD, who considered it quite important to the research community.

[W2]: Why Conv can provide positional information? Does kernel size and layer number affect the positional information?

Previous works [1][2] have demonstrated that convolution with zero padding can leak local positional information, as cited in our main paper. The kernel size does not affect the leakage of positional information. As shown in the table below, the extrapolation performance to 512x512 resolution remains nearly identical across different kernel sizes.

When it comes to layer number, we conducted experiments by incorporating convolutional layers at different depths of LEDiT and found that adding convolutions to the early layers yields better performance than adding them to the deeper layers. We hypothesize that leveraging local positional information extracted by convolutions at earlier stages is more beneficial for diffusion transformer to leakage positional information, thereby leading to improved overall performance.

[1] How much position information do convolutional neural networks encode? ICLR 2020.

[2] Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS 2021.

[W3]: Provide the positional index regression experiments to prove that causal attention can implicitly encode positional information

Thank you for your valuable comments. The positional index regression experiment further verifies that causal attention can encode positional information. Specifically, we trained an MLP to predict the position index of each token using the outputs of causal attention from a well-trained DiT as input. We conducted experiments on both 1D and 2D position regression tasks on ImageNet-256x256 to validate the effectiveness.

For 1D causal attention, the MLP predicts the 1D position index of each token (e.g., {1}, {2}, ... {256}). For 2D causal attention, the MLP predicts the 2D position index (e.g., {1,1}, {1,2},...,{16,16}) for each token. The MLP is trained using L2 loss.

We performed these experiments using DiT with 1D causal attention (variant (a) in the main paper) and DiT with 2D causal attention (variant (d)), and compared the results with DiT-NoPE, which cannot encode positional information. If the causal attention variants outperform DiT-NoPE, it demonstrates that the outputs of causal attention contain implicit positional information.

During inference, we generated images using both DiT with causal attention and DiT-NoPE. At each of the 250 denoising steps, the MLP predicts the positional index from the features output by causal attention. We generated 100 images, resulting in 2,500 tests in total. We report the L2 loss between the predicted and ground truth position indices. Since the MLP is trained to predict positional indices, it cannot generalize to unseen positional indices when extrapolating to higher resolutions. Therefore, we perform inference with the MLP at the training resolution. Nevertheless, the significant performance gap compared to NoPE provides strong evidence that causal attention can implicitly encode positional information. As shown in the table below, causal attention demonstrates a significant advantage over NoPE in both training loss and test error. This indicates that the position regressor can effectively learn positional information from the outputs of causal attention, providing further evidence that causal attention can implicitly encode positional information.

[W4]: Multi-dilation has been widely used in previous vision transformer literature[1,2], limiting the novelty of this work.

We believe T2T-ViT [1] and PS-ViT [2] do not mention or utilize dilation-related concepts or techniques. The multi-dilation training strategy is only one of the contributions presented in our paper. Our main contribution is improving the generative performance of DiT at resolutions higher than those seen during training. We introduce a novel Diffusion Transformer without positional encoding (LEDiT) and provide rigorous theoretical analysis as well as extensive experimental results to demonstrate the extrapolation capability of LEDiT. We believe our contributions are significant in the domain of length extrapolation.

We thank the reviewer for their positive feedback and are pleased that they found our paper well-written and easy to follow. We appreciate the recognition of theoretical and experimental demonstration. We are also encouraged that the reviewer acknowledges the notable improvements.

[W1]: Missing baseline: learnable position embeddings.

Similar to ViT and Swin Transformer, we replace the positional encoding (PE) in DiT with learnable positional embeddings. For length extrapolation, we interpolate the learnable positional embeddings to higher resolutions to ensure compatibility. As shown in the table below, at the training resolution, Learnable PE and LEDiT exhibit nearly comparable performance. When extrapolating to 512×512, we observe a significant drop for Learnable PE. This is likely because the interpolated positional embeddings at new spatial locations are not seen during training, leading to degradation.

Table 1: We report FID-10K for 512x512 and FID-50K for 256x256

[W2]: Scaling to higher resolutions such as 1024x1024.

We respectfully clarify that a comparison at 1024×1024 resolution is presented in Table 7 of the appendix, where our approach demonstrates a clear performance advantage. Please refer to the appendix for detailed results.

As discussed in Section 3.3 of the main paper, we have observed that the variance between local tokens becomes indistinguishable at higher resolutions. To address this, we introduce Locality Enhancement, which improves LEDiT's ability to capture local positional information.

Furthermore, we evaluate LEDiT trained on ImageNet-256×256 and extrapolated to 1024×1024 resolution (a 16× length extrapolation). As shown in the table below, LEDiT outperforms other methods. However, the high FID suggests that aggressive resolution extrapolation remains challenging and warrants further exploration.

[W3]: Justification of the assumption that attention scores and v are mutually independent.

Thank you for your insightful comments. We observe that the correlation between the attention matrix and the value vectors is low during the early stages of denoising. This empirical observation supports the validity of independence assumption. As discussed in the main paper (lines 128–143), our analysis primarily focuses on the early stage, where we show that causal attention mainly encodes positional information.

Theoretical Justification: The input sequence $\mathbf{x} = [x_1,...,x_n]$ can be approximated as i.i.d. Gaussian noise in the early denoising stage, as acknowledged by the reviewer. Each {$x_i$} is an independent Gaussian vector. The queries and keys are computed as: $q_i = x_i W_q$, $k_j = x_j W_k$ Since the {$x_i$} are i.i.d., the sets {$q_i$} are also i.i.d. after linear transformation, the same as {$k_j$}. The attention score is: $S_{ij} = q_i^T k_j$. Because the {$k_j$} are i.i.d., the set {$S_{ij}$} for $j=1,...n$ are identically distributed random variables. After applying softmax $A=Softmax(S)$, the attention matrix $A_{ij}$ approaches a uniform distribution, and $\mathbb{E}[A_{ij}] = 1/n$. From this perspective, the attention matrix and the value have low correlation: changes in the value vectors do not significantly affect the attention matrix, resulting in low correlation. We conduct a toy experiment with a sequence length of 256 and a head hidden dimension of 72, consistent with the DiT-XL/2 configuration. We randomly initialized $W_q$, $W_k$, $W_v$, generated random Gaussian noise $x$, and computed the correlation coefficient between the attention scores $S$ and the values $V$ over 1000 trials. The average correlation coefficient was 0.03, indicating very low correlation. This provides theoretical support for our assumption.

Empirical Justification: In practice, we also observe several layers show low correlation between the attention matrix and the value vectors during the early denoising stages of a trained diffusion transformer. Specifically, we report the correlation coefficients between the attention matrix and the value vectors across different timesteps and layers (see Table below). As shown, the correlation remains low in the early stages of denoising. While the correlation gradually increases in later stages—where the independence assumption no longer holds—we have demonstrated in the main paper that causal attention primarily encodes positional information during the early denoising steps. Therefore, this does not affect the validity of our justification.

[W3]: Missing experiment analyzing trend in Fig 4 and Appendix B for “Mask Lower-right Corner” variant.

Thank you for your valuable suggestions. The variance distribution under the "Mask Lower-right Corner" order is consistent with our main findings. As this is a 2D scan variant, the variance is expected to decrease progressively along the height or width axis. Due to rebuttal space constraints, we provide a single table each for in-distribution and out-of-distribution results. Additional visualizations will be included in the revised version. These results demonstrate that the 2D causal scan variants still exhibit the existence of causal attention in DiT that satisfies the conditions outlined in our theorem.

Additionally, we conducted a position regression experiment to further demonstrate that causal attention can implicitly encode positional information. For more details, please refer to our response to [W3] of Reviewer x8vf if interested.

Table 2: Timestep: 943, Layer: 20. In-distribution (256x256) settings.

Table 3: Timestep: 750, Layer: 15. Out-of-distribution (512x512) settings.

We thank the reviewer for their positive feedback and are pleased that they found our paper to be well-motivated and well-written. We appreciate the recognition of our novel implicit positional encoding and the analytical insights provided, which enable robust length extrapolation and demonstrate substantial improvements over existing methods.

[W1&W3&Q2]: The trade-offs of replacing self-attention layers with causal attention's unidirectional flow needs further analysis and discussion. Whether causal attention might potentially reducing generative diversity (recall); The insight on why the recall drops.

Thank you for your insightful comments. We have found that, by appropriately balancing causal attention and self-attention, our method achieves in-distribution performance comparable to RoPE, while significantly outperforming existing methods on extrapolation tasks. To provide a more comprehensive analysis, we conducted controlled ablation studies by varying the number of causal attention layers in LEDiT. Our results indicate that increasing the number of causal layers enhances length generalization, with only a minor reduction in in-distribution performance. Compared to previous methods, our approach achieves a more favorable trade-off between these aspects. Table 1 compares different LEDiT variants and other positional encoding methods under both in-distribution and out-of-distribution settings. Notably, with just a single causal layer, LEDiT significantly outperforms all previous methods at higher resolutions. At the training resolution, LEDiT matches the widely adopted RoPE in performance, demonstrating that LEDiT preserves strong in-distribution results while providing superior extrapolation capability. Increasing the number of causal layers further enhances extrapolation, with only a slight reduction in in-distribution performance. While the optimal trade-off may vary by application, our results indicate that the default configuration described in our paper (i.e., 14 causal layers) achieves an effective balance between in-distribution and out-of-distribution performance.

Table 1: Out-of-Distribution (512x512) and In-distribution (256x256) comparison between LEDiT variants and other methods. we report FID-10K for 512x512 and report FID-50K for 256x256

Whether causal attention might potentially reducing generative diversity (recall); The insight on why the recall drops.

Causal attention is not the cause of recall reduction. In fact, we emphasize that LEDiT even achieves slightly better recall than DiT in the in-distribution setting (see Table 1), indicating that causal attention does not compromise sample diversity.

In the length extrapolation scenario, both Table 2 of the FiT paper [1] and Table 1 of our work consistently reveal two distinct method categories:

• Category I: Methods with lower FID (e.g., LEDiT, DiT-VisionYaRN) tend to exhibit higher precision but lower recall.
• Category II: Methods with higher FID (e.g., DiT-NTK) tend to exhibit lower precision but higher recall.

For clarity, we reports the performance of various PE and LEDiT when extrapolated to 384×384 resolution, all trained on ImageNet-256×256.

Category I methods generate high-quality samples with limited diversity, whereas Category II methods frequently produce a substantial proportion of broken images, as shown in the visualizations in our appendix. We argue that Category I methods are preferable, consistent with the statement from the precision/recall paper[2] (page 5): "It is unclear which application might favor setup D (Category II), where practically all images are broken, over setup B (Category I), that produces high-quality samples at a lower variation."

The reduction in recall can be attributed to out-of-distribution (OOD) generalization challenges. When models are trained on low-resolution data, the learned data distribution $P_l$ does not fully capture the high-resolution manifold $P_h$. Consequently, when extrapolating to high-resolution, models tend to generate samples concentrated in the intersection region $P_e = P_l \cap P_h$. Samples from $P_e$ simultaneously satisfy both the higher-resolution requirements and the learned data distribution. However, since $P_e$ is a subset of $P_l$, this restriction reduces diversity, leading to lower recall.

[1] FiT: Flexible Vision Transformer for Diffusion Model, ICML 2024.

[2] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019.

[W2]: Training LEDiT on large-scale, diverse sets like LAION-5B to test generalizability.

Due to time constraints and limited computational resources, we currently face challenges in applying LEDiT to large-scale sets like LAION-5B. Following previous works DiT, SiT, REPA, we conduct extensive experiments on class-conditional and text-to-image generation tasks on ImageNet and COCO dataets. We also believe that LEDiT can improve training on large-scale datasets, which contain images with diverse resolutions and aspect ratios. By removing the constraints of explicit positional encoding, LEDiT can better accommodate images of varying resolutions, which may lead to improved training outcomes. In future work, we plan to progressively extend LEDiT to larer-scale data to evaluate its generalization and scalability in more complex generative settings.

[Q1]: How does the T2I alignment metric perform with the proposed architecture. It would be good to understand if the architecture would compromise semantic coherence for dense prompts

LEDiT maintains semantic coherence for dense prompts. We use 40,504 captions from the COCO dataset to generate images and compute the CLIP score as the text-to-image alignment metric. The comparison is presented in the following table. LEDiT consistently outperforms other methods in terms of semantic coherence.