Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations

Thanks for your valuable suggestions. We’re glad that you found our work insightful and interesting. Please see the responses to your comments.

Q1. Reason why massive activation is suboptimal for representation

Thank you for your valuable comments. First, we would like to clarify that we have provided insightful intuition in the Introduction section (Lines 35–39), e.g., “These massive activations share a few fixed dominant dimensions, resulting in a high degree of directional similarity for feature vectors.” We are pleased to see that you agree with our explanation. To further investigate the effect of massive activations, we conduct detailed experiments on Table i, and we elaborate on several key findings based on the results.

1) Massive activations significantly hinder spatial discrimination

Due to the directional homogeneity introduced by massive activations, cosine similarity becomes ineffective in distinguishing spatial feature vectors. As shown in Table i, discarding the massive activation dimensions in feature maps significantly improves correspondence performance (from 29.9 to 37.5). Moreover, since DiTs are originally pretrained for generation tasks, their features often contain fine-grained visual details, which may be irrelevant or even detrimental for dense matching tasks [1]. For example, as illustrated in Figure 5 of the main text and Figures in the supplementary material, the original features (pre-AdaLN) tend to preserve vivid color cues as well as lighting and shading, which are essential for generation but suboptimal for visual correspondence (especially semantic correspondence). Consequently, the co-existence of massive activations and irrelevant details weakens the representational robustness of DiT features for correspondence tasks.

2) AdaLN enhances representation by suppressing activations and irrelevant details

As shown in Table i, directly applying channel discard to the original DiT features provides only suboptimal improvement, suggesting that reducing massive activations alone is insufficient. In contrast, AdaLN not only suppresses the massive activations but also effectively activates other critical channels essential for dense perception tasks. This dual effect mitigates the impact of massive activations while simultaneously suppressing irrelevant low-level details (e.g., color and shading), leading to a more refined and semantically meaningful feature space (see Figure 5 and the supplementary material). Consequently, the collaboration between AdaLN and channel discard operation leads to substantial gains in performance (29.9 to 67.1).

Table i. Ablation study of our model DiTF$_{Flux}$ on the SPair-71k semantic correspondence benchmark.

Q2. Explanation for the geometric correspondence performance

Thank you for your comments. We agree that feature resolution plays a critical role in determining geometric correspondence performance. Specifically, given an input image of size $H \times W$, our DiTF framework produces features at a resolution of $\frac{H}{16} \times \frac{W}{16}$. In contrast, DIFT [2] leverages higher-resolution features of size $\frac{H}{8} \times \frac{W}{8}$, which are extracted from its 2nd upsampling block. These higher-resolution features preserve finer-grained texture and structural details, thereby contributing to better geometric localization. As shown in Table ii, when DIFT utilizes the lower-resolution features of size $\frac{H}{16} \times \frac{W}{16}$ from its 1st upsampling block, it achieves a reduced accuracy of 37.2%, which is lower than our DiTF$_{Flux}$ (41.9%). This comparison underscores the effectiveness of our framework.

Table ii. Comparisons between DIFT and Our DiTF$_{Flux}$ model on the HPatches geometric correspondence benchmark.

Q3. Typos and citations

Thank you for your feedback. In the final version, we will properly cite the related GAN literature and fix all typos and errors in Figure 5. In addition, we will highlight temporal correspondence results in the main text.

[1] Representation alignment for generation: Training diffusion transformers is easier than you think, ICLR 2025.

[2] Emergent Correspondence from Image Diffusion, NeurIPS 2023.

Thank you for your thoughtful review. We appreciate your recognition of our contributions on massive activations in DiTs and address your concerns as follows:

Q1. Clarification of core contributions and novelty

We would like to clarify that our method is fundamentally different from a simple reimplementation of [1] within the DiTs, and it introduces novel insights and contributions beyond them. In particular, we emphasize the following key aspects:

1) Unique characteristics and insightful analysis of massive activations in DiTs.

• Compared to [1], we identify distinctive characteristics of massive activations in DiTs, which differ fundamentally from those observed in large language models (LLMs) [1]. In DiTs, the massive activations are consistently concentrated in a few fixed dimensions across all spatial tokens (see Figures 2 and 3). In contrast, massive activations in LLMs typically occur in specific input tokens (e.g., starting token).
• Furthermore, our analysis reveals that the unique characteristic of massive activations in DiTs is closely associated with the channel-wise residual scaling factors introduced by AdaLN-Zero. Specifically, DiT feature updates are computed via residual connections: $z_{t}^{k+1} = z_{t}^{k} + \alpha_k A_k(z_{t}^{k}),$ where we observe a strong alignment between the dimensions exhibiting massive activations and those corresponding to large values of the residual scaling factors $\alpha_k$ (see Figure 4 and supplementary material). Such theoretical and empirical investigations demonstrate an insightful connection between massive activations and the scaling behavior of AdaLN-zero (recognized by Reviewer ntGm).

2) A reasonable and practical training-free framework for feature extraction with DiTs.

• Motivated by the observed relationship between massive activations and the residual scaling factors of AdaLN-Zero, we propose a reasonable and elegant feature extraction framework, DiTF, which can be directly applied to existing pre-trained DiTs without requiring any additional training (recognized by Reviewer ntGm). Specifically, we reuse the built-in AdaLN modules in DiTs to effectively identify and regulate these massive activations with adaptive channel modulation, significantly enhancing the quality of representations.
• Furthermore, strategies for handling massive activations in LLMs [1] do not transfer effectively to DiTs. Specifically, when directly applying the activation intervention strategy in LLMs [1] to DiTs, it fails to improve feature quality, as evidenced in Table i (29.9 vs. 30.1), highlighting the fundamentally different nature of massive activations in LLMs and DiTs. In contrast, our proposed DiTF enhances both feature semantics and discriminability, leading to substantial performance improvements on dense spatial tasks (29.9 to 67.1).

Table i. Performance comparison of Flux models with different massive activation tackling strategies on the SPair-71k semantic correspondence benchmark

3) A novel perspective on the poor performance of DiT features in dense tasks.

• When directly using DiT features for dense spatial tasks, it yields disappointing results (Figure 1). In this paper, we identify the massive activations within DiT feature maps as a key factor limiting performance (recognized by Reviewer gPKV). Unlike Stable Diffusion (SD2-1), DiTs exhibit massive activations concentrated in a few fixed dimensions across all image patch tokens (Figure 2). These dominant dimensions cause high directional similarity among feature vectors [3], which makes it difficult to distinguish them using cosine similarity, and ultimately degrades performance on dense spatial tasks (Lines 35-39).

4) Novel insights into future DiT training.

• REPresentation Alignment (REPA): REPA [4] aligns DiT hidden states with pretrained features (e.g., DINOv2) to facilitate training. However, due to the distinct feature characteristics: DiTs exhibit massive activations whereas DINOv2 does not, direct alignment may be suboptimal. Instead, aligning features after AdaLN modulation may improve compatibility and alignment quality.

Q2. On Theoretical depth and justification

We respectfully clarify that our conclusions are not solely based on visual inspection or simple correlations, but are derived from a rigorous and systematic study across various pre-trained DiT models and are supported by both comprehensive results and theoretical insights. Specifically:

1) Robustness and generality.

We examined a wide range of publicly available pre-trained DiT models, including SD3, SD3-5, Flux, etc., and consistently observed the same characteristics of massive activations. This consistent pattern suggests that our findings are not model-specific or incidental, but rather reflect a general property of DiTs.

2) Comprehensive experimental and visual support.

Our paper conducts systematic experiments complemented by carefully designed visualizations to thoroughly investigate the phenomenon of massive activations. Furthermore, we provide plentiful supporting evidence in the supplementary materials, collectively establishing a robust experimental foundation for our conclusions.

3) Theoretical justification for the proposed solution.

Our method is theoretically motivated and interpretable:

• Massive activation analysis: We have provided theoretical justification for massive activation analysis in the introduction section (Lines 35-41). These massive activations share a few fixed dominant dimensions across all spatial tokens, resulting in a high degree of directional similarity for feature vectors. This property makes it challenging to distinguish spatial feature vectors using cosine similarity, which hinders spatial discrimination of representation. In addition, this analysis theoretically supports the conclusion that "Massive activations hold little local information."
• Connection to AdaLN-Zero: DiT feature updates are computed via residual connections: $z_{t}^{k+1} = z_{t}^{k} + \alpha_k A_k(z_{t}^{k}),$ meaning AdaLN-zero directly affects the dominance and expressiveness of DiT features with residual scaling factor $\alpha_k$.
• Channel-wise modulation with AdaLN: Motivated by the above insights, we employ the built-in AdaLN to modulate massive activations in DiT features, formulated as:

$$\text{AdaLN}(z_{t}^{k}) = (1 + \gamma_k) \cdot \text{LayerNorm}(z_{t}^{k}) + \beta_k,$$

where $\gamma_k \in \mathbb{R}^C$ and $\beta_k \in \mathbb{R}^C$ are adaptive channel-wise scaling and shifting parameters. Specifically, AdaLN effectively identifies and suppresses dimensions associated with massive activations through the scaling factor $(1 + \gamma_k)$, thus enhancing representation semantics and discrimination.

Q3. Clarifying the Alleged Contradiction with 'Massive Activations in LLMs'

We would like to clarify that our method does not conflict with [1], as the operational setting and the nature of massive activations are fundamentally different.

• In [1], the authors directly intervene in the hidden states during inference by manually setting massive activation values to zero in a single layer, and then propagating the modified hidden state through the remaining model. This intervention aims to probe the functional role of massive activations in the internal computation of LLMs. In contrast, our method treats the Diffusion Transformer (DiT) as a frozen feature extractor. We extract features from an intermediate block, apply AdaLN to these features, and perform a channel discard (zero out) operation to weakly massive activations in the extracted features only. Importantly, no modifications are made to the internal computation of the model.
• Furthermore, the characteristics of massive activations in DiTs are notably different from those in LLMs [1]. The massive activations in LLMs [1] typically arise at specific tokens (e.g., starting token). In DiTs, such massive activations consistently occur at a small number of fixed channel dimensions, and appear across all spatial tokens (Figures 2 and 3). These massive activations with concentration across all spatial tokens in DiTs result in a high degree of directional similarity for feature vectors in the latent space [2]. Our proposed channel discarding strategy specifically addresses this issue by suppressing dominant but uninformative massive activations, thereby promoting feature discrimination and semantics.

Q4. Generalizability on various CV tasks

We have conducted semantic segmentation experiments on ADE20K (Table 6 of main text). Specifically, our approach boosts mIoU from 43.8 to 53.6. We further conduct linear probing on ImageNet classification using globally pooled DiT features, following the DINOv2 [3] protocol, and achieve a notable accuracy gain from 65.8 to 72.6 (DiTF$_{Flux}$), highlighting the effectiveness and generalizability of our method.

Q5. Typos and citations

Thank you for your valuable feedback. In the final revision, we will correct all the typos and citation mistakes. In addition, we will revise our assumptions of AdaLN-Zero and AdaLN to accurately reflect their differences.

[1] Massive activations in large language models, COLM 2024.

[2] Clearclip: Decomposing clip representations for dense vision-language inference, ECCV 2024.

[3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2025.

[4] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, ICLR2025.

Thank you for the clarification. I find the authors’ explanation—along with the newly added experiments—both clear and convincing. The distinction between their method and [1], particularly in terms of operational settings and the nature of massive activations, is well articulated. It is especially helpful that the authors emphasize how their method operates on extracted features in a non-invasive manner (I refer to it more like a masking operation in this work instead of the Set to zero operation in [1]), without intervening in the model’s internal computations, leading to different performances (30.1 and 67.1).

The additional experiment also strengthens the justification for their approach by highlighting the differences in activation patterns between LLMs and DiTs (i.e., token-specific vs. channel-specific and spatially consistent). The motivation for channel discarding is now well supported by the observed characteristics of massive activations in DiTs, and the intended impact on improving feature discrimination is clearly demonstrated.

I strongly encourage the authors to explicitly incorporate this clarification and the new experimental evidence into the revised manuscript. Doing so will greatly help readers understand the key distinctions between their method and related work.

Just one additional concern. While I understand that LLMs and vision models exhibit different massive activation patterns, I would appreciate it if the authors could further elaborate on why such differences also exist within vision models. Specifically, in [1], ViT and DINO do not exhibit massive activations concentrated in a few fixed dimensions across all spatial tokens, as observed in DiT in this work. Furthermore, the zero-set operation in [1] leads to a severe performance drop in ViT-based models, whereas in DiT, the same operation appears to yield a slight performance improvement.

Overall, I appreciate the thoughtful and detailed rebuttal. As my main concerns are mostly addressed (difference to [1] and openration clarification), the method demonstrates both conceptual soundness and strong empirical performance with a high-quality, I will update my score to reflect this positive clarification based on the final response accordingly.

Thanks for your suggestions and for recognizing the novelty and contribution of our work. Please see the responses to your comments.

Q1. Details of how AdaLN mitigates the massive activations

Thank you for your valuable comments. We have described the channel-wise modulation strategy using AdaLN to address massive activations in Sections 4.2 and 4.3. To further clarify the mitigation process, we elaborate on the rationale and mechanism below.

1) Why utilize AdaLN to mitigate the massive activations?

In DiTs, feature updates are computed via residual connections: $z_{t}^{k+1} = z_{t}^{k} + \alpha_k A_k(z_{t}^{k}),$ where $\alpha_k$ denotes the channel-wise residual scaling factor from the AdaLN-Zero layer. We observe that the massive activation dimensions in $z_{t}^{k+1}$ strongly align with large values of $\alpha_k$ in the corresponding channels (as shown in Figure 4). This reveals an inherent connection between massive activations and the scaling behavior of AdaLN. Based on this insight, we leverage the adaptive nature of AdaLN to identify and regulate these massive activations.

2) How to mitigate the massive activations with AdaLN?

The modulation process of AdaLN is formalized as:

$$\text{AdaLN}(z_{t}^{k}) = (1 + \gamma_k) \cdot \text{LayerNorm}(z_{t}^{k}) + \beta_k,$$

where $\gamma_k \in \mathbb{R}^C$ and $\beta_k \in \mathbb{R}^C$ are adaptive channel-wise scaling and shifting parameters, respectively. Applied to the DiT features $z_{t}^{k} \in \mathbb{R}^{C \times H \times W}$, this modulation allows AdaLN to adaptively identify and suppress dimensions with massive activation through the factor $(1 + \gamma_k)$. Consequently, the massive activations are effectively mitigated by the channel-wise modulation of AdaLN (see Figure 5).

Q2. Explanations of inconsistent dimensions in flux

Thank you for your comment. We emphasize that AdaLN effectively suppresses the original massive activations at dimensions 154 and 1446 through channel-wise modulation. The observed inconsistency in the dimensions of (weakened) massive activations stems from the adaptive nature of AdaLN, which applies distinct scaling factors to each dimension based on learned statistics. While AdaLN strongly attenuates the dimensions corresponding to massive activations, it may slightly amplify others, resulting in a shift in the top activated dimensions after normalization. This behavior reflects the rebalancing effect of AdaLN and does not negatively impact downstream performance.

Q3. Comparisons between models with and without the DiTF framework

We have provided detailed direct comparisons between the same base models (e.g., SD3-5 or Flux) with and without the DiTF framework in Figure 6. For better readability, we summarized these results in Table i. The comparison clearly shows that incorporating DiTF leads to consistent and significant performance improvements across different DiT-based models.

Table i. Comparisons between models with and without the DiTF on the SPair-71k semantic correspondence benchmark.

In the paper, SD3 refers to the SD3-Medium model, while SD3-5 corresponds to the SD3.5-Large model. Additionally, we conducted experiments on SD3.5-Medium, where our DiTF led to a substantial performance improvement from 38.8 to 58.3, demonstrating the effectiveness of our method. To improve clarity, we will provide more detailed descriptions of the different DiT variants in the final revision.

Q4. Generalizability on various CV tasks

We have conducted semantic segmentation experiments on ADE20K, as presented in Table 6 of the main paper. Specifically, our method significantly improves the performance of DiT features, increasing the mIoU from 43.8 to 53.6 (DiTF$_{Flux}$). Additionally, we perform linear probing experiments on ImageNet classification using DiT features, following the same protocol as DINOv2 [1]. We use the globally pooled DiT features as input. Our method again yields a substantial improvement, boosting accuracy from 65.8 to 72.6 (DiTF$_{Flux}$). These results demonstrate the effectiveness and general applicability of our approach across diverse vision tasks.

Q5. Discussion of limitations and future works

We have discussed the limitations and potential future directions in the supplementary material. In the final revision, we will expand upon these points and incorporate them into the main text to enhance clarity and completeness. In response to your suggestion, we will also apply color coding to Figures 1a and 2 for improved readability.

[1] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2025.

Thanks for your valuable suggestions. We’re glad that you found our work insightful and interesting. Please see the responses to your comments.

Q1. More theoretical insights into the unique characteristics of massive activations in DiTs

Thank you for your insightful comment. We would like to emphasize that our primary focus in this work is to investigate the impact of massive activations from a representation learning perspective. To inspire further research on visual generation, we provide a preliminary theoretical claim to offer insights into the underlying cause of this phenomenon.

Distinct training paradigms and supervisory signals result in different spatial locations of massive activations in LLMs and DiTs.

• LLMs are typically trained using an autoregressive next-token prediction objective, where each token prediction is conditioned on all preceding tokens. As the model generates the entire sequence token-by-token, the early tokens play a disproportionately important role in shaping the final output distribution. Consequently, the initial tokens (particularly starting token) tend to accumulate stronger activations, resulting in token-specific massive activations [1].
• In contrast, Diffusion Transformers (DiTs) are trained under a denoising diffusion paradigm, where supervision is applied uniformly across all image patches. Each patch is treated equivalently during training, and the model learns to reconstruct all patches simultaneously. This uniform treatment results in a spatially uniform distribution of massive activations across tokens.
• To verify our hypothesis, we conduct a comparative analysis of massive activations in two large language models LLaMA3-8B [4] and LLaDA-8B [5]. Notably, these two models share a similar architecture, and the primary difference lies in their training paradigms: LLaMA3-8B follows next-token prediction objective, while LLaDA-8B is trained using masked diffusion modeling. As shown in Table i, massive activations in LLaMA3-8B are concentrated in a specific token: the starting token (<|startoftext|>). In contrast, LLaDA-8B exhibits massive activations in dimension 3848 across all text tokens. This distinction aligns with our theoretical claim that the choice of training paradigm fundamentally influences the activation distribution patterns.

Table i. Distribution of activation values along the massive activation dimension in LLaMA3-8B and LLaDA-8B. The dimension associated with massive activations is indicated by (·). The input sentence is: "Summer is warm. Winter is cold.".

Q2. Computational cost analysis

To provide a more transparent comparison, we have included a detailed computational cost analysis in Table ii. All results are measured on a single NVIDIA L40S GPU (48 GB). Below, we elaborate on several key aspects relevant to this concern:

1) Training-free and zero-shot Inference:

DiTF operates in a fully training-free and zero-shot manner, leveraging pretrained Diffusion Transformers (DiTs) as universal feature extractors. This design enables direct application to a wide range of visual correspondence tasks without any task-specific fine-tuning. In contrast, supervised methods such as CATs++ [2] rely on costly and time-consuming task-specific training.

2) Efficient inference with simpler matching pipeline:

DiTF achieves faster inference despite using larger backbones, due to its lightweight and direct matching pipeline. Specifically, DiTF only requires a single forward pass to extract dense features from each image, followed by a cosine similarity computation for pixel-wise matching, where no trainable modules are introduced in the matching process. In contrast, supervised methods typically adopt a two-stage process:(1) Extract features via a CNN or Transformer backbone (2) Feed the feature pair into an additional trainable cost aggregation network to produce the final matching result. This additional network significantly slows down inference. As a result, although DiTF uses a large pretrained backbone, its inference time is faster than that of supervised as CATs++ [2] (42.0 ms vs. 55.9 ms), as shown in Table ii.

3) Negligible overhead from large-scale backbones:

The feature extraction stage consumes only a small portion of the overall runtime. For instance, with the billion-parameter DiT (Flux) model, the feature extraction time is 6.21ms (42.0ms in total), accounting for less than 15% of total inference time. This suggests that the performance gain from better features outweighs the minor cost introduced by a larger backbone.

| Model |Zero-shot|Inference Time $ms/pair$ | Accuracy | | :-----: | :-------: |:-------: |:-------: | | CATs++| No |55.9(1.99)|59.8| | DINOv2 |Yes|21.0(2.53)|55.6| | DIFT|Yes|26.7(4.08)|57.7| | DiTF$_{SD3-5}$|Yes|29.5(4.90)|64.6| | DiTF$_{Flux}$|Yes|42.0(6.21)|67.1|

Table ii. Computational cost comparison of different models on the SPair-71k semantic correspondence benchmark. Inference time for feature extraction is denoted by (·)

Q3. Hyperparameter analysis

Channel discard thresholds: In our study, we set the default discard threshold to $m=10$, where a feature dimension is discarded (zeroed) if its mean activation magnitude exceeds $m\times$the global median. To assess robustness, we evaluated thresholds $m\in$ {14,12,10,8,6}. As Table iii shows, our method maintains stable performance across all thresholds and different DiT models. Notably, the normalized (weakened) massive activations remain ≥14× the global median, ensuring a clear distinction from normal activations. Such a wide margin ensures that varying the threshold $m$ (from 6 to 14) has minimal impact on performance, highlighting the robustness of our method.

Table iii. Performance with different channel discard thresholds on the SPair-71k semantic correspondence benchmark.

Condition dependencies: We have already included a detailed ablation study in Table 5, demonstrating that the condition timestep t plays a critical role in enabling AdaLN to modulate massive activations (Lines 248-254). More investigations of the block index k and the timestep t are provided in Figure 4 of the supplementary material.

Q4. Related works

We appreciate the suggestion and will cite TinyFusion [3] along with other related works in the revised version.

Q5. Transparent discussion of limitations

In the revised version, we will move the limitations and future work section from the supplementary material to the main paper, and include a discussion emphasizing the need for a deeper understanding of why such massive activations emerge during DiT training and what functional role they play in visual generation tasks.

[1] Massive activations in large language models, COLM 2024.

[2] CATs++: Boosting Cost Aggregation with Convolutions and Transformers, TPAMI 2022.

[3] TinyFusion: Diffusion Transformers Learned Shallow, CVPR2025.

[4] The llama 3 herd of models.

[5] Large Language Diffusion Models.