Demystifying Singular Defects in Large Language Models

Thank you very much for your encouraging comments!

Thank you for the insightful comments! We will incorporate the suggestions in the revision.

Is causal attention really the key factor? Why ViTs have similar high-norm tokens?

We hypothesize that causal attention excites high norms in LLM in the following way. In the causal formulation, the possible output features at the initial position are finite ($=V$, the vocabulary size, e.g., 32K for LLaMA2). The softmax loss increases the norm of these features for each training sample, so their norms grow fast. By comparison, the possible output features at the 2nd position are 1 billion ($=V^2$). The push towards increasing their norms is thus dispersed, yielding no systematic high norms in the 2nd and subsequent positions.

The high-norm tokens in ViTs could stem from different causes as implied by their drastically different patterns.

1. They appear in the late stage of training, while High-norm tokens in LLMs appear in early stage (L324-327).
2. They grow progressively layer by layer, while high-norm tokens in LLMs explode abruptly (L157-159).
3. They appear at random locations, while high-norm tokens in LLMs mainly locate at the starting position (L160).
4. They can be repaired by SINDER without affecting performance, while the presence of high-norm tokens is critical for the performance of LLMs (L310).
5. The single-token assumption in ViTs is an ideal assumption to simplify the analysis. By contrast, in LLMs, the single-token assumption is real and valid for the initial token (L157 Column2).

These differences motivate the question "whether the theory of singular defects can be applied to LLMs" (L34-35 Column2) and our study gives an affirmative answer.

How about the behavior of high-norm vision tokens in the language models of VLMs? Are they working as the vision registers or behaving as the LLM high-norm tokens?

To answer this question, we forwarded an image-text mixed data using LLaVA-v1.5-7b, and located the image patch token with the highest norm in the ViT features. Here is its norm (round to integer) at different LLM layers:

764 (norm of input visual token), 764 (after 1st layer of LLM), 764, 765, 764, 764, 764, 762, 761, 760, 758, 756, 754, 752, 750, 748, 746, 744, 741, 740, 738, 738, 737, 736, 734, 734, 732, 732, 732, 730, 730, 729, 728

Unlike the high-norm text token in LLM, whose norm explodes and decays, the norm of high-norm vision token stays nearly the same throughout the LLM. In addition, the angle between the high-norm vision token and the high-norm text token is kept around 87 degrees across LLM layers. Based on these, we hypothesize that high-norm vision tokens neither work as registers nor behave as LLM high-norm tokens.

What about the differences between the attention sink and the high-norm tokens?

Attention sink refers to the phenomenon that the initial token receives large attention scores. It is a side-effect emerging from the fact that the initial tokens have high norms, as discussed in the massive activation paper by Sun et al.

We believe our efforts on demystifying high-norm tokens provide a better understanding of the underlying mechanism of attention sinks. For example, in Fig. 12, we show that model trained with window attention (Mistral) does not have initial high-norm tokens. As a result, Mistral can process arbitrary context length without the special treatment introduced in the attention sink paper.

What is the actual role of the high-norm tokens in LLMs?

In Fig. 13, we show that high-norm tokens appear in the very early stages of training. We therefore hypothesize that they are a way to accelerate network convergence by injecting some biases. To verify this, we added an extra layernorm at the output of the residual branch of each attention block and that of each FFN block to suppress any possible high-norm tokens. We trained the modified LLaMA2-1b model from scratch and observed that the loss decreased more slowly compared to the original structure.

Which kinds of text tokens are easier to trigger the high-norm token phenomenon?

Firstly, as shown in Fig. 7, any token in the initial position will trigger the high-norm phenomenon. Secondly, some high-frequency tokens in the corpus such as \n, <s> may appear as non-initial high-norm tokens. Yet, the set of non-initial high-norm tokens may change during training. For example, in Fig. 13 Column2, the '.' token is a non-initial high-norm token at training iteration 50k, but is no longer a high-norm token at iteration 143k. We will extend Fig. 6 to more LLMs in the revision and below we summarize their non-initial high-norm tokens.

Thank you for the detailed and thoughtful comments and we are encouraged by the remark that "applications section is particularly strong". We will incorporate the suggestions in the revision.

How it fits into the broader literature and what exactly new it contributes

While Sun et al. spotted high-norm tokens and SINDER provided a basic tool, our work advances the understanding from three aspects:

• Structural Characterization. We introduce explosion subspace and decay eigenvalue analysis, providing a structured framework for high-norm token evolution in LLMs.
• Causal Insights. We conjectured a link between high-norm token emergence and the causal self-attention.
• Practical Applications. We demonstrate their impact on quantization and introduce model signatures for lineage tracing.

We highlight our unique contributions and whether they relate to SINDER and Sun et al. below.

The linear model applies to one token and makes the analysis of noninitial tokens challenging

Fig. 6 shows that the noninitial high-norm tokens retain their high norms after removing all self-attentions. They effectively behave as independent single-token within the network (L262-264), which makes their analysis by linear model feasible.

Sun et al. found that HNs occur in the same locations in the vectors, which is the same as "high norm tokens share the same direction"

Their observation is not sufficient to reach our conclusion that "high norm tokens share the same direction". Firstly, they show no evidence that the relative magnitude of massive locations is consistent. For example, Tab. 1 and 2 of Sun et al. track the scalar magnitude by sorting, which discards the channel locations. Secondly, they did not quantify the influence of the non-massive locations (which are in majority) on vector direction. As such, they cannot rigorously infer our conclusion.

"Self-attention plays an insignificant role in noninitial HN tokens" is the same as Sun et al's "high-norm tokens act as fixed biases"

We respectfully disagree. Our statement refers to the development of noninitial high-norm tokens, namely, they still exhibit high norm after removing self-attention (Fig. 6). Contrastingly, self-attention is indispensable for initial high-norm tokens: When it is removed, initial tokens lose high norms (Figs. 7 and 6). This underscores a fundamental difference to Sun et al's.

The findings from removing the explosion subspace are equivalent to Sun et al.'s zeroing out massive activations

We will add the following clarification in the revision. "This observation echoes the experiments done by Sun et al., where the authors set the massive activations to zero".

How does the quantization procedure differ from that of LLM.int8()?

The core innovation of LLM.int8() is to decompose a matrix into an 8-bit part and a 16-bit part. We did not use this trick. Besides, they rely on row-wise quantization whereas we use tensor-wise quantization (L354-357 Column2). Finally, more than 50% of their matrices are affected (Fig. 3 in their paper), whereas ours only affects 2 matrices. We will add LLM.int8() to the related work.

Why high-norm tokens occur during training in the first place?

We have ruled out 11 factors (L308-313 Column2) and singled out the most probable reason: The causal self-attention. It is evidenced by the impact of different attention formulations (L318-329). With causal attention, the possible output features for the initial token are finite ($=V$, the vocabulary size, e.g., 32K for LLaMA2). The softmax loss pushes the norm of these features to grow for each training sample, resulting in high-norms. By comparison, the count of possible output at the 2nd position is 1 billion ($=V^2$), which disperses the push toward increasing their norms.

The "explanation for why there exists a set of fixed channels with high activations" is not supported

To avoid confusion, we will revise it to "This explains why the set of high-activation channels observed in Sun et al. is fixed".