Uncovering hidden geometry in Transformers via disentangling position and context

We thank the reviewer for reading this submission and providing feedback. We first present our replies to common concerns, and then respond to specific issues.

Common concerns

1. Significance of positional basis. We emphasize that the positional embeddings in many transformers including GPT-2 are trained from scratch during pretraining---which is different from the sinusoidal positional embedding originally proposed. We find this nontrivial because

   • as far as we know, the low-rank and low-frequency structure has not been examined extensively before.
   • The magnitude of the positional basis can change across layers (for aesthetic reasons, Figure 1 is presented as scale-free). We observed the positional basis increases significantly from layer 0 to layer 1. So residual connection is not the sole reason for consistent positional basis.
   • Our discovery helps elucidate confusing artifacts (e.g., spiral clustering in embeddings) people observed; see Section 6.
   • Smoothness is the key to Transformers. Our analysis on an arithmetic task in Section 5 shows a failure example where transformed exhibit discontinuous patterns that result in failure of length generalization.

2. Induction head. One of the most surprising phenomena about LLMs is the emergent abilities, and induction head---which functions as copying even in OOD data---is one simple example. Still, much is unknown about induction head.

   • Our decomposition-inspired analysis identifies 3 interpretable components. The first two (attention to self-token, attention to neighboring tokens) correspond to exactly positional basis or cvec.
   • This ``abstract ability'' learned by transformers may allow people to understand the inner workings of LLMs and lead to practical values.
   • It provides a new visualization idea---which decouples positional effects from non-positional effects.

Specific issues

• Q1 "The first two findings (a and b) sound relatively trivial". We disagree that the finding a is trivial. See our response to common concerns 1. Moreover:

  • We hope to draw the reviewer's attention to the fact that many Transformers use positional embeddings entirely trained from scratch, so the sinusoidal is not known in advance. Even so, the low-rank and low-freq structure is not expected.

  • Note that the positional basis lies approximated in a low-dimensional subspace. As far as we know, this is not known before, and is the reason for many artifacts people observed in embedding visualiation.

• Q2: "Why does Equation (7) not include the residual term?" In this decomposition, we are only interested in separating the positional effects vs non-positional effects, so $h$ is decomposed into $\mathrm{pos}$ plus $\mathrm{cvec}$. Note that $\mathrm{cvec}$ includes the residual term already.

• Q3: "How does transformer layer TFLayer enable this information flow". Our question is to understand how information flow is processed in the attention component, when passing through the self-attention in each layer.

  • The structural decomposition of $W$ shows that there is a positional low-rank part corresponding to exactly the positional subspace of embeddings. As far as we know, this gives the first consistent identification of interpretable components of pretrained weights.

We thank the reviewer for reading this submission and providing feedback. We first present our replies to common concerns, and then respond to specific issues.

Common concerns

1. Significance of positional basis. We emphasize that the positional embeddings in many transformers including GPT-2 are trained from scratch during pretraining---which is different from the sinusoidal positional embedding originally proposed. We find this nontrivial because

   • as far as we know, the low-rank and low-frequency structure has not been examined extensively before.
   • The magnitude of the positional basis can change across layers (for aesthetic reasons, Figure 1 is presented as scale-free). We observed the positional basis increases significantly from layer 0 to layer 1. So residual connection is not the sole reason for consistent positional basis.
   • Our discovery helps elucidate confusing artifacts (e.g., spiral clustering in embeddings) people observed; see Section 6.
   • Smoothness is the key to Transformers. Our analysis on an arithmetic task in Section 5 shows a failure example where transformed exhibit discontinuous patterns that result in failure of length generalization.

2. Induction head. One of the most surprising phenomena about LLMs is the emergent abilities, and induction head---which functions as copying even in OOD data---is one simple example. Still, much is unknown about induction head.

   • Our decomposition-inspired analysis identifies 3 interpretable components. The first two (attention to self-token, attention to neighboring tokens) correspond to exactly positional basis or cvec.
   • This ``abstract ability'' learned by transformers may allow people to understand the inner workings of LLMs and lead to practical values.
   • It provides a new visualization idea---which decouples positional effects from non-positional effects.

Specific issues.

• Q1: "Several topics discussed feel incomplete". We agree that some supporting arguments are not presented in the main paper; however, we feel that the appendix does contain enough details and supporting figures.
• Q2: "The discussion on induction heads is distracting". Please refer to our main response on "Induction head".
• Q3: "I find the presentation of the paper particularly after Sec 2 confusing".
  • We provide further analyses based on the decomposition and illustrate why nearly orthogonoal (incoherence) structure is useful. We decided to include Section 4.1 to show that the pos-cvec decomposition can lead to better visualization.
  • Section 4.2 shows that the weights exhibit a low-rank + diagonal structure. As far as we know, this provides the first interpretable decomposition of weights that is consistent in pretrained models.
  • Section 4.3 shows that the incoherence structure is quite related to the classical compressed sensing & dictionary learning literature. We provide an analysis via this lens, showing that incoherence does lead to cross interaction in the self-attention being easily captured.

Additional questions.

• "Do you have an intuition/interpretation for the spiral shape?" The spiral shape is related to low-frequency structure and smoothness property. In the top-left part of each subplot in Figure 3, we see that the Gram matrix $G$ is smooth. If we simply take $W$ to be an identity matrix (which is closely related to empirical observation in Section 4.2), then

  $$\mathrm{softmax} \big( x_q^\top W x_k \big) \approx \mathrm{softmax} \big( G_{qk} \big)$$

  gives high values if the positions of query and key are close to the diagnoal part of the Gram matrix. In other words, the attention values are large if and only if $|q-k|$ is small---this is exactly attention to neighboring tokens.

  Note that Gram matrix is used to analyze positional embeddings before [1], but only for the 0-th layer. Our analysis shows (both empirically and theoretically) a connection between Gram matrix and self-attention.

• "Any explanations on the non-spiral trends in Figs 9-11 in the appendix?" We checked the frequency coefficients and the low-frequnecy components are also dominant ones. The specific linear combination of Fourier basis may be different from GPT-2, but the low-freqency and smoothness properties still hold.

• "last paragraph `Why smoothness' of Sec 2". See the first reply in "Additional questions."

• "How the clustering property of the contextual part of embeddings on per document basis is informative?" Our context basis essentially ignores the order of input words. In this light, it is similar to the bag-of-words assumption in NLP before deep learning; see [2]. Clustering encodes the topic information.

[1] Benyou Wang, et al. On position embeddings in bert. In International Conference on Learning Representations, 2020.

[2] Arora, Sanjeev, et al. "Learning topic models--going beyond SVD." 2012 IEEE 53rd annual symposium on foundations of computer science. IEEE, 2012.

We thank the reviewer for reading this submission and providing feedback. We first present our replies to common concerns, and then respond to specific issues.

Common concerns

1. Significance of positional basis. We emphasize that the positional embeddings in many transformers including GPT-2 are trained from scratch during pretraining---which is different from the sinusoidal positional embedding originally proposed. We find this nontrivial because

   • as far as we know, the low-rank and low-frequency structure has not been examined extensively before.
   • The magnitude of the positional basis can change across layers (for aesthetic reasons, Figure 1 is presented as scale-free). We observed the positional basis increases significantly from layer 0 to layer 1. So residual connection is not the sole reason for consistent positional basis.
   • Our discovery helps elucidate confusing artifacts (e.g., spiral clustering in embeddings) people observed; see Section 6.
   • Smoothness is the key to Transformers. Our analysis on an arithmetic task in Section 5 shows a failure example where transformed exhibit discontinuous patterns that result in failure of length generalization.

2. Induction head. One of the most surprising phenomena about LLMs is the emergent abilities, and induction head---which functions as copying even in OOD data---is one simple example. Still, much is unknown about induction head.

   • Our decomposition-inspired analysis identifies 3 interpretable components. The first two (attention to self-token, attention to neighboring tokens) correspond to exactly positional basis or cvec.
   • This ``abstract ability'' learned by transformers may allow people to understand the inner workings of LLMs and lead to practical values.
   • It provides a new visualization idea---which decouples positional effects from non-positional effects.

Specific issues.

• Q1 "Could the authors detail how this new decomposition relates to ANOVA and previous work on positional embeddings and induction heads?" As far as we know, ANOVA used in analysis of transformers is new.

• Q2 "Moreover, what practical benefits does this decomposition provide?" We offer improved interpretations of tranformers---which is important considering the potential risks of LLMs. More specifically,

  • We highlight that smoothness is the key to Transformers by analyzing the positional basis and a failure study on addition tasks where discontinuous structure prevents length generalization.
  • We identify a consistent geometrical structure. This is connected to classical literature in compressed sensing and dictionary learning.
  • We also have a new visualization idea based on our decomposition and apply this idea to induction head.

Additional questions.

• O(1)-sparse representation by bases is defined in the paper in Section 4.3 Equation (11).
• Operator norm (usually denoted by "op") is commonly defined as $||A|| = \sup_{v} \frac{||Av||_2}{||v||_2}$ .
• Thank you for pointing out the typo in Equation (6). It should be $(F_{\le k}B)(F_{\le k}B)^T$.