The study is limited to fairly small transformers (GPT2-small, and Pythia-160m in appendix), and it’s hard to know whether the results will generalize to more complicated tasks or methods will scale to larger models. Additionally, It would also be helpful to know whether larger models also struggle with the item-recall task (and whether they have similar inhibition-mover subcircuits).

Thank you for raising this. Please see our rebuttal which primarily focuses on this point. In summary, yes larger models struggle with this task despite the simplicity and we argue this is an isolated mechanism that is part of a very basic and general language modeling mechanism learned by the model.

The composition score presented in Equation 1 should be presented in context (i.e. seems to have come from Elhage, et al. [3]? Is there earlier use of this score elsewhere?). Besides substituting individual SVD components in for QK or OV matrices, are there any other changes compared to how [3] measure the composition score?

As far as we know, the composition score does not appear anywhere before Elhage et al. And that is correct, this iso our only modification to get it to work

It is unclear to me whether the failure mode of GPT2 small on the item-recall task is due to (a) not correctly identifying duplicated objects, or (b) not suppressing correctly identified duplicated objects when going to copy. This is because Figure 4 shows the attention pattern of mover heads can be influenced by either duplicate token head channels or inhibition head channels. Is this because the duplicate token channel influences the inhibition signal? (i.e. its effect on the mover head is mediated by the inhibition head?)

The reviewer is correct in their last statement, it is because the duplicate token head affects the inhibition head, which affects the mover head. This is established in the circuit identified by Wang et al., 2023. If this is still confusing to the reviewer we can go into more detail. Also to be clear, when we do the edit here, we are only editing the duplicate token subspace at the point the inhibition head sees it, so it is following the logic of duplicate token head-->inhibition head-->mover and not duplicate token head-->mover head (where --> is read as 'affects'/'changes'). We expect a lot of familiarity with previous work, which is not entirely fair to all readers, so to remedy this we added a section to the appendix outlining the entire IOI circuit in GPT2 from that paper (as well as the circuit we find for IOI on pythia).

When you say a head is “highly active” on a prompt or task [e.g. Line 229, 255], what is meant by that? Is this measured by its attention pattern or something else? When would an attention head be "non-active"?

Yes this is measured by the attention pattern. We apologize for the lack of details, please see the rebuttal for an operationalization of these terms and how this analysis was carried out. They will be included in the final draft.

As one of the other well-studied/well-known subcircuits, have you done any analysis to understand whether the induction subcircuit is also dominated by a low-rank subspace? Or is this specific to the inhibition-mover circuit you study?

We also looked at the induction head in Figure 6 (left) and Appendix H and found that the one that we looked at was not low rank, so not every head composition is like this. However this is not specific to the inhibition-mover/duplicate-inhibition subcircuits because we found plenty of other compositions between random heads that were similarly sparse. Unfortunately, we didn't have space to also analyze those and assign function to them because they are not part of any known circuits (like the IOI components). This is a very interesting direction for future work. We'd be happy to discuss more with the reviewer about what information we could/should include regarding this, though.

In section 5, how does trying to learn optimal singular vector weightings either via gradient descent or regression compare to doing a grid search over 3-d points?

This is a very cool idea that we did not consider when writing/have time for in the rebuttal. We kept the 3-d points analysis extremely basic. Would this involve optimizing directly in that 3d space? Please re-raise this point during the discussion if the reviewer finds this an interesting addition to the paper.

We hope we have resolved all of the reviewer's concerns which we believe we properly addressed in the rebuttal document or here, by clarifying details from the paper.

Thank you for considering the rebuttal and other reviews, and suggestion to accept. We agree with the remaining points on related work and will make sure adequate space is provided in the camera ready.

Thank you for the review. We are happy that the reviewer enjoyed the paper and we appreciate the encouraging comments regarding static weight analysis, we think there is a lot of interesting work to be done in this area.

The proposed composition score does not work for models using relative positional embeddings like RoPE,

We'd like to point out that this is only for query/key composition and does not occur for composition between values and outputs. Pythia uses RoPE and we cover positive results on this model using composition in the appendix. We use the composition score to do static circuit analysis on part of the Pythia IOI circuit, which was not previously known before (which we later validate with path patching). See the below point. See also the rebuttal for more info.

Can the proposed composition score be used to identify compositions of attention heads without prior hypotheses?

Yes absolutely, the compositions can be taken between pairs of heads, and outliers can be searched for. From there, new connections can be found and later analyzed on data. We had to do IOI circuit analysis on pythia to find inhibition heads, and the composition score was used for part of this. After starting the circuit analysis and finding the inhibition heads, the value composition score revealed the induction heads that connect the circuit, which path patching later confirmed

The results could be significantly strengthened if the technique was used to conduct an unsupervised search for head compositions within the model, rather than focusing solely on the IOI task with a known circuit.

Thank you for raising this. We definitely hear the reviewer on this, but we're not sure how we could fit this in without it disappearing in the appendix. We hope that the above point about our analysis on pythia without a known circuit is satisfactory for the scope of this paper.

Could you elaborate on the intervention used to improve model performance on the laundry list task? Specifically, what operations do you perform to "set the model components in a certain area of the 3D space"?

Yes, in plain english, each inhibition head is responsible for one direction (we focus on three of them so that we can visualize the space without dimensionality reduction). We overwrite the output of each head to be some coordinate in its corresponding direction. That is, some scalar times the unit vector corresponding to the inhibition component we previously identified. A very important part of this intervention is that it makes it impossible for the head to attend to any of the previous context (the Q*K operation becomes unused), which is why we are able to say it is content independent: we control which item in the context is being attended to/ignored without letting those heads attend to them

Thank you for the review. We are very happy to hear that the reviewer found the results interesting, promising for facilitating future work, and the paper easy to follow. The reviewer had some concrete questions and concerns about the generalizability of the method to new models and results to new tasks. We believe we have satisfactory answers to these below:

The limitations of the method do not allow for a straightforward implementation on newer models with relative positional embeddings.

We would like to point out that this is only for composition with queries and keys in models with RoPE embeddings, in case it wasn't clear. Value-Output composition is entirely unaffected, and we have results with Pythia (a RoPE model) in Figure 12. This is certainly still a limitation, but we believe in the future it will be surmountable. In addition, composition with mlps (mlp to attn, attn to mlp, and mlp to mlp) is still a valid path for future work that we didn’t explore here, so we don’t think that our method will be limited to only certain kinds of models. Given the complexity of the mechanisms involved already, we figured it would be too much to find a satisfactory workaround to the query/key problem, work with MLPs, and provide our current analysis all in one paper. We'd also like to point the reviewer to the rebuttal which addresses some related points.

The proposed method of Communication Channel Interventions improves model performance on a synthetic task (Laundry List); however, it is not clarified how such modification will affect model's performance on other tasks.

We would argue that the mechanism we uncover is an extremely general component in broader language modeling capabilities and possibly affects most tasks that involve some form of recall. Our evidence for this is the activity on both the IOI and Laundry list tasks (which would seem to coincidental otherwise), the content independence of the mechanism (see Line 281), and the consistent general activity on open domain text (see rebuttal document). In our rebuttal document we outline some evidence for this idea

We will use the rest of the space to answer other questions:

How many heads were affected by the interventions in your experiments in Section 5? Do you have information on how the increase in performance depends on the number of modified heads?

We use three of the four inhibition heads (7.9, 8.6, 8.10). We chose to leave out the fourth (7.3) because we wanted to be able to plot the results in 3D without dimensionality reduction (Figure 5) since that often can’t be done. As for why we chose 7.3: per Line 182: “changing 7.3 does not have a strong effect on its own” (see Figure 2). We saw some marginal increase by also including 7.3 but it was computationally expensive to run the full sweep. Based on the evidence we do have, the scores would probably be at baseline if we performed an analogous intervention on three random heads, though.

You identify several types of attention heads: mover, duplicator, and inhibitor, but could a single head exhibit traits of multiple types? Also, is this classification not exhaustive (i.e., there are heads that do not fall into any of the aforementioned categories)?

We did not find, but don’t rule out that a single head could exhibit traits of multiple types. This classification is not exhaustive, and we do briefly study induction heads in the appendix. Still, we are excited by recent and evolving work in characterizing other types of observed specializations in heads (e.g., successor heads).

Thank you for the review. We are glad the reviewer found the methods interesting and well-motivated but disappointed the paper was difficult to read. This is a complicated topic that needs to fit in 9 pages, so it is difficult to catch everyone up and make a significant contribution in that time. Below, we answer questions, outline some proposed changes, and would also like to use this space to push back on the notion that we are overclaiming. After considering our points we would like to hear if the reviewer still feels this way/what they specifically feel is being overclaimed.

“Understanding Inter-Layer Communication”: Exactly what is the scope of this inter-layer communication? Is it primarily/always from layer l−1 to l, or something more?

We do not find a preference for more recent layers. Please see Figure 2: heads in layer 7 communicate with the mover head in layer 9. The duplicate token head (3.0, see Figure 4) communicates with inhibition head 7.9 (a difference of four layers).

“using subspaces we call communication channels” (Intro): this terminology already existed in Elhage et al. (2021)

We use this term to ground the paper to previously established literature but also provide a more specific definition rather than a notion of communication. Elhage et al., is extensively cited in this work, but this point is well taken because it was not our intention to appear to coin this term. This will be reframed in the camera ready to refer to this paper when introducing “communication channels”.

“QK/OV circuits (e.g., which weight matrices are we talking about exactly? why are they “low-rank””

These are low-rank because of their shapes. For example, in one head, the value matrix projects from the embedding dimension (768) to the head dimension (64). E.g., the value matrix is 64x768 dims. This is therefore lower rank than the embedding dim. This has been consistent in the transformer architecture since Vaswani et al,. 2017 so we do expect some familiarity from a general reader. If the reviewer agrees it would be helpful, we would be more than happy to define these terms in a glossary in the appendix.

Relatedly, we’d like to point out this isn’t a typo:

Page 4, line 128 (d \times d)

We will add something like “V \in d x d_h and O \in d_h x d, so OV \in (d \times d)” to make this very clear

There’s also a related issue of overusing informal analogies unnecessarily (attention heads “talking” to each other).

We tried to balance formalisms with intuitive analogies due to the complexity of the work. If the reviewer could, we would be interested in hearing which specific places more rigorous definitions would be helpful.

“all the overclaims in the paper really hurt the overall message”

We are generally a bit confused about what the reviewer finds to be overclaimed. Perhaps the above points help clarify our work more. If not we would appreciate some specifics if the reviewer still feels this point is relevant after the rebuttal.

In terms of the proposed Laundry List task, a possible caveat is that models more recent (and larger) than GPT2-Small may actually do a good enough job of solving them.

This is a great suggestion, please see our rebuttal doc. We tested multiple n-billion parameter models (up to 6.9B) from different model families and found that the larger/more recent models tend to get better but still struggle. We’d also like to point out some relevant related work: Liu et al., 2023: highlights the broader inability of LMs to use information when it’s in the middle of a long context. This negatively affects retrieval augmented generation (RAG) systems, which are highly relevant to current SotA systems. We don’t claim this is a failure of the exact same mechanism, but it definitely connects our work to a substantial current problem.

This point may also help answer the following:

”[a] confusion for me is: generally, what implications do we have about inter-layer communication in Transformers from these findings?”

Our work points to a concrete capacity limit in inter-layer communication that leads to performance degradation on certain recall tasks. This work provides a promising avenue for understanding and improving “lost in the middle (Liu at al) type failures.

“Page 7, line 227: In what sense are these results “strong”? Is there a baseline?”

Yes there is a baseline. In Makelov et al., 2023, they examine the IOI dataset. As a reminder, we report 97.5% and 35% with components directly taken from the weights. Their baseline achieves -8% FLDD 0.0% Interchange. By taking the gradient to directly optimize a single vector for this task they achieve 111.5% FLDD and 45.1% Interchange.

“In the intervention experiments, the choices of the exact subspace (the number of components) and the intervention method (e.g., scaling diagonally for 2D) appear a bit heuristic and need better justification.”

We do have simple explanations for these:

Intervention: We wanted to be able to plot the results in 3D without dimensionality reduction (Figure 5) because we thought it was very interesting to be able to do that. We chose to leave out 7.3 specifically, per Line 182: “changing 7.3 does not have a strong effect on its own” (see Figure 2). We entirely agree that this point is just too covert in the current draft, so we will raise this decision when introducing the interv. experiment

Choice of scaling: We use z score thresholding on the distribution of composition scores (>4) to choose which components to scale. On all components this led to choosing 1D except for the duplicate token head which was 2D. This is why we used diagonal scaling. The graphs in Figures 6 and 7 show these components compared to others, and we believe it’s agreeable that these are also visually very clear outliers and the choice is justifiable.

“It is well known that the matrix of the attention head is sparse, which naturally suggests using Singular Value Decomposition (SVD) to compress the original matrix… the conclusion that inter-layer communication is low-rank seems evident due to the inherent low rank of the attention head weight matrix”

Attention heads are inherently low rank because of the size of the matrices (e.g., 64x768, projecting down to 64 dims). This is much different from the low rank subspaces we identify here, which are 1d or 2d. We disagree with the claim that this is somehow self-evident, because we find that not all compositions between communicating matrices have this property. For example, see FIgure 6, left. We know that previous token head 4.11 composes with induction head 5.5 from prior circuit analysis (Wang et al., 2023), but they seem to do so with nearly the full-rank space of the attention head.

“Numerous related works, such as LoRA, have applied SVD to the original model weights.”

Does the above point address the concern about the connection to LoRA? If not, could the reviewer please expand on this? We are familiar with some of the work on LoRA that is relevant but we are not aware of any work that implies the results we find in this paper.

“the instructions provided in lines [148-152] are too brief, making the specific method difficult to understand.”

Thanks for raising this. We take the distribution of composition scores between component matrices and another matrix (such as mover head 9.9) and take the highest z score components (the outliers). We do not determine an optimal threshold in this work, but we use >4 (upon visual inspection the outliers are extremely obvious: see Figure 6 middle and right).

“In Figure 2, the Inhibition Score is presented, but it is not introduced until Section 3.3.1, which may confuse readers. The use of ( V ) in Equation 2 may confuse readers with the ( V ) matrix in the Transformer model. “

Thanks for pointing this out, we will address this in the camera ready by introducing the term earlier.

“This article merely reinforces that the inhibition head functions as an inhibitory mechanism” No other work has been able to find a known circuit within a model solely from the weights without running the model. We think this is a significant contribution to the field. We focus on a well known circuit to establish credibility without too much overhead of verifying a new circuit.

We localize the inhibition mechanism to a few dimensions in the attention heads. We establish that traversing this space (by intervening on the outputs of these heads) controls the position of the token inhibited completely independently of the content of the token. That is, the edit we make is ‘ignorant’ to the preceding context, yet the downstream effect is predictable. This has not been established by previous literature

“this article does not propose a new mechanism for inter-layer communication in the Transformer model.”

The new mechanism is the way that this is implemented in the model, with extremely low rank signals. ‘Bandwidth’ in the residual stream has been hypothesized as being in high demand (Elhage et al., 2021), but it has not been established how these signals are passed. We thoroughly justify the claim that this is one such method native to the transformer (see point 1).

We believe we have clarified the points raised by the reviewer. This paper makes heavy use of the appendix for supporting figures, which we know reviewers are not necessarily responsible for, so we have provided some pointers here. We hope the reviewer considers these in their determination of the paper.