We thank the reviewer for their helpful review of our paper.

Weaknesses:

1. Thank you for your suggestion about Figure 4. There is some overplotting in these figures, and we will improve them. Regarding Figures 11-17, indeed, they are proposed to be viewed in a separate file. We plan to release a Cytoscape file with Figures 11-17, which will enable them to be visualized interactively.
2. This is a very good point, thank you for pointing it out. We did not think about it, and indeed, using a rank correlation such as Spearman correlation, could improve the clustering. We will definitely investigate that in the revision.

Questions:

1. Good catch; we will fix those in the revision.
2. Also helpful and we will fix the figure order in the revision.
3. Thank you for the question. Indeed, it is quite surprising that a few signals are responsible for so many attention weights.
   To explain, we start by noting that what is happening in all these cases is attention sink – meaning that the head is attending to the first token (or punctuation token) because it does not have any important function to perform. In fact, for most of the attention heads and token pairs, the model only performs the attention sink operation. In other words, only a small subset of heads and token pairs are actually important for computing the model output in these tasks. This is perhaps the underlying cause of the effect.
   We connect this fact with the observation (from [19]) that when models represent features in superposition, they will tend to allocate a dedicated dimension for features that are used frequently. Because attention sink is so common, the model seems to allocate a dedicated, global feature direction to control this behavior. The simplicity of this function, which is merely to direct attention to the first token, could also explain why a single direction suffices.
4. We have preliminary evidence that some of the signals our method identifies carry semantics that are shared across different attention heads. For example, we have found that in the IOI task [42], different Duplicate Token Heads and Induction Heads write the exact same signal (same one-dimensional subspace) to be read by S-inhibition heads. Another example is the control signals we identified, which appear to represent consistent, meaningful instructions that are shared across multiple heads and layers. However, the extent to which signals in general are shared remains an open question. A more in-depth study of their semantics is an exciting direction for future work that we are eager to pursue.

We thank the reviewer for their helpful review of our paper.

Weaknesses:

• Thank you for pointing out this lack of clarity. These plots are over all the prompts, and over all the attention heads and respective token pairs (d, s) that appeared in a trace from our tracing algorithm. We see that our use of the term “relevant” didn’t clearly specify what they are. We will change the manuscript to make this part clearer.
• We can see that there are multiple kinds of counterfactuals that can get confused here. We discuss this bullet point and how we will clarify the issues as part of our response to the third “Questions” bullet below.
• Thank you for pointing this out. We will add a citation for the Courant-Fischer theorem. Regarding the “ersatz function”, another term would be “replacement function”. We can see that that term would be a bit clearer so we’ll make that change in the revision.
• Yes, the method scales well. All SVDs for a model can be computed just once and reused for every prompt. The bottleneck of this method is the same as the attention mechanism, which is quadratic in the number of tokens. We have a small discussion about this in the Appendix (lines 802-808).

Questions:

• We address the first question in the first bullet under Weaknesses above.
• Yes, something odd is going on in that case. What is definite is that our method is causal in the attention weights – this is a provable fact. In other words, if we intervene in the position (d, s), the boosting and removing interventions increase and decrease the attention weight in the (d, s) position, respectively.
  So what we conclude for this case is that increasing the attention weight on the token pair in the downstream head actually decreases the performance of the model. We did not investigate further, but one possible explanation is that what this attention head in the (d, s) position is adding to the residual stream is information about suppressing the correct prediction (something similar to the Negative Name Mover heads in the IOI paper). Therefore, boosting this signal will make the performance worse, while ablating it will make the performance better.
• Thanks for pointing out the need for clarification regarding “counterfactual inputs” along with the comparisons being made between alternatives at lines 249-251 and lines 759-771.

In the style of circuit tracing that is most common in the literature (causal mediation analysis) a counterfactual input is a prompt that is identical to an original input, except that it differs in the single feature under study. For example, the paper [42] – which was the first to trace the IOI circuit – uses counterfactual inputs that vary a name across three different versions of a prompt. Inputs, therefore, come in pairs—the original and the counterfactual—and these pairs form the basis of their causal mediation analysis method. For instance, for the IOI task, the original and counterfactual inputs could be:

• Original input: “When Mary and John went to the store, John gave a drink to”.
• Counterfactual input: “When Mary and John went to the store, Ryan gave a drink to”.

When we state that our method does not require “counterfactual inputs” (e.g., at lines 9-10), we are referring to this specific type of paired, minimally-different prompt As we note, constructing counterfactuals to test hypotheses about circuits can in many cases be a subtle and challenging task (eg, see [46]).

We also use the term “counterfactual” in a different context, in which we are discussing a hypothetical situation in which a particular token representation could have been different. This is the sense on lines 26, 33, and 42 and particularly on line 141. The intent in those passages is to highlight that the signals we identify are causal for a head’s attention. For example, at line 141 we are establishing causality by showing that, had a signal not been present, the head would not have attended to the token pair. These passages are not considering actual counterfactual token pairs, but rather imagined modifications of a single token.

Lines 249-251 are expanding on this hypothetical situation. These lines are providing a mathematical tool for assessing the impact on attention if a token had been different. So here, we are again not using actual counterfactual token pairs. Instead in these lines and those that follow, we are calculating the effects of an imagined modification to a token, in which an upstream component’s contribution to the token had been removed.

Turning to lines 759 - 767, here we are describing a method for measuring model performance. Again, for the IOI task, one can evaluate a model's performance by measuring the logit difference between the correct name (the IO name, "Mary") and the incorrect name (the S name, "John"). These logits are available after a single forward pass, and comparing them serves as a success metric. In all of our tasks, we use a success metric based on the logit difference between “success tokens” and “failure tokens.” For the GP task, the success token is either “he” or “she,” while the corresponding failure token is the other pronoun. Our use of these metrics is consistent with the extensive literature on these tasks.

Lines 768 - 771 are connecting the success metric with a “success direction” in activation space at the model output. For any success metric there is a corresponding direction, akin to an initial “signal”, that can be used to start our tracing method.

That said, we note that our method does not fundamentally rely on this evaluation scheme. We can trace any linear combination of the unembedding vectors, that is, any direction in the logit space, by defining a corresponding “success direction” (line 769). For instance, we could trace the direction corresponding to the top-k tokens for any given prompt.

We appreciate the opportunity to tease out these different threads, and these questions have been helpful in exposing the need for clearer language around counterfactuals, which we will add to the paper.

• Thank you for making this important connection. The concept of "binding" is indeed relevant to our work. We view the challenge of correctly associating attributes with entities as a crucial application of the more general attention-causal mechanisms we study. As the reviewer correctly notes, the four works cited identify specific low-rank subspaces that control these binding functions. We hypothesize that these "binding subspaces" are the concrete, circuit-level implementation of the causal mechanism our work aims to identify. Therefore, we expect our method not only to find similar low-dimensional directions but also to provide a mechanistic explanation for how attention heads manipulate information within these subspaces to achieve binding. We agree this connection deserves a more detailed discussion, which we will add to the related work section of the paper.

Other: Minor Typos

• Thank you for pointing these out. We will correct them in the revision.

Thank you for the response, particularly related to explaining your various uses of "counterfactual" and how they differ throughout the paper. It is more clear to me now and hope some of these changes will be incorporated into the final paper.

I have also read through the other reviewers' responses and rebuttals and I think the proposed changes (specifically in discussion with reviewer Zss5) will be helpful for increasing the clarity of the paper. With the rebuttal response addressing my concerns, I am inclined to leave my score as Accept.

We thank the reviewer for their helpful review of our paper.

Weaknesses:

1. Thanks for pointing this out. We’ll provide a clearer definition of $S^{\ell ads}$ in the revision. Specifically, each term in (3) is the amount that one singular vector adds to the relative attention $c^{\ell ads}$. We take the terms from (3) and sort them in decreasing order, and find the shortest initial sequence of this list that sums to $c^{\ell ads}$. Because most terms in (3) are quite small due to sparse decomposition, our method gives a small number of singular vectors. The indices of these singular vectors are $S^{\ell ads}$.
2. Thank you for pointing out the lack of explanation of notation. We will add an explanation of this notation to the paper in the location you pointed out. Briefly, the sets {$f^i$}, {$g^i$} are intended to represent sets of features present in a token. Each feature, eg, $f^0$, is assumed to be a single direction in activation space, following the linear representation hypothesis. We go into more detail in our next answer immediately below.
3. We appreciate the reviewer’s interest in theoretical justification for sparse decomposition of attention scores, which we share. We recognize that, due to space limitations, we were not able to provide a more detailed explanation of why attention scores should show sparse decomposition.

We start from the linear representation hypothesis, which holds that important features are often encoded in one-dimensional or low-dimensional subspaces. We provide a number of cites in the introduction to papers that demonstrate this phenomenon for an enormous variety of different kinds of features [12, 19-29]. Consider a specific feature, such as the fact that a token corresponds to a person’s name. This feature may be represented by a 1-D direction in activation space. We denote such a direction $f^0$. Hence we can think of the model activation carrying this feature as $x = b + f^0$.

Consider an attention head, one of whose roles is to attend to the personal-name feature. It calculates its attention score as $x^\top \Omega y$. Then the process of model training will tune $\Omega$ and $f^0$ so that a left singular vector of $\Omega$ will tend to be in the same direction as $f^0$. This is the significance of Lemma 2. Note that the other singular vectors cannot lie in the same direction as $f^0$, because singular vectors form an orthogonal set. Hence, if the personal-name feature were the only feature that the head attended to, then decomposing the attention score $x^\top \Omega y$ in the singular vectors of $\Omega$ would yield a sparse decomposition. That is, the one singular vector that is in the direction of $f^0$ would yield a large contribution to the attention score, and the other directions would not contribute significantly.

To understand how this extends to the situation where there are multiple features that the head attends to, we note that previous work [19] has shown that features that co-occur will naturally tend to be represented by orthogonal or nearly-orthogonal directions. That is, features such as “personal name”, “place name”, “organization name” will tend to be represented by vectors $f^0, f^1, f^2$ which are mutually orthogonal or nearly so. Then by the same argument, model training will tend to align different left singular vectors of $\Omega$ with each of $f^0, f^1, f^2$, again leading to a sparse decomposition of attention when one of those features is present in the input $x$.

Finally, in the general case where more features are important to the attention head than there are available singular vectors (dimensions $R$), the tendency will be for features to be approximately orthogonal as explained in [19]. In that case, each feature will tend to align with a small number of singular vectors of $\Omega$, still leading to a sparse decomposition of attention when one of those features is detected.

The above is not a formal proof, which would involve defining a distribution over features and a model of the training process, and would be outside the scope of our paper. Rather it is an explanation of why sparse attention decomposition can arise. A formal proof, along with experiments that demonstrate the emergence of sparse decomposition during the training process, is a subject of our current ongoing work.

Moreover, we note that the authors in [58] make assumptions that essentially follow the reasoning we present above, although they do not lay out the reasoning in detail as we do here.

4. Indeed, these descriptions are missing from the body of the paper. The tasks were described in Appendix J due to the lack of space, but we will add short descriptions of the tasks to the body of the paper to aid the reader.

5. Thank you for pointing this out. In attention rollout, the identities of input tokens are linearly combined through the layers based on the attention weights. Thus attention rollout is a heuristic method to approximately track information flow through the model. It does not attempt to identify causal communication. Rather, being a heuristic, it is typically used as a tool to make general observations about what tokens a model is attending to. In contrast, our method identifies the specific low-dimensional components of a token that are responsible for the fact that a head attends to the token. Accordingly, our method provides a more precise and reliable understanding of why an attention head attends to a particular token pair.

We agree that a more complete methodological comparison with attention rollout and other internal tracing methods would clarify our contributions. We also agree that having a more intuitive explanation of the entire methodology would make the paper more accessible and easier to understand. As discussed in our response to Reviewer 1, we believe we can use space currently allocated to Figure 1 to address these needs.

Questions:

1. We address this question in answer #2 (and #3) above.
2. Using the definition of $S^{\ell ads}$ that we gave in answer #1 above, the x-axis of Figure 2 is $\frac{|S^{\ell ads}|}{R}$, where $R$ is the dimensionality of the attention heads (line 93) and consequently the rank of the $\Omega^{\ell a}$ matrix. Therefore, the point of Figure 2 is that, for multiple models and tasks, the attention weights are sparsely decomposable in the singular vectors, almost always using a very small number of singular vectors.
3. The x-axis of the dendrogram represents all the control signals (D-dimensional vectors) used in a forward pass that were identified by our selection process (lines 332-333). The colors are used to represent the cluster that the control signal falls in, where clustering is based on cosine similarity.
4. We appreciate the opportunity to clarify how using attention-causal communication improves over previous methods.
   With respect to circuit tracing, the dominant strategies used for circuit tracing are activation patching or path-patching [7, 8, 11, 42-44]. These methods are still extremely time consuming and require the careful and subtle construction of counterfactuals to expose circuits. Each time one seeks a circuit for a new task, one must carefully design counterfactual task instances and then run many forward passes of the model, moving activations from the “corrupted” execution to the “clean” execution. Importantly, there are cases where activation patching can fail - instances where downstream behaviors by the model “undo” the effects of the patching.

Our method does not use activation patching, and so avoids problems in which downstream components “undo” the effects of the patching. Further, it allows for tracing without counterfactuals; the only prompt needed is the one that is to be traced. Finally, our method performs the trace in a single forward pass – which is much more efficient.

As we discuss in our response to Reviewer Zss5 above, our paper also improves on the tracing method in [51]. Briefly, the method in [51] does not work for models in general – it only worked in [51] for the single case of GPT-2. In fact, we have found that the method in [51] does not work for Pythia. Our method is based on a formal problem that we solve in a provable way, and thus works on any model (we show its success on GPT-2, Pythia, and Gemma-2).

Typos and Wording Suggestions:

• Thank you for pointing these out. We will correct them in the revision.

Thank you for the detailed rebuttal, which helped clarify some of my initial confusion. After reviewing the response and other reviews, I agree with Reviewer Zss5 that substantial revisions are needed both in rewriting the text and improving the figures. In its current form, the paper is not ready for publication. In particular, the lack of intuitive/theoretically grounded explanation limits the accessibility/rigor of the work. In addition, the absence of comparisons with relevant methods reduces the empirical soundness. While I see potential in the direction, the clarity, rigor, and completeness of the presentation need substantial improvement. I will keep my score and encourage the authors to improve the clarity and soundness of the work based on the feedback provided.

We thank the reviewer for their careful read of our paper and the helpful comments. We appreciate the opportunity to clarify the points raised.

First we’ll address the main thrust of the reviewer’s concerns and then we’ll address the detailed points individually.
The reviewer is concerned that our paper does not break new ground over [50, 51, and 58]. The closest paper to ours is [51], so we’ll start there.

As the reviewer notes, [51] provides a data dependent measure for selecting singular vectors, which then exposes meaningful communication. The strategy taken in [51] is to select a set of singular vectors that contribute an arbitrary amount (70% of total) to attention scores (ie, pre-softmax scores – not attention itself).

First of all, the signals found in [51] have no provable effect on attention itself. Nothing can be said about how much of a head’s attention is actually caused by these signals. That is, there is no formal basis for, nor is there any provable result from the signal identification method used in [51]. As a result, there is no assurance about the nature of the set of signals found in [51] – there may be both false negatives and false positives with respect to attention causality, for example.

In contrast, our paper starts with a formal problem definition of attention-causal communication (a concept that is a novel contribution of our paper) and then presents an algorithm that provably identifies attention-causal communication. This conceptual advance is important, because it provides a formal definition for what researchers seek in understanding why an attention head attends to a particular token pair. See, for example, the recent blog post “Progress on Attention” in which prominent researchers point to this question as a key open problem in LLM interpretability (“Our biggest remaining question is how to understand why models attend where they do.”).

Second, and related to this flaw, the method of [51] does not work in general. For example, it cannot be used on Pythia. We have worked with the method of [51] from the authors’ code and tried to use it on Pythia. We find that the key idea of [51] is based on assumptions that do not hold in general, and do not hold in Pythia specifically. This is perhaps not surprising for an approach with no formal justification. To explain, the key problem is that many attention scores in Pythia are actually negative. There is no way to apply the approach of [51] to situations where the attention scores are negative. The method of [51] fundamentally assumes that moving the attention score of a token pair towards zero will decrease the attention on the pair – but this is not true for negative attention scores.

Our development of relative attention as the proper measure for selecting singular vectors is therefore a critical advance over [51], and part of the reason it was possible for us to apply our method to all three of GPT-2, Pythia, and Gemma-2 (whereas [51] only worked with GPT-2). Furthermore, the decision to define and use relative attention is not obvious; it’s important that relative attention (a linear function) has a provable relationship with attention (a nonlinear function) as shown in Lemma 1. This fact is key to being able to attribute attention causality for a particular head to components upstream of that head, which relies on the linearity of relative attention.

The reviewer states that our paper “does not demonstrate why we should prefer” our paper over [51]. This is a fair point. Thanks to the reviewer’s comments we realize we should have made these points clearly in the manuscript and we will definitely make these points clearly in the revision.

Turning to paper [50], indeed that paper shows the existence of low-rank communication channels between attention heads. It is important to observe that [50] only provides one concrete example for the IOI task. This example is in GPT-2 and is a particular communication from four inhibition heads to one name mover head. Importantly, as the paper admits, the authors were looking for this channel because they knew from prior work that the inhibition heads communicate with the name mover. This example underscores the fact that the method used by the authors in [50] is entirely manual and heuristic, based on examining plots involving pre-selected pairs of heads. The point is that paper [50] does not provide, or attempt to provide, a general or automatic method for identifying all the important communication occurring in the IOI task, nor does it attempt to provide a general method for identifying which attention heads are communicating. In contrast, our paper presents a systematic, general method, with provable results, for finding all the attention-causal communication present, and identifying all the heads that are contributing, and it does so without assuming that circuits have been traced in advance (unlike [50]). We mention here again that the analysis of the IOI task in GPT-2 does not generalize to other models without making use of significant new ideas (such as relative attention) that we contribute.

Next, regarding paper [58], indeed that paper focuses on semantic feature matching in vision transformers, performed using singular vectors. The main goal in [58] is to show that vision transformers do not just attend to similar objects, but also (in later layers) perform contextual analysis. As such, we believe that [58] is complementary to our work. The authors in [58] are focused on exploring the meaning of visual features that correspond to singular vectors. Our work does not dig into the semantics of features but instead focuses on systematically extracting all the relevant features (signals) that are causal for attention. On the other hand, the fact that [58] shows that so many features found using singular vectors of QK matrices are interpretable adds motivation to our method, since we show how to extract those (attention-causal) features systematically in a manner that is robust across models and tasks.

Regarding the bullet points:

• Lots of jargon and formalisms mask that the paper essentially is a rehash of several prior works. While I think there are some interesting extensions on prior work [50, 51, 58], this paper provides some clarifying formalisms only to replicate known findings.
  We appreciate the opportunity to clarify the ways in which our work advances the field. We don’t claim that we are the first to observe the importance of singular vectors of QK matrices as tools for interpretability. Indeed, we cite [50, 51, 58] as important prior work showing that attention heads communicate in low-dimensional channels identifiable using singular vectors of QK matrices. The focus of our contribution in Section 4 is on showing how to turn this observation into a formalized tracing strategy that is more efficient and easier to use than prior methods (eg, more efficient than using counterfactuals and patching) and that has provable properties and is robust across model architectures (unlike [51]).
• Singular vectors of attention weight matrices encode distinct meaningful signals across layers [50, 51, 58]
  Indeed, as discussed above, this fact is an important background for our paper; we don’t claim this as a contribution of our paper.
• Specifically, the IOI circuit is solved by communication across singular vectors in attention heads [50, 51]
  While both [50] and [51] look at the IOI task via singular vectors, neither offer a robust and systematic solution, as explained above. The other tasks we study (GP, GT) have not been addressed before, and showing that the singular vector method works well for those as well is important evidence for the robustness of the method.
• [51] expands on the ideas of prior work by providing a data dependent measure for selecting singular vectors. This work provides their own version of this, but does not demonstrate why we should prefer there's and offers no novel insight
  As discussed above, we understand now that we did not explain in our submission why our method is a distinct advance over [51]. We explain that above, and plan to clearly articulate our advance over [51] in the revision.
• The "signals in the stream" analogy is just a rewording of findings from [18] and by extension, the analogy of low rank communication channels in [50]
  Understood. Removing it from the title would focus the title better on our paper’s contribution, which we will do in the revision.
• Section 4: "Tracing circuits" is not a novel contribution and the language around circuit tracing is very similar to the "circuit tracing" and results described in [51] and does not expand on it meaningfully
  While much prior work has focused on tracing circuits, the dominant strategies are still extremely time consuming and require the careful and subtle construction of counterfactuals to expose circuits. Tracing without counterfactuals, and in a single forward pass, offers the field a much more efficient tool.
  As described above, our work goes beyond [51] by providing a formal problem definition based on the new concept of attention-causal communication, presenting a provable and efficient solution to that problem, and developing new concepts (relative attention) needed for working with models beyond GPT-2.
• Distinct semantic feature matching performed by singular vectors of [58 <- vision models]. A lot of the analysis resembles [58]
  As described above, as [58] focuses on interpreting the features found via singular vector analysis of QK matrices, we see it as complementary to our work, and providing motivation for the approach we take.
• Regarding Figure 1, we appreciate the reviewer’s frank appraisal that this figure doesn’t work. This is helpful criticism, and we can see in that light that Figure 1 should be removed and the space used more profitably.