Transformer Circuit Evaluation Metrics Are Not Robust

We thank the reviewer for their detailed and encouraging feedback. We are glad they believe the paper will be a valuable contribution.

The reviewer presents an interesting perspective on the correct resolution to debates around faithfulness metrics. There are a range of views on this topic in the literature. For instance [1], (as cited by reviewer W9hJ), rejects explanations that enable users to make better decisions with the help of the model as a poor metric for the faithfulness of the explanation. Instead, they claim that this is merely “indicative of correlation between the plausibility of the explanations and the model’s performance.” This is not a debate which we feel our paper needs to resolve. As explained in our comments to other reviewers, our primary motivation is to critique the faithfulness claims of previous papers and warn against over-optimizing particular metrics.

We do however believe that better evaluation is important. This is a perspective shared by others, e.g. [2]. One example of a potential improvement to evaluation methodology could be to test the circuit across a very wide distribution, as compared with the narrow distribution used to discover it, to find whether it implements the same behavior more broadly. In particular, some of the authors are currently investigating a method which would ablate the circuit and test whether the tokens with the greatest decrease in loss across a large dataset are instances of the task that the circuit is supposed to perform.

Alternatively, the interpretability community has had some limited recent success in applying insights on 'features’ or 'representations’ to better control models [3], which is an example downstream application. It is an interesting question to ask whether the same may be possible for entire computational circuits. We suspect it will be found to be significantly harder. Concretely, it may be possible to leverage circuit-based insights to perform highly specific capability unlearning [4] or parameter efficient finetuning [5], as a generalization of recent work attempting to use features to do the same.

[1] https://arxiv.org/abs/2004.03685 [2] https://www.alignmentforum.org/s/a6ne2ve5uturEEQK7/p/f8nd9F7dL9SxueLFA [3] https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html [4] https://arxiv.org/abs/2403.03218 [5] https://arxiv.org/abs/2404.03592

Thank you for the detailed feedback.

We defer to the definitions of faithfulness used by the authors that discovered the circuits we study. These papers consider faithfulness to be the similarity of the explanation (in this case circuit) to the full model on the task - this is “fidelity” in the language of [1] - but they use a variety of metrics for their exact definition. We will include further discussion of this, and put this in the context of other definitions of faithfulness in the NLP literature in our revised paper.

We thought that diagrams would provide the clearest explanation for our discussions of ablation methodology, but we will also include mathematical descriptions for further clarity (using do-calculus notation). We will add the formula for logit difference recovered and appreciate that this may be easier to understand than our verbal description. A table summarizing prior methodological choices is a good idea, thank you for the suggestion.

We agree that Zhang and Nanda (2024) is highly relevant and we do cite it. They perform experiments studying different aspects of ablation methodology. Importantly, they do not arrive at our notion that circuit faithfulness can only be defined by specifying the circuit evaluation and ablation method. We agree these differences should be enumerated in the paper.

On the question of our initial assumptions, we wish to clarify that we are not arguing that faithfulness metrics “should” be robust to methodological choices, we are merely noting that they are not, because prior work implicitly assumed that optimal circuits could be defined by tasks only. We agree that some of the wording choices are too strong and that these can be made more precise. In particular, we like the suggestion of recommending that researchers describe their choices clearly, and use more rigorous and precise language to clarify the exact claims they support. We agree that “sensitive” is a better description than “not robust”, and will adjust wording where possible.

We have benchmarked our library against the most popular circuit discovery library [2] and found it to be much faster. We will include this graph in the final paper.

Differentiating by token position means considering a prompt of n-tokens to have n parallel computational graphs, that interact only through the attention mechanism. The graphs can be ablated independently.

[1] https://arxiv.org/abs/2004.03685 [2] https://github.com/ArthurConmy/Automatic-Circuit-Discovery

Thank you for the detailed feedback.

We agree that certain methodological points are omitted. We will include a more thorough description of methods at the beginning of each experiment. Regarding the specific points you raise:

• In our work, we test the evaluation metrics used by prior circuit discovery work. As such, we study the very same circuits found by those works, which are often in GPT-2. We follow [1] here, and many of the circuits that we study are canonical in the MI literature e.g. [2, 3, 4]. We agree that the model choice is another important dimension of circuit discovery, but disagree that the model is a choice to be made when evaluating some particular circuit. Note that we do perform experiments in the appendix on circuits found in different models (e.g. Pythia 2.8B).
• We will include a mathematical description of logit difference recovered. A logit diff greater than 100% means the logit diff output by the circuit is greater than the logit diff of the full model, which is suboptimal because it means that the circuit is not producing the same output as the full model.

Regarding Section 3.1.5, it is true that the correctness of the circuit is largely independent of the model performance on a task. The aim of localizing model behavior requires only that the circuit reproduces the model’s behavior of the task, not that the circuit performs well at the task. As mentioned in our comment to reviewer W9hJ, our use of faithfulness here is following previous work.

Ablating the complement of a circuit is valuable because it tests whether the circuit is sufficient to perform the task by eliminating any useful computations that might be done outside of the circuit. Ablating the circuit itself on the other hand is coarse – for instance, just ablating one important edge in a circuit may be sufficient to destroy performance.

Thank you for the idea to run a statistical significance test. The difference between Mean and Resample Ablation was significant when ablating nodes (p=1e-5), but not edges. We will include this result.

The IOI authors do consider the possibility of two distinct circuits that are combined with an OR operation in a late layer (in section 4.1 of their paper), but as they do not themselves notice the difference between the ABBA and BABA prompts, they do not propose this hypothesis.

[1] https://arxiv.org/abs/2304.14997 [2] https://arxiv.org/abs/2405.08366 [3] https://arxiv.org/abs/2310.04625 [4] https://arxiv.org/abs/2309.16042

Thank you for the detailed feedback. We apologize for the list of typos and will ensure to correct them.

To clarify the point on the Pareto frontier of faithfulness, completeness and simplicity, we are pointing out that if you consider the neural network itself as an explanation or interpretation for the neural network, this is actually perfectly faithful and complete. However it is not a useful description because it is too complex for humans to understand. We argue that there exist many optimal circuits, which make different tradeoffs between faithfulness, completeness and simplicity.

As stated in Section 3.1 “we consider activation patching to be a specific type of ablation, and use the term Resample Ablation interchangeably.” By MLP (multi layer perceptron) and QKV (attention head queries, keys and values) we refer to internal model activations of the transformer based architectures we consider. We will include a mathematical description of the transformer architecture.

We believe that Wang et al. would have done better to evaluate their IOI circuit at an edge level, because this is the level at which their circuit is specified. We perform this experiment and compare the result to the node-level circuit, which is what they measure.

In the IOI prompt format (e.g. “When A and B went to the shops, A bought milk for B” (ABAB)), the names are arbitrary, so BABA is the same as ABAB and ABBA is equal to BAAB. Our choice of prompt formats follows prior work [1]. We will clarify that we are not arbitrarily omitting an ordering of names.

We do not think it is very strange that we observe lower faithfulness scores with Resample Ablations than Mean Ablations. Conceptually, resample ablations induce the network to more strongly predict a particular wrong answer, whereas the mean activation attempts to simply destroy the signal of the clean prompt. We agree that methods may not necessarily be inconsistent if relative ordering is maintained. As we evaluate single circuits in this work (and don’t discover circuits ourselves), we do not study this relative ordering of circuits, but our null hypothesis is that all methodological choices matter. Also, absolute faithfulness scores are interpretable and meaningful.

Our recommendations for researchers are fairly limited as the focus of our paper is to highlight problems in existing circuit discovery research. See rebuttal to reviewer 3bug for further discussion of this point.

[1] https://arxiv.org/abs/2211.00593