Towards Best Practices of Activation Patching in Language Models: Metrics and Methods

Thank you for reviewing our work! We address your concerns below.

The work overall is very interesting but it feels a bit light on the novelty. Perhaps proposing additional novel methods for input corruption and improvements for the activation patching techniques can help to increase the novelty in this paper.

We believe that our work is significant, since it gives the first rigorous and comprehensive evaluation of a foundational technique. As we surveyed in the paper and a rebuttal comment above, a large number of recent papers in interpretability have all used activation patching (AP). Yet, many works use somewhat different variants in terms of metric or implementation details. Prior to our work, there was no comprehensive study on the consistency of AP. Our paper fills this gap by (i) rigorously evaluating a wide range of techniques within AP and (ii) providing recommendations for its best practices. Hence, while the main contribution of our work may be unglamorous, we believe it is crucial as it provides the rigor for the field moving forward.

In terms of novelty, our investigation into GN corruption suggests that it may introduce OOD inputs to the model's internal components. We further argue that it may lead to unreliable interpretability claims. To the best of our knowledge, this insight is novel in the literature. It provides an importance evidence for the limitation of the technique.

The way the paper is written it might be a good fit for a workshop.

Our hope was to write a paper that would be of value to practitioners in LM interpretability who use activation patching in their research, as well as of broad interest to the community. We would appreciate any advice on improving the presentation of our paper.

Some terminology could be explained in the paper. E.g. Name Mover It seems that the paper requires prior knowledge of another paper Wang et.al. Some concepts such as: 0.10 negative detection is not very clear.

It might be good to describe clearly what 0.10 negative detection is under the Negative detection of 0.10 under GN section.

Thanks for pointing this out! In the end of the first paragraph of Section 3.2, we added the definition that “we say a detection is negative if the patching effect of the component is negative (under a given metric).” Intuitively, this means that patching suggests the component hurts model performance. We also added Appendix B, an overview of the attention heads in the IOI circuit of GPT-2 small, as found by Wang et al. We hope these clarify the setting of Table 1.

How was the content of Table 1 computed?

As detailed in Section 3.1, we say that a head is detected if its patching effect is 2 standard deviations (SD) away from the mean effect. We have also clarified this in the main text: Table 1 is computed by patching each individual head, calculating the patching effects, listing the heads that are detected (according to the definition above) and contrasting with the results from Wang et al.

Thank you for reading and appreciating our work! We address your questions and concerns below.

My biggest concern is relevance to the community: other than Meng et. al, are there other papers using gaussian noise? It's not clear to me that this is a wide-spread issue.

We remark that the follow-up work of Meng et al (2023) and Hase et al (2023) both use activation patching with Gaussian noise.

In fact, Meng et at (2022), though fairly recent, is quite influential in the community, accumulating nearly 200 citations over the year. Thus, we believe more follow-ups of it may emerge in the future.

Moreover, the recommended course of action (STR) can be more difficult to actually implement.

We agree, but we also remark that this really depends on specific tasks. In the case of IOI, STR is not difficult to implement, as demonstrated by Wang et al. One can simply identify a collection of single-token names and use them for corruption. In the case of factual recall, it takes some effort, but in our work, we have managed to construct a dataset of 145 symmetric pairs of prompts, as detailed in Appendix C.

To be of most relevance to the community, it would be great if the authors suggested an approach that had the flexibility of gaussian noise without introducing as much bias.

One possibility is mean ablation and resample ablation. Here, mean ablation replaces the activation of a model component with the average activation (over a dataset), and resample activation averages the effects of different ablations; see Chan et al 2022 for further discussions.

We have added a note about this in the paper. We leave it as a future direction to compare ablations and activation patching for circuit analysis.

Clarity: Table 1 and in particular the part about negative detection was a bit hard to parse. I would make the discussion around those results more clear.

Thanks for pointing this out! In the end of the first paragraph of Section 3.2, we added the definition that “we say a detection is negative if the patching effect of the component is negative (under a given metric).” Intuitively, this means that patching suggests the component hurts model performance. We also added Appendix B, an overview of the attention heads in the IOI circuit of GPT-2 small, as found by Wang et al. We hope these clarify the setting of Table 1.

How much does different substitutions for STR or different samples of noise (for gaussian noise) change the interpretation? Are the methods at least consistent within themselves?

This is a great question! In section 6, we investigate the effect of varying the corrupted token position in STR. For example, instead of corrupting S2 (as done in section 3), what happens if we corrupt S1 and IO? As shown in Appendix G, the experiments suggest this leads to different attention heads discoveries. Moreover, in Appendix I.3, we experiment with corrupting all S1, S2 and IO by three random names, in the same way as Wang et al. The results are again somewhat different. We believe, however, that all these discoveries are different perspectives on the same circuit and simply complement each other, since different corruptions just trace different information; see Section 6 for a conceptual argument of the point.

For GN, the original work of Meng et al attempt at different noise levels and noise distributions and find pretty consistent results. We reached the same conclusion in our experiments, and hence didn’t delve into this issue again in the paper.

Thank you for evaluating and appreciating our work!

I am not very familiar with the details of the existing activation patching methods. Therefore, I am not sure whether the methods included in the paper are diverse and representative enough for the mechanistic interpretability community.

First, we note that activation patching (AP), as a generic method, is foundational and widely used in mechanistic interpretability. In particular, the third paragraph of our introduction lists 5 prior works that apply the method. We also refer the reviewer to consider the related work section that gives more examples. In total, one can identify over 20 papers using AP or its variants in the literature (which we list in a separate comment below).

Second, the metrics and particular variants considered by our work cover a large fraction of existing techniques in AP. Specifically, this includes techniques used in Meng et al., 2022, Wang et al., 2023, Hanna et al., 2023, Heimersheim & Janiak, 2023 and Stolfo et al., 2023. All of them are studied in detail in our paper. As we surveyed in the related work section and the comment below, these same techniques are used in other works as well. In addition, we also investigate KL divergence as a metric, used in Conmy et al., 2023.

Finally, in section 6, we consider the effect of varying the corrupted token position, a rather neglected aspect in the literature. This again adds to the comprehensiveness of our study.