Optimal ablation for interpretability

Thank you for the positive and helpful feedback, and we’re glad you appreciated our work!

Some of the applications lack depth as a result of the number of different methods studied. For example, it would have been interesting to further investigate the observations on factual recall.

Comment on Figure 2

We agree that it would be great to show further results on factual recall. We’ve extended our analysis to include localization results for different token positions and sliding window sizes as considered by Meng et al. (see PDF attached to global reply), which we will add to the appendix. We also made a few modifications, like increasing our dataset size for factual prompts from 500 to 4,000 and changing the y-axis to reflect a percent (%) difference between original and corrupted probability recovered rather than a difference in percentage points, and added appropriate standard errors to the figures, which is why the new Figure 2 looks different.

There are more directions we could pursue, like human mechanistic interpretability analysis to investigate how the recovered probability occurs, but in general, we acknowledge that we have limited space, and want to show as many different applications of OA as possible. We hope we’ve done enough to show that OA is a promising and versatile method; depth will hopefully come from further studies.

Optimal ablations still appear to perform worse than subtask-specific ablation methods in single-component ablations. However, the authors make a reasonable argument that these are subjective and require human and require manual effort, making them hard to use in some settings.

Note that the only place where OA performs worse than CF is the predictive power of single-component OA loss for inclusion in the manual circuit (the second line of Table 1 in the original submission). However, one element we should point out is that the manual circuit was discovered by using counterfactual ablations to investigate the model, which provides a strong inductive bias in the human studies toward selecting components that create high loss when ablated with CF. Thus, it’s not surprising that single-component CF has better predictive power for this particular circuit.

Instead, the point of this analysis is to confirm that OA produces measurements that are related to CF. To clarify, our stance is that, among previous methods, CF likely best reflects effect 1 (which we loosely consider “ground truth importance”) from the rewritten part of section 2.2 (see global reply), and we will clarify this point in the paper. Thus, it would make sense that, if OA were now the closest method to “ground truth importance,” then OA would also be the method closest to CF among existing methods, and we present the circuit prediction result in Table 1 to confirm this occurrence. However, after further reflection, we think this result may only confuse the reader, since we do not propose to use single-component ablation loss for circuit discovery, and propose to remove it from the paper (while keeping the correlation result).

Resample ablations are almost on par with optimal ablations in the single-component ablation setting (see Table 1), but perform significantly worse than mean and optimal ablations when it comes to making latent predictions (see Figure 3). Do you have a hypothesis as to why this might be the case?

Yes – one general theme is that replacing activations from one sequence with those from another sequence produces the least loss of coherence when the sequences have a similar format or share many tokens in common. The single-component ablation experiment uses IOI, a dataset with 13 similar-looking prompt templates, so we are resampling between sequences that mostly look the same. But in the lens experiments, we are trying to perform latent prediction on the entire OpenWebText dataset, and we are resampling activations between sequences that may have completely different structure and content, hence the significantly worsened performance of resample ablation. This result furthers the notion that OA is more universally applicable than resample ablation and, in particular, can be used for more diverse datasets.

Thank you for the positive and helpful feedback, and we appreciate your optimism about our work!

First, the argument in the paper, while being mathematically clear, is not philosophically rigorous

We agree and have rewritten this section (see global reply). The term “total ablation” clarifies what we consider ablation: replacing the value of component A with a random variable that is independent from the input X, so the new value of A cannot provide any information about X. We think the revised section 2.2 crystallizes why it’s desirable to reduce ∆.

it is not clear that optimal ablation is the answer to an intelligible question. For example, suppose we have a neural model that classifies people into “give loan” and “reject loan” categories, and that we know that a neuron in this model represents a person’s annual income in USD

Interesting example! In this case, we think OA does, in fact, answer an intelligible question: “How much worse does the model perform if it had to treat everyone as having the same income?” which is arguably extremely similar to “How much worse would the model perform if it had no information to distinguish anyone’s income?,” i.e. "How important is information about income to the model's performance?"

it doesn’t make sense to ask how “important” a component is. Importance is not well-defined concept

We agree it’s not clear “importance” is uniquely defined, but we think it’s still helpful to discuss! If we know a component represents income, there are specific interventions we can assess (e.g. input-counterfactual pairs). But if we aren’t sure what it represents, as is typically the case in interpretability work, we may make mistakes like (as you mentioned) changing its value from Alice to Carol without changing a different component that stores the “word starts with A” feature to a “word starts with C” feature, making the model lose coherence (whereas OA could have changed the component from Alice to a value that maintains consistency with whatever other components reflect). Even if we know what each component represents, it’s often too complicated to assess causal interventions between every pair of inputs, and it’s helpful to have a simple conception of importance that somehow aggregates causal effects to guide our search for relevant components a model uses to perform an algorithm. Our claim is that the average causal effect of setting an activation for every input to the same constant adequately summarizes importance (similar claims are made to justify other ablation methods).

while results are plenty promising on the first two case studies, optimal ablation seems to hardly improve over marginal resampling based on Table 1

We were also confused, and when we thought about this experiment more carefully, we realized we implemented the ablation methods inconsistently and were not doing an apples-to-apples comparison. For each ablation method, we can either perform some prescribed intervention at the sequence level, or iterate an intervention over each token position in the sequence (note that Prop 2.3 only holds if we are consistent in this choice). For resample ablation, we were performing sequence-to-sequence replacement by resampling between corresponding token activations in a different input sequence, but for mean/optimal ablation, we were performing token-level replacement by setting activations to the same value at all sequence positions.

We updated Table 1 to adopt the first approach for every method (i.e. conditioning on sequence position), which shows a much larger improvement of OA over resampling.

For completeness, we’ll also show in the appendix that if we implement all ablation methods in the tokenwise fashion (e.g. resampling all tokens individually), OA still shows much higher correlation with CF.

why then, does optimal ablation perform far better than counterfactual patching in Fig. 1, when counterfactual patching was just treated as something of a ground-truth in Sec. 2.

Just to clarify, our stance is that, among previous methods, CF may best reflect effect 1 (which we loosely consider “ground truth importance”) from the rewritten part of section 2.2, and we will clarify this point in the paper. Thus, it would make sense that, if OA were now the closest method to “ground truth importance,” then OA would also be the method closest to CF among existing methods. We present Table 1 to confirm this, and do not mean to present CF as a ground truth.

Even for single components, OA achieves lower ∆ on average than CF (0.0035 vs 0.0296), in line with what we see for circuits. The disparity may be larger for circuits due to ablating many components at once. Rushing and Nanda (2024) showed that models can self-repair if a few components contribute weird values; however, ablating many components with previous methods may contribute much more to effect 2 (“spoofing” the model with inconsistent values).

I really could not tell what the third case study was trying to show

See global reply

the uniform gradient sampling (UGS) method…is introduced without nearly enough detail

See global reply

I would recommend discussing some earlier work on how this entire debate has played out with feature attribution methods

Good suggestion and thanks for the references! We will add discussion.

it’s extremely non-standard for the references section to be full of references that do not appear in the paper

We will correct this and the phrasing corrections mentioned in the questions section.

The limitations could discuss some higher level points [and] could also include caveats about the proposed method from Sec. 2.

We agree and will add these points.

Thank you for the constructive feedback.

Writing and paper structure can be significantly better

We’ve revised the writing so that the structure makes more sense, improving flow and clarifying differences between sections. In particular, we’ve substantially rewritten section 2 to better contextualize our motivation and problem setup, and edited later sections to clarify the motivation and description for each application.

Justification for why ablation loss is the "right" way to measure "optimality" could be better.

We agree and have rewritten this section. See global reply.

it is unclear why making [ablation loss] the primary evaluation metric in the experiments is insightful

The ablation loss gap ∆ is the primary metric in previous circuit discovery work. While it is true that OA is guaranteed to have lower ∆ than zero, mean, and resample ablation, studying ∆ still produces several surprising results.

1. The amount that OA reduces ∆ (67-89% for components, 85-97% for circuits) is very large. In light of our motivation for minimizing ∆ (see global reply), this suggests most of ∆ for these methods is accounted for by “spoofing,” or inserting inconsistent information into the model, not removing important information.

2. OA produces lower ∆ than CF, which is not guaranteed, since CF is input-dependent and removes less information than OA.

3. Manual circuits achieve close to Pareto-optimal ∆ when evaluated with OA, but for other ablation methods, there are similar-size circuits that get much lower ∆ than the manual circuit. This is likely because there are components that do not serve relevant functions according to the manual analysis but need to be included in the circuit to avoid spoofing the circuit components with conflicting information when ablated. If you believe that manual circuits from previous work capture the important mechanisms, then this is evidence that OA is a better evaluation metric for circuits.

We agree with the general idea of having more evaluations. We added a result for OCA lens that shows that it elicits more truthful predictions than tuned lens (see global reply).

does it find smaller circuits that induce the same effect on model behavior?

We believe we show the answer is yes by looking at the Pareto frontier; for the same effect on model behavior, we can achieve smaller circuits. We will make this conclusion more clear in the paper.

Is the second-order effect on unrelated tasks less (as intended) due to optimal ablations?

While this is an interesting question, we’re not sure we should expect circuits to generalize to unrelated tasks. We are not trying to find a small general-purpose circuit; we are deliberately trying to isolate a circuit that performs one particular task, and we might even prefer the circuit does not perform unrelated tasks because we’d have wasted circuit complexity.

uniform gradient sampling method is not described well

See global reply

These applications are niche and specific to mechanistic interpretability subroutines.

Each of our selected applications is the subject of extensive discussion in interpretability. We acknowledge that in the original submission, we only cited a few relevant works. We’ve added far more citations for each of the applications and previous ablation methods.

Furthermore, as other reviewers have noted, our work presents a new approach to the longstanding question of feature importance and has broad implications beyond mechanistic interpretability. We focus on this area specifically for the sake of cohesion and because it is a growing subfield in which the use of ablation is particularly common and for which our method results in concrete empirical improvements.

factual recall (the second application) seems like a special case of the first one

We agree that these applications are mathematically similar, but they are answering substantively different scientific questions about a model, and they are separate lines of inquiry in previous work. Circuit discovery asks how a model performs an algorithmic task, and seeks to find components that perform steps of the algorithm. Factual recall asks where the model has memorized factual associations. We will clarify this point in the paper.

I would be interested to see optimal ablations improve model editing

Thanks for pointing us to this paper, and we agree it would be an interesting result. Our results on localization suggest that our method could do well here. But we think it would require significant extra thought – we’d have to make choices about how to edit the model, compare to various baselines, etc. Given our space limitations, we defer this to future work.

Novelty concerns. I am not sure if the contributions in this paper are strong enough right now, especially given related work in this area, e.g., https://arxiv.org/abs/2309.16042 also focus on systematically evaluating activation patching

As noted by other reviewers, our approach is fundamentally quite different from existing approaches, which manifests in dramatic differences in several important metrics. To be clear, unlike the linked paper, this paper is not about evaluating previous methods. Our evaluation is about showing our method outperforms others, including all methods that are evaluated in that paper.

I like the OCA lens idea, but it's a bit unfair to compare it to tuned lens because OCA lens is still using additional computation via learned ablated layers

See global reply for more context about why it is a fair comparison – both methods elicit predictions from the last token position without using activations at previous token positions. Indeed, OCA lens is actually a more constrained elicitation model than tuned lens: it has a factor of $d_\text{model}/n_\text{layers}$ fewer learnable parameters, since we train a $d_\text{model}$-parameter constant at each layer for OCA lens, compared to a $d_\text{model}^2$-parameter weight matrix for tuned lens.

Thank you for your generous feedback, and for recognizing the value of our work!