Towards Unifying Interpretability and Control: Evaluation via Intervention

Thank you for your suggestions! We appreciate your feedback on how to improve this work and address your concerns below.

Extension to complex, real-world applications and abstract tasks. While we agree that empirical results for more abstract concepts and applications would be highly interesting, we are limited by the feature spaces of the interpretability methods evaluated. In particular, we cannot specify the features learned by SAEs or encoded by Logit Lens or Tuned Lens. Thus, we can only evaluate them for the features they provide us, which are generally low-level token- or word-related features. Furthermore, we believe that if an explanation method doesn’t admit intervention on lower-level features, we cannot expect it to work on the high-level features we might care about in downstream tasks, thus making high performance on low-level features imperative. Additionally, we note that it is easier to measure low-level features due to their objective evaluation (e.g. labeling an output as referring to “yoga” is much easier than labeling an output as “safe”).

We include additional results for two more complex concepts: (1) responding in the French language despite being prompted in English and (2) responding by “yelling” (using all upper-case text) in Section B.3 of the Appendix (page 19). Both of these concepts are not tied to specific tokens, thus making them relatively complex. We find that supervised methods (probes and steering vectors) perform best on these complex features, aligning with our expectations and results in Section 4.6. Even so, these methods are highly suboptimal, and intervention often causes drastic loss in model coherence, despite the studied features being easy to encode through prompting. While results for safety-critical features would be interesting, we were unable to find such features that permitted steering in labeled and publicly available SAEs, and we further believe this application to be beyond the scope of this work.

Human inspection of data/prompts. All of the training datasets and prompts were evaluated by humans and are provided in our accompanying repository for inspection by the reviewers or the public.

Thank you for your suggestions! We appreciate your feedback on how to improve this work and address your concerns below.

Comparability across approaches. Despite the different dimensionalities or columns of the dictionaries learned by each interpretability method, all methods attempt to explain the same latent vectors by using similar features in the same space. More specifically, a single feature of a trained SAE might predict the presence of “San Francisco” in the model’s input text, as would a probe trained on a dataset that encodes for “San Francisco”, and high logit values assigned to San Francisco’s corresponding tokens in Logit Lens and Tuned Lens. Thus methods are still comparable despite their different feature spaces because they all operate on the same latent space. Furthermore, we note that the discrepancies between Ds is what warrants a unified framework for evaluation in the first place - if different methods are all attempting to explain the same feature using different Ds, how can we assess which one is most correct or faithful or which one is best for intervention and control?

Steerability / behavior of the base LLM. In order to ensure that the base LLM itself is steerable, we consider the baseline of simply prompting the LLM to increase the feature of interest (e.g. we ask it to talk more about coffee), as denoted by the teal stars in Fig. 4. If the model is able to be prompted towards a certain behavior, we believe that steering and intervention should also be effective. We found that prompting was more effective than all interpretability methods at intervening on model behavior, thus indicating that poor intervenability is more likely due to the interpretability methods than the LLM. Additionally, we craft our prompt dataset such that steering should be feasible given the open-ended nature of the prompts and the simple intervention topics considered.

More generally, we note that if an explanation method claims that the model is encoding information corresponding to some feature, then steering the model to further increase the presence of that feature should be possible. If such steering were not possible and the model does not actually encode for the feature, the interpretability method itself is likely incorrect and should not have provided an explanation including that feature in the first place.

Definition of coherence. We agree that using humans and LLMs to measure coherence both present different limitations, and we will be sure to expand on the discussion of this limitation in the final paper. However, we note that even if there is slight misalignment between the absolute values of coherence as rated by humans versus coherence as rated by an LLM, what matters more is the relative coherence between the clean LLM’s text and the intervened outputs. We have provided baselines of the coherence of clean outputs in the black lines in Figures 3 and 4 to better calibrate to the LLM rater’s scale of coherence. Furthermore, to validate using LLMs to measure coherence, we present results using a rules-based grammar checker in the supplementary materials of the rebuttal (also discussed below in point ‘Rigorous coherence metric.’ We note that prior literature frequently uses LLMs-as-judges due to the expense of human annotators, which is what motivated our use of LLMs to measure coherence.

Undefined notation. Thank you for noting this. We will be sure to clarify this notation in the final version of the paper, where x is the latent representation of the model and theta corresponds to the learned weights of the probe.

Fig. 1 Clarity. Thank you for bringing this to our attention. We are happy to make the changes suggested by the reviewer and will accordingly move the table further down in the paper.

Counterfactual prompts. We note that the prompts used to train the steering vectors and probes are not meant to be counterfactual but contrastive, as is standard for the steering vector method we implement (Contrastive Activation Addition [1,2]). We use the term contrastive in the paper and make no claims of counterfactuality because it is difficult, if not impossible, to create the types of counterfactual datasets the reviewer is suggesting. For example, if we take 10 sentences about San Francisco, it is likely that replacing the name “San Francisco” with any other city name would yield a factually incorrect sentence (e.g. “San Francisco is home to the Golden Gate Bridge” → “New York is home to the Golden Gate Bridge”). As such, we cannot only edit the phrase of interest to create a counterfactual prompt, but we must instead generate a contrastive prompt that does not contain the same feature. We apologize for not correcting reviewer LqMe’s use of the word counterfactual. We further note that this issue of potential confounders or correlations in the data trained to use steering vectors and probes is why they are fundamentally less causal in nature, as is frequently noted in past literature, including many of the citations in [3]. We acknowledge this in our paper, including in Section 4.6.

Causal mediation analysis. To clarify any confusion regarding this topic, in our discussion with reviewer LqMe, we “acknowledge that Intervention Success Rate can be thought of as an extension of IIA to feature-based interpretability methods.” We think of our work as working towards the suggestions for future work noted in Section 7.2.4 of [3], particularly an evaluation of interpretability methods for intervention. As noted by [3], many of the methods in 4.3 of [3] that extract bases of human-interpretable features are not systematically evaluated for intervention, and in fact these are the very methods that we evaluate in our paper (cited in [3] as Turner et al. (2023), Chen et al. (2024b), Templeton et al. (2024), Cunningham et al (2024)). As we noted to reviewer LqMe, we will be sure to include further discussion of causal mediation analysis in the final work, which was previously limited due to space constraints. We reiterate that the main goal of our work is to evaluate existing methods empirically, and that our framework is simply a method to standardize intervention across methods with different encoders, feature spaces, and decoders. We do not claim to be the first to propose evaluation via causal intervention, but that to our knowledge, we are first to present a standardized, systematic empirical study testing various existing methods for this ability.

Furthermore, we note that [3] was only put on ArXiv in August 2024, and we were unaware of this work at the time of submission. As according to ICLR policy, works published after July 1, 2024 and any non peer-reviewed conference proceedings or journals need not be cited by authors, as they are considered “contemporaneous.” Nonetheless, we will be sure to discuss this concurrent work in the final version of our paper.

[1] Panickssery, Nina, et al. "Steering llama 2 via contrastive activation addition." arXiv preprint arXiv:2312.06681 (2023).

[2] Ghandeharioun, Asma, et al. "Who's asking? User personas and the mechanics of latent misalignment." arXiv preprint arXiv:2406.12094 (2024).

[3] Mueller, Aaron, et al. "The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability." arXiv preprint arXiv:2408.01416 (2024).

Please let us know if you have any remaining questions. Otherwise, if you feel your concerns were adequately addressed, we ask you to consider raising your score. Thank you!

Thank you for your suggestions! We appreciate your feedback on how to improve this work and address your concerns below.

Related work, novelty, and position relative to causal interpretability methods. We thank you for pointing us to this review paper! Mueller et al. (2024) succinctly describes many of the motivations for our paper - mainly that while mechanistic interpretability began as a field that leveraged causal interventions to create mechanistic or causal abstractions of models, many popular interpretability methods now aim to create human-interpretable feature-based explanations of model representations by relying on correlational statistics rather than causal theory (e.g. Sparse Autoencoders, Logit Lens, probes). This is frequently noted in Mueller et al. (2024) and Saphra et al. (2024), and both papers call for better evaluation and more standard benchmarks for these methods (see Section 7.2 of Meuller et al. where they note that "currently, most studies develop ad-hoc evaluations” and that more "standard benchmarks for measuring progress in mechanistic interpretability" are needed). We believe this work to be a step towards such systematic evaluations.

We apologize for any confusion our wording may have caused, as we did not intend to say that we are the first to consider interventions for interpretability, but that to the best of our knowledge, we are first to present a systematic evaluation of various popular interpretability methods for intervention. We will be sure to update our language and make more references to past mechanistic interpretability work that focuses on causal interventions, including the citations provided by the reviewer, which were unfortunately previously omitted due to limited space.

Interchange intervention accuracy. Interchange Intervention Accuracy measures the extent to which the functionality of one model can be induced in another model, when their latent variables or activations are interchanged. This allows for making a causal claim regarding the effect of the latent variables. In contrast, Intervention Success Rate focuses on interventions made to human-interpretable explanations of model latent representations. While both metrics aim to measure causal effects on the outputs of the model, the input space for both varies quite drastically. Furthermore, while interchange intervention works on uninterpretable latent vectors, our intervention success rate is applicable to only “interpretable” latent vectors, allowing us to evaluate their interpretability and utility for model intervention. We are happy to note the similarities between Intervention Success Rate and IIA in the final paper, and we acknowledge that Intervention Success Rate can be thought of as an extension of IIA to feature-based interpretability methods.

Definitions of evaluation metrics. We apologize for the brevity of the descriptions of each evaluation metric due to the space constraints of the paper. We will provide more rigorous definitions of the metrics in the final version of the paper, including the additional coherence metric of the number of grammatical errors (proposed by reviewer 1BCv).

Number of methods. In total, we evaluate four interpretability methods: {SAEs, Logit Lens, Tuned Lens, probes} and one control (but not interpretability) baseline: {steering vectors}, There are only three methods in Table 1, as noted in lines 349-351, because only SAEs, Logit Lens, and Tuned Lens can be used to fully reconstruct the model’s latent vectors. As such, we cannot perform the reconstruction sanity check on steering vectors and probes, and they are omitted from the table.

Counterfactual prompts. We apologize for the confusion. The prompts provided in 316-318 are not the counterfactual prompts themselves but rather the prompts we used to generate the counterfactual datasets with an LLM. The full counterfactual datasets for each feature used to train the steering vectors and probes are provided in the code repository given in the supplementary material. For example, some positive prompts for the “dog” feature are {“The dog jumped into the pool,” “The dog howled at the moon,” “The dog had expressive eyes”} and some negative prompts are {“The pelican dived into the water,” “The wolf howled at the moon,” “The owl blinked its big eyes”}.

Thank you for your suggestions! We appreciate your feedback on how to improve this work and address your concerns below.

Lack of experiments on latest/larger models. We agree that the models we experiment on are relatively small; however, we were bound by the availability of trained, labeled SAEs and trained Tuned Lenses. Recently, SAELens released a trained and labeled SAE for a single layer of Llama3-8b, which we provide results for in Section B.1 of the Appendix, Rebuttal Supplementary Materials. Results on Llama3-8b are very similar to those for the models we included in the main paper, with Logit Lens and Tuned Lens outperforming the other intervention and interpretability methods, but underperforming prompting. Unfortunately, no trained SAEs/Tuned Lenses exist for larger variants (e.g. 70b-200b). We note that our framework should easily apply to larger models, and we have provided code that can be used for any models, lenses, and SAEs that may be released in the future.

Table 1 results missing. We apologize for the confusion. The results for Gemma2-2b Tuned Lens and Llama2-7b SAE are missing because there are no publicly available trained SAEs/Tuned Lenses for those two models. Furthermore, as noted in lines 349-352, we cannot perform the reconstruction sanity check with steering vectors and probes as they do not generate complete explanations and thus cannot be inverted back to the original latent vector. We will be sure to reiterate this in the table caption.

Low- vs high-level features. While we believe that intervention and control over high-level features is the end goal of many interpretability methods, we think that being able to intervene on simple, low-level features is a prerequisite to intervening on complex, high-level features. We also note that these are not necessarily distinct: we find anecdotally that upon steering the model toward a low-level feature (e.g. the word “yoga”), the topic of conversation also changes accordingly toward the underlying high-level concepts it represents.

Furthermore, we note that we are limited in our ability to pick arbitrary or interesting features by the interpretability methods evaluated, in particular, because SAEs and lens-based methods do not allow for the specification of features. However, we present additional results for the features (1) responding in the French language (with English prompts) and (2) responding by yelling (writing in upper-case text) in Appendix Section B.3, Rebuttal Supplementary Materials. We find that, as expected, lens-based methods do not scale well to abstract features, and that steering vectors and probes are more performant than SAEs. However, overall, we see that the relationship between intervenability and coherence remains suboptimal, aligning with the existing results and conclusions of the paper.

Task-related performance. Given that LLMs are used for a variety of downstream tasks but are generally built/tuned for chat performance, we believe that coherence is the best metric to maintain task-agnosticity, ensuring that the model retains its language modeling capability (i.e. its ability to generate coherent text without grammatical errors) after intervention. While we agree that using our framework to have fine-grained evaluation on various specialized downstream tasks is an interesting problem, we believe it to be beyond the scope of this paper.

Concept drift. If we correctly interpret your question to mean asking whether the mapping from the model’s latent space to the interpretable latent space will be affected by the concept drift resulting from fine-tuning/editing with guardrails, then it depends on the explanation method used within the framework. For instance, because Logit Lens only uses the unembedding weight of the language model, no retraining would be required to maintain accurate interventions, as the unembedding matrix would have been updated during the model’s finetuning itself. However, probes and SAEs would have to be trained on the activations of the finetuned LM to still be valid. Thus, this limitation is not one of our framework, but of the underlying interpretability methods that may require additional data.