InversionView: A General-Purpose Method for Reading Information from Neural Activations

Dear reviewer fZCU,

Thanks for your feedback on our paper!

Reply Regarding Questions

Questions 1-3: writing and demonstrating suggestions

Thanks for your suggestions. Most importantly, we have created two figures describing the decoder and the workflow, included in the global rebuttal PDF. We will also address your other two suggestions.

It seems like you needed the "tailored methods" to know where to apply inversionView in the first place, or am I misunderstanding sth?

Yes, there is a misunderstanding. There are two key parts in Wang et al. [39]: They (1) use path patching to identify important components, and then (2) study the function of these components using tailored methods. By "avoiding the need for tailored methods", we refer to (2). In order to know where to apply InversionView, we only need (1), i.e., a general-purpose method for finding circuits.

For example, [39] use path patching to identify certain heads ("S-Inhibition heads") affecting Name Mover head queries, and then use tailored patching experiments to show that they output both token and position signals. These experiments were designed specifically to disentangle these two effects, ablating or inverting token or position information. On the other hand, InversionView directly reads out these two kinds of information (Figure 17 shows an example for an S-Inhibition head containing position information), obviating the need for guessing possible information and tailoring patching experiments.

We will add this specific example to the paper.

Reply Regarding Weaknesses

Weaknesses 1-6: Writing and Presentation

Thanks for your suggestions, which we appreciate and which we will implement.

Key points:

• We will provide more specific information about our studies in the abstract.

• The two new figures described above will make it easy for the reader to quickly grasp how the method works.

• We will change the sentence in line 28 to "which in turn helps us identify how the information flows through the model. This is crucial to obtain the algorithm implemented by the model. "

• We respectfully submit that we do have formally described the method/decoder in Appendix C (decoder architecture), D.1, E.1, F.1 (decoder training details for individual tasks), and A.2 (sampling details). We will expand these sections with more discussion.

• We will revise the description of results to provide more high-level intuition. We would be grateful for any more detailed guidance.

The approach seems related to feature visualization/ adversarial example and counterfactual example generation

Thanks for pointing out other methods that also generate in input space. We explain the difference between InversionView and these methods below. We will add this comparison to the paper.

Feature visualization generates inputs that maximally activate a certain neural network unit, while InversionView finds inputs that result in the same vector. Whereas feature visualization interprets an individual neural network unit (e.g., neurons) to understand its general role across inputs, InversionView interprets specific values of inner representations of neural networks. When the input changes, the value and thus the interpretation may change.

Adversarial example or counterfactual example generation methods generate input that is similar to the original input but results in different outcome. While similar in input space, the adversarial/counterfactual input is likely to be quite different in internal representation space, leading to a different output. In contrast, we are interested in how different inputs in input space are represented very similarly in internal representation space.

Reply Regarding Limitations

On SOTA models, inversionView might become prone to human tendency for pattern matching

We would like to argue that pattern matching is fundamental for interpretation. Pattern matching is essential when interpreting neurons, studying the function of an attention head, or, as done in our paper, inspecting preimages. Neural activations may contain arbitrarily complex information, hard to capture by a fixed set of templates. Therefore, analysis by an intelligent entity (artificial or biological), capable of identifying novel patterns, is necessary. As we discuss in our paper, LLMs are useful but not good enough to replace humans at this task of pattern matching yet.

Moreover, InversionView is a method for generating hypotheses, not final interpretations. Generated hypotheses must be verified with intervention experiments, as we do in the paper. An incorrect hypothesis due to an error in pattern matching will be identified at this stage, ruling out dependence on subjectivity in pattern matching.

Polysemanticity might be very relevant for InversionView results but is not discussed

Thanks for raising this point. Polysemanticity is widely observed when one wants to find a unified interpretation across all inputs for a model component such as a neuron -- in contrast, InversionView interprets a specific activation (vector) on a specific input. Nonetheless, concepts similar to Polysemanticity may be defined in our setting. Specifically, we observe cases where the same (activation site, position) pair encodes different kinds of information on different inputs -- a certain kind of "polysemanticity". For example, in the factual recall task (see global rebuttal), some heads' outputs encode subject information on some inputs, and relation information on other inputs. InversionView is very helpful for studying this kind of "polysemanticity", because it decodes per-example information instead of average information. We will discuss in the paper.

Related work is not discussed in sufficient detail

We will add work related to feature visualization / adversarial example. We request the reviewer to suggest any other related work they find missing from the paper. We would be happy to include it too.

Dear reviewer P3CM,

Thanks for your feedback on our paper!

Reply Regarding Questions

It is possible to use other distance metrics besides L2-based metrics? Why do you believe the L2 distance is effective in this context? Is there something unique about the geometric space?

Yes, we think it's possible to use other distance metrics. Since our work explores a novel direction, we would like to start with simple and common metrics, in order to show that the idea works without complicated metrics.

Therefore, we use L2 distance because of its simplicity and straightforward intuition, so that readers can easily grasp the idea and realize how much we can learn from the geometric space, instead of regarding L2 distance as the optimal choice.

In addition, in the paper we show L2-based metrics are effective empirically, as the information obtained using the simple L2-based metrics is well supported by causal and quantitative experiments.

Reply Regarding Weaknesses

Applying the method to extremely large models and more complex tasks is challenging

We agree that interpretation of extremely large models tends to be more challenging in general. On large models and complex tasks, a potential challenge is the complexity of the information itself, but this can in principle be overcome by scaling the decoder so it can learn more complex inverse mappings, and basing the decoder on pretrained language models can help. We show the feasibility of this approach in the factual recall task on GPT2-XL (see global rebuttal), where we read information from a model 10x larger than in the IOI case study.

Selecting an appropriate distance metric and epsilon is challenging and requires tuning

As we mentioned above, we show that the method works with the most common and simple distance metric. There was in fact no need at all for effortful tuning of these aspects -- though using more sophisticated metrics could be a topic for follow-up work. Regarding the threshold $\epsilon$, we simply choose a small threshold for which the preimage shows meaningful patterns. Note that, unlike neural network training, where people train models repeatedly to tune hyperparameters, in the preimage returned by InversionView, one can immediately see possible interpretations resulting from different thresholds. We have more detailed discussion regarding the threshold, and our method's robustness to it, in Appendix A.4.

In summary, the fact that simple distance metric and threshold choice work highlights the robustness of our method.

Dear reviewer vTBG,

Thanks for your feedback on our paper!

Reply Regarding Questions

In line 115，what does $\mathbf{z}$ represent?

Sorry for not writing it clearly. The $\mathbf{z}$ is the same as $f(\mathbf{x})$ in the previous part of the paper. So it simply represents an arbitrary activation that is compared with the query activation $\mathbf{z}^q$. We will make it clear in the next version.

In line 108, is this input the same as the query input in figure 1?

Yes, "input" here refers to the same as the query input in Figure 1. It's a sequence of tokens. Note that we use the modifier "query" in the context of comparing it to its neighbourhood during interpration. Thus, in line 108, we do not use "query" because the input is not mentioned in the context of a neighbourhood.

Does this neighbour of geometry hypothesis always hold across all LLM activations?

Because transformers define continuous functions, the hypothesis should hold in principle. Of course, that doesn't guarantee it will always hold in a practically meaningful sense. As experimentally demonstrated in our paper, the hypothesis holds for the conducted case studies.

Reply Regarding Weaknesses

Another figure to demonstrate the whole training and evaluation pipeline

Thanks for your suggestion. We have made a new figure, which we have included in the global rebuttal PDF, and which we will add it to the paper in the next version.

Need large labour resources to extend to large-scale LLMs

1. Neural activations can contain any kinds of information, so that any kind of template-based interpretation is likely not expressive enough. InversionView allows decoding various kinds of information. We believe manual or LLM-based interpretation is likely necessary given the variability of the information that can be encoded.
2. InversionView is naturally suitable for using LLMs to automate interpretation, because it produces samples in input space that LLM can easily read, rather than abstract data structures.
3. As our experiment (Appendix J Automated Interpretability) shows, LLMs can detect the main information in the preimage, despite occasional hallucination. We use manual investigation throughout in order to ensure correctness, since this is a scientific paper. We firmly believe that automated interpretation can be further improved in the future with better prompt engineering and better LLMs.
4. Note that, as InversionView is a method for reading information from neural activations, it facilitates reverse-engineering (which by nature requires a lot of work) but it does not require reverse-engineering to work. Our paper aims to show that InversionView is one of the tools in the ecosystem of interpretability research, rather than an all-inclusive solution. In the paper, we reverse-engineer models and provide causal verification in order to show the correctness of the information given by InversionView. The number of samples one needs to inspect to interpret a single activation vector does not necessarily increase with the model size, as we show in the case of factual recall task. For larger models, one may use it to study a specific part of the model.
5. Exhaustive interpretation or reverse-engineering requires a lot of work -- this is a feature of mechanistic interpretability research in general. InversionView is a faster way to generate accurate hypotheses. When combined with existing methods, InversionView makes the overall workflow more efficient, thereby decreasing guesswork and promoting faster iteration cycles in mechanistic interpretability research.

Dear reviewer Mv2t,

Thanks for your feedback on our paper!

Reply Regarding Weaknesses

The precise algorithm for the proposed method is unclear

In the global rebuttal, we provide a new figure showing the training and sampling pipelines. In its caption, we also provide detailed explanation.

If you are unclear about the the decoder. We also provide a new figure showing decoder architecture in global rebuttal. The decoder model is basically a decoder-only transformer combined with some additional MLP layers. In order to condition the decoder on the query activation, the query activation is first passed through a stack of MLP layers to decode information depending on the activation sites and then made available to each attention layer of the transformer part of the decoder. There are also details about it in Appendix C in the paper (we should have linked section 2.2 to that).

We will add the new figures to the next version of the paper. Taken together, these changes will make the precise algorithm much more accessible to the reader.

How the proposed method scales with larger models

Thank you for raising this point. Please refer to the global rebuttal, where we describe our new experiment on a larger model, with 10x more parameters than in the IOI case study. We find that InversionView continues to produces interpretable results, and allows us to read out interesting information content.

Thank you. We are grateful for your support.