Can We Partially Rewrite Transformers in Natural Language?

(Weakness 1) Is there a good reason to use an LLM as a judge to predict the SAE activations?

Yes. Explanations are meant to enable a human or an LLM to better predict the activations of an SAE. Other works have shown that LLMs are about as good as humans at generating explanations, and at using those explanations to make predictions about whether latents are active or not in a given context. It would be infeasible to use human labor for this task, and using an LLM ensures consistency of the predictions across features and examples.

(Weakness 1) The submission would benefit from more clearly distilling what the findings are; the current conclusions along the lines of "we find that current sparse coding techniques and automated interpretability pipelines are not up to the task of rewriting even a single layer of a transformer" are too general given the experimental setup.

As we stated in response to reviewer TeiC, there is a definite upper limit to explanation quality, as long as we restrict ourselves to reasonably short explanations. In Interpretability as Compression, Ayonrinde et al. (2024) emphasized the importance of restricting or penalizing the length of an explanation when evaluating its quality, since a sufficiently long and detailed explanation could perfectly capture the activation pattern of a feature in a trivial way. Hence, while our pipeline is imperfect, we do not believe there is sufficient room for improvement to completely overturn our negative results.

(Weakness 2) If the authors find that the LLM is not able to predict the SAE activations, why would one expect that the only problem of the LLM is merely over-predicting the activations and that monotonous rescaling would yield anything useful?

We directly show in Figure 3 (left panel) that the LLM dramatically over-predicts the activation values before quantile normalization is applied. We saw this in early experiments and deduced that quantile normalization would fix this problem— which it indeed does (Figure 3, right panel).

(Weakness 3) How can we trust the SAE training to have been 'successful' in the sense that it yields similar results to those reported in the literature? Without establishing this, any statement about SAEs that goes beyond this specific study is hard to justify given the fact that SAEs are difficult to train properly.

We use the standard, widely used open source library called sparsify (https://github.com/EleutherAI/sparsify) to perform our transcoder training. Its default hyperparameters are known to produce reasonably high quality sparse coders. We can update the manuscript with numbers related to the final reconstruction loss, but the increase of CE loss shown in Figure 2, blue bar, should be sufficient evidence that the SAE was trained ‘successfully’, as these numbers are similar to the ones reported in the literature.

(Weakness 1-4) In general, I do not find the experimental setup very convincing and would highly appreciate if the authors could clearly motivate why this specific approach would make sense to explore.

Automated interpretability using an LLM as a judge is a well-established technique for evaluating the interpretability of SAE features; see for instance Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet by Templeton et al. (2024) and Automatically Interpreting Millions of Features in Large Language Models by Paulo et al. (2024). Our experimental setup is also grounded in the stated goals of the mechanistic interpretability field, which has always aimed at reverse engineering neural networks. For example, the 2024 ICML workshop, accessible at https://icml2024mi.pages.dev/, states: “One emerging approach for understanding the internals of neural networks is mechanistic interpretability: reverse engineering the algorithms implemented by neural networks into human-understandable mechanisms.”

(Weakness 5) I would appreciate if the authors could more clearly explain what aspects of their work they believe to be the root cause of the negative results, in particular in relation to other works that have shown promising results when it comes to either activation patching or sparse autoencoders.

No prior work has performed experiments similar to ours, so we do not believe that our negative results are in tension with prior work that may seem more positive about the prospects of sparse coders. While previous work on transcoders and SAEs show that one is often able to extract superficially interpretable features from LLM activations, features are rarely completely interpretable, and in fact a significant fraction of them can’t be easily understood. Our results stress tested the assumption that latents are easily interpreted by looking at their activation patterns. We believe that our negative results stem from deep facts about how large language models work. There is no guarantee that the collection of all of the features of transcoders and SAEs, filtered through their natural language explanations, can capture all the information in the original activations. Explanations are always incomplete, and sparse coders almost always have non-negligible reconstruction errors. It is likely true, as Hubert Dreyfus argued starting in the 1960s, that intelligence— whether natural or artificial— cannot be decomposed into a list of independent atomic features.

Have you experimented with few-shot prompts for the simulation? I would expect the calibration to be better if the model would have been presented a couple examples.

Our prompt currently shows 3 example sentences and their real activation values, but it is possible that having a larger number of few shot examples, which allowed the model to estimate the average firing frequency would help, although that would mean that on 100 examples shown, none should be active. We also noticed that explicitly telling the model that latents fire very infrequently helped make the predictions better calibrated.

You mention that you only explain latents that are active on more than 200 instances (L89), but report substitution of latents up to 100% in Figure 2. Does that mean you found more than 200 instances for all latents, or does the 100% in Figure 2 refer to all latents for which you were able to generate explanations? If the latter, how many latents were not active on at least 200 instances?

When reporting 100% in Figure 2 we mean 100% of the ones that fire more than 200 instances. Around 0.5% of features don’t fire more than 200 times.

The evaluation is limited to two small language models; Pythia-160 M and SmolLM-2 (125 M). Smaller models tend to learn lower-quality representations than their larger counterparts, so performance might improve with a larger model.

We agree this is an important limitation of our study. However, a significantly larger language model would have required a concomitantly larger computational budget. Our pipeline is, unfortunately, already quite expensive as-is.

The presentation in the paper is a little informal, and its readability could be somewhat improved. For instance, I'd suggest including titles on the left and right panels in Figures 2 and 3, or using bold when you refer to the left or right subplots.

Thank you for this suggestion. We will be sure to add titles to those subpanels and use bold to make it clear which subpanel we are referring to.

The main result of the paper is not particularly deep. I appreciated the quantile normalization analysis and also Section 4.1 on the specificity of the explanations. But I do wish that there was a little bit more of a takeaway. Do you have any guesses about why the current auto-interp pipeline fails? Ideas for how to improve it? It would be ideal if the paper had a bit more to say. What else can you say about what this result means for auto-interp or for interpretability more broadly?

We think the current auto-interp pipeline probably fails for deep reasons. It is likely fundamentally impossible to translate the inner workings of a large language model into interpretable operations on independent, atomic features. This point has been argued by philosophers and cognitive scientists like Hubert Dreyfus for decades.

On the other hand, incremental improvements may be possible with concerted work in two directions. One direction is that of improving the reconstruction error of sparse coders with architectural improvements. The other is to use more compute for generating explanations. Generating more explanations per feature, or using iterative methods to generate explanations might get us further improvements.

Do you think that your overall negative result is more likely to reflect the limitations of auto-interp or the limitations of SAEs themselves? I wonder whether, if we had stronger LLMs, which could, as agents, run experiments to test different hypotheses about latent explanations, or if we gave the simulator and explainer model other information (for instance about other internal latents in earlier network layers), whether something like this could eventually work?

It is possible that, by making the explanations sufficiently long and detailed, we could eventually get significantly better results. Trivially, a very long explanation could perfectly capture the activation pattern of a latent by listing the weights of the base model and the transcoder, and describing how to compute the latent using these weights. But such an explanation would be useless. At the present time we can’t precisely estimate how much of an improvement these approaches can bring, but we have not had too much success with this approach, although it is possible that that might be due to the extreme costs necessary to explain latents in such a way.

Summary: The paper examines whether we can rewrite an LLM using sparse coding (sparse auto encoder, transcoder). The paper proposes a pipeline using an LLM to find patterns in the target LLM's activations. They then propose to change the activations to control the output of the target LLM. The paper concludes that "we can't"

We appreciate the reviewer's comments; however, we believe that this summary misinterprets the core objective of our research. While we do employ a Large Language Model (LLM) to identify specific activation patterns within a sparse coder, which itself is trained on the hidden representations of an LLM, the primary intent was not to manipulate these activations to control the output of a target LLM. Instead, our goal was to leverage the explanations provided by sparse coders as "concept bottlenecks" to reconstruct the original LLM's behavior.

Strengths And Weaknesses: Although I think the idea of the paper is interesting, I found some drawbacks. The paper quite difficult to read, partly because of some quite subjective statements, for instance, in the abstract "The greatest ambition of mechanistic interpretability is to completely rewrite deep neural networks in a format that is more amenable to human understanding, while preserving their behavior and performance". Is there a reference for that? I don't think the ultimiate purpose of interpretability is limited to NNs.

Prominent researchers in mechanistic interpretability (MI) have consistently stated that the discipline's fundamental objective is to reverse-engineer the mechanisms of neural networks. For instance, the 2024 ICML workshop, accessible at https://icml2024mi.pages.dev/, explicitly says: “One emerging approach for understanding the internals of neural networks is mechanistic interpretability: reverse engineering the algorithms implemented by neural networks into human-understandable mechanisms.” Similarly, the most recent MI workshop at NeurIPS (https://mechinterpworkshop.com/) asserts: “The mechanistic interpretability field benefits from a rich diversity of approaches— (...) from reverse-engineering a model via bottom-up (...). But all are unified by the belief that there is meaning and structure to be found inside neural networks.”

While “interpretability” can include research on machine learning models other than neural networks, the field of mechanistic interpretability is indeed entirely focused on neural nets. If there are other statements that the reviewer finds troublesome we would be happy to correct them.

It's challenging to understand the logic of the paper. At the end of the introduction, the authors "puruse the ... idea", but then they state that "there are several roadblocks for this approach", yet "we see this test as a potential benchmark..." So why do the authors still believe in this approach?

This paper starts with a hypothesis. Sparse coders were introduced as a way to break down how models represent things internally. Assuming they can be interpreted in a clear way, we should be able to rewrite a language model's inner workings using everyday language. We worked on presenting evidence against this hypothesis by performing several experiments. The results of these experiments are the roadblocks mentioned in the introduction, which makes us posit that one of our initial assumptions is wrong: either sparse autoencoders don't really break down internal representations well enough, or we can't understand them well enough yet. It's easy to check the first assumption by looking at how well they reconstruct the model activations or how much the model's performance drops when we add them to the computation graph, but the second point is harder to measure. Because of this, we suggest our method as a way to test how well we can understand sparse autoencoders. If sparse autoencoders were fully interpretable, it should be possible to at least partly rewrite a transformer using natural language.

In the abstract. Line 9-12, the paper says that "current sparse coding techniques and automated interpretability pipelines are not up to the task". But the authors just try only ONE pipeline (proposed by them). How could they generalize that statement to other pipelines?

We agree with the reviewer that the language in this sentence might be a bit too general. We do use a single automated interpretability pipeline, but are confident that current automated interpretability pipelines all have the same problem, as they are based on the same idea for generating interpretations. We are not aware of any pipeline that would be able to generate explanations that would allow for a precision of 99%, without tanking the recall.

In the conclusion, the authors wrote "we proposed a new methodology for rigorously evaluating the faithfulness of natural language explanations...". I don't think anywhere in the paper shows any point about "faithfulness" in explanation.

Faithfulness normally refers to how well explanations conform to the model’s behaviour. If our explanations don’t allow the simulator LLM to accurately predict the activations of the latent, we can say that these explanations are not faithful.

In general, I can see that the paper present negative results using their approach, but I do think their claim is too wide in terms of generalization.

Throughout this paper, we have tried to balance claims of generality regarding our results with an awareness of their inherent limitations. It is true that our results are confined to the re-implementation of an MLP module using a transcoder. In this context, it is reasonable to conclude that the necessary precision of natural language explanations for this endeavor is not currently possible with current transcoder and automated interpretability technology.