Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders

We appreciate constructive feedback.

Downside of FF-KV and motivation of SAEs (Weakness 1 & 2)

We agree that a key motivation for SAEs is to disentangle more distinct concepts than the limited number of features permitted in MLP neurons. We agree with this theoretical advantage, and our experiments may suggest that, empirically, current SAEs may not disentangle concepts as much as expected, and the provided interpretability is nearly the same as the original MLP representations. More generally speaking, the advantage of SAE over conventional interpretability approaches, such as linear probe, has been debated, and our results will contribute to and encourage the estimation of SAE’s actual advantage.

Regarding Figure 1 (Question 1)

We apologise for Figure 1’s poor readability. Please see our reply to Reviewer euvs for the table replacement and our planned visual update. Note that a more detailed breakdown of Figure 1 is shown in Appendix (but we admit the room for presentation improvement)

Negative SCR scores (Question 2)

SCR can indeed be negative: a negative value simply means that using SAE features reduces performance relative to raw activations. Negative SCR values also appear in the SAEBench paper [1] (See their Fig. 7). In our updated table the layer-averaged SCR is non-negative; the negative points in the original figure were outliers.

[1] Karvonen et al., SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability.

Sparsity of MLPs and choice of L₀ (Question 3)

Reference [2] shows that MLP activations are already sparse. Appendix E lists the L0 values for all SAEs; following the Gemma-Scope recommendation, we set $L_0 \approx 100$ for every model, so sparsity levels are comparable.

[2] Z. Li, C. You, S. Bhojanapalli, D. Li, A. S. Rawat, S. J. Reddi, K. Ye, F. Chern, F. Yu, R. Guo, and S. Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations, 2023

Handling negative activations (Question 4)>

Please see our response to Reviewer hRu1 for a detailed explanation of how we treat negative activations.

Conclusion of our paper (Question 5)

We align with conclusion (2): SAEs themselves are not a mistake, nor do we advocate reverting to direct neuron interpretation. Rather, our findings suggest that current SAE implementations fall short of their goals, motivating both improved SAEs and new methods to achieve better conceptual disentanglement.

We thank the reviewer for recognizing our work’s contribution. We greatly appreciate these suggestions and will incorporate them in the next revision. Our answers to the questions appear inline below:

How do you define a “simple concept”?

Our example in the feature‑absorption evaluation merely illustrates that a concept can appear at different levels of abstraction, following existing metrics. In practice, we believe that researchers should choose the appropriate level of absorption, depending on specific research questions.

What does “Feature Origin Judgment” tell us about the model interpretability?

This experiment addresses the question: “Do SAEs produce activation patterns similar to FF-KVs qualitatively?” Our human annotators could not reliably distinguish top‑K FF‑KVs from SAEs or Transcoders, suggesting that SAEs and Transcoders do not provide fundamentally clearer activation patterns compared to FF‑KVs, supporting that FF-KV is nearly interpretable as SAEs/Transcoders.

We thank the reviewer for the insightful comments.

Detailed plan for complete update of Figure 1 (Weakness 1)

We will replace Figure 1 with tables with exact scores — sorry for the limited readability. Below, we showcase the Gemma‑2‑2B layer‑averaged results. Still, such averaged scores overlook layer-wise detailed changes; in the next version, we will also include additional figures in Appendix: one plot per coder, task, and model to improve legibility.

Here, we leave out some evaluation scores for random transformer, where the metric does not make sense or shows immediately bad scores (Feat. Alive, Explained Var. , RAVEL).
Note that a detailed breakdown of Figure 1 is shown in Appendix (but we admit the room for presentation improvement)

Missing Transcoder in explained‑variance plots (Weakness 1)

We apologize for this confusion. In Figure 5, the GPT‑2 transcoder is omitted because its explained variance is far below the plot range. The explained variance is −14 807.46 with a standard deviation of 38 729.33, indicating that the GPT‑2 transcoder is poorly trained; its result should therefore be interpreted with caution. We will make this explicit in the text.

paper is slightly overclaiming at several points (Weakness 2)

We will adjust or clarify the conclusions based on your concern. Our main claim is that SAEs do not provide a large interpretability advantage over the existing FF-KV view overall. This asserts that a strong, simple baseline is missing in the community, and also possibly highlights the limitations of the current SAE benchmark (if one expects the advantage of SAE). As you pointed out, the autointerp score is slightly lower in FF-KVs, but this is one facet of our evaluations, and entirely, we hope the above take-home message and the novelty of our findings generally hold. As for the chance score, we hope that our baseline of random Transformer already provides a good reference.

poor explained variance on Top‑K FF‑KVs and could use more discussion (Weakness 3)

Explained variance is computed on the FFN output. Because Top‑K FF‑KV zeroes all but the k largest activations (e.g., k=5 for 8000 hidden states corresponds to 0.125% of neurons), its output inevitably diverges from the original, reducing explained variance.

For a quick check, increasing k for Gemma2-2B (layer 13) easily alleviates this trend, as shown below:

We can easily extend such quick checks to all models and layers, but during author response, we did not have enough computational resources to run a sweep across all layers in all k settings.

Human evaluation is very small scale and relies a lot on author judgement (Weakness 4)

We will try to increase the annotation scale for camera-ready, but it is not so easy as the task is labour‑intensive. Let us excuse that the current manual analysis scale (200 features in total) and the setting (handled by authors) already exceed or match existing works —for example, Anthropic[1] (168 features) and the Transcoder study[2] (100 features), both annotated by the authors.

[1] Bricken et al.. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning.
[2] Dunefsky et al.. Transcoders Find Interpretable LLM Feature Circuits.

Limited originality (Weakness 5)

The NeurIPS reviewer guidelines explicitly value studies that yield new insights by rigorously evaluating existing methods. Our work fits this category, and if the reviewer is concerned about the fact that this study does not propose a novel method, we respectfully disagree with the assessment of limited originality.

Possible explanation of the contradiction between this paper and the Anthropic work (Question 1)

We believe the discrepancy arises from different sampling strategies. Anthropic uniformly partitions each feature’s activation range into equal bins (e.g. 0–10 %, … , 90–100 %) and samples evenly from all bins, thereby analysing low, medium, and high activations. Our study, by contrast, interprets each feature using its highest‑firing inputs, as is typically focused on in other metrics. This difference likely accounts for the contrasting findings.

We thank the reviewer for the insightful comments.

Why adopt the key–value memory lens? (Weakness 1)

Thanks for a good clarification question. SAE research investigates both feature activations and the geometry of their feature vectors, whereas the term “neuron” typically refers to the activations (FF keys) alone. The natural counterpart of the SAE feature vectors in the FF module is their value vectors through the lens of key-value memory, and in this sense, key-value memory will be a more accurate scope corresponding to SAE research. For example, in Section 6, we analyzed the alignment of feature vectors in SAE and FF, which will be beyond the scope of just observing activations (neurons).

 Applicability of the KV view to SwiGLU layers (Weakness 1 & Question 3)

This is a good point, and there will be at least two dimensions to be discussed:
A. Split‑gate factorisation (gate_proj, up_proj)
B. The presence of negative activation values produced by SwiGLU and its impact on the experimental results

For the concern (A), Zhong et al. (2025) [1] show that a SwiGLU FFN can be written as a signed key‑value sum (their Eq. 29):

$

\mathrm{FFN_{SwiGLU}}(x)=\sum_i v_i g_i (x) \mathrm{Swish}(k_i x)

$

where $g_i(x) \mathrm{Swish}$ is regarded as a key (activation) and v_i as a value vector.

As you may be concerned, the $\mathrm{Swish}(x)$ can take negative values, in contrast to the original key-value memory analysis in vanilla FFs — the concern (B).

Fortunately, the absolute scale of negative activation is typically very small (as it’s the approximation of ReLU). Our experiments rely on the top-k activations based on their magnitude, and we empirically confirmed that negative activations were not selected at all throughout the experiments, as their magnitude was almost zero. That is, the negative activations do not affect experimental results in this study.

[1] Zhong, Shu, et al. "Understanding Transformer from the Perspective of Associative Memory." arXiv preprint arXiv:2505.19488 (2025).

The inconsistency in autointerp scores with previous work (Weakness 2, Question 1, Question 2)

In the SAE fields, slightly different implementations (with different activation selection processes) are used for the auto-interpretaion task, and we adopted the implementation of Karvonen et al. [2] — this is the source of the discrepancy with Paulo et al. (2024) [3]. Our autointerp results yielded a similar score scale to [2] (see their Figure 5). Note that the implementation of [3] introduced a specific threshold (1e-5) for sampling activation (w/o reasonable justification), while [2] applies a two-step process, first a threshold-free sampling process, and second a dynamic threshold (1% of the max activation); we hope that the latter (i.e., our implementation) can handle concerns, such as different activation scale for different features/models.

[2] Karvonen et al., SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. Forty-Second International Conference on Machine Learning.

[3] Paulo, Gonçalo Santos, et al. Automatically Interpreting Millions of Features in Large Language Models. arXiv preprint, arXiv:2410.13928 (2024).

Improvement on Figure 1

We apologize for the dense, small plots and will replace Figure 1 entirely, as all reviewers noted its limited readability. For a detailed revision plan, please see our response to Reviewer euvs. Note that a more detailed breakdown of Figure 1 is shown in Appendix (but we admit the room for presentation improvement)

Dear Reviewer hRu1,

Thank you once again for dedicating your valuable time to reviewing our paper. As the deadline approaches, we kindly request your feedback on our rebuttal.

We have carefully addressed all your concerns in detail and hope that you find the response satisfactory. Particularly, similar concerns (negative activations in SwiGLU, slight score differences with existing studies, Figure 1) were also raised by other reviewers, and they indeed recognize that our response successfully handled these concerns and ultimately raise their review scores. Now, all the other reviewers lean toward the acceptance of this paper, and we hope you also might reconsider your score based on our response.

We are eager to engage in further discussion and address any additional concerns you may have. We sincerely appreciate your constructive suggestions.

Thank you! Best Regards, Authors