Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures

While to my knowledge, this is the first work evaluating the similarity of SAE features between Transformers and Mamba models, other works such as Paulo et al. 2024 and Sharma et al. 2024 already showed that many interpretability approaches such as logit lenses, steering vectors, and probes produce similar results across a variety of LLM architectures (Mamba, Transformer and RWKV). It is, hence, not particularly surprising that such findings extend to SAE decompositions of model activations.

Thanks for pointing this out. Paulo et al. 2024 is a highly relevant work which we did not cite in our submitted manuscript. We have included this in our new revision. Apologies for this.

We would like to point out that although our work mainly falls on the line of transferrability of interpretability methods across LM architectures, which has already been explored by existing literature, our findings provide new insights into the following aspects:

• The analysis of depth specialization of SAE features across models suggests the existence of an architecture-agnostic feature hierarchy for language modeling.
• The induction circuit similarity analysis, though being rather simple, serves as a supporting example of architecture-agnostic circuit motif for language modeling.
• Our complexity viewpoint of MPPC raises a broad concern for cross-architecture model diffing. For example, a recent promising model diffing method crosscoder is also prone to false positive of divergent features, where features activating on the same concept might be identified as different ones due to their respective preference for specific instances. This suggests the need for sanity checks of subsequent model diffing methods.

We would like to appreciate it again for pointing out a missing related work. And we are glad to have further discussion on the novelty of our findings.

What justified the choice of plain SAEs over more performant variants such as Gated or Top-K SAEs? It is currently hard to gauge the impact of this choice on the obtained results, and whether findings could have been different if improved SAE variants were tested.

Thanks for this valuable question. Vanilla SAEs are indeed less performant and somewhat outdated in a retrospective view. We conduct the main experiment and both skylines with TopK SAEs and get the following results:

We have incorporated these updates in Sections 4.4 and Appendix D.3 of the revised manuscript for further clarity and elaboration.

(Reviewer 4GEc) Have you performed any statistical significance tests to support your claims of feature similarity and universality?

(Reviewer jsst) In Section 4.4, no hypothesis is formulated regarding the expected similarity of features found across the four tested variants, and consequently, no significance measure for the correlation coefficients is reported.

This is an important part we missed in our submitted manuscript and we are very thankful for pointing this out.

We additionally establish a random baseline by calculating MPPC for each Pythia feature against a random SAE with matching feature count and sparsity level (by masking all but the TopK activating features) to the Mamba SAE to observe the impact of feature quantity and sparsity on MPPC distribution.

We denote mean MPPC as the random variable $ x $. To evaluate whether the result of our main experiment ($ x = 0.681 $) belongs to a distribution with the same mean as the random baseline, we conducted a Hypothesis Testing. Specifically, we tested the null hypothesis ($ H_0 $) that the mean of the experimental group is equal to the mean of the random baseline group ($ \mu_{\text{experiment}} = \mu_{\text{baseline}} $) against the alternative hypothesis ($ H_1 $) that the two means are different ($ \mu_{\text{experiment}} \neq \mu_{\text{baseline}} $).

The random baseline group consisted of 16 samples(repeated the baseline 16 times and calculate mean MPPC for each) with a mean of $ 0.1944 $ and a standard deviation of $ 0.00046 $. The experimental group consisted of a single sample with $ x = 0.681 $. Since the sample size of the experimental group is one, we applied a two-sample $ t $-test using the pooled standard deviation from the baseline group. Yielding a $ t $-value of $ -1026.24 $. The degrees of freedom are $ n_{\text{baseline}} + n_{\text{experiment}} - 2 = 15 $. Using a one-tailed $ t $-test, the resulting $ p $-value is:

$p = 4.54 \times 10^{-38}$

Given that the $ p $-value is significantly smaller than any conventional significance level ($ \alpha = 0.05 $), we reject the null hypothesis. This indicates that the experimental result ($ x = 0.681 $) is highly unlikely to belong to a distribution with the same mean as the random baseline.

We have incorporated these updates in Sections 4.1 and 4.4 of the revised manuscript for further clarity and elaboration.

(Reviewer 4GEc) How do you expect your findings to scale to larger models? Did you observe whether model size impacts universality between architectures? Could smaller or larger versions of Transformers and Mambas exhibit different degrees of feature similarity?

(Reviewer nuP5) (Minor) The size of LMs is limited. Only ~100m models are used in the experiments.

(Reviewer jsst) Without further experiments, it cannot be excluded that the limited capacity of tiny models might be the main motivation behind the high similarity features and circuits across the two architectures, and this could not be the case for more capable models with e.g. 1B or 7B parameters.

Thanks for pointing this out, which greatly helps improve our work.

We additionally conduct the experiment for 2.8B variants of both models, giving us the following results:

We have incorporated these updates in Sections 4.4 and Appendix D.4 of the revised manuscript for further clarity and elaboration.

We sincerely appreciate your recognition of our work, with highlights in motivation, interest, soundness and clarity. We would also like to acknowledge the thorough and constructive feedback and questions provided which help strengthen our work, which we summarize as follows:

• Novelty of the findings;
• Generality of findings for larger models;
• Statistical significance tests;
• Choice of SAE variants.

All of these aspects are helpful and insightful. In addition, thanks for pointing out the typos and clarification points in our submitted manuscript, which we have all fixed in our updated version.

Using a random SAE as a null hypothesis is a very low bar to claim feature similarity between the two models. Such tests may, at best, confirm that the similarity between latents is higher than random chance, which is unsurprising given that both models were trained on the same data. An evaluation comparing the similarity of features cross-layer vs. cross-architecture would have been more interesting in this regard (e.g., Is the similarity of features from upper layers of Mamba and upper layers of Pythia higher than between upper and lower layers in the same model?)

Thank you very much for your thoughtful feedback and for adjusting the rating. We greatly appreciate the time and effort you have invested in reviewing our paper.

We understand your concern, which, if we have interpreted correctly, points to the baseline being relatively weak. You note that the substantial performance of the main experiment over the baseline might not sufficiently reflect the extent of cross-model feature similarity.

To address this, we would like to clarify that our paper used the relatively small difference between the main experiment and Skyline1 to reflect this similarity. Specifically, we highlighted that more than 60% of features exhibit near-zero MPPC differences (absolute value < 0.05). This implies that, for the majority of features (and their corresponding semantics), the similarity between Mamba and Pythia is almost indistinguishable from the similarity between Pythia and models of the same architecture.

That being said, your suggestion is particularly insightful. Comparing cross-layer similarities to cross-architecture similarities could indeed provide a new and meaningful baseline for evaluating cross-model similarity. Inspired by your comment, we have conducted additional experiments to explore this idea. Specifically, we calculated the similarity between high layer(12~23) features and lower layer(0~11) features within Mamba, as well as the similarity between high layer features of Mamba(12~23) and Pythia(6~11). In the table below, the column headers represent the range of MPPC values calculated by mamba(all layers) to pythia(all layers), where lower values indicate relatively higher feature complexity (which sometimes may simply result from the absence of matching semantically similar features). The results are as follows:

It can be observed that for relatively simple features (the rightmost three columns), Mamba's higher layers and lower layers exhibit stronger similarity. For relatively complex features (the third, fourth, and fifth columns), Mamba's higher layers and Pythia's higher layers show stronger similarity(same in the third column). As for the leftmost two columns, we speculate that this might be due to some Mamba features failing to find matches in Pythia.

We hope this additional analysis addresses your concern and provides further clarity on the cross-model feature similarity presented in our paper.

Thank you once again for your valuable insights, which have helped us improve the rigor and depth of our study. Please let us know if you have any further suggestions or questions.

While the heatmap matrix in Figure 3c is mainly diagonal, I can see that there is a cluster of features located in the last layer of the Pythia and distributed fairly uniformly in the middle layers of Mamba. Can the authors clarify the meanings of these features?

Thanks for pointing this out. Apologies for not explaining this exception which is indeed confusing without further details. We expect this to be reasonable for the following reasons:

• The residual stream after the last layer is directly connected with the unembedding (interleaved by a LayerNorm), so it should most contain information about next token prediction, rather than that about past tokens, making it substantially different from lower-layer residual stream activations.
• One piece of evidence of this is the ultra-high norm of the last layer residual stream, which was reported in the third figure in this post. One can think of this as "Overwriting the whole residual stream to focus on predicting".
• For example, in this last-layer residual stream SAE, the interpretation for most features are only clear if one looks into the top logits they contribute to, which is much less often the case for lower-layer ones.

It might be worthwhile to explore other correlation measures, such as distance correlation, which could potentially yield better results.

Thank you so much for your insightful suggestion! We genuinely appreciate the thoughtfulness behind it. In our paper, we used Pearson Correlation as a metric to measure feature similarity. However, we agree that, influenced by factors such as feature complexity, Pearson Correlation may not fully capture the semantic similarity of features. Exploring more sophisticated and semantically aligned similarity metrics is indeed a promising direction for future research.

That being said, Distance Correlation, while conceptually compelling, presents significant computational challenges. Its time and space complexity scale as $O(n^2)$ with the number of sampled tokens $n$, compared to the much more efficient $O(n)$ complexity of Pearson Correlation. Given the resource constraints we faced, we were unable to employ Distance Correlation in this work. Nonetheless, we see great potential in this idea and are excited to explore it in our future research endeavors.

Once again, thank you for your valuable feedback—it has given us fresh perspectives and inspired directions for further improvement!

It would be interesting to explore more circuits through SAE, as suggested in [1] (see Section 4.3).

Thanks for this suggestion. We think that SAE feature circuits, despite some existing literature investigating this problem[1, 2, 3], is still not what we think a mature method to help support our claims due to its time and computation complexity. In addition, we expect findings of SAE feature circuits to be a finer-grained extension of head-level or block-level circuit analysis, which may not contradict with our findings.

[1] Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT

[2] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models

[3] Transcoders Find Interpretable LLM Feature Circuits

We sincerely appreciate your recognition of our work, with highlights in novelty, clarity, and impact. We would also like to acknowledge the thorough and constructive feedback and questions provided which help strengthen our work, which we summarize as follows:

• Exploring more circuits with SAEs;
• Exploring more correlation metrics;
• Clarify the implication of the last Pythia layer exception in depth-specificity analysis.

All of these aspects are helpful and insightful. And many thanks for suggestions to clarify the meaning of the acronym MPPC, which we have included in our updated version.

(Reviewer 4GEc) How do you expect your findings to scale to larger models? Did you observe whether model size impacts universality between architectures? Could smaller or larger versions of Transformers and Mambas exhibit different degrees of feature similarity?

(Reviewer nuP5) (Minor) The size of LMs is limited. Only ~100m models are used in the experiments.

(Reviewer jsst) Without further experiments, it cannot be excluded that the limited capacity of tiny models might be the main motivation behind the high similarity features and circuits across the two architectures, and this could not be the case for more capable models with e.g. 1B or 7B parameters.

Thanks for pointing this out, which greatly helps improve our work.

We additionally conduct the experiment for 2.8B variants of both models, giving us the following results:

We have incorporated these updates in Sections 4.4 and Appendix D.4 of the revised manuscript for further clarity and elaboration.

How does the feature mapping between RWKV and Mamba look like?

Thanks for this question. we provide Pythia-160m&Mamba-130m&RWKV-169m similarity results as shown below(mean MPPC of A->B):

We have incorporated these updates in Sections 4.4 and Appendix D.5 of the revised manuscript for further clarity and elaboration.

Do Transformers have such layer-specific behavior when it comes to inductive ability?

Is the layer-17 phenomenon robust to random initializations? I.e., if one retrains the SSM with another seed, would layer 17 still be the key in induction?

Thanks for this interesting question. It is an important hypothesis that the inter-layer structure of the final model should be very robust to initialization and even more other hyperparameters. We conjecture layer 17 will still be a pivotal induction layer with a retrained Mamba. One of the main reasons is that two Transformers trained with mostly the same configuration by two groups, Pythia-160M and GPT2-Small, have both been reported to mainly use their layer 5 (out of 12 layers) to perform induction.

• Both models have a hidden dimension D=768 and # layers = 12. The main differences are summarized as follows: | Model | Position Embedding | Embedding & Unembedding | Trained with Dropout | |-----------------|-----------------------|--------------------------|-----------------------| | GPT2-Small | Absolute, sinusoidal | Tied | Yes | | Pythia-160M | Rotary | Independent | No |
• GPT2-Small: We quote Section 4.2 in [1], "As a case study, we focus on GPT-2 Small [55], which has two induction heads in layer 5 (heads 5.1 and 5.5) "
• Pythia-160M: We perform path patching on this model, finding that head 5.0 (the first attention head in layer 5) is the most notable induction heads: | layer\head | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | |--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------| | 0 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | | 1 | 0.07 | -0.15 | -0.10 | 0.03 | 0.09 | -0.08 | -0.07 | 0.06 | -0.01 | 0.11 | 0.34 | -0.05 | | 2 | -0.14 | 0.07 | 0.10 | 0.14 | 0.14 | -0.13 | 0.60 | -0.03 | -0.14 | 0.10 | 0.04 | 0.03 | | 3 | -0.24 | -0.14 | -0.96 | -1.20 | -0.49 | -0.14 | 0.20 | -0.38 | -0.10 | 0.06 | -0.11 | -0.07 | | 4 | 0.13 | -0.26 | 0.09 | -0.16 | -0.10 | -0.02 | 0.89 | 0.13 | 0.09 | -0.28 | -0.14 | 0.30 | | 5 | 4.00 | -0.20 | 0.05 | 0.06 | -0.53 | -0.04 | 0.48 | 0.62 | 0.06 | 0.08 | 0.05 | -0.23 | | 6 | -0.04 | -0.23 | -0.04 | -0.22 | 0.02 | 0.09 | 0.04 | -0.33 | 0.02 | -0.04 | -0.38 | 0.04 | | 7 | -0.28 | 0.17 | 0.03 | 0.06 | -0.28 | -0.07 | 0.01 | -0.18 | -0.23 | -0.03 | -0.02 | 0.18 | | 8 | -0.07 | 0.03 | 0.50 | 0.00 | 0.15 | -0.02 | 0.01 | -0.22 | 0.02 | -0.02 | -0.08 | 0.38 | | 9 | 0.54 | -0.03 | 0.07 | -0.09 | -1.10 | -0.04 | 0.04 | 0.00 | 0.04 | 0.10 | -0.01 | 0.02 | | 10 | -0.01 | 0.03 | 0.00 | 0.00 | -0.03 | -0.10 | 0.01 | -0.01 | 0.00 | -0.04 | 0.03 | 0.01 | | 11 | -0.14 | -0.13 | -0.05 | -0.04 | 0.00 | -0.02 | -0.11 | -0.02 | 0.01 | -0.07 | -0.02 | 0.06 | [1] Interpreting Attention Layer Outputs with Sparse Autoencoders

The claims made in Section 6.2 need to be empirically supported.

Thanks for this suggestion. We are not sure we currently understand your suggestion correctly. We take it as "designing extra experiments to show that induction circuits are similar".

It is indeed necessary to quantify or validate cross-architectural circuit similarity with counterfactual experiments to further strengthen our claim that Mamba and Transformer implement the same induction algorithm. For instance, it may be a statistical method to reveal that model weights connecting similar features also tend to be similar. Or one can ablate matched pairs of features from both models and see whether downstream performance drops in the same trend. However, due to time and compute constraint, we are not currently able to conduct such experiments. Apologies for this. Nonetheless, since induction circuits have been widely studied for Transformers[1, 2], we think that the "previous token heads-local convolution" and "Induction heads-Layer 17 SSM State" analogies are strongly backed up this claim.

[1]A Mathematical Framework for Transformer Circuits

[2]In-context Learning and Induction Heads

The role of circuit analysis experiments is unclear.

They are indeed very interesting but I’m not sure how they contribute to building a mechanistic analogy between SSMs and Transformers.

Thanks for this question. Our thoughts on this question are mostly influenced by a lot of existing discussion of features and circuits in the field of Mechanistic Interpretability[1, 2]. We think that circuit analysis is investigating how features are connected from a more macroscopic viewpoint. If we have found that interpretable features in different models are universal, a natural research question to ask next is whether the weights connect them in the same way.

We conjecture the main reason we left you confused is that our circuit analysis seems not correlated to the SAE features investigated in Section 4 and 5. We are actually studying individual mamba blocks, SSM states or attention heads rather than what we claim to be "how the features are connected".

If this is the case, we think that SAE feature circuits, despite some existing literature investigating this problem[1, 2, 3], is still not what we think a mature method to help support our claims due to its time and computation complexity. In addition, we expect findings of SAE feature circuits to be a finer-grained extension of head-level or block-level circuit analysis, which may not contradict with our findings.

[1] Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT

[2] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models

[3] Transcoders Find Interpretable LLM Feature Circuits

We sincerely appreciate your recognition of our work, with highlights in novelty, soundness and significance. We would also like to acknowledge the thorough and constructive feedback and questions provided that help strengthen our work, which we summarize as follows:

• Clarifying the role of circuit analysis.
• Empirically supporting the "Universal Induction Algorithm" claim.
• Generalization to Larger Models
• Universal layer-specificity phenomenon.
• RWKV-Mamba feature similarity.

All of these aspects are helpful and insightful. In addition, thanks for pointing out the typos in our submitted manuscript, which we have all fixed in our updated version.

Dear Reviewer nuP5,

We hope this email finds you well. We are reaching out regarding our ICLR 2025 submission (Submission 4540). With the discussion period coming to a close, We wanted to follow up and see if there are any additional questions or concerns about our paper or the rebuttal we provided earlier that we could help clarify.

Your detailed feedback has been Insightful, and we have put significant effort into addressing the points you raised. If there are any remaining aspects where further clarification might strengthen your understanding of our work, please let us know—we would be happy to provide more information.

Furthermore, if you feel our responses have addressed your concerns effectively, we would greatly appreciate it if you might consider revisiting your initial rating of our submission. Your expert evaluation plays a crucial role in shaping the final outcome, and we sincerely appreciate your time and efforts throughout this review process.

Thank you again for your dedication to improving the quality of submissions. Please feel free to let us know if there is anything else we can assist with.

Best regards,

Authors of Submission 4540

Is there a risk that the Sparse Autoencoder pre-processing itself may impose a degree of alignment between features in Transformers and Mambas? Could the sparsity constraint inadvertently enhance apparent similarity?

We appreciate your insightful question. This is indeed a reasonable and inspiring concern.

There are two possibilities we can think of that this interpretability illusion holds: (1) Illusion in MPPC analysis: sparsity constraint causes the activation pattern of all features to be sparse, i.e., activating only on 1% of all tokens, leading to an increase in Max Pairwise Pearson Correlation. This turns out to be the case in the random baseline we established in Response 2/9, compared to one without being set to the same the sparsity level as a normal SAE. Nonetheless, it accounts for only a small portion of Pythia-Mamba feature similarity, as indicated by the statistical significance test. (2) Illusion in SAE Training, which we discuss as follows:

We think there is possibility that different model architectures actually learn different sets of features but SAEs inadvertently aligns them. If SAEs learn commonly-composed features, it may be the case that we are being over-confident about feature universality. It has already been suggested by [1] that SAEs may learn compositions of the "true" underlying features in a toy model due to the sparsity constraint. For instance, if two models encode the same concept in different ways (e.g. one identifies dogs with color and another identifies with breeds) but our SAEs are too small to capture such difference and only learn a "universal dog feature", we are then fooled and draw the wrong conclusion.

There are two reasons why we are not quite concerned with this possibility. (1) Holding all else equal, larger SAEs exhibit higher MPPC values (Response 4/9). This serves as negative evidence against the hypothesis above since if features of larger SAEs 'split' into different sub-features, there should be a drop in MPPC. We are also open to the possibility that we have not scale our SAEs enough yet. (2) Even if there exists interpretability illusion for some reason we currently are not aware of, our findings reliably suggest there at least exists a universal, sparse and interpretable, though lossy, decomposition of the models' hidden activations. It is an exciting topic to discover hard-to-notice divergence hiding in part of the activation space uncaptured by SAEs, probably with better SAE training techniques[2] or some well-designed probes.

We have incorporated the baseline updates in Sections 4.1 and 4.4 of the revised manuscript for further clarity and elaboration.

[1] https://www.lesswrong.com/posts/a5wwqza2cY3W7L9cj/sparse-autoencoders-find-composed-features-in-small-toy

[2] Sparse Crosscoders for Cross-Layer Features and Model Diffing

Including a more diverse set of models, such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs), could strengthen the universality hypothesis by offering a more comprehensive understanding of feature similarity across a wider range of neural networks. This would enhance the generalizability of the findings.

Thanks for the reasonable and consturctive suggestion. We would like to point out that we included results for Pythia-RWKV in Appendix D of our submitted manuscript, which is known as an RNN-like language model architecture. We have made this clearer in our revised version. In addition, we provide Pythia-160m&Mamba-130m&RWKV-169m similarity results as shown below(mean MPPC of A->B):

It is also reasonable to include more novel architectures like xLSTM, RetNet or even convolution-based language model architectures and will indeed enhance the generalizability of our findings. Currently, however, we do not have resources at hand to conduct these experiments and we will leave this for future work. Apologies for this.

We have incorporated these updates in Sections 4.4 and Appendix D.5 of the revised manuscript for further clarity and elaboration.

Can you provide more details on why the "Off-by-One" motif exists in Mamba models?

This is an important question to understand circuits in Mamba models. However, our conjecture about this phenomenon is not rigorously tested and we leave this for future work focusing on Mamba circuits. Our current hypothesis on this problem is that Mamba performs "Off-by-One" to implicitly revert the write-read order of its SSM States.

Write-and-Read Nature of SSMs: Write: $h_{i}^{(l)} = F_a(c_{i}^{(l)}) \circ h_{i-1}^{(l)} + F_b(c_{i}^{(l)})$. Read: $s_i^{(l)} = h_i^{(l)} (W_c^{(l)} * c_i^{(l)}) + W_d^{(l)} \circ c_i^{(l)}$. Concretely, an SSM block has three data-dependent transformations A, B, C and a shortcut D. It first write its input at token $i$ $c_i^{(l)}$ to the state space multiplied by B ($F_b(c_{i}^{(l)})$) and add this to its past state with a transformation $F_a(c_{i}^{(l)}) \circ h_{i-1}^{(l)}$. It then reads from this state space with matrix C plus a shortcut, where the input $c_i^{(l)}$ is directly transformed by D without interacting with the SSM state.

The gate branch $g$ might prevent the need to write into SSM states in-time. There are two ways for a Mamba block input, at step $i$ at layer $l$, $x_i^l$ to contribute to the block output: via the gate branch $g_i^l$ and via the conv-SSM branch. If the local convolution only merges information from the past timesteps rather than the current time step $i$, this means the conv-SSM branch are blocked for input $x_i^l$. However, $x_i^l$ can still affect subsequent computation via the gate. And it is not written into the SSM state until the next timestep $i+1$ due to Off-by-One.

This implicitly reverts the write-read order of its SSM States for easier token-mixing. In this case where the conv-SSM branch are blocked for the current timestep $i$, the SSM actually first reads from information in the past, and write information in $x_i^l$ in the next step. We conjecture this is beneficial in reducing one term in the state space so that it can more effectively retrieve past information.

Since there is relatively less work on Mamba interpretability compared to Transformers, we are open to the possibility that we missed something or made improper assumptions here. We do not dive deeper into this problem due to time and compute limitation and its being slightly beyond the scope of this paper.

Why did you choose OpenWebText as your primary dataset for analysis? How might the choice of OpenWebText as the dataset influence your results? Have you tested if the feature similarities hold across different domains (e.g., code, mathematics, or structured data)? Would analyzing domain-specific text reveal different patterns of architectural universality?

Thanks for these insightful questions. The main reason we choose OWT as our primary dataset is that OWT is a widely used comprehensive text corpus in this field[1, 2, 3]. This is probably because of the popularity of GPT2-Small, whose training data WebText has its open-sourced version OWT.

It can be the case that older datasets of lower quality and duplication can result in overhigh correlation. To address this, we additionally perform correlation analysis on two more datasets, namely SlimPajama[4], a more recent cleaned and deduplicated text corpus for pretraining, and Github subset of RedPajama[5] for domain-specific analysis. The results are shown as follows:

We again appreciate this question as it makes us decide to demonstrate all of our experimental results with ones obtained on SlimPajama, rather than OWT, for its comprehensiveness, higher quality and containing OOD text data for both models we tested. This helps improve the robustness of our conclusion.

We have incorporated these updates in Sections 4.4 and Appendix D.2 of the revised manuscript for further clarity and elaboration.

[1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

[2] Transcoders Find Interpretable LLM Feature Circuits

[3] https://www.alignmentforum.org/posts/f9EgfLSurAiqRJySD/open-source-sparse-autoencoders-for-all-residual-stream

[4] https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama

[5] https://www.together.ai/blog/redpajama

The use of GPT-4 for complexity scoring lacks rigorous validation. The paper does not provide inter-rater reliability metrics or comparisons with human annotations, nor does it discuss potential biases in the automated scoring.

Thanks for pointing this out. Without rigorously validating the reliability of autointerp scores, it is questionable to draw the conclusion in Section 5. We ask two human annotators to score 64 (out of 358 automatically evaluated) pairs for complexity scores and 32 (out of 72) for monosemantic scores. We take the average of human labeled scores and fit human-GPT-4 consistency. The fitted results in terms of (scope, R-square score) is (scope=0.45, R2=0.29) for complexity and (scope=0.38, R2=0.17) for monosemanticity, suggesting the existence of human-GPT4 scoring consistency. We notice that compared to GPT-4 labeled scores, human annotators tend to be more polarized, which may cause a lower R-square score.

We have incorporated these updates in Sections 5.2 and Appendix I of the revised manuscript for further clarity and elaboration.

The paper does not include ablation studies on SAE hyperparameters (dictionary/code size, training duration, etc.). There is no discussion of how SAE reconstruction quality relates to feature similarity.

We appreciate your constructive feedback. We have included more ablations studies on dictionary size, L1 coefficient and SAE architecture (TopK variant). The results are as follows:

L1 coefficient:

Dictionary size (Expansion factor * model hidden size D):

We do not include results with respect to training duration because our SAEs quickly converge after 30 minutes of training on an H100 GPU. Due to time and computation constraint we do not further ablate on this setting.

We have incorporated these updates in Sections 4.4 and Appendix D.1 of the revised manuscript for further clarity and elaboration.

(Reviewer 4GEc) How do you expect your findings to scale to larger models? Did you observe whether model size impacts universality between architectures? Could smaller or larger versions of Transformers and Mambas exhibit different degrees of feature similarity?

(Reviewer nuP5) (Minor) The size of LMs is limited. Only ~100m models are used in the experiments.

(Reviewer jsst) Without further experiments, it cannot be excluded that the limited capacity of tiny models might be the main motivation behind the high similarity features and circuits across the two architectures, and this could not be the case for more capable models with e.g. 1B or 7B parameters.

Thanks for pointing this out, which greatly helps improve our work.

We additionally conduct the experiment for 2.8B variants of both models, giving us the following results:

We have incorporated these updates in Sections 4.4 and Appendix D.4 of the revised manuscript for further clarity and elaboration.

How generalizable are your findings to other tasks beyond language modeling?

We do not further investigate this problem since it is slightly beyond the scope of this work. Nonetheless, we are optimistic about the generalizability for the following reasons.

• Our method (i.e., Sparse Autoencoders) has been shown to generalize to vision models[1, 2], Othello & chess models[3, 4] and protein language models[5] etc.
• There has been lines of evidences that interpretable vision neurons[6] and circuits[7] can be observed across a variety of vision model architectures. However, there is possibility that simpler features are neuron-aligned and more complex ones are stored in superposition and turn out do not match across architectures. We are also excited to see this line of work continue to ViT and conv model universality and we currently expect them to be similar as well.

[1] https://livgorton.com/inceptionv1-mixed5b-sparse-autoencoders/

[2] Towards Multimodal Interpretability: Learning Sparse Interpretable Features in Vision Transformers

[3] Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT

[4] Evaluating Sparse Autoencoders with Board Game Models

[5] InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders

[6] Zoom in: An Introduction to Circuits

[7] High-Low Frequency Detectors

(Reviewer 4GEc) Have you performed any statistical significance tests to support your claims of feature similarity and universality?

(Reviewer jsst) In Section 4.4, no hypothesis is formulated regarding the expected similarity of features found across the four tested variants, and consequently, no significance measure for the correlation coefficients is reported.

This is an important part we missed in our submitted manuscript and we are very thankful for pointing this out.

We additionally establish a random baseline by calculating MPPC for each Pythia feature against a random SAE with matching feature count and sparsity level (by masking all but the TopK activating features) to the Mamba SAE to observe the impact of feature quantity and sparsity on MPPC distribution.

We denote mean MPPC as the random variable $ x $. To evaluate whether the result of our main experiment ($ x = 0.681 $) belongs to a distribution with the same mean as the random baseline, we conducted a Hypothesis Testing. Specifically, we tested the null hypothesis ($ H_0 $) that the mean of the experimental group is equal to the mean of the random baseline group ($ \mu_{\text{experiment}} = \mu_{\text{baseline}} $) against the alternative hypothesis ($ H_1 $) that the two means are different ($ \mu_{\text{experiment}} \neq \mu_{\text{baseline}} $).

The random baseline group consisted of 16 samples (repeated the baseline 16 times and calculate mean MPPC for each) with a mean of $ 0.1944 $ and a standard deviation of $ 0.00046 $. We made the simplifying assumption that the variance of the experimental group is the same as that of the baseline group. The experimental group consisted of a single sample with $ x = 0.681 $. Since the sample size of the experimental group is one, we applied a two-sample $ t $-test using the pooled standard deviation from the baseline group. Yielding a $ t $-value of $ -1026.24 $. The degrees of freedom are $ n_{\text{baseline}} + n_{\text{experiment}} - 2 = 15 $. Using a one-tailed $ t $-test, the resulting $ p $-value is:

$p = 4.54 \times 10^{-38}$

Given that the $ p $-value is significantly smaller than any conventional significance level ($ \alpha = 0.05 $), we reject the null hypothesis. This indicates that the experimental result ($ x = 0.681 $) is highly unlikely to belong to a distribution with the same mean as the random baseline.

We have incorporated these updates in Sections 4.1 and 4.4 of the revised manuscript for further clarity and elaboration.

We sincerely appreciate your recognition of our work, with highlights in novelty, quality, depth, and significance. We would also like to acknowledge the thorough and constructive feedback and questions provided which help strengthen our work, which we summarize as follows:

• Statistical significance tests;
• Generalization to larger models and other tasks;
• Ablation studies on SAE hyperparameters;
• Manually validating Autointerp complexity and monosemanticity scores;
• Dataset Choice;
• Further Exploring Off-by-one motif;
• A broader examination of additional architectures;
• Could the sparsity constraint inadvertently enhance apparent similarity?

All of these aspects are helpful and insightful. We sorted them in the order we think are of decreasing importance and respond in the following comments:

Dear Reviewer 4GEc,

We hope this email finds you well. We are writing to follow up regarding our submission (Submission Number: 4540). We noticed that the discussion period is nearing its end, and We wanted to kindly check if you had any further questions or concerns regarding our paper or the rebuttal we provided earlier.

We deeply value your feedback and have aimed to address your comments comprehensively in our rebuttal. If there are any remaining points that we could clarify or elaborate on, please do not hesitate to let us know.

Additionally, if you believe our responses sufficiently addressed your concerns, we kindly ask if you would consider revisiting your rating for our submission. Your assessment is incredibly important to us, and we appreciate the time and effort you have dedicated to reviewing our work.

Thank you once again for your thoughtful review and contributions to improving our paper. Please feel free to reach out with any further questions or comments.

Best regards,

Authors of Submission 4540