Response to Reviewer WhEs

We thank the reviewer for their thoughtful engagement with our paper and their recognition of its theoretical and empirical contributions. We address their specific concerns below.

Regarding inconsistencies between synthetic and real data experiments

The reviewer correctly notes that there are differences in the methods evaluated on synthetic versus real data. This was primarily due to implementation and computational constraints:

1. SAE+ITO on real data: We did not implement SAE+ITO for the GPT-2 experiments due to the significant computational cost of performing inference-time optimisation on hundreds of millions of tokens. While feasible on synthetic data, this would have been quite expensive at LLM scale, particularly in addition to our LCA method.

2. LCA versus sparse coding: The LCA method used in our real data experiments is an implementation of sparse coding that uses competitive dynamics to achieve sparse inference. We chose LCA specifically because it has an established history in the sparse coding literature and offered a more efficient implementation path for our large-scale experiments than our synthetic sparse coding method.

We acknowledge that using the same set of methods across all experiments would have provided better consistency. In future work, we plan to implement a more unified experimental framework that can efficiently scale to larger models and datasets.

Qualitative feature visualisation

We agree that providing qualitative visualisations and examples of the features discovered by different methods would strengthen our paper. While our automated interpretability evaluation provides quantitative evidence for the superior interpretability of MLP features, examples would offer readers a more intuitive understanding of these differences.

In the final version, we will add an appendix section with representative feature examples from each method, including:

• Raw activation patterns on top activating tokens
• Generated feature interpretations
• Validation examples showing correct/incorrect feature activation predictions

We note that while recent work by Heap et al. (2024) raises important questions about interpretability pipelines, their concerns primarily relate to interpreting features in randomly initialised transformers, not to the comparison of different encoding methods applied to the same model. In addition to this, there have been demonstrations that trained transformers still out-perform randomly initialised transformers on autointerp baselines. Our work focuses on the relative performance differences between encoding strategies rather than absolute claims about interpretability.

Clarification on dimensionality settings (N > M versus M > N)

The reviewer raises an excellent question about the dimensionality settings and their implications. To clarify:

Our theorem addresses the case where N (number of sources/latent dimensions) > M (observed dimensions), which indeed matches the typical SAE setting where the latent space is larger than the activation space. The key insight is that in this regime, a simple linear-nonlinear encoder cannot perfectly recover all possible sparse codes from their lower-dimensional projections, even when such recovery is theoretically possible with iterative methods.

Regarding the contractive autoencoder settings with M > N, this represents a different regime than what SAEs typically address. In that case:

1. The encoding is undercomplete rather than overcomplete
2. The primary goal is often dimensionality reduction rather than disentanglement
3. The sparsity constraint becomes less critical since there is no inherent superposition

Our theoretical result does not imply that SAE architectures would be better suited to the M > N setting. Rather, it suggests that in the standard N > M setting where SAEs operate, more expressive encoders (like MLPs) can reduce the amortization gap and improve feature recovery.

Scaling to larger models

We acknowledge the limitation that our experiments are conducted on GPT-2 Small. While computational constraints prevented us from scaling to larger models, we have encouraging evidence that our findings would generalise:

1. Our synthetic experiments at larger scales (Appendix A.3) show that the amortisation gap becomes more pronounced as dimensionality increases
2. The theoretical result is independent of scale
3. Recent work applying SAEs to larger models (e.g., Gao et al., 2024) has shown qualitatively similar features and challenges across model scales

We believe our contributions provide valuable insights despite these limitations, and we hope to validate our findings on larger models in future work.

Thank you again for your thoughtful review and constructive suggestions for improving our paper.

Response to Reviewer STGo

We appreciate the reviewer's thoughtful analysis of our work and their recognition of the theoretical contribution. We would like to address several points regarding the practical implications of our findings.

Regarding the claim about optimality gap in LLM activations

We agree that the connection between our theoretical results and practical LLM applications deserves further elaboration. Our work demonstrates that the amortisation gap exists not just theoretically but also empirically across various settings, including LLM activations.

Model choice and scale

While GPT-2 Small (124M parameters) is indeed older than more recent models, it remains a standard benchmark in the mechanistic interpretability literature for several reasons:

1. Established baseline: Numerous interpretability papers continue to use GPT-2 as a testing ground, including recent work by Anthropic, EleutherAI, and others. This facilitates comparison with existing literature.

2. Computational accessibility: Our experiments involved training multiple models on hundreds of millions of tokens, which becomes prohibitively expensive with larger models.

3. Feature stability: The principles of superposition and sparse feature representation appear consistent across model scales, as demonstrated by recent work finding similar phenomena in models from GPT-2 to GPT-4 and Claude.

That said, we acknowledge that verifying our findings on larger models would strengthen our claims, and we plan to pursue this in future work.

Baseline improvement

Regarding the baseline SAE implementation, we intentionally tested a standard ReLU SAE as it represents the most widely used architecture in current interpretability research. We agree that newer architectures like JumpReLU and TopK offer improvements, and we discuss these in Appendix A.7.2.

Our theoretical result applies to this entire class of models, as they all rely on fixed function approximations rather than iterative optimisation. The empirical gap we observe provides an explanation for why these architectural innovations improve performance - they partially address the amortisation gap through more sophisticated function approximation.

Computational efficiency at scale

The reviewer raises an excellent point about computational efficiency comparisons at scale. While our LCA implementation was not fully optimised, we can provide some context:

• Training the LCA model took approximately 3x the computation time of the SAE model due to the additional gradient steps per batch.
• During inference, LCA required approximately 20x the computation of the SAE.

However, as noted in recent work (e.g., Nanda et al., 2024), inference-time optimisation approaches can be made significantly more efficient through techniques like matching pursuit and careful algorithm selection. The fundamental trade-off between amortised and iterative approaches remains, but the computational gap can be narrowed considerably.

On empirical significance beyond theory

While our theoretical contribution stands independently, we believe the empirical results are meaningful for several reasons:

1. The significant interpretability improvement of the MLP encoder (median F1 of 0.83 vs 0.6 for SAE) suggests that even modest increases in encoder expressivity can substantially improve feature extraction.

2. Our work offers an explanatory framework for why recent SAE variants achieve better performance, connecting theoretical understanding with practical innovations.

3. The results suggest promising directions for further research, such as developing more expressive encoders or hybrid approaches that balance computational efficiency and encoding quality.

We thank the reviewer for highlighting areas where our practical evaluation could be strengthened, and we hope to address these in future work with larger-scale evaluations across multiple model families and more sophisticated baselines.

Response to Reviewer JWnB

We thank the reviewer for their time spent evaluating our paper. We believe there are several misunderstandings in the review that we would like to address, as they appear to have led to an incomplete assessment of our work.

Regarding Theorem 3.1

The theorem is indeed rigorous and makes minimal assumptions about the sparse prior $P_S$. The only requirement is that the support of $P_S$ consists of vectors with at most $K$ non-zero entries. This is explicitly stated in the theorem: "Let $S=\mathbb{R}^N$ be $N$ sources following a sparse distribution $P_S$ such that any sample has at most $K \geq 2$ non-zero entries, i.e., $||s||_0 \leq K, \forall s \in \text{supp}(P_S).$"

The proof holds regardless of whether $P_S$ is deterministic or stochastic. It's a constructive proof that shows a specific set of vectors (the standard basis vectors and their sums) cannot be correctly encoded simultaneously by a linear-nonlinear encoder. Since these vectors are valid under any sparse distribution with max sparsity $K \geq 2$, the conclusion holds generally.

Regarding the conclusion after equation (8): This follows directly from linear algebra. If $S'$ must be diagonal to correctly represent the sparse codes after ReLU, but also must have rank ≤ M < N due to the dimensionality constraints, we have a contradiction since a diagonal matrix with non-zero diagonal entries has rank N.

Figure 3 Non-monotone Behaviour

The non-monotonic behaviour observed in Figure 3 (particularly for SAE+ITO) is a deliberate focus of our analysis, not an oversight. As stated in Section 4.2: "SAE+ITO initialised with SAE latents exhibits distinct, stepwise improvements throughout training, ultimately achieving the highest MCC." This pattern reveals interesting dynamics of how inference-time optimisation interacts with the learned dictionary during training.

Figure 7 Visualisation

The circles in Figure 7 represent statistical outliers in the distribution of F1 scores, following standard boxplot conventions. We will add this clarification to the figure caption to avoid confusion.

Interpretability of F1 Score Metric

The F1 score is a widely recognised and highly interpretable metric for classification tasks. In our case, it measures how well a second instance of GPT-4o can predict neuron activations based on explanations from a first instance. This is explicitly described in Section 6: "The model predicted which examples should activate the feature based on the first instance's explanation, allowing us to compute an F1-score against the ground truth."

The approach is standard in the field and follows established methods from Anthropic, EleutherAI, and other research groups cited in Appendix A.7.

Relevance of Synthetic Experiments

While our synthetic experiments use moderate dimensionality for clarity and computational efficiency, we demonstrate their relevance to practical problems in multiple ways:

1. We explicitly test larger-scale experiments in Appendix A.3 (N=1000, M=200, K=20), showing that our findings hold and even strengthen at larger scales.

2. We examine non-uniform feature distributions in Appendix A.4 to better match real-world latent spaces.

3. Most importantly, we validate our findings on actual GPT-2 residual stream activations in Section 6, showing that the principles discovered in synthetic settings transfer to real neural networks.

The synthetic experiments provide a controlled environment where ground truth is known, allowing us to rigorously evaluate the amortisation gap that forms the theoretical foundation of our work.

Model Complexity and SOTA Comparison

Our focus is on the fundamental mechanisms behind sparse encoding and dictionary learning, rather than specific architectural innovations. The models in Section 3.3 represent core architectural categories (linear-nonlinear encoders, MLPs, sparse coding) that underlie most advanced SAE variants.

In Appendix A.7.2, we discuss how our findings relate to advanced SAE architectures like top-k SAEs, JumpReLU, Gated SAEs, and ProLU activations. These are cutting-edge developments in the field, many published in the past six months. Our work provides theoretical and empirical foundations that explain why these architectural innovations yield performance improvements.

We appreciate the opportunity to clarify these points and believe our work makes significant contributions to both the theoretical understanding and practical application of sparse autoencoders for neural network interpretability.