Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality

The claim that k is arbitrarily chosen is valid, not being obvious how to define the best K à priori, but this hyperparameter is substituted with a new one, $\lambda$ , which the training process is very sensitive to as claimed by the authors.

As we included in the Limitation section, we agree that we have introduced an alternative hyperparameter $\lambda_{AFA}$. Nevertheless, we believe our contribution still provides meaningful value to the research community, and we would like to clarify the distinction between λ and 𝑘 in more detail. First, $\lambda$ is a training-time regularization coefficient, whereas $k$ directly relates to inference-time because it refers to the number of active features for each input example. In this sense, $\lambda$ and $k$ start from fundamentally different ideas. Second, our approach allows the model to adaptively determine the effective number of active features for each input, rather than enforcing a fixed $k$ across all inputs. This reflects the intuition that different examples may require different amounts of feature activation, which a fixed $k$ cannot capture (and which is grounded mathematically by the theory in our paper). Therefore, we feel that replacing the need to arbitrarily set $k$ by the training-time hyperparameter $\lambda$ is an advantage of our approach that is motivated by the underlying ideas of sparse autoencoders.

Why should we care about the relation between f and z? The point of sparse autoencoders seems to be that the reconstruction is good. Why should the norms have to match?

We thank you for this insightful question. One of the findings of our work is that, if the reconstruction is perfect ($D\mathbf{f} = \mathbf{z}$) in a widely-used single-layer SAE, the norms of z and the norms of true f must theoretically be approximately equal. Thus, if the reconstruction is simply good, checking whether the norms match can serve as a valuable diagnostic tool, especially since state-of-the-art methods like k-sparse autoencoders impose a strong and naïve inductive bias that exactly $k$ features should exist in every input. That is, when $||\mathbf{f}|| ≪ ||\mathbf{z}||$, we can now tell the SAE is under-activating the feature vector, implying that additional, nearly orthogonal features (as guaranteed to exist by the Johnson-Lindenstrauss lemma in high dimensions) should be added, not removed, to improve the reconstruction. When $||\mathbf{f}|| ≫ ||\mathbf{z}||$, it indicates either over-activation or poor orthogonality in the learned dictionary (For more details, see geometrical intuition in Appendix D).

The claim is that εLBO is more informative than just evaluating the MSE because the distribution has a more complex "shape". Could it just mean that the metric is more noisy, not informative or that it is evidence that the LRH and SH don't hold that strongly?

We agree that having a complex distribution shape does not by itself make a metric better, and we will revise or delete any parts of the paper that might have suggested this. However, the concern that εLBO may simply be a noisier metric can be addressed by the additional experiments we conducted as shown in the table below. These are the results of evaluating our final experiments (Figure 4 and Table 1, 2, and 3) not only with MSE but also with our proposed εLBO metric:

Notably, top-AFA SAEs reliably attain substantially lower εLBO values compared to top-k variants, suggesting that εLBO can be effectively minimized without compromising reconstruction quality. This shows that low-noise εLBO is not impossible to achieve per se, but rather that conventional k-sparse autoencoders fail to learn εLBO effectively.

Sparsity greater than 10^3 is probably useless and the learned features are probably not interpretable or at least as interpretable as neurons. Why use top AFA at all?

We agree that the usefulness of a sparsity as high as 1k-2k in a dictionary size of 12k is an important and open question. Although some studies limited $k$ to very small values less than 64 [1], recent work has also suggested that “sparsity is not a good proxy for interpretability” [2]. For example, [3] argues that maximizing sparsity is less important than minimizing description length, finding cases where sparser representations are less than optimal; thus, we do not feel that simply looking for small L0 is a fair comparison. Further, considering the diverse linguistic and semantic content that may be included in each of 128 tokens in each input context, it could actually be reasonable for the number of activated features to approach 2k-3k.

[1] Bussmann, Bart, Patrick Leask, and Neel Nanda. "Batch Top-k sparse autoencoders." arXiv preprint arXiv:2412.06410 (2024).

[2] Sharkey, Lee, et al. "Open Problems in Mechanistic Interpretability." arXiv preprint arXiv:2501.16496 (2025).

[3] Ayonrinde, Kola, Michael T. Pearce, and Lee Sharkey. "Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations." NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning.

We sincerely thank you for your positive evaluation.

For a paper that is so crisp and economical with its notation, built from the ground up, it is surprisingly free with obscure jargon that is specific to this subfield e.g. "meaningfully aligned" (line 37) and "residual stream" (line 79).

We thank you for pointing out terminology that is not clear to readers. Regarding “meaningfully aligned,” we will revise the sentence to more clearly emphasize a limitation in prior work: although existing methods aim to construct sparse features corresponding to inputs, the selection of sparsity levels in these methods is independent of the inputs themselves.

For the term “residual stream”, we will add a clarification in the main text to explicitly define it as the hidden embedding vector passed through the residual connections between transformer layers. We will also link this to Appendix B, where the technical background is described in more detail, and point to existing literature that uses this term to ensure consistency with prior work.

Unlike the rest of the paper, the abstract is a big mess; more of a story about how the paper is structured than what the paper delivered. Weirdly, the arbitrariness of $k$ is introduced in the first sentence, then completely ignored for the remainder of the abstract until almost the end, with nary a bit of logical connection in between. I thought this paper would be an impossible mess based on the abstract but it was (mostly) pretty clear. Please rewrite the abstract for clarity.

We agree and thank you for this candid feedback. We have rewritten it as below.

Updated Abstract

Sparse autoencoders (SAEs) are widely used in mechanistic interpretability research for large language models; however, the state-of-the-art method of using $k$-sparse autoencoders lacks a theoretical grounding for selecting the hyperparameter $k$ that represents the number of nonzero activations, often denoted by $\ell_0$. In this paper, we reveal a theoretical link that $\ell_2$-norm of the sparse feature vector can be approximated with the $\ell_2$ norm of the dense vector with a closed form error, which allows sparse autoencoders to be trained without the need to manually determine $\ell_0$. Specifically, we validate two applications of our theoretical findings. First, we introduce a new methodology that can assess the feature activations of pre-trained SAEs by computing the theoretically expected value from the input embedding, which has been overlooked by existing SAE evaluation methods or loss functions. Second, we introduce a novel activation function, top-AFA, which builds upon our formulation of approximate feature activation (AFA). This function enables top-$k$ style activation without requiring a constant hyperparameter $k$ to be tuned, dynamically determining the number of activated features for each input. By training SAEs on three intermediate layers to reconstruct GPT2 hidden embeddings for over 40 million tokens from the OpenWebText dataset, we demonstrate the empirical merits of this approach and compare it with current state-of-the-art $k$-sparse autoencoders.

We sincerely appreciate your verification of our theoretical contributions and positive evaluation of our overall findings. Your summary suggests a strong grasp of the paper. Are there any clarifications we could provide that would help raise your confidence in the assessment? Thank you again for your time spent reviewing our paper.

Question 3

How statistically significant are the performance improvements? What are the confidence intervals, and how many random seeds were used? This might help provide clarity on the improvement claims.

This is an excellent and important question. In line with prior SAE training setups that use online learning [1, 2], our evaluation was based on the L2 reconstruction loss measured on the final mini batch after 2K training steps. Typically, random seeds are not varied in this setting because SAEs are simple single-layer models with a large number of randomly initialized weights, and the effect of random seeds tends to be negligible. Nevertheless, we agree that assessing statistical significance is valuable. While our final mini batch included 4,096 input examples, we did not perform statistical testing when aggregating the results. To address this, we conducted a t-test over the final 100 batches, comparing top-AFA with top-k and batch top-k baselines:

The results show that the MSE improvements from top-AFA are statistically significant. Notably, even for Layer 7, where performance initially appeared weaker, the extended evaluation across more batches indicates that top-AFA actually outperforms the alternatives when tested more thoroughly. We plan to include these experimental settings and results in the revised version of the paper to strengthen our main experimental claims. We sincerely thank the reviewer for raising this point, and it helped us more clearly demonstrate the empirical contribution of our method.

Question 4 and 6:

Does the proposed approach improve the interpretability of learned features compared to standard top-k/batch top-k SAEs? Do the adaptively selected features provide better semantic coherence?

How do the top-AFA SAEs perform on downstream tasks that measure feature quality, such as those in SAE Bench?

Thank you for the question. Evaluating performance on downstream tasks such as those in SAE Bench is not part of the original motivation or scope of our work, which focuses primarily on theoretical insights and architectural analysis. We agree that such evaluations would be valuable for understanding the practical utility of our method, and we leave this, as well as a qualitative analysis of learned features, as an important direction for future work.

[1] https://en.wikipedia.org/wiki/Online_machine_learning

[2] Bussmann, Bart, Patrick Leask, and Neel Nanda. "Batch Top-k sparse autoencoders." arXiv preprint arXiv:2412.06410 (2024).

Question 2 and 5

The top-AFA algorithm involves sorting and cumulative sum operations for each input. How does this computational overhead compare to simple top-k selection? Is the improved performance worth the added complexity? What are the training time and memory requirements compared to baseline methods?

We agree that computational complexity is an important consideration and took care to address it during development. The following table provides layer-wise training time and the best reconstruction performance (NMSE), demonstrating that Top-AFA is competitive in practice:

(The experiment was conducted using a single A100 GPU.)

These results show that Top-AFA achieves similar or better NMSE compared to other methods with training time that remains in a comparable range. To support our claim and explain the empirical results, we also conducted the complexity analysis as follows.

The key difference lies in the top-k selection mechanism, where sorting dominates:

Since $k < h$, the leading term for Top-AFA is $O(Bhd + Bh \log h)$, which is not higher than other activation methods.

Memory complexity can be analyzed as follows, and all methods exhibit comparable complexity, which is $O(Bd + dh + Bh)$:

Thank you again for raising this important point. We will include this analysis in the appendix of the revised version.

We thank the reviewer for the detailed and constructive feedback.

Restructuring of the Main Content

We agree that the presentation can be improved. In the revised version, we will move the description of the top-AFA algorithm (currently Algorithm 1) and its geometric intuition (from Appendix D) into the main body, immediately after Theorem 2. We believe this will better connect the theoretical framework with its practical implementation and improve the paper's overall readability.

Induced Loss Term

Question 1

The reviewer’s main concern is with the introduced loss term. The authors claim to eliminate the need for tuning k, but introduce λ in the loss, which appears equally critical. This seems to lessen the strength of the main claim about reducing hyperparameter sensitivity. More clarity on the justification of the choice of the additional loss term would also be helpful.

The addition of an additional AFA loss term might shift the optima of the loss objective from proper reconstruction, so it might be worth exploring the effect of the additional loss term. This might also point to why top-AFA might not outperform its top-K counterparts except for specific hyperparameter settings.

We would first like to clarify that the added loss term does not shift the optimum of the reconstruction objective. The L2 reconstruction loss is minimized when $D\mathbf{f}=\mathbf{z}$, and under this condition, Theorem 2 guarantees that $\mathbb{E}_\varepsilon[\|f\|] = \|z\|$. We will add this explanation to the revised paper to better clarify the role and effect of the loss term. Further, as noted in our Limitations section, we acknowledge that the justification for introducing the additional loss term could be strengthened, and a deeper theoretical and empirical analysis of this loss is left for future work.

We thank the reviewer again for the valuable feedback. We would like to gently remind you that the experimental results we shared address your concerns and further support the contributions of our work. As we near the end of the discussion phase, we respectfully ask if you would consider revisiting your evaluation in light of these clarifications and additional results.