Scaling and evaluating sparse autoencoders

We thank the reviewer for the detailed review.

• For the probe loss baselines: the residual stream neuron baseline performs much worse; we do not present results on probes trained on the entire residual stream, but they perform substantially better than single latents.
• For N >> d, it’s intuitive that the best probe across N random directions will be better than the best probe across the d basis directions, especially since the residual stream doesn’t have a privileged basis (so the basis is arbitrary), so it would make sense for the loss to start out better than the single-channel baseline.
• We will add a sentence to the camera ready clarifying that the scores reported are averaged across probing datasets.
• We can add some additional interpretation of figure 7 to the main body.
• There was a latex error that caused all instances of L0 and L1 in the main text to go missing; this will be fixed in the camera ready.
• Line 384 "trie" is correct (https://en.wikipedia.org/wiki/Trie)
• We will cite Conerly (2024) in section 2.4 as concurrent work.
• The links are indeed broken for anonymization reasons and will be fixed in the camera ready version.

We thank the reviewer for the detailed review.

• The intuition for the irreducible loss term is that SAEs can only efficiently model the distribution if there is some structure (i.e there are features that compose together to explain the distribution well). If there is no structure, then it is very inefficient to model the distribution by increasing the number of latents. When we say “components with different amounts of structure”, what we mean is that the residual stream might be a linear combination of components that scale differently with number of latents due to different amounts of composability.
• We agree that the lack of shrinkage in topk is not surprising. Our goal in section 5.1 is just to double check this prediction, and to see whether this explains the entire difference between topk and relu.
• The probing datasets were chosen somewhat arbitrarily, based on a subjective sense of what kinds of features were likely to occur. We’re excited about future work further expanding the size and diversity of the set of tasks.
• There was a latex error that caused all instances of L0 and L1 in the main text to go missing; this will be fixed in the camera ready.
• The aux loss is often hard to interpret because as the main loss improves, the task represented by the aux loss changes. To see how much reconstruction quality we are paying for sparsity, we can compare autoencoders with different k (see figure 1b).
• We’re excited about the concurrent and subsequent work on jumprelu, MoE SAE, and sparse feature circuits. We can include citations to work on these directions in the related work section for the camera ready.

We thank the reviewer for the detailed review.

• We agree that feature splitting limits the usefulness of probes on individual latents; as a result, we chose to consider latents before applying the activation function, so that split features would still correlate strongly with our labels. In some unpublished experiments we explored the qualitative interpretation of the chosen latents but found that there was a tradeoff between higher qualitative quality of latents when considering post-activation, and greater robustness to feature splitting. We omitted this discussion from the paper because our results were relatively preliminary.
• We agree that restricting to exactly k active features is not ideal, and we’re excited about follow-up/concurrent work like batch topk and jumprelu. We also have some results on replacing the topk activation function at test time with a jumprelu with only a moderate hit to the MSE-L0 frontier. We can include these results in the camera ready. We can use N instead of n everywhere for number of latents.
• While we focus mostly on aggregated quantitative metrics, because we are interested in choosing between different autoencoder training setups, we are also excited about future work on more detailed qualitative investigation of these metrics. We do present qualitative results for some metrics, such as the n2g explainability metric.
• There was a latex error that caused all instances of L0 and L1 in the main text to go missing; this will be fixed in the camera ready.
• Yes, we used the scaling laws to choose hyperparameters for the largest runs. While it’s hard to know for sure whether the choice was optimal, our largest run appears to be on trend in our L(C) scaling law.
• There are several directions for evaluating precision that we’re excited about - one approach is to use a LM to generate sequences matching the explanation, and adjusting for the distribution shift; another approach is to use cheap heuristics as a first stage filter before applying more expensive explanation simulation techniques.

We thank the reviewer for the detailed review.

• We think variable k sparsity is an exciting direction for future work. We did some initial explorations of this direction but ultimately did not include any in the paper.
• We agree that experiments with a longer context length would capture more interesting features. We’re excited about future work exploring this in more detail.
• It is true that some of the metrics presented require substantial computational power. We believe that it is nonetheless valuable to present metrics which are expensive. In addition, we also present inexpensive metrics (probe loss, logit diff sparsity) for use when using expensive metrics is infeasible.
• The auxiliary loss represents a substantial simplification of methods used in previous work for dead latent prevention. Additionally, we find in practice that the optimization is not very sensitive to the auxiliary loss hyperparameters. We swept the auxiliary loss coefficient hyperparameter at small scale and found that it is not very sensitive. We believe that it is also not sensitive at large scale, but it would be prohibitively expensive to check. In appendix B.1 we give the recommendation to use a value of 1/32 for this coefficient.
• As for scalability, we demonstrate that our approach scales smoothly to the largest existing language models (GPT-4).
• We’re excited about future work demonstrating applications of sparse autoencoders to various tasks; however, it would be out of scope for this paper.
• The illusion of interpretability effect occurs when the explainability is determined only on the basis of documents that activate the latent (i.e it only measures recall). We use N2G precisely because we want an explainability metric that measures both precision and recall.
• Human judgment of interpretability is highly nontrivial and requires human interaction with the model in order to evaluate both precision and recall. We’re excited about future work exploring this direction but we felt it would be out of scope for our paper.

Given that the deadline for making changes is fast approaching, please let us know if there are any changes we can make to address your concerns.

Thank you for the follow-up comment.

• We have some results that demonstrate that it is possible to replace the TopK at test time with a fixed threshold (thereby allowing variable sparsity); furthermore, it is possible to train on multiple k's at the same time to create autoencoders where k can be varied at test time. We are unfortunately past the deadline for updating the paper in this phase of reviewing, but if the ACs will allow it, we can include these results in the appendix for the camera ready. Our work was in fact conducted before the batch-topk work, and posted to arxiv and was not reviewed at NeurIPS. The batch-topk work was inspired by our work and introduces an incremental improvement; it does not demonstrate the ineffectiveness of topk, but rather that topk performs almost as well as batch-topk.
• While it is certainly the case that longer contexts will involve more features, and it would have been ideal for us to also double check longer contexts in our paper, we believe that our experiments nonetheless provide strong evidence for scalability of our techniques. First, GPT-4 already exhibits highly nontrivial behavior within 64 tokens. Second, to explain the behavior on this context length required us to push the frontiers of sparse autoencoder training to train the largest sparse autoencoder (by # of alive latents) ever created, requiring the development of new scalable approaches. Third, it would be very surprising if there were very clean scaling laws at short context length, but no such clean scaling law at longer context lengths. Finally, it's not like GPT-4 is an ancient outdated model from an era when language models only supported 64 tokens; GPT-4 is a very recent model and models supporting thousands of tokens of context have existed for years, and we simply chose to only use 64 tokens of that context for reasons of convenience.
• The computational cost of these metrics comes primarily from factors other than context length - for downstream loss, contexts on the order of thousands of tokens costs negligibly more (because you get the loss on every token in the context anyways), and for explainability, the main source of cost is rarity of the features, not context length.
• Across an order of magnitude of choices of coefficient and auxk hyperparameters (4x4 runs in total), in one setting we explored, we found that the best and worst configurations had reconstruction losses varying between 0.0127 and 0.0122 (4% relative difference).

We thank the reviewer for the detailed review.

• One major advantage of being able to set the sparsity directly is it makes it much easier to do scaling laws where sparsity is held constant.
• We’re excited for future work on whether top-k finds the same kinds of latents as other autoencoder methods, how top-k behaves on toy models of superposition, whether longer prompts than 64 tokens have better results.
• We do briefly compare TopK and ReLU on probe loss and explainability in figures 23 and 24, but we’re excited about more in-depth future work on how different autoencoder methods perform on metrics other than reconstruction/sparsity.
• To avoid the issue of TopK preventing values from being calculated independently, the TopK activation function can be replaced by a JumpReLU at test time chosen to target the same L0. This leads to a small increase in MSE (at comparable L0), but this increase can be largely reduced by training with multiple k values. These results were cut from the submission, but we can add them back in the camera ready.
• We’re excited for future work that focuses on comparing and refining autoencoder metrics.
• We generally find that linear probes on the entire residual stream are substantially better than the best probes on single autoencoder latents.
• In some unpublished experiments we explored the qualitative interpretation of the chosen latents but found that there was a tradeoff between higher qualitative quality of latents when considering post-activation, and greater robustness to feature splitting. We omitted this discussion from the paper because our results were relatively preliminary.
• We focus on k=1 because of its conceptual and computational simplicity, but we’re excited about future work on k-sparse probes for k > 1 for autoencoder evaluation.
• While we don’t directly compare the explanations produced by n2g and auto interpretability in our paper, we do explore the auto explanation scores in figure 19. We generally find that auto interpretability scores are very noisy.

This basically breakes double blind. Maybe not a suitable comment to make.

I like the paper, but direct appeals to the referees from high-profile allies cannot be part of the review process. If public discussion opens the ACs to Oscar-style award campaigns, then ICLR shouldn't allow comments at all until after decisions are finalized.