• The synthetic data experiments take up the majority of the paper, but these experiments have a couple large problems. The first problem is that the scale of the experiments is extremely small compared to actual SAEs, so it is hard to know how the synthetic results generalize. E.g. the following is a table containing my estimates for each quantity in reality vs. the experiments:

• Second, the distribution of latents used to generate the synthetic data is gaussian, but actual data is very much not so, see the plethora of recent work on structure in SAE latents (e.g. Not All Language Model Features Are Linear 2024, The Geometry of Categorical and Hierarchical Concepts in Large Language Models 2024). To be fair, the authors do note this second point in their limitations section.
• Experiments on GPT2 only compare SAEs and MLPs; why not SAE w/ ITO and sparse coding? Is it because sparse coding is computationally prohibitive? If so, this feels like a weakness that should be mentioned.
• The performance difference of the models in section 5 may be mostly due to different amount of training compute (MLPs are more expensive to train) or the different number of dead neurons. The authors acknowledge that a weakness of their work is not using SOTA SAE architectues like topk, jumprelu, or gated; since these architectures lead to less dead features, I suspect that it might erase most of the difference between the models. (There now exist very good libraries for easily training SOTA SAEs, e.g. SAELens or the Eleuther SAE library).
• The real model experiments are limited to one model, one layer, one SAE size, one sparsity, and one SAE architecutre. It would be much more helpful to vary at least 3 or 4 of these options.
• Figure 1 is confusing on first reading it, I didn’t really understand it until I read the proof. At least in mech interp, people don’t usually think of the “decoder” as producing activation space, but instead as the part of the SAE responsible for mapping estimated latents to the reconstruction. I think you should replace “decoder” with something like “linear down projection”.
• The theory shows SAEs cannot perfectly reconstruct sparse overcomplete bases, but maybe they almost can. Do you have some sense of how close an approximation with a linear projection followed by relu could be?

• The experiments systematically comparing SAE performance with MLP encoders, SAE_ITO, etc. are done only on synthetic data which is generated with some strong assumptions that could not be representative of real LLM activation vectors. For instance, for the "Unknown sparse codes and dictionary" results, a dataset of 2048 samples was used, but real-world SAE training is typically done with hundreds of millions of samples in the single-epoch regime. There are limited experiments on real LLM activations.

1. The theoretical argument in this paper seems narrow and does not necessarily support the conclusion that SAEs cannot learn sparse codes. The proof of theorem 1 argues that since the weight matrix of the encoder has rank at most $M$ (signal dimension/number of measurements), the encoder cannot recover all sparse codes since that would require a matrix of rank $N$ (sparse code dimension). This seems to contradict prior evidence from sparse recovery algorithms and work on autoencoders. If we apply this argument to a sparse coding algorithm like Iterative Soft Thresholding / Matching pursuit which also uses the low rank matrix $D^\top D$ (where $D$ is an overcomplete dictionary), we arrive at a contradiction since such sparse coding algorithms can provable recover sparse codes from even noisy measurements. Moreover in [1], the authors show (in theorem 3.1) that one layer of a ReLU autoencoder can recover the support of the sparse code with high probability if the weights are close to the ground truth dictionary $D$. Reference [2] shows that autoencoders trained to minimize reconstruction error with gradient descent can recover the ground truth dictionary $D$. The discrepancy between theorem 1 and these observations can possibly be explained by the fact that both sparse coding algorithms like ISTA and ReLU autoencoders use a nonlinearity between the encoder $W_e$ weights and the decoder weights $W_d$, and they do not try to predict the sparse code directly. Analyzing SAEs in this setting can resolve the apparent "amortization gap" identified by the authors.

2. Connections between sparse coding and autoencoders have been explored extensively in the literature. The application of deep networks to compressed sensing/sparse recovery problems has also been widely explored. This paper does not cite or engage with any of this literature.

[1] Rangamani, A., Mukherjee, A., Basu, A., Arora, A., Ganapathi, T., Chin, S., & Tran, T. D. (2018, June). Sparse coding and autoencoders. In 2018 IEEE International Symposium on Information Theory (ISIT) (pp. 36-40). IEEE.

[2] Nguyen, T. V., Wong, R. K., & Hegde, C. (2019, April). On the dynamics of gradient descent for autoencoders. In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 2858-2867). PMLR.

[3] Arora, S., Ge, R., Ma, T., & Moitra, A. (2015, June). Simple, efficient, and neural algorithms for sparse coding. In Conference on learning theory (pp. 113-149). PMLR.

[4] Refinetti, M., & Goldt, S. (2022, June). The dynamics of representation learning in shallow, non-linear autoencoders. In International Conference on Machine Learning (pp. 18499-18519). PMLR.

[5] Kunin, D., Bloom, J., Goeva, A., & Seed, C. (2019, May). Loss landscapes of regularized linear autoencoders. In International conference on machine learning (pp. 3560-3569). PMLR.

[6] Zhang, Z., Liu, Y., Cao, X., Wen, F., & Zhu, C. (2021). Scalable deep compressive sensing. arXiv preprint arXiv:2101.08024. and references within for compressed sensing with deep networks.

[7] Gregor, K., & LeCun, Y. (2010, June). Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning (pp. 399-406).

[8] Monga, V., Li, Y., & Eldar, Y. C. (2021). Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(2), 18-44.

3. The experiments seem to show that deeper MLPs perform better than shallow SAE architectures in sparse code prediction and dictionary learning problems. This is again intuitive and has been explored in the context of deep unfolding networks [7,8] and deep network models for sparse recovery (check [6] and references within). It is unclear what the new finding here is.

4. The interpretability experiments in section 5 could use more elaboration, specifically the part that describes evaluation with GPT-4. On the surface, the idea of evaluating the interpretability of LLM features using another LLM seems circular. Explaining the foundations behind this evaluation can aid the reader in understanding this section. The MLP results in this section also do not seem to be significantly better than the SAE results. Is this due to the evaluation technique, or a fundamental limitation of MLPs? The authors do not explore this.