The main weakness of this paper is that the central quantities are not defined clearly enough for me to be able to properly understand what is going on, or the adequacy of their empirical experiments. My current understanding, rephrased in my own language is that the authors are making the following decomposition:

• Activation vector $x$
  • The sparse autoencoder reconstruction: $\textup{Sae}(x)$
  • The error of the SAE reconstruction: $\textup{SaeError}(x)$
    • Unlearned sparse features: $\sum_{i=m}^n w_i y_i$
    • Dense feaures which will not be learned by the SAE: $\textup{Dense}(x)$.
    • Additional error: (I don't know if the authors have a term for this)
      • Linearly predictable additional error: $Wx$
      • Nonlinear error: $\textup{NonlinearError}(x)$ (where my notation is that each bullet point is the sum of the bullet points nested at one level below it). This was my understanding after having spent quite some time re-reading Section 3, so I'll proceed as if this is correct, but I'm not confident and I'd appreciate clarification from the authors.

Assuming the above, I have several comments:

• The exposition in Section 3 needs to be substantially clearer for readers to be able to understand your definitions. For example, you should name the error mentioned in "the SAE introduces some error when making this approximation" and give it a consistent name. I also think equations (3) and (4) are misleading. According to my understanding, (4) is the central definition decomposing the SAE error and (3) is a consequence of (4) when you consider subtracting from $x$.
  • As an aside, the introduction of $W$ is poorly motivated here. Also, later on you're going to consider another sense in which the error can be linearly predicted from $x$ - namely using $a^T x$ to predict $\textup{SaeError}(x)$. I believe that these are not the same, but I was initially confused here and I don't understand the difference in intuition between them.
• In Figure 1, I don't see why Dense Features and Linear Error should be grouped together. They seem like quite different quantities with different interpretations to me.
• The changes of notation and setup in the first paragraph on page 6 are very unhelpful given there's already a lack of clarity. I'd strongly recommend that the authors stick to consistent notation throughout the paper, and choose a single - clearly distinct - term and corresponding notation for each component of their decomposition and stick with it throughout the paper.
  • As an aside, if we're dropping $Wx$ in this paragraph, why did we have it in the first place? I'm not sure I ever understood the intuition for having it.
• Since you are agnostic to the SAE architecture, you don't introduce any notation for the features that the SAE learns. This lead to me getting confused and originally not understanding that the SAE reconstruction error comes from essentially four places: learning wrong features, learning wrong feature weights, unlearned linear features, and (unlearned) dense features. My understanding is that the nonlinear error is effectively measuring the first two. Is that right? If so, a clarifying comment in this direction might be helpful.

Secondly, I did not understand the description of and intuition behind the experiment in the first paragraph of page 4. I think this might be downstream of the fact that I haven't completely understood the authors' decomposition of the SAE error. But in any case, either the authors need to offer a clearer definition of the quantities they are working with, or more intuition for why this experiment claims to measure what they are hoping - and probably both.

Given this lack of clarity in a couple of key places in the manuscript, it was hard for me to engage with the more detailed experimental results in Sections 5 onwards, since I couldn't understand what the authors were actually hoping to measure. The problem that the authors are trying to study seems fundamentally very interesting, and I'm optimistic that some version of this paper could be very solid, but as it stands it's not possible to appreciate the authors' contributions.

More minor points:

• I think the synthetic experiment setup in Section 4 is unlikely to be particularly realistic - particularly the assumption of Gaussian noise. (My read of Gurnee (2024) which the authors cite regarding pathological reconstruction errors makes the Gaussian assumption likely incorrect.) But, this is not a central concern.
• I suggest that the authors read over their manuscript for minor typographical errors. Ones that I caught include: issues with citation formatting in several places, summation indices in equations (3) and (4) being incorrect, and several stray commas.

Major

To be honest, I had a very hard time understanding this paper. Specifically,

1. How do you define the term “most common” (features..) on line 146? It is ambiguous.
2. Section 4.1 is not well written, which makes me hard to understand the later sections of the paper. Specifically:
   1. Line 193 and 194 do not make sense. For instance, Dense is nonlinear, how can the sum between Wx and Dense be linear component of the error?
   2. What leads you to make the statement on line 200 to 201? You don’t even know what the “true features” are. Please elucidate.
   3. It is questionable to claim “the percent of variance left unexplained by the regression will be an upper bound on the true variance explained by NonlinearError(x).” First, please provide a clear definition of “variance explained or unexplained”, perhaps in the appendix. Second, if this is a concept similar to the explained variance in PCA, I’m not sure if you can readily extrapolate that to nonlinear components. Some rigorous derivation is needed. If you are not sure, you need to stress this is an assumption.
   4. Here the so called synthetic setup is just for confirming that the SAE is approximately an identity function on its image, i.e., the linear subspace of $\mathbf{x}'$. There is no need for this long verbosity.
      1. Also, what is the motivation for this? I think there should be more clarification.
   5. Linear transformation $\mathbf{a}$ (line 161) should be denoted by $A$ for consistency (since you have $W\mathbf{x}$). You current notation makes it look like evaluating inner product. I don’t think this is the same as the one in Section 4.2, right? Also, clarify its output dimension.
   6. The Gaussian noise simulation on line 204 to 207 is kinda questionable.
      1. The set of $\mathbf{x}$ could be a curved low dimensional manifold. In this case the Gaussian noise—whose support is a practically a ball of the same dimension as $X$—cannot accurately simulate Dense(x).
      2. The output of Dense and NonlinearError could be correlated in reality.
      3. The functions Dense and NonlinearError could be continuous, so Gaussian noise may not accurately simulate them.
   7. Since SAE is approximately an identity function according to you (line 204 to 205), we can safely assume that $\mathbf{x}’$ and SAE($\mathbf{x}’$) are identical. Then adding Gaussian noises to each of them simply makes the simulated SaeError (i.e., $\mathbf{x}$ - SAE($\mathbf{x}$)) an Gaussian noise. I’m not sure how using a linear map $\mathbf{a}^\top \mathbf{x}$ can derive the results in Fig. 2, or maybe I just got it completely wrong since I can hardly understand your writing. I think you might need to elaborate on line 202 to 207.

I’d like to stop here since Section 4.1 has already baffled me enough, making me impossible to review the later sections. I think some revision is needed to streamline the narrative.

Minor

1. On line 135, I think you mean $\|\mathbf{w}\|_0\ll d$.
2. What is called the “linear subspace of $\mathbf{x}$” (line 187)? $\mathbf{x}$ is just a point in the activation space $X\ni\mathbf{x}$. If you mean a linear subspace of $X$, then is it proper, i.e., it cannot be $X$ itself?
3. Use the correct citation format \citep and \citet. For instance, line 35.

1. Large sections of the paper are difficult to understand

a. Some notation is defined in strange ways that require me to reread them many times. E.g. SaeError(x) := x - Sae(x) is defined with a -1 coefficient for Sae(x), but NonlinearError(x) := Sae(x) - Wx - \sum_{i=0}^{m} w_i \vec{y}_i is defined with a +1 coefficient for Sae(x). I think that most of these problems are downstream of defining some parts of the notation with an end state in mind, e.g. the weak linear representation hypothesis suggests that there exists sum ideal dictionary decomposition of the vector, but other parts of the notation is defined with the current state in mind, e.g. you slot SaeError(x) into the sum with the existing Sae(x) and x terms and nothing else. However, I'm not confident that this consistently considering solely the end state or current state is either necessary or sufficient for making the notation clearer.

b. Some statements do not make sense after several times re-reading. E.g. "The intuition behind this test is that if ... its existence is not guaranteed".

c. "If this test is accurate, we can use it to estimate the linear component of the error, Wx + Dense(x)" but Dense(x) was introduced as possibly non-linear, and Wx is non-linear, so how is this the linear part of the error?

2. I think the conclusions are too strong.

The paper states "We also find that the norm of the NonlinearError(x) is constant on a per token level as we scale SAE width", and while this section changes the definition of NonlinearError to be the error as determined by their method, in the conclusion the paper states "... the presence of constant nonlinear error ..." with no hedging.

I am not convinced that the methods in the paper are capturing true linear error and non-linear error. One reason this may be happening is that all the training on the error term x - Sae(x) may involve some vestigual x due to shrinkage (which could be boosted by 10x by the various predictors, since the appendix suggests there is 10% shrinkage). This would mean the various methods that predict linear or nonlinear error may be cheating and picking up on this shrinkage. Is this addressed in the paper? Why is there no hedging in the conclusion that the methods presented may not capture true (non-)linear errors, or even that the weak linear representation hypothesis may not be true. For example, in the extreme case where the SAE was entirely dead, we can predict SaeError(x) = I * x and so the SAE solely has linear error, which suggests there are some assumptions that need to be stated (that the SAE is sufficiently good at reconstructing I think).

Weaknesses The main weakness is that the method is only applied in one model, and even worse, only one layer in the model. The methodology is self described as empirical, but because only one context is analyzed, the findings are not sufficiently proven because the paper essentially shows only one data point. That is, is it possible that this only happens in this one layer for this one model? I would not think that the findings are exclusive to just this one context, but would it be possible to construct a LLM architecture for which this result does not hold? It should be easy to repeat the analysis for various open source models and different layers, and to thus illustrate that the observations and theory hold more widely. With just more demonstrations of the same trends across more models, the paper would be strong.

Another easy to correct shortcoming in the presentation is that details of the autoencoder training process are left out. There is a citation to GemmaScope, but given that this paper is trying to make a broader theoretical claim, a short description of the SAE training process would be apt. Similar to applying the method to more models and layers, it would also be interesting to verify if different SAE algorithms resulted in features that change the results.