Dense SAE Latents Are Features, Not Bugs

Thank you for your valuable feedback and for acknowledging the “novel and intriguing finding” of our work.

W1: Generality remains unclear: with experiments limited to Gemma 2, the findings might still be model-specific. Replicating the core result, especially the claim that dense latents are not SAE artifacts, on several other LLMs would strengthen the paper.

To address this point, we replicated the dense subspace ablation experiment from Section 3.1 on a second model, GPT-2. As with Gemma 2, we find that training an SAE on activations with the dense subspace ablated results in almost no dense latents, while ablating an equally-sized sparse subspace has no comparable effect. These additional results strengthen the evidence that dense latents are not artifacts of SAE training and generalize beyond a single model. We also note that Appendix A.1 already includes a replication of the antipodality results on GPT-2. We will include these additions in the updated version of the paper.

W2: The main text does not fully explain how each latent class is assigned. A unified, detailed description of the classification criteria (ideally consolidated in the appendix) is needed, including justification for heuristic thresholds such as the chosen Spearman cut-offs.

We detailed in Appendix E.1 the tests and cutoffs we used for each category of dense latents. We chose heuristic thresholds by identifying latents with outlier scores for the corresponding test (e.g. for Spearman cut-offs, see Figure 13 in Appendix E.2; for AUC cut-offs, see Figure 19 in Appendix E.6), and estimating a reasonable cut-off. We acknowledge that there is some noisiness to choosing fixed cut-offs in this way, and we do not claim that our classification is fully accurate but serves more as a guide. We will improve the clarity of the consolidated table in the final version, with references to the corresponding plots.

Q1: In Section 3.1, how exactly are the dense-latent components removed before retraining the SAE?

To ablate this subspace from the language model’s residual stream, we remove the component of the residual that lies within it. Formally, let $\mathbf{W} \in \mathbb{R}^{k \times d}$ be the matrix of the latents’ decoder weights and $x \in \mathbb{R}^d$ the residual stream. We compute the projection $\mathbf{p} = \mathbf{x} \mathbf{W}^* \mathbf{W}$ (where $\mathbf{W}^*$ is the pseudoinverse of $\mathbf{W}$) and remove it: $\mathbf{x}’ = \mathbf{x} - \mathbf{p}$. This yields the ablated residual stream $\mathbf{x}’$. We will add an appendix section to clarify this procedure in the final version of the paper.

Q2: Because the taxonomy relies on a properly trained SAE, could under- or over-fitting of the autoencoder bias the classification? Wouldn’t a direct statistical analysis of the raw LLM activations avoid this dependency?

This is a good question. We actually do study how the number of dense latents changes across training (Appendix A.2), and we observe that the number of dense latents stabilize early in training. Therefore, we do not expect that they are a result of under-/over- fitting.

As for the direct statistical analysis of the raw LLM activations, dictionary learning is necessary to isolate and identify the directions that are frequently present in the activations, and it is unclear how we would otherwise find the “dense” subspaces from the LLM activations alone.

Q3: Are there latents that play more than one role, for example, units that behave both as position latents and context-binding latents?

In our analysis, we did not find clear evidence of latents that play multiple roles. Due to the nature of our classification using the chosen heuristics and cutoffs, there are a few (3.6%) dense latents that fall into more than one category. Most overlaps are between sentence- and paragraph- tracking latents (as discussed in Appendix E.2, Figure 15), as well as between meaningful-word and nullspace/alphabet latents (likely due to the cutoffs chosen).

Q4: PCA latents are reported, but what interpretive value do they have—how should we understand their function?

We were unable to form hypotheses on interpreting PC-aligned latents by examining the activations. However, they could be related to rogue dimensions [1], as LLM activation spaces are highly anisotropic.

In conclusion, we thank the reviewer for their constructive comments. Their feedback prompted additional experiments and clarifications that we believe improve both the clarity and robustness of our work. We hope our responses address the reviewer’s concerns.

—

[1] Timkey, William, and Marten Van Schijndel. "All bark and no bite: Rogue dimensions in transformer language models obscure representational quality." (2021).

Thank you for acknowledging the originality of the work, and its significance towards understanding of SAEs. Your positive feedback is appreciated. We have addressed each of your questions below.

W2: The discussion of related work (background information) could be improved.

We agree and have significantly expanded the related work section in response. The new version provides a more thorough discussion of the previous studies that are most relevant to our work. The revised section is provided below (with newly added portions highlighted in bold).

—————————————————————————————————————————————

Interpreting SAE Latents.

As SAEs have gained traction, recent work has focused on interpreting the meaning of their latent features [28,44]. Building on the neuron interpretation methodology introduced by [45], several recent works aim to interpret SAE latents systematically. Templeton et al. [6] propose a rubric-based evaluation method in which a language model (Claude 3 Opus) scores how well a proposed feature description aligns with the contexts on which the latent activates. Similarly, Paulo et al. [46] propose a pipeline in which natural language interpretation for SAE latents are matched with different contexts and used by an LLM in different tasks that evaluate how good the interpretations are in predicting activating and non-activating contexts. Other recent efforts explore automated interpretation approaches based on self-interpretation strategies [47]. A recurring observation across multiple studies is the emergence of a substantial number of dense latents, which activate on more than 10% or even 50% of tokens [8,9]. Chen and Batson [10] take the 10 most densely activating latents in a cross-layer Transcoder trained on Claude and attempt to manually interpret them, finding plausible interpretations (e.g., "activates on commas", "activates on non-terminal tokens in multi-token words") for 6 of the 10 features. In contrast, Rajamanoharan et al. [11] view dense latents as an undesired phenomenon and propose a frequency-based regularizer to discourage their emergence during training. Whether these latents reflect meaningful internal computations or arise as undesirable artifacts remains an open question. Our work addresses this question directly, showing that dense latents are persistent, characterized by a specific geometry, and often track interpretable, functional signals in the model.

Dense Language Model Representations.

Prior work has also identified dense signals in language model representations more broadly (i.e., components that encode information consistently across many tokens). Gurnee et al. [1] present a taxonomy of universal neurons that appear across GPT-2 models trained with different seeds. Among these, they identify neurons that encode positional information at each input token. Chughtai and Lau [3] similarly identify dense positional features in an SAE trained on GPT-2’s layer 0, though they do not explicitly analyze their activation density. Finally, Stolfo et al. [2] describe neurons that regulate model confidence by tracking entropy and connect them to a component of the residual stream aligned with the quasi-nullspace of the unembedding matrix.

—————————————————————————————————————————————

W3.3: What is the meaning of "a linear combination of the "true" basis works for reconstruction too"?

We apologize this was unclear; we had to cut a longer discussion for space reasons. The idea here is that if some set of underlying LLM signals are dense and co-occur perfectly, then any basis of this dense space has the same sparsity penalty to get the same reconstruction (the sparsity penalty $L_0$ is simply the dimension of the dense space), so there is no reason that the "true" basis of features would get learned. Although dense latents are not perfectly co-occurring, some co-occurrence could still cause the learned basis to be slightly rotated from the "true" basis.

W5: Would it be interesting to plot distributions of latents (like Figure 11) depending on the frequency?

Good question. If we color the points in Figure 11c by activation frequency, the trend remains similar: dense latents cluster near (-1, -1), consistent with the antipodality pattern. This follows from the strong correlation between activation frequency and antipodal structure.

Q2: Do you know whether the presence of (almost) antipodal pairs is a consequence of the frequency of the latent (more frequent features need to be reconstructed more accurately)? Does reducing the dictionary size reduce the number of antipodal pairs?

We believe that antipodality is a consequence of the fact that dense features almost always are active, so it is hard for the SAE bias to correctly learn the "bottom" of the dense feature. To support this hypothesis, we see that reducing the size of the dictionary does not significantly reduce the number of pairs (12 pairs at 16k, 18 at 1m), but replacing the ReLU TopK activation function with an "absolute TopK" activation function (allowing activations to be both positive and negative, and taking the top-k activations by absolute value) does remove all pairs.

Q3: Do you know how much different types of dense latents overlap?

Based on our chosen tests and cutoffs, very few (3.6%) of all dense latents fall in more than one category. Most overlaps are between sentence- and paragraph- tracking latents (as discussed in Appendix E.2, Figure 15), as well as between meaningful-word and nullspace/alphabet latents (likely due to the cutoffs chosen).

—————————————————————————————————————————————

We also apologize for the clarity issues. We clarify each one below, and have added the corresponding explanations to the paper.

W1.1: Are the inputs of SAEs the features after adding the MLPs output to the residual stream?

Yes, these SAEs are "between layers", so e.g. the layer 0 SAE is after the first attention and MLP output.

W1.2: Is the subspace ablation based on encoder or decoder vectors?

We ablate based on decoder vectors.

W1.3: The definition of "antipodality score" lacks justification (the paper mentions its drawbacks), but I think that it accomplishes its goal.

Thank you for acknowledging the usefulness of our metric. We designed the antipodality score based on the intuition that antipodal pairs (latents that jointly reconstruct a single line) should exhibit cosine similarity close to -1 in both their encoder and decoder weights. We take the product of these two similarities to capture cases where both alignments are strong and quantify the extent to which a given latent participates in such a pair.

W1.4: Relatedly, in Figure 11 I find it unclear how the indices $i, j$ are chosen and whether the antipodality score (color) correspond to $i$ or $j$?

The indices are indexing SAE latents. For a given SAE latent $i$, we loop over all other SAE latents $j \ne i$, compute the similarity product from equation 1 for the pair of latent i and j, and then pick the latent $j$ that maximizes the quantity in equation 1. This latent $j$ is then used to calculate the antipodality score for latent $i$. Figure 11c then shows this antipodality score for each latent $i$.

W1.5: Why is encoder cosine similarity chosen rather than decoder cosine similarity?

For a single SAE latent, the encoder and decoder directions have high cosine similarity, so it is somewhat arbitrary which we chose to show (i.e., the decoder pattern would be similar).

W3.1: L36 "antipodal pairs, effectively reconstructing specific subspace directions." Perhaps something like "antipodal pairs, each effectively encoding a single direction" would be more clear.

Thank you, we will make this change!

W4.1: "many pairs" could be expressed more precisely.

We will change this to read many pairs ( for the SAE shown in Figure 1, there are 10 such pairs).

W4.2: Figure 11 a: Consider showing the 10% threshold.

To clarify, the plot is showing the median principle angles between each layer’s dense subspace, where the dense subspace is defined as the space spanned by only dense (frequency > 0.1) latents.

W4.3: The boxplots are a bit hard to recognize. What are the dotted lines?

The dots in the boxplots represent outliers outside the whiskers, based on the standard Plotly implementation. Specifically, whiskers extend to 1.5 times the interquartile range, and any data points beyond this range are marked as outliers. We will add a note clarifying this in the appendix of the final version.

W4.4: typo: ". and" -> ", and"

We will fix this, thank you!

W4.5: The context of Figure 16 seems to be missing. What are the run lengths of? What is the parameter of the Bernoulli distribution?

We apologize for this, this is a plot for an additional experiment that we forgot to describe in the text, and we have removed it from the paper (the average run length is the average number of tokens that each latent fires for).

W4.6: Consider including an overview of hyperparameters, models and datasets.

Thank you for this suggestion! We include chosen hyperparameters for feature cutoffs in Appendix E.1 and relevant experiment details in Appendix B, and additionally plan to open source our code.

In conclusion, we thank the reviewer for their thoughtful feedback. We hope that our responses have addressed their concerns and provided sufficient clarification.

Thank you for the response!

I have read all reviews and responses. The authors have clarified many things and provided additional analyses. I find the following particularly interesting:

• Replacing the ReLU TopK activation of the SAE with an "absolute TopK" activation, which allows negative activations, removes antipodal pairs.
• Different types of dense latents (according to the chosen thresholds) overlap very little (3.6%).

W4.2: Figure 11 a: Consider showing the 10% threshold.

I apologize for the confusion. I was probably thinking of Figure 1, but now I think it is fine either way.

Thank you for acknowledging the novelty of our analyses and the importance of studying dense SAE features.

It's not clear if the goal of the paper is to classify as many of these dense features as possible, or to argue that dense features are important/formulate a clear theory about their existence.

Our primary goal is to demonstrate that dense SAE latents are not artifacts of SAEs but instead represent meaningful internal model signals. To support this, we (1) show that these latents do not appear when a particular residual-stream subspace is ablated, and (2) provide functional interpretations for a wide range of them. The classification into six dense-latent types is not the central contribution, but serves as evidence that many dense latents correspond to genuine (and interpretable) dense signals in the language model, rather than SAE artifacts (e.g., aggregated sparse features or optimization noise). This directly challenges the previously hypothesized view of dense latents as optimization byproducts.

That said, we agree with the reviewer that some latent types, such as concept-binding features, deserve deeper analysis. In response, we have conducted new experiments to further explore these, which we describe below.

[clarifications about context-binding latents] W3: “if this is an actually meaningful latent or an artifact.” & “it is not clear to me if these bind to distinct types of concepts depending on the context (like latents shown in Fig 3), or if they also specialize on a single concept, but activate densely around it.” & Q1: “Why do we see an antipodal pair of features that track 'casino facts' vs 'looking for a buyer'. These concepts do not seem related in any way, except that they both feature in that particular context?”

Our hypothesis about context-binding latents is that the meaning they encode is context-dependent, rather than a globally monosemantic direction.

Traditionally, SAE features are assumed to activate on a consistent concept (e.g., “Golden Gate bridge”) across all contexts. We found that context-binding latents activate on consistent concepts only within each context, but the specific concept they represent varies across contexts. For instance, Figure 3 shows #7541 activating on “casino facts” in one document, and “healthcare” in another. This alone does not rule out the possibility that #7541 represents a generic “noun phrase” feature, for example. However, our experiments in Section 4.2 on steering then show that if we amplify such a feature, it influences the generation towards that specific concept, thus it less likely merely represents a generic “noun phrase” feature, but actually contains information about that specific concept.

For an antipodal pair, the two features never co-activate, thus they tend to represent two distinct ideas/themes in the passage, for instance “casino facts” and “looking for a buyer”.

To further validate our claim of “context-binding latents activate on specific but varying concepts”, we ran additional auto-interpretability experiments (adding onto Appendix E.5), and will include more examples in the paper, of within-context versus across-context explanations of the latent. Furthermore, if we ask an LLM judge to rate the specificity of explanations, dense latents have within-context explanations that are much more specific than across-context explanations—in contrast with interpretable sparse latents which have similar specificity within-context and across-context. We will include these additional results in the final version, and thank the reviewer for bringing up the clarification on this claim.

W1: What is the relevance of the antipodality? & Q1: Why is antipodality relevant in the context of density?

Antipodality is not the central phenomenon we seek to explain but rather a structural byproduct of how SAEs encode dense residual-stream signals. We hypothesize that antipodal pairs emerge due to the one-sided nature of ReLU activations: a single ReLU unit can capture a feature only if it also learns an appropriate bias value that shifts the activation range to match the full span of the signal. When this signal is dense (i.e., different from 0 most of the time, unlike sparse features) this offset is hard to fit, so the SAE learns a second latent to “complete” the signal, yielding a latent pair whose encoder/decoder weights lie on opposite sides of the same line (i.e., antipodal).

To test this hypothesis, we trained an “Absolute TopK” SAE, where we remove the ReLU to allow the activations of each latent to be both negative and positive. We enforce sparsity by using the TopK most extreme activations for reconstruction. This removes the one-sided constraint and should allow a single latent to encode the entire dense signal. The results confirm our hypothesis: the antipodal structure disappears under this activation. Furthermore, compared to the corresponding TopK SAE, the number of dense latents is smaller and the density of dense latents is higher, as antipodal pairs map roughly to single latents. We will include these results, along with the above explanation, in the final version. We thank the reviewer for prompting this clarification and follow-up experiment.

W1: “Some features, like entropy regulating and positional features have been discovered before, as the authors note in the related works section.”

We would like to clarify that only one of the prior works mentioned in our “Dense Language Model Representations” section actually studies SAE latents: Chughtai and Lau [48] identify positional signals in an SAE trained at GPT-2’s layer 0, but do not analyze their density. The other two works (Gurnee et al. [20] and Stolfo et al. [13]) focus on individual neurons in the model and not on SAE features. Stolfo et al., for example, study a confidence-regulation mechanism associated with the unembedding nullspace, but do not relate it to SAEs. We have clarified these distinctions in our revised related work section.

W2: The related works section is extremely brief. The authors mostly just cite work without discussing it. We agree and have significantly expanded the related work section in response. The new version provides a more thorough discussion of the previous studies that are most relevant to our work. The revised section is provided below (with newly added portions highlighted in bold).

—————————————————————————————————————————————

Interpreting SAE Latents.

As SAEs have gained traction, recent work has focused on interpreting the meaning of their latent features [28,44]. Building on the neuron interpretation methodology introduced by [45], several recent works aim to interpret SAE latents systematically. Templeton et al. [6] propose a rubric-based evaluation method in which a language model (Claude 3 Opus) scores how well a proposed feature description aligns with the contexts on which the latent activates. Similarly, Paulo et al. [46] propose a pipeline in which natural language interpretation for SAE latents are matched with different contexts and used by an LLM in different tasks that evaluate how good the interpretations are in predicting activating and non-activating contexts. Other recent efforts explore automated interpretation approaches based on self-interpretation strategies [47]. A recurring observation across multiple studies is the emergence of a substantial number of dense latents, which activate on more than 10% or even 50% of tokens [8,9]. Chen and Batson [10] take the 10 most densely activating latents in a cross-layer Transcoder trained on Claude and attempt to manually interpret them, finding plausible interpretations (e.g., "activates on commas", "activates on non-terminal tokens in multi-token words") for 6 of the 10 features. In contrast, Rajamanoharan et al. [11] view dense latents as an undesired phenomenon and propose a frequency-based regularizer to discourage their emergence during training. Whether these latents reflect meaningful internal computations or arise as undesirable artifacts remains an open question. Our work addresses this question directly, showing that dense latents are persistent, characterized by a specific geometry, and often track interpretable, functional signals in the model.

Dense Language Model Representations.

Prior work has also identified dense signals in language model representations more broadly (i.e., components that encode information consistently across many tokens). Gurnee et al. [1] present a taxonomy of universal neurons that appear across GPT-2 models trained with different seeds. Among these, they identify neurons that encode positional information at each input token. Chughtai and Lau [3] similarly identify dense positional features in an SAE trained on GPT-2’s layer 0, though they do not explicitly analyze their activation density. Finally, Stolfo et al. [2] describe neurons that regulate model confidence by tracking entropy and connect them to a component of the residual stream aligned with the quasi-nullspace of the unembedding matrix.

—————————————————————————————————————————————

In conclusion, we thank the reviewer for their constructive feedback. Their comments prompted several clarifications (regarding the role of context-binding latents and the broader framing of our contributions), as well as additional experiments on antipodality and context-binding features. We believe these additions substantially strengthen the paper. We hope these improvements will encourage the reviewer to reconsider their evaluation.

Thank you for highlighting our “careful empirical analysis” and recognizing the novelty of our study. Your positive feedback is appreciated. We address each of your points below.

W1: While the paper makes a strong case that dense latents are not artifacts, it could further contextualize this by considering known geometric properties of LLM representations

We appreciate this suggestion. We indeed observed a connection between the nullspace-aligned latents and outlier features [1]: the high norm that characterizes outlier features often lies in the unembedding quasi-nullspace, the same subspace tracked by nullspace-aligned latents. We will elaborate on this connection in the final version of the paper.

W2: Are the features learned by sparse latents of a different nature, or do they partially overlap in function? & Q1: How do the types of features recovered by sparse latents compare to those recovered by dense latents?

In general, dense latents appear to represent more “functional” or “structural” information (e.g., token position, lexical confidence) rather than semantic content (see Section 5.1 in Chen et al. [2] for a discussion of semantic and structural features). This is consistent with the intuition that semantically interpretable features typically appear sparsely across a text (e.g., a feature representing the concept of “dog” is expected to be active well below 10% of the tokens of a large corpus), whereas functional information related to confidence or token position is expected to be present more densely across tokens.

W3: Taxonomy completeness unclear & Q2: What fraction of dense latents do not fall into your taxonomy? Are there properties that reliably predict whether a dense latent is interpretable?

Our classification into six latent types is intended as evidence for the functional roles that dense latents can play. Coverage varies by layer (from 14% to 81% of dense latents per layer) amounting to an overall coverage of roughly 46%. We did not identify simple metrics that predict whether a dense latent is interpretable; interpretation generally requires specific geometric or activation pattern tests, as described in our methodology.

W4: Some empirical observations (e.g., context-binding latents activating on "specific but varying concepts") are not clearly explained and may need clarification or better examples.

Thank you for pointing this out. We will clarify that context-binding latents tend to activate on coherent concepts within the same input, but these concepts vary across inputs. For example, in Figure 3, we see that #7541 activates on “casino facts” in one context, but “healthcare” in another. Furthermore, these context-binding latents come in pairs, and since pairs are antipodal, each tends to activate on a distinct concept from its pair (e.g. #7541 activating on “casino facts”, and its pair #2009 activating on “looking for a buyer”, two concepts that are both present in the context but distinct).

Traditional autointerpretation of SAE latents [3] uses activating examples from the entire corpus (across-context), assuming a specific consistent concept for each latent. We suggest also autointerpreting SAE latents using activating examples within-context. We have since conducted more autointerp experiments (adding on to Appendix E.5), so we will provide more examples in the final version. For many sparse features, if explanations are specific within-context, the explanations are similarly specific with across-context examples (e.g. a feature always for “governmental bodies”). For context-binding latents, the explanations are specific within-context (e.g. “creation, sharing, or replication of digital content”), but vague across-context (e.g. “specific noun phrases, including proper nouns and descriptive phrases”). This is what we mean by “specific but varying”, and we have since quantified this claim further by using an LLM judge to judge specificity. We thank the reviewer for prompting this clarification.

Q3: In Section 3.1, how exactly is the dense subspace identified and ablated? How large is it, and what is the impact of its ablation on downstream SAE behavior?

We identify the dense subspace by collecting the decoder weight vectors of the dense latents from a first SAE. To ablate this subspace from the language model’s residual stream, we remove the component of the residual that lies within it. Formally, let $\mathbf{W} \in \mathbb{R}^{k \times d}$ be the matrix of these weights and $\mathbf{x} \in \mathbb{R}^d$ the residual stream. We compute the projection $\mathbf{p} = \mathbf{x} \mathbf{W}^* \mathbf{W}$ (where $\mathbf{W}^*$ is the pseudoinverse of $\mathbf{W}$) and remove it: $\mathbf{x}’ = \mathbf{x} - \mathbf{p}$. This yields the ablated residual stream $\mathbf{x}’$. For comparison, we apply the same ablation procedure using a random subset of sparse latents (ensuring dimensionality matches the dense subspace). In the experiments of Section 3.1, the dimensionality of the ablated dense subspace is 46 in the SAE with 16k latents, and 10 in the SAE with 32k latents. We did not directly quantify the impact of this ablation on downstream model performance, but expect that ablating a persistent, information-carrying subspace would substantially degrade model behavior. We will add an appendix section to clarify this procedure in the final version of the paper.

Q4: Could the antipodal structure of dense latents be an emergent by-product of L2 reconstruction loss rather than an interpretable representation structure?

This is a good observation. We hypothesize that antipodality emerges from the combination of dense signals and the one-sided nature of ReLU activations. Representing a dense signal with a single ReLU unit requires learning a strong negative bias, which the SAE may fail to achieve during training. As a result, the SAE learns a second latent to encode the missing component, forming an antipodal pair. To test this, we trained an Absolute TopK SAE by removing the ReLU allowing activations to be both negative and positive, and enforcing sparsity by taking the TopK most extreme activations for reconstruction. This removes the one-sidedness and allows a single latent to capture both sides of a dense signal. As expected, antipodal structure disappears under this activation. We will include this new experiment and explanation in the final version of the paper. We thank the reviewer for their detailed and constructive feedback. We hope that our responses have addressed their concerns and provided sufficient clarification.

—

[1] Timkey, William, and Marten Van Schijndel. "All bark and no bite: Rogue dimensions in transformer language models obscure representational quality." (2021).

[2] Chen, Alan, et al. "Transferring Features Across Language Models With Model Stitching." (2025).

[3] Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. "Automatically Interpreting Millions of Features in Large Language Models." (2024).