Steering Large Language Model Activations in Sparse Spaces

We appreciate your thorough review and the constructive feedback provided. Below, we address each of your concerns in detail.

Before diving into individual points, we would like to clarify a few aspects of our work that may not have been sufficiently emphasized in the current draft:

• SAS effectiveness: Our goal is not to outperform dense steering but to match its effectiveness while gaining interpretability and control.
• Sparse space and notation: “Sparse space” is a standard term and been used before [1]; here it refers to high-dimensional vectors in which most dimensions are zero. In the SAE literature, sparsity is constrained using non-linearities such as ReLU (or in our case JumpReLU), which enforce both sparsity and positivity. We clarify that $\sigma$ refers to JumpReLU, not sigmoid.
• SAE details: We use pretrained SAEs from Gemma Scope [2]; we do not train them ourselves.
• Evaluation diversity: We included multiple-choice, open-ended, and effectiveness evaluations on standard benchmarks (MMLU and TruthfulQA). The “LLM-as-a-judge” approach is now common in recent work (e.g., Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena).
• Detailed algorithmic steps: While Figures 1 and 2 should help visualize the method, detailed algorithmic steps are provided in Appendices E and F.

1. Comparison with Dense Activation Steering

In Section 4.1, we present a comparative analysis between SAS and dense steering, focusing on behavioral modulation. Figure 4 shows SAS achieves comparable performance. Our goal is not stronger steering, but more interpretable and controllable behavior modulation, so matching dense performance is meaningful.

2. Clarity of Notations

We will revise the paper to:

• Define all mathematical symbols and tensor dimensions.
• Clarify that $f_l$ and $a_\ell$ refer to encoder/decoder functions from pretrained SAEs.
• Explicitly state that $\sigma$ is JumpReLU.
• Clarify "sparse space" as the latent space of the SAE with many zeros, as commonly used in sparse coding literature.

3. Evaluation Metrics

While we primarily use LLM-as-a-judge, this has become a standard scalable metric. From Zheng et al., 2023, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena":

“LLM-as-a-judge is becoming standard practice in LLM evaluation pipelines due to cost and scalability.”

From Chiang et al., 2023, "Can LLMs be Better Judges than Humans":

”We find that LLMs often provide more consistent and informative feedback than humans.”

However, we agree that additional human evaluations or alternative metrics would be valuable, and we plan to explore these in future work. Nonetheless, a thorough set of generation examples for each steering behavior is provided in Appendix N, “Open-Ended Generations Examples.”

4. Sections 4.4 and 4.5 Evaluations

We view these sections as important for understanding not just whether SAS works, but also why it works and how it can be composed and scaled. They show:

• Different behaviors can be composed using SAS vectors to have simultaneous effect.
• Wider SAEs show a trend toward monosemantic SAS vectors. While the number of raw extracted features remains roughly the same—especially beyond a width of 65K (Figure 7, middle)—only a few correspond to SAS (Figure 7, left), indicating that concepts are becoming increasingly monosemantic.

5. Specific Questions and Clarifications

• "Sparse space": Latent space where SAE outputs have mostly zero entries.
• The same color coding is used in all figures, especially in Figures 1 and 2.
• Symbols and shapes: Will be explicitly defined in the revision.
• Encoder/decoder forms: From pretrained Gemma Scope SAEs (JumpReLU-based, for Gemma-2B/9B).
• Sparsity vs. non-negativity: Non-negativity is enforced by JumpReLU in the pretrained SAE; we use the same activation function since the decoder expects non-negative codes.
• Feature filtering heuristics:
  • Threshold $\tau$: Vary the number of active features (i.e. controlling sparsity); ablation provided in Appendix C.2.
  • Removing common features: Because common features almost cancel each other out and are concentrated around zero, removing them further makes the final SAS vector sparser with no performance downgrade; ablation provided in Appendix C.4.
  • Subtracting the two final vectors: This further helps reinforce the desired behavior and suppress the model tendencies already present in the activation; ablation provided in Appendix C.3.

Thank you very much for your thoughtful and encouraging review. Your remarks on the interpretability and compositionality advantages of Sparse Activation Steering (SAS) are especially motivating for us.

Response to your main suggestion — larger-scale models

New experiments now available. As noted in your comments, additional evidence on larger and more diverse LLMs would strengthen the work. Therefore, we have experimented with the multiple-choice steering experiments on the newly released Llama-3.1-8B SAE checkpoints from the Llama-Scope project [1]. The results closely track those reported for Gemma models. A preliminary figure is included below; the full experimental details (hyperparameters, and qualitative outputs) will appear in our next revision.

New Llama results: [reference link]

[1] He, Zhengfu et al. “Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders,” 2024.

Thank you for the thoughtful review. Below, we address your concerns and questions.

Limited backbone (Gemma).

New experiments now available. Recently, the Llama-Scope project [1] released SAE checkpoints for the Llama-3.1-8B model. We have reproduced the multiple-choice SAS-steering experiments on these checkpoints; the trends mirror those in Gemma (see preliminary plot below ($\tau=0.2, \lambda=\pm1$)). We will ablate more on these models in the next revision.

New Llama results: [reference link]

Why steering in sparse spaces yields improved modularity and compositionality.

From Toy Models of Superposition [2] we learn that when a network is forced to represent many sparse input features in too few dimensions, it stores them in superposition—several concepts share a single neuron, so an intervention on one concept inadvertently affects others. Once a sparse autoencoder (SAE) wrapped around, it discovers a sparse basis that separates these features, those different concepts become linearly independent directions.

Sparsity is therefore the mechanism that resolves superposition. Resolving superposition yields a clean dictionary of features, and a clean dictionary makes feature edits modular and their compositions linear—hence the improved modularity and compositionality we observe when steering in sparse spaces.

Questions

Cost of building a steering vector

To clarify, no gradient‐based training is performed when we build a Sparse Activation Steering (SAS) vector. For each behaviour, we simply run the model once on a small contrastive dataset (≤ 1,000 prompt pairs) and obtain the sparse representation with the SAE encoder.

Cost at inference

At inference, we load one SAE (encoder + decoder) for a chosen layer and perform:

1. SAE encode → K active features
2. Add $\lambda \times v_{b, \ell}$ to the sparse representation
3. SAE decode (plus the $\Delta$ correction)

This adds a single extra general matrix–matrix multiplication at one layer:

Gemma-2 2B's residual stream dimension is 2304.

 Practical comments (Exploration and Efficiency)

• Exploring different layers or $\tau$-thresholds only requires re-scoring already-cached sparse activations—no extra passes through the model.
• Wider SAEs yield sparser, more monosemantic vectors without hurting steering power, so practitioners can trade a modest RAM increase for better interpretability.
• If a tighter budget is needed, the SAE can be 8-bit-quantized.

[1] He, Zhengfu et al. “Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders,” 2024.

[2] Elhage, Nelson, et al. "Toy models of superposition." arXiv preprint arXiv:2209.10652 (2022).

Thank you for the thoughtful review. Below, we address your concerns and questions.

Limited range of models.

• What is already in the submission. Appendix D (Transcoder) reports multiple-choice results on Llama-3.2 1B using Transcoder sparsifiers (an alternative to SAEs that operate only on the MLP block). This preliminary result shows the same trend as in Gemma models; however, MLP representations are less expressive than the residual stream, and broader experiments are needed to draw firmer conclusions.
• New experiments now available. Recently, the Llama-Scope project [1] released SAE checkpoints for the Llama-3.1-8B model. We have reproduced the multiple-choice SAS-steering experiments on these checkpoints; the trends mirror those in Gemma (see preliminary plot below ($\tau=0.2, \lambda=\pm1$)). We will ablate more on these models in the next revision.

New Llama results: [reference link]

Limited range of representation-engineering techniques.

Our goal is to show that contrastive steering—originally implemented in dense space by CAA—transfers to sparse space without losing efficacy while gaining interpretability (sparsity, feature compositionality, and progress toward monosemanticity).

Limited scope of compositionality.

We agree that there is a lack of a systematic compositionality benchmark. While our experiment shows the effectiveness of SAE-steering on this particular task, creating such a benchmark is a natural next step, and we will highlight this explicitly in the future work section.

Why the combination of SAS and non-SAS vectors?

To clarify our compositionality study: it is designed to probe one specific question—Can a complex behaviour (e.g., Myopic Reward) be cleanly combined with a simple, atomic feature (e.g., Gender) ?

This choice follows directly from the motivation in the introduction:

• Prior work that labels individual SAE features assumes monosemanticity (one meaning per dimension), such as the non-SAS steering vectors from Neuronpedia [2].
• In practice, (i) many dimensions remain unlabeled, and (ii) complex behaviours such as myopic reward rarely align with a single feature. They are typically represented by a small bundle of partially disentangled features because the SAE has not yet fully resolved superposition—especially at smaller scales, as discussed in Section 4.5 (Scaling Monosemanticity).
• Conversely, simple and highly frequent attributes like gender can be captured by a single, confidently labeled feature.

Therefore, the Myopic × Gender case illustrates the mixed toolkit we expect practitioners to use in the sparse setting:

• derive SAS vectors for complex behaviours, and
• reuse well-labeled non-SAS (i.e., from Neuronpedia) features for atomic attributes.

Finally, note that the same SAS approach could also identify gender-specific features, but doing so would require constructing a small contrastive dataset.

Questions

Which token representation.

We always take the last token of the prompt—the hidden state used to predict the first generated token.

Sycophancy and MMLU (Figure 6).

Sycophancy steering nudges the model toward answers that closely match the majority-held, human-endorsed view of each question. While deeper investigation is required, one hypothesis is that, because MMLU’s ground-truth labels correspond to these majority-endorsed answers, sycophancy increases the probability of selecting them.

Compositionality of Hallucination and Refusal.

Although we did not show an explicit linear-combination experiment in the main text, we analyzed the relationship between these two behaviours and every other pair in Appendix L (Behavioural Correlation), Figure 25d. That figure measures, for every pair of SAS vectors, the fraction of features they share. Refusal(+) and Hallucination(–) exhibit one of the highest overlaps, which explains why steering with either vector alone already improves factual correctness in Figure 6 (right).

AS compared to SPARE?

SPARE uses SAEs to resolve knowledge conflicts—that is, it decides which knowledge source the model should trust when the current context contradicts what is stored in memory. SAS, on the other hand, is designed to steer an LLM’s behavioural traits at inference time. Because the two methods address different goals, they are complementary rather than directly comparable.

Why using $\tau = 0.2$ for transcoders.

Pre-trained transcoders are far sparser (~32 active features) than SAEs (~150). A lower $\tau$ therefore preserves a comparable absolute number of features. We will add this rationale in Appendix D.

[1] He, Zhengfu et al. “Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders,” 2024.

[2] Johnny Lin. Neuronpedia: Interactive Reference and Tooling for Analyzing Neural Networks, 2023.