Analyze Feature Flow to Enhance Interpretation and Steering in Language Models

Thank you for valuable questions.

To further validate the goal in Section 3.3 of “tracking the evolution of feature”, it would be nice to improve this step by ensuring the features considered are rarely activated on other datasets so they are tailored to the examined distribution.

We appreciate this suggestion. To address feature specificity, we estimated activation frequencies across 100k non-special tokens from FineWeb and categorized features into quantiles based on their activation rates. As expected, the results vary significantly depending on the quantile selected, supporting the need for careful feature selection.

https://anonymous.4open.science/r/icml_rebuttal-5064/quantiles.png

It is unclear whether there are preceding features that are co-activated with the target feature with high probability, but cannot be identified via cosine similarity search.

To evaluate this, we calculated the following score: the fraction of features for which the top-1 match by cosine similarity also appears in the top-k matches by Pearson correlation.

https://anonymous.4open.science/r/icml_rebuttal-5064/top_k_corr.png

https://anonymous.4open.science/r/icml_rebuttal-5064/top_k_gpt.png

https://anonymous.4open.science/r/icml_rebuttal-5064/top_k_pythia.png

We observe strong agreement for residual features in Gemma and GPT-2, while MLP and attention features show lower consistency. However, we have not yet validated whether Pearson-based matches reliably predict causal relationships or steering outcomes—a valuable direction for future work, and we thank you for highlighting this.

is it possible that a similar feature is reactivated again in later layers, even though it appears to be eliminated when examining the current target feature?

Yes, this is very common. Since deactivation typically applies only to a small subset of features, and rescaling coefficients are often modest, hidden states undergo only minor perturbations, allowing for later reactivation. We hypothesize this stems from the model’s self-repair mechanisms, which can recover information even after layer pruning (cf. [1–3]). To mitigate this, effective deactivation may require not just zeroing target features but also adding another steering vectors to the hidden state to prevent such recovery (discussed in Appendix C.2).

To test the occurrence of reactivation, we used three strategies: deactivating only the first layer in a graph, deactivating a random layer in a graph, deactivating the first half of layers in a graph. Subsequent deactivations were measured on residual nodes that comes after the deactivated layer, confirming that reactivation is present, but it depends on intervention method and strength.

https://anonymous.4open.science/r/icml_rebuttal-5064/reactivation.png

From Figure 10, how could the conclusion that “cumulative intervention outperforms the single-layer approach” be drawn, given that the blue curve shows a better best score than the orange curve?

Thank you for highlighting this. Cumulative steering generally achieves comparable effects at lower rescaling coefficients (Figure 10), improving stability. While single-layer steering can yield higher total scores, it requires either precise selection of highly impactful layers, or spanning multiple layers with a well-constructed feature graph.

This introduces additional optimization challenges not faced by cumulative steering. We will revise this conclusion in the updated manuscript to reflect this nuance.

References:

[1] [Your Transformer is Secretly Linear] (https://arxiv.org/abs/2405.12250)

[2] [What Matters in Transformers? Not All Attention is Needed] (https://arxiv.org/abs/2406.15786)

[3] [The Hydra Effect: Emergent Self-repair in Language Model Computations] (https://arxiv.org/abs/2307.15771)

Thank you for your insightful comments and questions.

1. Some layers share less cosine similarity with others, e.g., the first layer; does this affect the mapping/steering?

We observe that layers indeed vary in feature-matching quality, though we have not yet quantified this systematically (e.g., via similarity analysis of feature descriptions). Your suggestion to assess this further is valuable, and we will incorporate such analysis to strengthen our paper.

Notably, we find that certain layers (e.g., the 4th, 9th, and 18th in Gemma) exhibit differences in how well features match their top-1 similar counterparts in the MLP versus the previous residual stream. This aligns with feature clustering patterns, as shown in Figures 3 and 18b: early layers lack strong cluster separation, leading to poorer matching, while later layers converge toward a single dominant cluster with less distinct groupings.

https://anonymous.4open.science/r/icml_rebuttal-5064/out_mlp_scatters.png

https://anonymous.4open.science/r/icml_rebuttal-5064/out_attn_scatters.png

We have not identified generalizable differences in steering effectiveness across layers, as our analysis so far has focused on a limited set of features and graphs. However, we note that steering MLP or attention features often produces negligible effects, regardless of the target layer or steering coefficient. We hypothesize that this could stem from imperfect matching or the possibility that these features operate differently (e.g., requiring other active features in the residual stream to function). Further investigation is needed here.

2. More charts/graphs about the dynamics of cosine similarity?

We have measured the mean cosine similarity between hidden states at layer outputs and three SAE-related positions, plotted against relative layer distance:

https://anonymous.4open.science/r/icml_rebuttal-5064/cossim_models.png

The divergence in Pythia’s results may arise from its parallel residual stream architecture: $\text{Layer}(x)=x+\text{MLP}(\text{norm}_1(x))+\text{Attn}(\text{norm}_2(x))$. We hypothesize that higher cosine similarity between hidden states correlates with better matching quality, though this depends on SAE and model architectures—an aspect we plan to explore further.

These findings may also explain why steering MLP and attention features is less impactful than steering residual features. For additional results on correlation matching and group dynamics in Pythia and GPT-2, please see our response to Reviewer jyW6.

3. Can the co-existence of activation values or the transference be further evidence of the evolution claims?

Yes. If you refer to transference across forward layers, we find that matching via the residual stream generally works well, whereas matching with MLP or attention features in the next layer is often poor. We have prepared example graphs of activations for specific texts, available at.

https://anonymous.4open.science/r/icml_rebuttal-5064/g_3_t_0.5.png (g_3_t_0.8.png/g_3_t_0.85.png)

4. How about the sensitivity of the steering results to different threshold values?

The steering effect varies significantly depending on the graph and target feature. For instance, steering the "London" feature barely influences London-related tokens unless the coefficient is very high, at which point it introduces fashion-related terms (see [1] and Appendix E). Some features even show minimal response to steering.

Thus, different graphs react differently, and a systematic study of threshold sensitivity would require extensive experimentation across many features and setups. While we did not aim to optimize thresholds (given their dependence on model/SAE architecture, SAE quality, etc.), we acknowledge this as an important direction for future work.

For our steering experiments:

• Deactivation did not use thresholds, i.e. full graphs from 0th to 25th layers were used.
• Activation thresholds were chosen empirically: higher thresholds sometimes diluted the theme by including weakly related features, while lower thresholds reduced theme prominence.

5. How was the threshold selected?

Initially, we derived thresholds from feature group analysis (Section 5.1) and correlation studies. For example, in Gemma’s 18th layer, we colored features with >0.65 correlation to MLP (orange) or residual (blue) features, then applied linear separation:

https://anonymous.4open.science/r/icml_rebuttal-5064/thresholds.png

For experiments, we determined thresholds $s^{(R)}=0.5$ and $s^{(M, A)}=0.15$ by computing feature activations across texts, building graphs with active features highlighted, and by manually examining those graphs for threshold values that preserved semantic consistency and co-activation of features in graphs. While expert input remains helpful, we believe this process could be automated in future work.

[1] https://arxiv.org/pdf/2411.02193

Thank you for your careful reading and insightful questions.

---

Q1. The purpose of Figure 4 is indeed to show that these groups differ significantly. Below are the raw counts of elements in each group:

See Figure 13 for an evaluation of how frequently groups intersect. Those values are calculated as $|G_1\cap G_2|/|G_1|$, where $G_1$ is a row group and $G_2$ is a column group.

Q2. Figure 5 alone cannot confirm feature emergence. We instead analyze group proportions across layers. Features are classified as emergent if closer to module (MLP/ATT) features than residual stream features (Figures 12b, 18c), placing "From MLP", "From ATT", and "From MLP & ATT" in this category. Translated features are those closer to the previous residual stream ("From RES"). See also our answer to Q6.

Q3. Comparing FineWeb and Python Code in the later layers reveals a higher presence of “From nowhere” features than “From RES,” distinguishing this dataset from others. The same trend appears for Llama (see Figure 18b) and Pythia-70M (mentioned in our response to reviewer jyW6).

Q4. We may have misparsed your comment—did you mean "I did notice"? Model differences exist but stem from varying architectures, parameter counts, and SAE training procedures. A full explanation requires deeper study (e.g., identically architected SAEs on shared data). We will provide additional discussion and experiments on Pythia and GPT-2 in the revised manuscript to address this further.

Q5. We believe there are two primary causes for features labeled “From nowhere.” First, there may be a matching error—artifacts from SAE training or a situation where the true predecessor is not the top-1 but perhaps the top-2 or top-3 similar feature. Second, some features could be combinations of two or more features, making them harder to detect via a simple top-1 match; they could also transform (e.g., rotate) as they pass through the layer, as reviewer jyW6 pointed out. Using top-5 cosine similarity, as in our deactivation experiment, reduces the “From nowhere” group substantially, but it is unclear if it can ever be made negligible. Expanding the SAE dictionary might also risk increasing false positives under the top-1 strategy, so this trade-off requires more investigation. We will extend our discussion of the “From nowhere” group to clarify these points in the revised version.

Q6. We currently think of two general approaches for validating these shorthands.

The first involves analyzing their semantics and activation patterns. For instance, if a target feature activates on tokens like “the, a”, while its predecessor in the residual stream fires on “the, The” and the corresponding MLP feature activates on “a, A, an”, this suggests the target feature emerges from their interaction—we categorize it as “processed.” Features with nearly identical descriptions and activation patterns across layers likely belong to the “translated” group, indicated by high $s^{(R)}$ values. Conversely, features without semantic ties to the residual stream but aligned with MLP or attention counterparts may be “newborn”—new linear directions introduced by those modules. Expanding this analysis beyond top-1 matches (e.g., to top-5) could further enhance its reliability.

Figures 3 and 18b illustrate clusters in the top-left corner, where features lack $s^{(R)}$-similar predecessors in the residual stream. This could imply either new feature introduction by the MLP (e.g., stored knowledge) or transformation of an initially dissimilar residual feature via MLP processing. In contrast, Figure 5 shows a clear predominance of the ”From RES*”* group, suggesting most features propagate across layers with minimal modification.

While this analysis could be automated, it hinges on either reliable feature explanations or raw activation data.

The second approach tests feature roles through steering experiments. For example:

1. Steering only the target feature,
2. Steering its residual predecessor, or
3. Steering the residual predecessor while deactivating active MLP/attention predecessors.

Comparing these setups could reveal mechanistic differences, though we have not yet implemented this systematically. Preliminary versions of (a) and (b) appear in Section 5.3, though (b) in our current work involves steering the entire flow graph, not just the residual stream.

Thank you for your valuable and extensive review. We aim to address your concerns below.

Claims And Evidence

Our work examines SAEs at various points in the model and linking their features allows us explore its computational structure—such as the technique described in Appendix F, which mimics a transcoder approach. Thus, we can shed more light on the model’s behavior than just geometry. Permutations [2], top-1 correlation [3], and top-1 cosine similarity (our work) are all special cases of this feature mapping approach.

We respectfully disagree with labeling our approach as non-"data-free", since we use this term following the prior work [2]. While SAEs could be trained on the LLM’s data, we solely analyze model weights, not external data, justifying the "data-free" description.

Methods And Evaluation Criteria

We appreciate your suggestion for a data-driven baseline. While useful, such methods struggle with sparse SAEs, demonstrating the advantage of our data-free approach where adjusting k in top-k matching can better address these limitations.

We tested Pearson correlations on 100K non-special tokens from the FineWeb “default” subset for each feature in Gemma Scope’s even layers and all layers of Pythia-70M-Deduped and GPT-2. Using 500 samples (instead of 250) as described in Appendix A.1, we identified feature groups.

https://anonymous.4open.science/r/icml_rebuttal-5064/gemma_corr_cos.png

https://anonymous.4open.science/r/icml_rebuttal-5064/pythia_corr_cos.png

https://anonymous.4open.science/r/icml_rebuttal-5064/gpt_corr_cos.png

Correlation-based matching reduced the "From nowhere" group and better identified attention module predecessors, although the mismatch with Gemma Scope SAEs still lowered quality. For Pythia, results aligned closely with Llama Scope, reflecting clearer attention features.

However, correlation-based matching did not consistently outperform top-1 cosine similarity and performed worse on out-of-distribution Python code (further from FineWeb). Top-1 cosine and top-k correlation predecessors showed strong agreement for Gemma Scope and GPT-2 residual SAEs but weaker alignment for module-based SAEs, consistent with prior feature propagation findings.

These results broadly characterize correlation-based performance. We plan to include these comparisons but welcome requests for specific details to enhance our discussion.

Behavioral and Coherence scores follow the setup from [1], with details and the system prompt described in Appendix B. In brief, we ask a model to assess whether a specific theme is present (Behavioral) and to rate the text’s language quality (Coherence), assigning each an integer score from 0 to 5. These scores are then normalized to the [0, 1] range. We acknowledge that this explanation belongs in Section 4.2 and will revise it accordingly.

As you noted, multiplying the scores ensures that moderate Coherence and high Behavioral (or vice versa) results in a moderate overall score as expected. Both scores are illustrated in Figure 9.

Experimental Designs Or Analyses

We define the activation change as $1-\mathbf{z}^{new}/\mathbf{z}^{old}$, where $\mathbf{z}$ is a feature activation after applying JumpReLU.

Each feature may have up to three predecessors (residual, MLP, attention). To identify active predecessors, we use four methods (Appendix A.2).

For features with active predecessors in both residual (R) and MLP (M), we label them “From RES & MLP” and run forward passes deactivating R, M, or both (“deactivated one at a time”). In top-k method, up to five features may be deactivated per predecessor (though few are typically active).

Figure 7 subplots group results by predecessor type; bars show deactivation effects (e.g., deactivating “mlp” in “From RES & MLP” yields ~0.25 mean activation change).

Other Strengths And Weaknesses

While we did not explicitly justify cosine similarity as a metric early on, Section 5.2 and Appendix F compare it with a permutation-based method.

By “removal” or “suppression,” we refer to specific linear combinations of features (e.g., “king – man”), similar to combinations that create new semantics (e.g., “woman + power”). We suspect these combinations are common.

Questions For Authors

We use the L2 norm, treating vectors as sliced decoder columns.

While our presentation assumes B < A, this is not strictly necessary.

Thank you again for your valuable feedback. We will reorganize our paper, improve clarity, and add the missing reference. We hope these clarifications sufficiently resolve your concerns and encourage you to reconsider your evaluation. Thank you for your time, and please let us know if you have any further questions or requests.

References:

[1] https://arxiv.org/pdf/2411.02193

[2] https://arxiv.org/pdf/2410.07656

[3] https://arxiv.org/abs/2410.08869