Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning

We thank the reviewer for their feedback and for clearly spending a significant amount of time with our work! We start by responding to the summary:

What I found to be most hand-wavy is the interpretability analysis: the authors seem to cherry-pick [...].

We appreciate the reviewer's critique. While we acknowledge that latent interpretation remains challenging, we'd like to clarify that the practical interpretability examples comprises less than half a page of our main paper. Its purpose is to provide concrete examples showing that crosscoder latents can capture interpretable chat-specific behaviors, not to serve as comprehensive validation. We'd like to clarify our approach and its limitations.

Systematic selection: For the BatchTopK crosscoder, 9 of 15 displayed latents rank within the top 14 by Δ-norm (out of 73,728 total), with none ranking higher than 131 across our metrics. We first examined the top 20 latents by Δ-norm, then systematically reviewed latents meeting below certain chat-specificity thresholds.

Quantitative support: Beyond these examples, Section 3.2 demonstrates that many chat-only latents activate on template tokens, and our autointerp analysis shows higher interpretability scores for lower ν values—providing quantitative evidence for meaningful behavioral differences.

While we agree that more rigorous validation would be valuable (which we'll add to our limitations: "Further, an extensive interpretability analysis of all chat-only latents would be valuable."), we believe this focused analysis serves its intended purpose of illustrating that crosscoders can identify interpretable differences between models. We are happy to reformulate certain claims, if the reviewer has specific proposals.

It seems to me that the practical value of the insights has not really been tested and shown -- when it comes to interpretability -- the authors do not provide really any "objective" measure of interpretability gains

We acknowledge this important distinction between interpretability and practical utility. While our primary contribution is methodological—identifying and fixing artifacts in crosscoder training —we do provide evidence of practical value:

1. Objective interpretability metrics: Using established autointerp methods [1], we quantitatively show that our latent scaling method identifies more interpretable latents in L1 crosscoders, with BatchTopK latents generally showing higher interpretability scores for low ν values.

2. Template token analysis: Our finding that many chat-only latents activate on template tokens provides actionable insights for understanding how chat fine-tuning affects models.

We agree that demonstrating more downstream applications would strengthen our work and view this as an important direction for future research.

why did you select "chat-tuning" as the phenomena of analysis w.r.t. crosscoders? [...] most limiting factor, which reduces the scope of the analysis [...]

We initially chose chat fine-tuning as it represents one of the most prevalent and impactful model adaptations in practice, and aligns with prior work which first introduced the crosscoder (Lindsey et al., 2024). However, we fully agree with the reviewer that the value our paper provides to the community applies more broadly. We have now added two new model comparisons trained on different domains to verify that this is not merely a base vs. chat phenomenon:

• Medical domain finetuning.

• Long RL-finetuning on reasoning data.

We observe two consistent patterns across all models: (i) The Δ metric in L1 crosscoders consistently identifies a large number of latents as finetuning-only that actually display Complete Shrinkage or Latent Decoupling. (ii) When using Latent Scaling to identify finetuning-only latents, BatchTopK crosscoders consistently identify more such latents.

These results demonstrate that our findings reproduce across narrow domain finetuning (medical & reasoning), different architectures (Qwen & Llama), and alternative finetuning algorithms (RL tuning), supporting the generality and robustness of our analysis. We will add these results to the Appendix and make sure to properly refer to them in the main text.

We provide a table with the results from those runs in our rebuttal to Reviewer A8m4 (Reviewer 1) in Section “New generalization results” at the end. We apologize for this, the full details didn’t fit into this answer.

I feel like the motivation for the work and practical impact would be clearer if the experiments involved several model pairs.

We completely agree with this suggestion. We are currently developing a comprehensive evaluation framework for a follow-up study that will systematically compare multiple model diffing techniques (including trivial baselines) on their ability to surface expected behavioral differences across controlled fine-tuning scenarios.

Remaining Weaknesses and Questions

We respond to weaknesses and questions that haven't been addressed above.

Low-level hyperparameter choices and their impact

Choice of L0, Δ-norm thresholds and class boundaries as well as how the choice of L0 affects the observed phenomenon.

Thank you for these detailed questions. We chose L0=100 as a reasonable balance between sparsity and reconstruction quality, consistent with SAE literature practices (e.g. [2, 3]). For the Llama 8B experiments, we used L0=200 and observed the same patterns. We are currently training multiple crosscoders for Gemma-2 2B with different target L0 values as well as different dictionary sizes to verify robustness and will include these results in the camera ready. We expect the artifacts to persist across different dictionary sizes and sparsities since they stem from the fundamental properties of L1 regularization.

The Δ-norm thresholds have intuitive interpretations: Δ=0.5 means equal decoder norms, Δ=1.0 means zero base decoder norm (pure chat latent), and Δ=0.0 means zero chat decoder norm (pure base latent). We chose the 0.9 threshold for "chat-only" to capture latents strongly dominated by the chat model.

We acknowledge that the missing analysis of some factors is a limitation of our work. We added the following sentence to the limitations: "We haven't systematically explored how hyperparameter choices (e.g. layer selection and Δ thresholds) affect our results."

Why exactly 3176 chat-only latents?

3176 is simply the number of latents in the L1 crosscoder that meet our chat-only criterion (Δ>0.9). To ensure a fair comparison in the KL experiments, we selected the same number of top-ranked latents from the BatchTopK crosscoder.

Why layer 13?

We follow the methodology of Lindsey et al. (2024), who trained crosscoders on middle layer activations. Since Gemma-2 2B has 26 layers, layer 13 represents the middle of the network. While analyzing multiple layers would be valuable, computational constraints limited us to a single layer analysis. Middle layers are where we expect to find the richest and most interesting representations [4].

Question 2: Addition Orthogonality Constraint

This is a good input. Orthogonality constraints present an interesting trade-off. While they could reduce redundancy, they might also create undesirable artifacts. For instance, semantically related concepts (like different aspects of refusal behavior that we observed) naturally share representational structure, and forcing them apart could harm interpretability.

Your question connects to well-known phenomena in SAE literature:

• Feature splitting [5]: Where semantically related concepts distribute across multiple latents. Wider SAEs often split concepts into finer-grained representations (e.g., "math" → "algebra", "geometry").
• Feature absorption [6]: Often some latents absorb their child latents, reducing interpretability.

For crosscoders specifically, we believe orthogonality constraints would be problematic, as illustrated by our example in Appendix D. Consider a concept C (e.g., "dangerous concept") that appears in some contexts in both models, but where the fine-tuned model has additionally learned to activate it in new contexts where the base model remains inactive. Ideally, we would want two separate latents: one that activates solely on shared contexts (where both models activate equally) and another that activates only where the fine-tuned model fires.

An orthogonality constraint would likely force these into a single latent, creating a clear misrepresentation of the actual behavior. This would either lead to shrinkage of the base decoder (since when only the fine-tuned model should be active, the shared latent becomes unhelpful for base model reconstruction) or cause the latent to miss contexts where only the fine-tuned model should fire—thereby missing the new behavior entirely.

It is also worth mentioning the "Deep Causal Transcoding" blog post on LessWrong by Andrew Mack (2024) that proposes an - in our view super exciting - alternative to finding interesting and causal latent directions in space. They use a constraint similar to what you describe and we are currently looking into how we can leverage their method for model diffing.

[1]: Paulo et al. “Automatically Interpreting Millions of Features in Large Language Models.”

[2]: Rajamanoharan et al. “Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders.”

[3]: Lieberum et al. “Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2.”

[4]: Skeal et al. Layer by Layer: Uncovering Hidden Representations in Language Models

[5]: Bricken et al., "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning”

[6]: Chanin et al., "A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders"

Dear Reviewer wpTV,

Thank you again for your review and let us know if there are any remaining questions or concerns we can address.

Best,

The authors

We thank the reviewer for highlighting the strengths of our paper. We now address the weaknesses individually:

1. Limited Scope

We agree with this concern and would like to emphasize that our claims about BatchTopK and the Complete Shrinkage and Latent Decoupling phenomena represent general issues with any L1 crosscoder used for model diffing. We chose to examine chat vs. base models as a case study following the approach of Lindsey et al. (2024), who first introduced crosscoders. To demonstrate that these phenomena generalize and also extend beyond base vs. chat comparisons, we have conducted additional analyses:

In Appendix I, we already extended our analysis to Llama 3.2 1B and Llama 3.1 8B models, covering different architectures and model sizes. We now added the KL experiment on Llama 3.2 1B which are consistent with our findings on Gemma:

Further, we have now added two new model comparisons trained on different domains to verify that this is not merely a base vs. chat phenomenon:

• Medical domain finetuning: We compare google/gemma-2-2b-it to OpenMeditron/Meditron3-Gemma2-2B (part of the meditron3 suite). We train our crosscoders on 50M tokens from lmsys and 39M tokens of medical data (a mixture of bio-nlp-umass/bioinstruct, FreedomIntelligence/medical-o1-reasoning-SFT, and MedRAG/pubmed).

• Long RL-finetuning on reasoning data: We compare deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B to nvidia/Nemotron-Research-Reasoning-Qwen-1.5B, which applies "extended RL training periods that facilitate deeper exploration of reasoning strategies" and significantly boosts reasoning performance. We train our crosscoders on 50M tokens from lmsys and 50M tokens of reasoning traces from open-r1/OpenR1-Math-220k.

For both comparisons, we trained L1 and BatchTopK crosscoders with comparable L0≈100 on the validation set and measured how many latents are truly specific to the finetuned model as determined by Latent Scaling. The table below shows results across all investigated models, including the number of finetuning-only latents based on the relative norm difference Δ:

We observe two consistent patterns across all models: (i) The Δ metric in L1 crosscoders consistently identifies a large number of latents as finetuning-only that actually display Complete Shrinkage or Latent Decoupling. (ii) When using Latent Scaling to identify finetuning-only latents, BatchTopK crosscoders consistently identify more such latents.

These results demonstrate that our findings reproduce across narrow domain finetuning (medical & reasoning), different architectures (Qwen & Llama), and alternative finetuning algorithms (RL tuning), supporting the generality and robustness of our analysis. We will add these results to the Appendix and make sure to properly refer to them in the main text.

2. Improving Clarity

We take this very seriously and have tried our best to implement this. We are happy to provide more details in the discussion but due to the character limit, we can only provide high level bullet points. To summarize our changes, we have:

• We added a glossary with all the important terms and mathematical notations to the appendix.
• At the beginning of the method Section 2, we added an additional few sentences about crosscoders to build intuition for readers not familiar with the SAE literature: "To build intuition, the crosscoder's goal is to learn a dictionary of interpretable concepts (latents) that can explain the activations of both models. It consists of an encoder and a decoder. The encoder takes the activations of the base and chat models and projects them into a shared high-dimensional sparse space, where each dimension corresponds to a potential concept. We call the encoded activations latent activations. The decoder takes the latent activations and tries to reconstruct the activations of both models. The key insight is that while both models share the same latent activations for a given input, the crosscoder learns separate decoders for each model, allowing concepts to have different importance or manifestation in each model. Sparsity ensures that only a small number of concepts are active at any time, which should promote monosemanticity. Practically, sparsity is measured by the L0 norm of the latent activations - the number of non-zero latent activations. We train the crosscoder to minimize the reconstruction errors of both models."
• More careful introduction of mathematical notation in Section 2, e.g. "Let $D$ be the dictionary size and $\mathcal{J}=\{1, \dots, D\}$ the set of all latents. For each latent $j \in$..." instead of "For each latent $j \in \mathcal{J}=\{1, \dots, D\}$, where $D$ is the dictionary size, the $j^\text{th}$ latent activation function..."
• Added intuitive explanation of the $\Delta_{norm}$ function into the main text: "A value of 1 for latent $j$ means that $\|{\mathbf{d}^{base}_j}\|_2 = 0$, indicating that the latent is not used by the base model. A value of 0 for latent $j$ means that $\|\mathbf{d}^{chat}_j\|_2 = 0$, indicating that the latent is not used by the chat model."
• Additional intuitions about what the Latent Scaling ratios measure and why high values indicate Complete Shrinkage/Latent Decoupling.
• Made sure to accompany pure mathematical notation with their intuitive names, e.g., "We compute $\nu^r_j$" → "We compute the reconstruction ratio $\nu^r_j$".
• Added more intuition about what we observe in Fig. 2.
• Better explained what we mean by "we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents" and why we make this choice: "To compare the absolute number of chat-specific latents in both crosscoders, we choose the same number of top $\Delta_{norm}$ latents from both models and compare for how many of them both ratios $\nu^r_j$ and $\nu^\epsilon_j$ lie below a range of thresholds $\pi$. Specifically, we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents based on $\Delta_{norm}$ values from the BatchTopK crosscoder."
• We moved Figure 11 from the Appendix back into the main paper. It provides a high-level intuitive visualization of the KL experiment, which helps with understanding these experiments.
• Section 3.2: Added more intuition about why we expect a good chat-specificity metric to have lower KL for the lower 50% than higher 50%.
• Better explanation of how to read Figure 3: The main goal is to compare how the metrics organize latents, not to show that the latents of the BatchTopK crosscoder are more causal.
• Added one sentence at the end of 3.1 and 3.2 to summarize takeaways.

If the reviewer has any specific paragraph that they found hard to parse, we’d be happy to share our current version of it. Due to an oversight with the font type which made our text longer, all of these changes fit within 10 pages.

Questions

Line 78: By "model-only latents," do you specifically mean base-only and chat-only latents?

Yes, exactly. While our analysis focuses on chat-only latents, the described phenomena (Complete Shrinkage and Latent Decoupling) are symmetric and apply equally to base-only latents. As demonstrated in our extended experiments above, these issues persist across different types of model comparisons beyond base vs. chat. We will clarify this in the paper by revising the statement to: "Our conclusion is that the crosscoder loss does not actually have an inductive bias that helps to learn better model-specific latents, whether they are chat-only, base-only, or specific to any other type of finetuning."

Line 171: Should the denominator here be $\beta_j^{r,chat}$?

Yes, you are absolutely right. Thanks for pointing this out!

Thank you for the response. I appreciate the authors' additional efforts. While clarity can be addressed in the final version, I still think the scope is somewhat narrow and limited to specific cases. Therefore, I will maintain my original score (still positive) for the paper.

We thank the reviewer for highlighting the strengths of our paper. We now address the weaknesses individually:

1. Improving Clarity

We take this very seriously and have tried our best to implement this. We are happy to provide more details in the discussion but due to the character limit, we can only provide high level bullet points. To summarize our changes, we have:

• We added a glossary with all the important terms and mathematical notations to the appendix.
• At the beginning of the method Section 2, we added an additional few sentences about crosscoders to build intuition for readers not familiar with the SAE literature: "To build intuition, the crosscoder's goal is to learn a dictionary of interpretable concepts (latents) that can explain the activations of both models. It consists of an encoder and a decoder. The encoder takes the activations of the base and chat models and projects them into a shared high-dimensional sparse space, where each dimension corresponds to a potential concept. We call the encoded activations latent activations. The decoder takes the latent activations and tries to reconstruct the activations of both models. The key insight is that while both models share the same latent activations for a given input, the crosscoder learns separate decoders for each model, allowing concepts to have different importance or manifestation in each model. Sparsity ensures that only a small number of concepts are active at any time, which should promote monosemanticity. Practically, sparsity is measured by the L0 norm of the latent activations - the number of non-zero latent activations. We train the crosscoder to minimize the reconstruction errors of both models."
• More careful introduction of mathematical notation in Section 2, e.g. "Let $D$ be the dictionary size and $\mathcal{J}=\{1, \dots, D\}$ the set of all latents. For each latent $j \in$..." instead of "For each latent $j \in \mathcal{J}=\{1, \dots, D\}$, where $D$ is the dictionary size, the $j^\text{th}$ latent activation function..."
• Added intuitive explanation of the $\Delta_{norm}$ function into the main text: "A value of 1 for latent $j$ means that $\|{\mathbf{d}^{base}_j}\|_2 = 0$, indicating that the latent is not used by the base model. A value of 0 for latent $j$ means that $\|\mathbf{d}^{chat}_j\|_2 = 0$, indicating that the latent is not used by the chat model."
• Additional intuitions about what the Latent Scaling ratios measure and why high values indicate Complete Shrinkage/Latent Decoupling.
• Made sure to accompany pure mathematical notation with their intuitive names, e.g., "We compute $\nu^r_j$" → "We compute the reconstruction ratio $\nu^r_j$".
• Added more intuition about what we observe in Fig. 2.
• Better explained what we mean by "we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents" and why we make this choice: "To compare the absolute number of chat-specific latents in both crosscoders, we choose the same number of top $\Delta_{norm}$ latents from both models and compare for how many of them both ratios $\nu^r_j$ and $\nu^\epsilon_j$ lie below a range of thresholds $\pi$. Specifically, we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents based on $\Delta_{norm}$ values from the BatchTopK crosscoder."
• We moved Figure 11 from the Appendix back into the main paper. It provides a high-level intuitive visualization of the KL experiment, which helps with understanding these experiments.
• Section 3.2: Added more intuition about why we expect a good chat-specificity metric to have lower KL for the lower 50% than higher 50%.
• Better explanation of how to read Figure 3: The main goal is to compare how the metrics organize latents, not to show that the latents of the BatchTopK crosscoder are more causal.
• Added one sentence at the end of 3.1 and 3.2 to summarize takeaways.

If the reviewer has any specific paragraph that they found hard to parse, we’d be happy to share our current version of it. Due to an oversight with the font type which made our text longer, all of these changes fit within 10 pages.

2. Generalizability and robustness of claims

We agree with this concern and would like to highlight that in Appendix I, we have already extended the analysis to Llama 3.2 1B and Llama 3.1 8B models, covering different architectures and model sizes. We added the KL experiment on Llama 3.2 1B which are consistent with our findings on Gemma:

We have also added two new model comparisons trained on different domains to verify that this is not merely a base vs. chat phenomenon:

• Medical domain finetuning: We compare google/gemma-2-2b-it to OpenMeditron/Meditron3-Gemma2-2B (part of the meditron3 suite). We train our crosscoders on 50M tokens from lmsys and 39M tokens of medical data (a mixture of bio-nlp-umass/bioinstruct, FreedomIntelligence/medical-o1-reasoning-SFT, and MedRAG/pubmed).
• Long RL-finetuning on reasoning data: We compare deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B to nvidia/Nemotron-Research-Reasoning-Qwen-1.5B, which applies "extended RL training periods that facilitate deeper exploration of reasoning strategies" and significantly boosts reasoning performance. We train our crosscoders on 50M tokens from lmsys and 50M tokens of reasoning traces from open-r1/OpenR1-Math-220k.

For both comparisons, we trained L1 and BatchTopK crosscoders with comparable L0≈100 on the validation set and measured how many latents are truly specific to the finetuned model as determined by Latent Scaling. The table below shows results across all investigated models, including the number of finetuning-only (FT-only) latents based on the relative norm difference Δ:

We observe two consistent patterns across all models: (i) The Δ metric in L1 crosscoders consistently identifies a large number of latents as finetuning-only that actually display Complete Shrinkage or Latent Decoupling. (ii) When using Latent Scaling to identify finetuning-only latents, BatchTopK crosscoders consistently identify more such latents.

These results demonstrate that our findings reproduce across narrow domain finetuning (medical & reasoning), different architectures (Qwen & Llama), and alternative finetuning algorithms (RL tuning), supporting the generality and robustness of our analysis. We will add these results to the Appendix and make sure to properly refer to them in the main text.

Question: Significance of identifying misattributed chat-only latents

Can the authors better clarify the significance of misattributing concepts as unique to the chat model?

Misattributing base model concepts as chat-only has significant consequences for both research and practical applications:

• Misleading interpretability insights: Researchers studying what makes chat models act in specific ways might incorrectly conclude that certain capabilities (e.g., harmlessness, basic reasoning, code understanding) were introduced by chat-tuning, when they actually existed in the base model. This fundamentally misrepresents how alignment training works. More generally, researchers studying what finetuning modifies in a base model, will make false conclusions.
• Inefficient safety interventions: If we believe a harmful behavior is chat-specific when it's actually in the base model, we might waste resources trying to remove it through chat-tuning modifications rather than addressing it at the pretraining level.
• Reduced research efficiency: If 50% of presumed chat-specific latents are actually noisy artifacts, researchers waste significant time analyzing spurious features while potentially discarding genuine but subtle chat-specific latents that appear artifact-like.

By correctly identifying truly chat-specific latents, our method enables more precise model control, better safety interventions, and accurate scientific understanding of how fine-tuning transforms language models.

Formatting concerns

We apologize for this oversight. We have corrected the font to match the official NeurIPS style guide, which has actually saved us half a page. Thank you for pointing this out!

Dear Reviewer EoXg,

Thank you again for your review and let us know if there are any remaining questions or concerns we can address.

Best, The authors

We thank the reviewer for highlighting the strengths of our paper. We now address the weaknesses individually:

1. Improving Clarity

We take this very seriously and have tried our best to implement this. We are happy to provide more details in the discussion but due to the character limit, we can only provide high level bullet points. Summary of our changes:

• We added a glossary with all the important terms and mathematical notations to the appendix.
• At the beginning of the method Section 2, we added an additional few sentences about crosscoders to build intuition for readers not familiar with the SAE literature. We show the first few sentences of this: "To build intuition, the crosscoder's goal is to learn a dictionary of interpretable concepts (latents) that can explain the activations of both models. It consists of an encoder and a decoder. The encoder takes the activations of the base and chat models and projects them into a shared high-dimensional sparse space, where each dimension corresponds to a potential concept. "
• More careful introduction of mathematical notation in Section 2, e.g. "Let $D$ be the dictionary size and $\mathcal{J}=\{1, \dots, D\}$ the set of all latents. For each latent $j \in$..." instead of "For each latent $j \in \mathcal{J}=\{1, \dots, D\}$, where $D$ is the dictionary size, the $j$th latent activation function..."
• Added intuitive explanation of the $\Delta_{norm}$ function into the main text.
• Additional intuitions about what the Latent Scaling ratios measure and why high values indicate Complete Shrinkage/Latent Decoupling.
• In response to “There ended up being a lot of notation that would benefit from replacing with english (for example, $\nu^\epsilon$)” we made sure to accompany pure mathematical notation with their intuitive names, e.g., "We compute $\nu^r_j$" → "We compute the reconstruction ratio $\nu^r_j$".
• Added more intuition about what we observe in Fig. 2.
• Better explained what we mean by "we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents" and why we make this choice: "To compare the absolute number of chat-specific latents in both crosscoders, we choose the same number of top $\Delta_{norm}$ latents from both models and compare for how many of them both ratios $\nu^r_j$ and $\nu^\epsilon_j$ lie below a range of thresholds $\pi$. Specifically, we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents based on $\Delta_{norm}$ values from the BatchTopK crosscoder."
• We moved Figure 11 from the Appendix back into the main paper. It provides a high-level intuitive visualization of the KL experiment, which helps with understanding these experiments.
• Section 3.2: Added more intuition about why we expect a good chat-specificity metric to have lower KL for the lower 50% than higher 50%.
• Better explanation of how to read Figure 3: The main goal is to compare how the metrics organize latents, not to show that the latents of the BatchTopK crosscoder are more causal.
• Added one sentence at the end of 3.1 and 3.2 to summarize takeaways.

If the reviewer has any specific paragraph that they found hard to parse, we’d be happy to share our current version of it. Due to an oversight with the font type which made our text longer, all of these changes fit within 10 pages.

2. Interpretation of Figure 3

Even though the numbers directionally confirm multiple hypotheses in Figure 3, the absolute improvement introduced by BatchTopK seems relatively small [...]. Is this a correct comparison?

We'd like to clarify the primary purpose of Figure 3, as there appears to be a misunderstanding about our claims. The key insight from this figure is not that BatchTopK provides more causal latents (which, as you correctly note, it does not significantly). Rather, we aim to demonstrate that norm-based analysis (Δ) is insufficient for identifying causally influential latents in the L1 crosscoder, while it works well for BatchTopK. Specifically, this effect is most prominent on the first 9 tokens:

• In Figure 3b): In the L1 crosscoder the lowest 50% of latents identified by Δ as "more chat-specific" actually show lower causal effect than the highest 50%. This counterintuitive result highlights the limitation of the L1 crosscoder when using Δ alone, since the loss term interacts with the decoder norms. In contrast, for BatchTopK: The Δ metric successfully identifies causally relevant chat-only latents because, as we note in the text, the Δ and Latent Scaling metrics are highly correlated for this crosscoder.

• This comparison provides evidence that latent scaling metrics are necessary (and effective) to identify truly causal latents when using L1 crosscoders, while BatchTopK's training objective makes simple norm-based analysis sufficient.

We appreciate you pointing out this potential confusion and will make this distinction clearer in our revision.

3. Small contribution

We appreciate the reviewer's perspective and understand why our modifications might appear incremental at first glance. However, we respectfully argue that our contributions address fundamental issues in the field:

• We show that the approach by Lindsey et al. (2024) produces critical and significant artifacts due to the L1 loss function. This finding has immediate practical implications for researchers using crosscoders for model diffing.
• We propose the Latent Scaling to provide a principled way to verify whether identified latents genuinely represent model-specific features. This technique extends beyond crosscoders—we've successfully applied it to standard SAEs (details forthcoming in future work).
• To show that our analysis holds more generally than just for base and chat models, we added additional analysis, where we show that the results hold across models architectures, domains and training algorithms:
  • Different model scales (Llama 3.2 1B and 3.1 8B in Appendix I)
  • New domains: medical fine-tuning and long RL-tuning on reasoning data
  • Consistent findings across all scenarios, demonstrating robustness

We provide detailed results for the new domain analyses at the end of this response.

4. More principled approach to model diffing

We strongly agree that the field would benefit from a more principled framework. While BatchTopk is effective, it addresses specific observed issues rather than starting from first principles. We're actively working on this challenge. For our follow-up study, we're developing a comprehensive evaluation framework that will:

• Systematically compare multiple model diffing techniques
• Include appropriate baselines
• Test methods on controlled fine-tuning scenarios with known behavioral changes

We'd be happy to discuss this further and would welcome any specific properties you think such methods should satisfy.

Formatting concerns

We apologize for this oversight. We have corrected the font to match the official NeurIPS style guide, which has actually saved us half a page. Thank you for pointing this out!

New generalization results

We have added two new model comparisons trained on different domains to verify that this is not merely a base vs. chat phenomenon:

• Medical domain finetuning: We compare google/gemma-2-2b-it to OpenMeditron/Meditron3-Gemma2-2B (part of the meditron3 suite). We train our crosscoders on 50M tokens from lmsys and 39M tokens of medical data (a mixture of bio-nlp-umass/bioinstruct, FreedomIntelligence/medical-o1-reasoning-SFT, and MedRAG/pubmed).

• Long RL-finetuning on reasoning data: We compare deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B to nvidia/Nemotron-Research-Reasoning-Qwen-1.5B, which applies "extended RL training periods that facilitate deeper exploration of reasoning strategies" and significantly boosts reasoning performance. We train our crosscoders on 50M tokens from lmsys and 50M tokens of reasoning traces from open-r1/OpenR1-Math-220k.

For both comparisons, we trained L1 and BatchTopK crosscoders with comparable L0≈100 on the validation set and measured how many latents are truly specific to the finetuned model as determined by Latent Scaling. The table below shows results across all investigated models, including the number of finetuned-only (FT-only) latents based on the relative norm difference Δ:

We observe two consistent patterns across all models: (i) The Δ metric in L1 crosscoders consistently identifies a large number of latents as finetuning-only that actually display Complete Shrinkage or Latent Decoupling. (ii) When using Latent Scaling to identify finetuning-only latents, BatchTopK crosscoders consistently identify more such latents.

These results demonstrate that our findings reproduce across narrow domain finetuning (medical & reasoning), different architectures (Qwen & Llama), and alternative finetuning algorithms (RL tuning), supporting the generality and robustness of our analysis. We will add these results to the Appendix and make sure to properly refer to them in the main text.