Redefining Experts: Interpretable Decomposition of Language Models for Toxicity Mitigation

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we address the key concerns and suggestions raised.

Addressing the Missing Citation of Prior Work

We appreciate the pointer to the work by Balestriero et al. (ICML 2024):

Balestriero, R., Cosentino, R. and Shekkizhar, S., 2023. Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generation. arXiv preprint arXiv:2312.01648.

We acknowledge the significance of the referenced work and appreciate the reviewer for highlighting it. We will incorporate a proper citation and provide a detailed discussion in the Related Work section of the revised manuscript, clearly outlining both the similarities and key differences between their approach and ours.

Scalability to Larger Language Models

Thank you for the valuable suggestion regarding evaluation on larger models. We fully agree with the importance of demonstrating scalability and real-world applicability. In response, we have extended our evaluation to include several larger LLMs. The results below highlight the effectiveness of our method across scales:

These results confirm that our method maintains strong performance even when scaled to more powerful and practically relevant LLMs.

We are deeply committed to building safe and interpretable AI systems. EigenShift is a meaningful step in that direction: lightweight, transparent, and generalizable. We appreciate the reviewer’s thoughtful questions and hope these clarifications demonstrate the rigor, novelty, and practicality of our approach.

We respectfully request reconsideration of our score in light of these improvements and experiments.

Sincerely,

Authors

We sincerely thank the reviewer for their positive assessment of our work's core contributions, including:

• The novel distinction between "detection experts" and "generation experts"
• Our principled eigen-decomposition approach
• The systematic cross-lingual evaluation

We appreciate the thoughtful concerns raised and address them below.

W1: The major concern that I have is how such decomposition methods would influence the performance of large language models. Without extensive evaluation on this, it's hard to conclude how those methods would be useful for the community.

[W1] Response:

We believe our paper provides an extensive and direct evaluation of the performance impact:

• Direct Performance Measurement:
  We use perplexity, the standard metric for language model fluency, to measure performance impact. As stated in our Evaluation Setup, perplexity is computed on a fixed snapshot of the English Wikipedia corpus, a widely used benchmark for this purpose.

• Comprehensive Experimental Results:
  Table 2 presents a detailed, side-by-side comparison of performance trade-offs between EigenShift and three other baselines across five different LLMs. This clearly illustrates changes in perplexity versus toxicity reduction the most direct evaluation method for our goal.

• Negligible Impact of SVD Itself:
  To isolate the impact of the decomposition from the intervention, we analyzed the reconstruction error from SVD. As shown in Appendix C.4 and Table 5, the Frobenius reconstruction loss is negligible (e.g., 8.00×10⁻⁵ for LLaMA-7B), confirming that decomposition does not measurably affect perplexity.

• Qualitative Performance:
  In addition to quantitative results, Table 4 provides qualitative examples showing that while baseline methods lead to "incoherent generation" or "catastrophic forgetting", our approach preserves fluency and semantic intent, replacing toxic terms with neutral alternatives.

We believe these four points, perplexity benchmarking, cross-model comparisons, reconstruction analysis, and qualitative insights, together demonstrate a thorough evaluation of performance impact and the practical utility of EigenShift.

W2: While the SVD decomposition is mathematically sound, the paper lacks rigorous theoretical justification for why eigen-directions should correspond to semantic concepts like toxicity.

[W2] Response:

We agree that a formal theoretical connection between eigenvectors and semantic concepts is a challenging open research problem. Our contribution lies in proposing a novel and interpretable hypothesis, supported by robust empirical evidence across multiple models and languages.

• A Principled Hypothesis:
  Our hypothesis is grounded in model architecture: the final linear layer (lm_head) serves as a semantic decision bottleneck. The eigenvectors of this transformation are hypothesized to correspond to principal axes of semantic choice during generation.

• Validation Through Rigorous Experimentation:
  The core of our paper is an empirical validation of this hypothesis. We demonstrate that:

  1. Identifying eigen-directions via activation differences isolates toxicity signals (see Equation 1), and
  2. Selective damping of these directions reduces toxic generation while preserving fluency (Table 2).

This empirical validation is a key contribution, and we hope it lays the groundwork for future theoretical investigations.

• Interpretability Justification:
  LLMs are widely regarded as black-box systems. Our work takes a significant step toward interpreting their internal decision mechanisms, bridging the gap between structure and semantics.

If these clarifications address the concerns and misalignments in understanding, we kindly ask you to consider revisiting your score.

We remain deeply committed to building safe and interpretable AI systems. EigenShift represents a meaningful step in that direction it is lightweight, transparent, and broadly generalizable. We sincerely appreciate the reviewer’s thoughtful questions and engagement with our work. We hope that the clarifications and additional experiments presented here underscore the rigor, novelty, and practicality of our approach.

We respectfully request a reconsideration of our score in light of these responses.

Sincerely,

The Authors

We sincerely thank the reviewer for their thoughtful review and constructive feedback. We appreciate the time you have taken to analyze our work and are glad you found our contributions valuable. We address your concerns and questions below.

Rebuttal to Weaknesses

W1: A 58% increase in perplexity is substantial.

We thank the reviewer for highlighting this important point. We agree that a 58% increase in perplexity (from 6.23 to 9.84 for LLaMA-7B) is significant and merits close scrutiny.

However, it is crucial to view this increase in context. Our method, EigenShift, prioritizes a careful balance between toxicity reduction and linguistic fluency. Unlike other zero-training intervention approaches that severely degrade language quality, our method achieves this trade-off with minimal sacrifice to coherence. As shown in Table 2:

• Det-0 increases perplexity from 6.23 to 43,517.97 — a staggering 700,000% increase.
• Damp increases it to 741.65 — an 11,724% rise.
• Aura, the best-performing baseline prior to our work, raises perplexity to 19.30, a 210% increase.

In contrast, EigenShift yields a final perplexity of 9.84, which, although increased by 58%, represents a massive preservation of fluency relative to existing methods. To quantify this trade-off, we introduced the Toxicity-Perplexity Harmonic (TPH) Score, where EigenShift attains a leading value of 60.37%, demonstrating its superior balance of toxicity mitigation and coherence.

To further address concerns regarding potential degradation of the model’s core abilities, we evaluate our intervened models on widely accepted downstream benchmarks using the MMLU (Massive Multitask Language Understanding) suite across diverse reasoning domains. Results below indicate that model capabilities remain largely intact:

MMLU - Algebra (Mathematical Reasoning)

MMLU - Logical Fallacies (Logical Reasoning)

MMLU - U.S. Foreign Policy (Factual Knowledge)

These results confirm that EigenShift’s intervention maintains the reasoning, factual knowledge, and problem-solving skills of the base models. Therefore, while there is a moderate perplexity increase, it is a carefully engineered and justifiable trade-off that avoids the catastrophic degradation seen in prior work.

W2: The AUROC for the layer-wise method...is still ~60%...not a very high accuracy for a classifier.

We appreciate the reviewer's observation. We wish to clarify that the goal of our layer-wise AUROC analysis was not to build a state-of-the-art classifier, but to address RQ1 and RQ2 by proving that layer-wise representations are a more stable source of signal for toxicity than individual neurons. Our results in Table 1 confirm this, showing a +15.84 percentage point improvement in AUROC on the Jigsaw dataset over neuron-based methods.

The main contribution, EigenShift, uses this signal for a targeted intervention, not classification. Its success is therefore measured by the final generation quality and toxicity reduction, not the intermediate AUROC score. The table below, extracted from our LLaMA-2 results, highlights the final performance where EigenShift excels:

This demonstrates that even a moderate signal, when used in a principled way, can lead to state-of-the-art intervention outcomes.

We thank the reviewer for their detailed feedback and insightful questions. We are encouraged that the reviewer found our work to be "technically solid" with good quality, significance, and originality. We address each feedback and question below.

[W1/Q1] We thank the reviewer for this practical concern. EigenShift has two hyperparameters and, unlike baselines that apply a uniform damping factor, our method first identifies and ranks novel 'generation experts' based on their directional influence (Δi​). There is no existing metric to pre-determine the optimal number of these experts to target. Therefore, the Top-k parameter is a necessary and natural mechanism to select the most influential directions for intervention. The damping factor, α, then controls the intensity of this intervention.

Our ablation study in Appendix C.5 (Table 6, Figure 4) was included to provide clear guidance and avoid the need for a complex search. Our findings show that setting the damping factor α=0.9 is a relatively stable choice that consistently preserves model fluency (perplexity) across various Top-k values, as shown in the Table below. This simplifies tuning by effectively reducing the search to a single dimension. With α fixed, Top-k becomes a direct and interpretable control knob for the user to manage the toxicity-fluency trade-off.

The table below is a snapshot from Table 6 in the paper (c.f. Appendix C.5 for the complete table).

[W2] We ensured a fair comparison by re-running the baselines from their original source. We used established configurations and standard (consistent) practices for these baselines. The hyperparameter selection for all methods, including our own, was guided by optimizing the trade-off between toxicity and perplexity, which we unified with our proposed TPH score (which is also consistent throughout the work). The use of automatic metrics is addressed in W3.

[W3 / Q7] We utilized widely accepted and rigorous automated benchmarks from the previous literature [1, 2, 3]:

RealToxicityPrompts for toxicity and the Wikipedia corpus for perplexity. To bridge the gap with human perception, we included a qualitative case study in Table 4 (c.f. Page 8 ).

This analysis demonstrates a real-world example where EigenShift successfully steers a toxic generation (e.g. "...who allegedly rapd...") to a neutral alternative ("...involved in the assault..."*) while preserving the original intent and coherence, which other methods fail to achieve. This provides qualitative evidence that our method's improvements are meaningful.

The RoBERTa-based classification model (s-nlp/roberta_toxicity_classifier) employed in our work is a standard toxicity classifier, widely used in prior studies [1], and has achieved an inter-annotator agreement (IAA) score of $0.66$ with toxicity 41.2 on the Jigsaw dataset. The following table provides a deeper insight into how human perspectives were integrated into the modeling process and highlights the rationale behind selecting s-nlp/roberta_toxicity_classifier as our primary toxicity classification model.

[W4/Q5] We acknowledge the concern regarding the potential for catastrophic forgetting due to the suppression of generation experts, and the subsequent impact this might have on the model’s performance on downstream tasks. However, since our intervention selectively dampens only the generation experts, our findings show that the core semantic and reasoning capabilities of the model remain largely unaffected.

To validate this, based on your suggestion, we conducted further experiments to assess the impact of our intervention on general-purpose capabilities. Specifically, we evaluated the intervened models on the widely-used MMLU (Massive Multitask Language Understanding) benchmark, which spans a diverse set of reasoning domains, and report the accuracy metric below. The results, presented below, demonstrate that the model retains its core abilities with minimal degradation.

MMLU - Algebra (Mathematical Reasoning)

MMLU - Logical Fallacies (Logical Reasoning)

MMLU - U.S. Foreign Policy (Factual Knowledge)

These results confirm that EigenShift’s intervention maintains the reasoning, factual knowledge, and problem-solving skills of the base models. Therefore, while there is a moderate perplexity increase, it is a carefully engineered and justifiable trade-off that avoids the catastrophic degradation seen in prior work.

[W5] We appreciate the reviewer’s suggestion. While we do state in the abstract that “our method requires no additional training or fine-tuning, incurs minimal computational cost, and is grounded in rigorous theoretical analysis,” we agree that it would be beneficial to reiterate this more clearly in the main text. We will ensure that this point is explicitly addressed in the camera-ready version.

[W6 / Q3 / Q4] The primary objective of this work is to interpret deep models, which are often considered black box, through an intuitive framework. To the best of our knowledge, this is the first approach that leverages such simplicity for interpretability while reducing toxicity without requiring any additional training. Our method incurs minimal computational overhead, which is comparable to a single forward inference pass, and mitigates toxicity with negligible catastrophic forgetting. This distinguishes our approach in terms of both computational efficiency and interpretability.

[Q2] We introduced the Toxicity-Perplexity Harmonic (TPH) Score precisely to serve as this selection criterion. The hyperparameters for EigenShift reported in Table 2 were selected by identifying the configuration that maximized the TPH score, ensuring a balanced optimization of both toxicity reduction and fluency preservation. For the baselines, we followed the code in their repos and configurations recommended in their respective original publications to ensure a fair comparison.

[Q4] This is a key strength of EigenShift. Our method is training-free and operates directly on the model's pre-existing weights. The data used is solely for identifying the toxic eigen-directions, not for fine-tuning the model's parameters. This process requires a relatively small set of example generations. Therefore, EigenShift is inherently well-suited to low-resource settings where the large, high-quality datasets required for fine-tuning are unavailable. Its advantage over training-based methods would indeed be more pronounced in such scenarios.

[Q6] This is a relevant suggestion. Figure 4 in the appendix already provides a visualization of this trade-off surface by plotting the TPH score against Top-k for different α values. However, we agree that a direct 2D plot of perplexity versus toxicity reduction would make the trade-off frontier even more explicit. We will incorporate such a visualization into the final version of the paper (as we are not allowed to post an image/URL here).

[Q8] Our work is premised on the hypothesis that the final layer is a critical semantic choice as it is the decision layer of an LLM. However, instead of intervening on unstable, individual neurons within that layer, we propose a more robust and interpretable approach. We also acknowledge in our future work that exploring how individual layers decompose to provide insights is an important research direction. Currently, this is computationally expensive, as LLMs often have many decoder layers (e.g., average 7B models have 32 layers), making it relatively complex to analyze in depth. We consider this a valuable area for future research.

[Q9] The neuron-based baseline in Table 1 corresponds to the expert identification method proposed in prior work, which we evaluate to answer RQ1 and RQ2. We compare this fine-grained neuron-level classification with our proposed layer-wise approach, and show that the latter provides a more stable and generalizable signal across examples. We emphasize that this neuron-based analysis is distinct from the intervention baselines (e.g., DetZero, Damp, and Aura) evaluated in Table 2, which focus on modifying model behavior rather than interpretability. For all baselines, we reproduced results using the official implementations or publicly released code from prior literature. Where applicable, we adhered to the default hyperparameter settings used in those works to ensure a fair and consistent comparison.

[Q10] Yes. We state this explicitly as a key advantage and future direction in our Limitations and Future Work section, as well as our discussion of RQ3. The EigenShift framework is designed to be concept-agnostic. By providing examples of any desired semantic concept (e.g., formality, sentiment, specific writing styles), one can identify the corresponding eigen-directions and steer the model's generation accordingly. This transforms the "black-box" LLM into a set of interpretable semantic axes, opening promising avenues for multi-objective control, which we see as an exciting area for future research.

We are deeply committed to building safe and interpretable AI systems. EigenShift is a meaningful step in that direction: lightweight, transparent, and generalizable. We appreciate the reviewer’s thoughtful questions and hope these answers clarify your points.

References:

[1] Suau, X., Delobelle, P., Metcalf, K., Joulin, A., Apostoloff, N., Zappella, L. and Rodríguez, P., 2024. Whispering experts: Neural interventions for toxicity mitigation in language models. arXiv preprint arXiv:2407.12824.

[2] Geva, M., Caciularu, A., Wang, K.R. and Goldberg, Y., 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680.

[3] Wang, Y. and Demberg, V., 2024. RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework. arXiv preprint arXiv:2410.19109.

Thank you for the detailed response from the authors. Most of my questions were answered through the author's response. I do not have any further questions, and I'm still in favor of accepting this paper. I will also check all the other reviews and responses and decide whether to change the ratings or not.