Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons

Thank you for your positive feedback and valuable suggestions. Please see our responses below.

W1 The supplementary material is submitted, the source code is empty

We sincerely apologize for the oversight. Upon reviewing the supplementary material, we find it is indeed empty. We are unsure at which stage this error occurred, but the outermost folder of our codebase is present, and the structure is indeed consistent with our codebase. Rest assured, we will make the entire codebase publicly available in the future.

W2 Lack of sufficient discussion of related work

Here is our discussion of related work. The discussion will be added to the paper if more space is permitted.

Existing interpretability research on LLM safety can be broadly categorized into two perspectives: Representation Engineering (RepE) and Mechanistic Interpretability (MI). We acknowledge the importance of RepE-focused studies, as they often demonstrate strong practical effectiveness in steering model behavior. For instance, [1-2] are firmly grounded in the RepE perspective, and [3][7] also incorporate some perspectives of this approach.

In contrast, our work adopts the MI perspective, which seeks a bottom-up understanding of models’ inner workings. This perspective emphasizes the importance of localizing model functionality to the most fundamental operational units—a core principle of mechanistic interpretability. In the case of transformers, MLP neurons constitute approximately two-thirds of the model's parameters and serve as the foundational units for functionality. Therefore, we focus our study on neurons as the target of analysis to uncover safety mechanisms.

For articles categorized under the MI perspective, [4] has been discussed in our paper, where we point out that toxicity is an incomplete part of model safety concerned in our work, a view also acknowledged in recent work [7]. [5] adopts a different definition of “neuron”, which describes individual parameters rather than complete functional units in this paper. Since features in transformers are usually represented as vectors, it is difficult to interpret how different parameters in a single vector play different mechanistic roles. [6] identified attention heads critical for LLMs’ safety. We believe that the functionalities of neurons and attention heads are not in conflict; instead, complex functions like safety are more likely to result from their collaboration. We plan to further explore their relationship in future work. [7] adopts a safety layer perspective, which we consider too coarse-grained compared to neurons and attention heads for providing a mechanistic understanding. [8] finds safety neurons based on their contribution to the residual stream of safety-related data. This ablation-based method may be influenced by the existence of the "Hydra effect", which suggests that ablating certain components in LLMs may trigger compensatory behaviors in other components.

[1] Zou, Andy, et al. "Representation engineering: A top-down approach to AI transparency." arXiv 2023.

[2] Arditi, Andy, et al. "Refusal in language models is mediated by a single direction." NeurIPS 2024.

[3] Lee, Andrew, et al. "A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity." ICML 2024.

[4] Yang, Yushi, et al. "Ablation is Not Enough to Emulate DPO: How Neuron Dynamics Drive Toxicity Reduction." MINT: Foundation Model Interventions. 2024.

[5] Wei, Boyi, et al. "Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications." ICML 2024.

[6] Zhou, Zhenhong, et al. "On the Role of Attention Heads in Large Language Model Safety." ICLR 2025.

[7] Li, Shen, et al. "Safety Layers in Aligned Large Language Models: The Key to LLM Security." ICLR 2025.

[8] Zhao, Yiran, et al. "Understanding and enhancing safety mechanisms of LLMs via safety-specific neuron." ICLR 2025.

Q1 Could you tell me the differences from the following existing work on task vector? What is the merit of focusing on neurons?

Neurons are the fundamental computational units in neural networks, and they inherently possess interpretability attributes. For example, a neuron's "key" (a row in the MLP up-projection matrix) determines the type of input that will activate it, while its "value" (a row in the MLP down-projection matrix) defines its influence on downstream components, such as other neurons or attention heads. Through the residual stream, neurons can directly impact the output. This allows for a more granular explanation of the model’s internal mechanisms.

In contrast, the task vector is in the entire model weight space. Although the task vector can steer the model’s behavior, it is a global feature of the model involving all the model parameters; understanding how it operates requires examining all the model activations, while the safety neurons identified in this paper are local.

Thank you for your questions and suggestions. Please find our responses below.

W1 Some technical content is difficult to understand.

In the process of generating each token, we use activation patching to modify the model’s forward pass, which results in responses that differ from those of the original model. We then assess the safety gap between the responses generated by the patched model (base/SFT) and the model used for patching (DPO). The smaller this gap, the more precisely the safety neurons capture the safety mechanisms. Thank you for your suggestion; we will include a more detailed explanation of the underlying intuition in the revised version.

W2 The Mechanistic Explanation in this paper seems insufficient

We have attempted to interpret the activations of these neurons by checking the semantic meaning of activating tokens (Table 2), like many previous works, but there is no clear pattern observed. This indicates that more in-depth interpretability techniques are required for interpreting safety mechanisms, which is a major challenge, and we leave it to future work. The contribution of this work lies in developing techniques for localizing safety mechanisms, which is the first step for interpreting them.

W3 The proposed method seems analogous to traditional model compression techniques

Indeed, our approach is similar to model compression, but there are two key differences:

1. Unit of Focus: Model compression typically targets individual parameters, which are not necessarily interpretable. In contrast, we focus on neurons, which have been shown to have specific, interpretable functions in LLMs.
2. Intervention and Causal Effect: Our method allows for direct intervention on identified neurons, demonstrating their active role in model inference. While neuron-level model compression exists, our comparisons (Lines 177-185) suggest that applying model compression for interpretability may not fully capture the causal effects.

Additionally, beyond the technical differences, localizing neurons for interpretability is a well-established research direction, with distinct goals from model compression [1-3]. The research community generally recognizes these as separate lines of study.

[1] Dai, Damai, et al. "Knowledge Neurons in Pretrained Transformers." ACL 2022.

[2] Wang, Xiaozhi, et al. "Finding Skill Neurons in Pre-trained Transformer-based Language Models." EMNLP 2022.

[3] Gurnee, Wes, et al. "Finding Neurons in a Haystack: Case Studies with Sparse Probing." TMLR.

Thank you for your questions. We are happy to discuss these questions further, and below are our responses to them.

W1 Scalability to larger frontier LLMs

Due to time constraints, we are unable to conduct a full set of experiments during the rebuttal period. Below, we provide partial results for Llama2-13B (5% safety neuron activation patching improves the safety of Base and SFT models, similar to the 7B model). Experiments with Qwen2.5-14B and Qwen2.5-32B are currently in progress, and we will include those results in a future revision.

W2 & Q1 Incremental novelty and contribution

The novelty of the method builds on activation patching

Activation patching is a technique used to examine the causal effects of specific components within a model. Previous works that employ activation patching typically identify components with causal effects through exhaustive enumeration. However, enumeration at the neuron level is infeasible due to the vast number of neurons. To address this gap, we propose inference-time activation contrasting, which first filters candidate neurons and then applies activation patching to identify safety neurons.

About the prior neuron-steering work

The primary goal of our work is not to provide a practical model steering method, but rather to serve as a first step toward understanding the safety mechanisms and safety alignment in large language models (LLMs). Therefore, we focus on decoding safety alignment into the most fundamental computational units—neurons. Representation Engineering [1] adopts a top-down approach, starting from high-level representations. While it is currently more practical, its contribution to interpretability is relatively limited. Circuit Breakers [2], as a follow-up work, shares a similar philosophy. Sparse Autoencoder (SAE) is a promising tool for interpretability, but whether SAE features are faithful to the original models remains an open question [3]. We also plan to investigate safety neurons from the perspective of SAEs in future work.

[1] Zou, Andy, et al. "Representation engineering: A top-down approach to AI transparency." arXiv 2023.

[2] Zou, Andy, et al. "Improving alignment and robustness with circuit breakers." NeurIPS 2024.

[3] Leask, Patrick, et al. "Sparse autoencoders do not find canonical units of analysis." ICLR 2025

Q2 Sparsity Stability

From our current results, we observe that the sparsity of safety neurons is more closely related to the extent of model training rather than model size alone. Models trained during the same period—such as Llama-7B/14B, Mistral-7B, and Gemma-7B—exhibit similar levels of sparsity. In contrast, Qwen2.5-3B shows higher sparsity compared to these models (it has a similar number of neurons as Llama2-7B despite its smaller model size). This observation is also reflected in related work [4], which suggests that as model capabilities increase, the model's abilities become increasingly concentrated in a smaller, sparser subset of neurons. We think this is an interesting phenomenon for future research rather than a weakness of our work.

[4] Cao, Yixin, et al. "Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric." arXiv 2025.

Q3 Robustness to Jailbreaks

To verify whether safety neurons faithfully capture the safety capabilities of aligned models, we conducted evaluations on diverse red-teaming datasets. These datasets include carefully crafted prompts, some of which are capable of successfully jailbreaking ChatGPT.

Q4 Activation Timing

We discuss this point in Appendix D.2, where we compute neuron importance using activations from different token positions: all tokens in the prompt, only the last token of the prompt, and all tokens in the generation. Our results show that using activations from the generation tokens yields the best performance. This suggests that safety neurons may play a more important role in influencing model behavior during output generation, while identifying harmful prompts may be more aligned with the function of attention heads[5].

[5] Zhou, Zhenhong, et al. "On the Role of Attention Heads in Large Language Model Safety." ICLR 2025.

Q5 Safeguard Rationale

1. We have not yet established whether the firing of safety neurons directly causes the model to refuse harmful queries; the relationship between the two may be more complex. However, our results indicate that the safeguard mechanism does indeed provide further improvements for DPO-aligned models.
2. In addition to enhancing safety, the safeguard mechanism can reject certain outputs before the model actually generates a response, thereby significantly improving inference efficiency. This point is also discussed in lines 313–316 of our paper.

Q6 Practical Patching

Our patching method is input-agnostic; that is, all experimental results in our paper are obtained by applying patching to all prompts, which does not need to identify harmful prompts beforehand.

Thank you for your recognition of our work and your valuable suggestions. Here are our responses to the weaknesses and questions.

W1 Generality beyond PEFT

We chose to employ $(IA)^3$ because full fine-tuning modifies the weights of neurons, making it challenging to ensure the neurons in the base model and fine-tuned model are consistent. We greatly appreciate your insightful suggestion about adding an ablation. Recent studies [1, 2] have indicated that full fine-tuning may only modify a small subnetwork of the model, suggesting that safety-related neurons might indeed be preserved during this process. As a result, we plan to investigate this aspect further in our future work.

[1] Wu, Taiqiang, et al. "Shadow-FT: Tuning Instruct via Base." arXiv 2025.

[2] Mukherjee, Sagnik, et al. "Reinforcement Learning Finetunes Small Subnetworks in Large Language Models." arXiv 2025.

W2 Relation to steering-vector work

Due to space limitations, we are currently unable to provide a comprehensive discussion on this matter in the paper. Our ultimate goal is to investigate the safety mechanisms of LLMs. Steering effectiveness, in this context, serves as a metric to assess the accuracy and reliability of the research objects we identify. Unlike papers focused on providing practical steering methods, this paper focuses on providing mechanistic interpretations at the neuron level. We believe there are intrinsic connections between neuron-level mechanisms and vector-steering, and we leave the detailed explorations for future work.

W3 Predictive safeguard feels expected

There may be two potential mechanisms through which safety neurons operate: one involves distinguishing harmful inputs, and the other increases the likelihood of rejection, or a combination of both. These mechanisms are not necessarily bound together. In other words, even if we can identify harmful inputs based on the model’s internal state, it does not guarantee that we can predict whether the model will reject the response. Our experimental results suggest that safety neurons may play a role in the latter mechanism, though the full mechanism remains to be explored.

Q1 safety alignment is done in both SFT and DPO stage

Our approach requires a pair of aligned and non-aligned models. However, we found that most open-source instruct models, either intentionally or unintentionally, have already undergone some form of safety alignment. As a result, we decided to start training from the base model. However, directly applying DPO safety alignment to the base model is not feasible. Therefore, we introduced a non-safety-aligned SFT phase as an intermediary step. The SFT stage here does not involve safety data but focuses on training basic instruction-following abilities to prepare safety alignment (DPO). We will add more details to the paper to make it clear.