Understanding and Enhancing Safety Mechanisms of LLMs via Safety-Specific Neuron

Dear Reviewer WgZu,

We appreciate the time and effort you have put into providing valuable feedback. However, we respectfully believe there might be some misunderstanding regarding our work. We would appreciate the opportunity to clarify a few points and address your concerns as follows:

Concern #1 Distinction of the paper

Our safety neuron detection method builds upon the approach introduced in [1], but our work significantly extends this foundation. Instead of proposing a new detection method, our key contribution is applying this technique to safety scenarios. We define, identify, and verify safety neurons while conducting a comprehensive analysis to explore their features. Furthermore, based on our novel and insightful findings, we develop two novel methods: SN-Tune for efficient safety alignment and RSN-Tune for robust safety alignment.

Another paper you mentioned [2] similarly applies existing techniques to safety scenarios rather than introducing new detection methods. It relies on SNIP scores [3] which require ground truth labels, and implements [4]'s uniform sparsity constraints. In contrast, our method achieves superior performance without these limitations - identifying less than 0.3% of parameters as safety neurons versus their 3%, while maintaining higher accuracy. Moreover, we extend beyond neuron analysis to introduce practical safety alignment methods (SN-Tune and RSN-Tune), an aspect not addressed in [2].

We respectfully note that multiple reviewers have recognized our contributions: Reviewer 9nQR noted our "interesting verification of safety neurons," Reviewer Uh2w highlighted our "originality," "experimental rigor," and "practical impact," Reviewer Qmyn praised our "strong defense performance," and Reviewer R3v2 emphasized our "empirical insights" and "relevance of results for practitioners."

Concern #2 Features of safety neurons

1. Safety neurons are also used for general capability.

We would like to first clarify “pruning” that you mentioned in the review and “deactivating” that we used in our experimental settings. In the context of neuron level, pruning cannot be directly applied to the self-attention layer, as it would alter the number of key-value pairs in each attention head. This limitation can only be overcome by either enforcing uniform neuron removal across heads or eliminating entire attention heads. In contrast, deactivation can be implemented in any structure by simply setting the parameters of the neuron to zero. While the sparsity approach employed in [2] is applicable to self-attention structures, it proves inefficient due to its high computational cost and low accuracy, as it requires maintaining uniform sparsity across columns or rows. This flexibility enables our method for more accurate safety neuron detection.

Moreover, we acknowledge that completely separating safety-related neurons from general capability neurons is challenging, as safety itself represents a specific model capability. This observation aligns with [2], where the phenomenon is even more severe. As shown in their Figure 2(a), the safety mechanism completely breaks down when general capability accuracy decreases from 0.58 to 0.52. Additionally, while [2] evaluates general capabilities using mostly classification tasks, our evaluation involves the more challenging GSM8K reasoning tasks.

2. Safety neurons in different structures.

Since self-attention layers contain approximately half as many neurons as feed-forward layers, the higher concentration of safety neurons in self-attention shown in Figure 3 indicates that safety neurons are disproportionately located in self-attention layers. To further clarify and facilitate understanding, we list the number of safety neurons in each architectural component.

Concern #3 Experiment details for SN_Tune and RSN-Tune.

1. Safety Neuron detection time

Thanks for the comment. We acknowledge this oversight in the paper. For clarification, we utilize the training documents from [5] as noted in line 158, and sample 200 documents for detection. Due to the parallel detection approach described in Equations (4) and (5), the entire detection process is completed in under one minute.

2. Difference between SN-Tune and RSN-Tune

According to Table 4, the RSN-Tuned model demonstrates reduced harmfulness compared to the SN-Tune, with scores decreasing from 38.0 to 26.0 for Llama2-7b-chat and from 72.0 to 41.0 for Mistral-7B-Instruct-v0.2. As noted in lines 412-413, "A complete harmful score reduction to 0.0 is not achievable due to an insufficient number of non-overlapping safety neurons."

Concern #4 Unclear explanation

We clarify several details from the paper that may have caused confusion. We will update the draft to make it clearer.

1. We explain the portion of safety neurons from line 245 to line 247. “In Llama2-7B-Chat, the leap appears when deactivating 0.3% neurons, while the number is 0.15% for Llama3-8B-Instruction and is 0.1% for Mistral-7B-Instruct-v0.2.”
2. Thanks for your valuable suggestion. However, helpfulness is not explored in this paper and we mainly focus on harmfulness and general capability, a setting same as [5]. Similarly, LLama2-7b-chat is not reported in Table 2 because its harmfulness has already been very low and leaves no space for us to improve, also the same setting as [5].
3. Section 2.3.2 analyzes safety neurons from a multilingual perspective, as safety alignment across different languages poses significant challenges. In Figure 5, languages X and Y represent the language pairs for which we analyze safety neuron overlap, following the notation used in Equation (288).
4. Thanks for reminder. We ignore detailed description of language-specific neurons in the paper, which are a subset of neurons responsible for processing multilingual queries, as defined in [1].

[1] Zhao, Yiran, et al. "How do Large Language Models Handle Multilingualism?." arXiv preprint arXiv:2402.18815 (2024).

[2] Wei, Boyi, et al. "Assessing the brittleness of safety alignment via pruning and low-rank modifications." arXiv preprint arXiv:2402.05162 (2024).

[3] Lee, Namhoon, Thalaiyasingam Ajanthan, and Philip HS Torr. "Snip: Single-shot network pruning based on connection sensitivity." International Conference on Learning Representations (2022).

[4] Sun, Mingjie, et al. "A Simple and Effective Pruning Approach for Large Language Models." The Twelfth International Conference on Learning Representations. (2024)

[5] Zou, Andy, et al. "Improving alignment and robustness with circuit breakers." The Thirty-eighth Annual Conference on Neural Information Processing Systems. (2024)

Dear Reviewer WgZu,

Thank you for your thoughtful comments and valuable feedback. We deeply appreciate the time and effort you have invested in reviewing our work.

We have carefully addressed your comments in the rebuttal, clarifying key details to respond to your concerns, suggestions, and misunderstandings. We kindly ask for your prompt attention to our response.

Thank you again for your time and consideration.

Best regards, Authors

Thank the authors for their response. However, there are concerns as below.

Concern #2 Features of safety neurons

1. Safety neurons are also used for general capability.

• What you referred to as “deactivating” is, in fact, magnitude (a.k.a. weight or unstructured) pruning, which I referred to.

• As for the fact that this submission still has a huge overlap with [1], the response was not convincing. All the exact same formulations are found in [1] except for (3), where the authors feed harmful queries and see the activations. In addition, [2] also identified safety weights. Hence, considering those closely related papers, the contribution and novelty of this submission are not significant.

• The authors’ claim in the paper “Regarding general capability, deactivating the safety neuron shows minimal impact” conflicts with the claim in the response, “completely separating safety-related neurons from general capability neurons is challenging” Also, I noticed that although the authors stated them as “safety neurons” in the paper, in the response they slightly change the wording as “safety-related neurons.”

2. Safety neurons in different structures.

• The table confirms that the authors’ statement was overclaimed, “Safety neurons predominantly reside within the self-attention layers.”. 8:5 or 5:3 ratios are hard to say as “predominant.”

Concern #4 Unclear explanation

I note that it is not fair that the authors selectively do not show helpfulness and results on Llama2 instruction-tuned models, especially for fair and confident comparisons with other safety studies in LLMs. Hence, I am not fully convinced by this work.

By considering all the points above, I maintain the original score.

[1] Zhao, Yiran, et al. "How do Large Language Models Handle Multilingualism?." arXiv preprint arXiv:2402.18815 (2024).

[2] Wei, Boyi, et al. "Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications." ICML (2024).

Thank you Reviewer WgZu for your detailed reply! Let us address your remaining concerns as follows:

Llama2 instruction-tuned models

Like we mentioned earlier, we followed our baseline [1] and did not report results on Llama2-7B-Chat results in table 2 because it is already safe and resistant to safety attack methods.

However, we conducted experiments to use SN-Tune to make Llama2-7b-base safe, which can be found in table 3. The results show that our method can effectively make the base model even safer than the released chat model meticulously trained to ensure its safety by a big company.

We will include this result in the paper.

Contribution of the paper

We have elaborated the contribution of the paper in our previous response. Having [2] as our prior work by no means diminishes our contribution. Our paper and [2] studies two completely different fields: multilinguality and safety. None of our proposed methods, experiments and findings appear in [2].

The other reviewers gave 8, 6, 8, 6 for a reason. We identify individual neurons and train them and it’s different from the methods in prior work [3] that you mentioned. First, the biggest difference in method is that they study pruning methods to identify relevant parameters while we identify neurons and train them. They only show the impact of pruning those parameters instead of trying to make the model safe without sacrificing the capacity. As a result, our method as shown in table 2 can achieve safety with equal or better capability in reasoning tasks, while their studied methods lead to significant drop in classification accuracy when they prune those parameters as shown in the Figure 2(a) of their paper. Lastly, we only identify 0.3%, 0.15% and 0.1% of the neurons for various models to be safety neurons or safety-related neurons while they identify 3% of the parameters. We hope that this clarifies your question about the comparison with [3].

To us, being able to train individual neurons is like performing surgery on neuron networks. We do not know how general this method is but it is at least interesting and may have the potential to be used in other domains.

8:5 or 5:3 ratios are hard to say as “predominant.”

We searched the meaning of predominant in legal documents. According to this database [4], predominant means more than 80% or 50%. The average being 65% roughly matches 8:5 and 5:3 ratios. We are happy to change this word to “largely” if that is a better word. But we certainly did not say “almost all”, which would be an inaccurate description. Thank you for pointing this out!

general capability

As shown in Table 1, Deact-SN is 1 or 2 points lower than Deact-R in Avg. Capability when the Avg. Capability scores are 34.8, 67.6 and 50.3 which implies that "deactivating the safety neuron shows minimal impact" and “completely separating safety-related neurons from general capability neurons is challenging” are both true at the same time. We think it is fine to say safety-related neurons here.

“deactivating” is, in fact, magnitude (a.k.a. weight or unstructured) pruning

We’ve provided the definition of deactivating in line 708 and line 698 of the paper. We do not think there is any ambiguity here. Magnitude pruning is a specific terminology in the pruning field which is not familiar to the safety community and general LLM community, in addition to the other implications we discussed in the previous reply. Thus we will use our original terminology “deactivating”.

score

It is interesting that the reviewer does not mention any strengths of our work except that we work on safety, which is an important topic. This definitely is a completely different view than our own views and other reviewers’ views. We have done our best to address the reviewer’s comments but we are honestly not sure if the reviewer is willing to change their opinion.

[1] Zou, Andy, et al. "Improving alignment and robustness with circuit breakers." NeurIPS (2024).

[2] Zhao, Yiran, et al. "How do Large Language Models Handle Multilingualism?." NeurIPS (2024).

[3] Wei, Boyi, et al. "Assessing the brittleness of safety alignment via pruning and low-rank modifications." ICML (2024).

[4] https://www.lawinsider.com/dictionary/predominantly

Dear Reviewer R3v2,

Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:

Concern #1 Experiment detail

Thanks for the suggestion. To provide further clarity: In the safety neuron detection process, we utilize the training documents from [1] as noted in line 158, sampling 200 documents for detection. Specifically, the training set [2] contains harmful queries across various categories, including terrorism and violent extremism, self-harm, and political campaigning, etc. This diverse dataset helps ensure the generalizability of the detected neurons.

Furthermore, our analysis examined how the number of input documents affects safety neuron detection. The results demonstrate that 200 documents are sufficient to reliably identify safety neurons. We will update the draft accordingly.

When identifying foundation neurons, we apply the same detection algorithm but utilize a general corpus sampled from Wikipedia rather than harmful queries, as detailed in line 404. We analyze 10,000 documents, following the experimental setup established in [3]. These neurons demonstrate stable activation during fundamental language processing, validating their classification as foundation neurons.

Concern #2 Notation and presentation in Section 2.1

Thanks for the reminder. We would like to further clarify. Mask is an identity matrix with 1's on the diagonal and 0's elsewhere. For the parallel feature of neuron detection, Equations (4) and (5) operate on all neurons l in layer i simultaneously, rather than processing each neuron individually. This parallel computation across neurons enables faster execution compared to sequential processing of each neuron.

Concern #3 Safety neurons in the base model

The base model's lack of safety alignment complicates the classification of these neurons as true "safety neurons." Essentially, these neurons are activated when processing potentially harmful queries. They account for less than 1% of parameters and share a similar pattern with safety neurons in safety-aligned models. The difference is that, since the base model lacks inherent safety alignment, it consistently generates harmful outputs, regardless of the activation state of these neurons. Therefore, SN-Tune is introduced to address this by exclusively training these neurons to produce safe responses.

Suggestion #1 More related work

We appreciate your suggestion. The papers you recommend [4] [5] support our findings and serve as inspiration for our paper. We will cite them in the updated draft.

[1] Zou, Andy, et al. "Improving alignment and robustness with circuit breakers." The Thirty-eighth Annual Conference on Neural Information Processing Systems. (2024)

[2]https://raw.githubusercontent.com/GraySwanAI/circuit-breakers/refs/heads/main/data/circuit_breakers_train.json

[3] Zhao, Yiran, et al. "How do Large Language Models Handle Multilingualism?." arXiv preprint arXiv:2402.18815 (2024).

[4] Kotha, Suhas, Jacob Mitchell Springer, and Aditi Raghunathan. "Understanding Catastrophic Forgetting in Language Models via Implicit Inference." The Twelfth International Conference on Learning Representations. (2024)

[5] Jain, Samyak, et al. "Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks." The Twelfth International Conference on Learning Representations. (2024)

Dear Reviewer Qmyn,

Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:

Concern: Description of the experiments

Thanks for the suggestion. To provide further clarity: In the neuron detection process, we utilize the training documents from [1] as noted in line 158, sampling 200 documents for detection. Specifically, the training set [2] contains harmful queries across various categories, including terrorism and violent extremism, self-harm, and political campaigning, etc. This diverse dataset helps ensure the generalizability of the detected neurons.

Furthermore, our analysis examined how the number of input documents affects safety neuron detection. The results demonstrate that 200 documents are sufficient to reliably identify safety neurons. We will update the draft accordingly.

[1] Zou, Andy, et al. "Improving alignment and robustness with circuit breakers." The Thirty-eighth Annual Conference on Neural Information Processing Systems. (2024)

[2]https://raw.githubusercontent.com/GraySwanAI/circuit-breakers/refs/heads/main/data/circuit_breakers_train.json

Concern #5 Comment on related work

[2] employs various neuron detection methods to identify safety neurons and analyzes their brittleness through pruning. Their approach differs from ours in two key aspects. First, regarding neuron detection, [2] uses the SNIP score [3], which requires correct labels for gradient calculation, and implements [4], which enforces sparsity constraints like fixed sparse numbers in each column/row. In contrast, our method operates without requiring correct labels or uniform sparse parameters across columns/rows, enabling precise identification of 0.1% of parameters as safety neurons compared to their 3%, resulting in higher accuracy. Second, beyond analyzing safety neurons, we propose SN-Tune and RSN-Tune to enhance LLMs' safety alignment, an aspect not addressed in [2].

Related works [5], [6], and [7] explore model safety mechanisms through different approaches. [5] limits its focus to the feed-forward layer, leading to imprecise detection and identifying 5% of parameters as safety neurons. [6] introduces additional structural elements, while [7] improves attacks by targeting parameters with minimal training changes—an approach that proves impractical due to the black-box nature of LLM training.

[1] Qi, Xiangyu, et al. "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!." International Conference on Learning Representations. 2024.

[2] Wei, Boyi, et al. "Assessing the brittleness of safety alignment via pruning and low-rank modifications." arXiv preprint arXiv:2402.05162 (2024).

[3] Lee, Namhoon, Thalaiyasingam Ajanthan, and Philip HS Torr. "Snip: Single-shot network pruning based on connection sensitivity." International Conference on Learning Representations (2022).

[4] Sun, Mingjie, et al. "A Simple and Effective Pruning Approach for Large Language Models." The Twelfth International Conference on Learning Representations. (2024)

[5] Chen, Jianhui, et al. "Finding Safety Neurons in Large Language Models." arXiv preprint arXiv:2406.14144 (2024).

[6] Hsu, Chia-Yi, et al. "Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models." arXiv preprint arXiv:2405.16833 (2024).

[7] Zhang, Zhengming, et al. "Neurotoxin: Durable backdoors in federated learning." International Conference on Machine Learning. PMLR, 2022.

Dear Reviewer 9nQR,

Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:

Concern #1 Theory guarantee

We appreciate your interest in the theoretical foundations and guarantees of our neuron detection method. Our work focuses on empirical analysis, defining, identifying, and verifying safety neurons, and exploring their features comprehensively. Building on our novel insights, we have developed two methods: SN-Tune for efficient safety alignment and RSN-Tune for robust safety alignment. While theoretical frameworks are valuable, they often involve numerous assumptions. We plan to explore these theoretical aspects in future work.

Concern #2 Equations Clarification

1. Equations (4) and Equation (5) have the same value for different l in layer i.

Yes, you are right. Equations (4) and (5) operate on all neurons l in layer i simultaneously, rather than processing each neuron individually. This parallel computation across neurons enables faster execution compared to sequential processing of each neuron.

2. $h_{ffn}$ and input of $W_{down}$ in Equation (7) and (8).

We apologize for the typo. The dimension of $h_{ffn}$ should be $R^{l \times d_{inter}}$. I wrote $W_{down}(x)$ to indicate that the whole equation $|(h_{ffn}\cdot Mask[k]) W_{down}(x)|\_2$ involves $x$ as an input variable. This notation is not rigorous enough since $W_{down}$ does not directly take $x$ as input - the input information from $x$ is already contained in $h_{ffn}$. We will removed this notation in the updated version.

Concern #3 Implement RSN-Tune on Alpaca.

Following your suggestion, we employ the setting of Table 3 in [1], where we train Llama2-7B-Chat and Mistral-7B-Instruct-v0.2 by Alpaca. Detailed results are as follows. We find that RSN-Tune can still enhance models’ safety robustness during Alpaca tuning.

Concern #4 Implement RSN-Tune on SN-Tune Mistral.

Thanks for your suggestions. We respectfully disagree with the reviewer's assessment that Mistral-7B-Instruct-v0.2 is not safety-aligned. While a 36.0% harmful response rate indicates room for improvement, Mistral-7B-Instruct-v0.2 successfully refuses nearly two-thirds of harmful requests, demonstrating meaningful partial safety alignment. Furthermore, we conduct experiments on both well-aligned and partially aligned LLMs to verify the generalizability of RSN-Tune.

Following your suggestion, we applied RSN-Tune to the SN-Tune-aligned Mistral-7B-v0.1 model. The results demonstrate that this combination enhances the model's safety mechanisms and makes them more resilient during downstream tuning. While the RSN-Tuned Mistral-7B-Instruct-v0.2 still shows a harmful behavior score of 41.0 after tuning on downstream task, our combined approach performs better due to SN-Tune's ability to partially separate safety neurons from general neurons—a capability not achievable through traditional safety alignment methods.

Dear reviewer 9nQR,

Thank you for your thoughtful comments and valuable feedback. We deeply appreciate the time and effort you have invested in reviewing our work.

We have made our utmost effort to address your comments in the rebuttal. We clarified several details to address your converns and suggestions. If you feel that the clarifications align with your feedback, could you please consider raising your score in light of our efforts? Many thanks for your time; we are extremely grateful!

Best regards, Authors

Dear Reviewer Uh2w,

Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:

Suggestion #1 Theoretical foundation

We appreciate your interest in the theoretical foundations and guarantees of our neuron detection method. Our work focuses on empirical analysis, defining, identifying, and verifying safety neurons, and exploring their features comprehensively. Building on our novel insights, we have developed two methods: SN-Tune for efficient safety alignment and RSN-Tune for robust safety alignment. While theoretical frameworks are valuable, they often involve numerous assumptions. We plan to explore these theoretical aspects in future work. Regarding the reviewer's concern about neurons in the final layers qualifying as "safety neurons," we note that these neurons are empirically identified as important in safety-related query handling. We find no compelling reason to exclude them from our definition of safety neurons merely because they can not independently manage safety mechanisms. We agree this is an interesting open question and have added it to our discussion of future work.

Suggestion #2 More Exploration

We appreciate your thoughtful suggestions regarding additional aspects of safety neurons, including multilingual safety analysis, the impact of jailbreaking prompt positioning, and the effects of context window length. While these research directions would undoubtedly yield valuable insights, they extend beyond the current scope of our investigation. We plan to explore these promising directions in our future work. Furthermore, regarding safety neuron dynamics, SN-Tune exclusively modifies identified safety neurons while keeping other parameters fixed, which prevents non-safety neurons from transforming into safety neurons during fine-tuning.