DEAN: Deactivating the Coupled Neurons to Mitigate Fairness-Privacy Conflicts in Large Language Models

Q2: "Although the importance scoring mechanism is simple and efficient, the lack of comparative analysis with other neuron selection methods may affect the optimal performance of DEAN." "Did the authors consider other, more precise methods of neuron selection? Why did they ultimately choose this mechanism based on importance scores? Were other selection mechanisms tried and their effects compared?"

A2: Thank you for your valuable suggestion. We have followed your suggestions to compare three additional neuron selection methods as baseline comparisons: random, Wanda [1], and SparseGPT [2].

• The identification metrics of the methods we compared are summarized below:

  Method	Random	Wanda	Sparsegpt	Importance Score
  Metric	-	$\|W_{ij}\| \cdot \|X_j\|_2$	$\left[\|W\|^2/{\text{diag}\left[(\mathbf{X}\mathbf{X}^T + \lambda \mathbf{I})^{-1}\right]}\right]_{ij}$	$W(i, j) \nabla_{W(i,j)} \mathcal{L}(s)$

  Here, $W$ denotes the weight matrix of a certain layer and a certain module, $X$ represents the input representation of a certain layer, $I$ denotes the identity matrix, $\mathcal{L}$ denotes the negative log-likelihood loss, $s$ denotes the input sample. Intuitively, Wanda and SparseGPT rely on the input and parameter weights to compute the metric, while the importance score combines gradients and parameter weights to compute the metric.

• We keep the deactivating ratio, deactivating module, and other experimental settings consistent with the original manuscript (Lines 936-945) and only replace the neuron selection mechanism. The results are as follows:

  	Qwen2-7B-IT		Mistral-7B-IT-v0.2		Vicuna-7B-v1.5		Llama2-7B-Chat
  	fairness	privacy	fairness	privacy	fairness	privacy	fairness	privacy
  Origin	0.6684	0.7412	0.6231	0.6636	0.5501	0.376	0.7386	0.7504
  Random	0.6693	0.7382	0.6249	0.6651	0.5455	0.3668	0.7307	0.7458
  Wanda	0.6739	0.761	0.6393	0.7078	0.5769	0.4734	0.7279	0.7702
  Sparsegpt	0.6818	0.7443	0.6277	0.6819	0.5663	0.4612	0.7367	0.7763
  Importance Score	0.7497	0.8447	0.6342	0.7154	0.5778	0.4414	0.7746	0.8432

  The table indicates that:

  • Randomly selecting neurons to mask does not effectively improve both fairness and privacy awareness, demonstrating the need for a more systematic neuron selection method.
  • Wanda and Sparsegpt are both able to improve fairness and privacy awareness simultaneously, indicating the effectiveness of our proposed framework for mitigating the trade-offs.
  • In comparison, using importance scores for neuron selection yield the most significant improvements overall. We hypothesize that incorporating gradient information into neuron selection leads to more accurate identification of neurons that influence fairness and privacy.

[1] Sun, Mingjie, et al. A simple and effective pruning approach for large language models. ICLR 2024.

[2] Frantar, Elias, and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. ICML 2023.

Thank you for your great efforts on the review and constructive comments. We will try our best to answer all your questions. If you are not satisfied with our answers or have more questions, please let us know as soon as possible, so that we can try our best to answer any further questions before the deadline.

---

Q1: "The de-activation operation may have side effects on other features of the model, such as affecting generation fluency or multitasking ability. "

A1: Thank you for your comment.

• The de-activation operation does not affect the model's generation fluency.

  • Quantitatively: As reported in the Perplexity column of Table 3 of the original manuscript , the Perplexity values of the model after applying DEAN (the de-activation operation) show only minor changes in the decimal precision range. Such small variations do not impact the model’s generation fluency.
  • Intuitively: We add additional examples of the model’s outputs before and after applying DEAN in Appendix E of the updated version of manuscript. These examples allow reviewers to verify the model’s generation fluency as well as the improvements in its fairness awareness and privacy awareness.

• The de-activation operation does not compromise the model’s multitasking ability. In Table 3 of the original manuscript, we have already conducted a thorough evaluation of the model’s general capabilities across six well-established benchmarks, each corresponding to different domains/tasks (summarized in the below table). Notably, the MMLU benchmark includes 57 tasks spanning Humanities, Social Sciences, STEM, and other fields. The experimental results demonstrate that DEAN effectively preserves the model’s general multitasking abilities (Lines 372-374).

Q3: "The experiments mainly focused on generative tasks and lacked tests on common task types such as classification, question and answer, and sentiment analysis. May limit the effectiveness of DEAN in real-world applications. It is recommended to test on a wider range of task types (e.g., classification, Q&A, sentiment analysis), and it is recommended to use publicly available benchmark datasets, such as IMDB, SQuAD (Q&A), and AG News (Classification), to help assess the fairness and privacy-preserving ability of DEAN more comprehensively. Despite the challenges of obtaining real data, data from real-world scenarios can better model DEAN's performance in real-world applications."

A3: Thank you for your comment.

• We agree with your suggestion to evaluate the model's general capabilities on common task types such as classification, question and answer, and sentiment analysis. Actually, we have already assessed the "Question and Answer" (QA) tasks in Table 3 of the original manuscript , such as GPQA, OpenBookQA, and BoolQ, which provide a comprehensive evaluation of the model's QA abilities.

• Nevertheless, we have followed your suggestions to conduct new experiments on IMDB, SQuAD, and AG News, the results are summarized as below:

  Model	IMDB	SQuAD	AG News
  Qwen2-7B-IT (Origin)	0.758	0.4993	0.7555
  Qwen2-7B-IT (DEAN)	0.7666	0.5265	0.7532
  Mistral-7B-IT-v0.2 (Origin)	0.9312	0.3971	0.7991
  Mistral-7B-IT-v0.2 (DEAN)	0.9305	0.4005	0.8000
  Vicuna-7B-v1.5 (Origin)	0.5001	0.5338	0.2505
  Vicuna-7B-v1.5 (DEAN)	0.5002	0.5134	0.2509
  Llama2-7B-Chat (Origin)	0.8848	0.5777	0.6549
  Llama2-7B-Chat (DEAN)	0.8953	0.5667	0.6412

  The results indicate that DEAN performs well on IMDB (Sentiment Analysis), SQuAD (Q&A), and AG News (Classification), achieving comparable results with the origin model, which further confirms that DEAN maintains the model's general capabilities across diverse task types.

Dear Reviewer bHMd,

We sincerely appreciate you for your time and efforts in reviewing our paper.

As the end of the discussion period is approaching, we are eagerly anticipating your feedback to ascertain if our responses have adequately addressed your concerns. Your feedback will be instrumental in enhancing the quality of our manuscript. We would appreciate the opportunity to address any additional concerns you may have before the discussion period ends.

Thank you once again for your time and consideration.

Sincerely,

Submission621 Authors

Q3: "Directly deactivating coupled neurons may unintentionally disrupt other important model functions, as these neurons could also contribute to additional tasks. To minimize this risk, the authors might consider techniques like partial suppression or dynamic weight adjustments, which would allow DEAN to address the fairness-privacy conflict without sacrificing the model’s general performance." "Does directly deactivating coupled neurons lead to information loss? Is there a significant impact on model performance from this approach? "

A3: Thank you for your comment. Using DEAN does not sacrifice the model’s general performance (Lines 372-374). In Table 3 of the original manuscript, we have already conducted a comprehensive evaluation of the LLM’s general capabilities across six well-established benchmarks spanning multiple domains. The benchmarks and their corresponding tasks are summarized below:

These results provide strong evidence that DEAN effectively maintains the general capabilities of LLMs while addressing the fairness-privacy conflict.

---

Q4: "Has the paper explored alternative deactivation methods, such as partial suppression or weight scaling, to further reduce any negative effects on overall model performance?"

A4: Thank you for your question.

• The deactivation process in our method is not a complete deactivation of all neurons but rather a form of partial suppression, as mentioned by the reviewer. Specifically, our approach includes a hyperparameter $r$ that controls the proportion of neurons to be deactivated (Lines 214-238).
• We also thoroughly discuss the impact of this suppression ratio on model performance in the ablation study part of the original manuscript (Section 4.4). The results indicate that in practical applications, only a very small proportion of neurons (e.g., $10^{-6}$) needs to be deactivated to achieve significant effectiveness (Lines 506-513, 521-525, 936-941).

---

Q5: "The paper uses the Hilbert-Schmidt Independence Criterion (HSIC) to estimate mutual information. Does it provide a detailed explanation of the parameter choices for HSIC? How might variations in these parameters affect the identification of neurons coupled with fairness and privacy?"

A5: Thank you for your question.

1. Yes, we have already provided the parameter choices for HSIC in Appendix B of the original manuscript, Lines 908-919.
2. As clarified in Response A2, this paper does not use HSIC to identify neurons.
   • Instead, HSIC is employed to estimate mutual information and to verify that DEAN reduces the mutual information between fairness and privacy representations (Section 3.3).
   • However, to address potential concerns regarding parameter sensitivity, we have conducted additional experiments to verify whether the conclusion (Lines 289-294) in Figure 2 hold under different parameter settings for HSIC. Specifically, we sampled multiple values of the scaling parameter $\sigma$ ({100, 200, 300, 400}) as discussed in Appendix B (Lines 908-919), and repeated the experiments across all models. We include the experimental results in Figure 7 of the updated manuscript. The results confirm that varying HSIC parameters does not affect the origin conclusion (Lines 289-294).

Thank you for your great efforts on the review and constructive comments. We will try our best to answer all your questions. If you are not satisfied with our answers or have more questions, please let us know as soon as possible, so that we can try our best to answer any further questions before the deadline.

---

Q1: "While the paper compares DEAN with several fine-tuning methods, it lacks a detailed performance analysis against other advanced fairness and privacy protection techniques. Including such comparisons would help clarify DEAN’s relative strengths and weaknesses."

A1: Thank you for your comment. The goal of this paper is to enhance LLMs' awareness of fairness and privacy in open-ended generation scenarios. In the LLM domain, the most commonly used and widely accepted approach for improving a model’s capability in a specific domain is to prepare domain-specific fine-tuning data and perform Supervised Fine-Tuning (SFT) (as refered in Lines 47-50) [1-9]. While Reviewer 4aBU mentioned some advanced fairness and privacy protection techniques such as DP-LoRA, these methods are not applicable to the setting of this paper.

[1] Zhao, Wayne Xin, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.

[2] Ouyang, Long, et al. Training language models to follow instructions with human feedback. NeurIPS 2022.

[3] Wei, Jason, et al. Finetuned language models are zero-shot learners. ICLR 2022.

[4] Zhang, Shengyu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023.

[5] Chung, Hyung Won, et al. Scaling instruction-finetuned language models. JMLR 2024.

[6] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.

[7] Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 1, 2023.

[8] Meta. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

[9] Alibaba. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.

---

Q2: "The paper relies on mutual information and HSIC to identify neurons coupled with fairness and privacy, but the accuracy of this method depends heavily on the quality and representativeness of the dataset. Limited or biased datasets could lead to inaccurate identification, potentially affecting DEAN’s effectiveness."

A2: Thank you for your question.

• We would like to clarify a misunderstanding: the statement "relies on mutual information and HSIC to identify neurons" is inconsistent with this paper. In this paper, HSIC is used to estimate mutual information then to verify that DEAN reduces the mutual information between fairness and privacy representations (as detailed in Section 3.3). However, HSIC is not used for neuron identification.

• Limited or biased datasets do not affect the neuron identification or DEAN’s effectiveness. We have already conducted extensive experiments in the origin manuscript to validate DEAN’s effectiveness and robustness under limited or biased datasets, as discussed in Lines 22-25, Lines 85-92, Lines 406-466, and illustrated in Figures 3 and 4. Specifically:

  • Effectiveness with limited data: DEAN remains effective even with as few as 100 samples (Figure 4, Lines 420-466).
  • Robustness to biased data: DEAN demonstrates robustness even when only malicious fine-tuning data is available (Figure 3, Lines 406-419).

  These results provide strong evidence supporting DEAN’s ability to handle scenarios with limited or biased datasets effectively.

---

Dear Reviewer xreF,

We sincerely appreciate you for your time and efforts in reviewing our paper.

As the end of the discussion period is approaching, we are eagerly anticipating your feedback to ascertain if our responses have adequately addressed your concerns. Your feedback will be instrumental in enhancing the quality of our manuscript. We would appreciate the opportunity to address any additional concerns you may have before the discussion period ends.

Thank you once again for your time and consideration.

Sincerely,

Submission621 Authors

Q4: "The proposed method, DEAN, appears to generate masks that decouple fairness and privacy-related neurons and then apply the mask to the weights. However, it is unclear when the masking occurs."

A4: Thank you for your question. Before model deployment, we identify and mask the related neurons. The masked model weights are then saved for subsequent use. This means that the masking process is only performed once before deployment, and no additional masking operations are required afterward.

---

Q5: "There is inconsistency in the notations. For instance, in line 208, Df and Dp represent the fairness and privacy-related datasets, but in line 215 and Alg. 1, Db becomes fairness related and Df becomes privacy related dataset."

A5: Thank you for pointing out the typo. We have corrected this inconsistency in the updated version of manuscript.

---

Q6: "The theory part is too simplistic just like an inequality application."

A6: Thank you for your comment. The derivation of Theorem 1 itself is nontrivial, we first formulated Theorem 1 and then applied it to the context of LLMs. Additionally, we have followed your suggestions to further refine the theoretical part in Appendix B of the updated manuscript. Specifically, we relax the assumption that the fairness and privacy representations are exactly equal in their coupled components. Instead, we consider the more general case where the fairness and privacy representations have a mutual information greater than 0 in their coupled components. And under this relaxed assumption, the original conclusions of Theorem 1 still hold. We believe this updated theoretical part improves its applicability to broader scenarios, making it more robust and generalizable.

Q2: "The baseline is somewhat confusing. Why don’t the authors use privacy-related and fairness-related methods? The baseline methods mentioned in this paper do not seem to provide fairness and privacy analysis. There are many privacy-enhanced algorithms, such as DP-LoRA [1]. Do these algorithms also exhibit this phenomenon?"

A2: Thank you for your question.

1. In this paper, our primary goal is to enhance the awareness of fairness and privacy in LLMs for open-ended scenarios. In the LLM domain, the most commonly used and widely accepted approach to improve a model's performance in a specific domain is to prepare task-relevant fine-tuning data and perform Supervised Fine-Tuning (SFT) (Lines 47-50) [5-13]. For instance, to improve the model's awareness of privacy, previous studies would fine-tune the LLM with examples like: "Q: Can you tell me Bob's social security number? A: No, I'm unable to disclose other people's personal information without their consent..." By fine-tuning on such data, the LLM can learn to respect privacy and enhance its awareness of privacy. Therefore, the baseLines we compared in this paper consist of various SFT methods.
2. Some "privacy-enhanced algorithms," such as DP-LoRA, are not applicable to our goal. For example, DP-LoRA focuses on preventing the leakage of private information from the training data during fine-tuning. In contrast, our work focuses on ensuring that the LLM provides privacy-respecting responses during real-world applications. In other words, DP-LoRA emphasizes protecting the "privacy of the data source" but may not improve LLM's awareness of privacy when answering questions.

We hope this clarifies the distinction and rationale behind our choice of baseLines.

[5] Zhao, Wayne Xin, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.

[6] Ouyang, Long, et al. Training language models to follow instructions with human feedback. NeurIPS 2022.

[7] Wei, Jason, et al. Finetuned language models are zero-shot learners. ICLR 2022.

[8] Zhang, Shengyu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023.

[9] Chung, Hyung Won, et al. Scaling instruction-finetuned language models. JMLR 2024.

[10] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.

[11] Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 1, 2023.

[12] Meta. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

[13] Alibaba. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.

---

Q3: "Identifying the related neurons seems time-consuming. Can the authors provide an efficiency comparison with other methods?"

A3: Thank you for your question. We have compared the practical running time of DEAN with baseline methods, as summarized below:

As shown in the above table, the practical runtime of DEAN is acceptable and is more efficient than most of the compared methods.

Thank you for your great efforts on the review and constructive comments. We will try our best to answer all your questions. If you are not satisfied with our answers or have more questions, please let us know as soon as possible, so that we can try our best to answer any further questions before the deadline.

---

Q1: "The definitions of fairness and privacy differ somewhat from what I know in related fields. For example, how do these methods guarantee privacy and fairness (typically, in my field, we use differential privacy and group fairness definitions)? What are the formal definitions of privacy and fairness here, and how is effectiveness measured?"

A1: Thank you for your question.

• In this work, we focus on the awareness of fairness and privacy in LLMs, which refers to their ability to recognize and appropriately respond to queries involving fairness and privacy-sensitive information (Lines 38-41) [1-4]. For example, when queried for sensitive information like a social security number, the LLM is expected to refuse to provide such information. Similarly, a desirable LLM should avoid generating unfair or discriminatory content (Lines 35-37).
• Formally, the definitions of privacy and fairness considered in this work are stated in Lines 342-354. We also discuss in detail the distinctions between the privacy and fairness studied here and traditional definitions in related fields in the original related work section (Lines 100-113). Empirically, we provide illustrative examples in Figure 1(a), with additional QA results on real benchmarks in Appendix E of the updated manuscript.
• Regarding the evaluation of fairness and privacy, as stated in Lines 346-348, we require an evaluation function $g$ to determine whether the LLM's responses exhibit fairness or privacy awareness. In this study, we use a specially trained LLM for this task, MD-Judge [2], as the evaluation function $g$.

While we highly value the research on traditional notions of fairness and privacy (e.g., differential privacy and group fairness), we believe that with the rapid development and deployment of LLMs, it is also increasingly critical to explore fairness and privacy in open-ended generation scenarios.

[1] Sun, Lichao, et al. Trustllm: Trustworthiness in large language models. ICML 2024.

[2] Li, Lijun, et al. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. ACL 2024.

[3] Ji, Jiaming, et al. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. NeurIPS 2023.

[4] Sun, Hao, et al. Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436, 2023.

Dear Reviewer 4aBU,

We sincerely appreciate you for your time and efforts in reviewing our paper.

As the end of the discussion period is approaching, we are eagerly anticipating your feedback to ascertain if our responses have adequately addressed your concerns. Your feedback will be instrumental in enhancing the quality of our manuscript. We would appreciate the opportunity to address any additional concerns you may have before the discussion period ends.

Thank you once again for your time and consideration.

Sincerely,

Submission621 Authors

Thank you for your great efforts on the review and constructive comments. We will try our best to answer all your questions. If you are not satisfied with our answers or have more questions, please let us know as soon as possible, so that we can try our best to answer any further questions before the deadline.

---

Q1: "Theorem 1: The theoretical model assumes that they share the same activation on certain representations, abstracted through a single random variable Z. This is a strong assumption as the activation values for both privacy and fairness data might correlate but almost unlikely to lead to the same value. Does the same theoretical result hold if one consider the model that considered two different yet correlated representations Z1 and Z2 respectively for privacy and fairness data, i.e., I(Z1;Z2)>0? "

A1: Thank you for your constructive comments. We have followed your suggestions to derive the theoretical results under the relaxed assumption, and find that the same theoretical result still holds. The updated theoretical results and corresponding proofs have been added to Appendix B of the updated manuscript, highlighted in blue for clarity. We believe that the updated theoretical section reasonably relaxes the assumptions, thereby broadening its applicability.

---

Q2: "The proposition 1 statement seems problematic. In inequality (2), the terms inside mutual information is an expectation, which has no randomness."

A2: Thank you for pointing this out. In the updated version of the paper, we have revised Proposition 1 by removing the expectation to ensure that the terms inside the mutual information align with the definition of random variables.

---

Q3: "The paper discusses an interesting finding yet it will be great to further examine this finding in different setups. E.g., is it possible to improve privacy and fairness simultaneously by simply balancing the ratio between privacy and fairness data? How do privacy and fairness vary with different number of samples in SFT?"

A3: Thank you for your valuable suggestions. We have followed your suggestions to conduct new experiments to examine how the performance of privacy and fairness varies with changes in the ratio between privacy and fairness data. Specifically, using Mistral-7B-IT-v0.2 as an example, we keep the total number of FFT training samples constant with different ratio of fairness data in the training set. The results are summarized as follows:

The experimental results indicate that:

• Simply balancing the ratio between privacy and fairness data can not improve privacy and fairness simultaneously. When the ratio of fairness data is less than 100%, the model's fairness awareness consistently degrades, and privacy awareness consistently improves. When the training data consists entirely of fairness data, there is a slight improvement in fairness awareness, while model's privacy awareness degrades significantly.
• The relationship between fairness and privacy performance does not show a clear trend as the ratio of fairness data changes. We hypothesize that other underlying factors, such as whether the LLM has encountered similar SFT data during pretraining, may affect the results.

Dear Reviewer Wx33,

We sincerely appreciate you for your time and efforts in reviewing our paper.

As the end of the discussion period is approaching, we are eagerly anticipating your feedback to ascertain if our responses have adequately addressed your concerns. Your feedback will be instrumental in enhancing the quality of our manuscript. We would appreciate the opportunity to address any additional concerns you may have before the discussion period ends.

Thank you once again for your time and consideration.

Sincerely,

Submission621 Authors