Collapsed Language Models Promote Fairness

We truly thank reviewer 6ic8's time and effort in reviewing our paper!

Q1: More comprehensive theoretical explanation.

Thanks for this suggestion! We will further consider using the layer-peeled model [1] to analyze neural collapse in language models in the context of fairness.

Q2: Regarding non-binary protected attributes.

Our approach is directly applicable to non-binary protected attributes. The challenge is only from the data collection and evaluation side.

Most related works (that we compare with) only study lists of binary genders of only 'female' and 'male' elements. If better word lists beyond binary genders or other non-binary attributes, and corresponding evaluation metrics become available in the future, we would be happy to conduct related experiments.

Q3: Regarding for decoder-only models.

Thanks for this great suggestion!

It is possible to extend our work to decoder-only models.

However, as mentioned in paper [2], which provided a thorough summary of fairness in language models. For decoder-only models, prior methods have primarily focused on Llama or GPT. Moreover, most previous fairness and debiasing approaches have relied on prompt tuning [3,4], rather than model fine-tuning. Improvements in fairness achieved through these prompt-based methods cannot be explained by neural collapse, as they do not change the model's weights.

Due to the limited timeframe, we are unable to test this extension during the rebuttal period, but we will try to add it to the final version if the paper is accepted. If more debiasing approaches for decoder-only models are proposed in the future, we will also be happy to conduct further research.

[1] Zhu Z, Ding T, Zhou J, Li X, You C, Sulam J, Qu Q. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems. 2021 Dec 6;34:29820-34.

[2] Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint arXiv:2308.10149, 2023a.

[3] A. Tamkin, A. Askell, L. Lovitt, E. Durmus, N. Joseph, S. Kravec, K. Nguyen, J. Kaplan, D. Ganguli, Evaluating and mitigating discrimination in language model decisions, arXiv preprint arXiv:2312.03689.

[4] J. Mattern, Z. Jin, M. Sachan, R. Mihalcea, B. Scholkopf, Understanding stereotypes in language models: Towards robust measurement and ¨ zero-shot debiasing, CoRR abs/2212.10678. arXiv:2212.10678, doi:10.48550/ARXIV.2212.10678.

Q7: How the vocabulary is obtained and the impact of random words.

First, as we considered using gender words provided by the ASE work, we are not able to tell the precise efforts of collecting the list of sensitive words like those for gender. We will reach out to authors of ASE for details instructions on collecting sensitive words and estimate if this process is challenging or not.

Second, following your suggestion, we also randomly added 100 high-frequency words and 100 low-frequency words to the gender word list. The total amounts of random words we added (200) is approximately 50% of the number of gender words in the original list (425, see Appendix A). The results with these additional words are shown in the table below. We can observe that adding unrelated words has a negative impact on the model's bias.

Q8: Explanation of the visualizations in Section 5.

From Figure 2 in the main text, we can observe that when we compare the gender word clusters with and without adding $L_{nc3}$ to the training, the word embeddings within each class become more collapsing together. This aligns with the definition of NC3 and provides evidence that our training is effective.

Moreover, Reviewer also acknowledged our visualization: “provide helpful visualizations and analyses that illuminate how their method affects model representations.”

[1] Wu R, Papyan V. Linguistic Collapse: neural collapse in (Large) Language Models. arXiv preprint arXiv:2405.17767. 2024 May 28.

[2] Papyan V, Han XY, Donoho DL. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences. 2020 Oct 6;117(40):24652-63.

[3] Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint arXiv:2308.10149, 2023a.

[4] A. Tamkin, A. Askell, L. Lovitt, E. Durmus, N. Joseph, S. Kravec, K. Nguyen, J. Kaplan, D. Ganguli, Evaluating and mitigating discrimination in language model decisions, arXiv preprint arXiv:2312.03689.

[5] J. Mattern, Z. Jin, M. Sachan, R. Mihalcea, B. Scholkopf, Understanding stereotypes in language models: Towards robust measurement and ¨ zero-shot debiasing, CoRR abs/2212.10678. arXiv:2212.10678, doi:10.48550/ARXIV.2212.10678.

[6] Jacqueline He, Mengzhou Xia, Christiane Fellbaum, and Danqi Chen. Mabel: Attenuating gender bias using textual entailment data. arXiv preprint arXiv:2210.14975, 2022

[7] Bartl, Marion, Malvina Nissim, and Albert Gatt. "Unmasking contextual stereotypes: Measuring and mitigating BERT's gender bias." arXiv preprint arXiv:2010.14534 (2020).

[8] M. De-Arteaga, A. Romanov, H. M. Wallach, J. T. Chayes, C. Borgs, A. Chouldechova, S. C. Geyik, K. Kenthapadi, A. T. Kalai, Bias in bios: A case study of semantic representation bias in a high-stakes setting, in: Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT, 2019, pp. 120–128.

We truly thank reviewer fRsu's time and effort in reviewing our paper!

Q1: Further discussion on the results.

Thanks for the suggestion! We have included more results and discussions on BEC-Pro (Table 5) in Appendix F. We will add more discussions on our results and move some discussion from appendix to the main text in our camera ready.

Q2: Clarify of Tables.

Thanks for this suggestion! We have updated our Table 8 with underlines for better performance. We can see that on natural language understanding (NLU) tasks, the baseline and our method are comparable, validating that our method preserved language models’ performance on NLU tasks.

Q3: Regarding for Large Language Models (LLMs).

Thanks for this great suggestion!

It is possible to extend our work to LLMs.

However, the challenge is that most previous fairness/debiasing methods for LLMs have been dominated by prompt tuning [4,5], instead of model fine-tuning, as pointed out by the survey paper [3] on fairness in language models. Improvements over fairness by these prompt tuning methods are not related to neural collapse, as model weights are unchanged.

Due to the limited timeframe, we are unable to test this extension during the rebuttal period, but we will try to add it to the final version if the paper is accepted. If more debiasing approaches for LLMs are proposed in the future, we will also be happy to conduct further research.

Q4: Why we primarily focus on gender words.

We mainly focus on fairness about gender because we follow previous works. Works in [6,7] primarily focused on gender or extensively explored the associations between other words and gender. Additionally, a relatively comprehensive framework for evaluating bias [8] in gender-related words was established, allowing us to clearly observe whether NC is effective in debiasing.

Q5: Clarify of Notations.

Thanks for pointing out! We have updated our draft with the definition of $\sigma$ in Sec. 3.1.

Q6: The rationale behind using classifier weights and the exclusivity of NC3's effectiveness.

As explained in our Sec. 3.2.1 (Line 209-215), the neural collapse behavior manifests under certain conditions, including zero training loss and clean labels with balanced classes. However, these conditions deviate more in language model fine-tuning than vision models.

Such a difference between the language model and the vision model has a direct consequence: The model predictions align more with classifier weight instead of class means in language models. This phenomenon highlights a key difference between:

• Language models: class means are noisy and deviate from classifiers, as shown in Fig.1 (right) in [1] - the “Classifier Agreement” is less than 40%

vs.

• Vision models: class means are reliable and less noisy, as shown in Fig.7 of [2] - mismatches between class means (${\mu}_c$) and classifiers (${w}_c$) are less than 10%

This is why we use classifier weight instead of class means in the denominator in NC1(w) for language models.

Moreover, we further measure the token-prediction accuracy using either class means ($\underset{c \in \mathbb{V}}{argmin} \|{h} - {\mu}_c \|_2$) or classifiers ($\underset{c \in \mathbb{V}}{argmax} \langle{w}_c, {h} \rangle + b_c$) for language models. As shown in the table below, using class means leads to much worse predictions. These two observations motivate us to replace class means with classifiers in NC1.

We truly thank reviewer adpT's time and effort in reviewing our paper!

Q1: Clarify of Tables.

Thanks for the suggestion! Table 9 had bold for the best results in our previous submission. We have revised our Table 8 accordingly with underlines

Q1: Regarding for generation tasks.

Thanks for this great suggestion!

It is possible to extend our work to generation tasks.

However, the challenge is that most previous fairness/debiasing methods for generation tasks have been dominated by prompt tuning [1,2], instead of model fine-tuning, as pointed out by the survey paper [3] on fairness in language models. Improvements over fairness by these prompt tuning methods are not related to neural collapse, as model weights are unchanged.

Due to the limited timeframe, we are unable to test this extension during the rebuttal period, but we will try to add it to the final version if the paper is accepted. If more debiasing approaches for generation tasks are proposed in the future, we will also be happy to conduct further research.

[1] A. Tamkin, A. Askell, L. Lovitt, E. Durmus, N. Joseph, S. Kravec, K. Nguyen, J. Kaplan, D. Ganguli, Evaluating and mitigating discrimination in language model decisions, arXiv preprint arXiv:2312.03689.

[2] J. Mattern, Z. Jin, M. Sachan, R. Mihalcea, B. Scholkopf, Understanding stereotypes in language models: Towards robust measurement and ¨ zero-shot debiasing, CoRR abs/2212.10678. arXiv:2212.10678, doi:10.48550/ARXIV.2212.10678.

[3] Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint arXiv:2308.10149, 2023a.

Q5: Why we choose to use the difference in association scores between female and male on the BEC-pro.

We greatly appreciate your valuable feedback, which is a meaningful complement to our work. We think that both your comment and our explanation are correct, corresponding to two different but reasonable fairness definitions in this context.

1. Your interpretation of the association score is correct. It measures the correlation between a <person word> and a <profession> (smaller the less correlated). However, the association score itself is not an indicator of gender bias. For example, the associations of male and female could be both small, but any of their difference indicates a bias over genders (i.e. the profession has different inclinations to male and female).
2. When we measure the difference in the association scores between <female> and <male> with respect to a <profession>, this represents gender bias when associating with professions, thus serving as a more appropriate intrinsic measure, one that is specifically related to gender.

To facilitate your understanding, we have included an additional experiment below, where we also optimize NC3 in BEC using different sets of words. As shown in the table below, using gender or profession words can both reduce the association scores, but only regularizing NC3 on gender words can make the association with female and male less distinguishable (reduced diff), whereas regularizing on profession actually enlarges the gender bias.

Q6: Regarding causal language models.

Thanks for this great suggestion!

It is possible to extend our work to causal language models.

However, the challenge is that most previous fairness/debiasing methods for causal language models have been dominated by prompt tuning [4,5], instead of model fine-tuning, as pointed out by the survey paper [3] on fairness in language models. Improvements over fairness by these prompt tuning methods are not related to neural collapse, as model weights are unchanged.

Due to the limited timeframe, we are unable to test this extension during the rebuttal period, but we will try to add it to the final version if the paper is accepted. If more debiasing approaches for causal language models are proposed in the future, we will also be happy to conduct further research.

[1] Wu R, Papyan V. Linguistic Collapse: neural collapse in (Large) Language Models. arXiv preprint arXiv:2405.17767. 2024 May 28.

[2] Papyan V, Han XY, Donoho DL. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences. 2020 Oct 6;117(40):24652-63.

[3] Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint arXiv:2308.10149, 2023a.

[4] A. Tamkin, A. Askell, L. Lovitt, E. Durmus, N. Joseph, S. Kravec, K. Nguyen, J. Kaplan, D. Ganguli, Evaluating and mitigating discrimination in language model decisions, arXiv preprint arXiv:2312.03689.

[5] J. Mattern, Z. Jin, M. Sachan, R. Mihalcea, B. Scholkopf, Understanding stereotypes in language models: Towards robust measurement and ¨ zero-shot debiasing, CoRR abs/2212.10678. arXiv:2212.10678, doi:10.48550/ARXIV.2212.10678.

We truly thank reviewer ypXD's time and effort in reviewing our paper!

Q1: Clarify of Notations.

Thanks for these comments! We have revised these notations in blue in our updated draft (Eq. 3 on page 3).

Q2: The reason we use classifiers instead of class means.

As explained in our Sec. 3.2.1 (Line 209-215), the neural collapse behavior manifests under certain conditions, including zero training loss and clean labels with balanced classes. However, these conditions deviate more in language model fine-tuning than vision models.

Such a difference between the language model and the vision model has a direct consequence: The model predictions align more with classifier weight instead of class means in language models. This phenomenon highlights a key difference between:

• Language models: class means are noisy and deviate from classifiers, as shown in Fig.1 (right) in [1] - the “Classifier Agreement” is less than 40%

vs.

• Vision models: class means are reliable and less noisy, as shown in Fig.7 of [2] - mismatches between class means (${\mu}_c$) and classifiers (${w}_c$) are less than 10%

This is why we use classifier weight instead of class means in the denominator in NC1(w) for language models.

Moreover, we further measure the token-prediction accuracy using either class means ($\underset{c \in \mathbb{V}}{argmin} \|{h} - {\mu}_c \|_2$) or classifiers ($\underset{c \in \mathbb{V}}{argmax} \langle{w}_c, {h} \rangle + b_c$) for language models. As shown in the table below, using class means leads to much worse predictions. These two observations motivate us to replace class means with classifiers in NC1.

Q3: Comparison of using and not using normalization on (G)NC2(w).

For (G)NC2(w), we studied both with and without normalization. As shown in table below, we found that the impact of normalization was minimal and will not change our conclusion.

Q4: Visualizing the gaps of NC across different word lists.

Thanks for your comment!

Our main conclusion is that: debiased language models collapse more on fairness-sensitive words (instead of the whole vocabulary and random words). The significant differences between Table 1 vs. Table 2 or 3 support the conclusion. Indeed, based on your suggestion, we can illustrate this conclusion with a better presentation. In the table below, we show absolute differences (on NC1) between the biased model and the debiased model. We can clearly see more significant gaps on gender words (than whole vocabulary and random words).

In our revised paper, we also show a bar plot of this Table in Appendix G for better illustration.