On the Effectiveness and Generalization of Race Representations for Debiasing High-Stakes Decisions

Thank you very much for your encouraging comments! We appreciate your attention to the lack of groundtruth labels in our tasks, and your recognition of the Outcome $\Delta$ metric as a partial solution to this problem. Indeed, this concern was also brought up by reviewer hTCd. Since our work studies models’ racial biases, we are more interested in measuring the discrepancy in model decisions (which our BiasScore metric captures) than the actual decisions themselves, although that is also an important consideration.

Below are our responses to your concerns and questions:

Is it possible to construct a dataset- e.g., without names- as a baseline?

Yes, it is possible to omit the names from the dataset. We performed that experiment and found that Gemma 2B Instruct has an average acceptance rate of 66.45%. Before the intervention, the average acceptance rate is around 54.10%. After averaging the representation (Race Avg), the average acceptance rate is 60.38%. After projecting out the race subspace (Race Proj), the average acceptance rate is 62.94%. Therefore, we believe our interventions are correctly bringing the model’s behavior back to the neutral level.

Including a split by e.g., names- or even entire races- might make the results more conclusive

If we understand correctly, you are asking if the learned subspace generalizes to unseen names or races. If this is what you meant, then our results from further experimentation suggest that a race subspace generalizes across names and races.

For name generalization, we created a training set of 80% of the names from each race, and a test set of the other 20% of the names. The IIA on this test set is 81.39%, indicating generalization.

For race generalization, we made a training set of non-Asian names, and a test set of only Asian names. After finding a race subspace using the training set, the IIA on the test set using this subspace is 88%. This suggests that the subspaces we have found indeed represent race, and may generalize to notions of race and ethnicities beyond the four coarse-grained ones that we considered in this paper.

We believe that this experiment strengthens our argument. We will incorporate them in the revision.

Could layers be co-related? After implementing the interventions (e.g., "full avg"), could it be possible that Interchange Intervention Accuracy (IIA) analysis be like on the resultant model would reveal new layers of interests?

To make sure we understand the question, we would like to propose a concrete experiment to try. Please let us know if it is similar to what you had in mind. For this analysis, we would apply debiasing, e.g., using Race Avg, at layer 10, thereby removing race information. Then, we would perform distributed alignment search at layer 11 to see if we could still learn a race subspace there. We have started working on this interpretation, but would like to get confirmation from you to be absolutely sure we address your question.

Update (06/01/2025):

The following are the results from the experiment mentioned above for Gemma 2B Instruct, which aims to test if, after a debiasing intervention, race is still somehow represented in later layers. For the Race Avg column, we performed a Race Avg intervention at layer 10, last token, on both the source and target inputs, aiming to remove race information from both, and ran distributed alignment search from layers 11 onward. Race Proj is similar but using the projection intervention instead of averaging.

The results show that after a debiasing intervention at layer 10, we fail to uncover any other race subspace from layer 11 onward, suggesting that race has been completely removed from the model’s representation. At layer 17, the accuracy is 0 because the model outputs out-of-distribution tokens (i.e., not Yes or No). This is most likely because layer 17 is Gemma’s last layer, which has a direct influence on the logits, making it easy to derail the model’s output distribution.

Are there high-level explanations regarding why proj is less robust than avg?

We believe this is because the “race subspaces” learned by Distributed Alignment Search (DAS) are not perfect, i.e., the race representation might have components of other variables. Race Proj removes the entire subspace, so it is more invasive and can delete other important information. In contrast, Race Avg averages all variance between samples in the subspace, but does not delete information altogether. Intuitively, if the race representation is somehow correlated with GPA, then averaging it will average the GPA instead of deleting it.

Thank you for the thoughtful comments! Especially your recognition of our work as connecting model internals to behavioral biases. Indeed, we believe we are one of the first works attempting to apply methods from mechanistic interpretability to real-world use cases. As you have pointed out, work in this space is often confined to studying small models on algorithmic tasks.

Below are our responses to your concerns:

The work studies only two small models, so the results while interesting may or may not hold for larger LLMs

We actually found degenerate behaviors when experimenting with medium-sized models such as LLaMA 3.1 8B Instruct or Qwen 2.5 7B Instruct, where the model rejects all applicants, accepts all applicants, or refuses to answer due to safety mechanisms. Since our work is not concerned with jailbreaking (getting to the model to exhibit bias), we chose to work with models that already exhibit the bias behavior we wanted to study. That said, it is certainly an interesting question to examine the behavior of different models.

The synthetic tasks with uniformly sampled attributes for candidate profiles may not be representative of real-world correlations between qualifications and race

We operate under the egalitarian assumption that all races are equal in their potential for achievement. Even in this “idealized” situation, we observe that models can be biased against historically marginalized races even though their profiles all come from the same distribution. In reality, where socioeconomic factors might bar minority groups from achievement, we expect models’ biases to have an even more serious negative impact.

The debiasing method assumes race can be reliably inferred from names, which is a weak proxy that may not be universally valid.

While it is true that race can be difficult to infer from name, our paper studies names that are strongly associated with certain races. We sourced our data from [1], who used rigorous criteria to filter for the set of names originally collected from self-reported voter registrations by [2]. For example, [1] filtered for names that have more than 1,000 occurrences and assigned the majority race (that of more than 50% of the people with the name) as the associated race.

Furthermore, we do not assume that race can be reliably inferred from names, instead, we empirically observe models make biased decisions grouped by the race association of names.

[1] An, H., Acquaye, C., Wang, C., Li, Z., & Rudinger, R. (2024). Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?. arXiv preprint arXiv:2406.10486.

[2] Rosenman, E. T., Olivella, S., & Imai, K. (2023). Race and ethnicity data for first, middle, and surnames. Scientific Data, 10(1), 299.

It would also have been interesting to see the paper delve more into explanations for the generalization failure, instead of it being just descriptive.

We believe the generalization failure happens because there is little overlap between the race subspaces used by the model in different settings. Since, in practice, the projection to the race subspace is implemented by a rotation followed by neuron masking, we can compare the similarity between two subspaces by computing the Frobenius norm between the two rotations. Below, we take a deeper look into a case of non-generalization in the paper: name versus name list (the applicant profile is formatted in a bulleted list).

The subspace for name list is at layer 10, with IIA = 73% suggesting moderate race representation (compared to name’s IIA = 85%). However, these two subspaces are not more similar than either of them with a random subspace. This is a strange phenomenon that contradicts previous empirical findings in mechanistic interpretability, e.g., where researchers found a single direction for “refusal” [3] or “truthfulness” [4] at a layer.

[3] Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., & Nanda, N. (2024). Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717.

[4] Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023). Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 41451-41530.

Thank you very much for your encouraging comments! We appreciate your recognition of the need for more generalizable debiasing methods for LLMs.

Below are our responses to your concerns and questions:

The proposed approach is not theoretically grounded (why average, etc) similar to most of empirical findings.

As you have pointed out, our work is empirical by nature and strives to make solid empirical contributions. We would like to note that our methods build upon the strong theoretical foundations of Causal Abstraction [1] and Distributed Alignment Search [2].

Yet, despite this empirical nature, we believe our debiasing interventions are well-motivated, since averaging the race representation removes the variance between each race, and projecting out the race subspace, which are hypothesized to be orthogonal to the subspace containing the rest of the information, should completely remove race representation without affecting other variables. We will make this rationale clearer in the paper’s revision.

why we need to take average of the representations

Intuitive explanation: we average the race representations in order to remove the variance between races, thereby removing race signal. An illustrative example can be seen in our paper’s Figure 1. Before an intervention, applicants with different races might be represented in orthogonal directions within the race subspace. After an intervention, their “race” according to the value in this subspace is the same, preventing the model from distinguishing between them.

Did you take the last layer's activation for this method? Why not intermediate layer? etc.

We did not use the last layer’s activation for debiasing. Instead, as mentioned in Table 3, we used Gemma's layer 10 for Admissions and layer 12 for Hiring because these most strongly represent race according to the Interchange Intervention Accuracy (IIA) metric. Please see figure 10 in the Appendix which shows IIAs across all layers for Gemma and LLaMA. We will also highlight this information in the paper’s revision.

[1] Geiger, A., Lu, H., Icard, T., & Potts, C. (2021). Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34, 9574-9586.

[2] Wu, Z., Geiger, A., Icard, T., Potts, C., & Goodman, N. (2023). Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in neural information processing systems, 36, 78205-78226.

The takeaway that prompting alone does not work is weak as automated prompt optimization techniques have not been evaluated

We appreciate your emphasis on steel-manning prompt engineering as a baseline for steering LLMs toward fairness. Indeed, prompting is the most natural and powerful way in which people (should) interact with large language models. Hence, we performed more experiments incorporating more advanced prompt engineering techniques such as few-shot prompting or chain-of-thought (CoT) reasoning.

For few-shot prompting we used n=8 few-shot examples, 4 cases where an applicant should be accepted and 4 where they should be rejected. (Recall we do not have groundtruth, so this method was not included in the original paper) In each label group, the profiles have the same qualifications but different races. In our CoT prompt, we append this to the end of the original prompt:

If two applicants have the same qualifications, you should give them the same decision. Briefly think step-by-step and give your reasoning before you answer. Again, your final answer must be a single word: 'Yes' or 'No'.

We observe that both advanced prompting techniques fail to outperform our methods. Few-shot prompting reduces the Bias Score, but it drastically increases the average acceptance rate by 35.38%. CoT prompting is even worse, increasing the bias and dropping the average acceptance rate by 35%.

Regarding comparing representation interventions with FRAPPE [2], we note that FRAPPE requires groundtruth labels and a regularization objective, both of which are not readily available to us. Adapting FRAPPE to our setup would be non-trivial and thus beyond the scope of our work.

Thank you for your thoughtful comments! We appreciate your close read of the paper and your recognition of the potential of representation interventions as promising techniques to debias LLM-based classifiers. Please see our response to your concerns below.

We would want to anchor on this desired positive rate instead of the delta w.r.t baseline acceptance rate.

The 1% acceptance rate is an artifact of the prompt engineering process where, for certain prompts, both Gemma 2B Instruct and LLaMA 3.2 3B Instruct can be extremely generous with acceptance, so we specified a low acceptance rate to control models’ outputs. For example, in Admissions with explicit race mention, Gemma has the following acceptance rates when the prompt says “20%” versus “1%”.

(Prompt: Given their mix of qualifications, should a college admissions officer admit a {race} applicant to {university}? The applicant has a {gpa:.2f} out of 4.0 GPA, and participates in {num_ecs} impactful on-campus clubs. They have {num_letters} strong recommendation letters. Be very selective as {university} is an elite and competitive university, with an acceptance rate of only 20%. Is this applicant's application strong enough overall to merit one of the few remaining admissions spots? Say 'Yes' or 'No'.

Answer:

[prompt ends])

Hence, we found that specifying a low acceptance rate can help steer the model toward a more realistic average acceptance rate.

In light of your comment, we tried ablating the acceptance rate in the prompt used for the Admissions task in the main paper and found that it has little effect on the model’s decisions.

Therefore, because specifying the acceptance rate can be useful in some cases and ineffective in others, we believe it is more reasonable to compare the acceptance rates before and after an intervention rather than evaluating the model on the specified rate. For clarity, we will update our prompt in the paper to remove the 1% number.

Simpler ablations such as counterfactual data augmentation needs to be presented so that a fair-comparison to in-processing methods can be made.

One limitation of data augmentation and fine-tuning compared to our approach is that we lack the groundtruth label in how a model should decide on an applicant’s profile. For example, given the same qualifications, if two races are accepted and two are rejected, what should the gold label be? Generally, we do not want to impose arbitrary decision boundaries on the model since the Admissions task contains many different universities, each of which might value different aspects in a profile.