Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs

Thank you for recognizing the importance of this work. Please see our responses below to your specific questions and concerns.

---

Evaluation with other LLMs

Thanks for the suggestion to incorporate additional LLMs. We have expanded our analysis to include 3 additional models: 2 latest OpenAI models (GPT-4-Turbo and GPT3.5-Turbo Nov.) and 1 public model (Llama2-70b-chat). Upon evaluation across all 24 of our datasets, we found our findings to generalize to these models and the persona-induced biases to be prevalent across models (Figs. 12, 16, and 18). Thus, it is clearly not an issue with just ChatGPT or the OpenAI family of models, but is more fundamental to how the LLMs are currently developed. Please refer to the general response and newly added Appendix D for additional results and analysis.

---

Should Abstentions be penalized?

We want to clarify that ChatGPT primarily abstains by citing various incorrect (a physically disabled person would not be able to analyze legal situations) and stereotypical (a religious person does not possess any expertise in physics) presumptions about the socio-demographic personas it is emulating. The abstention examples in Table 3 and Appendix H demonstrate this troubling trend and clarify this key distinction from regular model refusals (e.g. ”I will not provide a bomb making recipe”). This undesirable behavior would be detrimental to specific groups, e.g. personalized assistants (LLMs with assigned personas aligned with the end-user) responding incorrectly or inappropriately.

---

Better exploration of the prompt space for bias mitigation?

We explored prompts that go from providing a nudge (“Try your best”) to strongly suggesting to answer a question (“Don’t refuse”) to even making each persona a subject expert. We have extended this set to include results for 8 additional de-biasing prompts (see the general response above). Our mitigation experiments now cover 13 debiasing prompts to more systematically cover the space of variations. We did not see any notable differences in mitigation in the results. Note that the focus of this work is to identify and characterize the reasoning biases in persona-assigned LLMs, and we are the first ones to do so to the best of our knowledge. We leave more sophisticated bias mitigation approaches such as aligning LLMs on our dataset of 1.5 Million released model predictions to future work.

---

Why do certain personas experience a larger performance drop?

This is a very interesting question. We believe that many factors are at play here. Firstly, modern language models go through multiple training stages like pretraining, instruction-tuning, alignment, etc. Biases within the datasets used at each stage can favor or disfavor certain socio-demographic groups (e.g. text snippets like ‘religious individuals lack interest in sciences,’ or ‘Jewish individuals possess high intelligence’), consequently shaping the induced biases in these models. Secondly, the alignment efforts guided by red-teaming for these LLMs might primarily target widely recognized harms and biases, potentially leaving other biases unaddressed, in turn contributing to some personas exhibiting larger performance drops.

Using open pre-training corpuses (e.g. C4, Dolma) one could in principle run ablations on the pre-training corpus to identify why certain personas have a much higher performance drop. E.g. If pre-training on a corpus excluding certain subreddits results in reducing the bias, we can identify that subreddit as a potential cause of this drop. However, in practice such ablations would be prohibitively expensive to run.

We will add this to the discussion section in the final version of the paper.

---

How are the “human” and “average human” baselines implemented?

To create these baselines, we prompt the LLM to adopt the persona of a “Human” and “Average Human” respectively and measure the resulting performance on our 24 dataset benchmark. Similar to the personas studied in this work, we simply replace the “{persona}” placeholder in persona instructions with “Human” or “Average Human”. We have clarified this in Section 3.1 of the updated paper (see the changes in blue).

---

Why sample from ChatGPT with temperature 0?

We use a temperature of 0 as it helps us quantify and characterize the bias from the most likely (and least stochastic) responses from these models.

---

Error bars and variation across model runs

Thank you for pointing this out. We have added Fig. 23 to Appendix F that shows the variation in performance across runs. We found that the standard deviation of the performance across runs (when aggregated over all 24 datasets) is very small (less than 0.006 and is thus barely visible in the figure) and thus doesn’t affect our findings. We have updated the paper to drop the “significant” terminology when discussing the variation across runs.

Thank you for recognizing the importance of this work and sharing your valuable feedback! Please see our responses below to your specific questions and concerns.

---

Would different models perform differently?

We have significantly expanded our analysis to include Llama2-70b-chat, GPT-4-Turbo, and GPT3.5-Turbo Nov. We find that these models do perform differently (compared to ChatGPT) both in the extent and pattern of the bias. For instance, we observe that the latest version of GPT3.5-Turbo exhibits significantly higher levels of bias, whereas GPT-4-Turbo has reduced levels of bias across personas. Additionally, we see a difference in the nature of the bias as well. For instance, compared to ChatGPT, there is a significant reduction in the amount of bias against Physically Disabled in Llama-2, whereas the amount of bias against the non-binary genders goes up in Llama-2.

---

Non-binary genders & Intersectional biases

Thanks for the suggestion to extend the scope of our study. We have added results for 3 non-binary gender personas and 13 compound personas. With these additions, our bias study now includes insights from 1.5 Million model predictions (more than 200 Million generated tokens) and spans 32 socio-demographic personas, 24 datasets, and 4 LLMs.

We found evidence of bias against non-binary genders across models (Figs. 5, 12, 16, 18). We additionally found that intersecting 2 personas with high levels of bias can at times reduce the overall amount of bias, whereas intersecting personas with high and low biases usually results in a level of bias that is in between the bias levels of the intersecting personas. Please refer to the general response for additional findings. The results for the non-binary genders are included in the main body of the updated paper (Figs. 2, 3, 4, 5), and the results and analysis for the compound personas are in the newly added Appendix E.

---

More sophisticated approaches to address the bias issue?

Thanks for the suggestion. We have extended our set of debiasing prompts and have added results for 8 additional prompts (see the general response above). Our explored prompts are stylistically and semantically diverse that range from providing a nudge (“Try your best”) to strongly instructing to answer a question (“Don’t refuse”) to even making each persona a subject expert. In total, we tried 12 de-biasing prompts exploring this space and did not see any notable differences. Note that the focus of our work is to identify and characterize the reasoning biases in persona-assigned LLMs, and to the best of our knowledge, ours is the first work to do so. We leave more sophisticated bias mitigation approaches such as aligning LLMs on our dataset of 1.5 Million released model outputs to future work.

---

More in-depth discussion; reason behind these observations.

Section 4 of our paper analyzed the model behavior and identified two key manifestations of the biased reasoning: explicit abstentions and implicit errors. We showed in Sec. 4 that while abstentions form a large percentage of the model errors (Fig. 8), the bias goes beyond abstentions and implicitly affects the model’s reasoning, leading it to make more mistakes for certain socio-demographics (Fig. 9).

For a more causal understanding of these observations, we can consider the model development process. Given the key sources of data in these models are the pre-training corpus, instruction-tuning datasets and the RLHF training data, we believe these are the key reasons for this observed bias. Biases within the datasets used at each stage can favor or disfavor certain socio-demographic groups (e.g. text snippets like ‘religious individuals lack interest in sciences,’ or ‘Jewish individuals possess high intelligence’), consequently shaping the induced biases in these models.

Given the access to these datasets (e.g. C4), one could possibly run ablations to identify the more specific cause. E.g. If pre-training on a corpus excluding certain subreddits results in reducing the bias, we can identify that subreddit as a potential reason behind these observations. However, such ablations would be pretty costly and not even feasible with GPT models where the entire model building process is private. We will add this to the discussion section in the final version of the paper.

---

Question: Why does the religious persona perform lower than each of the more fine-grained demographics?

It is hard to say for certain. The model’s opinions about a persona are not guaranteed to be consistent with the hypernym-homonym relations. We theorize that models pick up certain incorrect stereotypes about the religious people from the pre-training corpus (e.g. “religious people are not open-minded”, “religious people don’t accept sciences”) that are probably not that commonly expressed in written text for more fine-grained religions.

Thank you for taking the time to review our paper! Please see our responses below.  

---

Significance of the results

 

Significant bias for nearly all personas

• Figure 2 serves the purpose of demonstrating that there are disparities in performance across personas. Note that this figure plots the accuracy micro-averaged over all 24 of our datasets and thus even small differences in performance can be statistically significant. We explicitly mention in the text whenever the differences are not statistically significant (e.g. the note regarding differences not being statistically significant for Caucasian, Man, and Woman personas in Section 3.1).
• Our claim that “there is a significant bias inherent in LLMs for nearly all personas” can be substantiated through Figure 3 where we show that relative to the “Human” persona, all personas have at least one dataset where there is nearly a 10% drop in the performance. We think a drop of 10% in the model performance just by changing the socio-demographics of the assigned persona is significant and harmful for the unassuming end-users of these personas-assigned LLMs. We will include this clarification in the final version of the paper.
• We have included an additional plot in Appendix F (see Fig. 24) that complements Fig. 2 and 3 and plots the number of datasets with a statistically significant drop in performance for all personas relative to the baseline “Human” persona. We can see in this figure that 16 out of 19 personas have at least one dataset with a statistically significant drop in the performance.

---

Additional clarifications

• "accuracy being so low across the board .. doesn’t seem to be significant": Firstly, note that 65% is not a low score for MMLU tasks using zero-shot prompting. For example Fu et al ‘23 reported 5-shot scores for gpt-3.5-turbo as 67.3% (compared to our 65% with a zero-shot prompt). Secondly, we run statistical significance tests for all reported findings and clearly state when the differences are not statistically significant.

• "physically disabled and religious seem to have performance worse than a coin toss.": Note that the datasets from MMLU have 4 answer choices and thus the coin toss performance would be 25%. None of the personas report an aggregate score below 40%.

• "two bars between man and woman are almost indistinguishable and also very close to human persona": This is true, and we explicitly mention in Section 3.1 that the performance differences for Caucasian, Man, and Woman personas are not statistically significant.

 

---

Soundness of the analysis

 

Figure 3 and dataset-level comparisons

Figure 3 shows the distribution of dataset-level %change in performance for all personas and primarily serves the purpose of depicting the extent of the bias across personas (there is no direct comparison between the personas at the dataset granularity in this figure) e.g. Physically-Disabled, Religious, and Trump Supporter personas have datasets with 50%+ drop in the performance; nearly all datasets have at least one dataset with a 10% drop; the distribution of %change varies across personas. For dataset-level comparisons, please refer to Fig. 4 and Fig. 5 which provide insights for various persona groups (e.g. Politics, Race), as well as the newly added Fig. 24 (in App. F) that provides insights for individual personas.

---

Error bars

We have added Figure 23 to Appendix F to show the variation in performance across runs. We can see that the standard deviation of the performance across runs (when aggregated over all 24 datasets) is very small (less than 0.006 and is thus barely visible in the figure), and thus the variation across runs doesn’t affect our findings (hence the omission of error bars in the figures). We will add references to Figure 23 and Figure 24 in the final version of the paper.

---

Why switch from 'Accuracy' to error rate in Figure 8?

Thank you for bringing this up. As mentioned in Section 4, we switch from accuracy to error rate because at this point in the paper we have moved to analyzing the errors and quantifying the contribution of abstentions to the overall error. We have added an additional note about this to the caption of Figure 8 in the updated paper for further clarification (see the change in blue). We hope this clarifies the confusion. We scale the plot down to 0.6 as it is the highest error rate observed across personas.

Thank you for taking the time to review our paper! We are glad to hear that you found our study intriguing and our findings robust. Please see our responses to your questions and concerns below.

---

Analyses on other LLMs

Thanks for this suggestion. We have expanded our analysis to Llama2-70b-chat, GPT-4-Turbo, and GPT3.5-Turbo Nov. We observed that the persona-induced biases are not exclusive to GPT-3.5 and even extend beyond the OpenAI family of models. We additionally found the extent and nature of the bias to vary across the models. Please refer to the general response and the newly added Appendix D for additional results and findings.

---

How does intersectionality alter the results?

Thanks for this interesting question. We have added a systematic analysis of 13 compound personas that intersect the existing socio-demographic personas. We found that intersecting a low bias persona (e.g. ‘a man’) with a high bias persona (e.g. ‘a physically-disabled person’) does mitigate some of the bias in the high bias persona but not completely. Interestingly, we found that intersecting two low bias personas (e.g. ‘a physically-disabled person’ and ‘a religious person’) can at times achieve a lower overall bias compared to the participating personas. Please refer to the general response and the newly added Appendix E for more details.

---

Why not just prompt for "expert in the subject" or "domain expert”?, Why the need to explicitly state “Chemist”?

This is a very interesting suggestion. We experimented with 3 new (’domain expert’) prompts based on this suggestion and found them to have little to no effect. We also experimented with 5 other stylistically different debiasing prompts (see the general response above) and found them to have little effect as well (more details in Appendix C).

It's interesting that a generic instruction of “act like a domain expert” does not work but grounding the expertise to a task-specific occupation (e.g. Chemist) works so well. It is possible that the domain expert terminology still leaves room for the LLM to surface its biases (e.g., by assuming that different personas have different levels of domain expertise), whereas specifying an occupation potentially unifies the perceived capabilities across personas.

---

How does the LLM perform when no persona is provided?

This is a good question. We evaluated ChatGPT with a no persona prompt (i.e. only providing the task instruction) and found that it basically performs the same as the “Human” persona. E.g. With no persona, ChatGPT achieves an aggregate score of 0.66 on our 24 dataset benchmark as compared to the 0.65 score for the human persona (a non statistically significant difference). We also didn’t observe any statistically significant change on any of the datasets between these two personas. Thus, all of our bias findings hold relative to this ‘no persona’ baseline as well.

---

Be consistent with the abbreviation.

Thank you for pointing this out. We have fixed this and use a consistent abbreviation in the updated version of the paper.

---

Specify explicitly what the "human" and "average human" persona prompts are.

You are correct -- these are generated by replacing the {persona} placeholders in our instruction templates. We have clarified this in Section 3.1 of the updated paper (see changes in blue).