More of the Same: Persistent Representational Harms Under Increased Representation

We thank the reviewer for their incredibly detailed review and insight provided! We appreciate all of the questions and feedback provided!!

Our paper’s main contributions are the development of an evaluation methodology to surface representational differences in generated text between groups without specifying group membership in the prompt, and an empirical analysis utilizing this evaluation methodology in the gender and occupation domain, where we demonstrate that gendered associations persist in biographies and personas generated by prompts not specifying gender and these associations correspond to stereotypes and harms outlined in the social science literature. Our evaluation methodology is as follows: 1) generate text, 2) associate each generation with a group, 3) statistically test whether differences between groups are significant, 4) analyze meaningful words for patterns and potential harms, grounded in social science scholarship. To more clearly communicate this, we will add a figure outlining this evaluation methodology and add the following to the beginning of our methodology section:

We develop an evaluation methodology to surface representational differences in how groups are represented in generated text displayed in Figure 1. First, we generate text both with and without specifying a group in the prompt. Second, we associate each generation with a group. Third, we statistically test whether differences between groups are significant. We then analyze these meaningful differences for patterns and potential harms grounded in social science scholarship.

Q1

We agree with the reviewer that it may be expected that women-associated words persist in biographies about women. However, our goal is not to simply show such associations persist. Instead, our empirical contribution is to identify statistically significant words that differentiate how men and women are represented in specific occupational contexts—where the relevant associations are not obvious and often differ from generic gender markers (e.g., ‘women’-‘mother’ association). For example, ‘ambitions,’ ‘accomplishments,’ and ‘countless’ are statistically significant words for female software engineers, but only ‘countless’ is statistically significant in both generations where gender is specified and associated. Similarly, the word ‘passion’ is statistically significant for women in many occupations but not broadly in other domains, suggesting occupation-specific representational differences. These nuanced representational differences are essential for understanding how biases persist and proliferate in downstream tasks.

We also emphasize our methodological contribution: an evaluation framework that identifies representational differences between groups in a context-dependent manner. This enables researchers and practitioners to perform a more nuanced, statistically grounded analysis of how different groups are characterized. As LLMs are increasingly used across domains, our method offers a practical tool for identifying and measuring representational biases in generated text.

Finally, the papers cited by the reviewer are important but differ in scope. While both examine gender bias in unprompted contexts, neither investigates how different groups are represented. One paper shows that male bias persists in generated medical cases but does not explore representational differences between men and women or offer a method to analyze them. The other examines gender bias in cover letters across French and Italian but focuses primarily on the frequency of gender markers, without assessing broader patterns in how men and women are described. Moreover, its approach may not generalize well to languages with fewer explicit gender markers. In contrast, our work directly analyzes representational differences and introduces a generalizable methodology for surfacing these patterns—addressing a critical gap in the literature.

Q2

The Calibrated Marked Words method is an instance of the Fightin’ Words method in Monroe et al., using a prior that is a mixture of in-distribution context-specific text (i.e., language model-generated biographies), and a corpus representative of English language (see Section 3.1 and A.2 for details). We found this prior was more effective at not including common English words from our list of words with statistically significant differences in usage, while negligibly affecting other (context-specific) statistically significant words, compared to the prior used in the Marked Words method of Cheng et al (which only used in-distribution text as the prior). Other than selecting the hyperparameter governing mixture weights, there is no additional manual removal of common words from our significant words list, and we directly follow the Fightin’ Words methodology.

As described in Section 3.3, Chamfer-based SRBS solves the following problem: given a list of words C, and two lists of words T1 and T2, measure how much more similar C is to T1 than to T2. We note that in our setting, all of C, T1, and T2 have already been selected as statistically significant in their respective contexts (i.e., C is from generated personas with inferred genders, T1 is specified male, and T2 is specified female). We use the Chamfer distance to account for simple invariances in the word lists, e.g., synonyms should not be considered different.

We would like to emphasize that other methods in the literature address fundamentally different problems, and are not suited for our task. For instance, the GenBit method solves the following problem: given two lists of words T1 and T2, and a co-occurence matrix, assign each word a score based on how much more it co-occurs with T1 than T2. Importantly, in our setting, T1 and T2 arise from different distributions than C (i.e., use different prompts), so a co-occurrence matrix between the two lists is not meaningful, as it would cross significant words / their meanings from two different contexts. Moreover, GenBit does not appear to encode semantic invariances such as synonyms, beyond basic lemmatization.

Reviewer 2KRr mentioned the WEAT method, but it does not address our goal of comparing differences in similarities between lists. WEAT adapts the Implicit Association Test to evaluate bias in word embeddings and requires carefully curated word sets to be meaningful (e.g., male/female names vs. career/family terms). WEAT relies on the mean cosine distance. The choice of mean of cosine distance is arguably appropriate for WEAT because the lists of attributes are carefully curated such that each list of attributes refers to a particular category (e.g. family). While we could use mean cosine distance (and doing so would not affect the novelty of our proposed contribution), we believe Chamfer distance is much preferable because the interpretation and value of the mean cosine distance is less clear when the lists are not curated and may differ from one another in a multitude of ways.

Q3

Our primary historical comparison is to older models (GPT-3 and earlier), which prior work has shown to exhibit strong gender bias [3] (see L 80–89 & 244–246). While we also compare GPT-3.5 and GPT-4o-mini to show recent shifts, the main contrast is between current models and earlier generations. We’ll clarify this with an added sentence.

Q4

Our findings align with prior work showing a shift toward female preference in GPT-3.5 and GPT-4o-mini. For example, [1] finds these models favor female candidates over male ones in hiring tasks, consistent with our observation that generated personas are more often women. However, other work shows stereotypical gender biases—e.g., GPT-3.5 assigns higher salaries to male candidates [2]. These results are not contradictory, as they involve different tasks, and increased female representation does not imply bias has been eliminated. Notably, [1,2] use gendered names in their prompts, while our analysis does not.

Q5

Our gender association method is meant to be a lightweight component of our pipeline, and we evaluated its accuracy in Appendix A.1. We did not mean to claim that it is a significant technical innovation, and due to its simplicity, it is quite similar to many other gender inference methods – we will add text clarifying this in a revision.

Q6

We appreciate the reviewer highlighting the importance of statistical significance. All words discussed in Section 5.3 are statistically significant identified with the Calibrated Marked Words method selected using p-values <0.05. Differences in SRBS between genders are also statistically significant, as confirmed by t-tests with p-values well below 0.05 across models (see Table 9 and Figure 2). However, we had not reported statistical significance for differences in Figure 3 between GPT-3.5 and GPT-4o-mini. We have now run a t-test on SRBS scores conditioned on gender and found the differences to be statistically significant. We will add this result to the appendix and note it in the main text.

We found the standard error for female representation in each occupation was consistently below 0.01 across all models, allowing us to estimate percentages to the nearest point with high confidence. For transparency, we will add a table of standard errors for each occupation-model pair to the appendix and note this in the main text.

Q7

We do not discuss new stereotypes in our analysis, as we are not defending the validity of the stereotypes and harmful patterns surfaced, but rather are grounded in existing social science work (L 318-331) that point to the harm of these stereotypes and patterns.

Misc improvement/Weakness 5: We thank the reviewer for identifying text improvements and will address in a revision.

[1] Zhang et al. 2025. Hire Me or Not?...

[2] Nghiem et al. 2024. "You Gotta be a Doctor, Lin": An Investigation...

[3] Kirk et al. 2021. Bias out-of-the-box...

We thank the reviewer for their review and appreciate that the reviewer found the work provides “novel methodological contributions,” has a “rigorous empirical design,” and discusses “critical social implications.”

U.S.-centric bias: Occupational gender distributions sourced from U.S. BLS, ignoring cultural variability.

We do acknowledge this as a limitation in lines 372-374.

Non-binary exclusion: <10% representation prevents meaningful analysis (acknowledged but unaddressed).

We do discuss non-binary representation in lines 208-210. Detailed figures are provided in Figure 6 in the appendix. Further analysis is minimal on non-binary representation due to the lack of text generated that can be associated with the non-binary gender identity.

Pronoun dependency: Gender Association Method may miss cultural/linguistic nuances (e.g., gender-neutral language).

The Gender Association Method is tailored to our context, and although pronoun reliance is limiting, it is sufficient for our use case as demonstrated by the accuracy metrics of this method in Table 1 in the Appendix.

Subjective cluster interpretation: Stereotype labels (e.g., "passion," "excellence") applied post hoc to word clusters lack inter-rater reliability metrics.

To mitigate the issue of subjective cluster labels, in a revision, we will replace the bar graphs with a table containing the actual clusters and their prevalence for each model and gender pair. An excerpt of the table is provided below:

Speculates about "bias mitigation interventions" but lacks access to model training data/RLAIF details to verify causality.

As the reviewer pointed out, we “lack access to model training data/RLAIF details,” thus we are unable to explain why this shift in representation occurred. Thus, we join the call for greater transparency for model training, post-training methods, and bias-mitigation methods used to better understand and anticipate unintentional impacts of these methods.

No ablation on prompt phrasing impact (e.g., "biography" vs. "persona").

We do provide gender distribution breakdowns by prompt type in Figure 5 of the Appendix.

Groups all stereotypes under "representational harm" without severity gradation (e.g., "empathy" vs. "subservience").

Many representational harm taxonomies have been developed [1,2,3]. The purpose of our paper is not to develop another representational harm taxonomy, but rather, to demonstrate that some of the differences surfaced between generations associated between women and men are associated with representational harms and stereotypes. As it is outside the focus of our paper, we do not explicitly distinguish between severity of stereotypes and representational harms.

Neglects intersectional biases (e.g., race/class + gender).

Our application of the evaluation methodology does not account for intersectional biases, but rather provides a detailed analysis for gender. We agree this is important work, and leave future work to extrapolate our evaluation methodology to intersectional groups and to develop a systematic contextually-aware algorithm for associating race/ethnicity, class, or other groups with generations.

[1] Chien & David. 2024. Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigation

[2] Katzman et al. 2023. Taxonomizing and measuring representational harms: A look at image tagging

[3] Shelby et al. 2023. Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction.

We thank the reviewer for their detailed review and insight, and especially appreciate how the reviewer pointed out that we “address a crucial and nuanced issue in AI safety” and our “proposed bias measurement pipeline is technically sound.”

What do your results imply for practitioners? Would interventions at the RLHF or fine-tuning stages be more effective in targeting “how” people are represented?

Our results highlight the importance of addressing how groups are represented—not just who is represented, as we demonstrated how gender representational biases associated with how women and men are represented persist despite increased representation of women. Practitioners can use our evaluation methodology to detect representational biases and assess the impact of mitigation efforts on representational biases. While RLHF or fine-tuning may help reduce biases in how people are represented, further research is needed to evaluate how impactful these techniques would be, which we leave to future work. It is also essential to consider potential unintended effects of these interventions and utilize methods to surface or evaluate for these effects and implement means to best address these effects.

Can your methods be applied to domains outside occupation modeling? For example, do you observe similar representational harms in education, family, or politics contexts? What about dealing with intersectional identities (e.g., race + gender)?

The general pipeline we propose applies to any situation where there is a protected group (e.g., women) and a non-protected group (e.g., men) in some context (e.g., generated biographies specific to an occupation), and we want to understand stereotypes specific to that context. In principle, the protected group could be selected based on gender, race, religion, politics, or any intersections thereof (provided it is sufficiently well-represented to obtain statistically meaningful conclusions), and the context does not need to be occupation-related. We chose to focus our paper on the specific protected group/context combination that we selected because we felt it was sufficiently demonstrative of our problem setting and methodology, and performing a thorough study of this particular case already merited a somewhat lengthy investigation. However, we believe it is a promising future direction to apply our techniques to other settings of interest, potentially augmenting our pipeline with more task-specific tools in case there are challenges that may arise outside of gender and biographies. To assist in utilizing our evaluation methodology in other contexts, the code we release will be modular and easy to use outside of the gender occupation domain of our empirical analysis.

How is your Subset Representational Bias Score substantially different or more effective than other existing bias quantification tools, such as WEAT, StereoSet, or Marked Personas alone?

As described in Section 3.3, Chamfer-based SRBS solves the following problem: given a list of words C, and two lists of words T1 and T2, measure how much more similar C is to T1 than to T2. We note that in our setting, all of C, T1, and T2 have already been selected as statistically significant in their respective contexts (i.e., C is from generated personas with inferred genders, T1 is specified male, and T2 is specified female). We use the Chamfer distance to account for simple invariances in the word lists, e.g., synonyms should not be considered different.

While WEAT has a similar goal to our SRBS evaluation method, it applies in a very different setting. WEAT modifies the human psychology Implicit Association Test (IAT) to evaluate biases in word embeddings. As such, it requires the lists of words to have particular characteristics that lend the test its meaning, making it much less flexible than our approach. For example, the gender-career IAT “categorizes typical male names and typical female names as well as words associated with career and family” [5]. In terms of the approach to measure the distance between words, WEAT relies on the mean cosine distance. The choice of mean of cosine distance is arguably appropriate for WEAT because the lists of attributes are carefully curated such that each list of attributes refers to a particular category (e.g., family). While we could indeed use mean cosine distance (and doing so would not affect the novelty of our proposed contribution), we believe Chamfer distance is much preferable because the interpretation and value of the mean cosine distance is less clear when the lists are not curated and may differ from one another in a multitude of ways.

To give an illustrative example, the mean cosine distance used by WEAT applies best when the female word list has a coherent theme (e.g., {“she”, “mother”, “mrs.”}), and there exists a list of male words at the opposite extreme (e.g., {“he”, “father”, “mr.”}) However, in our setting, not only are the female stereotype words less coherent semantically (e.g., {“passionate”, “resilience”, “empathy”}) due to their context-specific usage, it is also unlikely that the antonyms of these words are representative of male biographies – just because women are described as passionate does not imply that men are described as dispassionate.

On the other hand, StereoSet (more concretely, the CAT method in StereoSet) and Marked Personas are solving fundamentally different problems than the one addressed by our SRBS metric. CAT requires the specification of stereotype and anti-stereotype words (typically crowdsourced), and evaluates language models based on their preference of word. This is a different setting than we study; we do not provide any stereotype at all when prompting the language model, and automatically produce stereotypical word lists based on a statistical test (i.e., our Calibrated Marked Personas method, an earlier stage of our pipeline before applying the SRBS metric). Marked Words is a simpler variant of our Calibrated Marked Words method (which again, occurs at a different stage of our pipeline than the SRBS evaluation, and hence addresses a different problem of producing stereotypical word lists rather than evaluating their similarities); the difference lies entirely in the prior, see our response to Reviewer keQ9, Question 2 for details.

There is little discussion about how findings might guide practical LLM development beyond “do better.”

We agree that diagnosing the root causes of these issues is challenging, in part due to limited transparency from model developers. Without access to training data, bias mitigation methods, fine-tuning practices, etc., it is difficult to understand which aspects related to model development have impacted model biases as well as how interactions between these aspects may have affected model biases. This makes providing more targeted recommendations to practitioners challenging. In response, our work offers a scalable evaluation methodology that practitioners can use to surface and track representational harms—an important step towards aiding practitioners in understanding whether representational biases persist after methods and techniques to address biases have been utilized. We also join ongoing calls for greater transparency to enable deeper diagnostic work [1].

Limited originality in framing: The underlying insight—that improving demographic parity (e.g., 50/50 gender distribution) doesn't eliminate harmful stereotypes—is widely discussed in fairness and safety literature.

We agree that the limitations of demographic parity have been well-documented in the fairness and safety literature [2, 4]. However, to our knowledge, prior work has not examined how increased representation—particularly achieving or exceeding demographic parity—fails to mitigate how groups are represented in generative models. While some work has addressed the persistence of stereotypes in text-to-image models despite mitigation efforts [3], we are not aware of studies that demonstrate stereotype persistence under conditions of increased representation. Our findings show that for the majority of occupations analyzed, women are represented at or above 50%, yet gendered representational biases persist. If there is relevant literature we have missed, we would be grateful to be directed to it.

What does "English language" in line 153 refer to? What does "topic prior" in line 663 refer to? What dataset is it?

The “English language” prior refers to the prior containing the distribution of words in the English language. The “topic prior” refers to the prior containing the distribution of words in the topic, which, in our case, is the generated biographies and personas. We will clarify this in Appendix A.2.

[1] Cheng et al. 2023. Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models

[2] Kenfack et al. 2024. A Survey on Fairness Without Demographics

[3] Bianchi et al. 2023. Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale

[4] Mouzannar et al. 2020. From Fair Decision Making To Social Equality

[5] Project Implicit Harvard

Thank you for the clarification. I will maintain my score because I still have some concerns about the weaknesses mentionedabove.

We thank the reviewer for their thoughtful engagement with our work and for recognizing the importance of studying who is represented within generative AI and how they are represented. We are especially grateful for the reviewer's appreciation of our methodology and the importance of this work.

overall the analysis and approach fails to dive deep into the question of generated image which in fact uses text to generate an image vs text to text generation

We agree with the reviewer that previous work has investigated gender occupation bias in image generation. However, the task our paper focuses on is surfacing representational differences in text generation, which faces challenges distinct to those posed by images. In the context of images, researchers first associate an image with a gender using approaches such as gender classification models or using a VQA model to caption an image, as in [1]. With images, features related to the presentation of gender are prominent. Such features include hair length, facial features, makeup, and so on. In contrast, with generated biographies and personas, identifiers associated with gender are less prominent. In text, discussion of an individual’s appearance is often not included and primary gender markers are the pronouns used. Furthermore, studies on TTI, such as [1,2,3,4], focus on the distribution level representation but do not focus on differing features related to how groups are represented. Thus, although we agree work in image generation has demonstrated gender biases in occupation, our paper surfaces representational biases in how women and men are represented of which limited work exists within image generation, and we develop an evaluation methodology for surfacing these representational biases.

Would be nice if the authors add more context in how such representational harms harm more women even when the occupations described here have more representation.

Representational harms may disproportionately affect women because they are now more frequently generated in model outputs. As representation increases, so does exposure to harmful stereotypes—e.g., if a biased portrayal appears in 90% of generated female biographies instead of 30%, the overall harm increases. Furthermore, the majority of stereotypes and harmful patterns analyzed related to women rather than men, as fewer statistically significant words for male generations were associated with stereotypes and representational harms. We'll expand upon these points within the paper to provide greater context. 

[1] Luccioni et al. 2023.Stable Bias: Evaluating Societal Representations in Diffusion Models

[2] Naik & Nushi. 2023. Social Biases through the Text-to-Image Generation Lens

[3] Cho et al. 2023. DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models

[4] Wan & Chang. 2025. The Male CEO and the Female Assistant: Evaluation and Mitigation of Gender Biases in Text-To-Image Generation of Dual Subjects