Probing Social Bias in Labor Market Text Generation by ChatGPT: A Masked Language Model Approach

Thanks for the detailed and thorough comments. Below please find our responses to specific comments:

---

W1: We acknowledge and face challenges in evaluating social bias in text:

1. The evaluation benchmarks in this field are scarce and underdeveloped so we can only turn to human labeling. This is also the key motivation for developing our method, which aims to provide a reference bias score for a field lacking domain-specific labeled data.
2. Compared to related papers in this field, our evaluation is more robust. Many existing evaluations only rely on heuristics, such as counting gendered words like “he/she” [3,8].
3. Evaluating gender bias is particularly challenging due to its subtle and multifaceted nature. This complexity makes it difficult to use crowdsourcing effectively, leading us to rely on expert labeling. Our paper significantly advances existing research by providing expert validation across multiple dimensions of gendered language in the labor market context. While not perfect, our approach represents the first of its kind to provide such comprehensive expert validation, addressing a critical gap in the literature.

Specifically, we have used the following strategies for validation:

1. For human evaluation, we work with experts in related fields (sociology, psychology, management, human resources), with an iterative multi-round coding process. This ensures high labeling quality, especially compared to non-expert crowdsourcing workers (who may be less attuned to the intricacies of labor market dynamics). We have provided more details about the expert labeling process in the global response.
2. In addition to human evaluation, we also conducted benchmark validation. The task has more than 250,000 examples and the dataset is from a different domain about classifying biography.
3. Bias is inherently contextual; what is biased in one context may not be in another. To ensure accuracy and relevance, we focused specifically on the labor market, going beyond generic language use that may not apply well to labor market processes.

Finally, the main goal of this paper is to evaluate the bias within LLMs. Please refer to our response to your next question for additional details.

---

W2: This is a really good and difficult question.

First, we want to emphasize that our final goal is to assess LLM bias, i.e., we are investigating the change in the score between input text and output text. By using the same MLM for input text and output text, which allow us to isolate and measure the additional bias introduced by the LLM during the text generation process, thereby controlling for the baseline biases present in the MLM used for evaluation.

Additionally, we conducted Control Experiments with Different MLMs, we selected two more MLMs: BERT-large and DistilBERT. The results for both the human validation score and the LLM correlation plots are provided in the additional PDF. These results demonstrate that changing the MLMs produces consistent outcomes in the scatter density plot for evaluating bias for LLM, and statistical testing results are also highly consistent. Also in lines 507-510, the use of the ranking method helps mitigate bias within MLMs.

Indeed, the bias for each MLM affects the bias score calculation, but we would like to highlight that related research, such as using word embeddings for bias score[3], also faces similar questions regarding the bias inherent in the embeddings themselves. This is a big challenge in the field, but we do our best to conduct control experiments and validation methods to mitigate this issue.

---

W3: We include baselines in the benchmark validation (see Lines 210-218 and 482-490). One baseline uses pure counting while the other uses static word embedding (word2vec). Our method is the first of its kind that uses contextual information with an MLM, i.e., the information of the whole sentence, other than looking at each word individually. From Figure 4b in the paper, we can observe a better performance compared with other baselines.

---

W4: The full word list is the integration of the existing literature in sociology, specifically references[5,6,7] and we directly adopted them without modification. These word lists are already publicly available. And we will publish the wordlists and the corresponding code.

Below is the table for the length of each word list:

As well as some job posting examples:

We would like to express our sincere gratitude to the reviewer for the time and effort in providing the comments and suggestions. Below, we address each of your points in detail.

---

W1: Thank you for your question, which we address in three aspects:

1. Use Case: In our settings, we used pre-validated word lists that are widely used in existing research. Our method aims to provide a biased score based on words chosen by these lists that are grounded in expert knowledge of labor market processes across the disciplines of sociology, management/human resources, psychology, and linguistics. Therefore, we are confident in the validity and reliability of the widely used and social science expert-validated word lists.
2. Algorithmic Aspect: The design of our algorithm incorporates robustness measures. Specifically, after combining the two sets of probabilities, we only take the top alpha percentage for calculation.
3. Numerical Experiment: We have conducted additional experiments to assess the impact of the word-list quality. As shown in Figure 4(b) in the new PDF, the correlation with human labeling decreases as we remove more words from the word list. This additional analysis helps underscore the importance of a comprehensive and accurate word list for reliable bias evaluation results.

---

W2: This is an important point. We want to first explain that biases are contextual, i.e. some words are biased in one domain but not in another. So in order to be correct and precise, the word lists we use are grounded in multidisciplinary literature on labor market processes, which goes beyond generic language use that may not necessarily apply well to the labor market context. Our method is easily generalized to other domains by changing the word lists (to other domain-specific ones). Moreover, drawing on interdisciplinary collaboration, we believe our work is a good illustration of combining ML and social science, and we believe our paper fits well into the Neurips stream of ‘Machine learning for social sciences’.

---

W3: Thank you for pointing this out. We have fixed the issue you raised and moved the table titles so that they appear above the tables.

---

Q1: same as W1

---

Q2: In terms of the specific example you noted, established psychological methods and evidence shows that language associated with characteristics such as “soft” is perceived by readers as “feminine”, which tends to appeal to and resonate more with women rather than men. In the case of job posting and application, this would mean that (1) the use of feminine language (e.g., soft) in job postings tends to appeal more to women (than men) applicants; (2) the use of feminine language in job applications tends to be evaluated more favorably for a post stereotypically perceived to a feminine orientation. Therefore, our definition of bias is embedded in the broader social science and psychological literature on the “biased” impact of language use. We have clarified our approach to bias in our revised paper.

This question also pertains to the validity of the word list we use, which we clarify in three aspects:

1. The word lists are established, widely used, and validated in the literature, specifically those developed by [5,6,7]. They are theoretically grounded in social science research.
2. The word lists are specially informed by extensive sociological, psychological, and management research focusing on gender in the labor market.
3. Lastly, we conduct a rigorous internal validation process, achieving a high level of intercoder reliability with experts specializing in sociology, management, and labor market research.

---

Limitation: We appreciate that some may view this as a limitation, but we view this as a strength. A great deal of social science research in psychology, linguistics, and sociology has established the relationship between language and social bias. We are able to draw on the strengths of decades of research to inform innovations in ML/NLP. Therefore, we believe our paper fits well into the Neurips stream of ‘Machine learning for social sciences’ by illustrating the strengths of interdisciplinary cross-fertilization.

Thanks for the detailed and thorough comments. Below please find our responses to specific comments:

---

W1: the use of LLMs like ChatGPT for generating job applications is indeed increasingly common among job seekers. This trend has been highlighted in discussions and concerns raised by employers and HR professionals, as reported by high-profile news such as CNBC (May 6, 2024, article: "Exact same cover letters word for word: Career consultant says Gen Z are misusing AI") and Vox (Mar 8, 2023, article: "Maybe AI can finally kill the cover letter: Jobs still require cover letters. Apps like ChatGPT can help."). Moreover, the academic community is beginning to explore this use case, focusing both on its technical applications[1] and ethical/bias implications[2]. Our study contributes to this burgeoning body of literature by addressing key public, organizational, and scholarly concerns regarding the use of LLMs in generating job applications.

We explicitly identify the two major harms resulting from bias exacerbation in LLM-generated job applications:

1. Structural Harm: LLM-generated job applications that perpetuate gender stereotypes contribute to the reinforcement of gender inequalities embedded in language used in labor market processes. Job postings and applications are critical steps in these processes and play a significant role in the reproduction of gender inequalities. By failing to challenge these biases, LLMs can inadvertently support the perpetuation of these inequities.

2. Practical Harm: As gender-biased LLM-generated job applications become part of the training data for future AI applications in HR(both in generating job postings and assessing applications), there is a risk of further entrenching gender biases in language use. This entrenchment can lead to cascading effects, such as increased labor force gender segregation, which have significant societal implications.

---

W2: This is a really good question. We want to emphasize that our goal is to assess LLM bias, i.e., we are investigating the change in the score between input and output text. Using the same MLM for input and output text, allows us to isolate and measure the additional bias introduced by the LLM during the text generation process, thereby controlling for the baseline biases present in the MLM used for evaluation.

Additionally, we conducted Control Experiments with Different MLMs, we selected two more MLMs: BERT-large and DistilBERT. The results for both the human validation and the LLM correlation plots are in the additional PDF. These results show that changing the MLMs produces consistent outcomes in the scatter density plot for evaluating bias for LLM, and statistical testing results are also highly consistent.

Indeed, the bias for each MLM affects the bias score calculation, but we would like to highlight that related research, such as using word embeddings for bias score[3], also faces similar questions regarding the bias inherent in the embeddings themselves. This is a big challenge in the field, but we do our best to conduct control experiments and validation methods to mitigate this issue.

---

Q1: You are correct about our intended meaning, that words are stereotypically or historically associated with certain genders. We clarified that our word lists capture feminine and masculine orientations that are grounded in and associated with gender norms and stereotypes, as supported by extensive sociological, psychological, and linguistics research[4,5,6]. Notably, while some of this work is more historical in nature(evidence from decades ago), more recent research cited below also establishes the continuing importance of gendered language in contemporary settings.

---

Q2: We would like to clarify that the word lists used in our experiment were not created by us specifically for this paper. Instead, we directly adopted them from existing literature across the social science research as referenced in [5,6,7], without modification. These word lists are publicly available, and we will publish the word lists along with the corresponding code.

Using pre-validated word lists is a key advantage of our method, allowing us to easily incorporate established, rigorous, and theoretically grounded word inventories from social science research (see lines 132–136).

Additionally, this paper also draws on the expertise of several co-authors specializing in sociology, management, and labor market research to provide further internal validation of the word lists.

---

Q3: Rank has several advantages over probabilities:

Theoretical Properties: Ranks exhibit normality and allow for a rigorous formulation of the test statistic and its asymptotic result as presented in Theorem 1, which is based on the Wilcoxon rank sum test. Probabilities do not possess these important properties.

Sensitivity and Robustness: Direct use of probabilities can be overly sensitive due to the highly skewed nature of prediction distributions. By using ranks, we can mitigate this sensitivity, and enhance the robustness of our results. This robustness enhancement also helps mitigate the inherent biases present in these pre-trained models.

---

Q4: From a Bayesian view, when we lack prior information, the mean serves as a robust estimator of central tendency. However, we acknowledge that for future work, incorporating prior information for a weighted average or other aggregation methods could potentially improve performance.

The asymptotic distribution of S is normal. To validate this, we have included a histogram of S in the PDF. Additionally, we conducted the Shapiro-Wilk test, which resulted in a p-value of 1.00, confirming that S is indeed normally distributed

---

Q5: We employed the GPT-3.5 Turbo, as it was the most accessible and widely used free version at the time of our experiment. This choice was made to ensure that our results are representative of the typical user experience.

We are very grateful for the thoughtful comments and heartwarming acknowledgment, as well as the time and effort devoted to the reviewing process. Below please see our response to the specific point in the reviewing comment.

---

W1: Our experiments were specifically conducted using ChatGPT due to its wide accessibility and prevalent usage. This choice was deliberate to ensure that our findings are representative of the typical user experience with one of the most commonly used models. However, we acknowledge the importance of testing additional models, and we agree that future work should include experiments with a broader range of models to validate and extend our findings.

---

W2: This is a great suggestion, pointing out an important direction for future research, which we have now acknowledged in our revised paper. We focused on gender, as gender inequality in the labor market is well-documented, persistent, and associated with gendered language, making this a prominent and exemplary type of social bias. We were also motivated in part to include just one aspect of bias due to the limited space of the paper. Adding additional dimensions would require providing background on each form of inequality, as well as locating established word lists of linguistic cues associated with those biases. In our revision, we have more clearly articulated that gender biases are chosen as a focal case of a broader range of social biases and that gender bias is important because it underpins one of the most prominent and persisting forms of inequality in the labor market. We have also suggested that future research explore additional forms of bias and labor market inequality

---

Q1:Thank you for your question. Because we assign lower numerical ranks to higher probabilities. As illustrated in Figure 2 in the paper, a rank of 1 represents the highest probability. Therefore, a lower mean rank indicates higher probabilities, while higher ranks (larger numerical values) correspond to lower probabilities. We have clarified this in our revised paper to ensure the scoring logic is explicitly defined.

---

Q2: The validity of our word lists is grounded in in three aspects:

1. The word lists are established, widely used, and validated in the literature, specifically those developed by [5,6,7]. They are theoretically grounded in social science research.
2. The word lists are specially informed by extensive sociological, psychological, and management research focusing on gender in the labor market.
3. Lastly, we conduct a rigorous internal validation process, achieving a high level of intercoder reliability with experts specializing in sociology, management, and labor market research.:

---

Q3: Thank you for pointing this out. We will include this additional information in the appendix:

To ensure the scientific rigor of the evaluation, we paid particular attention to inter-rater validity and reliability. Specifically, each phase included individual labeling of data conducted by four independent experts specializing in labor market inequalities associated with gender, work, and family, and Equity, Diversity, and Inclusion (EDI) in the labor market. In each round of labeling, individual labeling was followed by group sharing and discussion of the preliminary outcomes among the four experts. This combination of individual and group analysis allowed the team to trace the score labeling and the validation of word inventory, ensure inter-rater reliability in each phase, and contextualize each phase within relevant scholarly literature, policies, and definitions.

The score labeling and the validation were further validated by three additional expert labelers (management, human resource, and social science scholars) from the team. Through a double-blind labeling approach, the three additional experts independently assessed the dimensions and labels produced by the first four experts, demonstrating a high level of consistency. The scores and word lists were then finalized through further deliberation among the four experts and the three additional validators.

Because we used an iterative multi-round coding process, the inter-coder consistency rate in the developmental coding varied between 0.6 and 0.8. Notably, the final validation by three fresh validators within the team achieved a high level of inter-coder consistency exceeding 0.8.