Towards Effective Discrimination Testing for Generative AI

You mention liability (and accountability) a few times throughout the paper - do you have any thoughts on how this intertwines with the legal landscape? As stated, this is a challenge particularly in GenAI, could be useful to discuss this a bit more, or reference discussions on it.

Liability in AI systems is particularly complex because the development and deployment processes are often separate. Developers create the systems, while users or deployers integrate them into real-world applications, often with limited understanding of the underlying mechanics or data. We attempt to summarize some discussion of these points here in the response, but provide more discussion in Appendix B.

Historically, discrimination law has primarily focused on the entities using or deploying systems, holding them accountable for discriminatory outcomes and decisions. In contrast, other legal frameworks, such as product liability, have centered on developers or manufacturers of products. For AI systems, and particularly for GenAI, the emerging approach is to distribute liability across both developers and deployers, sometimes with different requirements. For instance, the EU AI Act includes provisions that apply to both developers and users of GenAI/AI systems– for example, Article 52 outlines requirements for general-purpose AI providers to conduct risk assessments, implement mitigation measures, and ensure transparency, regardless of the specific application for which the AI is eventually used, and Article 29 requires deployers of all high-risk AI (not only GenAI) systems to monitor the operation of their systems. It is worth noting that the proposed EU AI Liability Directive, which is under negotiation, leans more heavily toward addressing developer accountability, particularly where defects in the system’s design or training contribute to harm. However, the Directive does not exclude users from liability when users directly violate discrimination laws.

In the U.S., liability for discriminatory outputs of GenAI systems is typically addressed through a patchwork of domain-specific laws, which apply in contexts like employment, lending, or housing. These laws generally hold users or deployers responsible for discriminatory practices, regardless of whether those practices result from an AI system. However, recent litigation highlights the evolving application of anti-discrimination law to AI technologies that shifts more burden to the developer. In a notable case, the U.S. Equal Employment Opportunity Commission (EEOC) supported a lawsuit against Workday, a developer—not a deployer—of an AI system, alleging that its AI-powered job application screening tools disproportionately disqualified candidates based on race, age, and disability. A federal judge allowed the proposed class-action lawsuit to proceed, emphasizing that Workday’s tools could be viewed as performing tasks traditionally associated with employers and were therefore subject to federal anti-discrimination laws. This case illustrates that developers can face liability, and it highlights the often-blurred lines between developers and deployers. Similarly, New York City’s AI bias audit requirement for hiring tools (Local Law 144) places obligations on deployers to audit and disclose information about tools they may not have developed. Our analysis provides yet another reason to not view this distinction as straightforward, given that harm can arise from a user’s specific implementation or customization of the AI system. Importantly, this is an ongoing issue and, as we note in the paper, AI researchers can add to the conversation by creating more ways for developer-deployer collaboration on discrimination testing.

I would have found the first case study much more compelling if it were using some of the state of the art benchmarks for LLM fairness. Moreover, I found it a bit suspicious that the authors used Llama 2 for the first case study, but not for the later ones. This made me question whether that model (together with the others) was cherry-picked to make the point they were trying to make – that's fine (because ultimately you're just trying to show that this kind of phenomenon can happen), but it does detract from the impact of your point, as it makes one question how common this kind of phenomenon actually is.

We do agree with the reviewer that it would be interesting to explore whether other technical notions of fairness might be correlated or not with these context-dependent outcomes, but we believe such studies are outside the scope of our paper. However, as we argue and demonstrate throughout the paper, it cannot be taken for granted that the latest research techniques are guaranteed to achieve specific real-world outcomes.

Regarding the inclusion of Llama-2 in the first experiment, we would note that it offers the best summary quality according to ROUGE among a set of similarly sized models, and thus seemed reasonable for inclusion in this example. In recognition of this concern of experiment consistency, we have added Llama-2 to the red teaming experiment.

Moreover, the first case study suggests to use alternate fairness metrics, showing that they are more successful. This feels a bit to me like re-inventing the wheel, as there are already other evals which are specifically tailored for measuring fairness, and my guess would be that such evaluation scores are already more correlated with actual fairness of outcomes in the setting that is tested. Another smaller qualm with the setup is that there were no error bars reported in Figure 2, and the number of resumes generated was quite small, when considering that they are AI generated, so the additional generation costs are minimal. This makes it hard to assess the significance of results.

As noted above, we do agree with the reviewer that it would be interesting to explore whether other technical notions of fairness might be correlated or not with these context-dependent outcomes, but we believe such studies are outside the scope of our paper.

We recognize that the size of our dataset is relatively small; however, we note that all resumes were summarized 4 times (with different names) by each of 5 models, and then each of those were scored by a 70B parameter model, meaning that the overall compute costs for this experiment (or those across the whole paper) are not low (at least with our academic GPU resources).

In response to the concern regarding error bars, we have added standard error bars to plots wherever possible (Figures 2 (left), 4, 5, 6, 8).

Despite the second case study being well executed, the takeaways are a little underwhelming – as mentioned above, I think they are already the standard practice for any entity that knows what they're doing.

We feel that it may not be fair to assume that any organization with sufficiently advanced ML researchers could anticipate all of these or other undemonstrated shortcomings of existing fairness techniques. However, even if this were the case, we hope that the techniques themselves might in the future be built to meet the needs of deploying organizations without such personnel, of which there will likely be many.

I had trouble understanding the third case study, especially the experimental setup. How is the interaction history created exactly? Do you have an example of what an interaction looks like somewhere?

We have updated the main paper and appendix to clarify the details of the multi-turn experiment, including an example of an interaction history in the appendix (Table 9). To clarify, an interaction history is created from a set of (input query, LLM response) pairs, where the input query comes from the GSM8K or MedQuad dataset and the LLM response is generated by Gemma-2-9B-instruct.

(3) I am also concerned that Llama is not a good proxy for this setting of resume evaluations per se. There’s a difference between summarizing the content of the resume and grading it. The description in the appendix includes the line, “In order to simulate interview decisions, we prompt Llama-3-70B to score each candidate 1-10 based on the summary of their resume, where a score of 9 or greater results in an interview.” Summaries from language models make sense for LLM applications, but grading criteria is more likely to be applied on top of these summaries as either a classifier or some other learned (specialized) tool…While the authors explicitly state that realism is not their focus, the descriptions of these experiments as they stand are not sufficient, and further detail is needed.

In response to concerns about the LLM decision maker, we have performed new experiments in order to understand whether the LLM decision maker itself is the source of the bias in the selection rates shown in Figure 2. To do so, resumes are summarized without an applicant’s name by Llama-2-7B, and then fed to the decision maker with stereotypical names from each of 4 groups. Selection rate results are shown in Figure 8, showing the decision maker to be significantly less biased when Llama-2-7B produces race-blind summaries, demonstrating that the main source of discrimination is likely the summarization model.

Also, with respect to the appropriateness of simulating these decisions with Llama-3-70B, we emphasize that our experiment is not meant to be a high-fidelity simulation of a real world application, but instead to demonstrate a core tension between GenAI bias evaluation and discrimination regulation. We also note that using generative models for evaluation is an increasingly common paradigm. Overall, we believe the inclusion of the LLM decision-maker facilitates a more complete illustration of the issue at hand.

Experimental Setup for E2 This experiment involves using a RedLM (there are apparently 7) to generate question templates. The blanks in these templates are then filled (unclear how) and fed into the LLM of interest. In this way, the LLM of interest is then evaluated by its responses in aggregate, which are measured for toxicity etc. I think the experiment is being compared to a baseline measure of fairness/bias, but the baseline is not described. In other words, I do not understand how this experiment is intended to show that variability is a concern ( though I buy the premise). Further, there are two instances where the RedLM and the LLM of interest are the same... is there a reason why it's okay here to generate and then evaluate using said content? I don't know that in this specific case it is too much of an issue, but it is odd from an empirical perspective.

We would like to clarify specific points about the red teaming experiment for the reviewer. In this case study, we are highlighting how red teaming, as prescribed by an increasing number of policy documents, might produce variable results, where the appearance of discrimination in red teaming is highly sensitive to the choice of red team (or underlying technique, model, hyperparameters, etc.). The idea is that a deploying organization will have a set of models that they consider, and want to show regulators or auditors that among these models, they have chosen one that is relatively non-discriminatory. We examine how degrees of freedom available to a red team using existing, highly-cited attack frameworks may enable them to produce a report with arbitrary fairness results, intentionally or not. Though anecdotal, we have been told by members of a sophisticated industry research lab that they are encountering difficulty in reproducing and stabilizing the results of a recently-published, high-cited attack framework. We thus feel that showcasing this in our work is a meaningful contribution to the bias testing literature.

With respect to generating and evaluating from the same model, this was the case in the original Perez 2022 paper. Also, we think it is reasonable that attacks might come from a different model, for example if a set of attacks is produced once and reused over time. Thus we consider both scenarios. In response to this concern, we have added clarification of this point in writing in Section 4.2.

Thank you to the reviewer for considering our rebuttals. We are glad that we were able to respond to many of your concerns, and that you found our responses helpful. We appreciate the concern regarding the amount of contents in the appendix, and would make an effort to use the space in the main paper more effectively before a camera-ready. Also, we politely note that Figure 8, with the probing experiment results, does have error bars. We recognize that they may be thin and difficult to notice, and we will make them more visible in an updated version of the paper.

We agree with the reviewer that the strength of the paper is its coverage of important recent regulatory discussion motivating the work, and that it touches on a number of important questions. We also agree that these case studies could be extended into standalone research directions, and we are excited by that promise. However, the goal of our work is to connect technical phenomena in generative AI with anti-discrimination regulation, a cornerstone of societal efforts to address unfair practices in important domains like medicine, housing, lending, and education. By addressing generative AI discrimination risks proactively, our paper fills a critical gap at a pivotal moment in the history of this technology.

We argue that as the development and enforcement of emerging GenAI frameworks increasingly relies on technical expertise, aligning technical and regulatory considerations is essential for society to achieve effective outcomes. We believe that our work is an important first step in achieving this alignment, and that providing ML researchers with the knowledge necessary to support real anti-discrimination efforts in their work is a more important goal than individual research efforts along one or more of the axes we highlight.

We thank the reviewer again for the time taken in reviewing our paper, and hope that they may consider our work in light of the potential larger societal impact, as opposed to its potential to produce incremental technical progress.