Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective

Thank you for acknowledging that FiSCo tackles an important blind spot. Yes, we will opensource our benchmark datasets for researchers to use. We address your concerns with the following clarifications:

Because the entailment judgements come from a single LLM, errors or biases in that checker could leak into FiSCo’s final verdict; an experiment with an alternative NLI model—or at least a discussion of this risk—would strengthen the claim of robustness.

Answer: Thank you for raising this important point. To assess the robustness of FiSCo to the choice of entailment checker, we evaluated our method using several alternative LLMs: Claude 3.0 Haiku (bedrock/anthropic.claude-3-haiku-20240307-v1:0), GPT-4o Mini (gpt-4o-mini-2024-07-18), and GPT-4.1 Nano (gpt-4.1-nano-2025-04-14).

We sampled 400 response pairs across our dataset and computed similarity scores as defined in Equation (1). We then computed the Spearman rank correlation between each model’s similarity scores and those from our baseline model (LLaMA3-70B: bedrock/meta.llama3-70b-instruct-v1:0). All models showed high consistency, with Spearman ρ > 0.5 and p value < 1e-26, indicating stable ranking behavior across models [1]. Additional validation results are provided in Tables 3 and 4 of RefChecker [2].

What was the inter-annotator agreement on the 4920 human-labelled pairs?

Answer: For the initial 4,920 annotated pairs, inter-annotator agreement yielded a Cohen’s κ of 0.395 (95% CI: [0.330, 0.461]), which is within the expected range for nuanced textual entailment tasks. As described in Appendix B, disagreements were reviewed by a verifier, and cases with labeling errors were re-annotated to ensure quality. We will clarify this verification step in the final version.

Did you try prompts that combine a gendered name and explicit pronouns, and if so do the measured gaps widen?

Answer: Yes, we conducted additional experiments where prompts were augmented with both gendered names and explicit pronouns. For example, the prompt: “I am 44 years old. My name is Abgaile. I grew up in Massachusetts and I am currently a Hotel Manager...” was rewritten as: “She is 44 years old. Her name is Abgaile. She grew up in Massachusetts and is currently a Hotel Manager...”

This change led to an increase in Claude 3 Sonnet’s FiSCo score from 0.05 to 0.07, indicating that including both names and pronouns can amplify measurable gender response differences.

[1] Akoglu H. User's guide to correlation coefficients. Turk J Emerg Med. 2018 Aug 7;18(3):91-93. doi: 10.1016/j.tjem.2018.08.001. PMID: 30191186; PMCID: PMC6107969.

[2] Hu, X., Ru, D., Qiu, L., Guo, Q., Zhang, T., Xu, Y., ... & Zhang, Z. (2024). RefChecker: Reference-based fine-grained hallucination checker and benchmark for large language models. arXiv preprint arXiv:2405.14486.

Names often bundle in extra information such as social class or geographic origin, so the measured “gender” or “race” effect may actually mix several biases. Have you considered adding non-name demographic cues (for example explicit age phrases or organisation names) to disentangle name-based confounds?

Answer: You're absolutely right — names can carry unintended social signals such as region or class. To mitigate this, we explicitly include non-name demographic cues in our prompts, such as age, geographic origin, and occupation (see Table 6). These attributes are held constant across group comparisons to isolate the effect of the group variable (e.g., gender or race), reducing name-based confounds.

The competing metrics are presented mainly as raw agreement numbers; adding confidence intervals or statistical tests for those baselines would make the comparison more convincing.

Answer: Thank you for the suggestion. In Tables 1 and 2, we report confidence intervals for agreement scores and perform paired t-tests comparing our method to baselines like Sentence-T5 and SimCSE. These analyses confirm that FiSCo’s performance improvements are statistically significant.

Although FiSCo flags problems, the paper does not explore how the score can steer mitigation, so the practical loop is not yet closed.

Answer: FiSCo is intended as a diagnostic tool that supports fairness mitigation. Once biased cases are identified, practitioners can use topic modeling or clustering to isolate high-risk themes (e.g., question types, topics, demographic contexts). This enables practical interventions such as:

(i) Excluding or refining prompts in high-risk applications

(ii) Augmenting training data with counterfactual or group-balanced examples

(iii) Adjusting prompt templates or response structures to reduce disparities

We will expand on this in the final version to highlight FiSCo’s value in supporting responsible AI as a whole.

Could FiSCo handle more than two groups at once by replacing Welch’s test with one-way ANOVA?

Answer: Excellent suggestion — one-way ANOVA is a natural extension of Welch’s test and would allow FiSCo to handle comparisons across multiple demographic groups. While our current implementation focuses on binary comparisons, we plan to incorporate ANOVA-based evaluation for multi-group fairness assessment in future work.

As the discussion phase progresses, I would be grateful if you could take a moment to consider our responses and let us know if there are any further clarifications or concerns we can address. We are hopeful that our clarifications help resolve the issues raised and would appreciate any additional feedback you might have.

Any reference for using name to guess gender or race? What is the failure rate that the first name does not correctly retrieve the gender or race information -- for example, gender-neutral first names: Alex, Dakota, Morgan, Taylor...

Answer: We follow the protocol used in the BOLD dataset [1], which employs first names as proxies for gender and race identification. In our experiments, we deliberately use only representative names with high demographic association to minimize misclassification risk. We exclude gender-neutral or racially ambiguous names (e.g., Alex, Taylor, Morgan) to ensure clarity in group assignments and reduce noise. This strategy allows us to isolate group-based behavior without confounding due to name ambiguity.

If the hyperparameters of the similarity should be tuned for different applications, how to tune the hyperparameters for less clear applications.

Answer: In our experiments, we use a fixed configuration of α=1, β=γ=0 to focus exclusively on shared entailments — a setting suited for high-stakes applications where consistency across groups is critical. We also conduct an ablation study (Appendix F.2) to assess the impact of varying β.

For real-world deployments, FiSCo's weighting scheme can be adapted to reflect the sensitivity of the application:

(i) High-stakes scenarios (e.g., hiring, healthcare, education): Use α=1, β=γ=0 to emphasize content consistency across demographic groups.

(ii) Moderate-stakes or content rewriting tasks (e.g., rewriting for tone/style): Set α=β=1, γ=0 to allow some variation while still preserving shared meaning.

(iii) Low-stakes domains (e.g., movie recommendations): Bias evaluation may be optional or more lenient.

When application stakes are unclear, we recommend using the stricter configuration (α=1) as a conservative default to reduce the risk of underestimating harmful disparities.

[1] Dhamala, Jwala, et al. "Bold: Dataset and metrics for measuring biases in open-ended language generation." Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 2021.

Why does the paper consider suitability: “Is ChatGPT appropriate for answering this question?” Based on the examples in appendix B.1.2., it seems unlikely that the models tested would have come up with unsuitable responses to the prompts defined in the appendix (given the specificity).

Answer: We performed the validation procedure to ensure that the questions themselves (not the responses) were a good representation of realistic use cases that LLMs would be expected to handle. As noted in Appendix B, we identified two types of unsuitable questions: (1) those that violate ethical, moral, or legal standards, and (2) those that are improbable, incoherent, or meaningless.

LLMs may still generate answers to such prompts, but these questions do not reflect realistic or appropriate user–LLM interactions. Evaluating model behavior on such inputs would dilute the signal we aim to measure with FiSCo.

Additionally, unspecific questions lacking necessary background information can also be unsuitable, as they do not allow for meaningful or grounded comparisons. This is illustrated in Table 9 (row 3), where the lack of context renders the prompt ineffective.

Is asking annotators “Is a human likely to ask this question?” a reasonable substitute to, for example, gathering questions from the WildChat dataset (Zhao et al.)? The latter may have better ecological validity for real-world user queries in the wild.

We synthesized questions that were likely to elicit bias-related behaviors from LLMs. To do this, we first adopted the Advice and Insight Generation templates to generate question topics. We then augmented these templates with demographic attributes (e.g., gender, race, age) as potential bias triggers.

Answer: After adding these attributes, we conducted a likelihood validation step, where annotators judged whether a human would realistically ask each question. This helped ensure that the final prompts were not only diverse and bias-relevant but also realistic and contextually plausible. As shown in Table 9 (row 1), we excluded questions where personal information was irrelevant to the query — i.e., prompts that no real user would plausibly ask.

We agree that the WildChat dataset is a valuable resource for collecting real-world user–LLM interactions and may serve as a strong starting point for question topic inspiration. However, once bias-triggering factors are inserted, human validation is still essential to ensure the resulting questions remain ecologically valid and contextually grounded. In our work, this two-stage process (template-driven synthesis + annotator validation) was necessary to maintain both bias relevance and user realism.

You raise an excellent point about the importance of synthetic dataset quality. We apologize for not including our sampled dataset in the supplementary materials. We will share the dataset upon the final decision of the paper to support a thorough review of question quality and to enable replication of our results.

To operationalize age as a bias variable, we included a statement such as "I am XX years old" in the prompt, while keeping all other contextual attributes the same (see Table 6). In our setup, we defined individuals above age 50 as ‘older’ and those between 20 and 50 as ‘younger’, and we generated an equal number of examples for both groups. We will add these details explicitly in the final version.

Regarding the figure suggestion: repositioning is a great idea. We initially planned to show the t-SNE figure in the introduction. However, we felt that explaining baseline comparison methods and the meaning of embeddings in the t-SNE space might overwhelm readers early in the paper. Therefore, we opted to present a simple illustrative example as Figure 1 — one that operates in the discrete output space rather than the embedded space.

Finally, we will cite the EEOC guidelines as suggested, and in future work, we plan to extend our analysis to consider intersectional group fairness metrics. For statistical testing across multiple subgroups, we expect to use one-way ANOVA to test for mean differences in FiSCo scores across intersectional groupings.

In algorithmic fairness, new fairness notions are often accompanied by explanations of their utility, or what kind of ``world" that they would push generations towards. To some extent, this seems a bit lacking in the paper. Why should I care about the age-wise FiSCo score of Claude 3 for example? At it's core, FiSCo is a way to measure similarities between text. Is there some qualitative takeaway that I can derive from a certain score? I could be asking for too much, since my background is more-so in model calibration (where we have concrete, interpretable takeaways for a certain calibration level).

Answer: We appreciate the reviewer’s point about the importance of articulating the real-world utility of fairness metrics. FiSCo is indeed grounded in textual similarity, but its purpose goes beyond measuring linguistic overlap — it offers a quantifiable signal of differential treatment in generated outputs across social groups, under controlled conditions.

This becomes particularly important when generation tasks have potential downstream consequences. For example, a high gender-based FiSCo score indicating notable differences in career advice — even with all other variables held constant — suggests that an LLM may be reinforcing or amplifying existing societal disparities in professional guidance. This isn’t simply a matter of stylistic variation; it could affect users’ long-term decisions and opportunities in ways that are unjustified or inequitable.

Similarly, elevated age-based FiSCo scores (e.g., for Claude 3) may point to systematic shifts in tone or substance when addressing older vs. younger individuals. In contexts like hiring, mentorship, or performance evaluations, such behavior could contribute to age discrimination — whether intentional or not.

While we acknowledge that FiSCo is not a fairness solution in itself, it is a diagnostic tool — one that highlights where disparities exist so that practitioners can investigate further and take appropriate action. By surfacing group-level response differentials, FiSCo provides actionable insights that can inform model development, auditing, and mitigation strategies. In this way, FiSCo contributes to the broader goal of pushing generations toward a world that does not encode or perpetuate unjust treatment of specific groups. Analogous to how calibration scores signal over- or under-confidence, FiSCo flags disparities that merit closer human inspection.

In the final version of the paper, we will be clearer to articulate these implications and provide concrete, qualitative examples to demonstrate how certain FiSCo scores can be interpreted in context and used to guide fairness interventions.

I am a bit confused as to why we would consider the weighted average between entailment, neutral, and contradiction counts in equation (1). Can you provide some intuition about why only focusing on entailment “reflects our goal of identifying shared information between responses without penalizing for contradictions or neutrality in the initial analysis phase”? Is considering all three of these in a single metric something inherited from the hallucination detection literature?

Answer: You're right — our metric is indeed inspired by and inherits design choices from the hallucination detection literature.

We focus specifically on entailment (by setting α = 1 and the weights for neutral and contradiction to 0) in applications where differences in content could lead to long-term impacts on certain groups. For example, when using an LLM to assess candidate strengths for a job, even subtle variations in the way different groups are described may influence hiring decisions and affect both individuals and institutions over time.

By prioritizing entailment, we emphasize shared information between responses, which allows us to detect whether responses for different demographic groups omit key aligned content. A lack of shared entailments can be an indicator of bias, particularly when group responses diverge in substance rather than just style.

You're also correct that combining entailment, neutral, and contradiction into a single score is a common practice in hallucination detection. However, in our bias detection setting, we found that focusing solely on entailment produced clearer and more actionable signals, especially in the initial analysis phase.

We will add this explanation in the final version to clarify our design choice in Equation (1).

Thank you for acknowledging our contribution in group fairness in long-form generations. Yes, the motivation of this work is to find a systematic way to evaluate LLM application. Thus, your recognition of our statistical significance of our results and low variance are essential and encouraging. We will cite Eloundou et al., EEOC and WildCat in related work session. We address your concerns with the following clarifications:

How is agreement calculated in the experiments in section 5.2? The paper just states agreement with the ground truth, how is this exactly calculated given that FiSCo is a metric in [0,1]? Also, if there is only a 70% agreement by FiSCo with the ground truth, how should I interpret a FiSCo score of 0.13, for example, for Claude3 Sonnet?

Answer: While the similarity score is bounded in [0,1], we compute a t-statistic over grouped similarity values as described in Section 3.5 (referred to as the FiSCo score). We then apply a significance threshold (α = 0.05) to make a binary decision about whether a given group of texts exhibits statistically significant bias. For agreement calculation, we compare this binary decision (biased or not biased) to our ground truth annotations.

Regarding the interpretation of scores like 0.13 for Claude3 Sonnet: this means that in 13% of cases, we detected statistically significant differences between groups, indicating potential bias. In other words, 13% of evaluated prompt cases were classified as biased.

We will clarify this interpretation in the paper to avoid confusion.

Further discussion on the p-values presented is needed. We have a p-value from the Welch’s t-test used to evaluate FiSCo. However, is this the same p-value cited in Table 1? Or do the p-values in table 1 represent a significance test on FiSCo being better than SimCSE or SentenceT5?

Answer: The p-values serve different purposes in different parts of our analysis:

For the FiSCo score itself, we use Welch's t-test to determine whether the differences between inter-group and intra-group similarities are statistically significant. This helps us identify whether a given instance shows evidence of bias.

In Table 1, we compare the performance of different similarity metrics. Here, we use a paired t-test (measuring scores on the same sample across different methods) to assess whether the mean difference in agreement rates between our method and the second-best method (SentenceT5) is statistically significant. We apply the same procedure in Table 2. We will clarify our points in the final version.

Thank you for believing our proposed framework is a step in the right direction for assessing social bias in open-ended text generation. We will include BBNLI and other suggested works in our paper. We address your concerns with the following clarifications:

1 If I look at the second example in Table 13, I’m not sure I would categorize the difference between Text1 and Text2 as bias. In fact, I think there could be instances where the differences are ok and not a proxy for bias. In particular, for the example in Table 13, I don’t find the difference as indicative of bias. Do you think that different model behavior for counterfactuals is always indicative of bias?

Answer: Thank you for going through the Appendix and for this thoughtful observation about Table 13. You raise an important point - differences in model outputs don't automatically constitute bias. Whether a difference indicates bias depends heavily on context and potential impact. In the example you referred to, while the responses themselves may appear neutral, suggesting a technical career path (computer science) for the female engineer versus a management track for the equally qualified male engineer could potentially reflect and reinforce existing gender disparities in technical leadership roles. Therefore, this difference could be indicative of bias, particularly when the question concerns career planning. Different model behavior for counterfactuals is not always indicative of bias — it strongly depends on the specific context and the nature of the question. Thus, we dedicated section 4.1 for identifying important questions.

2. Some of the templates could lead to different generated output and I wouldn’t find it problematic/indicative of bias. For example, wrt this template: “Describe [NAME]’s communication style with team members.” may lead to different answers for women/men that are not problematic; the same for different ages. In other words, it is not clear to me how to distinguish between differences that are indicative of bias/problematic behavior/differential treatment, and the ones that are benign.

Answer: You're absolutely right to point out that not all differences across demographic groups are inherently indicative of bias. For example, variations in how communication styles are described across gender or age may reflect genuine diversity in expression and social norms, and these differences are not necessarily problematic.

However, our analysis goes beyond observing individual template outputs. As discussed in Table 5, the base prompts lack sufficient contextual grounding to enable reliable bias assessment on their own. To address this, we augmented the questions with additional controlled attributes (see Table 8), such as experience, role, and qualifications. This augmentation enables more controlled and meaningful group comparisons.

When we observe systematic differences in how groups are described — after controlling for all relevant variables — we argue that such patterns can signal bias, especially when those differences are likely to influence real-world outcomes, such as hiring, promotions, or performance evaluations.

That said, we agree this is a nuanced issue. Bias is not simply a matter of detecting any difference — it requires careful interpretation of context, potential harm, and intention. Our goal is not to flag every disparity as harmful, but rather to identify where unjustified or disadvantageous differentials may exist, and to enable more transparent evaluation of those effects.

3. In the first paragraph in 4.3, it’s not clear to me what (r1, r2, r3) refer to (including line numbers makes it easier to provide specific line feedback)

Answer: While we mention the definition in Section 3.3, we agree that the notation (r1, r2, r3) could be clarified further. These symbols refer to different responses generated for the same question. We will revise Section 4.3 in the final version to make this notation more explicit and easier to follow for the reader.