B-score: Detecting biases in large language models using response history

Thank you for your suggestions!

Summary: We experimented with confidence baselines, highlighted our focus on detecting bias, and provided empirical evidence showing that LLMs can self-calibrate in multi-turn due to their inherent ability to do so.

The experiments lack meaningful baselines. The verbalized confidence score is not commonly used for LLM calibration and the authors ignore other potential model calibration baselines.

We'd like to clarify that our work focuses on the proposed B-score, which detects bias in an LLM’s output rather than on model calibration. B-score measures the bias in the output, which differs from standard confidence calibration that aligns overall probability estimates with correctness likelihood. On the other hand, all the works mentioned by the reviewer face a similar issue with verbalized confidence scores on detecting bias. For example, in [1] and [2], the confidence score is computed based on the option distribution, which ends up being the same score for all options. This is not what we expect for bias detection, which should be high for the biased option.

We will cite the reviewer's suggested references, making clear that our work tackles a different problem (bias detection vs. confidence calibration). Moreover, prior works that the reviewer mentioned either required rephrasing prompts using other LLMs [3], training auxiliary models [4], or accessing internal weights [5]–[8]. In contrast, our method only needs to repeat the same question in single-turn and multi-turn conversations, without fine-tuning or extra training.

Tab. R1 Accuracy on verification task of GPT-4o (%). Mean Δ=29.1%

However, we still appreciate the reviewer’s advice and have conducted experiments comparing our method with the Agreement-based [1,2], Entropy-based [2], and FSD Confidence Scores [2]. These baselines are closely similar to our single-turn probability. They can be computed even for closed-source models, but our single-turn is different for each option, unlike confidence scores. The results (Tab. R1) show that our B-score significantly outperforms confidence score baselines on the verification task:

• In our evaluation framework, B-score (85.9%) alone achieves higher verification accuracy than FSD (55.2%), Entropy (76.7%), and Agreement-based (76.7%) confidence scores; similarly, in the BBQ benchmark, B-score (85.8%) outperforms FSD, Entropy, and Agreement-based confidence scores (34.9%for all).
• B-score can still help improve verification accuracy when combined with confidence scores in our evaluation framework and BBQ bias benchmark (Mean Δ=29.1%).

---

The proposed method lacks justification, it's unknown why LLMs can self-calibrate in multi-turn conversation - even with explicit refinement instructions. There is neither empirical nor theoretical explanation to defend the method to be universally benefit in LLM calibration.

As mentioned in the paper, LLMs can self-calibrate in a multi-turn conversation because they inherently possess this capability. Multi-turn settings simply trigger their actual capability by allowing them to see their response history, providing a form of feedback or context that helps adjust subsequent answers.

Tab. R2 Distribution of Answer Percentages (%)

To provide an empirical explanation, as the reviewer suggested, we conducted an experiment and the results indicated that LLMs (e.g., GPT-4o, GPT-4o-mini) are able to generate well-known distributions (i.e., Gaussian, Uniform; Tab. R2). This is the fundamental reason why LLMs can self-calibrate in multi-turn. For example, the prompt we used for Gaussian is:

I have a random variable X that takes 10 integer values between 0, 1, 2, 3,...,9. Sample X 100 times following a Gaussian (mean=4.5, std=2.0) distribution, and return a list of 100 integer numbers.

Thank you for your detailed and constructive feedback!

Summary: We've carefully extended the experiment on the BBQ bias benchmark to address concerns about dataset size, clarifying our results in Tab. 3 vs Tab. 4 and number of queries in single-turn/multi-turn, revised our writing (e.g., quotes) and claim,...

My main issue though is that the set of questions over which a lot of the analysis is done is very small

Please check our response for Reviewer ENzm regarding this issue. On well-known BBQ bias benchmark, our conclusions remain the same.

...in Table 3 the smaller and the larger model variants win almost equally (as measured by the mean across tasks) based on your tasks, but not in Table 4 for standard metrics, where almost universally the gain is evident for smaller models...

In Tab. 4, we did not experiment with GPT-4o (larger model) on HLE as we did for GPT-4o-mini (smaller model), which may give the impression that smaller models have gained more. To address this, we've added HLE for fair comparison. GPT-4o clearly surpasses GPT-4o-mini in Mean Δ (+2.9% vs. +1.8%). Thus, in total (if we consider Command R/R+ in Tab. 4), the large model performs almost equally to the smaller models across benchmarks when HLE is fully added.

Tab. R1 Accuracy (%) across CSQA, MMLU, HLE on verification task

Is it true that B-score “attempts to indicate whether a model is biased due to its imbalanced training data”?...

We agree that B-score by itself does not identify the source of the bias, only the presence of a bias. We acknowledge that other factors (e.g., model architecture, decoding algorithms) could also contribute to these behaviors. We will rephrase this claim based on your valuable suggestion.

...I am not sure there is much signal in the “easy” category however...

Easy category is included as a baseline (B-score~0) and contrasts with other categories. In Tab. 2, N/A entries for fully correct answers inadvertently increased the Easy mean. Since an ideal bias metric should be 0 when no bias exists, we modified Tab. 2 so that Easy now has a mean B-score of +0.06, much lower than Hard (+0.15), and Random (+0.41).

Computing the B-score potentially incurs significant expense given the need to probe using the query 30 times each for the single-turn and multi-turn queries.

Q1: How was using 30 samples for the single-turn and multi-turn queries selected, and is this number required to be the same for all tasks? What is the effect of making this number of samples smaller? ...

Tab. R2: Mean B-score with different k across 8 LLMs (GPT, Gemini, Command R, LLama)

We replicated our evaluation with k=10, 20 queries. Different k's results are still similar (0.22 ~ 0.23). Thus, reducing #queries count does NOT significantly change B-score and can save computation resources. We chose k = 30 as an upper bound to ensure reliability, but in practice, a smaller k may suffice. k should ideally be 2-3 times the number of answer options to ensure sufficient coverage. We will add a section to discuss this in detail.

...it would be interesting to compare measured/reported confidence for models for which the weights are available and can be run locally.

For models that can be run locally with accessible weights, confidence is often measured directly using log probabilities. However, this isn't possible for closed-source models where internal logits aren't exposed. In such cases, our single-turn method offers a practical proxy: we repeatedly ask the same question and aggregate the model’s responses to estimate confidence. The single-tun approach effectively simulates log-prob-based confidence, which is discussed in Sec 4.4.

Overall I feel that the scope of the study is on the small side for ICML, and the paper might be better suited to a conference dedicated specifically to fairness/bias.

We thank the reviewers for prompting us to clarify its broader impact, but respectfully argue that our study is not too narrow for a general ML venue. Bias/fairness in LLMs is critical for the broader ML community. In fact, our submission was to the category as a paper in Trustworthy ML (fairness, interpretability,...), one of the topics of interest of ICML this year. Additionally, we hope the reviewer sees that LLMs can self-correct biases via their own response history is a general insight that could inspire new training/prompting/evaluation techniques in ML at large.

Thank you for your thoughtful feedback!

Summary: We've extended our experiments to the BBQ benchmark to address the reviewer's concern about test size and clarified our rationale for using four distinct question categories to capture different aspects of bias.

This is a relatively small dataset, which may limit the generalizability of the findings.

The results may be overly dependent on specific dataset choices

We respectfully note that our 36-question evaluation framework is consistent with prior work in this line of research (e.g., [R1] uses only 6 questions, [R2] uses only 30 questions in test set). In our work, each question is rigorously tested (30 single-turn and 30 multi-turn queries over 10 runs), yielding statistically reliable results. We believe increasing the sample size would not change the observed trends. Instead, the next natural step is to expand the dimensions of bias, which we already do by categorizing questions into Subjective, Random, Easy, and Hard. Moreover, we have complemented these experiments with evaluations on standard benchmarks such as CSQA, MMLU (Easy), and HLE (Hard). For Random, our multi-turn setup consistently yields uniform answer distributions (Fig. 4), a finding that we believe is very interesting and will persist even with large-scale testing.

However, in response to reviewer's feedback, we have extended our evaluation to the BBQ (ambig category) [R3], a well-known bias benchmark (Subjective). First, we removed the unknown option, forcing the model to choose between the remaining two options. For each binary-choice question, we compare the higher single-turn prob (Higher) option with the lower one (Lower). On well-known BBQ bias benchmark, our conclusions remain the same:

• For Higher options, the Single-turn prob drops significantly in the Multi-turn (0.94→0.77; Tab. R1), indicating that the model adjusts its answer distribution when allowed to look into response history
• Confidence scores remain constant (0.63 for both options; Tab. R1), confirming that they fail to capture the output's distribution and thus are unsuitable for bias detection
• The B-score differentiates clearly between 2 options (Tab. R1): a positive B-score (+0.17) for the Higher option and a negative B-score (-0.16) for the Lower option, showing its effectiveness as a bias indicator
• The B-score substantially improves verification accuracy (Mean Δ = 45.7%; Tab. R2)
• The B-score (89.6%) alone also performs significantly better than other metrics individually (Tab. R2), such as Single-turn prob (20.9%), Multi-turn prob (33.9%), and Confidence score (77.6%)

Tab. R1: Results for Higher Single-Turn Prob (H) and Lower Single-Turn Prob (L) Options

Tab. R2: Verification accuracy (%). Overall Mean Δ = 45.7%

---

However, these categories do not appear to be conceptually parallel—subjectivity vs. difficulty vs. randomness are distinct properties rather than a single dimension of classification

We've clarified this distinction in the paper. These categories were intentionally chosen to span different aspects of bias rather than a single spectrum. Our goal is to ensure coverage of scenarios where bias can manifest in distinct ways: Subjective test preference; Random test for random ability; Objective (Easy, Hard) test whether there is a bias toward the incorrect option.

References

[R1] Forcing Diffuse Distributions out of Language Models. COLM 2024
[R2] The Woman Worked as a Babysitter: On Biases in Language Generation. EMNLP 2019 
[R3] BBQ: A Hand-Built Bias Benchmark for Question Answering. ACL 2022