Benchmarking Cognitive Biases in Large Language Models as Evaluators

(Q6) ”I see that random is calculated for different biases, however, for better models like ChatGPT, the egocentric bias may be unfair because the generation of ChatGPT is indeed better…”

We agree with your statement that biases such as Egocentric and Saliency may be difficult to fairly measure. However, we highlight two important aspects regarding the identification of these biases:

1. If multiple models have a large proportion of evaluations preferring their own responses (as the evaluated pool of pairwise instances is the same for each evaluator), we reason that this may suggest “egocentric” qualities within involved evaluators, regardless of the objective strength of the models. Moreover, we see this effect is especially demonstrated between the more powerful models as well (GPT4 & ChatGPT) that suggest the presence of Egocentric evaluations from their disagreement.
2. We employ various strategies in order to mitigate these confounding variables and isolate each analysis as much as possible. For example, we employ a “hierarchical” rubric, where some biases take priority in an evaluation. Specifically, if an evaluation shows signs of order bias by choosing A in (A first, then B) and B in (B first, then A), we do not evaluate it for Saliency or Egocentric bias.

To get further insight into decoupling them, we provide additional statistics below examining the proportion of Egocentric samples where the (self-preferred) model’s generation was longer/shorter than the other generation.

Overall, we view that most models (9/16) exhibit a self-preference for their own generations often when their own generations exhibit longer token length. As above, we see that Salience may be associated with higher quality generations, as we see that the strongest models (GPT4, ChatGPT) often prefer their own responses when their generations are longer. Furthermore, even in smaller models (e.g. Cohere, Koala) prefer their own generations more often when they are longer. However, as we previously emphasized, if multiple models observe a self-preference for their own generations, it is difficult to associate with Salience as there is disagreement that is indicative of an Egocentric bias.

We will include this additional insight into the Appendix section of the final draft for more context into decoupling the confounding factors.

(Q5) “... The inter-annotator agreement (IAA) among the workers is not notably high in both ranking and pairwise tasks. The IAA aligns with the agreement between humans and machines. While the authors' claim that "LLMs are still not suitable as fair and reliable automatic evaluators" is not incorrect, it appears that humans may also fall short in this regard.”

We thank you for your feedback and acknowledge that the current IAA score (0.47) between humans is not high compared to agreement amongst humans/machines.

We realize our current calculation procedure may not precisely capture human or each machine preference tendencies by simply aggregating all of them, which, as a result, produces higher agreement between machine and humans and may not be a fair representation of the IAA. Hence, we provide a correction to our previous calculation for a more precise result by computing the RBO between each individual annotator and machine preferences, obtaining an average RBO of 0.357 from $6 \times 15$ different RBOs. This way, we present a more complete and accurate representation of the involved parties' preference behaviors and thus their agreements. We see that this correction further supports our claim that there is a misalignment between humans and LLMs, compared to IAA among humans. We will add these supplementary results from this new approach in the final draft.

For the pairwise task measuring biases in human preferences, we acknowledge that there would be lower IAA among humans, as the study involves 75 annotators for each bias (225 annotators) which we argue still achieves reasonable agreement given the complexity, and scale of agreement to calculate between humans. Additionally, we clarify that the main objective of this human pairwise setup is to measure the impact on human evaluation quality via our bias benchmark rather than measuring agreement. The main takeaway is that humans generally were less impacted by most of the biases than LLMs which continue to support our original claims. We will clarify these intentions in the final version of the manuscript.

(Q1) “The indicators of the benchmark is overly complicated…”

We thank you for your feedback. We presented individualized scores across each dimension in order to be as precise as possible to represent a fair comparison and evaluation of each language model as an evaluator. However, we acknowledge that the current organization of scores across several dimensions may complicate their interpretability. In order to increase the readability of Table 2, instead of breaking down Order and Compassion Fade into “First Order” and “Last Order” columns, we will merge them as a “positional” measure and take the more intense value to simplify each of the benchmark metrics to a single score. Additionally, we will move the current representation down to the Appendix to provide a full breakdown of the bias metrics at the reader’s leisure.

---

(Q2) “More analysis with human's own biases on these dimensions are needed…”

We thank you for your comments. We underline that our main objective of the pairwise human bias experiment (Section 5.2) is to measure how much humans were also affected by different types of cognitive biases, thus comparing them with LLMs. Due to time constraints and the main scope of this entire paper, however, we acknowledge that we were not able to run additional analysis with the biases in human preference.

We will put more in-depth analyses of human biases in our future work. Thank you.

---

(Q3) “The details are quite unclear. The use of 50 examples is insufficient to evaluate 15 LLMs”

We understand that the instruction set of sample size 50 may be small; however, we emphasize that the actual number of total evaluations is very large, as we construct pairwise examples between 50 generations of all 15 models for each LM-as-evaluator (totaling 10,500 samples as outlined in Section 4.2). We limited our evaluation size to 50 Q/A instances due to the large number of choices for considering each bias. We also draw upon this setting from previous works in which [1] selects 80 questions to evaluate pairwise instances for only 6 models, totaling 1,200 pairwise examples, and [2] selects computes pairwise comparisons across 12 models and 40 questions.

Furthermore, we underline that considering our task, setting up a test for the significance of results from a specified sample size is relatively difficult as there is no baseline to compare to other than evaluation results from other models. Thus, we provide results from testing the statistical significance of differences in bias scores among the 15 models below from a 1-way ANOVA test. Our calculated p-value of 0.02 highlights that the difference in bias scores between the 15 models is indeed statistically significant to further support the significance of our results over 50 question-answer instances:

---

(Q4) “How these evaluations help in the improvements of LLMs?”

We believe our contribution can be utilized for the future development of models in reducing these identified cognitive biases. For example, our preference dataset can be utilized as subsequent RLHF training that can assign negative signals with evaluations associated with a bias from our benchmark. Although employing this in practice remains out-of-scope for this work, we iterate that the contribution and novelty of our work come from our large-scale study of several models and the introduction of a benchmark composed of six cognitive biases, providing valuable insight (as underlined by Reviewer 1K6b) that has not been previously investigated in depth. However, we underline that this direction remains an important and exciting area to explore in future research.

(Q1) “In this work, the authors primarily focus on pairwise evaluation based on the coherence criterion, without considering other evaluation formats.”

Thank you for the suggestion, and we completely agree that different types of evaluations could be used and that might result in completely different outcomes. However, please note that adapting these evaluation formats poses some technical difficulties and computational costs. For instance, not all models can generate or evaluate document-level inputs. Also, the interactive evaluations pose a computational challenge (which we iterate consists of over 630k comparisons) and formats such as explanations or discussion-based preferences are challenging to benchmark.

Instead, our work focuses on comprehensiveness and scalability, so we had to simplify the evaluation setup to be a Q/A and preference-based comparison. Extending our study to other formats of evaluation could be an exciting direction for future work.

---

(Q2) “It would be better to explore more diverse evaluation settings to ensure a comprehensive assessment of LLM-based evaluators.”

We acknowledge that in our evaluation setting, we ask each evaluator to analyze generation quality along one aspect (coherence) with respect to the reference and that this setup may confine the diversity of assessment of LLM-based evaluators. We highlight that it is difficult to conduct independent, more fine-grained evaluation setups due to the scale of our experiments across all 15 models as evaluators.

We also conjecture that these cognitive biases still remain regardless of evaluation aspects. To validate our conjecture, we conduct an additional experiment incorporating different dimensions of evaluation criteria into our pairwise evaluation prompt and report their results below. In our modified prompt, we ask each evaluator to judge responses based on their “coherence, accuracy, factuality, and helpfulness” following [1] and [2]. (See the table and prompt below)

We find that the proportions of evaluations are still affected by Order bias considering ties. We see that by including diverse perspectives in the evaluation setting, for each model, some values become more pronounced (i.e. Cohere for egocentric) or bias decreases (i.e. Vicuna for egocentric). However, we see that for the majority of models in each of the tested benchmarks, the proportion of biased evaluations stays relatively consistent, which remains our conclusion that models still show a large skewness in bias tendency as evaluators along our benchmark.

We appreciate your suggestion again and will include the fine-grained evaluation in our final manuscript. – The following is the prompt and result of the fine-grained evaluation: For context, the prompt we provide is:

### Instruction: Which system's response is more coherent, accurate, factual, and helpful considering the reference and instruction?
The instruction is: {{instruction}}
The reference is: {{reference}}
'''
{model1_name}: {model1_response}
{model2_name}: {model2_response}
'''
Please response directly in the following format: System _ is better
Do not provide any additional text or explanation:
### Response:
Due to limited computation resources and time, we only run additionals experiment for two models at each size range (+ all API-based models) and present the results below: 

For visual clarity, we only display the bias ratio with the highest proportion and denote with subscript $x_F$ or $x_L$ for first- or last-ordered bias, respectively.

Q3) “It would be valuable to explore the potential contributions of current generation techniques, like self-consistency, in reducing bias.”

We thank the reviewer for the comment and acknowledge that exploring other mitigation techniques for each bias is a crucial area to explore within our bias benchmark. As mentioned in Section 4.2, we do include prompting measures such as self-consistency for every bias evaluation by prompting each pairwise instance twice in both orders (i.e. A first, then B, and B first, then A). We underline that the contribution of our work is proposing a benchmark of cognitive biases along six dimensions, testing the ability of automatic annotators to make quality evaluations without being impacted by other implicit (or induced) artifacts that may exist in the prompt. To this, we do try to incorporate some mitigation methods, such as the self-consistency checks to confirm the presence of a bias for fairness, and design our evaluation prompts according to the three criteria outlined in Section 3. However, exploring other mitigation methods to defend against each cognitive bias remains an invaluable task to investigate but remains out-of-scope for this work.

---

(Q4) “The example provided in Table 1 does not align with the definition of compassion fade bias”

We clarify that we partially borrow the definition of compassion fade from psychology, employing only the portion of the phenomena with respect to the influence from recognizable names as opposed to its real meaning defined in Section 3.1, which is difficult to examine through prompting. We highlight that our use case of presenting recognizable names in automatic evaluations is to measure the impact on the quality of said evaluations in comparison to ones with anonymized names.

Thus, the example provided in Table 1 is intended to display the differing evaluation behavior between evaluation settings presenting anonymized names and recognizable ones. Specifically, System Star is associated with Model Alpaca, and System Square is associated with Model Vicuna, where the selection of preferred responses (bolded) is inconsistent between anonymized names and recognizable names. We will make sure to clarify this detail for Table 1 and in the definition of Compassion Fade in Section 3.1 in the final manuscript. Thank you for your feedback.

---

(Q5) “There seems to be confusion regarding the number of models that human evaluators and model-based evaluators are required to rank in Section 5.2. Clarifying whether it is 15 or 4 models is necessary. Additionally, the statement about human consensus being modest but reasonable while model evaluations not aligning closely with human preferences is contradictory, as the average RBO values are very similar.”

Thank you for this comment. For the first question, the number of models ($N$) that human and LLM evaluators ranked (as shown in Sec 5.2) is 15. Appendix C.2 shows only the preliminary results of $N=4$ via a ranked-list evaluation setting versus the pairwise study that was conducted. We will add further clarification to this detail in the final draft.

Additionally, we acknowledge that the current IAA score (0.47) between humans is not high compared to the agreement amongst humans/machines.

We realize our current calculation procedure may not precisely capture each human or machine preference tendencies by simply aggregating all of them, which, as a result, produces higher agreement between machines and humans and may not be a fair representation of the IAA. Hence, we provide a correction to our previous calculation for a more precise result by computing the RBO between each individual annotator and machine preferences, obtaining an average RBO of 0.357 from $6 \times 15$ different RBOs. This way, we present a more complete and accurate representation of the involved parties' preference behaviors and thus their agreements. We see that this correction further supports our claim that there is a misalignment between humans and LLMs, compared to IAA among humans. We will add these supplementary results from this new approach in the final draft.

(Q1) “The main weakness is that the number of instructions is small…”

We understand that the instruction set of sample size 50 may be small; however, we emphasize that the actual number of total evaluations is very large, as we construct pairwise examples between 50 generations of all 15 models for each LM-as-evaluator (totaling 10,500 samples as outlined in Section 4.2). We limited our evaluation size to 50 Q/A instances due to the large number of choices for considering each bias. We also draw upon this setting from previous works in which [1] selects 80 questions to evaluate pairwise instances for only 6 models, totaling 1,200 pairwise examples, and [2] selects computes pairwise comparisons across 12 models and 40 questions.

Furthermore, we underline that considering our task, setting up a test for the significance of results from a specified sample size is relatively difficult as there is no baseline to compare to other than evaluation results from other models. Thus, we provide results from testing the statistical significance of differences in bias scores among the 15 models below from a 1-way ANOVA test. Our calculated p-value of 0.02 highlights that the difference in bias scores between the 15 models is indeed statistically significant to further support the significance of our results over 50 question-answer instances:

---

(Q2) “The experiments do not consider ties in the pairwise evaluation”

This is a valid point. Our original experiments did not include the “tie” option in order to avoid the possibility of gray areas where models-as-evaluators may assign a tie preference for all pairwise instances. Thus, in order to retrieve realistic results, we only gave the option for models to choose strictly between the two options presented.

We present a modified version of the prompt that considers ties in each pairwise preference in the following table. The results show evaluations affected by Order bias with considering ties. For Salience, if a pairwise sample was labeled as “Tie,” we do not consider it for length bias.

We see that the inclusion of the tie option does view a considerable change in the bias benchmarks. Notably, the strongest and smallest models (GPT-4, ChatGPT, Baize, WizardLM) do not exhibit any change. However, we see that the mid-range models (Alpaca, Vicuna) and InstructGPT display a large preference for assigning the tie label ($\geq 90 \\%$) that does not present any valid results, to which we had originally only prompted two options for each LM-as-evaluator to avoid this issue. The only model that demonstrated an improvement from previous bias behavior was Cohere. We will include these supplementary results in the Appendix within the updated manuscript via your suggestion.

### Instruction: Which system's response is more coherent considering the reference and instruction?
The instruction is: {{instruction}}
The reference is: {{reference}}
'''
{model1_name}: {model1_response}
{model2_name}: {model2_response}
'''
Please response directly in the following format: System _ is better

If you believe each response is equally sufficient simply respond with: Tie

Do not provide any additional text or explanation:
### Response:

Due to limited computation resources and time, we only run additional experiments for two models at each size range (+ all API-based models) and present the results below:

For visual clarity, we only display the bias ratio with the highest proportion and denote with subscript $x_F$ or $x_L$ for first- or last-ordered bias, respectively. No subscript means same preference.

(Q4) “Some models display low rates of valid responses… Consequently, the claim that "most LLMs exhibit cognitive biases" is not sufficiently justified, given that weaker models may exhibit biases more frequently.”

We acknowledge that some models display inferior performance when extracting evaluations from them, which may skew our results for benchmarking their behaviors.

However, we underline that uncovering a correlation between valid response rates and bias is not within the scope of our findings. If a model is not strong enough to produce valid outputs, we assume those models are not strong enough to be used for evaluations. Weaker models that were not good enough to produce valid inputs were not taken into consideration in this study. And as we don’t consider invalid responses within the study, we emphasize that we claim that only from models that were strong enough to produce valid outputs, most models exhibit cognitive biases from our benchmark. We will add this additional clarity to the final version of the manuscript.

Additionally, we understand that some low valid response rates ($< 80\\%$) may not appear sufficient in precise benchmarking performance estimation and identifying cognitive biases in their evaluations; we would like to emphasize that we attempt to compensate for this by the scale of our experiments. We still observe that most (13/16) models on each benchmark provide a performance rate of $\geq 50\\%$ that represents at least 2,625 samples evaluated, which we believe still provides a considerable source of evidence of some cognitive biases in the tested LLMs-as-evaluators.

(Q5) “... The inter-annotator agreement (IAA) among the workers is not notably high in both ranking and pairwise tasks. The IAA aligns with the agreement between humans and machines. While the authors' claim that "LLMs are still not suitable as fair and reliable automatic evaluators" is not incorrect, it appears that humans may also fall short in this regard.”

We thank you for your feedback and acknowledge that the current IAA score (0.47) between humans is not high compared to agreement amongst humans/machines.

We realize our current calculation procedure may not precisely capture human or each machine preference tendencies by simply aggregating all of them, which, as a result, produces higher agreement between machine and humans and may not be a fair representation of the IAA. Hence, we provide a correction to our previous calculation for a more precise result by computing the RBO between each individual annotator and machine preferences, obtaining an average RBO of 0.357 from $6 \times 15$ different RBOs. This way, we present a more complete and accurate representation of the involved parties' preference behaviors and thus their agreements. We see that this correction further supports our claim that there is a misalignment between humans and LLMs, compared to IAA among humans. We will add these supplementary results from this new approach in the final draft.

For the pairwise task measuring biases in human preferences, we acknowledge that there would be lower IAA among humans, as the study involves 75 annotators for each bias (225 annotators) which we argue still achieves reasonable agreement given the complexity and scale of agreement to calculate between humans. Additionally, we clarify that the main objective of this human pairwise setup is to measure the impact on human evaluation quality via our bias benchmark rather than measuring agreement. The main takeaway is that humans generally were less impacted by most of the biases than LLMs which continue to support our original claims. We will clarify these intentions in the final version of the manuscript.

We thank the reviewer tfnE for dedicating their time and effort to evaluate our manuscript. In response to your valuable feedback, we carefully addressed and clarified the raised concerns and comments below.

---

(Q1) “I believe that most of the biases discussed have been previously identified…”

We respectfully disagree with this. Although some biases, such as Order or Salience, have previously been identified in other works, we highlight the novelty in our work lies in confirming their presence in using LLMs as automatic evaluators in an extensive study of 6 different biases over 15 models. We believe that our focused, in-depth, and larger-scale exploration of these biases in LLMs as evaluators remains a very important research question to present the reliability and vulnerability of LLM evaluations. Overall, we believe the results of our benchmarking procedure provide “several interesting insights” (as highlighted by Reviewer 1K6b) that have not been previously explored at this scale and give valuable context to the current state of foundation models as automatic evaluators.

---

(Q2) “The observed effect might closely resemble that of injecting random names.”

Our intended evaluation setup for compassion fade was to measure its impact on the quality of model evaluations when presented with recognizable names compared to anonymized ones. We conjecture that if models were fairly judging based on the quality of outputs rather than placing weight on the name of the response model, we would expect similar results (in terms of order and egocentric bias) to the ones observed in the Order bias experiments. As demonstrated in Table 1, the disparity between Order and Compassion Fade results supports our hypothesis that the presence of recognizable names indeed influences evaluations by each evaluator in contrast to anonymized ones.

---

(Q3) “Observing a correlation between the biased tendency and the percentage stated in the injected sentence could provide deeper insights.”

This is a valid point. In our additional experiment, we show a modified statistic for the biased model: "0% of people prefer {model}.” If bias tendency were indeed correlated with the statistic, we would expect the evaluator model to have 0 preference for bandwagon response. Due to limited computation resources and time, we ran additionals experiment on the bandwagon test with 0% statistic for two models at each size range (+ all API-based models) and present the results below:

Here one can see that the preference choices for the bandwagon statistic greatly change (besides GPT4 and Vicuna) which suggests that indeed the biased tendency is correlated with the bandwagon statistic. However, we see that Vicuna, in particular, is not greatly affected by the statistics. This suggests that within the prompt, the model only focuses on the phrase “people believe that {model} is better” instead of the statistic. Similarly, this may be the case for Alpaca and InstructGPT as well. We also present the results of the bandwagon test by randomly choosing a percentage between 50% and 85% below to support a correlation between biased tendency and the statistic. Most models show slightly lower bias preference as the statistic can range lower down to 50% that further supports a correlation: