Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation

The paper lacks any kind of human evaluations which serves two purposes. First, human evaluation acts as a standard to understand the quality of rankings and ratings data acquired from the LLMs. In general, folks do not use very small models like Qwen-3B or LLama-3B as judges as they would achieve very poor agreement with humans. Secondly, it is useful to study the extent to which distractor features distract humans under these preference protocols too. Such experiments could be conducted at a smaller scale but are critical.

It is crucial to understand both the quality of model responses and the agreement of LLM judgments with human preferences. Our focus, however, is specifically on LLM-as-judge settings, which are increasingly used in practice for automated evaluations and leaderboard rankings. The use of smaller models in our study was intentional to highlight scale effects in evaluation bias. To address concerns about representativeness, we note that even the 3B models used in our study demonstrate reasonable alignment, achieving around 60% agreement with ground-truth human judgments in the MT-Bench dataset.

The paper does not seem to talk much about the current way of performing pairwise evaluations since LMArena has a style-controlled leaderboard and AlpacaEval2 has length-controlled version.

Style-controlled Chatbot Arena mainly addresses length and formatting biases, and length-controlled AlpacaEval2 similarly focuses on verbosity. In contrast, our work targets specific distractors—assertiveness, prolixity, and sycophancy—to analyze how they bias evaluation protocols.

It would be helpful to have some experiments on actual LLMs (e.g., GPT-4) that are used as judges in popular datasets like AlpacaEval, ArenaHard

We agree it's important to evaluate with models commonly used as judges. Below are the results with GPT-family models on the MT-Bench dataset, replicating the leaderboard analysis presented in Section 5:

These results are consistent with our main findings: pairwise preferences are more susceptible to distracted evaluation.

What is the distribution of ratings on some of the datasets? If the evaluator keeps on predicting very similar scores (e.g., 5) for each response, the plausibility of flipping in absolute rating is lower due to the sheer low quality of the rating method.

Thank you for the question. In the IF-EVAL-TweakSet experiments, the responses are explicitly designed to be either of identical quality (in the fixed-quality case, Section 3.1) or varied in a controlled manner (in the variable-quality case, Section 3.2). In these cases the score distribution reflected the intended quality differences.

For the leaderboard analysis in Section 5, where the response quality varies more naturally for MT-Bench, we observe a well-distributed range of scores, showing that the evaluator models are sensitive to quality variation. For the MT-Bench experiments, the score distribution of the initial responses when evaluated using Qwen-2.5 models is as follows:

Thank you for your feedback! We understand that there are some typos and notational inconsistencies which we plan to address and fix in the future releases of this paper. Addressing some of your concerns below:

What is the meaning or implication of the bold and underlined values in Table 2? The bold and underlined values were originally included to highlight particularly large differences between absolute and relative scores. We plan to clarify these annotations in the future versions of the paper. Since absolute scores can be converted into pairwise preferences by comparing them, what is the performance impact when using the proposed distractor feature in that context?

We appreciate the question. In Table 2 and Figure 3, we address this by converting absolute scores into pairwise preferences to compute win rates, allowing a direct comparison with explicitly collected pairwise evaluations. The results show that models tend to incorrectly favor distractor-augmented responses under pairwise evaluation, while absolute scoring remains more consistent with the intended quality differences.

The experiments evaluate the influence of distractor features on feedback protocols during evaluation. Given that LLM feedback is also used in post-training, it would be useful to know whether distractor features have a similar impact during post-training. We agree this is an important direction which we have also been considering and are actively exploring it for follow-up work. Our current focus is on the LLM-as-judge protocols and we believe that studying the LLM-as-judge setting in isolation is also crucial, especially given the growing reliance on automated evaluations in the literature and benchmarks.

Prominent models from the GPT-family were excluded from the experiments due to potential biases from dataset generation and technical constraints like positional bias and lack of logit access. This limits the generalizability of the findings to a significant portion of currently used LLMs

The reviewer presents a valid point. We conducted additional experiments using GPT-family models with the IF-EVAL-TweakSet in the fixed quality case and found results consistent with our main findings: pairwise preferences are more susceptible to distracted evaluation. The effect persists and highlights the need to account for such biases in evaluation.

We can update the paper with this analysis as well!

The analysis of distracted evaluation focused on just three distractor features—assertiveness, prolixity, and sycophancy—and used instruction following as the primary task. It's unclear how well these findings extend to other distractors or evaluation dimensions like factuality or reasoning.

We agree this is an interesting direction and our experiments can be easily extended to other distractor features and evaluation dimensions. The work focused on instruction following with verifiable instructions to ensure a clear ground truth. For tasks like factuality or reasoning, such a ground truth might be harder to establish and LLM-based annotations would likely be needed, adding additional complexity.

While highlighting the strengths of absolute scoring, the paper acknowledges its inherent limitations, such as its difficulty in capturing subtle differences between responses and potential for inconsistency or poor calibration across different evaluators

The paper does not aim to promote one protocol as universally superior, but rather present the limitations of both absolute scoring and pairwise preferences to inform more rigorous evaluation setups. We argue that absolute scoring performs better in the specific context of mitigating bias, while acknowledging its limitations and avoiding overgeneralization.

Finally, some of the paper's findings are not particularly surprising. For example, Jeong et al. (2024) (https://arxiv.org/abs/2406.12319) —which is not cited or discussed—show that pairwise evaluation amplifies evaluator bias and performs significantly worse than absolute scoring or pointwise evaluation on adversarial examples. They also propose a hybrid method that combines independent pointwise reasoning with a final pairwise decision, reducing bias and improving LLM evaluator performance across datasets.

Thank you for pointing to this related work, we plan to cite it in the revision. Our work is complementary but takes a slightly different perspective, focusing on a narrower setting of distracted evaluation and leaderboard-style evaluation protocols. We analyze the impact of three specific distractor factors through win-rate comparisons under both absolute and pairwise scoring, using a setup with verifiable ground truth. In contrast, Jeong et al. (2024) presents an empirical study on how pairwise preferences exaggerate bias in adversarial evaluations and propose an interesting hybrid mitigation strategy. Together, these two papers help build a comprehensive understanding of evaluation biases under the different protocols.

The paper purports to compare feedback protocols, but neither term is accurate for describing the scope of this paper.

We agree that the study is more accurately described as comparing ‘LLM-as-judge scoring protocols’ rather than general-purpose feedback protocols. In our setup, "feedback" specifically refers to scores produced in automated evaluation contexts, not in RLHF pipelines. To avoid confusion, we are happy to rephrase the terminology to reflect the narrower focus on studying LLM evaluation and not generalise to RLHF training pipelines. With respect to downstream effects on the generator trained with RL, this is an exciting direction that we are exploring for follow-up work. However, in this work the focus is on evaluation and we believe that studying the LLM-as-judge setting in isolation is also crucial, especially given the growing reliance on LLM evaluations in the literature and benchmarks.

The experiment only involves LLM-as-judge with two Qwen models and two Llama models; so, at best, we can expect the results from this paper to generalize to LLM-as-judge protocols, not protocols (e.g., for collecting human judgments) in general.

Prior work like Bansal et al. (2024) focuses on human annotation protocols and judgment discrepancies and our work complements this by specifically analyzing biases within LLM-as-judge evaluation settings.

In addition to the two model families in the paper, we also conducted experiments using GPT-family models with the IF-EVAL-TweakSet in the fixed quality case. We find these results to be consistent with our main findings: pairwise preferences are more susceptible to distracted evaluation. The effect persists and highlights the need to account for such biases in evaluation.

We can update the paper with these results in any future versions of the paper!

The distracting features are added to model responses post-hoc. Despite the discussion of hacking in Section 5, it's unclear if these results will generalize to spontaneously occurring distracting features in LMs, like sycophancy. As a result, it's unclear whether these results are relevant to RLHF, where the sampling is controlled in-house, so only spontaneous distractors can occur.

Thank you for raising this point. While our setup does not involve RLHF and uses post-hoc injection of distractors, this choice was intentional to study the effect of distractors in a controlled way. Injecting distractors synthetically gave us a clear and measurable signal, which would be harder to achieve in a fully generative setup with more confounding factors. Additionally, prior work (Singhal et al., 2023, Sharma et al., 2023) has shown that similar distractor traits such as verbosity and sycophancy do emerge spontaneously in LLM outputs supporting the relevance of our findings beyond the synthetic setup. [1] https://arxiv.org/pdf/2310.03716 [2] https://arxiv.org/pdf/2310.13548

Additionally, it's unclear how relevant this is to leaderboards (whether it's plausible for a hacker to post-edit LM responses).

The point raised here is very valid. While we injected distractors post-hoc for control and clarity, we note that models can be trained to generate responses with such features–intentionally or otherwise. In that sense, post-editing is not a strict requirement for these biases to appear in practice.