Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

We thank the reviewer for their insightful feedback while acknowledging the novelty, effectiveness and simplicity of our method. We appreciate your valuable suggestions in further discussing the real-world cost. With the additional results with respect to beam size and model calls, we hope that our response sufficiently addresses the reviewer’s concerns and the reviewer could consider improving their score.

---

Presentation in Section 3.2

We sincerely appreciate the time and effort you have invested in carefully examining our algorithm and the corresponding demo presented in Algo.1, Fig.3 and 4. We acknowledge that our method is intricate and our presentation may not have been optimally structured to facilitate a clear understanding. We will expand the text and captions to improve the clarity of our work.

---

Beam size, number of model calls and performance

We agree that detailed information on the number of model calls can help readers better understand the complexity of our methods and baselines. Number of model calls is affected by many factors, including beam size, uncertainty threshold, number of candidates to rank and the distribution of model’s output logits. Therefore, we conducted an additional ablation study on beam size selection, keeping the uncertainty threshold at 0.6. We include the Spearman correlation results along with the number of model calls in ():

As mentioned in Section 3.4, direct scoring methods generally have complexity of O(N), while existing pairwise baselines are generally O(N^2). PairS provides an O(NlogN) solution and a trade-off between performance and complexity between O(NlogN) and O(N^2).

In conclusion, PairS is more efficient than pairwise baselines and utilizes computation more effectively than some score-based evaluation methods. For instance, G-Eval uses a weighted average of 20 inference repetitions, resulting in a total computation of 20N. In comparison, PairS-greedy requires Nlog2(N) in the worst case. Specifically, for the 16-candidate SummEval example mentioned above, the factor is only 2.5N.

---

Asymptotic complexity of PairS scaling variant:

In the second stage of PairS-beam-scaled, the complexity is N*log2(Anchor size). With an anchor size of 200, the complexity is 7.6N, which is comparable to some direct scoring method such as G-Eval.

We thank the reviewer for their valuable feedback and for recognizing the well structured presentation, novelty and superiority of this work to the research community. We appreciate the suggestions for listing the number of pairwise comparisons in our work to further enhance the presentation. Following this additional experiment, we hope our response could sufficiently address the reviewer’s concerns.

---

A better title of "aligning with human judgment."

Thank you for your suggestions. We are aware that the current title may overlap with the line of work in fine-tuning LLMs or calibration for a better alignment with human values. Our original intent was to highlight PairS' ability to leverage a smaller gap in evaluation standards between humans and LLMs. We will endeavour to figure out a more appropriate title for future versions.

---

Number of pairwise comparisons

Thank you for the suggestion to report the actual number of model calls. We agree the actual number of model calls can be more helpful than the computational complexity. We conducted an additional ablation study on beam size selection, keeping the uncertainty threshold at 0.6. We will include the Spearman correlation results along with the number of model calls in ():

We will include this ablation study with a detailed figure in the later version.

We thank the reviewer for insightful reviews and the acknowledgement of the novelty and presentation of our work.

---

Pairwise Baselines

We agree that pairwise baselines can better demonstrate our method's efficiency. We propose comparing PairS with two baselines: 1) Win-loss rate and 2) ELO rating, both using randomly sampled pairs without replacement. We present the mean and std of Spearman corr. over 10 repetitions, vs # of comparisons

Uncalibrated PairS-greedy achieves the same performance as the baselines using only 34% and 31% of the comparisons. Calibrated PairS-greedy, which doubles the number of comparisons, can sometimes outperform baselines that use O(N^2) comparisons.

---

Ranking as evaluation is not motivated

We respectfully disagree. Ranking offers a much finer granularity compared to score rating, as the number of candidates is typically much larger than the number of score categories. Even in rare cases where the candidate number is smaller, ranking still provides valuable information. As discussed in Sec 3.1, scoring suffers from challenges in aligning evaluation standards across raters.

---

PairS may be expensive

We acknowledge that pairwise evaluations are generally more expensive than pointwise methods. However, we would like to highlight:

1. Performance should be considered alongside cost. We show that PairS outperforms most pointwise methods, such as G-Eval, which only obtain marginal gains with increased computation.
2. PairS is highly efficient compared to pairwise baselines.

---

The limitation in Section 2

In Section 2, we show that calibration cannot effectively bridge the gap between predictive posterior distributions. This serves as empirical evidence that leads to the conclusion that there is yet a gap between the likelihood terms of Eq.1., which motivates us to investigate pairwise evaluation to mitigate the gap in evaluation standards.

It is important to note that this paper does not focus on calibration, and we are not showing that our method can better eliminate the bias in model’s prior. Instead, our experiments are designed to show that we achieve better final evaluation performance, which can be interpreted as a smaller gap between the posterior terms.

For GPT-4, as it does not support the output of token probabilities, we set ‘n = 20, temperature = 1, top p = 1’ to sample 20 times to estimate the token probabilities.

We thank the reviewer for the feedback and have addressed them below.

1. Pairwise Baselines

This ablation study with other two pairwise baselines using a different format than our result Table 1. In this table, 50, 100, 240(max) represents the number of randomly selected candidate pairs used to calculate the win rate or the ELO rankings. For SummEval dataset, there are 16 candidates for each datapoint, resulting in a total of 240 pairs. (N^2, including swapped comparisons and excluding self-comparison.)

For PairS-Greedy, we cannot control the exact number of comparisons needed for the ranking. By averaging the results over 10 repetitions, we use 39.2 and 44.3 comparisons to achieve the global rankings for Mistral-7B and GPT-3.5. When using calibrated PairS-Greedy, (where each comparison uses the average probabilities of pairwise comparison with permutated positions), the number of comparisons are 41.52 and 46.32. The reason we mention calibrated result is that the 240 total comparison pairs include comparisons with swapped positions.

Both uncalibrated and calibrated PairS-Greedy, our method are significantly more efficient than the win rate and ELO baselines. Presenting these results in a table may not be optimal, and we plan to use line figures in the formal revision to better illustrate the findings.

---

2. Ranking as evaluation is not motivated

If you compared M systems for N pairs, wouldn’t the computational complexity be O(M^2 * N^2)?

We believe the reviewer has misunderstood the setup of pairwise ranking. We are not comparing the whole LLM systems, but a list of candidates. The candidates can be generated by different systems, but we only evaluate the quality of those candidates. If we have M candidates to rank, we can sample up to M*2-M pairs. What we meant by "as the number of candidates is typically much larger than the number of score categories", is M is usually larger than the scale of score categories (5 for likert scale). Therefore, providing the ranking of M candidates is more fine-grain than scoring.

---

3. pointwise evaluation and G-Eval

We acknowledge the statement in G-Eval paper. We believe the reason why G-Eval cannot handle API model efficiently is beyond the discussion of our paper. In our paper, what we claim is pointwise evaluation methods generally have complexity of O(N) and our argument is still hold: pointwise evaluations can only gain marginal performance with more computations.

---

In conclusion, we believe our experiments and ablation studies provide a comprehensive evaluation of the proposed PairS method. We have compared with multiple baselines including both pointwise and pairwise evaluation methods. Our results show that PairS is more efficient and effective than pairwise baselines and achieves higher human agreement than both pointwise and pairwise baselines.

Furthermore, we have clarified and provided explicit support for the motivation behind ranking evaluation. We also have address reviewer's misunderstanding regarding our discussion of the limitation of direct scoring in Section 2.

We have made a sincere effort to respond to the reviewer's concerns and to provide additional evidence and clarifications to support our claims. In light of our response, we respectfully request that the reviewer consider improving their score, as we believe our work makes a valuable contribution to the field of LLM evaluator.

We thank the reviewer for their insightful feedback while acknowledging the quality of our work. We hope that in light of the response that addresses all the reviewer’s concerns, the reviewer could consider improving their score.

---

PairS performance vs Direct scoring, and the motivation in 3.1.

While our main results in Table 1 show that PairS generally outperforms direct scoring, we acknowledge that there are a few exceptions. These variations are expected due to challenges in LLMs' capability to handle long-context input and their positional bias. The exceptions occur in the consistency aspect of SummEval, as explained in Section 5.2, which is caused by the highly concentrated distribution of human scores, and in the performance of Llama2-7B and Mistral-7B on HANNA, which we attribute to the difficulty in understanding long-form creative writing.

To further support our motivation stated in Section 3.1, we conducted a complementary experiment by randomly sampling 50 summary candidate pairs for each source text in SummEval and calculating the 3-way agreement rate accuracy between direct scoring vs. human scores and pairwise comparison vs. human scores. The results show that pairwise comparisons have consistently better agreement than scoring.

---

Significance test on Table 1 results:

Thank you for raising this point. To address this, we conducted z-tests on 30 repetitions for cases where direct scoring and PairS have relatively similar performance. The results show that PairS-greedy significantly outperforms direct scoring for Mistral-7B on SummEval coherence (p<0.01). For GPT-4, the differences are significant on fluency (p<0.001) and weakly significant on consistency (p<0.15). Furthermore, the difference is significant for GPT-3.5 on relevance (p<0.001).

---

Beam size, # of model calls and performance:

The search space of the candidate ranking is determined by both the beam size and the uncertainty threshold. A large beam size allows for more ranking permutations to be considered when the uncertainty threshold is fixed. We provide the following ablation study with an uncertainty threshold of 0.6 and reporting Spearman corr. along with the number of model calls in ():