Prediction-Powered Ranking of Large Language Models

[Application of PPI] To the best of our knowledge, existing work has always applied PPI to construct confidence intervals for numerical quantities. In contrast, our work is the first to apply PPI to construct rank-sets and does so in a very timely domain, LLM evaluation.

[Rank set vs numeric metric] We would like to first point out that rank-sets have been used in the literature as measures of uncertainty in rankings (see, for example, [48, 50]). In our setting, we believe rank-sets may provide useful information—the rank-sets tell the practitioner to what extent the difference in win-rates across models, after taking into account the uncertainty summarized by the confidence ellipsoid, are sufficient for each model to be ranked above or below other models. That being said, a practitioner may decide to use our framework to obtain not only estimates of the rank-sets but also estimates of the win-rates, which is a numeric metric, and the confidence ellipsoid used to construct the estimates of the rank-sets. We will clarify this in the revised version of the paper.

[One dataset / generalization] Since our computational framework is rather generic, it is true that it could be readily applied to other benchmarks. However, this would require significant funding and time and, given the on-going discussions about the reliability of LLM evaluation, we believe the NeurIPS community may benefit more from our computational framework if published early. Further, given the rapid development of both new LLMs and benchmarks, we also feel that any claim of superiority of a LLM over others based on the estimated rank-sets on (static) benchmark data may be quickly outdated and thus have limited value.

[Baselines] Since the regression fit is just an empirical estimate of a set of means (the win-rates), we do not find any reason to raise doubts about the baseline. Nevertheless, during the rebuttal period, we have carried out an additional evaluation of our framework in a synthetic setting. Since in the synthetic setting, the ground truth ranking is known and given, we do not need to rely on any processing to validate the theoretical coverage guarantees. Refer to the (general) author rebuttal and attached pdf for more details. We will include this additional evaluation in an Appendix in the revised version of the paper.

[Pareto frontier] In the conclusions we have drawn from Figure 1 in lines 264-278, we do not (mean to) claim that the PPR methods achieve pareto frontier against all baselines. In lines 266-272, we claim that, in terms of baseline intersection probability, PPR methods are better than baseline methods using only (the same number of) comparisons by strong LLMs. In lines 273-278, we claim that PPR methods indeed achieve pareto frontier against a baseline method (HUMAN ONLY) using the same number of human comparisons as PPR methods since they are better both in terms of baseline intersection probability and size of the rank-sets. To avoid any misunderstanding, in the revised version of the paper, we will explicitly highlight that the baseline methods using only comparisons by strong LLMs GPT 4 and Claude 3 return smaller rank-sets than the PPR methods.

[Flow of the paper] Following the reviewer’s suggestion, we will improve the flow of the paper and the description of Figure 1 in the revised version of the paper.

[Limitations] Following the reviewer's suggestion, we will expand the discussions of the limitations and highlight that, in practice, active learning may be used to gather human pairwise comparisons.

[Small set of human pairwise comparisons] The small set of human pairwise comparisons has length = $n > 1$ and we evaluate the performance of our computational framework for different values of $n$ in Figure 2.

[Erroneous human comparisons] We agree that it would be interesting to explore the potential error propagation in the rank sets from erroneous human comparisons due to, e.g., bias, lack of truthfulness or strategic behavior, however, as pointed out in the limitation section, this is left as future work.

[Self-recognition problem] If the strong LLM tends to rank its generation higher due to self-recognition, this is corrected by our method, as PPR accounts for possible biases of the strong LLM using the small set of $n$ pairwise comparisons by humans. In fact, in our experiments, Claude 3 suffers from the self-recognition problem and our framework corrects for such bias. More specifically, in Figure 9 in Appendix D.3, LLM CL3 prefers to rank the two Claude 1 models higher than the baseline but PPR CL3 corrects for this bias.

[Evaluation metrics] In our work, we focus on rank-sets as a measure of uncertainty in rankings and thus our experimental evaluation aims to assess the quality of the rank-sets estimated using our method and several baselines. In this context, we think that the ranking metrics proposed by the reviewer provide little information about the quality of the rank-sets. More importantly, we are unsure how to operationalize them in our setting. On the one hand, precision/recall @ k, MAP and NDCG are used in settings in which there is a set of items to be ranked and, for each item, one can measure whether an item is relevant or is not relevant. However, in our setting, it is difficult to measure whether a model is relevant or is not relevant based on win-rates. On the other hand, RBO is used in settings in which there is a ground-truth ranking. However, in our setting, we do not have access to a ground-truth ranking.

That being said, following up on the reviewer's comment, we have carried out an additional evaluation of our framework in a synthetic setting where the ground-truth ranking is known. Then, in this synthetic setting, we have computed RBO using the most probable ranking, as suggested by the reviewer, as well as the empirical coverage. Refer to the (general) author rebuttal and attached pdf for more details. We will include this additional evaluation in an Appendix in the revised version of the paper.

[Synthetic experiments with true rankings] As discussed in the previous response, during the rebuttal period, we have carried out an additional evaluation of our framework in a synthetic setting. Since in the synthetic setting, the true ranking is known, we have been able to validate the theoretical coverage guarantees without using a baseline. Refer to the (general) author rebuttal and attached pdf for more details. We will include this additional evaluation in an Appendix in the revised version of the paper.

[Value of machine labeled information in Figure 3] In Figure 3, note that, by using machine labeled information, PPR GPT4 allows us to draw the same conclusions as BASELINE using significantly fewer human comparisons ($n$ vs. $N+n$). In Figure 1, the value of the machine labeled information is perhaps more apparent. Therein, the results show that PPR GTP4 achieves narrower confidence bounds (y axis) and higher baseline intersection probability (x axis) than HUMAN ONLY, a baseline that uses the same number of human comparisons but no machine comparisons.

[Additional insights] In our work, we do not aim to make a comprehensive empirical evaluation of rank-sets as an uncertainty measure or the quality of machine labeled data. Therefore, we leave the study of the sensitivity of rank-sets to minor changes in $\mathcal{M}$, practical uses of rank-sets, and the quality of machine label data as interesting venues for future research.

[Other benchmarks] Since our computational framework is rather generic, it is true that it could be readily applied to other benchmarks. However, this would require significant funding and time, and given the on-going discussions about the reliability of LLM evaluation, we believe the NeurIPS community may benefit more from our computational framework if published early. Further, given the rapid development of both new LLMs and benchmarks, we also feel that any claim of superiority of an LLM over others based on the estimated rank-sets on (static) benchmark data may be quickly outdated and thus have limited value. Under [Baseline method], we discuss selection bias of the test set and the evaluators.

[Reasoning tasks] The goal of our experiments is to showcase and validate our computational framework, and not to make a comprehensive evaluation of LLMs across different tasks. Therefore, we do not claim that the conclusions derived from the rank-sets estimated using data from the LMSYS Chatbot Arena platform correlate with real-world performance on reasoning over long form inputs or complex reasoning tasks. We will clarify this in the revised version of the paper.

[Metrics and tasks] We agree with the reviewer that each type of task may need to be evaluated using a different set of metrics. However, since our computational framework is generic and does not make any assumption about the metrics considered for rating each response, we do not find any reason for our framework not to be applicable. That said, as discussed under [Other benchmarks], conducting experiments on other tasks and/or benchmarks would require significant funding and time and, given the on-going discussions about the reliability of LLM evaluation, we believe the NeurIPS community may benefit more from our computational framework if published early.

[Baseline method] We would like to clarify that our claim is that the baseline intersection probability is a reasonable proxy for the coverage with respect to the true rank-sets induced by the distribution of queries used by users at LMSYS and the distribution of human preferences of the users at LMSYS. In this context, we would further like to clarify that, in Appendix D.2, we have also verified that, using a more conservative baseline metric, our conclusions also hold.

Relatedly, we do acknowledge that, if the users at LMSYS are not representative of the target query distribution and distribution of human preferences, the estimated rank-sets both by our method and the baseline method may be inaccurate (due to selection bias and distribution shift). However, studying such a setting is left as future work. We will clarify this in the discussion of the limitations in the revised version of the paper.

[Few-shot prompting] We did not try few-shot prompting of the strong LLMs but instead used (almost) the same prompt used by Zheng et al. [12]. We agree with the reviewer that it would be interesting to investigate and optimize the type of prompting used to elicit pairwise comparisons by strong LLMs. However, this is a research question on its own and is also left as future work.

[Empirically test coverage guarantees] During the rebuttal period, we have carried out an additional evaluation of our framework in a synthetic setting. Since in the synthetic setting, the true ranking is known, we have been able to validate the theoretical coverage guarantees without using a proxy metric. Refer to the (general) author rebuttal and attached pdf for more details. We will include this additional evaluation in an Appendix in the revised version of the paper.