Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

We thank Reviewer [ hqxB] for their positive feedback including observing our “detailed” and “appropriate” experiment setup as well as noting the breadth of ablation including “sequence arrangements, the number of games, the K value, and win rates”. We appreciate the comment that the rigor of the analysis provides “good reliability” in the findings. We take this opportunity to address R hqxB helpful feedback and questions below:

This paper explores the experimental results with up to three players. It would be more meaningful if the conclusions could be further generalized to settings with more players.

We thank the reviewer for this suggestion. Introducing each new model necessitates a comprehensive comparison with all previous models, which can quickly become overwhelming. To provide an overview of the landscape, we targeted a wide range of models with different architectures (encoder-decoder and decoder-only), training data (public, private), and model sizes (3B to 100s of billions). Additionally, our simulated experiments are devoid of any model artifacts and potential biases. This approach helps contextualize and establish our findings within the real world. Lastly, in the shared response above, we added analysis of a recently released dataset including 20 models (such as GPT-4, Claude-v1, and Palm-2) and observed confirmation of our findings in the manuscript. We will update the final manuscript with these results.

In reality, Nperms can be set very large without obvious computational cost to stable the evaluation results. Therefore, in multiple experiments, I would prefer to see results with settings like Nperms = 10k, similar to those in Figure 2.

We agree with the reviewer that the cost of large Nperm is offline and we can increase this number to an extent. We compute and update Elo ratings for "$N_{\text{pairwise games}} \times N_{perm}$" instances. As the Number of pairwise games ($N_{\text{pairwise games}}$) grows by increasing the number of models and the number of prompts in the dataset, running a high number of permutations will slow down the evaluation process. More importantly, our study shows that we do not need to run unnecessarily high numbers of permutations. As shown in Figure 2 in the paper, we observe stable results with $N_{perm} = 100$ with minuscule standard errors of the mean and conclude that there is no need to run a higher number of permutations. With the new Chatbot Arena dataset added in the shared response section of the rebuttal, we have 33k prompts and observed stable results with $N_{perm} = 1000$.

We hope that we were able to clarify the reviewer’s questions. If so, we ask that you consider raising your score to reflect. If you have any further questions or concerns during the rebuttal period, please let us know. We are happy to provide any necessary clarifications.

We thank Reviewer [ NZSD] for their helpful feedback and observations, noting that our study is “an interesting” and “meaningful research topic” and find our axiom definitions “reasonable and useful”. We take this opportunity to address R NZSD concerns and feedback below.

The writing is not very friendly to the reader unfamiliar with the Elo rating. In line 93 you mention the K factor, but it is hard to understand for the reader unfamiliar with the Elo rating.

Thank you for your feedback. We agree that this could be more kind to the reader, we will provide a more in-depth introduction to the Elo rating system in the final manuscript, including the impact of the $K$-factor on it.

How can the Elo rating be generalized to multiple-task evaluation?

This is indeed an exciting question. One method would be to just run experiments per task and obtain an “Elo per task" rating and then average these ratings across different tasks. We want to note that even the single task setup currently used with LLMs is complex enough since the "game" (prompt) changes every round and the win condition is subjective (due to human judgment). However adding a multi-task angle to capture the complexity of various domains and human feedback would be interesting. Lastly, there are more sophisticated rating methods such as Trueskill that can be useful for such scenarios. TrueSkill [1] is a Bayesian ranking system that can handle multiple tasks using a vector of skill parameters, each representing a different task.

Why not conduct experiments on more widely used API-based LLMs, such as ChatGPT, Gemini, and Claude-3?

We thank the reviewer for this question. To start, we want to explain how the core of our findings are model-agnostic. Relative differences and win rates are all that matter when we are studying the dynamics of Elo. With introducing more powerful models, the results could be reduced to one of the two cases we already have: model pairs that are close in performance or model pairs that have a large performance gap. The only new dynamics that may occur with many SOTA models is saturating Elo scores at the top. But that isn't very relevant here since it's easy to still find prompts that all LLMs are not good at so we haven't found a saturation point yet. Introducing each new model necessitates a comprehensive comparison with all previous models, which can quickly become overwhelming. The primary reason for not using API-based LLMs is their potential for behind-the-scenes updates that can alter performance over the evaluation time and impact replicability [2]. Hence, for this treatment, we restricted consideration to open weights which were highly performant and we have control over when collecting preference feedback over a period of time. Lastly, to cover a larger model list we looked at the Chatbot Arena dataset with 20 models and present the findings in the shared response of this rebuttal. We confirm our previous findings with this dataset.

[1] Ralf Herbrich, Tom Minka, and Thore Graepel. 2006. TrueSkill™: a Bayesian skill rating system. In Proceedings of the 19th International Conference on Neural Information Processing Systems (NIPS'06). MIT Press, Cambridge, MA, USA, 569–576.

[2] Luiza Pozzobon, Beyza Ermis, Patrick Lewis, Sara Hooker. 2023. On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research

We hope that we were able to satisfy the reviewer’s asks by running additional experiments on a much larger dataset and clarifying the details in question. If so, we ask that you consider raising your score to reflect and please do let us know if anything requires further clarification.

We thank Reviewer [9SNU] for their helpful feedback and their observations noting that our study is “important” and the experiments are “extensive”. We also hugely appreciate the reviewer’s note that this paper provides “several interesting lessons for using Elo ratings for LLMs” that “can (positively) influence how we compare models”. We take this opportunity to address R 9SNU helpful feedback and questions below:

Table 1 is somewhat difficult to read, why not make a plot similar to Figure 1 but with multiple lines for the different models?

We thank the reviewer for the helpful feedback to make this table more readable and understandable. We will take the great suggestion from this reviewer into account to update the manuscript. To clarify, the main takeaway from Table 1 is ​​the red star values for which we observe the cases where the Elo score fails to accurately reflect the expected hierarchy of models.

In the introduction -- implications of our work -- it is listed that the work is particularly important in situations where model performances are closely matched, which is a common occurrence in many real-world scenarios, whereas in 4.2 it is listed that the result is relevant because skewed win-rates are common in real-world scenarios (or maybe that is not what is meant, if not I am not sure why it is mentioned that there are skewed win-rates in real world scenarios)

Thanks for pointing out this unclarity in our manuscript. We believe that both scenarios (closely matched models and skewedly performing models) are equally important and they both occur in real-world model comparisons as is evident on the Chatbot Arena ranking. Our paramount goal in this work is to study the reliability and robustness of comparison methods such as Elo across all scenarios. We will clarify this detail in both the introduction and section 4.2.

It is not clear with what K-factor the main experiment (4.1) is conducted (or at least I couldn't find it)

Thanks for pointing out this missing point. We use $K=16$ for all experiments where we don’t explore different values of $K$. We will make this clear in the manuscript.

It is not clear with what K-factor the main experiment (4.1) is conducted (or at least I couldn't find it) in 4.1 it is stated that for P(A beats B) >= 0.6 Elo results are stable; I can't match this result with Figure 1 (middle column), where there seems to be quite some variation for P(a beats B) = 0.55 as well as P(a beats B) = 0.6*

We thank the reviewer for giving us the opportunity to clarify the results presented in Section 4.1.

• $P(A \text{ beats } B) > 0.6$: The Elo results for models where the win probability is $P(A \text{ beats } B) > 0.6$ are presented in Figure 6 (and not Figure 1). There we observe that with both one sequence and 100 permutations (row 2 and row 3 respectively in Figure 6), the Elo convergence is stable, i.e., the rankings of the models don’t change over the number of games played.

• $P(A \text{ beats } B) = 0.6$ and $P(A \text{ beats } B) = 0.55$: In Figure 1 we observe a slight instability with one one sequence but no change of model orders with increasing the number of games ($N_{perm} = 100$): model A achieves a higher Elo score with the first few games and stays on top as we increase the number of games.

• $P(A \text{ beats } B) = 0.51$: we observe a different pattern where depending on how many games we conduct the model order changes: at 100 model B is better than A and at 250 model A is better than model A. We make sure that the Figures 1 and 6 are consolidated and the description of the figures are more clear in the final manuscript.

The idea of comparing a single sequence with 100 different orderings is generally strong, but doing only one single sequence and one series of permutations is not very informative in this [...]

Reviewer [9SNU] raised a valid concern about the potential bias from comparing one sequence with $N$ permutations, as the single sequence could be unusually lucky or unlucky. To address this, we conducted new experiments using a dataset with 33k prompts, as detailed in the shared response section above.

In 4.2 it is listed that faster convergence is observed for higher K-factors, but I don't understand the evidence for that. I don't think this is visible from Figure 3, and I couldn't find an appendix plot either.

We would like to thank the reviewer for pointing out this vagueness in Figure 3. We will clarify the convergence details in this figure by adding the time dimension and showcase the speed of convergence more clearly. To explain the current figure we don’t currently show the traditional notion of convergence, but rather show stability. Here we present the final Elo score differences $(SA − SB)$ for different parameter values (different $K$-factors and win probability scenarios). For all matches model A beats model B (albeit with different probabilities of win) and a positive value for this difference is desired. For larger $K$ values, we observe larger score differences $(SA − SB)$ indicating that the system is more stable and confident in predicting model A is better than model B. Nonetheless, we will ensure that the final manuscript is clear on this detail.

If I understand correctly, the number of permutations is the number of game orders, why not use a word that matches that closely, e.g. 'game orders' or 'orderings'?

That is true. We mainly use permutation because of its convention of use as a mathematical definition to describe a number of orderings and common use in combinatorics. We will use “game orders” and will define this term more explicitly and clearly in the final manuscript.

We hope that we were able to clarify the reviewer’s questions and concerns. If so, we ask that you consider raising your score to reflect. If you have any further questions or concerns during the rebuttal period, please let us know. We are happy to provide any necessary clarifications.

We thank Reviewer [ jqvp] for their helpful feedback and observations, noting that our study “offering valuable insights for future Elo rating applications”. We thank R jqvp for noting the comprehensive nature of the evaluation, “on both synthetic and real human feedback”, and involving many ablation variants such as order of matches and K-Factor. We take this opportunity to address R jqvp concerns and feedback below.

My biggest concern is the relatively small amount of real data used in the experiments. Only 500 prompts and three model pairs are included, with just one pair having similar performance (Dolly-v2-7b vs. Dolly-v2-12b). Given that Chatbot Arena currently includes dozens of LLMs and has released a large amount of preference data, the volume of real data in this study seems insufficient.

We thank the reviewer for the suggestion and agree that additional data will strengthen our findings. We ran additional experiments during this rebuttal to include this new dataset (see shared response above for results).

As the authors mentioned, not considering ties appears to simplify the widely used "win/tie/lose" practice.

Thanks for the comment. In the case of a tie, the adjustment in Elo rating points is typically smaller, as ties are considered less informative about the relative strengths of the models. Ties are useful to upper and lower bound the pool of models (with very hard prompts that lead to "both bad" ties and vice versa) and otherwise they slow down the eventual convergence to true rankings.

The empirical guidelines provided may have limited applicability since most papers on automatic evaluation of LLMs report win rates rather than Elo ratings. Also, running more than 100 permutations seems costly and lacks practicality.

Reporting win rates for model evaluation is only feasible when we have a small number of models to compare because we need to play every two combinations of models against each other. Additionally, to get a reliable signal of performance, they are often run on various benchmarks and a large number of prompts. The advantage of the Elo score is that we can obtain a ranking of a large number of models and position a newcomer model well in this relative ranking with only a few pairs of games instead of covering every head-to-head evaluation. We note that widely used leaderboards such as Chatbot Arena [1] use Elo due to these advantages, so our study is timely at illustrating the limitations.

If I want to automatically evaluate k models to get their final Elo ratings, do you think the size of k will affect the robustness of the Elo rating?

Yes, we believe so. With a larger k we increase the reliability of the Elo rating because this system has more information to work with. More head-to-head comparisons help smooth out anomalies and provide a more accurate representation of each model's true performance. Elo ratings might converge more slowly because each model must be compared against a larger number of other models. However, once they do converge, the ratings are likely to be more accurate. A large and diverse prompt pool will also stabilize the Elo scores, especially regarding real-world performance. If the prompt pool used for Elo diverges from IRL prompts, the Elo scores will not accurately reflect actual performance. We will update section 4 in the manuscript to include this detail.

Why does Dolly-v2-7b have a slightly higher win rate than Dolly-v2-12b in Table 2?

While surprising, this is consistent with the evaluation performance self-reported by the model authors over 7 benchmarks: OpenBookQA, ARC (easy), WinoGrande, HellaSwag, ARC (challenge), PIQA, and BoolQ. In the model cards, it’s shown that the average performance of the 7B model and 12B model is 0.573 and 0.567 respectively [2].

[1] https://arena.lmsys.org/ [2] https://huggingface.co/databricks/dolly-v2-7b

We hope that we were able to satisfy the reviewer’s asks by running additional experiments on a much larger dataset and clarifying the details in question. If so, we ask that you consider raising your score to reflect and please do let us know if anything requires further clarification.

Now that the discussion is underway, we wanted to ask Reviewer jqvp if there are any follow-up points we can clarify. If there are no further points of clarification regarding the manuscript and size of the dataset (with the new experiments shared here: https://openreview.net/forum?id=Pc9LLjTL5f&noteId=Kv4lOJURpk), the practical applicability of our guidelines, and the impact of the number of models on robustness, we would ask that reviewer Reviewer jqvp consider increasing their score to reflect the additional experiments, improvements and clarifications we have provided. We are very happy to continue to engage and answer any questions.