Re-evaluating Open-ended Evaluation of Large Language Models

> Q9: Around line 60, why is this setting a necessity?

We believe this is a misunderstanding. On L59 we italicised “adversarial” and the “necessity” refers to the need for Balduzzi 2018 [1] to restrict themselves to a two-player zero-sum gamification. We did not argue that game-theoretic evaluation is a necessity but merely that Balduzzi 2018 modelled evaluation as a two-player zero-sum agent-vs-task game out of necessity as max-entropy Nash equilibrium is no longer unique beyond this setting.

We thank the reviewer for pointing out this source of confusion and we have revised our writing in the latest revision.

> Q10: This paper uses gemini-1.5-pro as the judger. Why not more powerful models?

The main reason is that the Gemini Pro model is relatively affordable and reasonably capable at the time of our experiment. We are aware that alternative models exist and that LLM judges exhibit self-preferences and we have caveated accordingly in L200-L203.

> Q11: The text repeatedly mentions skill, skill entropy. What exactly is the definition of skill in this paper?

We believe discussions around the notion of “skill entropy” are primarily restricted to Section 1.1 where we proposed a simulated thought experiment that serves to demonstrate the potential danger of chasing after ever-higher Elo ratings. Please see L88-L90 for the setting of this simulated experiment for a definition of a skill.

In real-world LLM development, model developers may optimise models for other considerations than improving on LMSYS Elo ratings and users who submit prompts may not solely focus on separability. As such, we believe a simulated thought experiment can provide insights more clearly and help draw attention towards more principled interpretation of ever-increasing Elo scores.

To summarize, Section 1.1 shows that equilibrium ratings promote well-rounded model improvements across skills whereas models can always improve their Elo ratings by improving on a single skill dimension that is over-represented in the prompt set.

[1] David Balduzzi, Karl Tuyls, Julien Perolat, and Thore Graepel. Re-evaluating evaluation. Advances in Neural Information Processing Systems, 31, 2018.

[2] Luke Marris, Marc Lanctot, Ian Gemp, Shayegan Omidshafiei, Stephen McAleer, Jerome Connor, Karl Tuyls, and Thore Graepel. Game theoretic rating in n-player general-sum games with equilibria, 2022b. URL https://arxiv.org/abs/2210.02205.

[3] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024a.

> Q6: Figure 5 is confusing…

We have now limited the visualisation to show the top-5 models only and we agree the earlier version is overwhelming and significantly simplified Figure 5 by grouping rebel models into family of models which revealed interesting game dynamics. We have also performed the same visualisation in Appendix D in a game of rock-rock-paper-scissors to help provide intuition.

Intuitively, each king model receives a scalar equilibrium rating, which can be explained by and is a sum of contributions by each rebel model. The rebel model making the most negative (positive) contribution is the one that the king model performs the worst (best) against relative to its equilibrium strategy.

> Q7: How much advantage does this evaluation method have over existing methods (such as various Elo-based methods and previous game-theoretic methods mentioned in the related works section)?

Our proposed methods are invariant to redundancy; prior Elo-based methods and game-theoretic methods are not. We do not have a quantitative measure of quality for rating methods. However, we believe a good rating method should be one that is suitable for the community to hillclimb on.

We argue that Elo-based ratings are not:

1. it can be manipulated by introducing redundant prompts (Figure 3) and;
2. it does not lead to well-rounded improvement (Figure 1). Figure 3 (Right) also show that naive equilibrium ratings are not clone-invariant [1, 2].

We argue that our equilibrium rating methods, combined with our definition of affinity entropy, provides ratings that are robust to redundancy and as a result, lead to well-rounded improvements on diverse skills.

> Q8: The dataset is too small at 500 prompts.

To the best of our knowledge, ArenaHard 500 is the largest publicly available prompt set for which we have model responses from a diverse set of 17 models for each prompt. This lets us pairwise compare all models on all prompts using an automated judge LLM.

We believe the size of the dataset is sufficient to make the point that our method is clone-invariant and provides intuitive rankings of models. However, we agree with the reviewer that 500 is too small to suggest that our method can scale. If the reviewer is concerned about our method being able to scale to larger prompt sets, we can include a runtime analysis for a much larger (synthetic) dataset in a follow-up revision.

Nevertheless, please let us know if there are other datasets we should be aware of.

> Q2: The differences between the method in this paper and other rating methods, other Elo-based methods, and other game-theoretic methods are not explained in detail.

In the Related Works section we have made references to Elo-based rating methods, methods based on Social Choice Theory and methods based on game-theoretic solution concepts. The differences between our approach and Elo-based methods are discussed throughout our experiments. The differences between our work and [1] has been discussed in detail in L50-L60.

Nevertheless, we recognize that we can be more specific in our discussions in the Related Works section and we have revised accordingly in our latest revision.

> Q3: What specific shortcomings of the Elo-based method does it aim to address? Is it the robustness and sensitivity to redundancy mentioned in the abstract?

Yes.

Section 1.1 (L135-143) suggests a non-obvious failure mode of hill-climbing on Elo scores due to their sensitivity to redundancy. If the set of openly crowd-sourced prompts over-represent one capability over another, then we might not see well-rounded improvement of models since specializing on the over-represented skill can be more effective.

> Q4: what are the essential advantages of this over traditional LLM-eval-based methods (such as AlpacaEval)

We hope our clarification and revision in the manuscript would help address this point of confusion. AlpacaEval family of works are not, according to our interpretation, open-ended evaluation. Specifically, the set of evaluation prompts are curated by the authors and we trust the authors would apply reasonable precautions to ensure a certain level of quality of the evaluation prompt set (e.g. removal of exact or near redundant prompts). In this case, ranking by average win-rate can be effective.

Our method offers additional benefits beyond redundancy too:

1. it provides quality ratings of prompts (see Figure 4) and;
2. it provides interpretable ratings (see Figure 5 and Appendix D-E which shows how one player’s actions contribute to another player’s action ratings).

Indeed, these benefits could be helpful in the AlpacaEval setting too to highlight discriminative prompts and to understand the comparative strengths and weaknesses of each model.

> Q5: Figure 3 is confusing…

Figure 3 shows that one can intentionally and arbitrarily manipulate the Elo rating of a model (in this case gemini-1.5-pro) by introducing replicas of prompts where gemini-1.5-pro performed poorly. Note that for LMSYS evaluation, this type of attack is in principle feasible by a motivated actor. Our equilibrium ratings are invariant to such manipulation. NE(-a) [1] and CCE(-a) [2] show the necessity of the affinity entropy we introduced.

Reading from the top of Figure 3, it suggests that gemini-1.5-pro remains top-ranked under NE ratings and CCE ratings. The dotted line shows it is also top-ranked according to Elo ratings. However, its rating decreases arbitrarily, as we duplicate prompts where gemini-1.5-pro performed poorly.

We hope this helps and we have revised the Figure caption for clarity.

We thank reviewer 56qC for recognising the relevance and urgency of our work. We also thank the reviewer for highlighting several points of confusion which has helped us improve our manuscript.

We would like to start by clarifying what we meant by open-ended evaluation. We believe the reviewer understood the notion of “open-ended evaluation” in the sense that LLMs exhibit open-ended language capability. This is not our intention. In fact, our method applies more generally, for example, to the setting of RL agent evaluation by Atari games, as in [1]. Instead, open-ended evaluation refers to evaluation systems such as LMSYS where the set of prompts (equiv., environments/tasks) or models (equiv., agents) come from open-ended user submission: anyone can freely submit evaluation prompts with minimal gatekeeping from the LMSYS designers. Note, this is not how evaluation systems have worked traditionally; MNIST, ImageNet, Kaggle, Atari environments, etc. are all carefully curated and controlled evaluation benchmark sets. This new open-ended evaluation setting requires new thinking and tools.

As such, we argue that if anyone can submit a large number of redundant or near-redundant prompts, the rating methodology ought to be robust against intentional or accidental redundancy. We then demonstrate in Figure 3 that the Elo rating system does not have this property and we can manipulate the rating of the Gemini model arbitrarily.

It is also in this sense that game-theory is a natural direction to investigate: having 5 copies of rock in a rock-paper-scissors game does not change the fact that at an equilibrium, each action is still played ⅓ of the time and all actions would receive an equilibrium rating of zero whereas the paper action would look like a much better choice if Elo ratings are used.

> Q1: The Background section is too general; … The Related Works section also fails to fulfill its intended purpose and function in the whole paper. …

Although we have motivated our work in the context of LLM evaluation, with LMSYS one of the most prominent open-ended evaluation systems, we believe our work should be compared to prior works that propose rating methodologies rather than their applications in a specific domain --- our method is not restricted to LLM evaluation nor does it require LLMs in its use. For instance, in a rock-paper-scissors game, our method equally applies, as we have done so in Appendix D to provide intuition. It is for this reason that the description of the 'king-of-the-hill' game is relegated to the method description, as it’s only an example of gamification that can and should be reconsidered for different application domains.

Nevertheless, we agree with the reviewer that the Alpaca papers should be cited, given their prominence in the broader LLM evaluation landscape and we have cited them in our latest revision.

> Q3:... Could it be beneficial to consider including more benchmark comparison methods, such as other evaluation systems based on game theory or related equilibrium solutions

We have indeed thought about several other baselines but we concluded that Elo remains the most relevant method to compare against due to its widespread use. We have therefore designed our experiments around different failure modes of Elo, owing to its sensitivity to redundancy and the limited information it reflects in terms of the “game dynamics” in the evaluation data (i.e. as opposed to the equilibrium structure underlying our proposed ratings, which we have exploited for Figure 5).

Other possible baselines we have considered include:

• Nash averaging: [1] proposed to use expected payoffs of each action at the max-entropy NE (MENE) but MENE is only unique in two-player zero-sum games. We did not compare to [1] as computing MENE is difficult in n-player general-sum games and would not be unique (see Appendix E);

• (C)CE-ratings: [2] proposed computing action ratings at the ME(C)CE of the game. This approach is akin to the setting of CCE(-a) as shown in Figure 3, which we showed to be affected by redundancy. Our max-affinity-entropy criterion provides a practical CCE rating method that remains invariant. Using (C)CE for ratings as a direct alternative to Elo is also open to debate, as it can assign high ratings to specialist models that excel at a (small) subset of the prompts. By contrast, both Elo and NE ratings tend to highlight strong generalists;

• LLE-ratings [2]: the authors of [2] further hinted at, but did not experiment with the possibility of using LLE for computing a set of NE ratings (recall that LLE defines a unique NE at the limit).

• Social Choice Theory [3]: there has been recent interest in using social choice methods for evaluation. However, the invariance to redundancy is usually given among candidates (i.e. models, if we consider prompts to be individual votes) whereas our methods provide invariance to redundancy for all players. SCT methods, as in the case of Elo scores, also do not provide additional insights regarding the importance of prompts;

• AlphaRank [4]: in theory, AlphaRank could be a reasonable alternative but we find it challenging to scale in practice due to its excessive memory requirements. Specifically, the transition matrix is quadratic in the number of joint actions ((500 x 17 x 17)^2 elements for the size of the game we study).

Overall, the current practice in the evaluation of LLMs is heavily centred around the Elo rating system and we believe a careful discussion of the practical limitations and failure models of Elo would be a timely contribution to the community.

A good rating system should scale to practical problems and should be non-trivial to manipulate. As we have shown in Figure 1, improvement against an equilibrium rating system leads to well-rounded improvement. In Figure 3, we have shown that our rating system is invariant to exact (and in Supp. Figure 11, near) redundancy. In Figure 5, we have shown that our approach provides an interpretable view of the comparative (dis-)advantages of different models. To the best of our knowledge, our proposal is the first to satisfy these desiderata simultaneously.

[1] David Balduzzi, Karl Tuyls, Julien Perolat, and Thore Graepel. Re-evaluating evaluation. Advances in Neural Information Processing Systems, 31, 2018.

[2] Luke Marris, Marc Lanctot, Ian Gemp, Shayegan Omidshafiei, Stephen McAleer, Jerome Connor, Karl Tuyls, and Thore Graepel. Game theoretic rating in n-player general-sum games with equilibria, 2022b. URL https://arxiv.org/abs/2210.02205.

[3] Marc Lanctot, Kate Larson, Yoram Bachrach, Luke Marris, Zun Li, Avishkar Bhoopchand, Thomas Anthony, Brian Tanner, and Anna Koop. Evaluating agents using social choice theory. arXiv preprint arXiv:2312.03121, 2023.

[4] Omidshafiei, Shayegan, et al. "α-rank: Multi-agent evaluation by evolution." Scientific reports 9.1 (2019): 9937.

We thank the reviewer rQkm for recognising the importance, novelty and careful design of our work as well as providing constructive feedback.

> Q1: LLE is also an important concept, yet it is only mentioned in section 3.2, causing some logical confusion.

This is an excellent remark and we apologise for its omission. Indeed LLE is a solution concept that refines the more popularized Nash equilibrium and a key element behind the validity of our approach. LLE is unique, has connections to the Nobel prize winning notion of risk-dominance and benefits from a rich body of literature in psychological studies and behavioural sciences that supports its intuitive nature.

We have moved its discussion earlier to the Background section and we hope this would help readers follow the design choices made around LLE in our method sections.

> Q2:... "p" represent in the context of Affinity Entropy… needs more explanation

The hyper-parameter p, sometimes called the entropic-index, controls the shape of the Tsallis entropy function, most importantly the steepness of the gradient of entropy near pure strategies. As p approaches zero, Shannon entropy is recovered and the gradient becomes infinitely steep at the boundaries. Note that we have updated our definition of Affinity Entropy with a small modification; it is now concave for all nonnegative p.

We thank the reviewer rQkm for recognizing the relevance of our work and for their constructive criticism.

> Q1: … expanding this section with additional intuitive explanations and also expanding some of the descriptions in the appendix.

This is an excellent remark and we have now included an additional analysis using a toy game of rock-rock-paper-scissors to help build intuition in Appendix D. We have also included in the Background section a definition of the LLE solution concept to better set the stage for the rest of the method section.

The technical complexities of our method primarily reside in the definition around different game-theoretic solution concepts, equilibrium computation and their selection. While most techniques employed in our rating methods have been discussed in prior works in game theory [1, 2], the definition of the affinity entropy is new and bridges the gap between game-theory and in-the-wild LLM evaluation.

Appendix E has an example Chicken game where the equilibrium that is of max-entropy would be different from the one with max-affinity-entropy. We hope this helps provide intuition as to how similarity kernels would react to redundancies via a concrete example.

> Q2: Figure captions could be improved by concluding with important takeaways.

We agree with the reviewer and added brief concluding summaries to figure captions where appropriate.

[1] David Balduzzi, Karl Tuyls, Julien Perolat, and Thore Graepel. Re-evaluating evaluation. Advances in Neural Information Processing Systems, 31, 2018.

[2] Luke Marris, Marc Lanctot, Ian Gemp, Shayegan Omidshafiei, Stephen McAleer, Jerome Connor, Karl Tuyls, and Thore Graepel. Game theoretic rating in n-player general-sum games with equilibria, 2022b. URL https://arxiv.org/abs/2210.02205.

With the discussion period coming to an end, please do let us know if our responses and revision to the main text have adequately answered your earlier questions or if we can help clarify any further. Thank you!

We thank reviewer v6QS for recognizing the importance and urgency of the problem we tackle.

> Q1: … especially for those fine-tuned from the same foundational model.

In particular, we appreciate this insight which we should highlight more in our writing as there are (increasingly) many models that are fine tuned from the same foundational models (e.g., Llama models) --- performing well against a specific, large family of redundant models would be one way to inflate redundancy-sensitive ratings such as the Elo scores.

> Q2: … it would be better to provide some information on redundancy rather than directly applying the inferred results.

We agree with the reviewer. Indeed, we have updated Figure 4, which now visualises a game instance with artificially introduced redundant prompts, demonstrating how redundant actions would be handled. We observe that redundant actions would split their probability mass at the equilibrium we select, explaining why redundant adversarial prompts had no effect on equilibrium ratings as shown in Figure 3.

We would like to clarify that although we focused our presentation on the specific use-case of side-by-side LLM comparison, our proposed rating framework is easily generalised in several ways.

> Q3: … concerns about the focus on the utility of separability.

On gamification: the game design (i.e. utility functions, number of players, …) depends on the applications and the behaviors we would like to see at an equilibrium. The separability utility function we chose would lead to model ratings that focus on performance on prompts discriminative among strong models. Other game designs could be equally valid and valuable as the reviewer pointed out.

> Q4: … Coarse Correlated Equilibrium (CCE), which is more accessible compared to the commonly used Nash Equilibrium (NE)

On solution concept: which solution concept to use depends on downstream applications. If we want the model ratings to reflect strengths of a model in any strategic niche, then the CCE solution concept could be preferred. If we want model ratings to reflect how generally capable each model is, then NE ratings would make more sense. Our rating framework leaves room for practitioners to lean one way or another depending on applications.

> Q5: How can we identify redundant prompts or models? Would it be necessary to include human verification of redundancy?

On redundancy: how we define redundant prompts or models is flexible, and in fact, this flexibility to re-interpretation of redundancy is precisely a strength we designed for in desired property P4 (section 3.3). In experiments, we chose to call two prompts similar if they rank all model pairs identically (see distance matrix D in equation 5). However, for example, distance matrix D in equation 5 could instead be defined to be the expected squared difference between payoffs + the edit-distance between the two prompt strings or cosine-distance between their embeddings.

To clarify this distinction further, we are not arguing one way or another how to correctly measure prompt redundancy. It would make sense to verify a definition of prompt similarity with human raters prior to passing this definition to our method. We are proposing a game-theoretic technique that rates models/prompts in a way that respects a user-provided definition of redundancy (and also approximate redundancy). In that setting, our proposed approach is uniquely capable of handling redundancy.

> Q6: … Scalar rating evaluations still suffer from intransitivity issues, as seen in Rock-Paper-Scissors scenarios.

Our equilibrium ratings account for intransitivity and we could indeed make it clearer in writing. We have now introduced a new visualisation of our rating method applied to the Rock-Rock-Paper-Scissors game in Appendix D. All four actions: Rock, Rock, Paper, Scissors would receive identical ratings under our rating framework, with an interpretable contribution plot (similar to Figure 5) explaining their ratings. For instance, we see the paper action receives a rating of 0 because if it were to play paper instead of its equilibrium distribution (uniform over R/P/S), it would lose and win in equal proportion against scissors and the two rocks respectively.

> Q7: Are there risks in excluding some common evaluation prompts, even if they seem redundant?

This is a great question. There are many equilibria in real-world evaluation games that we construct. Indeed, we have empirically computed 128 Nash equilibria in Supplementary Figure 6. For CCE, there’s typically a continuous polytope of (infinitely many) equilibria.

As such, which equilibrium we select matters and some of these equilibria would indeed unduly ignore “common prompts” that, for example, rank the overall stronger models high and weaker models low. However, these do not tend to occur with our selection procedure. For the NE solver, we propose to select the LLE of the game which tends to be risk-dominant. For CCE we propose to select the max-affinity-entropy CCE. Both procedures tend towards highly-mixed prompt distributions, if such an equilibrium exists.

In other words, our method is robust against (exact or near) redundancy, but would not unduly ignore “common prompts”. The degree to which two prompts are considered similar is also controlled by the users via the similarity kernel: using a small RBF kernel variance would guard against redundancy in the most strict sense, which is what we did in experiments (we used a kernel variance of 1e-6, see Supp. Section E.2).

> Q8: Related works on game balance research.

Indeed this line of work becomes relevant once we consider LMSYS evaluation in the context of game design. The equilibrium prompt distribution would in effect re-adjust the prompt distribution such that no single topic overshadows others. This is similar to the pattern shown in the "NE prompts" column in Figure 2. We have revised our Conclusion section accordingly.