Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation

We thank the reviewer for their detailed review and insightful questions. We are happy to hear that they find our work provides a solid motivation which Polyrating directly addresses and that they appreciate our theoretical analysis. Below, we address their remaining questions.

Why did you not include other multivariate rating systems in your experiments?
We did not include other multivariate rating systems in our experiments because they fundamentally differ from Polyrating in how they handle bias and model-specific parameters, making them categorically unsuitable for the experimental results we obtain.

Traditional multivariate rating systems are all based on approximative algorithms like Elo and Glicko, which were designed to handle time-dependent ratings of millions of objects or people. Thus, relying on these rating systems to obtain task-specific ratings would significantly reduce effectiveness compared to the optimal MLE-based ratings. Fig. 4 in Appendix A shows just how poor these approximations work in our LLM-based setting: the loss associated with Glicko (an improved version of Elo) is more than an order of magnitude higher compared to the MLE-based ratings! In contrast, Polyrating provably converges to the same ratings as the MLE-based one, not losing any utility in the rating. Furthermore, the implementations of these traditional multivariate rating systems require non-trivial adjustments to apply in the LLM-based setting due to their time dependence and various other differences that do not translate.

Length-Controlled Alpaca-Eval is the only system based on the Bradley-Terry (BT) model like Polyrating that can also incorporate various factors in its modeling. However, it misses several key features which make it inapplicable for the experiments we performed. First, it only includes model-pair-specific parameters without model-agnostic or model-specific parameters. This makes it categorically impossible to obtain task-specific ratings with it. This restriction also makes it impossible to arrive at single-parameter estimates for the overall bias impact, a core requirement for our bias detection experiments. Furthermore, it does not contain any priors on parameters. This means that even if it were adjusted to incorporate model-specific parameters, it would still be mathematically equivalent to run the univariate baseline as there is no coupling between the rating of two tasks.

We have now clarified these points in our Related Work section.

How are sample efficiency improvements (e.g., "38%" in line 361) calculated?
We measure this efficiency improvement by first obtaining the loss associated with Polyrating ratings at 10,000 samples. We then compute the lowest number of samples the univariate baseline requires to match this loss. For instance, in the case of 38%, the univariate case requires 16,130 samples to obtain the same loss since $(100 - 38)\\% \cdot 16,130 = 10,000$. Generally, for less than 10,000 samples this efficiency improvement would be larger, while for more samples the efficiency improvement would be smaller. We chose 10,000 samples as a good representative of a reasonably sized dataset that one could obtain.

We have clarified this point in the paper when we first introduce these numbers in the experiments.

Why are different optimization methods (Newton's method and L-BFGS) used for different parameters?
After testing various optimization methods, we found that they converged to the same ratings, but at different rates. Specifically, Newton’s method showed faster convergence for model-specific parameters, while L-BFGS performed better for model-agnostic parameters. Therefore, we chose to use the faster method for each parameter type to optimize the efficiency of our rating system.

What specifically makes Mixtral's performance different for the Chinese and English tasks in Table 3?
By "different," we mean that Mixtral’s performance on Chinese and English tasks deviates significantly from its base performance, an effect that is due to Mixtral’s relatively narrow multilingual focus compared to some other models. In particular, Mixtral’s ratings reveal a notable gap: it performs significantly better on English tasks and significantly worse on Chinese tasks than it does overall.

However, the univariate baseline overlooks these task-specific variances, assigning a fixed rating to Mixtral across all tasks. In this case, it artificially lowers Mixtral’s rating (and therefore all model ratings) by around 25 points for English tasks while raising it by approximately 37 points for Chinese tasks. This shift causes inaccuracies in model comparison across tasks. In contrast, Polyrating accounts for such variability, preserving Mixtral’s true performance differences across tasks. Without Polyrating, these nuanced performance differences would be masked, limiting the interpretability of Mixtral’s ratings in both Chinese and English contexts.

We thank the reviewer for their detailed review and insightful questions. We are pleased that they found our paper easy to follow, Polyrating effective with meaningful real-world applications, and appreciated our comprehensive experiments. We will address their question regarding applicability across domains in our response. If the reviewer has further concerns or suggestions for improvement, we would be happy to consider them. Otherwise, we would appreciate consideration for an improved score.

Will Polyrating also be applicable to domains other than large language models such as computer vision, information retrieval, etc.? What is the cost of the adaptation to these domains?
Yes, Polyrating is applicable to other domains, including multimodal models and domain-specific tasks like information retrieval, as long as preference data (outcomes and judgments) is available. We have also successfully applied Polyrating beyond AI evaluation, such as in chess.

To adapt Polyrating to new domains, only minor adjustments are needed:

• Data Availability: We used open-source preference datasets for LLMs in our work. For other domains (e.g., multimodal), open-source datasets on the same scale do not yet exist, though Polyrating requires no modifications. Thus, creators of such preference datasets for their specific application can directly apply Polyrating.
• Custom Factors: Domain-specific factors may need to be defined (e.g., type of visual content in multimodal models). These factors depend on the specific use case for which Polyrating is applied. However, the manual work involved in this feature specification is limited to negligible.

These simple adaptations allow Polyrating to retain its core benefits—improving sample efficiency and reducing evaluation costs—across various applications.

We hope that this addresses all the reviewers' questions and concerns. We are happy to discuss any further questions the reviewer might have.

We thank the reviewer for their detailed review and insightful questions. We are happy to hear that they consider our paper well-written and a good practical contribution and that our experiments are comprehensive and show significant improvements. Below, we address their remaining questions.

What is the computational complexity of the optimization process and how does it scale with the number of models and tasks?
The optimization process for Polyrating requires no more than 6 hours on a single CPU to handle a substantial dataset consisting of one million samples, 100 models, and ten tasks. Consequently, the computational cost of running Polyrating is negligible when compared to the expense of acquiring such a dataset. Moreover, the process is easily executable on commodity hardware, making it highly accessible.

We also experimented with varying the number of models and games but did not observe a clear trend in computational complexity. This lack of a clear trend can be attributed to several implemented optimizations, such as grouping games with identical features into a single term within the optimization process, but also due to varying number of iterations of the optimization process that were required to converge to the optimal solution.

How sensitive is the system to the choice of priors? Could you provide guidelines for selecting appropriate priors?
Yes, the choice of priors, specifically the variances associated with those priors, is crucial for the performance of Polyrating. Low variance introduces heavy regularization, potentially slowing convergence and limiting adaptation to new tasks, while high variance may reduce the regularizing effect of priors, making them less impactful. Fortunately, these parameters are optimized automatically via cross-validation on the training dataset, allowing us to identify priors that best support predictive accuracy. Empirically, we found that optimal standard deviations, identified via this process, ranged from 1 to 100. Further, based on our experiments in Figure 1 and Figure 3, these variances adjust dynamically based on dataset size: with fewer samples, lower standard deviations are typically optimal, and as sample count increases, the optimal standard deviations also tend to increase. This adaptive approach reduces manual tuning effort and improves robustness across varying data conditions. Thus, one should always use this cross-validation-based approach to find the appropriate hyperparameters. If this is not a possibility for some reason, or to obtain initial results, standard deviations around 50 provide a good default value. We note that for all our experiments, we only optimized over 10 possible values of the variances.

How does the system handle rapid changes in model capabilities over time?
We thank the reviewer for raising this interesting point and have now added an experiment addressing changing model capabilities, specifically model versioning, in Appendix C. As discussed there, Polyrating can incorporate model updates by imposing an extra regularization condition on the base rating of two model versions, similar to the regularization between a model’s base rating and its task-specific rating. This small adjustment enables Polyrating to converge 38% faster when obtaining 10,000 samples for new model versions. Notably, the experiment performed in the appendix only considers publicly available versions. If a model developer is evaluating small, incremental updates to their model, this convergence rate is likely to be much higher as these small updates are much more likely to lead to very similar ratings. Unfortunately, we do not have access to a (large) dataset of preferences that keep track of model performance across many different internal versions and can therefore not evaluate Polyrating in this setting.

How would Polyrating perform with extremely sparse preference data?
Polyrating manages sparse preference data through a Bayesian approach that incorporates priors and deviations. For models with limited task-specific data, the system reflects this uncertainty in the associated rating, clearly signaling to users that fewer observations inform the rating. In extreme cases with no data for a particular task-model pair, Polyrating initializes the rating based on the general prior, indicating an expected rating equal to the base rating but with high uncertainty. This is demonstrated in Table 2 and Table 7, where models with fewer samples, such as Claude-3.5-Sonnet, show higher variance in rating estimates. This mechanism not only ensures robust initial estimates in low-data scenarios but also communicates the confidence level associated with each rating, enhancing interpretability for users.

We hope that this addresses all the reviewers' questions and concerns. We are happy to discuss any further questions the reviewer might have.

We thank the reviewer for their detailed review and insightful questions. We are pleased to hear that they find our notation clear and rigorous, our justification for Polyrating robust, and appreciate our extensive experiments that demonstrate its effectiveness. Below, we address the remaining questions and aim to clarify any outstanding points.

Could you provide more detailed information on the feature selection per task?
Certainly. In this response, we discuss this for each experiment separately.

Bias Detection. In our bias detection experiments, we started by identifying biases that could exist in judges and that we would like to measure. We then map each bias conceptually into a function that quantifies it within an answer. For example, for readability, we use the Flesch Reading Ease score to estimate the influence of readability on a judge’s preference. The functional form of each of these biases is given in Table 4 on page 20. Once the functional form is specified, Polyrating can be applied to measure the influence of the bias.

Sample Efficiency. In Section 4.2, we specify the applied rating modeling formula in each experiment. In all these experiments, feature selection serves to establish meaningful priors across tasks. For example, the use of $R_\text{base}^m + \beta_\text{task}^m [[ g \in D_\text{task} ]]$ with a prior on $\beta_\text{task}^m$ ensures that a model's performance on a task and its base performance is reasonably close, which supports improved convergence. The only exception to this approach is the inclusion of log length as a feature in LLM-based judges. This feature counteracts the well-known increased length bias in LLM judgments compared to human judgments, aligning the priors more effectively.

Multivariate Leaderboard. In our final experiment, we add a feature per task to measure performance directly. These task-specific features are designed as described in L191-199, using $\beta_\text{task}^m [[ g \in D_\text{task} ]]$ to capture the performance on each task.

Could you clarify how Polyrating initializes weight for newly added models? Can Polyrating efficiently update ratings for new models without retraining the rating system?
Polyrating is a statistical method that initializes ratings for new models through priors. For base ratings, this manifests as a uniform prior across all ratings, while for features we employ a normal prior centered at zero. As new preferences are added, the system automatically incorporates these priors when optimizing the loss function. Thus, Polyrating does not require retraining and can seamlessly add new models and preferences on the fly. One would simply continue the optimization process from where it left off, but now with the incorporation of the new models or preferences. Even if full retraining were necessary, the process would take under six hours on a single CPU core, which is negligible compared to the total time required to obtain a significant number of samples that require LLM inference on expensive GPUs and human judgments.

We hope that this addresses all the reviewers' questions and concerns. We are happy to discuss any further questions the reviewer might have.