am-ELO: A Stable Framework for Arena-based LLM Evaluation

We would like to express our sincere gratitude for your high appreciation of the contribution and novelty presented in our paper. Your positive feedback means a lot to us. We also appreciate your valuable suggestions and questions for the computational complexity and experiments aspect of the paper. Here are the responses to each of your comments:

Q1: This paper might benefit from a discussion on other related work.

We sincerely appreciate your suggestion. We have referred to these work, which have been very enlightening for us. We will consider incorporating these related works into our article in future versions.

Q2: While the paper demonstrates the robustness of am-ELO through perturbation experiments, it would be useful to see how the method performs in real-world scenarios with varying levels of annotator quality.

Thank you very much for your question. We've discussed on the motivations behind using perturbations in experiments and plan to collect real-world datasets and test am-ELO online in future work, which can be seen in Comment 5 for reviewer Nkad.

Q3: How does he computational complexity of the proposed am-ELO method compare to the traditional ELO method?

Thank you for your question. The time complexity of am-ELO is equivalent to that of the MLE gradient descent, which is $O(1/\epsilon^2)$ where $\epsilon$ is the calculation accuracy of the GD method. However, since the MLE of m-ELO is a concave function, its time complexity is sub - linear, specifically $O(1/\epsilon)$ [1].

In the experiments, there was no significant difference in the efficiency between am-ELO and m-ELO when running on a GPU. Moreover, as tools for LLM evaluation, their inference costs are far lower than those of large - model inference. Hence, these costs are entirely acceptable.

Q4: Can the authors provide insights into how the proposed framework could be extended to handle more complex evaluation scenarios, such as multi-class or multi-label evaluations?

Thank you for your question. In my opinion, this paradigm can be applied to more complex annotation scenarios, such as multi-label evaluation, as long as the probability density function can be reasonably defined.

Q5: The lack of a detailed sensitivity analysis regarding some parameters.

We sincerely appreciate your suggestion. We assert that due to the convergence required by m-ELO and am ELO, the learning rate and epoch are almost unaffected. In addition, we’ve carried out sensitivity analysis experiments on two parameters. The results of these experiments will be integrated into subsequent versions of our paper.

Regarding scale factor K (Learning Rate) specifically, both m-ELO and am-ELO necessitate extensive training and exhibit insensitivity to this scale factor. Therefore, we focused our analysis on the consistency of the traditional ELO method across different values of K. The findings are presented in Table 1:

Table 1

Our analysis reveals that as the scale factor K increases, the model demonstrates enhanced data fit. However, this improvement in fit comes at the cost of reduced consistency. Furthermore, we carried out a hyperparameter sensitivity experiment on the minimum number of annotations per annotator. The results of this experiment are detailed as follows:

Table 2

As our research indicates, both the traditional ELO and m-ELO show resilience to the variation of annotation count parameters. However, their performance patterns diverge significantly. The traditional ELO consistently exhibits inconsistent results regardless of parameter fluctuations. In contrast, m-ELO demonstrates a tendency to converge towards consistency, underscoring its enhanced stability. Regarding am-ELO, it’s true that it’s highly sensitive to the annotation count. When faced with sparse annotations, the consistency of am-ELO is indeed compromised. But we’ve developed a practical solution. By implementing a screening process for annotators, we can effectively address this issue. This screening process ensures that only reliable annotators with sufficient annotations are included in the analysis, thus improving the consistency of am - ELO.

Reference:

[1] Introductory Lectures on Convex Optimization: A Basic Course. 2014.

Thank you for your valuable feedback! Regarding the questions you raised, we have carefully considered each point and have made the following responses:

Q1: The annotator modeling in the paper is somewhat simplistic. It mainly focuses on the annotator's discriminatory ability and consistency with other annotators and may not fully capture the broader capabilities of annotators.

Q2: How do you plan to extend the annotator modeling to better capture the diverse capabilities of annotators?

Thank you for your suggestion. Indeed, our work has primarily focused on proposing a stable framework that can simultaneously model both annotators and models, rather than comprehensively modeling the annotators. In our subsequent work, we will conduct research on how to comprehensively model the annotators while evaluating the model capabilities.

Q3: Can the am-ELO method still be applicable in scenarios where the annotator is not a human but Judge LLM?

This is indeed an issue we plan to research in the future. We believe that this method is applicable to both human annotators and Judge LLMs. In fact, by using a combination of human and Judge LLM annotations, we can potentially reduce annotation costs and evaluate the capabilities of Judge LLMs. However, we haven't yet figured out how to validate the effectiveness of this method.

Q4: Is there any other baseline method for identifying abnormal annotators?

In Arena systems, historical annotation records are typically used to identify abnormal annotators through hypothesis testing [1]. However, these hypotheses often rely on a rather strong assumption, that is, all annotators in the historical records are normal. Unfortunately, it is extremely difficult to verify this assumption. The premise of am-ELO is only that most annotators are normal annotators, and the premise for use is simpler, which is also the advantage of am ELO compared to hypothesis testing methods.

In future work, we plan to focus on identifying and comparing our am-ELO method on other scenes specifically tailored to the large-scale arena evaluation context [2]. This will enhance the comprehensiveness of our evaluation and more accurately position the contributions of our research.

[1] Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. 2024.

[2] Decentralized Arena via Collective LLM Intelligence: Building Automated, Robust, and Transparent LLM Evaluation for Numerous Dimensions. 2024.

Thank you for your feedback on our manuscript. We sincerely appreciate your time and effort in evaluating our work, and we will explain your questions and suggestions one by one below:

Q1: The evaluation criteria still lack a detailed sensitivity analysis.

We've conducted sensitivity analysis on scale factor K and minimum number of annotations. Results will be added to future paper versions.

As the K increases, the model demonstrates enhanced data fit. However, this improvement in fit comes at the cost of reduced consistency.

This shows traditional ELO gives inconsistent results across parameter changes, while m-ELO converges towards consistency, highlighting its greater stability.

Q2: The analysis fails to adequately address the impact of noisy or sparse annotation datasets on MLE stability.

You’re right—our theory doesn't directly counter the impact of noisy or sparse annotation datasets on MLE stability. But our analysis shows m-ELO has at most one maximum regardless of the dataset, maintaining stability as seen in Table above. In contrast, am-ELO, which models annotators explicitly, is sensitive to sparse data.

In section 4.3, we proposed selecting annotators with enough annotations. As shown in Table above, this strategy effectively lessens the negative impact of sparse datasets on MLE stability.

Q3: Experiments on one dataset provide limited evidence of method robustness.

Currently, open-source arena datasets are scarce. The Chatbot Arena platform is one of the few that offer public data. One NeurIPS 2024 paper also used just one real-world dataset[1]. Besides the dataset in our study, there’s the MTBench dataset[2]. Here’s its statistical information:

However, after filtering, MTBench had a severely limited number of annotators and models. This scarcity made it inadequate for fully validating the stability of our research method.

Despite this, MTBench is still valuable for demonstrating the superiority of our am-ELO modeling:

We’ll incorporate this finding in later paper versions.

Q4: There’s no clear justification that artificial perturbation methods accurately represent realistic annotator behavior.

We designed artificial perturbations to stress-test the ELO. By setting extreme perturbations, we explored the system's robustness.

In LLM evaluation, real annotator behavior is uncertain. Extreme perturbations can better expose ELO's weaknesses and help evaluate its stability.

Going forward, we'll explore artifact perturbation. First, we'll collect a larger real dataset to develop realistic perturbation methods. Second, we'll conduct online testing of am-ELO to validate its real-world effectiveness.

Q5: No baseline comparison beyond the traditional ELO.

Currently, ELO algorithms are widely used in arena platforms like Chatbot Arena. Modified ELO algorithms used in traditional competitive scenarios, such as ELO++[3], incorporate temporal information. But this conflicts with the static nature of LLM evaluations in our paper. Shuffling the dataset for these methods yields unreliable temporal information, making them unsuitable.

Crowdsourcing and arena scenarios differ fundamentally. Arena scenarios lack ground-truth annotation values, while crowdsourcing assume their existence[4]. So, applying annotator-modeling techniques directly to the arena is inappropriate.

In future work, we'll focus on evaluating our am-ELO method in other arena scenarios to enhance evaluation comprehensiveness.

Reference:

[1] Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. 2023.

[2] Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. 2023.

[3] How I won the "Chess Ratings - Elo vs the Rest of the World" Competition. 2010.

[4] Learning from Crowds with Annotation Reliability. 2023.

Thank you for your valuable feedback. Regarding the questions you raised, we have carefully considered each point and have made following responses:

Q1: One potential weakness is the simplicity of the annotator ability modeling, which primarily focuses on discriminatory ability and consistency.

Thank you for your suggestion. Indeed, our work has primarily focused on proposing a stable framework that can simultaneously model both annotators and models rather than comprehensively modeling the annotators. In our subsequent work, we will conduct research on how to comprehensively model the annotators while evaluating the model's capabilities.

Q2: There are a few minor typos and formatting issues that could be addressed in the final version. For example, the descriptions of some formulas are not particularly clear in Section 4.2

Thank you for your kind reminder. We will correct these issues and refine the formulas in subsequent versions.

Q3: How does the proposed am-ELO framework perform in scenarios, especially the number of annotators is very large

This is a highly worthy question for discussion. In Comment 2 of Review vzb8, the relationship between time complexity and precision was discussed, indicating that this complexity can be neglected when compared to the inference of large-scale models.

As the number of annotators gradually increases, since each annotator is associated with only a single parameter, the impact on the total number of parameters of the entire model is relatively small. Moreover, with the help of a GPU, the results can be easily computed.

Q4: Could the authors discuss potential limitations of the proposed method when applied to highly imbalanced datasets, where some models are significantly stronger or weaker than others?

We believe that the methods we proposed are hardly affected by the dataset, especially the m-ELO method. Theorem 4.1 proves that the MLE of this method has at most one extreme point. When there is a significantly stronger model, there is usually no extreme point, or rather, the extreme point occurs at infinity. Although the MLE does not converge in this case, we can still obtain a stable ranking.