Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions

We thank the reviewer for the comments and would like to reply to some of the concerns below:

1. Code anonymity.

Thank you for pointing it out. Upon closer inspection, The warnings displayed in one of the jupyter notebook cells may risk revealing the identity of an author. There was no critical information such as full name or institution present. We apologize for the oversight and have fixed it.

2. The paper only shows “this framework's effectiveness compared to existing evaluation methods”, which seems incremental. Better to show “significant new findings about model capabilities”.

First of all, the primary contribution of the paper is Auto-Arena, an automated evaluation framework. It proposes a method to tackle the task of LLM Evaluation.

Therefore, task performance is measured by the evaluation method’s effectiveness compared to previous ones. Compared to previous evaluation methods, Auto-Arena is the first to fully automate the LLM evaluation process in an end-to-end way. As no human efforts is involved, it still reaches the highest correlation with human preferences, validating the evaluation method’s robust performance. In the age of fast-developing LLMs, the fully automated feature opens up possibilities for deriving fast evaluations without manual efforts.

As to your suggestion of showing “significant new findings about model capabilities”, we would like to note that the results of the evaluation (specific model capabilities, etc) are not the focus of the paper. The focus is the evaluation method itself. Therefore, the results are presented and analyzed, but not studied in detail.

3. “Cost efficiency compared to previous methods”? “Detailed breakdown”?

A cost analysis is already provided in Appendix H of the paper. As shown in Table 12, Auto-Arena’s evaluation costs around USD 5 for one model, which is lower than previous LLM-judge-based evaluation methods. Despite the lower costs, the evaluation performance (correlation with human preferences) is higher.

As to your question on the detailed computational resource breakdown, during our tournament, many different models are used and they all have various API costs and inference times. The $5 was derived from the resulting overall cost for our main experiment.

To help better understand the computational resources, we have also summarized the table below, which estimates the computational resources for each component based on input/output tokens. Please see our summarized table below:

Table 1. Cost breakdown (in terms of input and output tokens) for one evaluation (a comparison between two models on one question)

For example, if all agents (candidates and judges) have costs and inference times that are on par with GPT-4o, the API costs would be USD 0.22 per evaluation question. Evaluating our set of 40 questions would cost USD 8.8. In the tournament, however, cheaper and non-proprietary models are engaged as well, which drives down the costs.

Analyzing the table carefully, we see that question generation is the cheapest component. The biggest effort is actually on the committee judgments and discussions, where bringing in several judges to discuss increases the costs.

We thank the reviewer again for his/her comments. If there are remaining concerns or lingering questions, we’d be happy to discuss them in more detail.

Dear reviewer j7qN,

Thank you once again for taking the time to review our paper. Could you let us know whether the questions have been addressed to your satisfaction? If not, we'd be happy to discuss further! If it has, would you mind adjusting your current ratings?

Best regards, Authors

Thank you for the reply, we reply to your concerns here:

Q. goal of building evaluation benchmarks is to inform users about the specific strengths and weaknesses of models.

R: We'd like to respectively clarify that our paper proposes a comprehensive framework that can be used to evaluate LLM capabilities and performances. We verify Auto-Arena's effectiveness by measuring the alignment with human preferences, where we show consistent improvements in correlation metrics. The method can be expanded to various domains and cover many LLMs. Previous important LLM evaluation papers also focus on the evaluation method itself, without analyzing specific model results: G-Eval[1] conducts experiments and compares to correlations to human judgments of other meta-evaluation benchmarks. MT-Bench[2] compares the LLM judge's performances across different judge designs and evaluates the agreement with humans. It does not talk about pitfalls and strengths of specific models. AlpacaEval[3] shows correlations with Chatbot Arena compared to other benchmarks. ChatEval[4] shows correlations and accuracy of the evaluation results compared to other benchmarks. The list goes on. Therefore, we believe it's unfair to reject the paper on the grounds that it doesn't analyze the weaknesses and strengths of specific models.

Moreover, besides quantitative evaluation, we do analyze the framework and the behaviors of specific debates and models. As mentioned in L403-405, we notice that models claimed to have multi-lingual efficiency perform better on the Chinese experiment compared to the English one. In section 5, we investigate various behaviors of the LLM candidates and highlight specific strengths of the models.

Q. How might the experiment results help users understand model-specific performance insights beyond the rankings?

R. In the main experiment, our goal is to derive a leaderboard on general LLM capabilities, which is similar to Chatbot Arena. The main advantage is that it's fully automated and, therefore, can derive timely evaluations of different LLMs. If, for example, besides a quantitative metric, the user would like to understand specific behaviors of specific models during debates, we release the debate samples online (link included in paper footnote 3), which the users can use to derive subjective observations on further model improvement. If the users are interested in deriving the leaderboard for model performance in a specific domain, Auto-Arena can be extended to other domains or languages by altering the language or aspect specification of the prompts. We provided Chinese as an example in the main paper section 4.2.

[1] G-EVAL: NLG Evaluation using GPT-4 with Better Human Alignment

[2] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

[3] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

[4] CHATEVAL: TOWARDS BETTER LLM-BASED EVALUATORS THROUGH MULTI-AGENT DEBATE

Previous important LLM evaluation papers also focus on the evaluation method itself, without analyzing specific model results

In Table 7 of MT-Bench [1], they gave a clear breakdown of the win rates of each model on different categories (roleplay, math, coding, etc.) In their analysis, they said "we see GPT-4 is significantly better than others. Vicuna-13B is noticeably worse than GPT-3.5/4 in reasoning, math, and coding categories. Note that in math/coding category, GPT-3.5 and GPT-4 have similar overall win-rate because they both failed to answer some hard questions, but GPT-4 is still significantly better than GPT-3 in the direct pairwise comparison or single-answer grading." This is informative in telling users the specific model results that I was referring to.

It seems like the prompts used for experiments in Auto-Arena are broken down into different domains/categories, similar to what MT-Bench did. However, there are no or very little analysis on how the models perform on these different domains. Case studies would've helped in explaining this part as well.

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Question 2: Why not “using the original debaters as judges”?

We explicitly avoid such a scenario to reduce self-enhancement bias [2], which is where the LLM judges would prefer the answers generated by themselves. Therefore, to select judges, we follow the 2 criteria: 1) the judge is not from the same model family as the current debaters (to avoid self-enhancement bias. 2) at each run, we select the 5 top-performing models from the current leaderboard. Note that the current leaderboard consists of all debaters that have participated so far. We select the better-performing as the better LLMs could serve as better judges (shown in Appendix F).

Question 3: “Incorporating external information sources to enhance performance?”

Thank you for the suggestion. It would be a good direction to consider in the future when we are evaluating information-rich questions that require retrieval from external sources. Then, introducing RAG could aid in the LLM judge’s ability to diagnose factual mistakes. As most of the current tasks we evaluate (math, coding, writing, roleplay, etc) are not very reliant on external knowledge, we haven’t included such mechanisms so far. But we would be happy to explore such directions in the future.

Thank you again for the helpful comments. Please let us know if the above has clarified some of your concerns or if there are still lingering concerns. We would be happy to address them.

[1] Humans or LLMs as the Judge? A Study on Judgement Biases.

[2] Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge.

We thank the reviewer for the constructive comments and would like to reply to some of the concerns below:

Weakness 1: Evaluations could be “unreliable”, “tied to the foundational model’s performance”.

We agree with your point: the performance of a single judge model may be inconsistent. This is exactly the motivation behind our multi-judge design: We incorporate multiple strong-performing judges to engage in discussions to improve judgment performance. As shown in ablation (“w/o committee discussion” in Table 2), committee discussion improves judgment correlations with human preferences.

“Data Leakage”.

We also agree with the data contamination concerns you have raised. Therefore, by dynamically drafting and updating the questions, Auto-Arena is more contamination-proof compared to static benchmarks, such as MMLU. We show this effect in the contamination analysis in Appendix C.

“Easily extended to other domains and languages”?

To clarify this claim, Auto-Arena would be quite useful for extending into many of the fine-grained usage scenarios currently. For example, if a gaming company wants to use LLMs for NPC dialogue, this would be a very fine-grained application where no datasets have been developed before. In this case, Auto-Arena can help develop a reliable evaluation. In this sense, we believe Auto-Arena would be immensely helpful to the industry that tries to incorporate LLMs.

Due to the limited resources, we only included Chinese as an example (Section 4.2) for the extension.

As for your comment on “testing on newer tasks and data—ones that no LLM agent has seen”, we are afraid this would not be feasible. In order for the agents to function properly (engage in debates, draft the questions, etc), the agents have to grasp a certain degree of prior knowledge of the task. For example, the agents can evaluate “Chinese performance” because it has seen and known Chinese during pre-training. If it has never seen this language before, it would not be able to “speak” Chinese, let alone debate in Chinese.

Weakness 2: “Analyzing instances where LLM judgments differ from human judges”.

Thank you for this note. There are papers that have specifically studied how LLM-based judges differ from humans. [1] discovers that LLM judges possess Misinformation Oversight Bias, Gender Bias, Authority Bias, and Beauty Bias. In contrast, humans also have strong Misinformation Bias and Beauty Bias, but very minimal Gender Bias. As Auto-Arena is also built with LLM-as-a-judge, the differences with human judgments would be quite similar. Although this is not the main focus of the paper, we agree that such case studies would make for an interesting case study and would aim to incorporate such a study on the website or in a later version of the paper.

Question 1: Detailed Computational Resources?

Thank you for your question about the detailed breakdown of costs. During our tournament, many different models are used and they all have various API costs and inference times. The $5 was derived from the resulting overall cost for our main experiment.

To help better understand the computational resources, we have also summarized the table below, which estimates the computational resources for each component based on input/output tokens. Please see our summarized table below:

Table 1. Cost breakdown (in terms of input and output tokens) for one evaluation (a comparison between two models on one question)

For example, if all agents (candidates and judges) have costs and inference times that are on par with GPT-4o, the API costs would be USD 0.22 per evaluation question. Evaluating our set of 40 questions would cost USD 8.8. In the tournament, however, cheaper and non-proprietary models are engaged as well, which drives down the costs.

Analyzing the table carefully, we see that question generation is the cheapest component. The biggest effort is actually on the committee judgments and discussions, where bringing in several judges to discuss increases the costs.

Dear reviewer PycX,

Thank you once again for taking the time to review our paper. Could you let us know whether the questions have been addressed to your satisfaction? If not, we'd be happy to discuss further! If it has, would you mind adjusting your current ratings?

Best regards, Authors

Thank you for the reply and concerns. We really appreciate the opportunity to discuss.

Q1. I mean how might Auto-Arena handle tasks that require highly specialized or emerging knowledge (e.g., newly reported events)? Basically, if a task only requires common sense, we would not really need LLMs to do the discussion. In real-world application scenarios, we often need to work on something requires highly specialized or emerging knowledge. How does Auto-Arena perform in those cases?

R1. Currently, Auto-Arena focuses on testing the general capabilities of LLMs, i.e. how helpful it is to human users. The tasks span domains such as writing, roleplay, and math.

If we are to extend to domains that require highly specialized knowledge, we believe it would then be beneficial to enable RAG for all the agents, which could incorporate external knowledge into this framework. RAG would allow the debaters to utilize specialized knowledge and examiners to detect discrepancies accurately. This would be an interesting direction to explore in the future.

Even besides the RAG component, the judges can use other dialogue techniques to evaluate the quality of the debate when they don't possess the specialized knowledge themselves. For example, in paper [1], the examiner can interact with the debaters and employ a cross-examination technique to detect the candidate's loopholes in logic and memory.

Therefore, it is a possible and interesting use case to extend into scenarios that require specialized knowledge. But to ensure performance, there would be other designs and techniques involved to guarantee the performance of the judges. We will consider such extensions in the future.

Q2 LLMs are found to be good generators but sometimes not effective as verifiers. The authors avoid discussing when their model might not align with human judges. It is acceptable to acknowledge these limitations so that future researchers can fully understand the implementation and its potential challenges.

We will include the previously mentioned biases caused by LLM-as-a-judge in the limitations section as possible pitfalls of Auto-Arena. Thank you for raising it.

We thank the reviewer for his/her constructive comments and would like to address some of the concerns below.

1. Limitations?

We didn’t include limitations discussion as it wasn’t required by ICLR. However, we have included limitations in an earlier version and would be glad to add it back to the discussion in the conclusion section:

“There still exist some limitations to the current approach. For example, the distribution of question domains is artificially designed, which may deviate from real-life distributions. Currently, Auto-Arena focuses on 1-to-1 peer battles, which limits its usage in multi-player scenarios.”

2. “Human in loop evaluations rather than comparing with static baselines”

Do you mind clarifying what the question refers to? If you mean that we should consider comparing to human-based evaluations, we’d like to note that the main results of Auto-Arena is derived exactly based on its correlation to human preference ratings, where we take chatbot arena as ground truth. Besides static baselines, we also compare to model-based ones (Alpacaeval, MT-Bench, Arena-Hard).

If you mean that we should include human in the loop in Auto-Arena, it would conflict with our overall motivation of eliminating human efforts in the task of LLM evaluation.

3. “30 questions” only? More detailed analysis of these questions?

As noted in the L258-267, we include 40 questions, with 5 from each of the 8 categories. This number was selected so that our experiments are on the same scale as MT-Bench. The question design and types also follow the design of MT-Bench dataset and has a similar distribution. We have included the question types and demos on page 18.

4. “Time-stamp cutoffs” for unseen data?

Thank you for the suggestion. It would be a good component to include in the future when we are evaluating information-rich questions. As most of the current tasks we evaluate (math, coding, writing, roleplay, etc) are examining the model capabilities and not very reliant on external knowledge and new information, we haven’t included such mechanisms so far. But we would be happy to explore it in the future.

Thank you again for the helpful comments. Please let us know if the above has clarified some of your concerns or if there are still lingering concerns. We would be happy to address them.

We thank the reviewer for his/her constructive comments and reply to some of the concerns below:

1. “Multi-agent debate in automatic evaluation is not particularly novel”. Discussion of ChatEval.

Yes, multi-agent debate has been used before for LLM-as-a-judge, which corresponds to the committee discussion component in our framework. We are aware of prior works and discuss them in L513-519.

The main novelty of our paper is the first completely automated evaluation framework, from question drafting to LLM reviewers. Therefore, it innovatively eliminates the need for any manual efforts in the process. Although prior works show that using multi-agent-debate for evaluations could mitigate bias and improve performance, they are still constrained to evaluating LLM performances on existing datasets (manually constructed).

Thank you the suggestion on including ChatEval. In fact, it was included in an earlier version but was deleted later due to the page limit. Currently, we mention DRPE and PRD in L514-517, which are similar to ChatEval and use multi-agent-debate for LLM jugdments. We will include ChatEval in L517 as well!

2. “No formal guarantees on the reliability of the evaluation.”

We agree that, because the method is inherently empirical, it’s not feasible to derive formal guarantees. The extensive study on an extensive set of 15 models provides empirical evidence that the method is robust and expandable.

3. “How sensitive are the results to the choice and number of initial committee members and their ranking initialization? Would including low-capability LLMs in the judge committees reduce overall performance?”

The committee members are updated after each debate according to best 5 models in the current ranking. Therefore, the choice stablizes through the iterations.

In the table presented in Appendix F, we do note that better-performing LLMs can serve as better judges. However, as mentioned previously, as we include more and more LLMs in the ranking, the judge committee will stablize to a group of high-performing ones.

4. “Examples of cases where Auto-Arena's evaluation significantly diverged from human preferences?”

Thank you for this note. There are papers that have specifically studied how LLM-based judges differ from humans. [1] discovers that LLM judges possess Misinformation Oversight Bias, Gender Bias, Authority Bias, and Beauty Bias. In contrast, humans also have strong Misinformation Bias and Beauty Bias, but very minimal Gender Bias. As Auto-Arena is also built with LLM-as-a-judge, the differences with human judgments would be quite similar. Although this is not the main focus of the paper, we agree that such case studies would make for an interesting case study and would aim to incorporate such a study on the website or in a later version of the paper.

5. “Ensure that the questions generated by LLMs are sufficiently challenging”?

In our current prompt to the examiner (shown in Appendix A.1), we ask the examiner to devise “complicated and difficult, requiring in-depth understanding and analysis of the subject”. The prompt design ensures that the questions are challenging and debatable.

Moreover, as the LLMs evolve, we will also update the examiner accordingly to be the best-performing LLM to ensure that it has the highest capability itself. Therefore, the question the best-performing LLM deems to be challenging would be challenging to other LLMs, too.

Last but not least, as the most important design is the peer-debate component, the LLM debaters raise questions to each other during the debate process, which democratizes the question generation process. The initial question, in this case, will only serve as seed question/topics for the debate. In the process, LLMs candidates also try to generate the most challenging question it could to further attack its opponents.

Thank you again for the helpful comments. Please let us know if the above has clarified some of your concerns or if there are still lingering concerns. We would be happy to address them.

[1] Humans or LLMs as the Judge? A Study on Judgement Biases.[2] Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge.