GameArena: Evaluating LLM Reasoning through Live Computer Games

Q3: It would be great if you can provide some lessons learned on your work to inspire future research. Any insights on constructing the data collection? Any discussion/trend on the reasoning evaluation results? Any future work direction?

Our key insights are:

• by using designated gaming rules, we can better control human-AI interactions towards evaluating LLM reasoning to disentangle different model capabilities
• relying solely on outcome metrics may lead to biased, narrow conclusions, highlighting the need for procedural metrics to provide a more comprehensive evaluation.

Our evaluation results show the following findings:

• GameArena’s ranking aligns with other static reasoning benchmarks (LiveBench-Reasoning, GPQA).

• Models with strong reasoning capabilities and multi-turn instruction-following capabilities, such as claude-3.5-sonnet (Appendix D.1) and GPT-4o (Appendix D.5) are ranked high in GameArena.

• Models that excel at short conversations but with poor reasoning in extended game sessions usually rank low in GameArena (example in Appendix D.5).

Here are some future directions we would like to pursue:

• Adding more games (e.g., social deduction games) to evaluate LLM reasoning in multi-player and multi-agent settings.
• Incorporating more advanced reasoning techniques.
• Exploring how the collected data can be used for improving LLMs’ reasoning capabilities.

Thank you for your valuable questions. Here are our responses to your questions and concerns:

W1: Regarding Table 4 on comparing GameArena, ChatArena, GPQA ranking, having n = 5 seems too small to have stat.

We initially considered including weaker models but found that they performed extremely poorly and negatively impacted the user experience. This reinforces that GameArena is a challenging reasoning benchmark. As a result, we chose to focus on a set of very strong models for meaningful comparisons, ensuring the experience remains fun and engaging for users.

W2: Does not mention the potential limitations of the work: it could relate to the benchmark topic distribution, the annotators distribution and any things that are worth mentioning if people are going to use this benchmark.

We will acknowledge the limitations of our benchmark in the revision:

• Potential bias in user-generated topics: user-generated topics may not be diverse enough. We plan to release GameArena publicly in the future, enabling data collection in the wild with more diverse topics.

• Limitations of participant pool: while our participant sample is diverse, its size is limited. We will release the games publicly to enable larger-scale data collection from a broader pool of participants.

• Limitations of retrospective analysis: we currently use simple CoT reasoning techniques and retrieve LLM hidden thoughts in retrospective analysis. However, the revealed hidden thoughts may not fully align with the original inference rounds. In our future work, we will incorporate advanced reasoning techniques that provide reasoning traces during gameplay (similar to o1 model).

W3: The notion of this work being the first to apply GWAP techniques for benchmarking LLMs (line 525) is not fully correct. Researchers in other subfields (e.g. HCI: human-AI collaboration, AI-augmentation) have investigated this for a long time. It is better to cite some recent relevant works on annotation and benchmarks.

Thank you for the pointers. We will incorporate these work into the related work section and explicitly acknowledge the connections. Our primary goal is to evaluate the state-of-the-art LLM reasoners with the games.

Q1: Can you provide benchmark statistics (e.g. topic) and annotator demographics to show if the users inputs are diverse and give a bigger picture of this benchmark?

Benchmark statistics: the distribution target words and statements cover a diverse set of topics. We used the o1 model from OpenAI to retrieve a breakdown of different topics in each game:

1. Akinator game: among all 693 user-generated objects, object duplication rate is lower than 2%. For all distinct objects, 18.75% belongs to electronics & appliances, 13.54% belongs to kitchenware & dining, 8.33% belongs to furnitures, 8.33% belongs to personal care, 7.48% belongs to vehicles, 7.29% belongs to sports, 7.1% belongs to home decorations, 6.25% belongs to office supplies, 5.21% belongs to tools, and 17.72% belongs to other categories (see Appendix B for examples).

2. Bluffing game: among all 615 user statements, only 3 statements are duplicated. 27.53% belongs to personal experience, 23.51% belongs to professional experience, 15.92% belongs to personal skills, 14% belongs to factual information, 8.51% belongs to significant personal achievements, 10.53% belongs to others (see Appendix B for examples).

3. Taboo game: we hand-curated a diverse set of target words from different topics (available via this link).

User demographics: all participants were from the United States, with 50% male and 50% female. Age distribution was 13% age 18-24, 34% age 25-34, 23% age 35-44, 18% age 45-54, and 12% age 55+. The distribution of educational attainment was close to that of the US population [1], with 22% holding a high school diploma, 31% a bachelor’s degree, 12% a master’s, and 5% an advanced degree.

[1] U.S. Census Bureau. 2016. Educational Attainment in the United States: 2015. U.S. Department of Commerce. (March 2016).

Q2: In Section 4.2 (line 327), for the comparison between Chatbot Arena and GameArena, is GameArena also collected in the wild during the same period? It seems unfair to compare the data efficiency between two if GameArena gives money incentives but ChatbotArena depends on community engagement (which is reasonable to have lower efficiency).

We will clarify that our data was not collected in the wild in the revised paper. Additionally, we will reframe it as an exploratory study rather than a comparative study.

We plan to release the games publicly in the near future and will keep the community informed on the results of data collection in the wild.

Thank authors for providing all the details (demographic, dataset detail) and responses. I like the idea on constructing the benchmark using gamified approach but the responses still cannot resolve some of my concerns:

We initially considered including weaker models but found that they performed extremely poorly and negatively impacted the user experience. This reinforces that GameArena is a challenging reasoning benchmark. As a result, we chose to focus on a set of very strong models for meaningful comparisons, ensuring the experience remains fun and engaging for users.

I understand and agree the decision to only focus on the very strong models to construct the challenging benchmark. However, this does not justify my concern whether the reader can trust the comparison due to the small sample (n=5). Does each model have 50 gaming sessions for testing? (and you calculated the statistics by aggregated scores for sessions?) If yes, authors may consider to compute some statistical sign. tests to justify the statistics reported.

Also, the authors' suggestion of 'unsatisfactory' performance by weaker models makes me wonder if this test could be used to differentiate those weaker models. It seems like this might limit the usability of the evaluation. Please correct me if I’ve misunderstood.

I understand and agree the decision to only focus on the very strong models to construct the challenging benchmark. However, this does not justify my concern whether the reader can trust the comparison due to the small sample (n=5). Does each model have 50 gaming sessions for testing? (and you calculated the statistics by aggregated scores for sessions?) If yes, authors may consider to compute some statistical sign. tests to justify the statistics reported.

Thank you for your reply and the question. Each of our models has ~120 gaming sessions for evaluation and we evaluated the model’s performance using aggregated metrics across gaming sessions. To address your concern, we conducted two hypothesis tests:

1. Kendall’s tau: Testing the null hypothesis that there is no association between the two rankings (i.e., tau = 0). We conduct one-tailed test for the alternative hypothesis that tau > 0.

2. RBO: Performing a permutation test to determine if the rankings are randomly related (i.e., RBO resulted from chance). For each comparison pair, we fix one ranking, and sample 1000 random rankings. This allows us to obtain a null distribution of RBO scores. We then compute p-value using the observed RBO score.

Here are the results (*: p-value < 0.05):

The results show a strong correlation between GameArena’s rankings with other reasoning benchmarks (LiveBench Reasoning, GPQA), where null hypotheses are successfully rejected (*). In many comparisons against Chatbot Arena, they fail to reject the null hypothesis and show that GameArena’s rankings are only weakly correlated with Chatbot Arena.

Also, the authors' suggestion of 'unsatisfactory' performance by weaker models makes me wonder if this test could be used to differentiate those weaker models. It seems like this might limit the usability of the evaluation. Please correct me if I’ve misunderstood.

With the rapid advancement of LLMs, weaker models will likely become obsolete as more capable ones emerge [1]. We believe evaluation benchmarks designed today should aim to evaluate the capabilities of cutting-edge models, rather than distinguishing between weaker models.

That said, our benchmark can still evaluate weaker models (e.g., GPT-3.5 and Haiku). We used a small portion of gaming sessions for their evaluation to minimize disruptions to user experience. For example, our benchmark is capable of distinguishing between the Mistral-large and GPT-3.5 model.

If there are additional ways we can address your feedback or enhance our work, we would be delighted to discuss them further. Your evaluation is greatly appreciated, and we hope you’ll consider raising the rating if we’ve addressed all your concerns.

[1] Matthew Lynley. An AI model deprecation dilemma, July 2023. https://www.supervised.news/p/an-ai-model-deprecation-dilemma.

Thank you for your acknowledgement and valuable questions. Here are our responses to your questions and concerns:

W1: The motivation, especially how using this benchmark can have findings different from / aligning with other benchmarks, such as Chatbot Arena, GameBench, or GTBench, needs further explanation. How can the findings generalize to other games / downstream tasks?

Our findings include:

1. Compared to other static reasoning benchmarks: GameArena’s ranking aligns with other static reasoning benchmarks (LiveBench-Reasoning, GPQA in Table 4). As a dynamic benchmark, GameArena is less prone to data contamination.

2. Compared to Chatbot Arena: GameArena’s ranking is weakly correlated with Chatbot Arena (Table 4) as they are designed for different evaluation goals (Chatbot Arena for overall performance across a diverse set of real-world tasks).

3. Disentangling response styles from reasoning capabilities: comparing user ratings of different models we collected (Akinator, Taboo and Bluffing) with GameArena rankings (Table 2 and 3), we found that user ratings often failed to accurately reflect the models’ reasoning capabilities. Instead, users tended to base their ratings on style factors like the engagement or appeal of LLM responses rather than reasoning performance [1].

[1] Tianle Li, Anastasios Angelopoulos, and Wei-Lin Chiang. Does style matter? Disentangling style and substance in Chatbot Arena, Aug. 2024. https://lmsys.org/blog/2024-08-28-style-control/.

Q7: Some existing work also uses social deduction games to evaluate LLMs’ reasoning abilities, like Werewolf [1], Avalon [2,3], Taboo game [4], and Who is Spy [5]. What is the contribution of this paper except using some different games?

GameArena is a dynamic, incentivized benchmark focused on reasoning. Static reasoning benchmarks you mentioned are prone to data contamination and saturation (GameBench, GTBench, PlanBench, any reasoning games etc.). A detailed comparison is made in the table below.

Our vision includes expanding to multi-agent and multi-user interactions in future work. This involves adding complex social deduction games for evaluations.

W2: Results need deeper analysis (question 3, 6). Q3: Why GPT-4o performs the best in the Taboo game in terms of win rate, but is not the best in terms of multi-hop reasoning and abductive reasoning?

GPT-4o has the highest win rate in Table 2 but is outperformed by Claude3.5-sonnect in Table 3 due to the following reasons:

1. We observed that GPT-4o tends to be cautious in responding to users’ questions (see examples in Appendix D.5 from GPT-4o’s responses). Refusals happen more often to GPT-4o than the other models, resulting in a higher win rate.

2. In terms of procedure metrics, GPT-4o is outperformed by Claude3.5-sonnect since claude-3.5-sonnet is more capable than gpt-4o in predicting the secret words.

Therefore, evaluating models’ performance solely based on win rates might be problematic, which motivates us to design procedural metrics for a more comprehensive evaluation.

Q6: Is there any superficial correlation (shortcut learning) in LLMs’ responses? For example, after responding “Eggs, of course” in Figure 2, the system prompts the LLM that the target word has appeared. But it is obvious that the word is Egg – one can tell even without previous context.

While there might exist some superficial correlations, we found most cases require complex reasoning strategies (see Appendix D.2, D.3). Figure 2 is a simplified example to illustrate the game rules, and we will replace it with an actual game conversation in the revision.

We show that indeed in most gaming sessions, abductive/deductive/abductive and multi-hop reasoning are involved. For example, in Appendix D.3, the model uses information from all conversational rounds to gradually narrow down the possible range of words and eventually identifies the secret word. To strengthen this point, we will add case studies for a detailed analysis of reasoning results.

Thank you for your recognition! We have modified Figure 2 and section 2.1 (the AI Taboo game paragraph) to provide clearer explanations.

Thank you for your acknowledgement and valuable comments. Here are our responses to your questions and concerns:

W1: In lines 82 to 84, the author states the effectiveness of data collection sorely based on reasoning assessment. The comparison on the effectiveness of data collection may not be fair since Chatbot Arena aims to evaluate human preference across a gigantic number of tasks and each pair-wise comparison contributes to the ranking.

In the data efficiency comparison, we consider all pairwise comparison data (with voting) across all topics in Chatbot Arena as effective when comparing it to GameArena.

W2: The description of the Taboo game in Figure 2 is confusing.

We will refine the description to: “The LLM includes the target word in its response but fails to guess it correctly” and update an example of human wins in the revision.

W3: In line 249, the author mentions "lower the disparity ratio, the higher information gain". But in the Akinator game, I suppose both "yes" or "no" narrow down the range of the actual object. A positive does not contribute more information than a negative response.

We agree that both "yes" and "no" responses can narrow down the answer space. So the key point is the balance of the "yes" and "no" responses (i.e., disparity ratio), which determines how effectively the question narrows down the possibilities. Ideally, a well-balanced question helps reduce the search space efficiently by evenly splitting the answers. If most objects give the same answer, the question provides little information in differentiating them.

Q1: In section 4.3, a brief demographic information could be described for the recruited 100 participants.

We will include participant demographics in section 4.3: All participants were from the United States, with an even gender split (50% male and 50% female). Age distribution was 13% age 18-24, 34% age 25-34, 23% age 35-44, 18% age 45-54, and 12% age 55+. The distribution of educational attainment was close to that of the US population [1], with 22% holding a high school diploma, 31% a bachelor’s degree, 12% a master’s, and 5% an advanced degree.

[1] U.S. Census Bureau. 2016. Educational Attainment in the United States: 2015. U.S. Department of Commerce. (March 2016).

Q2: Other than reasoning evaluation, is it possible to assess the preference factors from the comparison data?

In our user study, we also asked users to rate the performance of models from 1 to 10 and collected user ratings on different models for each game. You can find the ratings of all models for Akinator, Taboo and Bluffing.

We found that user ratings often failed to accurately reflect the models’ reasoning capabilities. Instead, users tended to base their ratings on style factors like the engagement or appeal of LLM responses rather than reasoning performance. For example, we observed that In the Bluffing game, users preferred LLaMA3.1-405B-chat due to its vivid, human-like tone, despite its lower win rate.

In the Akinator game, users favorred Gemini-1.5-pro for asking diverse and interesting questions, even though it typically requires more rounds to guess the target word correctly.

These findings align with recent studies on Chatbot Arena, which shows the style of responses strongly influences human preferences [2].

[2] Tianle Li, Anastasios Angelopoulos, and Wei-Lin Chiang. Does style matter? Disentangling style and substance in Chatbot Arena, Aug. 2024. https://lmsys.org/blog/2024-08-28-style-control/.

W4: the comparative study against Chatbot Arena is too shallow and uses limited data. The conclusions attributed to it don't seem solid nor significant for the overall paper. Similar comments apply to the ranking comparisons in 4.5.

Q6: As the comparison with Chatbot Arena and ranking comparisons don't seem to add much to support the main paper contributions, perhaps these could go into appendices to open space for more detailed analysis of the reasoning results.

Our comparison goals are the following:

1. GameArena is a dynamic, incentivized benchmark focused on reasoning.

2. GameArena disentangles reasoning capabilities more effectively than dynamic benchmarks like Chatbot Arena by reducing confounding variables.

3. GameArena has a gamified interface that improves data collection efficiency (see section 4.2).

The ranking comparison with Chatbot Arena provides evidence for the first two points (Table 4):

1. GameArena’s ranking aligns with other static reasoning benchmarks (LiveBench-Reasoning, GPQA).

2. GameArena’s ranking differs from Chatbot Arena as they are designed for different evaluation goals (Chatbot Arena for overall performance across a diverse set of real-world tasks).

Q2: If Table 2 lists GPT-4o as the best model for Taboo, why don't the metrics in Table 3 support that?

GPT-4o has the highest win rate in Table 2 but is outperformed by Claude3.5-sonnect in Table 3 due to the following reasons:

1. We observed that GPT-4o tends to be cautious in responding to users’ questions (see examples in Appendix D.5 from GPT-4o’s responses). Refusals happen more often to GPT-4o than the other models, resulting in a higher win rate.

2. In terms of procedure metrics, GPT-4o is outperformed by Claude3.5-sonnect since claude-3.5-sonnet is more capable than gpt-4o in predicting the secret words.

Therefore, evaluating models’ performance solely based on win rates might be problematic, which motivates us to design procedural metrics for a more comprehensive evaluation.

Q3: The calculation of disparity ratio in 3.2.2 is also not detailed enough. The authors claim the metric quantifies how effectively each step divides the answer space into two categories. But it only counts yes/no responses ratios?

Given a list of possible objects and a set of questions to choose from, 20Q algorithms use yes/no disparity ratios to select the best question [1]. The idea is that an effective question should evenly divide the answer space into "yes" and "no" responses, halving the search space each time in a way similar to binary search. If most objects yield the same answer, the question provides little value to differentiate them. We build on this concept to develop a metric for evaluating the effectiveness of question generation.

[1] Burgener, Robin. "Artificial neural network guessing method and game." U.S. Patent Application No. 11/102,105.

Q4: In 4.1 the authors state the prompt optimization process uses 200 of the collected game sessions. Does this mean all data collection uses one set of prompts and the analysis uses the different optimized prompts?

To clarify, we initially use a hand-crafted system prompt to collect 200 game sessions. This data is then used to generate five optimized system prompts using DSPy. Both the data collection and analysis are based on these five optimized prompts.

Q5: Will the code used to calculate the different metrics be released openly?

We have included the code for game implementation and data analysis in the supplementary material and available via this link. We will also publicly open-source the code, game data, and analysis results.

Thank you for your acknowledgment and valuable questions. Here are our responses to your questions and concerns:

W1: Firstly, the analysis of reasoning capabilities depends on a "replay" of a concluded game session round by round. At each round the replay prompt asks the models to output their "intermediary thought process". As LLMs are known for hallucination and fabricating rationalizations, this could heavily affect the proposed approach. There is no guarantee that whatever "evidence" being generated actually directly corresponds to the original inference rounds. A more robust approach to collect the data might have been to ask for such output during the game.

Asking LLMs to generate reasoning traces during gameplay could be an alternative approach. Which method to use is a trade-off between providing better user experience versus getting more reliable reasoning traces in real time. We didn’t use this method due to the following issues:

1. Increased latency: generating intermediary thoughts during gameplay will significantly slow down response times and disrupt gameplay experience.

2. Potential disruption of game flow: parsing exact answers from long, complex LLM responses is unreliable and may cause errors that break the game.

Since GameArena relies on user participation to collect data, we prioritize user experience and chose our current method. In our future work, we plan to incorporate advanced reasoning techniques that can provide the reasoning traces during the game (similar to the o1 model). For now, we developed a simplified proof of concept.

GameArena cannot fully eliminate hallucinations, as this remains an open challenge for all LLMs. However, our retrospective analysis keeps the context unchanged and is demonstrated reliable in many cases (see Appendix D for examples).

W2-1: The selected games also do not necessarily require the level of abstract reasoning being attributed to them in the paper. Akinator/Q20, for example, can be seen as a simple bisect search.

We agree that a model that performs efficient bisect search is able to perform well on the akinator game. However, GameArena requires a good AI to reach the correct guess in as few steps as possible, which requires the model to ask questions strategically and guesses effectively. Note that:

• We have tested weaker models (e.g., GPT-3.5 Turbo, Open-Mixtral-8x22b, Claude-3.5-Haiku-Latest), which usually failed to identify objects within 20 rounds.

• To evaluate this, we use one of our outcome metrics — the number of game rounds (Table 2). Winning in a smaller number of rounds implies a stronger reasoning capability.

We will modify our paper and make these points clearer.

W2-2: Perhaps to properly evaluate deductive/abductive/inductive reasoning, the responses first should have been turned into such problems and then LLMs re-queried to check for response matches.

We assume by “turned into such problems and then LLMs re-queried to check for response matches”, you are referring to the way users interact with AI during gameplay.

GameArena wants to keep users incentivized and needs to support game-based natural conversation. As a result, we embed them within a broader context to reflect the model’s performance in a natural conversational setting for reasoning evaluation. We hope this addresses your concern, if not, we are willing to provide further explanation.

W2-3: It's also been shown that LLMs don't necessarily perform multi-hop reasoning/answering, but infer a final response based on other context. To claim multi-hop reasoning, more evidence would be required.

Q1: Claims about reasoning, especially specific types of reasoning, require stronger evidence to support them. I'd expect to see at least a couple case studies to support the claims presented in the evaluation. Not only some metrics and no detailed analysis.

W3: The text of the paper tends to overclaim, both in terms of reasoning conclusion without enough evidence, but in using terms like "exceptional multi-hop reasoning"

We show that indeed in most gaming sessions, abductive/deductive/abductive and multi-hop reasoning are involved. For example, in Appendix D.3, the model uses information from all conversational rounds to gradually narrow down the possible range of words and eventually identifies the secret word. You can find more examples in Appendix D.1 and D.2. To strengthen this point, we will add case studies for a detailed analysis of reasoning results.

Dear reviewer kg44,

Thank you for taking the time to review our work. We would appreciate your feedback on the clarifications and additional analysis we provided:

• W1: tradeoffs between using retrospective analysis versus collecting reasoning traces during gameplay.

• W2, Q1, W3: How game sessions could involve sophisticated multi-hop, inductive, deductive, or abductive reasoning with case studies (Appendix D.1, D.2, D.3).

• Q2: why win rates alone are insufficient as a metric for the Taboo game, as outcome metrics often conflict with procedural metrics.

• W4, Q6: clarifications on the objectives of our comparisons and the key insights derived from ranking analyses.

We are happy to address any further questions or concerns you may have.

Best,

Authors

Dear reviewer kg44,

As the discussion deadline approaches, we would like to ask you kindly review our responses and paper revisions. We are glad to provide any additional clarifications that you may need.

Best,

Authors