Are Large Vision Language Models Good Game Players?

Q.1 Finding 3 - I find it less sound. Maybe I am missing something, but - Yes, VLMs produce long answers, and the expected answers are short, but this is an evaluation problem, not the model’s fault. There are several ways to overcome it - few-shot prompts with short answers, parsing the answers with regex to a closed set of answers, a model-based LLM, and so on.

A.1 We appreciate the reviewer’s insightful feedback. Regarding Finding 3, our observation highlights a difference in instruction-following capability between open-source and commercial models. Specifically, commercial models tend to produce long responses due to human preference alignment via RLHF. However, our evaluation does NOT penalize models simply for producing lengthy answers; rather it focuses on their inability to adhere to the expected or specified response format.

For example, in the question-answering task, we provide multiple examples as references for the expected answer format. For a question such as:

"Which player has more pieces on the board, 'Black', or 'White'? Please respond with 'Black', 'White', or 'Equal'."

The model is expected to choose one of these three options. However, we observed that commercial models frequently fail to follow this expected output format and instead produce verbose but irrelevant explanations that do not align with the task requirements.

Recognizing the limitations of the existing evaluation approach, we have revised the question-answering task to a closed set of answers (multiple-choice) format based on the reviewer’s suggestions. This modification avoids formatting issues and ensures a clearer focus on the reasoning and perception abilities that the Q&A task aims to evaluate. More details are provided in our response to the reviewer’s second question.

Dear Reviewer iEip, Thank you for your valuable feedback on our submission. We hope that our responses have addressed your concerns. Please let us know if there are any remaining issues or further questions, and we will be happy to provide additional clarifications. We appreciate your time and effort during the discussion period.

Q.1 One concern relates to the scale of the overall contribution. As the authors note, there are previous efforts such as SmartPlay (Wu et al., 2023) and GAMA-Bench (Huang et al., 2024) that explore the use of games for evaluating LLMs. The SmartPlay work, for example, also proposes a suite of games and assesses the individual capabilities (such as instruction following, reasoning, and planning). As the authors highlight, it is true that these works only operate in text space (SmartPlay converts image input into text-based visual descriptions for games that require it, rather than feeding in RGB images to the models). This is an important distinction, particularly given the focus of the current submission on assessing detailed perception. Nevertheless, the proximity of prior work somewhat diminishes the contribution of this submission.

A.1 We appreciate the reviewer's thoughtful feedback and acknowledgment of the importance of evaluating LLMs in game-based environments. While it is true that previous works, such as SmartPlay (Wu et al., 2023) and the concurrent GAMA-Bench (Huang et al., 2024), have explored the use of games for evaluating LLMs, our work introduces significant distinctions and contributions that go beyond these efforts. Specifically, the major differences and advancements provided by our submission include:

• Multi-modal Input. Existing works, such as SmartPlay and GAMA-Bench, focus exclusively on pure-text language models and rely on text-based descriptions of game states as inputs. However, this design limits its applicability in multi-modal scenarios and overlooks many unique challenges that arise from processing visual inputs. To bridge this gap, our framework incorporates multi-modal input by directly using screenshots of game states instead of converting them into text. This approach enables the evaluation of detailed perception and visual reasoning capabilities of LVLMs in a way that purely text-based systems cannot.

• Incorporation of Competitive Environment. The games selected in existing works are predominantly single-player, focusing only on individual capabilities without considering the competitive abilities of LLMs. In contrast, 4 out of the 6 games in our framework are two-player competitive games. These games require LVLMs to operate in adversarial settings, where they must not only strategize effectively but also counter the opponent's moves. This focus on competition highlights a novel and critical skill set not addressed in prior works.

• Granular Assessment. Unlike SmartPlay and GAMA-Bench, which primarily assess end-to-end game-playing performance, our work decomposes the evaluation into multiple sub-tasks. These include perception, question answering, rule-following, and end-to-end gameplay. This decomposition enables a more granular assessment of different LVLM capabilities in game environments. While SmartPlay and GAMA-Bench provide only overall game performance metrics, our approach allows for a deeper understanding of LVLMs' strengths and weaknesses across distinct abilities.

• Systematic Definition of Evaluation Framework. We systematically define the abilities required for game-based tasks, including perception, reasoning, decision-making, and adversarial gameplay, along with their level of difficulty in each game. This comprehensive framework enables multi-faceted evaluation and facilitates in-depth analysis, helping identify specific limitations of existing LVLMs. Such insights are crucial for guiding the future development and enhancement of LVLMs.

Therefore, although our work shares some similarities with existing works on game-based environment evaluation for LLMs, the fact that we go beyond merely adapting previous text-based approaches to a multi-modal setting highlights the novelty of our contributions. Our work not only introduces a multi-modal evaluation framework but also provides new insights and unveils unique challenges, offering fresh perspectives on LVLM evaluation and development. We have incorporated these distinctions into the related work sections of the updated version of our paper.

Q.3 I’m curious to better understand the motivation behind the authors’ choice of using "unbeaten rate" to measure gameplay performance?

A.3 Thank you for the thoughtful question. The reason we chose to use "unbeaten rate" as the metric is that for some games, such as Tic-Tac-Toe, it is theoretically impossible to achieve a perfect win rate if the opponent plays optimally. In such scenarios, using "success rate" would not accurately reflect the model's performance, as a draw could still indicate strong gameplay capability. Therefore, we adopted "unbeaten rate" (which includes both wins and draws) to better capture the model's overall ability to avoid losing.

Q.4 Given access to games simulators which enable repeated games, this seems like a very high variance estimator of capability.

A.4 Thank you for raising this important point. We understand that evaluating gameplay performance can be sensitive to specific conditions, such as the initial game state or the opponent's strategies. To ensure the robustness of our evaluation, we opted to simulate real gameplay interactions between the LVLMs and their opponents, rather than relying on a fixed set of examples. This approach provides a more dynamic and comprehensive assessment of the models' performance. To minimize potential variability and ensure reliable results, we conducted one thousand gameplay simulations for each setting and calculated the average performance metrics. This large-scale evaluation helps smooth out any inconsistencies and provides a stable estimate of the model's capabilities.

Q.5 Moreover, it is tightly anchored to the choice of opponent implementation for each game. Did the authors consider alternatives (for example, creating a pool of opponents at different thresholds and computing Elo ratings?) Did the authors consider evaluation against non-search-based opponents?

A.5 Thank you for the reviewer's thoughtful feedback. In our framework, we implemented various embedded AI-opponent agents with different skill levels. For example, we used the SunFish chess engine, which allows us to configure opponents with varying Elo ratings to simulate different levels of difficulty. Similarly, for other search-based opponents, we adjusted the search algorithm's hyperparameters to control their strength. While this flexibility in opponent design exists, our findings suggest that the primary bottleneck for current LVLMs is not the level of the opponent, but rather their fundamental ability to process visual inputs and play complete games effectively. Nonetheless, we acknowledge the value of using a broader pool of opponents, including non-search-based ones, and plan to incorporate this in future work.

Q.6 Taking Tic-Tac-Toe as an example, presumably minimax will produce optimal play, so the best that can be achieved is a sequence of draws?

A.6 For Tic-Tac-Toe, it is true that an optimal minimax strategy always leads to a draw if both players play perfectly. To account for this, we used the "unbeaten rate" as a performance metric, which includes both wins and ties. This ensures that even if all outcomes are ties, the LVLM's ability to avoid losses is properly captured in the evaluation.

Dear Reviewer 2f5P , Thank you for your valuable feedback on our submission. We hope that our responses have addressed your concerns. Please let us know if there are any remaining issues or further questions, and we will be happy to provide additional clarifications. We appreciate your time and effort during the discussion period.

Q1. I noticed that in Findings 6. "Gameplay Data Could Enhance LVLMs' Reasoning During SFT," the early work MiniGPT-4 was used as a base model. Since the SFT stage of MiniGPT-4 only used detailed description data, it is difficult to prove that the performance gains are due to the VQA data from the game, rather than other factors like data format. Therefore, I believe that the experiments in Table 5 are not sufficient to fully support your conclusions. I suggest supplementing this with more robust base models, such as LLaVA-1.5 or LLaVA-OneVision.

A.1 We thank the reviewer for their valuable feedback. We selected MiniGPT-4 as the base model for its simplicity and ease of implementation during our preliminary exploration. However, after considering the reviewer's insightful comments, we recognized the limitations of this choice and conducted additional experiments with a more robust base model, LLaVA-OneVision, to further investigate the impact of game-based instruction data for SFT on LVLMs.

LLaVA-OneVision employs three training stages: Language-Image Alignment, High-Quality Knowledge Learning, and Visual Instruction Tuning. However, since the intermediate models from these stages are not publicly available and the whole instruction tuning process involves nearly 5M instruction data, replicating such large-scale instruction tuning experiments within the response period was not feasible. Instead, we conducted a preliminary exploration by fine-tuning the final LLaVA-OneVision model using game-based instruction data. The detailed configurations of our experiment are as follows:

• Model: LLaVA-OneVision-0.5B
• Adaptor: LoRA(r=32, lora_alpha=32, lora_dropout=0.1, target_modules=['q_pro', 'k_proj', 'v_proj', 'o_proj'])
• Learning Rate: 2×10^-6
• Trainable Module: LoRA adaptor; all other modules such as Vision-encoder, Vision-Language Projectors are fixed.
• Instruction Data: 2k × 6 Games; 5k × 6 Games; 10k × 6 Games.
• Epoch: 1

We then evaluated the fine-tuned model on the MME benchmark, which encompasses a variety of multi-disciplinary tasks for evaluating LVLMs. The performance differences compared to the original LLaVA-OneVision (baseline) model are summarized below:

From the experiment results, we did not observe improvements in the MME benchmark. This could be due to several reasons:

• Extensive Pre-training: LLaVA-OneVision was extensively pre-trained across three stages with nearly 10M high-quality vision data, making it challenging to observe notable performance gains from a relatively small-scale fine-tuning with game-based instructions.
• Intermediate Models Unavailable: Since the intermediate models from the training stages were unavailable, we directly performed SFT using the game-based instructions on the final fully fine-tuned LLaVA-OneVision model, which might not have been optimal and could have even degraded performance.
• Training Scope and Resources: LLaVA-OneVision employs a very large batch size of 512 during its training, where the full model, including the projectors, vision encoders, and other components, is trainable. In contrast, due to resource constraints, our fine-tuning experiments were limited to updating only lightweight LoRA parameters while keeping the rest of the model frozen. This difference in training might have further limited the performance impact.

Given these observations and the reviewer's valid concerns, we acknowledge that Finding 6 in the original paper requires more in-depth analysis to draw definitive conclusions. To avoid potential confusion, we have removed this finding in the updated version of the paper. We leave a more detailed and comprehensive exploration of this topic for future work.

We sincerely thank the reviewer for pointing out this important issue, which has helped us refine our analysis and future directions. Importantly, the removal of Finding 6 does not affect the overall contribution of LVLM-Playground, as the benchmark's core strengths lie in the comprehensive evaluation framework for perception, reasoning, decision-making, and adversary abilities across structured game environments. We believe these contributions remain robust and we hope the benchmark could be used for advancing the evaluation and development of LVLMs.

Dear Reviewer ciTF, Thank you for your valuable feedback on our submission. We hope that our responses have addressed your concerns. Please let us know if there are any remaining issues or further questions, and we will be happy to provide additional clarifications. We appreciate your time and effort during the discussion period.

Q.1 The perceiving task prompts the models to output matrices that correspond to the game states, which not only tests for perception but also requires strong instruction-following ability.

A.1 We thank the reviewer for the insightful comment. We agree that the perceiving task inherently tests not only perception but also instruction-following ability, as the task requires the model to output matrices corresponding to game states. However, instruction-following is a fundamental capability of LLMs, and it is linked to almost all other abilities, such as perception, reasoning, and decision-making. This interdependence makes it inherently challenging to fully isolate instruction-following from other capabilities in practical evaluations.

To address the reviewer’s concern and provide a deeper analysis, we conducted additional experiments on the perceiving task, focusing on how board density may affect model performance:

Taking Gomoku as an example, we varied the number of pieces on the board across five levels, ranging from sparse to dense, and evaluated the model's perception accuracy under these conditions. Based on the results, we found that most models are highly sensitive to board density. For example, InternVL2-8B performed exceptionally well on sparse boards, even outperforming all commercial models in the 0-45 pieces set, but its performance dropped sharply to 0.097 due to unexpected outputs (such as looping behavior described in Finding 1 of the main paper) on the densest boards (181-225 pieces).

This result highlights that while the models possess instruction-following capability to output the game state in the required matrix format, their ability to follow instructions is significantly influenced by their perception performance. If a model struggles to accurately perceive the detailed game state, it often fails to generate the expected matrix, demonstrating that perception and instruction-following are interdependent.

Therefore, we believe the design of the perceiving task reasonably reflects and enables a fair comparison of the perception capabilities across different LVLMs, while also accounting for their instruction-following abilities in practical scenarios.

Dear Reviewer giEs, Thank you for your valuable feedback on our submission. We hope that our responses have addressed your concerns. Please let us know if there are any remaining issues or further questions, and we will be happy to provide additional clarifications. We appreciate your time and effort during the discussion period.

Dear Reviewer giEs,

As the author-reviewer discussion period is nearing its end, we were wondering if our response has adequately addressed your concerns. We are open to providing any further clarifications if needed and greatly value your feedback.

Thank you again for the effort you have dedicated to reviewing our paper and for offering many constructive suggestions that have helped us further improve our work.

Thank the authors for the detailed responses and updated results based on the suggestions! They addressed some of my concerns, with a few ones left:

Regarding Q.2 Part-1, I'm still skeptical about the numbers for commercial models -- are they low because the answer extraction can be improved? or do the authors believe that the results suggest commercial models have worse perception abilities than some of the open source models?

The reviewer also appreciates the error analysis in A.2-2 but still has questions: was the error analysis done manually or automatically (and if the latter, how)? Also, "Incorrect selection" is a very broad category and lisiting out the error count doesn't help understand why commercial models have higher error rates than open source ones, making it still unclear to the reviewer if their worse performance is due to weaker perception, worse generalizability or the task formulation -- can the authors provide deeper qualitative analysis into these particular results?

Regarding Deeper Qualitative Analysis

We acknowledge that Incorrect Selection is a broad category that may not fully capture the weaknesses in perception, generalizability, or task formulation. To address the reviewer's concern and provide deeper qualitative insights, we conducted an additional analysis where models were prompted to explain their reasoning after making a selection. This approach helps us better understand why specific options were chosen.

We used the Tic-Tac-Toe game as an example for this analysis. Specifically, we modified the task prompt format from:

....
Your Option:

to

.... Then explain the reason why you chose that option.
Your Option:
Explanation:

We added an additional instruction: "Then explain the reason why you chose that option." to the end of the original prompt.

This modification explicitly requests the model to provide a reasoning process alongside its answer, enabling a more detailed qualitative analysis of the model's decision-making. Since this explanation-based prompt sometimes broke the multiple-choice structure and the original regex-based parser failed to extract correct answers, we conducted a manual review of the responses and primarily focused on qualitative analysis.

By enforcing the models to provide an explanation for their choice, several limitations in their reasoning and instruction-following abilities become apparent, aligning with our initial findings in the Q&A task. From the new qualitative results, we found that commercial models often struggle to fulfill the given instructions and are prone to providing nonsensical or irrelevant responses. For example:

• Claude: The model repeatedly provides responses such as, "I apologize, but there is no multiple-choice question provided in your message for me to answer." or "Given this game state, if I were to suggest a move for the 'O' player, I would recommend: B2." These examples show that the model fails to understand and follow the given instructions.
• Gemini: Similar issues arise, where the model anticipates playing the game instead of answering the question. It often suggests moves such as, "Placing an X in A1 will complete three Xs in a row (A1, B1, C1) making X the winner," or "Placing an X in A1 will create a three-in-a-row victory for X. This is the only available move that achieves an immediate win."
• GPT-4o: The model also tends to interpret the task as gameplay instead of answering the questions. For example, it produces responses like, "To determine the best move, the aim is to either win the game or block the opponent from creating a winning line. As you are playing as 'O,' you should look for opportunities to create a line of three O's or prevent 'X' from doing so." This demonstrates a clear lack of adherence to the instructions provided in the prompt.

On the other hand, when we ran the open-source LVLMs with the exact same prompt, they were more likely to follow the instructions and answer the questions correctly. For example:

• Qwen2-VL: The model consistently generates reasonable explanations for its choices, even when selecting incorrect answers. For example:
  "The red marks are represented by the letter 'X'. In row C, there are two red marks, one in cell C1 and another in cell C3. Therefore, the correct answer is B. 2." and
  "There are three 'O's on the board: one in row A, column 1; one in row C, column 1; and one in row C, column 3."

• InternVL2: Similarly, this model demonstrates a good understanding of the task and adheres to the Question Answering baseline, providing logical reasoning. For example:
  "Upon examining the Tic-Tac-Toe board, we can see that there are three 'O's present. One 'O' is in the top right corner (position 2), another 'O' is in the middle left corner (position 2), and the third 'O' is in the bottom right corner (position 2). Therefore, the correct answer is B, as there are three 'O's on the board."

Although these open-source models make incorrect choices due to poorer perception capability, they generally follow the instructions and explain their reasoning in a coherent manner.

Based on the above qualitative analysis, we draw the following observations to address the reviewer's concerns:

• Since the commercial models generally performed much better on the perception task than the open-source models, we do not believe the performance gap in the Q&A task is due to weaker perception abilities. Instead, it appears to stem from difficulties in understanding and following the question-answering instructions.

• The multiple-choice format forces models to choose from predefined options, requiring them to follow the instructions. Although this enforcement ensures that commercial models select an answer from the candidate options, it may obscure the fact that they do not authentically understand or adhere to the instructions. This often results in responses that resemble random selections from the available options, which explains their low performance.

• We believe the commercial models' low performance may stem from the gap in task prompts. Many studies have shown that discrepancies between training and testing prompts can lead to poorer performance for LVLMs. As mentioned in the introduction section of our paper, existing evaluation frameworks often rely on varying prompts, which can significantly impact model performance. This motivates LVLM-Playground to provide a more consistent evaluation environment, where all models are evaluated using the exact same task prompts. In LVLM-Playground, we intentionally use casual, consistent task descriptions rather than meticulously designed, model-specific prompts. We believe that with more carefully tailored prompts, commercial models might achieve significantly better performance. However, the existing task prompts may favor open-source models, which are less influenced by RLHF, while highlighting potential weaknesses in commercial models to generalize across different prompt styles.

Since the response system does not support figure attachments and has character limitations, we have included these new qualitative results and findings in the revised paper, as well as a few example evaluation logs of each model. The reviewer may refer to the appendix, starting from Page 43 to 48, for the complete analysis and examples.

We sincerely appreciate the reviewer's constructive comments and suggestions. We will extend these deeper qualitative analysis to all other games in the LVLM-Playground and provide all evaluation logs in the final release of LVLM-Playground.

We sincerely thank the reviewer for their thoughtful feedback and valuable insights. We understand the concerns raised and would like to address them as follows:

Q.1 The authors mentioned that their analysis suggests commercial models are worse than open source models because (1) their failure in understanding and following the QA instructions; and (2) they do NOT generalize well to the specific task prompts used, which are favorable to open source models -- both points are very specific to the particular task formulation in this work instead of generalizable findings about commercial vs. open source models (e.g. it doesn't mean commercial models are always worse than open source models in following QA in general) and might be misleading if such general claims are made in the paper.

A.1 We appreciate the reviewer’s thoughtful comments. First, the authors would like to clarify that we never made any general claim that commercial models are always worse than open-source models in our paper. In fact, our experimental results show that commercial models generally outperform open-source models across most tasks, except for some Q&A tasks. Based on this observation, we hypothesized that RLHF might hinder instruction-following ability in specific cases based on detailed qualitative analysis, but we did not make any strong conclusion or general claim that commercial models are universally inferior to open-source models under such scenarios.

When we first observed the worse performance of commercial models on the Q&A task, we were similarly surprised and conducted multiple validations to ensure the reliability of the results. Additionally, following the reviewer’s suggestion, we converted the semi-open-ended Q&A task into a multiple-choice format to avoid potential penalties due to template-following issues. However, under the existing settings, all verified experimental results consistently demonstrated that commercial models show a performance gap compared to open-source models in this specific Q&A task.

We acknowledge the general expectation that commercial models typically outperform open-source models in most tasks. However, as the reviewer rightly noted, “it doesn't mean commercial models are always worse than open-source models,” it could also mean that “open-source models may not always be worse than commercial models in certain specific contexts.” We appreciate this nuance and understand the reviewer’s concern about whether our findings, based on specific task prompts and formulations, are generalizable. We address the reviewer's concerns as follows:

• We set the task prompts at the very beginning of this study, and once the prompts for each game were established, we intentionally did not modify them or introduce prompt engineering to optimize the performance of specific models. Our goal was to obtain vanilla results that minimize human intervention and reflect the models' inherent capabilities. As we emphasized in the introduction, many LVLMs tune their system prompts to optimize performance for specific tasks during evaluation. While we could similarly enhance the performance of commercial models by fine-tuning systems and task prompts, we believe that such modifications could lead to misleading results, suggesting strong performance in specific tasks while lacking true generalizability.
• All our analyses are based on the specific experimental settings provided by the LVLM-Playground framework, particularly the structured game environments and tasks designed within it. We do not aim to make a general claim about universal model performance; instead, as clarified in our motivation, our goal is to explore the challenges current LVLMs encounter under structured game-based environments. Based on the results and findings, we do not believe the results in the paper are misleading but rather reflect the models' behavior in this specific context.

In summary, we acknowledge that the failures of commercial models in a single QA task within LVLM-Playground may not reflect a universal performance gap between commercial and open-source models. This is why we divided the evaluation into different sub-tasks, such as perception, rule-following, and end-to-end gameplay, where commercial models generally outperform open-source models as expected. For the Q&A task, we believe the current experiments highlight some potential limitations in existing LVLMs, which are not confined to commercial models but also apply to open-source ones. Therefore, we retained the original results based on the consistent, initial task prompts rather than optimizing performance through meticulous prompt engineering. We do not consider this approach misleading but rather a reflection of the models' behavior under a uniform and controlled evaluation framework.

Q.2 Further, the authors even mentioned the commercial models "tend to interpret the task as gameplay" instead of question answering -- doesn't it mean that commercial models could be better game players (to answer the question posed in the paper title)? but these results could mislead readers to believe they are not. It makes the reviewer doubts if the task formulation in the paper truly represent gameplay.

A.2 It is true that commercial models tend to interpret the task as gameplay instead of question-answering. However, such responses do NOT demonstrate that commercial models are already proficient game players.

• The Q&A task was not designed to evaluate the general gameplay capability of the models but rather their ability to understand game states and game rules. We believe this is a critical component of gameplay. Therefore, providing gameplay-related responses to these tasks does not indicate that the models can play the games effectively.
• To assess end-to-end game-playing capabilities, we designed a specific E2E Playing task for this purpose. In this task, all existing LVLMs, whether commercial or open-source, failed to outperform a basic search-based AI opponent even in the simplest game, Tic-Tac-Toe. This highlights their limitations in handling multi-modal input environments based on game screenshots. It also reflects the misalignment between their multi-modal reasoning capabilities and their pure text-based reasoning capabilities.
• In addition, as per our response to Reviewer ciTF's #Q.3 regarding the intermediate metric for E2E gameplay, the results demonstrate that commercial models outperform all open-source models under the intermediate metric. Although they struggle to complete full gameplay for games like Sudoku and Gomoku, this experiment suggests that commercial models exhibit stronger capabilities in gameplay-related tasks compared to open-source models under the E2E setting. We have incorporated these additional results into the paper, ensuring clarity. Therefore, we do not believe our paper or experiments in a way mislead readers into thinking that open-source models are better suited for game environments than commercial models.

Given the performance across these subtasks, which separately evaluate different aspects of gameplay, the existing models performed poorly in end-to-end gameplay. Therefore, the authors do not believe the results or the task formulations are misleading. Instead, these findings provide an accurate reflection of the models' limitations in structured, multi-modal environments.