BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games

We thank the reviewer for their thoughtful feedback and for acknowledging the timeliness of our benchmark, and the potential impact of having identified the “knowing-doing” gap.

We address your concern and question below:

[Q1] Comprehensiveness of the benchmark and VLM results

We appreciate the reviewer’s concern regarding the comprehensiveness of the benchmark and its evaluation of VLMs. While it is true that the visuals in video games may be somewhat out-of-distribution (OOD) compared to the training data of current VLMs, many models have been exposed to a variety of multimodal datasets, potentially including content related to video games. This makes their underperformance in these scenarios an interesting result worth further investigation, as it points to challenges in leveraging visual information for decision-making in dynamic environments.

AgentQuest, as a benchmark, aims to provide a rigorous framework for evaluating models in complex, long-horizon tasks. The fact that VLMs struggle in certain tasks is not a limitation of the benchmark but rather a reflection of the current state of model capabilities. These results highlight specific gaps in the decision-making abilities of VLMs, particularly in leveraging visual context in dynamic settings—a challenge also shared by standard LLMs. We believe this is precisely the role of a good benchmark: to reveal weaknesses that inspire further research and innovation.

To further address your concern, we conducted additional experiments with claude 3.5 sonnet in VLM mode. Interestingly, this model showed improvements in performance when using visual input compared to its language-only counterpart, suggesting that some VLMs can benefit from visual observations even in challenging OOD scenarios. We include these updated results in our fully anonymous leaderboard website https://agentquestanon.github.io/ and in the revised paper.

[Q2] Clarifying reasoning in line 72

We apologize for the confusion. The line action = self.llm_cilent(reasoning) contains an error. Instead of “reasoning” it should be written “history”. We thank the Reviewer for spotting this and we have fixed the issue in the revised version of the paper.

Conclusion

We thank the reviewer for their constructive feedback, which has helped us strengthen the comprehensiveness and clarity of our work. We hope the added experiments and clarifications addressed your concerns and hope that based on this you will consider increasing the score for our paper, or point out what still stands in the way for a higher rating.

Thank you for your feedback. Games have a long tradition as a strong metric for AI due to their high complexity, long horizon and interactivity. Identifying tasks that are both long-horizon and interactive outside of games that are appropriate for both LLM and VLM evaluation is challenging.

Web-browsing tasks typically involve shorter decision chains compared to games, lacking the deep sequential dependencies that define long-horizon challenges.

While coding benchmarks can be considered somewhat long-horizon and interactive, they lack the dynamic, real-time decision-making and feedback loops that games inherently provide and that we are interested in. Moreover, there are already numerous coding benchmarks available.

We would also not expect a particular benchmark to cover all long-horizon interactive tasks one can imagine. However, if you have specific suggestions or domains in mind, we'd greatly appreciate your input.

We thank the reviewer for their thoughtful feedback and for recognizing the strengths of AgentQuest, including its integration of diverse environments and comprehensive evaluation metrics. We appreciate your constructive criticism, which has helped us improve the clarity and scope of our work. Below, we address each point raised.

[Q1] Evaluating more agentic models

We appreciate the reviewer highlighting the importance of evaluating models specifically designed to enhance agentic capabilities. AgentQuest establishes a baseline for widely-used instruction-following models in a zero-shot setting, providing the research community with a foundation for future innovation. As a benchmark paper, our primary goal is to enable the research community to evaluate and improve agentic models systematically.

That said, we fully agree that testing more advanced models that have been designed to enhance agentic performance is important. AgentQuest is designed to facilitate this kind of evaluation of fine-tuned models, inference-time strategies, and reinforcement learning approaches, as described in section 2.2. To address your concern, we have expanded our experiments in the revised paper to include additional models, specifically we added Claude 3.5 Sonnet, smaller Llama-3.1/3.2 variants (1B/3B/8B)-it, and more. Claude 3.5 sonnet has been trained on computer interaction [1], which likely enhances its capabilities in agentic tasks like AgentQuest with visual inputs, which may explain its state-of-the-art performance on the benchmark in VLM mode.

We are also excited to highlight that AgentQuest will be open-sourced alongside a public leaderboard, enabling researchers to submit their models. You can see a fully anonymous mock version of the website here https://agentquestanon.github.io/. We hope the community will leverage this platform to address the limitations of current approaches and drive further innovation.

We thank the reviewer for the positive comments on our work and for recognizing that AgentQuest is a well-rounded benchmark that addresses key limitations in current LLM/VLM agentic capabilities with metrics allowing for a detailed analysis on model performance.

We address your comments below:

[Q1] On the underperformance of VLMs

We agree with the reviewer that the underperformance of VLMs when visual information is included deserves careful investigation. This challenge, also noted in recent work [1][2][3][4][5], is an important area for ongoing research and we have conducted more experiments to explore it further.

Specifically, in our new experiments with Claude 3.5 sonnet on AgentQuest, we observed improved performance in the VLM mode compared to the language-only mode using identical prompts. This suggests that some VLMs can utilize visual information effectively. Claude-3.5-Sonnet's superior performance could be attributed to its training on computer interaction [6], which likely enhances its capabilities in agentic tasks like AgentQuest with visual inputs. We note that Gemini 1.5 Pro also was able to properly utilize visual inputs, with its performance in vision-language mode being comparable to language-only mode. We believe AgentQuest will be a valuable benchmark to measure progress in VLMs going forward.

[Q2] Including more long-context models

We thank the reviewer for suggesting additional models to evaluate. In response, we expanded our evaluations to include several additional models:

• Qwen2.5 (7B/72B)-instruct,
• Qwen2 VL (7B/72B)-instruct,
• Claude-3.5-sonnet,
• Llama-3.1/3.2 variants (1B/3B/8B)-it. You can find the results in our updated paper or on our fully anonymous submission website: https://agentquestanon.github.io/ and in the table below:

LLM mode

VLM mode

We also evaluated LongAlpaca-7B and LongChat-v1.5-7B-32k as suggested. However, these models failed to produce meaningful evaluations due to their design as chat-oriented models rather than true instruction-following models. Specifically, these models are fine-tuned from Llama 2 chat variants and exhibited behavior better suited for conversational tasks, often generating verbose and irrelevant responses instead of reliably outputting the required actions. As a result, their trajectories consistently scored 0% across all environments in the benchmark.

Our additional findings highlight the crucial role of robust instruction-following capabilities for successful performance in AgentQuest. Models must be capable of reading the rules of the game, understanding the action space, and outputting actions to navigate and complete the tasks evaluated by the benchmark. We have added this consideration to Section 3.1 of the revised paper.

[Q3] Adapting the benchmark to new multimodal architectures

AgentQuest is designed to be extensible and forward-compatible to new architectures, as shown in section 2.2. The AgentQuest team is going to support and keep up with newly released multi-modal models, making it as easy as possible for future users. We are excited for users to submit models utilizing such type of video reasoning, however, we also recognize that currently it is very expensive to use videos with multimodal LLMs. We do plan on supporting video observations once prominent models with efficient video-processing capabilities become available, as highlighted in section 6 of the revised paper.

Dear Reviewer, to further address your concerns on VLM performance, we conducted additional experiments using a 3D Minecraft-like rendering of the Crafter environment developed by us. The findings are included in our updated shared rebuttal and detailed in Appendix B.3 of the revised paper. We kindly invite you to review these updates and share any further thoughts.

We thank the reviewer for their thoughtful feedback and for acknowledging the strengths of our work, including the comprehensive experiments, diverse environments, insightful analyses, and reproducibility of our benchmark. We deeply appreciate your constructive suggestions, which have helped us refine and improve our paper.

Below, we address each of your comments:

[Q1] More related environments should be compared

We appreciate the suggestion to discuss additional environments. A key design criterion for AgentQuest is that all included environments are solvable entirely by LLMs alone, with minimal reliance on computationally expensive visual processing. While environments such as MineDojo and MineRL are excellent for exploring open-ended agentic tasks, their high computational requirements and complex multimodal inputs would hinder the parallelizability and accessibility of our benchmark. In contrast, NetHack offers a similarly challenging environment but remains resource-friendly, making it ideal for AgentQuest.

Regarding multi-agent collaboration environments like Overcooked, we agree these are valuable for evaluating teamwork. While AgentQuest currently focuses on single-agent capabilities to probe foundational decision-making skills, we don’t exclude that in the future we may support multi-agent environments as well. We have added a discussion on these environments and their tradeoffs to Section 5 of the revised paper.

[Q2] Visualization of the environment

We agree with the reviewer that additional visualizations would enhance the intuitiveness of our analyses. Unfortunately the constraints of a pdf paper restrict us from having videos or a large number of images.

AgentQuest logs agents’ trajectories, making post-evaluation visualization straightforward. Upon open-sourcing, we will launch a website with a leaderboard linking to trajectory logs for all submissions. This will allow users to explore agent behaviors in detail. You can find a work-in-progress, anonymous version of this website here https://agentquestanon.github.io/, including some example gifs of the environment visuals.

We believe this solution will make AgentQuest’s visual analysis accessible and comprehensive.

[Q3] Multi-Agent LLM decision Making

Thank you for raising this question! We interpret it in two ways and address both:

Including multi-agent environments: AgentQuest currently focuses on single-agent scenarios to evaluate core decision-making skills such as long-term planning and systematic exploration without added complexity. While multi-agent environments are valuable, expanding to include them is a compelling future direction. Multi-Agent decision-making in single-agent environments: AgentQuest already supports multi-agent decision-making strategies applied to single-agent tasks. Researchers can integrate methods such as debate [6][7] or collaborative prompting. We anticipate that such techniques could significantly enhance performance, as noted in Section 6.

We have added these considerations in the revised paper.

[Q4] Hallucinations and invalid actions in AgentQuest

AgentQuest handles hallucinated or invalid actions by: Providing feedback to the model that the action was invalid. Defaulting to a standard “do-nothing” action if supported by the environment, or a movement to “north/up” direction otherwise. Logging invalid actions in trajectory statistics for user analysis. These details have been added to the revised paper. Thank you for highlighting this.

[Q5] On the limitations of VLMs

We appreciate the reviewer’s suggestion to discuss further cases of the limitations of VLMs. From our experiments, we observe significant variability in VLM performance. While some models, like Llama 3.2, perform worse in VLM mode, often failing to integrate visual information into coherent decision-making, other models such as Gemini 1.5 Pro and the newly run Claude 3.5 sonnet, effectively utilize visual inputs.

These findings align with challenges documented in recent work [1][2][3][4][5], which identify key limitations in VLMs, such as biases toward natural image-text pairs and optimization for image description rather than action-based reasoning. We have added the above and further considerations in Section 6 of the revised paper.

Conclusion

We thank the reviewer for their constructive feedback, which has allowed us to refine AgentQuest further. We hope that the revisions and additional analyses adequately address your concerns. We hope that based on this you will consider increasing the score for our paper, or point out what still stands in the way for a higher rating. Thank you for your time and effort in reviewing our work!

Dear Reviewer, we would like to offer a contrasting view on why the absence of multi-agent environments does not limit the completeness of our benchmark. While we agree that multi-agent research is extremely important, it is orthogonal and complementary to the goals of this benchmark. Multi-agent environments are valuable for studying coordination and communication, whereas AgentQuest is designed to probe foundational agentic capabilities such as sequential decision-making, long-term planning, spatial reasoning, and systematic exploration.

We are not claiming that AgentQuest addresses all potential agentic challenges. Instead, it provides a rigorous and focused framework for evaluating core capabilities of LLMs and VLMs. Adding further multi-agent environments would increase complexity and resource requirements for running the benchmark, potentially hindering accessibility for the broader research community, while targeting different research questions.

That being said, we value your concerns and have conducted additional experiments to explore VLM limitations. Specifically, we developed a 3D Minecraft-style rendering of the Crafter environment to investigate whether enhanced visual representations improve performance. We invite you to review these findings, which are included in our updated shared rebuttal and detailed in Appendix B.3 of the revised paper.

We hope this discussion has provided clarity and convinced you to reconsider your view of AgentQuest as a complete benchmark in its current scope. Please let us know if you have any further thoughts or feedback, and we will get back to you swiftly.

Dear Reviewers,

We conducted a new set of experiments to answer common questions raised by the reviewers related to the underperformance of some VLMs on AgentQuest. Specifically, we investigated whether the visual representation could affect model performance in the Crafter environment.

To address this question systematically, we developed an augmented version of the Crafter environment that renders scenes in 3D using Minecraft-style models and textures, instead of the original 2D tiled textures, see Figure 4 in the revised version of the paper for visualization. We hypothesized that VLMs might perform better with 3D renderings, given that natural images which were probably a major part of the training dataset, represent three-dimensional environments.

Interestingly, our results revealed that the switch to 3D rendering did not lead to improved performance. We observed that in several instances, the models performed slightly worse in the 3D-rendered environment compared to the original 2D version. This sheds new light on the existing results. These results suggest that the underperformance of VLMs in our benchmark may stem from more fundamental limitations in their architecture or reasoning capabilities, rather than from the specific visual representation format. The models struggle with complex vision-based reasoning tasks regardless of whether the visual input matches the distributions they were likely trained on.

These new findings have been included in Appendix B.3 of the revised paper, and invite you to explore the 3D Crafter renderings and results presented there. While this sheds new light on existing results, we acknowledge that further investigation into this area is warranted.

We hope these updates address your concerns and look forward to hearing your thoughts.