VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents

Thanks for your thoughtful advice and review on VisualAgentBench! We really learn a lot from your comments. Here are our responses:

1. Details on Evaluation

Thanks for your question. Our detailed prompts and few-shot examples (if used) are presented in Appendix B to F in the original submission. In the interaction rounds for embodied and GUI agents, output of environments as images are sent to models in the next round as the observations follow prior practices.

2. Analysis on Error Recovery Behaviors

Thanks for your suggestion! According to your advice, we have updated detailed error recovery examples in the Appendix G.6. Basically, error recovery works in planning as an agent realizes the outcome of certain action has led to unexpected results. It usually involves two steps: return to the original state, and make another trial. We believe it is fundamental to agents due to their imperfect decision making (and so it is for us humans).

Additionally, as you suggest we analyze the average steps needed for agents to recover from error to the correct directions (as below, based on proprietary gpt-4o and opened glm-4v, only for those finally successful tasks). We find that GUI tasks usually require more steps to recover, as the action spaces for them are very large (e.g., any clickable elements on the web pages). And fine-tuned glm-4v has shorter mean error recovery steps compared to gpt-4o, probably because it can only recover from simpler errors. Due to the limited time period of rebuttal, we will provide more analysis and statistics regarding error recovery in the final version.

3. Detailed Analysis on Agent Error Modes

Thanks for your comment. We first update example successful and failed trajectory case study in the Appendix G for readers’ intuitive understanding. In fact, we find agents to fail due to a variety of reasons, which can be a bit hard to comprehensively attribute in such a short period of response time. However, we endeavor to provide some statistics about major types of errors we observe, by sampling around 20 error traces in each environment for gpt-4o and internvl-2. More results and analysis will be presented in the final version of the paper.

• Visual Grounding Error: Wrong detection or recognition of objects/elements in the visual observation.
• Invalid Action: Outputting wrong formats of actions.
• Loop: Agent repeats generating the same actions without quitting.
• Task Limit Exceed: Agent does not accomplish the goal within reasonable maximum steps.
• Hallucinated Task Completion: Agent makes wrong judgment on whether it has accomplished the task.

Thanks for your thoughtful advice and encouragement on VisualAgentBench! We really learn a lot from your comments, and spare no efforts during this response stage to make improvements according to your questions. Here are our responses:

1. On Training Data and Subsequent Scaling Method

Thanks for your advice. The initial purpose of providing training trajectory in VAB is to provide baseline data to include open LMMs into benchmarking, which performs too badly at instruction following without training. From experimental results, the goal is well accomplished as ability gaps between different LMMs’ on agents are distinguished after training.

And indeed, we can scale up training data according to the synthetic methods we have described. The most probable one is LMM Agent Bootstrapping, from which we can bootstrap and filter high-quality new trajectories based on existing LMM agents. The challenge for the method lies in building a strong automatic filter, which we would like to leave for future work due to its complexity.

2. Verify and Conduct RL training

Thanks for your great suggestion. Despite its feasibility to support RL, we do not intend to claim the support of it in the existing implementation of VAB. VAB is actually a very initial work in this field which aims to first set up the basic framework, tasks, and environments for future algorithmic study, such as RL.

However, it is possible to use RL given our implementation, since VAB has provided interactive environments for online sampling. For example, recent works have shown the feasibility to run RL training for GUI agents in both Android Virtual Device and WebArena-Lite environments [1,2] similarly adopted in VAB. Therefore, VAB could offer an ideal testbed to foster RL algorithms for effective LMM agent training.

3. RL Specialist Training

We agree that it is interesting to see if RL specialists could work in these environments. However, considering the implementation costs for customized training and architectural designs, we believe it is better to leave the idea for future study.

4. General Multimodal Abilities

Thanks for your question. In VAB, for the purpose of benchmarking, we fine-tune LMMs merely on agent trajectories for simplicity. However, the maintenance of general multimodal abilities can ben addressed by mixing agent-domain with general-domain data, as indicated in literature [3,4].

5. Providing Example Trajectories

We have uploaded 20 example trajectories (including both successes/failures and proprietary/open LMMs). Thanks for your advice.

6. Use of Vision Input in Evaluation

Thanks for your question. We adopt the strategy because of two reasons: 1) most LMMs when the work was done only support single-image input. 2) Consider the long-horizon multi-turn interactions in agent tasks, it would introduce overlength context tokens if we keep all historical visual observation (i.e., images).

However, the strategy is still multiturn interaction & conversation, since we only remove the historical visual content but leave other input and model-generated text contents in the context.

References:

[1] DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning. NeurIPS 2024.

[2] WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. arXiv:2411.02337.

[3] Agenttuning: Enabling generalized agent abilities for llms. Findings of ACL 2024.

[4] Fireact: Toward language agent fine-tuning. arXiv:2310.05915.

Thanks for your thoughtful advice and review on VisualAgentBench! We really learn a lot from your comments. Here are our responses:

1. Advantages and Comparison to Domain-specific Benchmarks

Thanks for your comment. The interactive environment for all tasks in VAB is a crucial advantage, which clearly distinguishes it from many existing benchmarks. For example, the RoboVQA is a video-based static planning dataset against prepared trajectories. Its static nature prevents it from reflecting agents’ real performances in interactive real-world environments, where agents could present typical behaviors of exploring and error-recovering that are vital but only seen in interactive evaluation as VAB-OmniGibson does. In addition, VAB-AndroidLab and VAB-CSS environments are also among the first to provide interactive evaluation in their specific domains. VAB-Minecraft is the first to provide a publicly available evaluation set for agent evaluation (while previous Minecraft works fail to open-source their evaluation data). For web tasks, it is certain that the VisualWebArena and WebArena are quite good, so we only do some refinements to them in VAB (e.g., fix problematic judging functions).

More importantly, the emphasis of VAB is the comprehensive benchmarking across multiple environments for interactive LMM agent evaluation. In the context of LLMs and LMMs, researchers wish to evaluate them across multi-domain datasets to verify their generalizability, which is a key to these models. However, existing domain-specific benchmarks fall short to satisfying the need due to their design features. Thus one of the VAB’s unique values lies in its LMM-oriented design and comprehensive evaluation sets.

2. On Quality of Training Trajectories

Thanks for your comment. In fact, except for WebArena and OmniGibson, training trajectories for all other 3 environments are either human-annotated or LMM-bootstrapped to ensure diversity. But in fact, the adoption of program-based trajectory collection, while sacrificing some task diversity (alleviated by task instruction rewriting), actually improves the overall quality in boost of data accuracy as we have discussed in Section 3.2. Take the web data you mentioned as an example. Human-collected web browsing trajectories often exhibit some problematic features that could harm trained LMM performances. For instance, human trajectories usually present repeated scrolling ups and downs for information seeking. LMMs trained on such data would present patterns to repeat scrolling actions without knowing when to stop. Program-based collection avoids the problem by eliminating such up and down loops.

Additionally, the training trajectories provided in VAB serve as a foundation for benchmarking open LMMs, which otherwise struggle significantly with instruction following without training. Experimental results demonstrate that this goal is successfully achieved, as training clearly distinguishes the capabilities of different LMMs when applied to agents. Overall, we position VAB not as a finalized solution, but as a robust starting point for the community to build upon and synthesize more effective data for agent training. We hope VAB will pave the way for further advancements.

3. Function-calling for Agent Tasks

Function-calling capability is a core requirement for language agents and a necessary condition for enabling them to take actions across different environments. To clarify, our ultimate goal is to advance LMMs into generalist agents capable of operating in diverse domains, each characterized by a unique set of actions (i.e., functions). For instance, in our five environments, each has its own distinct action space. To effectively instruct an LMM to complete tasks in a specific domain, it must be able to invoke the appropriate functions (actions) based on the environment’s specifications. Therefore, VAB is not “biased” toward models with strong function-calling capabilities. Instead, achieving better performance through robust function-calling is an inherently desirable property for generalist agents

4. Physical Dimension of Robot Arms

In VAB-OmniGibson, we allow the LMM agent to grasp objects within a 1.2-meter radius. In Figure 1 Round 3, the banana is within this range, so the LMM agent can successfully grasp it.

1. I still don't get the rationale behind why these tasks are chosen. The robotics, Minecraft, UI and webpage design are good domains but are not the only domains that matter. Why not include driving, drone and flat design, etc as well? Since VAB is not an exhaustive benchmark, how to choose each task is crucial. Also I don't know what does a comprehensive benchmark mean by the authors in the rebuttal. Does that mean if a VLM could do well in the VAB benchmark then it can do well in most cases?
2. Neither human-annotated nor LMM-bootstrapped ensures a high diversity. Only the careful task space design could achieve that goal. Maybe the authors could elaborate more about how the task lists are carefully crafted to maximize the diversity for each domain?
3. I don't understand why a model without function calling ability could not be a good VLM agentic model across different environments. I think Qwen-VL 1 is quite solid a VLM and could do well in many driving, robotics and UI tasks when fine-tuned as a VLA model without any function calling ability.
4. This is simply wrong, most robot arms like UR5 or franka have a workspace of 700mm-1000mm, if you allow the VLM to grasp objects within 1.2m there will be tons of failures. This makes me wonder if the VAB-OmniGibson really executed the action of VLM?

I think the authors fail to address most of my concerns and decide to downgrade my rating.

4. Regarding the workspace of robot arms in OmniGibson

We sincerely thank you for this important technical complement about robot arm workspaces. Given our previously mentioned goal and context of VAB, in OmniGibson we make a simplification: as long as the agent is within the radius of the target object, we would recognize the grasping as successful. The 1.2m we originally chose, is a loose median value after our surveying (we must acknowledge our limited expertise in robotics). We did observe some robot arms with working radius up to 1.3-1.7m [10-11].

Following your recommendation, we tested gpt-4o with the 1000mm radius. The results show:

While this adjustment slightly affects performance, it's unlikely to significantly alter VAB's main conclusions. We are committed to conducting a complete re-evaluation with this more accurate parameter.

We truly value your thorough review and expert insights, which have helped us identify important areas for improvement. Please don't hesitate to raise any additional concerns - your guidance is essential for strengthening this research.

Reference:

[1] Yao, Shunyu, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR 2022.

[2] Park, Joon Sung, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. "Generative agents: Interactive simulacra of human behavior." UIST 2023.

[3] "AgentBench: Evaluating LLMs as Agents." ICLR 2023.

[4] Yang, Jingkang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang et al. "Octopus: Embodied vision-language programmer from environmental feedback." ECCV 2025

[5] Wang, Guanzhi, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. "Voyager: An open-ended embodied agent with large language models." TMLR 2024

[6] Zhu, Xizhou, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang et al. "Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory." arXiv preprint arXiv:2305.17144 (2023).

[7] Hong, Wenyi, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang et al. "Cogagent: A visual language model for gui agents." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290. 2024.

[8] Si, Chenglei, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. "Design2Code: How Far Are We From Automating Front-End Engineering?." arXiv preprint arXiv:2403.03163 (2024).

[9] Hendrycks, Dan, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. "Measuring Massive Multitask Language Understanding." ICLR 2021

[10] UR10: https://www.rarukautomation.com/collaborative-robots/ur-6-axis-collaborative-robots/ur10-robot/

[11] CR20A: https://www.dobot-robots.com/products/cra-series/cr20a.html

Thanks for your thoughtful advice and review on VisualAgentBench! We really learn a lot from your comments, and spare no efforts during this response stage to make improvements according to your questions. Here are our responses:

1. More Explanation and Analysis on Poor-Performed LMMs

Thanks for your comment. We first update example successful and failed trajectory case study in the Appendix G for readers’ intuitive understanding. In fact, we find agents to fail due to a variety of reasons, which can be a bit hard to comprehensively attribute in such a short period of response time. However, we endeavor to provide some statistics about major types of errors we observe, by sampling around 20 error traces in each environment for gpt-4o and internvl-2. More results and analysis will be presented in the final version of the paper.

• Visual Grounding Error: Wrong detection or recognition of objects/elements in the visual observation.
• Invalid Action: Outputting wrong formats of actions.
• Loop: Agent repeats generating the same actions without quitting.
• Task Limit Exceed: Agent does not accomplish the goal within reasonable maximum steps.
• Hallucinated Task Completion: Agent makes wrong judgment on whether it has accomplished the task.

2. Clarification on Role of Visual Information

In VAB, visual information is indispensable for all tasks. We provide a detailed explanation in Appendix A.2 to address your concern. In most cases, the agent must rely on visual input to determine the affordances for its actions, such as identifying objects to interact with in a room or buttons to click on a website. For your example of “teleport,” you are correct that this represents a rare case of a global action available at any time. However, deciding whether it is reasonable to use this action requires the agent to recognize that it is trapped in certain locations, which can only be inferred using visual information, specifically the last several frames of gameplay.

3. On Clearer Annotations for Figure 2

Thanks for your comment. We have updated more annotations in the figure and the corresponding caption.