Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

Thank you for taking the time to review our work and providing constructive feedback! We greatly appreciate your recognition of our contribution in the realm of pure-vision based autonomous GUI agents, including our grounding-then-planning training pipeline and open-sourcing the large-scale, curated data. We are also delighted that you recognized the promising performance of our approach in both offline and real-world online benchmarks, which highlights its generalizability across multiple digital platforms. We have fully open-sourced our roadmap towards building pure-vision based autonomous agents, and are committed to support further research in this field.

We also noticed you have some constructive questions about our work, and we're happy to elaborate further below!

C1: The first paragraph in Section 3.3 claims that “we evaluated AGUVIS across four comprehensive benchmarks”. But only three benchmarks are mentioned here.

Thank you for pointing out this inconsistency! We will correct it in the revised version of the manuscript.

Q1: Which metric is used for the result comparison in Table 1?

For Table 1 (results on Screenspot), we report Click Accuracy as the evaluation metric, following the definitions provided in the original benchmark paper [1]. Specifically, Click Accuracy measures the proportion of test samples where the predicted location falls within the ground truth element's bounding box.

[1] SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. Cheng et al., 2024.

Q2: Why are the evaluation metrics used on different benchmark datasets different? For example, the computational cost analysis is only conducted on Mind2Web-Live, and the element accuracy and Operation F1 are only conducted on Multimodal Mind2Web.

Thank you for your thoughtful feedback! The differences in evaluation metrics arise because each benchmark is designed to assess different aspects of the agent's performance, and we adhere to the metric definitions specific to each benchmark. This approach ensures that our results remain comparable to previous methods and provides a clear understanding of the agent's performance under distinct conditions.

• Offline vs. Online Evaluations: Some metrics, such as element accuracy and Operation F1, are more suitable for offline evaluations (e.g., Multimodal Mind2Web), where the goal is to evaluate step-level accuracy and action predictions without environments. In contrast, online evaluations like those on Mind2Web-Live focus on final trajectory-level execution results, which offer a more holistic view of the agent's ability to complete tasks in dynamic environments.

• Benchmark-Specific Metrics: Different benchmarks introduce their own unique metrics to capture specific aspects of performance. For instance, Mind2Web-Live introduces the USD Efficiency Score, which evaluates the efficiency of resource utilization during task execution. This metric provides insights into the agent’s performance in real-world settings, where efficiency is a key factor.

While we acknowledge the importance of following standardized evaluation metrics, we also agree that it is valuable to align the evaluation criteria with the specific needs and goals of each benchmark. Developing unified metrics across benchmarks requires significant effort but remains an important direction for our future work.

Q3: Based on the results in Table 8, it seems that the proposed method paired with GPT-4o for planning, the result is better than that of the AGUVIS-72B. I was wondering about the results of the same configuration on other benchmarks.

We appreciate your interest in this observation! In addition to the AGUVIS-7B paired with GPT-4o for planning shown in Table 8 of OSWorld, we have also applied this configuration to other online GUI agent benchmarks, such as Mind2Web-Live, AndroidWorld, and MobileMiniWob, as detailed in Tables 4 and 5.

Q4: For the VLM-based trajectory augmentation process, the current inner monologue components are generated by GPT-4o. Can other latest VLMs be used and compared?

Our data processing pipeline is model-agnostic and can be extended to more recent VLMs. Our data pipeline utilizes a VLM annotator to generate inner monologue reasoning from human-annotated actions. This is a natural fit for human behavior, where humans easily generate actions but find it costly to record their inner monologue. Conversely, current VLM struggle to make accurate action decisions but can more easily infer inner monologues given action decisions. This novel approach provides a promising pipeline for building agent trajectories with reasoning. We believe such annotations could equivalently be generated by open-source VLMs. Although time and cost constraints during rebuttal have prevented us from fully re-collecting data and retraining with alternative models, we believe that exploring other open-source VLMs for this process is an exciting avenue for future work!

We sincerely appreciate your thorough review and positive assessment of our work. We're particularly encouraged by your recognition of our comprehensive evaluation benchmarks, the engineering effort involved in dataset aggregation and standardization, and the value of our open-sourced models and datasets to the research community.

Regarding your questions and comments:

Q1: On Inner Monologue's Contribution to Grounding Performance

Thank you for this insightful question. You correctly observed that while Inner Monologue is introduced in Stage 2, it also enhances grounding performance, which might seem counterintuitive. Let us clarify: In standard trajectories without inner monologue, the data structure is:

(High-level goal G, observation $o_1$, action $a_1$, observation $o_2$, action $a_2$, ...)

When we augment with inner monologue, we introduce low-level instructions, transforming it to:

(High-level goal G, observation $o_1$, low-level instruction $a_1^{inst}$, action $a_1$, observation $o_2$, low-level instruction $a_2^{inst}$, action $a_2$, ...)

This transformation creates high-quality explicit instruction-action pairs ($o_i$, $a_i^{inst}$, $a_i$) within each step, essentially embedding "grounding examples" throughout the trajectory. The model learns to:

1. Interpret high-level goals into precise low-level instructions
2. Ground these instructions to specific UI elements
3. Generate appropriate actions

As shown in Table 6, removing inner monologue reduces performance on ScreenSpot from 84.4% to 79.3% and also has a strong impact on AndroidControl Low-Level tasks (80.5% → 69.1%). This suggests that the ability to decompose tasks into explicit low-level instructions significantly improves grounding precision.

Q2: On Self-Plan vs. Enforced Plan Settings

Thank you for pointing out this confusion. As shown in Appendix E.2.1 and Figure, the difference between these settings lies in how we prompt the model in response:

Self-Plan Setting:

<|im_start|>assistant<|recipient|>
[The model decides whether to plan first with all or directly execute actions with os]

In this setting, the model autonomously determines whether to generate planning thoughts based on task complexity. For simple tasks like "Click the 'Buy' button," it might directly output:

<|im_start|>user
Click the 'Buy' button.
<|im_end|>
<|im_start|>assistant<|recipient|>os
pyautogui.click(0.34, 0.45)
<|im_end|>

While for some implicit tasks, it might choose to plan first:

<|im_start|>user
Send current webpage.
<|im_end|>
<|im_start|>assistant<|recipient|>all
Thought: To share the current page, I need to find and click the share icon, which is typically represented by a network or link symbol. This icon is usually located in the browser's toolbar or menu.\nAction: Click the share icon in the browser to share the current page.
<|im_end|>
<|im_start|>assistant<|recipient|>os
pyautogui.click(0.34, 0.45)
<|im_end|>

Enforced Plan Setting:

<|im_start|>assistant<|recipient|>all
Thought: [The model is forced to generate planning thoughts before actions]

The enforced plan setting explicitly requires the model to engage in high-level reasoning before taking actions. As noted in our error analysis (Section 4.5), this enforced planning resolves approximately 20% of grounding errors by encouraging the model to carefully consider the task context, potential ambiguities, and available UI elements before committing to action.

We will further clarify this part in our revised manuscript. Thanks for helping improve our work!

Q3: On AGUVIS Model's Generation Approach

Yes, AGUVIS generates the next step iteratively rather than first generating an overall plan and then executing steps. At each time step, given the current observation and task history, the model:

1. Generates thoughts about the current state in relation to the goal
2. Determines the appropriate next action
3. Executes the action and receives a new observation
4. Repeats the process until task completion

This iterative approach allows AGUVIS to adapt to changing UI states and unexpected outcomes during task execution, rather than rigidly following a predetermined plan.

Thank you for your thoughtful review of our AGUVIS paper. We appreciate your recognition of our comprehensive roadmap for developing a pure-vision GUI agent, particularly our data curation approach and training strategies. Your positive assessment of our experimental validations across multiple benchmarks is encouraging.

Q1. Recently, many co-current works have proposed similar unified model architectures, such as UI-TARS, OS-ATLAS, ShowUI, and CogAgent-9B. Could the authors compare their approach with these works, particularly in terms of data? The proposed method in this paper appears to be one of the best in balancing data utilization efficiency and model performance.

We're pleased to see the growing interest in GUI agent research through concurrent works like UI-TARS, OS-ATLAS, ShowUI, and CogAgent-9B[4]. In comparison with these impressive efforts, we believe that AGUVIS makes a unique and complementary data contribution to the GUI agent community in several important ways:

• Reasoning via Inner Monologue: A defining feature of AGUVIS is the use of reasoning via inner monologue, which is not present in many concurrent works such as OS-ATLAS, ShowUI, and CogAgent-9B. This inner monologue allows AGUVIS to perform reasoning during interaction, which is crucial for the effective handling of complex tasks across multiple platforms (mobile, desktop, web). As shown in Section 4.3, AGUVIS achieves strong generalization by using a unified action space after inner monologue reasoning, enabling knowledge transfer across diverse environments. Additionally, the inner monologue surprisingly enhances GUI grounding performance, as detailed in Section 4.2 and Appendix E.1.2, as well as in our response to Reviewer cBz9 Q1. We believe that these contributions jointly underpin our balance between data utilization efficiency and model performance.
• Open-Source Data Collection and Pipeline: While recent concurrent works, such as OS-ATLAS and ShowUI, focus on grounding-centric training data, and more recent UI-TARS has made significant strides in leveraging in-house human-annotated trajectories with reasoning, AGUVIS offers a unique advantage with its large-scale, open-source data collection. Our dataset collection not only includes both unified grounding and trajectory annotations but also integrates reasoning into the data pipeline. This transparency and open-source nature of our data collection make AGUVIS a valuable resource for the community to build upon and extend our work more easily.

We greatly appreciate your recognition of AGUVIS as one of the leading approaches in balancing data efficiency and model performance. We believe that our open-source data and the novel inner monologue reasoning offer complementary contributions that will drive continued progress in the development of autonomous GUI agents.

We sincerely appreciate your thoughtful review and the opportunity to further clarify our contributions.

W1: Technical Innovation in Training Method

Thank you for your insightful comments regarding the training methodology! We recognize that recent approaches share similar high-level components, such as grounding pre-training and decision training. However, AGUVIS's primary innovation lies in its unified framework that incorporates inner monologue reasoning. This enables the agent to operate effectively in new, previously unseen platforms without additional retraining. This integration, along with our training methods, distinctly sets AGUVIS apart from recent methods [1,3,4].

These innovations are empirically validated:

• Section 4.1 shows that our two-stage training strategy outperforms joint training and ablations.
• Section 4.2 highlights how inner monologue enables explicit planning and task decomposition, surpassing reactive action decision methods.
• Section 4.3 demonstrates cross-platform generalization, where AGUVIS trained on web/mobile tasks transfers well to desktop GUIs in OSWorld.

We would also greatly appreciate it if you could review these contributions highlighted by other reviewers as well. We strongly believe that the framework design, analyses, open-source model, and data contributions could significantly benefit and advance the GUI agent community.

W2: Claim of “first fully autonomous vision-based GUI agent that operates without relying on closed-source models”

Thank you for highlighting this important aspect of our work. Our claim specifically emphasizes that AGUVIS, once trained, operates fully autonomously during task execution without dependence on closed-source models. Crucially, the model itself—including all training data, architectures, and procedures—is completely open-sourced, enabling full transparency and reproducibility.

While our data pipeline leveraged GPT-4o to generate inner monologue reasoning from human-annotated actions, this step was employed solely to enrich reasoning data rather than define action policies. And we believe such annotations could equivalently be generated by open-source VLMs, and exploring this capability with open-source models is definitely an important part of our future work.

Moreover, while recent research ([1, 2, 3]) has similarly aimed at competitive performance using open-source models, AGUVIS uniquely provides a unified vision-based framework capable of seamlessly operating across multiple diverse GUI environments. We will further clarify this comparative context in our revised manuscript to better highlight AGUVIS’s contributions and clearly differentiate it from concurrent work.

W3. Testing settings with multi-image trajectories vs. text history + single image.

We appreciate the suggestion. AGUVIS currently uses text history + single image, balancing context richness and computational feasibility. Incorporating multiple images poses significant token cost challenges (~1200 tokens/image), especially when paired with inner monologue reasoning and 72B model size.

Nonetheless, we recognize the potential of multi-image context and plan to explore it in future iterations. Our open-source format supports multi-image inputs, enabling future work to build on this. As models like Qwen2.5-VL with advanced video ability, we anticipate AGUVIS will scale to multi-frame settings more efficiently.

W4. Evaluation on other datasets, such as AITZ.

Thank you for this suggestion. We evaluated AGUVIS on the AITZ benchmark. Results are summarized below and demonstrate AGUVIS’s advanced performance:

These results will also be included in our revision to strengthen the paper’s empirical validation.

Q1. Discrepancy between results on Mind2Web-Live and AndroidWorld.

We appreciate your attention to the detailed results. We think the discrepancy between Mind2Web-Live (Table 4) and AndroidWorld (Table 5) may stem from the differences in how GPT-4o understands and interacts with web interfaces versus mobile interfaces. We observed that GPT-4o tends to be distracted by extraneous details in high-resolution, information-rich web interfaces, which can lead to failures in planning. We will clarify these environmental factors and their impact on agent performance in our revision.