Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

Thank you for taking the time to review our work and providing constructive feedback! We greatly appreciate your recognition of the comprehensive experiments we conducted across multiple digital platforms, which demonstrate the the generalizability and promising performance of our approach. Additionally, we are pleased that you highlighted our integration of planning with action grounding, which is clearly key factor of advancing autonomous GUI agent. We are committed to fully open-sourcing our roadmap to foster future research.

We also noticed you have some constructive questions about our work, and we're happy to elaborate further below!

W1: Dataset ablation studies on impact of datasets from different device domains with unified action space.

A: We understand your concerns and would like to further clarify this concern:

• We would like to respectfully emphasize that observation and action spaces are typically coupled. For example, with text-based observations, the corresponding click action would be click(element_id), while with pure vision observations, it would be click(x, y). Simply mixing non-unified data makes it difficult for the model to benefit mutually from different data sources, which is exactly the challenge we aim to address.

• By utilizing pure vision inputs and a unified action space, we not only achieve mutual performance gains but also pave the way for developing more generalizable GUI agents. This approach eliminates the dependency on system-level APIs to access UI source code like HTML or Accessibility trees, relying solely on screenshots instead. Consequently, it enhances the agent's applicability across different platforms and devices.

• However, we understand your concerns about dataset ablation. We have added new experiments to verify the impact of cross-platform data on downstream tasks. We trained models using the Mind2Web training set, expanded web-only data, and the full Stage 2 data including both web and mobile, then tested their performance on web tasks. The results are as follows:

Our results indicate that expanding the web domain data enhances performance on web tasks. Notably, incorporating mobile data leads to additional improvements in web domain performance, as evidenced by the higher performance when using the combined Web + Mobile dataset.

These findings demonstrate that unified agent trajectory data can indeed achieve cross-platform benefits. The improved performance with the inclusion of mobile data validates our motivation to use a unified action space and pure vision inputs, as it enables the model to generalize better across different domains.

W2: L248 states that 'This approach significantly accelerates training by maximizing the use of each image without compromising accuracy, but the experiment confirming this point is nowhere to be seen. Additionally, the expression "we assume" and "robust" are informal.

A:

• In our preliminary experiments, we investigated the effect of the grounding packing strategy. We trained two models on the SeeClick web dataset, one employing the strategy and the other excluding it. We observed that grounding packing could significantly accelerate training efficiency, reducing overall training GPU hours from 6 to 1. Moreover, we found this strategy didn't hurt performance and even slightly improved the performance (76.8 vs. 73.3 on the ScreensSpot Web split). These results have been added to Appendix Section C.2.

• Additionally, Thank you for pointing out our inappropriate wording. We have revised the term "assume" and "robust," replacing it with more scientifically rigorous language to ensure clarity and precision.

W3: Justification reasoning step (inner monologue) generation

A: Thank you for your insightful feedback. The role of reasoning (inner monologue) is indeed significant. We think that the role of inner monologue can be attributed to two factors:

• The generation of an inner monologue elicits the reasoning process concerning current actions.
• The produced reasoning inner monologue provides informative context that informs subsequent planning stages.

To further validate the importance of inner monologue, we conducted ablation studies to demonstrate the performance improvements achieved by incorporating inner monologue (IM) in training:

Our findings indicate that:

• Inner monologue clearly enhances performance on planning benchmarks.
• Inner monologue can also improves low-level instruction grounding. This is because the low-level instructions in our augmented trajectory act as atomic instruction and grounding action pairs, enhancing GUI grounding ability.

Overall, this ablation study clearly demonstrates the performance improvements brought by incorporating inner monologue in training, further justifying its effectiveness and supporting our analysis.

W4: Due to the use of a more powerful pre-trained Qwen2-VL, it is difficult for current comparison to prove the superiority of AGUVIS.

A: Thank you for bringing up this important concern. We acknowledge that using a powerful backbone like Qwen2-VL could make it challenging to attribute performance gains solely to our AGUVIS methodology. To address this, we emphasize that AGUVIS is designed to be model-agnostic, enhancing GUI agent capabilities through two essential stages including grounding and planning&reasoning.

In our original submission (Section 4.2 and Table 8), we demonstrated that applying AGUVIS to a weaker backbone model like LLaVA still results in surpassing previous SOTA performances and achieves results comparable to those using Qwen2-VL.

To further substantiate that the improvements stem from our methodology rather than the inherent strength of the backbone model, we conducted comprehensive ablation studies on both Qwen2-VL and LLaVA backbones. The results are summarized below:

The ablation results reveal several key insights:

1. Importance of Stage 1 (GUI Grounding): For ScreenSpot benchmark, both backbones show significant performance drops when Stage 1 is omitted, highlighting the necessity of diverse GUI grounding data provided in Stage 1 to handle the comprehensive evaluations of ScreenSpot.

2. Impact of Stage 2 (Planning & Reasoning): For planning benchmarks like MM-Mind2Web & AndroidControl, excluding Stage 2 leads to notable declines in performance, especially on planning and reasoning tasks.

3. Model-Agnostic Effectiveness: Despite starting with lower baseline performance, LLaVA experiences substantial gains after applying the AGUVIS pipeline, surpassing previous SOTA methods and achieving results comparable to Qwen2-VL.

These comprehensive ablation studies confirm that the superiority of AGUVIS arises from our innovative training methodology, rather than the inherent capabilities of the backbone model. By systematically enhancing both GUI grounding and planning abilities, AGUVIS delivers significant improvements across various benchmarks and backbones.

Dear Reviewer pn3z:

We sincerely thank you for these constructive comments and evaluation of our paper. With the ICLR public discussion phase ending in two days, we kindly ask you to take a look at our responses. Our rebuttal also provided additional experiments in response to your concerns. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Thank you for taking the time to review our work and provide detailed feedback! We sincerely appreciate your recognition of our work as a comprehensive roadmap for developing a pure vision GUI agent, particularly in terms of data curation and training strategies. Constructing the entire pipeline for this framework is indeed challenging, as the pure vision setting is a novel and promising area in this field. Previous data and evaluation methods needed significant adaptation for this context. The challenge is heightened by our model's ability to plan independently during online realistic evaluations, rather than relying on closed-source models like GPT-4o. This makes AGUVIS an autonomous pure vision agent model. We are committed to fully open-sourcing our roadmap to support future research in this field.

We also noted that you have concerns about some details of our work and we are pleased to explain further:

W1: Limited novelty - mainly pre-training VLMs with data combinations

A: We sincerely thank the reviewer for their constructive feedback. Below, we address each concern regarding the novelty of our work:

1. Insights Provided:
   • In Section 4, we presented extensive training ablations, demonstrated the model-agnostic capabilities of our approach with LLaVA, highlighted the efficiency advantages of pure vision modeling, and provided error analyses that emphasize the benefits of improved planning.
   • To further address your concerns, we have added Appendix E, which includes additional training ablations on both Qwen2-VL and LLaVA models, analyses of dataset synergy, visualization examples from online evaluations, and experiments demonstrating generalization to real-world environments. These additions offer valuable insights into the viability and advantages of pure vision GUI agents, and we believe they will guide future research in this area.
2. Addressing Computational Intensity:
   • Exploring Model Scalability: GUI tasks require complex decision-making. Therefore, it's important to explore the potential of high-capacity model. It worth to conduct evaluations across model scales (7B-72B) which can provide comprehensive insights into how model size impacts performance.
   • Efficiency Enhancements: Our pure vision modeling approach naturally reduces token consumption compared to previous methods. To further mitigate computational demands, we have implemented packing techniques to accelerate training, as detailed in Appendix C.2. These strategies enhance efficiency without compromising performance.
3. Dataset Integration as a Novel Contribution:
   • Novel Integration Pipeline: Integrating trajectory data with vision-based datasets was a significant challenge. We developed a novel data pipeline that ensures unified action space modeling and incorporates VLM augmentation with inner monologues, which are critical for task performance, as validated in our experiments.
   • We created an agent data pipeline that effectively leverages existing datasets and augments them with additional inner monologues. In the future, it will incorporate more agent trajectory datasets. We committed to fully open-source this data pipeline as a resource for scalable agent research.
4. Novel Module Design:
   • Pure Vision Agent Model: Our pure vision agent represents a novel approach to GUI agent modeling by unifying planning and grounding within a single framework. This design overcomes previous scalability and dependency issues, enabling broader applicability without relying on closed-source models or specific environments.

We believe that our work presents significant novel contributions both in methodology and practical applications. We hope that these clarifications address your concerns regarding the novelty of our work. We are happy to provide further details or engage in additional discussions to showcase the impact of our contributions!

Dear Reviewer Vo6b,

Hello! We appreciate your suggestions. Since the discussion period has started, we kindly ask you to take a look at our responses. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address them. We appreciate your time and consideration!

Authors

I want to thank for the author effort for including Appendix and include many information.

Efficiency Enhancements: Our pure vision modeling approach naturally reduces token consumption compared to previous methods. To further mitigate computational demands, we have implemented packing techniques to accelerate training, as detailed in Appendix C.2. ''' This is still unclear to me (regarding token consumption), could the author provide more explanation on this part?

Novel Integration Pipeline: Integrating trajectory data with vision-based datasets was a significant challenge. We developed a novel data pipeline that ensures unified action space modeling and incorporates VLM augmentation with inner monologues, which are critical for task performance, as validated in our experiments. ''' How does the integration pipeline work and how it differs from previous approaches.

Regarding Novel Integration Pipeline: Integrating trajectory data with vision-based datasets was a significant challenge. We developed a novel data pipeline that ensures unified action space modeling and incorporates VLM augmentation with inner monologues, which are critical for task performance, as validated in our experiments.

Can the author provide more explanation how this integration pipeline works and how it differential or novel than previous methods such as GUI-Course?

Training cost:

Expensive training costs might be challenging for other scholars to afford.

Could you provide details on the training costs for 7B/72B models at each stage? Are these models fully trained, or do they utilize LoRA for fine-tuning? Additionally, how does the performance compare across these methods? which will provide valuable reference for community. Thank you.

Q3: My primary concern remains with the focus of this work. While it emphasizes large-scale pretraining and various engineering optimizations, it still lacks significant technical innovation regarding the vision perception or language agents.

A: Thanks you for your recognition for our training pipeline and engineering optimization effort! Exploring the whole pipeline to build AGUVIS really challengging but exciting. We appreciate your focus on the techinical innovation and we are more than happy to clarify the key contributions of our work and its value to the field!

The core focus of our work is the development of an autonomous pure vision GUI agent framework capable of generalizing across diverse GUI environments and platforms. Autonomous means that the agent model needs to generate repliable plan and reason on its own, while pure vision perception with unified coordinate-based action space significantly enhances generalizability across various environments. This is akin to human interaction with GUIs, where actions are based on visual input rather than abstract representations like HTML elements or code snippets. (We comprehensively elaborated these challenges for GUI agents in L51-96.)

However, one of the major challenges we address is the lack of generalizable agent data due to the fragmented nature of existing GUI agent trajectories. These trajectories are often environment-specific, recorded with different observation and action formats, making it difficult to use them for training an autonomous agent. For instance, training a model to read HTML or select a target element using its ID is fundamentally different from training a model to click a target on a screen using pixel coordinates. Simply mixing these data formats compromises the model’s ability to generalize across varied modalities.

Furthermore, most prior methods do not integrate explicit planning and reasoning capabilities, often relying on closed-source models or training agents to make decisions without any explicit reasoning process. This dichotomy leads to either weak grounding in GUI interactions or insufficient reasoning abilities. For an autonomous agent to function effectively across multiple platforms, incorporating planning and reasoning is essential. Unfortunately, existing trajectory data typically consists only of actions and observations recorded by human annotators, without capturing the explicit reasoning that drives each decision. Annotating such reasoning is costly and labor-intensive, which is why no existing model autonomously completes tasks across different platforms without relying on external reasoning systems. Therefore, currently no agent model can autonomously complete tasks across various real-world platform environments or online evaluations, such as AndroidWorld (Mobile), Mind2Web-Live (Web), and OSWorld (Desktop), without assistance from closed-source models.

AGUVIS is the first open-source model that addresses these challenges. It is capable of autonomously completing tasks across diverse GUI environments, including mobile, web, and desktop platforms, with a unified data pipeline and vision-based perception system. Our results on the OSWorld benchmark—evaluating performance in a challenging real desktop environment—demonstrate AGUVIS’s superior performance (10.46), which approaches that of the recently released Claude computer-use API (14.90).

However, we still recognize that this is just the beginning. Our work on AGUVIS is the first step toward building autonomous GUI agents via imitation learning with large-scale, unified data collection. We hope that our transparent framework and pipeline will serve as a foundation for future research and inspire further advancements in the field!

Q4: Could you provide details on the training costs for 7B/72B models at each stage? Are these models fully trained, or do they utilize LoRA for fine-tuning? Additionally, how does the performance compare across these methods? which will provide valuable reference for community. Thank you.

A: Thank you so much for your attention to the training cost! All models mentioned in our paper were fully trained, and we shared our training details with cost in Appendix C.2. Since we built AGUVIS as a foundational GUI agent model, we did not use LoRA, as our experience suggests that full fine-tuning generally yields better results. Although we have made every effort to provide as many additional results as possible during discussion phrase, it is challenging to re-implement and train the LoRA baseline for comparison in the limited time remaining. We will strive to include these results in the next version. We sincerely hope for your understanding.

We deeply agree that training cost is challenging for many researchers. We are committed to open-sourcing all our models as foundation models so that others can fine-tune them to specialize them with new abilities and achieve better results.

We deeply appreciate you taking the time to respond to us and improve the rating. Your recognition of our comprehensive insights and additional experiments we conducted in a short period is truly encouraging for us to continue improving our work. We are very grateful for you taking the time to read our reply and raise more valuable questions. We are definitely more than happy to provide more explanations about these questions and discuss our contribution and future further.

Follow Up Q1: This is still unclear to me (regarding token consumption), could the author provide more explanation on this part?

A: We are happy to provide further details regarding reducing token consumption, which we believe is one of the core advantages of pure vision GUI agent modeling. To control a GUI, most previous methods required access to the source code of the GUI, which on the web is represented by HTML, and on operating systems (like desktop and mobile OS) is represented by the accessibility tree. These text-based data structures represent the interactive elements in the interface, and the agent selects an element and performs an action to complete the task. A clear drawback of this approach is that each observation is very long, and the length increases as the complexity of the GUI grows. This results in large encoding costs for the agent model. For instance, even after extensive cleaning and pruning of these textual trees, the average token consumption for an accessibility tree on OS is around 6k tokens per observation, and 4k tokens for web HTML. These long observations lead to high overhead in both the training and inference phases.

To address this, our pure vision approach uses screenshots of the interface as observations and controls the GUI through generated pyautogui actions and coordinates. For a 720p (1280*720) screenshot, thanks to dynamic resolution strategy of the NaViT image encoder, AGUVIS consumes only 1196 tokens while maintaining the resolution and aspect ratio of the screenshot. This significantly reduces token consumption, which improves training and inference efficiency. The efficiency advantage of this pure vision modeling, combined with its generalization as a unified representation of the GUI, further demonstrates the benefits of pure vision observation.

You can also find more motivation explanation in our paper (L51-59, L151-155), as well as details about dynamic resolution strategy in the NaViT paper itself[1]. We also highlight the advantage of token consumption in Section 4.3 with Figure 2. If you'd like to know more about GUI observation and NaViT, we would be more than happy to provide further explanations!

Reference

[1] Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution. Dehghani et al., 2024.

Follow Up Q2: Can the author provide more explanation how this integration pipeline works and how it differential or novel than previous methods such as GUI-Course?

A: Yes, we are pleased to provide further insights into our crucial data pipeline in building AGUVIS!

The core motivation behind developing the AGUVIS data pipeline is to unify the observation and action space for effective data scaling, while systematically incorporating data augmentation techniques into both Stage 1 (GUI grounding) and Stage 2 (planning & reasoning). This unified pipeline fundamentally enables two key improvements in AGUVIS: pure vision perception with a pluggable, unified action space and autonomous planning and reasoning capabilities.

Data Pipeline Overview

The integration pipeline begins by standardizing both grounding and trajectory data across different environments. This step is crucial because existing datasets are often designed for distinct environments with mismatched observation and action spaces, making direct integration between them a challenge. Our approach extracts all observations as images and converts action annotations into a unified coordinate-based grounding format, which is compatible with PyAutoGUI for consistent action execution across different platforms. We illustrated the unified action space and an example of plugin action space for mobile as an example in Appendix A.1 and A.2.

This standardization allows for significant and effective data scaling, expanding the grounding data to 1 million screenshot samples and the trajectory data to 35K tasks with 300K actions. Compared to previous works, this represents an order-of-magnitude increase in dataset size and complexity. The unification of observation and action spaces also facilitates cross-platform compatibility, enabling AGUVIS to work seamlessly across various GUI environments. We have further demonstrated the effectiveness of cross-platform benefits in Table 16, where we demonstrate unified mobile trajectories can effectively improve the performance of web browsing.

Novel Data Augmentation Strategies

Once the data is unified, we introduce two novel data augmentation strategies designed to improve the model's performance in both Stage 1 (GUI grounding) and Stage 2 (planning and reasoning):

Stage 1: GUI Grounding Augmentation

In Stage 1, our focus is on grounding the model’s perception of GUI elements and improving its basic GUI action capabilities. This involves using a template-based data augmentation strategy that transforms and reformats referring expressions and bounding box pairs into diverse instruction-action mappings. This step prepares the model for a wide range of GUI control tasks, enhancing its ability to generalize across different environments.

1. Atomic Action Augmentation We generate a variety of templates for basic GUI actions, such as click, doubleClick, rightClick, and moveTo. These templates directly map grounding referring expressions (e.g., GUI element labels) to PyAutoGUI-compatible actions. For example:

   • Original Data: ("Share Icon", [0.1, 0.3, 0.2, 0.4])
   • Augmented Instruction: Click Share Icon
   • Corresponding Action: pyautogui.click(0.15, 0.35)

   By varying these templates and adjusting bounding box coordinates, we ensure that the model is exposed to a wide variety of basic GUI actions, thus broadening its coverage of PyAutoGUI functions.

2. Primitive Skill Augmentation In addition to basic actions, we augment the dataset with instructions requiring multiple actions in sequence, such as dragging, highlighting, or copying. These skills are foundational for the model to complete more complex tasks. For example, given a text element ("Price: $100", [0.1, 0.3, 0.6, 0.8]), the model can perform the following actions:

   • Highlighting Text:

     User: Drag to select "Price: $100"
     Agent:
     pyautogui.moveTo(0.1, 0.55)
     pyautogui.dragTo(0.6, 0.55)
     

   • Copying Text:

     User: Copy "Price: $100" as answer.
     Agent:
     pyautogui.moveTo(0.1, 0.55)
     pyautogui.dragTo(0.6, 0.55)
     pyautogui.hotkey('ctrl', 'c')
     

   These scenarios enable the model to internalize common action patterns in Stage 1, establishing a solid foundation for more complex tasks in Stage 2.

Stage 2: Planning and Reasoning Augmentation

For Stage 2, which involves trajectory data and task planning, we introduce GPT-4o-based inner monologue augmentation. Specifically, we provide GPT-4o with a prompt that includes:

• A high-level task goal
• Previous action history
• The current ground truth action
• The current observation image, with the target object highlighted in a red box

GPT-4o is then tasked with generating inner monologue including explicit reasoning and detailed low-level instructions. The output is returned in JSON format, facilitating easy parsing and integration into the model’s training process. We have shown the prompt in Appendix B.2 for your reference. We also illustrated our augmented trajectory data in Figure 4 and compared the difference with previous method in Figure 5.

Thanks to our unified data pipeline, we can effectively apply this pipeline on all trajectory data. By incorporating inner monologue into the trajectory data, we significantly improve the model's ability to plan and reason about complex tasks, which is crucial for autonomous decision-making in real-world GUI environments. We have demonstrated the importance of inner monologue in W3.3 and Table 17.

Key Differences from Existing Approaches

We would also like to highlight our approach introducing several key advancements over previous methods such as GUICourse:

1. Unified Observation and Action Space Design: Previous methods like GUICourse usually operate on fragmented data spaces for web and mobile, which are often tailored to specific environments and requires fine-tune on difference downstream training data for platform/environment adaptation. Our unification data collection along with the use of PyAutoGUI for action commands, ensures that AGUVIS works seamlessly across different platforms and environments, a major advantage over previous systems.
2. Effective Data Scaling for Cross-Platform: Our pipeline supports a much larger data collection, expanding the grounding data to 1 million screenshot samples and trajectory data to 35K tasks with 300K actions—an order-of-magnitude increase in scale compared to prior works as shown in Table 10 & 11. We unify observation and action modalities, enabling broader cross-platform performance benefits as shown in Table 16.
3. Dual-Stage Augmentation: We integrate two complementary augmentation strategies—template-based grounding augmentation—in Stage 1, alongside VLM-based planning and reasoning augmentation in Stage 2. This design effectively enhances the model’s ability to handle both low-level actions during stage 1 and complex task planning during stage 2. In contrast, GUICourse didn't include inner monologue for trajectory data, which we demonstrate its importance for planning and reasoning in W3.3 and Table 17.

We deeply acknowledge and appreciate the contribution of previous methods such as GUICourse. But we also think this approach differs fundamentally from our work, which focuses on introducing explicit planning and reasoning with unified pure vision design to build a generalizable GUI agent. We truly believe our novel contribution can complement previous efforts to further advance agent capabilities in real-world evaluations.

We greatly appreciate your recognition of our work! We are pleased to hear that you acknowledge our motivation for creating AGUVIS, showcasing unified pure vision GUI agent design and autonomous reasoning with grounding capabilities through training with our valuable augmented data collection. To construct the collection, we built a data pipeline framework to integrate different data sources into a standardized format, and then implement multiple augmentation strategies we introduced in our paper to augment these datasets. We are continuously adding more datasets to this framework and are preparing to publish this repository with processed training data to benefit future research.

Meanwhile, we noticed that you have some constructive questions about our work. We are pleased to explain further below.

W1: Unclear formalization of belief state update 'belief state $b_t$ '

A: At L122, we stated:

At each time step t, the agent receives an image observation $o_t$ from the GUI environment, updates its belief state $b_t$ based on its previous actions and observations, and generates an inner monologue.

Our original intention was to describe how the model performs internal implicit reasoning based on the current situation and then generates an explicit inner monologue. However, since this does not refer to a specific part of the model output, it might cause confusion with the explicit reasoning thought denoted as $h_t$. We have removed this term to avoid confusion and reflected this change in the uploaded revised version. Thank you for your suggestion!

W2: Which VLM was used for inner monologue generation and how data quality was ensured?

• During our data curation, we use GPT-4o to augment agent trajectory by adding explicit reasoning thought and low-level instruction. As shown in newly added Appendix Section B.2, we present the input prompt to GPT-4o for generating detailed, low-level instructions. Specifically, the prompt is consist of high-level goal, previous action history, the current ground truth action command, and the current observation image, where the target object is highlighted with a red box. We then ask GPT-4o to complete the reasoning process and respond in JSON format for easy parsing.
• According to our Table 1, GPT-4o indeed performs poorly in GUI grounding, but we found that it performs well in understanding and interpreting actions with GUI screenshots, especially when provided with ground truth action commands and highlighted target objects. We leveraged this to annotate reasoning process of trajectories.
• Our qualitative evaluation shows high accuracy of the augmented data. Through a human study of samples from each datasets, as described in Appendix Section B.3, 86.7% of the augmented data successfully demonstrated intermediate reasoning that aligned with both the ground truth actions and overall goal intention. Only 7.8% of cases were affected by dataset noise, and 5.5% showed misinterpretation of action intention in clean data.

W3&4: Showcase and illustrate the prompt organization and resulting planning & reasoning trajectory examples

Thank you for your suggestion! In Appendix Figure 4, We illustrate the generated trajectories schema. In Appendix Section C.1, we have visualized our trajectory template to showcase the training prompts. Additionally, we highlight how our AGUVIS collection differs from existing data in Appendix Figure 5.

W5: More examples and explanation of the 'self-plan' setting

A: We appreciate that you noted the improvement of self-plan setting and it definitely deserve further explanation.

• In Appendix C.1, we present the training templates for Stage1 and Stage2. We use the special token <|recipient|> along with os or all to control whether the message content is an inner monologue or a pyautogui action command. Thanks to this design, we can use <|recipient|> during the inference phase to control the content generated by the model.
• In the self-plan setting, we do not add any word after <|recipient|>, so the model can choose to generate os to directly produce a pyautogui command, or generate all to first create natural language reasoning and then generate a pyautogui command.
• To demonstrate the effectiveness of planning, we visualized several hard examples of ScreenSpot in Appendix E.2.2. These examples illustrate how an additional reasoning step helps the model achieve more accurate grounding.

Dear Reviewer RjsL:

We sincerely thank you for these constructive comments and evaluation of our paper. With the ICLR public discussion phase ending in two days, we kindly ask you to take a look at our responses. Our rebuttal provided more clarification of our framework and additional experiments in response to your concerns. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Thank you for recognizing our work and providing constructive feedback! We're delighted to hear about your interest in our unification design and your acknowledgment of our extensive evaluation efforts to verify the model's promising performance. Creating a unified model is a significant challenge, as it requires unifying both the model's training data and the evaluation framework. We developed a comprehensive offline GUI agent evaluation framework and integrated our pure vision model into online evaluation environments for each platform. These, along with our training data, code, and models, should serve as valuable contributions to the community. We are in the process of releasing all these resources to support further research in general autonomous GUI agent studies.

We also noticed you have some constructive questions about our work, and we're happy to elaborate further below!

W1: Comparison to previous methods like AMEX and SeeClick which also use pure-vision representation and both grounding/planning datasets

A: Both of these previous work laid important groundwork in vision-based GUI interaction. Compared with these work, we would like to highlight two foundamental aspects of our work:

• Thanks to the unified design, AGUVIS is a general cross-platform pure-vision based agent instead of specialized agent for single platform for either desktop or mobile. Through unified training with pluggable action space, our model can be directly deployed on these platform. In contrast, AMEX is designed for mobile and SeeClick is required to fine tune for each environment/benchmark.
• AGUVIS introduces a novel explicit planning and reasoning framework that differs from previous approaches. We illustrate these differences with visual examples in Figure 5. While earlier approaches fine-tune models with multi-step trajectories, they typically generate action commands directly, bypassing the reasoning process and low-level instructions. On the one hand, the previous approach fails to elicit the reasoning ability of VLMs during generation; On the other hand, this force the model to predict next action conditioning on less informative coordinates action representation(e.g. pyautogui.click(0.868, 0.610)) rather than informative natural language low-level instruction (e.g. click on the buy button.). Therefore, we propose introducing an explicit reasoning process with a large-scale data collection to improve pure vision GUI agents by elicit reasoning ability and also use the low-level instruction as informative previous action history. Moreover, this explicit reasoning process provides explainability for analyzing, improving, and overseeing GUI agent behaviors. We believe this approach could serve as a foundation for future GUI agent research.

W2: Unclear benefits of using VLM to construct CoT dataset

A: Thank you for your insightful feedback. We think that the role of inner monologue can be attributed to two factors:

• The generation of an inner monologue elicits the reasoning process concerning current actions.
• The produced reasoning inner monologue provides informative context that informs subsequent planning stages.

To further validate the importance of inner monologue, we conducted ablation studies to demonstrate the performance improvements achieved by incorporating inner monologue (IM) in training:

Our findings indicate that:

• Inner monologue clearly enhances performance on planning benchmarks.
• Inner monologue can also improves low-level instruction grounding. This is because the low-level instructions in our augmented trajectory act as atomic instruction and grounding action pairs, enhancing GUI grounding ability of model.

Overall, this ablation study clearly demonstrates the performance improvements brought by incorporating inner monologue in training data, further justifying its effectiveness and supporting our analysis.

W3: Lack of examples showcasing pyautogui's advantages for adaptation to new actions.

A: PyAutoGUI has predefined GUI-related atomic operations, which can be effectively transferred to new GUI interfaces. When applied to new environments like mobile, additional actions can be seamlessly integrated into the system prompt in a plugin format, allowing the model to utilize new actions. This approach enables the agent model to combine internally learned PyAutoGUI actions with newly added ones to complete tasks efficiently. We have included the mobile environment prompt used for Android World in Appendix Section A.2 as an example of adapting to new actions.

Dear Reviewer KN7M

As the discussion phase nears its conclusion in a few hours, we would like to sincerely thank you for your thoughtful feedback and constructive engagement throughout this process.

With the final hours upon us, this is our last chance to receive and address any remaining concerns or questions you may have. If you have a moment, we kindly invite you to review our previous response and let us know if it fully addresses your feedback.

We deeply appreciate the time and effort you’ve devoted to this discussion and look forward to hearing your thoughts!

Best regards,

The Authors