Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents

We thank the reviewer for their detailed feedback and for recognizing the Agent S2 architecture, SOTA performance, and soundness of our ablations and analyses. We appreciate the opportunity to address the concerns raised.

Agent S2 generalizes beyond a particular benchmark.

We would like to clarify that our contributions are general, and we do not spend engineering efforts tailored specifically to OSWorld for the following reasons:

Demonstrated Generalization.

Beyond OSWorld (Ubuntu), Agent S2's architecture demonstrates SOTA performance on WindowsAgentArena (Windows OS) and AndroidWorld (smartphone). This cross-platform success demonstrates that our framework is general and is not overfit or engineered to a particular environment.

Environment-Agnostic Innovations.

• Proactive Hierarchical Planning (PHP) operates in a semantic planning space. All of the Manager’s planning is based on the task description and current state of the desktop, without any human intervention.
• Mixture of Grounding (MoG) uses experts that are broadly applicable across any operating system. For instance, the Textual Grounding expert employs Optical Character Recognition (OCR), a general technique for extracting text from any screen. Similarly, the Structural Grounding expert provides a programmatic way to interact with tabular data, a common element across various apps and environments. Popular real world computer use tasks that extend beyond OSWorld, such as software engineering and online research, require interaction with textual documents and tabular data, and MoG is consistently applicable in all of these scenarios.

Regarding Novelty of Proactive Hierarchical Planning.

We emphasize our final revision will highlight and further compare with OS-Copilot (Wu et al., 2024), OS-Genesis (Sun et al., 2024), AgentStore (Jia et al., 2024), and Spider2-V (Cao et al., 2024) in more detail. However, our Proactive Hierarchical Planning (PHP) offers the following distinct characteristics:

• In Agent S2, the Manager refines its list of subgoals after the termination of each individual subgoal, allowing continuous adaptation at the Manager level. From our understanding of the OS-Copilot (Wu et al., 2024) codebase, their Planner reactively replans depending on judgement/repair status. Proactive Hierarchical Planning activates the Manager to continually replan, avoiding scenarios where future subtasks become outdated.
• The hierarchical framework effectively distributes cognitive load across different models. While learning multi-step planning with a single model is promising, as in OS-Genesis (Sun et al., 2024), our ablations demonstrate that no individual model is perfect, and leveraging the strengths of different models for different roles obtains better performance.

Regarding Fairness of Comparison of MoG.

Overcoming the bottlenecks of pixel-level grounding is precisely a strength of our framework. It is important to note that the baselines we compare against are also frameworks and not just base models. We show that a monolithic framework like Claude Computer Use that uses the same model for both planning and grounding performs worse than Agent S2. Thus, using separate MoG experts is a deliberate choice to address grounding bottlenecks, while also preserving the reasoning ability of generalists.

MoG is also comparable to previous works such as Omniparser which parses an image to extract interactable elements, Agent-S which uses OCR to augment its accessibility tree input, as well as standard Set-of-Mark approaches. We emphasize that Agent S2 does not use any privileged information from the environment. In fact, while a range of previous works use additional information like Accessibility Trees and HTML content, our approach relies purely on screenshot input.

Response for Questions to Authors

MoG Routing Strategy: The selection of the grounding expert is dynamically decided by the Worker module at each execution step. This is not enforced through rules. The key difference in AgentStore (Jia et al., 2024) is that their MetaAgent preselects application-specific agent(s) to perform a full task/subtask based on special tokens. In contrast, Agent S2 proposes a small set of very general classes of grounding experts: Visual, Textual, and Structural, that are always available and dynamically selected at the step level.

We reference Figure 4 as an example. For the subgoal: “select last paragraph”, the Worker initially tries the visual grounding expert by calling the agent.click() action. Immediately after, it decides to use the textual grounding expert by calling the agent.select_span() action after observing that the initial click did not highlight the paragraph properly. We do not enforce any heuristics or restrictions, such as assigning an expert per subtask. Instead, the ability of Agent S2 to reason and reflect on the desktop state is fully responsible for which grounding expert is used at each step.

We sincerely thank the reviewer for their positive assessment, strong support for our paper, and insightful questions regarding the practical deployment of Agent S2. We are encouraged by the reviewer's recognition of our state-of-the-art results and the transparency of our analyses.

We will address the questions concerning the deployment and modularity of Agent S2:

Regarding the handling of specialist modules in diverse environments and their potential unavailability.

The Worker module dynamically selects the most appropriate grounding expert based on its reasoning. This implies that if a particular type of interaction is not available or the agent encounters a failure, the Worker will naturally fall back to alternative actions. This dynamic selection allows flexible extension and replacement of various experts without breaking the system.

Additionally, the grounding experts used in Agent S2 are broadly generalizable and applicable across various different applications and operating systems.

• The Visual Grounding expert relies on screenshot input and language descriptions to locate elements, a general approach applicable to any GUI. This can be implemented with smaller models (UI-TARS-7B, UGround-V1-7B), which are cheaper to deploy.
• The Textual Grounding expert, which utilizes Optical Character Recognition (OCR), can identify and locate text on any screen, irrespective of the underlying OS or application. Various open source softwares and libraries like pytesseract, paddle-ocr, as well as services from providers like Google, Amazon, Mistral etc. can be used for the textual grounding expert.
• The Structural Grounding expert can work with any sheet-based application. While we use unotools to interface with Libreoffice calc, other sheet–based applications provide alternatives like Openpyxl for Excel, Google Apps Script for Google Sheets, AppleScript for Numbers etc.

Regarding the deployment and maintenance of Agent S2 in real-world settings, particularly integration, robustness and latency.

• The modular design of separating responsibilities across the Manager, Worker, and distinct Grounding experts, enhances the system's architectural robustness. This means replacing or ablating individual modules can be done without system failures.
• The modular design also helps reduce work for future development, as users can easily plug-and-play newer models, allowing the agent to evolve with minimal changes to the framework and codebase.
• Furthermore, both generalist planning models (e.g., Claude, GPT-4o) and specialist grounding models (e.g., UI-TARS) are currently supported via serverless APIs, which alleviates the difficulties of local or user-managed deployments.
• We acknowledge that latency is a critical consideration for real-time interaction in real-world deployments. The current work has focused on establishing the efficacy of the Agent S2 framework and achieving state-of-the-art performance on complex, long-horizon tasks. While our work does not focus on latency, users of Agent S2 can easily swap in smaller proprietary or open-source models to decrease latency.

Thank you for your insightful review and for recognizing the potential of Agent S2. We appreciate your positive comments on the framework's demonstrated improvements, the quality of our experiments and ablation studies, and its generalization potential. We welcome this opportunity to clarify the key innovations of our work and provide further details on our ablation studies. As per the reviewer’s request we also report additional experimental performance with an open source planner model.

Regarding key innovation.

• Agent S2's primary innovation is the empirical demonstration that a compositional framework that combines Generalist models for planning/reasoning with Specialist models for grounding can outperform stronger monolithic models.
• We present a concrete description of the hierarchical architecture used in Agent S2 (in Section 3), consisting of a Manager module ($M$), Worker module ($W$), and Grounding experts ($G_i$).
• We also introduce two key strategies, the Proactive Hierarchical Planning (Section 3.2) and Mixture of Grounding (Section 3.1) and quantify the benefits of these strategies through ablation studies in Figure 5.
• Our results show that Agent S2 achieves State-of-the-art results across two computer use benchmarks and 1 smart phone use benchmark
• We will open-source the code of our framework after the revision for complete reproducibility..

Regarding ablation on base models.

We would like to clarify that our paper ablates on different foundation models as the common backbone for Manager and Worker. The results in Table 2 show the categorized performance of using different generalist models as the Manager and Worker, demonstrating that Agent S2 amplifies the performance of all underlying models.

Furthermore, as per your suggestion, we run a full 15 step evaluation on OSWorld using the open-source Qwen2.5-VL-72B-Instruct as the backbone planning model. Agent S2 with Qwen2.5-VL-72B-Instruct obtained a success rate of 18.29%, more than doubling the performance over the reported baseline Qwen2.5 model score of 8.83%. Our framework still demonstrates large improvements, even when plugging in open-source VLMs.

We would also like to point out that our paper already ablates on different open-source models as the visual grounding expert. Our ablation study on the Visual Grounding expert (Figure 6) illustrates how smaller, open-source specialist models like UI-TARS-7B-DPO (29.23% success rate) and UGround-V1-7B (24.61% success rate) can match and even outperform a large generalist model like Claude-3.7-Sonnet (24.61% success rate) when employed as a dedicated visual grounding expert within our modular Agent S2 framework.

Our focus in this study was discovering the framework design methodology that works best, and comparing against the best available baselines to demonstrate that well motivated, open-source frameworks can outperform closed models and frameworks. This is why we chose to use strong generalist models in our study. Exploring the performance trade-offs with a wider range of open-source models represents an important avenue for future work and our open-source framework allows future researchers to easily compare and conduct these experiments.

Thank you for your detailed review of our work, which has prompted us to clarify several aspects of our paper and highlight the contributions more effectively. We will address the concerns below:

Source of improvement between framework and base models

We wish to address the misunderstanding about stronger base models, and clarify that the evidence strongly points to the Agent S2 framework improving the task success performance. Below we present a table showing the success rates of various frameworks using screenshot input, grouped by the underlying base model on OSWorld.

• The table shows that Agent S2, with the same base model outperforms the previous SOTA frameworks.
• Agent S2 w. Claude-3.5-sonnet (new) outperforms Claude Computer Use framework with the better Claude-3.7-sonnet. This further validates that Agent S2 improves performance by amplifying the abilities of the base model.

We used Table 1 to compare with SOTA baselines, irrespective of input modalities and special fine-tuning. However, we will include the above table in our revision to directly compare frameworks grouped by the underlying base model.

Generality of action space and framework contributions

• We point out that action spaces are an integral part of a framework. All baselines used in this work have their own action space. The purpose of the framework and its action space is to elicit the abilities of the base model for the provided task.
• We also clarify that our action space is not tailored specifically for the OSWorld benchmark. Our action space generalizes to various applications (Web navigation, desktop software, text editors, etc.) and operating systems (Ubuntu, Windows, and MacOS).
• The actions set_cell_values and highlight_text_span, are interfaces to our MoG method. Their functionality is enabled by the grounding experts.
• The same action space is used in OSWorld and WindowsAgentArena underscoring the general effectiveness of our framework and action space. For AndroidWorld, we adapt actions like swipe, back, home, and long_press which also exist in the baselines that we compared against.

Fine-grained baseline comparisons

Amongst the state-of-the-art baselines we compare with, only Aria-UI and Agent S report their categorized success rate, specifically on the 15-step benchmark. We will include this table in the revised version.

Baseline comparisons in other benchmarks

For WindowsAgentArena (WAA) and AndroidWorld (AWD), we already report comparisons with the previous SOTA frameworks. The cited baselines on WAA are the only reported scores at the time of the submission, to the best of our knowledge. For AWD, we already picked the three top-performing baselines for comparison, but we will include following additions in our revision.

I thank the authors for their efforts in addressing my concerns. I would like to follow up on a few remaining points below:

Regarding “Generality of Action Space and Framework Contributions”

I kindly disagree. Unless the manipulation of the action space is a central contribution of the paper, it should remain consistent across baselines, or at the very least, unaltered from the target environment, to ensure fair comparisons. Otherwise, it becomes too easy for the agent designer to introduce new actions that simplify the environment, potentially provide an unfair advantage and trivially improve performance. I strongly encourage the authors to include a more detailed ablation study on the added actions to clarify their impact.

Regarding “Baseline Comparisons in Other Benchmarks”

Thank you for the additional details. However, as previously noted, comparing agent frameworks without fully disclosing the underlying LLMs is strongly discouraged. For example, it is unsurprising that Agent S2 paired with Claude-3.7-Sonnet outperforms another framework using a weaker LLM such as LLaMA-2-7B. Without controlling for backbone differences, such comparisons are difficult to interpret.

Regarding “Novelty”

I remain unconvinced by the claimed novelty of the proposed framework. The "Proactive Hierarchical Planning" mechanism appears to be standard replanning upon receiving new observations, while the "Mixture of Grounding" seems to be a rebranding of the existing tool-use paradigm.