Agent S: An Open Agentic Framework that Uses Computers Like a Human

Thanks for feedback by Associate Program Chairs. The two suggestions are actually the same issue, and I will restate my concerns here.

Agent S is a system agent based on MLLM, with various additional modules such as memory, evaluator, etc. to assist MLLM in making decisions. It is not enough for the baseline to use only pure GPT4o, claude-3 and other models. The same agent system should be used for comparison to increase the persuasiveness of the experimental results.

Cradle[1] and claude 3[2] are the methods I know that can be used as baselines, so they need to be compared. Cradle is also an MLLM agent with GPT4o to control mouse and keyboard. Claude-3 is emphasized because Osworld itself uses claude-3 as a baseline. Agent s should compare the difference between agent s w/Claude -3 and pure Claude - 3, rather than comparing the difference between agent s w/Claude -3.5 and Claude - 3.

Reference: [1]Tan W, Zhang W, Xu X, et al. Cradle: Empowering Foundation Agents towards General Computer Control[C]//NeurIPS 2024 Workshop on Open-World Agents. [2]Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024. URL https://api.semanticscholar.org/CorpusID:268232499.

We thank reviewer 7E6Z for their detailed review of our work. We would like to take the opportunity to answer their questions and respond to their comments about our work.

W1: MLLMs are used with MMAgent in the Baselines

The baselines in Table 1 are not just MLLMs with tree and image inputs but rather MLLMs used within the MMAgent framework (Xie et al.), which incorporates working memory management, accessibility tree filtering, detailed prompts, parsing, and stepwise code execution. MMAgent achieved state-of-the-art results on the OSWorld benchmark at the time of our submission. We thank reviewer 7E6Z for highlighting this, and we have updated the presentation to clarify that the baselines include the MMAgent framework.

W2: Additional comparisons with existing agents

Here, we report the results of OpenInterpreter[3], FRIDAY[4], and CRADLE[1] as other agentic frameworks that are suitable for the OSWorld benchmark. Scores for OpenInterpreter and Friday (represented by *) were evaluated by a third party. Cradle and MMAgent scores are as represented in the respective papers. We have also updated these scores in Table 1 of the paper.
The CRADLE agent [1] achieved a 7.81% success rate on the OSWorld benchmark, which is much lower than our best result of 20.58%. While vision-only agents can potentially be powerful, the visual grounding abilities of current MLLM models are not sophisticated enough to handle precise grounding for UI manipulation [2, 5] The Claude-3-opus model with MMAgent [2] gets a 4.41% score in Screenshot + Accessibility Tree input and only 2.42% in Screenshot-only mode, reiterating the limitations of current MLLMs at reasonable visual grounding in UIs.

Q1: Do all GUIs have accessibility trees?

Accessibility trees are widely supported across all major OSs, including Windows (via UI Automation), macOS (through Apple's Accessibility API), Linux (using AT-SPI), as well as Android and iOS with their respective Accessibility APIs. Browsers like Chrome, Safari, Firefox, etc. all provide accessibility trees based on the Document Object Model. These accessibility frameworks are integral to modern OS design, driven by regulatory requirements such as the EAA[6], and influenced by industry standards like WCAG[7]. Frameworks like Selenium and Appium use accessibility trees for GUI testing and automation. Commercial vendors like UiPath and Automation Anywhere also use accessibility trees for Robotic Process Automation for GUIs. In general, accessibility trees are a common approach for GUI automation and are supported by all OSs.

Q1: Systems like Ubuntu and Windows have similar forms of accessibility trees

Ubuntu (AT-SPI), Windows (UIA) and MacOS (AXTrees) all represent the UI as a tree structure, starting from a root element (such as the desktop or main application window) and branching out to various UI components like windows, menus, and controls. In each framework, UI components are represented as accessible objects with properties such as role, name, and state. In our experiments with Windows and Ubuntu, we observed that the main differences came from the nomenclature used across the different frameworks which is quite easy to adapt from the respective documentation. We have added examples of Accessibility trees from Windows and Ubuntu in XML format to the Supplementary material for a better idea.

We thank you for your detailed review of our work. We appreciate your acknowledgment of our work as following a clear line of thought and having smooth transitions, as well as our experiments being conducted in well-considered environments that demonstrate the effectiveness and generalization of the framework.

We are sincerely grateful for the effort you have invested in helping us improve our work. Your feedback on our work has been invaluable, and we have carefully addressed your concerns in our response. With the discussion period deadline approaching, we would greatly appreciate any further comments you might have!

We thank reviewer KnHM for their review and insightful comments. We have updated our paper based on their points (marked in red). We address their concerns as follows:

W1: Limited Problem Definition

We have added descriptions motivating the GUI automation problem as a Partially Observable Markov Decision Process in Section 3. We have also added details about Hierarchical Planning and its general Workflow to clarify planning and execution in Section 3.. Finally we have introduced Grounding to UI Elements in Section 3.3 to give readers a better background about the problem and our method. We thank the reviewer for bringing this presentation deficiency to our notice.

W2 and Q2: Presentation Issues and Figure Organization:

While we did consider combining figures 3 and 4, we decided to separate them out as Figure 3 is a general overview of the framework, while Figure 4 describes the learning process - self-supervised exploration and continual learning. We have simplified Figure 4 to explicitly refer only to the learning and memory constructions and clearly delineated the two stages. We have also expanded Appendix A. with an illustration describing the overview of our Agent Computer Interface.

Q1: Citation Format Issues

The OSWorld paper (Xie et al., 2024) is accepted at NeurIPS 2024, resulting in the forward citation. In our bibliography, we have used the DBLP entry corresponding to this paper. The WindowsAgentArena paper (Bonatti et al., 2024) is a concurrent work available as a preprint. This work was highly suitable for testing the generalization of our framework and allowed us to demonstrate quick and easy adaptation of our method to a new setting. We have also updated the entry for this paper from DBLP in our updated manuscript.

We thank you for your insightful comments and your positive comments about our framework as having a Novel and Effective Memory Mechanism, Innovative ACI design, showing Strong Empirical Results, and providing an Insightful analysis of Agent Computer Interaction. We have taken your points in the review into consideration and updated our manuscript accordingly.

We are sincerely grateful for the effort you have invested in helping us improve our work. Your feedback on our work has been invaluable, and we have carefully addressed your concerns in our response. With the discussion period deadline approaching, we would greatly appreciate any further comments you might have!

W4 and Q1: Average time cost for Openai and Self-hosted models

We calculated the average time cost per task for running OpenAI models: GPT-4o (428.4 seconds), GPT-4o-mini (308.3 seconds), and self-hosted model: Llama-3.2-90B (366.9 seconds.) This average is calculated on a subset of 65 tasks. We would like to clarify that the performance of current GUI agents lags significantly behind human performance. In this work, we focus on the general task of improving the ability of agents to successfully complete tasks over the more engineering-oriented task of optimizing the time-cost. Consequently, Agent-S nearly doubles the performance of the baseline MMAgent. Future work could focus on optimizing the framework by reducing the number of MLLM modules, using KV cache optimization, and fine-tuning smaller MLLMs for GUI tasks.

Q2: Continued memory growth does not expand the context length

We do not keep the agent's experience in the LLM’s context window; rather, we store it separately in a database. Relevant experiences are retrieved from the memory based on a query formulated using the current observation and the task instruction. The retrieved experience does not include the full trajectory of the experience but rather a concise summary. Thus, the continued memory growth does not impact the Agent's input context length. Our Worker, which retrieves subtask level experience, has an average context size of 19184.6 tokens, and Manager, which retrieves task level experience, has an average context size of 4,719.7 tokens. This size depends only on the observations and does not increase with memory growth.

We thank the reviewer YYG1 for their valuable comments. We address their comments and concerns as follows:

W1: Fairness of comparison

While OSWorld is a benchmark paper, it also presents the MMAgent, the state-of-the-art result on this benchmark at the time of submission of this paper. We thank the reviewer for bringing up the point about the lack of other baseline evaluations in our presentation. We have added the following agent baselines to our paper as they are suitable agents for the OSWorld environment:

W2: Practicality of the self-supervised exploration process

Our Self-supervised exploration process does not need any human annotation and is performed on new, LLM-generated tasks in an offline phase prior to actual deployment on test tasks (which can be considered as the “training” phase of our in-context learning method). We do not perform any exploration on tasks from the test set. Currently, we use 427 tasks to explore; however, their number can be scaled up, and they can be extended to a large number of domains, applications, websites, and OSs to enlarge the knowledge base. This exploration allows our Agent to better understand the environment and interfaces without overfitting to specific tasks. In addition to using the experience gained from self-supervised exploration, we also utilize web knowledge for planning. If the retrieved experience is not helpful to the task, we rely on web search. We use an MLLM as a Knowledge Fusion module to judge whether the retrieved experience and search results are helpful and reliable, then combine them into integrated knowledge. Through this module, Agent S generates an end-to-end plan that effectively utilizes the most appropriate information without over-reliance on either source.

W3: The ACI proposed in this paper can only act on selected elements in the accessibility tree

While our ACI limits the agent's interaction with elements, this tradeoff between precision vs. flexibility works highly in favor of our Agent. Current MLLM models struggle with coordinate-level grounding, resulting in a very high grounding error [1]. By ensuring more precision of grounding, we can improve agents along the dimensions of planning and execution while visual grounding models catch up. Furthermore, our ACI can be easily extended to include more general instructions that can directly refer to coordinates instead of elements in the future. Our ablations study (Figure 6) also shows that ACI allows more effective learning from experience.

W4: Novelty of Agent S framework

Agent-S is a new agentic framework that integrates experience-augmented hierarchical planning, self-supervised continual memory update, and an ACI for MLLM-based GUI agents to perform complex computer tasks at a keyboard-mouse level.

1. Self-supervised exploration is an important contribution as it shows that agents can explore without ground-truth labels or training sets to better understand the environment. Since the exploration can be performed in an offline phase prior to actual deployment, it is possible to scale up the coverage of this exploration.
2. The addition of direct web knowledge is both novel and useful, as it can be easily accessed, gets frequent updates, can be directly utilized by GUI agents at runtime, and has been missing in previous GUI automation works.
3. Both self-supervised exploration and web knowledge retrieval synergize well with hierarchical planning to decompose complex tasks into executable subtasks, which is a novel finding.
4. The design of ACIs for GUI agents allows MLLM-based agents to operate computers more precisely and focus on improving planning and execution without worrying about grounding and missing feedback. We also show that an ACI is highly effective at improving the ability of GUI agents to learn from experience (Figure 6).

[1] Tianbao Xie, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. CoRR, abs/2404.07972, 2024. doi:10.48550/ARXIV.2404.07972.

We thank you for your valuable comments and appreciate the acknowledgment of our framework as showing good performance on OSWorld, demonstrating systematic evaluation, and having good presentation and visualization. We have addressed your concerns about our work in detail and provided additional results to answer your questions.

We are sincerely grateful for the effort you have invested in helping us improve our work. Your feedback on our work has been invaluable, and we have carefully addressed your concerns in our response. With the discussion period deadline approaching, we would greatly appreciate any further comments you might have!

We thank reviewer meL5 for their positive feedback and insightful comments. We want to take the opportunity to answer their questions and respond to their comments about our work.

W1: Agent performance under increased load

Agent S approaches each new task independently, and the volume of tasks does not impact the overall performance. Our agent stores experience memories externally, separate from the language model's context. These memories are retrieved as required. The only source of variation brought about by the increased volume of tasks can be retrieval costs as the KB expands. We precompute all embeddings for saved experiences in advance, so the only overhead during retrieval is calculating the cosine similarity with the current query. Our measurements of retrieval times across different sizes of the experience memory indicate that they remain constant. While our current knowledge base is relatively small, future work could utilize approximate nearest-neighbor algorithms to achieve better retrieval times with larger knowledge bases.

While complex workflows can affect the performance of our Agent, this is a very general factor, and any Agentic framework attempting complex tasks will suffer from similar issues due to the larger time and steps required for completion.

W2: Adapting to scarcity of reliable web knowledge and rapid interface changes.

Scenarios with missing web knowledge and rapidly changing interfaces are some of the key challenges for GUI agents. In situations where web knowledge is missing—such as with new applications or niche interfaces that lack comprehensive online documentation—we employ self-supervised exploration to supplement our agent's understanding. This method does not rely on ground-truth labels; instead, the agent engages in dummy tasks to learn about various workflows within these settings. Through this exploration, the agent can build knowledge about the application interfaces independently, ensuring robust performance even when up-to-date web information is unavailable. The web search and experience retrieval in Agent S distinguishes it from baseline methods, which are bounded by the pre-trained knowledge of MLLMs.

Q1: Balancing between web search and Experience Retrieval

Agent S differentiates between retrieving external web knowledge and leveraging internal memories by first formulating a focused query for each task using a Multimodal Large Language Model (MLLM). This query is used to retrieve relevant information from both the web and the agent's internal knowledge base, which contains narrative memories of prior experiences. A Knowledge Fusion Module (MLLM) takes both of these along with the Task instruction as context to generate relevant integrated knowledge. By dynamically balancing external and internal knowledge using an MLLM, Agent S generates an end-to-end plan that effectively utilizes the most appropriate information without over-reliance on either source.

Q2: Cold-start problem and addressing scenarios where experience is not available

Agent S addresses the cold start problem by performing self-supervised exploration to populate its Knowledge Base (KB) with information from dummy tasks and tasks derived from existing OSWorld tasks, relying on web knowledge and pre-trained LLM knowledge. Moreover, when encountering a new task with no prior experience, it benefits from multi-level retrieval, as new tasks often include familiar subtasks like opening file menus or closing pop-ups. By performing independent subtask-level retrieval, Agent S can leverage saved experiences from unrelated tasks.

In the most challenging scenarios where neither the task nor its subtasks are related to anything in the KB, the agent relies on the pre-trained knowledge and common-sense reasoning abilities of Multimodal Large Language Models (MLLMs), as well as real-time information from web searches. Additionally, Agent S employs a continual learning mechanism that updates its Knowledge Base upon encountering novel tasks, enabling it to adapt more effectively to new tasks and applications in the future.