We thank the reviewer for the valuable feedback. We are encouraged that the reviewer finds our benchmark to complement existing works, and provides a useful parallelization infrastructure. We address each question/concern below:

Additional benchmark improvements

• We agree that benchmark additions like multilingual tasks, improved metrics (e.g. cost, steps, time to completion) can improve agent evaluation. We note that our current tasks and metrics provide ample coverage across the most utilized Windows functions in terms of representative user workflows. Beyond our current analyses of human evaluation (Appendix A.5) like number of steps, difficulty ratings, and agent/task job runtimes (Appendix A.7, Table 11), we are happy to include additional details.
• We agree; the points raised by the reviewer are improvements that we plan to make over time on the open-sourced codebase nonetheless; this includes improving benchmark tasks in quantity and quality (with a focus on multi-application tasks), adding new applications/domains, new modalities to the benchmark (e.g., audio-capable agents), and adding Chinese, Japanese, and Spanish versions of tasks to our benchmark.
  • Given that our goal is to first make the benchmark scalable/parallelizable to enable faster evaluation/reiteration, our planned improvements can be easily added atop our framework, along with other features, as we co-develop our benchmark with the research community on Github—especially as model/agentic capabilities evolve.

Comparison with other agents

• Our main goals with Windows Agent Arena are to provide a new benchmark for Windows OS and offer an accessible, preliminary agent (Navi) which researchers can adapt for their own needs. Navi is based on set-of-mark prompting and VLM reasoning, which is a common agent architecture.
• There is a rich body of literature with multiple agent types, but implementing all these variants was out of scope given our main focus is on the construction of a scalable, parallelizable benchmark for the Windows OS environment. However, we still make a best effort to ablatively study our agent's capabilities as we compare 32 different Navi variants (Table 4) using multiple screen parsing models and VLMs (from small to large models) to help users understand the pros/cons of each configuration.
  • Our intention for Navi is to illustrate the use of our benchmark and serve as an accessible but relatively capable starting point/template for the community. Together with our benchmark, our hope is that our contribution will enable faster reiteration in terms of research and development of AI agents.

[Q1] What are our contributions / How is this different from OSWorld?

• Please see [N2] in "General Response to Common Questions from Reviewers" at the top.

[Q2] Failure analysis

• Section 4.2 details common failure causes for Navi: i) visual-language misalignment (selecting the wrong element ID despite a reasonable textual explanation), and ii) incorrect screen parsing (e.g. grouping multiple elements under the same ID).
• Figure 6 provides visual examples of these failures. Appendix B (Figure 16) provides another example. We are happy to include more examples in the appendix in the revised manuscript. We hope these examples help researchers when developing their own agents.

[Q3] Comparison with OSWorld’s agent

• OSWorld's agent uses as input the accessibility (a11y) tree along with the raw screenshot. Navi has a similar implementation, but instead uses the Windows UIA tree and other models (e.g., Omniparser) to create SoMs on the screenshot to aid element selection.
• Moreover, several of our agent tests/choices already share substantial overlap with OSWorld’s including GPT-4V (which delivered the best on-average performance in OSWorld’s experiments and received the most analysis).
• We also include new models released since OSWorld (e.g., OpenAI’s o1 for planning/reasoning or Omniparser for screen parsing) as well as other types of models not covered by OSWorld or other benchmarks (e.g., small language models like phi3).
• Overall, we study and analyze 32 variations of Navi (using different screen parsing backbones and VLMs for reasoning/planning).

[Q4] Human performance

• Appendix A.5 provides details on human evaluation/performance.
• Participants operated without search tools or the internet; failures were mostly due to not knowing the lengthy steps needed to change specific settings in media players and office apps. In several cases, participants were not as familiar with the program/application (e.g., more familiar with another media player). In others, the participants’ typical workflows were more casual which did not require advanced use cases in their day-to-day tasks compared to what the task required. Our tasks intentionally reflect a range of difficulty levels.

Please let us know if there are any other questions and we would be happy to address them!

We thank the reviewer for the response and we apologize if we missed parts of your inquiry or if our earlier response was unclear. We address your remaining concerns below.

W3:

• Re: task curation details, we refer the reviewer to the section on “Task Curation for Windows OS“ or Section 3.2 in our paper.
  • To summarize: changes include modifying file paths, converting Linux commands into Windows PowerShell, establishing a reverse proxy to communicate with Edge/Chrome via ports on the host machine, revisions to task instructions to match OS/program specifics, etc.
  • We are happy to expound upon this but this process to modify/adapt tasks is already in the current paper.
• Re: difficulty ratings, see Section A.5 of our paper which also contains some detailed analysis.
  • To summarize: participants rate tasks subjectively. We found that time-based and step-based metrics were susceptible to noise outside of their control (e.g., connection quality, OS updates) and exhibited high variance and disagreement rates, making them uninformative.

Q1:

• Regarding our contributions, we refer to general comment note [N2] above but highlight/reiterate our key ones below.

Benchmark Content + Tasks

• Only 10% of OSWorld's tasks focus on Windows---in contrast, our benchmark focuses exclusively 100% on Windows OS.
• This means that, in absolute terms, our benchmark has more than 3 times the number of Windows tasks (>150) than OSWorld (only ~50)---as a result, we have much more in-depth analysis and more task variety/diversity re: agent performance on Windows.
  • This is important given Windows is a very widely-used and prevalent OS but less well-represented in agent benchmark environments to see how agents perform with Windows' unique environment. Even OSWorld's own analysis notes that performance varies significantly between Linux and Windows OS.

Benchmark Scalability + Accessibility

• Our efficient, scalable benchmark infrastructure is cheap and does not require large amounts of storage or GPU resources, etc -- but still allows for a full evaluation of our benchmark in under an hour with what would normally otherwise take multiple hours or over a day despite the benchmark's size or # tasks, making agent research and development faster and more accessible. See general comment [N2] above for details.
• Agentic benchmarks are typically large/complex due to their need to test various aspects of agentic performance, slowing down evaluation, reiteration, and research. Our benchmark does not suffer from that bottleneck.

Q3:

• We emphasize that our intended key contribution is our benchmark: our primary purpose of providing an agent is to simply demonstrate our benchmark and provide the community with a starting template for experimentation.
• We also point out that comparisons with OSWorld's agent are not practical and, in several cases, not possible. Just a few examples:
  • Some closed-source models used as agents (e.g., OpenAI) by OSWorld have undergone several changes/updates, resulting in changed performance and depreciated models. This makes some comparisons impossible (e.g., the version of gpt4-vision-0125 used by OSWorld is depreciated and no longer available).
  • Due to differences in the programs/domains we create tasks for in Windows (detailed in our paper) versus those in OSWorld, how success rates are averaged/grouped together by domain will also be different, making direct comparisons very difficult and not intuitive.
    • Also, running our agent on OSWorld also represents a significant experiment/request in light of the amount of time permitted for the discussion period due to OSWorld's sheer size -- which is also exactly why we believe our benchmark provides a novel contribution in its speedy evaluation and scalability.
  • OS-specific differences: a11y trees (Linux) and UIA trees (native to Windows) differ in implementation/usage across Windows and Linux due to platform-specific frameworks and conventions. Since these are used as inputs to agents but differ depending on the OS, there is no direct comparability. In fact, this is why we believe it is important to have a benchmark that focuses on Windows' features.
  • Nonetheless, a subset of our results already possess some comparability with OSWorld’s agent: e.g., the overall performance in our results in Table 4, Section 4.2 using Omniparser (Set-of-Marks) and UIA trees are comparable/analogous to their counterparts in OSWorld using Set-of-Marks and a11y trees.
• We will make revisions in our manuscript to make these points clearer.

In light of these clarifications, could the reviewer clarify which specific concerns among W3, Q1, and Q3 still remain?

We'd be happy to provide answers, more context, revisions, etc.

We thank the reviewer for the valuable feedback. We are encouraged that the reviewer finds that our benchmark is complementary to existing works in the field, and adds a more efficient evaluation. Below, we address each concern in turn:

Navi’s novelty and Navi’s role in our paper

• To clarify, we do not claim in this work that Navi is an entirely new type of agent. Yes, as noted by the reviewer, our goal/aim for Navi (and its variants) was to have it serve as a reasonable, capable, but preliminary starting point and baseline for users, researchers, and the open-source community at large to develop their own agents using our new benchmark.

• Together with our novel contribution in making this benchmark scalable and parallelizable, we aim to provide an accessible way for the community to train/test agents and reiterate efficiently.

• Navi follows similar approaches to other web / OS agents which use screen parsing + VLM reasoning. In Table 4 and in the results section of our paper, we provide a thorough analysis of 32 different combinations of Navi agents with distinct backbone models, which can help guide researchers in this area to develop and customize their own solutions.

• We have also open-sourced these backbones to allow researchers to create their own agents which might contain better solutions and move the benchmark forward. In addition, we benchmark Navi on a more established, “traditional” agent benchmark (Mind2web) to verify its performance.

• Currently, we do not plan on testing Navi on other benchmarks for this paper since our main goal for this paper is not to make Navi competitive. Rather, our paper’s main contribution and novelty are developing a Windows OS benchmark that is scalable through its parallelization infrastructure (for fast evaluation) and realistic for training/testing agents in a Windows environment.

• To demonstrate the promise of agent development and help showcase the benchmark, Navi serves as both a beginning baseline and openly available template for users to play with and develop in this respect.

We hope our responses have addressed your questions. Please let us know if there are any other questions or suggestions and we would be happy to address them.

We thank the reviewer for the valuable feedback. We are encouraged that the reviewer finds that our benchmark fills a gap in current agent evaluations. Below we attempt to address each of your concerns within the space limit.

Accessibility and proprietary OS

• We are aware of this limitation which is why we have made significant effort so that anyone can access/use Windows OS in our benchmark; despite not being an open-source OS, our code repository enables users on to access a free evaluation copy which can be used for benchmark deployment with easy continued renewal after.
• To the best of our knowledge, our benchmark is one of the few that provides users open/free access to the Windows OS and computing environment for research/development. Please see [N1] in "General Response to Common Questions from Reviewers" Official Comment at the top.

Resource-intensiveness

• See (2) of [N2] under "General Response to Common Questions from Reviewers" Official Comment.
• In short, our benchmark does not need any high-performance computing setups and large storage capacities.
• After each run, the internal state of the system/VM is reset to ensure that progress/completion on one task does not interfere with others. This preserves the integrity of the OS environment so tasks can be continuously run and parallelized without interruption or issue. This also removes all created files etc. resulting from agent action, further mitigating the need for a large storage capacity.

Configuration/maintenance

• Our benchmark and the underlying infrastructure make it easy to install additional software either on the VM directly or via the Docker image. This way, users can design new tasks that require new programs and add them to the benchmark as necessary.
• Together with the ease in which a user can design their own tasks through a task configuration file/JSON (e.g., see README and Appendix A.6), our benchmark is easily configurable, flexible, and extensible to new tasks, programs, etc., with minimal risk of breaking the benchmark's functionality or integrity.
  • In defining their own custom tasks, users can leverage our existing library of configuration or evaluation functions or write their own by modeling theirs after ours. We are happy to include even more detailed instructions on task creation, hosting, and configuration beyond what we already have.
• Unfortunately, many agentic benchmarks with computer/OS environments (e.g., OSWorld, AndroidWorld, etc.) share similar weaknesses when it comes to maintenance: as OS systems undergo refreshes, updates, etc., it can be difficult to ensure continued compatibility with the benchmark.
  • To mitigate this, we have frozen automatic update and instead conduct manual review/updates (e.g., monthly) to re-adjust and mitigate things breaking. Since we are adding new features, modalities, and tasks, we will be continuously maintaining the benchmark in terms of both our needs and those of the wider community (e.g., Github).

Ease of benchmark infrastructure

• A major contribution of our benchmark is precisely making it easy for researchers to scale up agent experiments. As such, we have made significant effort to automatically build/install all necessary programs, scripts, images, etc. to simplify evals. We also include resources to help add/configure new tasks.
• Cost-wise, a full parallel run costs in the range of 10 USD in total with CPU machines in Azure. Model calls cost about $100-200 depending on tokens utilized and model choice. We will include these details in our revised manuscript alongside data we already report (Table 10 in Appendix A.7).

Windows-specificity / Why only Windows?

• Please see [N1] in the comments we made for all reviewers.

What are our contributions / How is this different from OSWorld?

• Please see [N2].

Tasks

• Please see [N1].

Navi’s novelty

• To clarify, we do not claim that Navi is an entirely new type of agent.
• We aim for Navi to serve as a reasonable starting point and baseline for researchers developing their own agents.
  • As pointed out, it follows similar approaches as other web / OS agents which use screen parsing + VLM reasoning.
• What we provide in Table 4 and in the results section is a thorough analysis of 32 different combinations of Navi agents with distinct backbone models to help guide researchers in this area.
  • Out of those models we also open-source a few of these backbones to allow researchers to create their own agents which might contain better solutions and move the benchmark forward. We also benchmark Navi on a more well-established “traditional” agent benchmark (Mind2web) to verify its performance.

Reward structure

• We believe that the completion-based reward function is a reasonable assumption for the benchmark. However, we do plan to add number of steps, time, and cost as auxiliary metrics to the benchmark in our open-sourced codebase.

We thank the reviewer for the valuable feedback and are encouraged that the reviewer finds our benchmark to be well-designed and diverse. We address each concern below in turn:

Why only Windows?

• Please refer to note [N1] in the "General Response to Common Questions from Reviewers" Official Comment at the top.

What are our contributions / How is this different from OSWorld?

• Please refer to note [N2] in the "General Response to Common Questions from Reviewers".

Extension to other OS

• Generally speaking, there are two aspects/issues to consider if a researcher wants to extend our benchmark to another OS:
  • Task configurations/evaluation scripts, and benchmark parallelization infrastructure. While OSWorld already, and primarily, covers tasks with Linux OS in mind, it is not yet equipped with fast and scalable parallel evaluation in the cloud. Our approach/environment not only provides benefits when it comes to scalability and faster/parallelizable evaluation but also a more secure environment as well.
  • Furthermore, OSWorld, to the best of our knowledge, does not natively support a full version of Windows OS to the extent that we do. A developer would be able to borrow infrastructure from Windows Agent Arena, but that project is outside of the scope of our paper. Porting the benchmark to MacOS would require both very heavy engineering work for porting tasks and infrastructure due to the structural differences between OS architectures.

Uniqueness of tasks and capability assessment

• We refer to [N1] for more detail. In short, because Windows OS has distinct UI, system-specific functions and command line, system-specific apps, idiosyncrasies, etc. than Linux OS, we expect agentic behavior, performance, and perception to vary considerably---as a result, the same task can have vastly different completion trajectories depending on OS.
  • As just one example, UIA trees that are used as input (in addition to screenshots and SoMs) to help multi-modal agents understand/access UI elements on the screen exist on Windows; however, different structures exist on Linux, like AP-STI or ATK among others, which naturally will impact agent performance in how well it traverses these structures to perceive different UI/visual elements, etc. These will ultimately impact how the agent plans, reasons, and acts across different OS. We will clarify these aspects in the revised manuscript.
• Only 10% of OSWorld's tasks are for Windows (43 out of 412) whereas the entirety of our tasks (150+) focus on Windows. Moreover, we include a considerable portion of newly designed tasks in addition to tasks we converted or transformed to suit Windows function and usage. We will add more discussion re: these differences in our manuscript.
• Rather than focusing on Navi's evaluation across different benchmarks, our main goal is to provide a new benchmark for Windows OS, and offer an accessible, well-performing but preliminary agent (Navi) which researchers can adapt for their own needs. Navi is meant to be a capable starting point and template for the research/open-source community to develop and customize their own agents; together with our benchmark, our hope is that our contribution will enable faster reiteration in the research and development of AI agents.

Benchmark cost

• Our open-sourced repository README file provides a cost estimate for running the benchmark. This estimate includes both the VM cost (about 8 US dollars for all machines running in parallel for the duration of the evaluation), plus the models' costs (which can vary from 0 US dollars if the user runs a local VLM model, all the way to about 15 US dollars for GPT4o-mini or about 100 US dollars for GPT4o).
• Appendix A.7 in our paper also includes a table of VM specs/costs and times. For times, we note that typical serial evaluation of a benchmark on the scale of Windows Agent Arena (or larger) would take significantly longer (likely multiple times longer) than our parallel setup.
• We note that these costs are similar to costs associated with evaluating state-of-the-art closed source models on other agentic benchmarks as well. We also note that as part of benchmark’s infrastructure, we do not need expensive high-performance compute as our parallelization can be done on inexpensive CPU cores (see [N2] in the "General Response to Common Questions from Reviewers") so any additional costs from running our benchmark beyond closed-model APIs are near-negligible.

Mistake in line 263

• Thank you for highlighting this mistake. We meant to point the reader to appendix A.6 (TASK DEFINITION & CONFIGURATION). We will change this in our revised manuscript.

We hope our responses have addressed the questions that reviewers have. Please let us know if there are any other questions or suggestions.

We thank the reviewer for the valuable feedback. We are encouraged that the reviewer finds our work easy to deploy. We address each of your questions/concerns below:

What are our contributions / How is this different from OSWorld?

• Please refer to [N2] in the "General Response to Common Questions from Reviewers" Official Comment at the top.

Uniqueness of Windows tasks

• Even for web browsing tasks (which are not inherently OS-related), a significant amount of time in terms of engineering and testing was required to modifying the configuration and evaluation functions so that they would work natively on the Windows OS given differences across OS kernels/support.
  • As just one example for web browsing tasks using Chrome and Edge, functions that access browser cookies, and settings in cached files are usually significantly different between Linux and Windows, and each task had to be modified and verified manually. We also added Edge browser support for several Web tasks, which did not exist in OS World.
• Additionally approximately double the number of hours was spent creating new tasks for Windows-exclusive apps (Microsoft Paint, File Explorer, Settings, Notepad, Clock). These newly designed tasks are not only meant to work under Windows OS but also aligned with the actions and parts of workflows of common users of Windows---we will better clarify and describe the differences between adapted tasks, converted tasks, and newly designed tasks in Section 3 of our revised paper.

Number of tasks

• With 154 task templates, our benchmark falls in line with major benchmarks in the domain of execution-based evaluation such as MiniWoB++ (114 templates), AndroidWorld (116), Web Arena (241), and OSWorld (369). We are also actively monitoring the open-sourced repository and adding new tasks that we create/source ourselves as well as those made by the research community to expand the benchmark, including users who have already designed their own tasks within our benchmark.

Agent design back-end

• In Table 4 and Section 4, we describe the 32 different agent back-ends we tested along with different combinations of screen parsing and VLM reasoning models. These models range in size and other aspects from small local and cloud models (e.g. Phi3.5 and GPT4o-mini) to large cloud models (GPT4o and o1). We will better clarify and describe the design of our agent Navi in Section 4 of our revised manuscript along with the individual components and their specific considerations.

Please let us know if there are any other questions or suggestions and we'd be happy to answer!