macOSWorld: A Multilingual Interactive Benchmark for GUI Agents

Thank you for your overall favorable review and for recognizing the significance of our work. We are happy that you mentioned:

　

I am looking for new techniques that have not been reported in existing work in order to rate it higher.

Below, we briefly summarize three key innovations in macOSWorld that were not fully detailed in the paper.

　

Innovation 1: Fast macOS Snapshot Recovery

• Prior method (OSWorld on AWS):
  • Relies on full instance recreation
  • Each single recovery takes 75+ minutes for macOS
• Ours:
  • At host machine OS-level reboot, we hot-swap the root volume instead of recreating the VM
  • ~10 minutes full environment recovery

　

　

Innovation 2: Compliant & Reproducible macOS Environments

• Prior method: cannot satisfy both (legal compliance & OS reproducibility)
  • App-level recovery (AndroidWorld): legally compliant, but cannot restore changes the agent made outside specific apps
  • VM snapshots (OSWorld): OS-level recovery, but not compliant for macOS (violates macOS EULA)
• Ours:
  • Legally compliant: Leveraging AWS virtualization
  • Reproducible: Pack environment as AWS AMIs that are bit-identical across users

　

　

Innovation 3: Native, Fine-Grained Step-Wise Rewarding

• 64 out of 202 macOSWorld tasks were annotated for partial credit at each key step
• Two examples:

• Why not reported in the paper? Multiple valid completion paths (e.g., GUI vs. Terminal) make it hard to define fair, path‑agnostic scores.
• Why open-sourced? These annotations can facilitate future RL‑based reward functions or development of fairer, fine-grained evaluations.

We are grateful for your constructive and overall positive review, and we look forward to addressing the questions you raised.

　

Q1: Binary evaluation metric neglects intermediate progress of agent.

Although not written in the paper, 64 out of 202 macOSWorld tasks include 0-1 scoring based on partial completion. These are fully implemented and available in our open-source release. Following are two examples:

Why not reported in the paper? Multiple valid completion paths (e.g., GUI vs. Terminal) make it hard to define fair, path‑agnostic scores.

Why open-sourced? These annotations can facilitate future RL‑based reward functions or development of fairer, fine-grained evaluations.

　

　

Q2: Could evaluate more agent frameworks like [1, 2].

Thank you for the suggestion. We will cite AgentS and ExACT. Due to current time and resource constraints, we were unable to benchmark them during the rebuttal, but we plan to include them in future updates of macOSWorld.

　

　

Q3: Why 15 screenshots/30 dialog limit?

We match OSWorld's limits. Since agents like Claude CUA fetch a screenshot every two turns, capping at 30 dialogs yields up to 15 screenshots.

　

　

Q4: How is the action space defined in detail? Will similar operations be mapped to a single function with different parameters?

• Design: The action space design aligns with the action-space design and implementations in OpenAI CUA and Claude CUA.

• Implementation: Your understanding is mostly correct. All GUI operations wrap a small set of underlying vncdotool calls with different parameters:

　

　

Q5: Can this work be extended to support accessibility-tree included methods? If so, how to obtain the accessibility trees?

Yes, macOS accessibility trees can be obtained in two ways:

1. Using an App: Xcode > Accessibility Inspector
2. Using system command-line: AppleScript AX API
   • Example command obtaining a table cell value: osascript -e 'tell application "Numbers" to tell table 1 of sheet 1 of document 1 to get value of cell "C4"'
   • Natively supported by macOSWorld via SSH access point

We expect more methods supporting macOS-exclusive accessibility trees, and we will benchmark them in the future.

　

　

Q6: Does this work consider some infeasible tasks?

Not in the current version. All tasks in macOSWorld are designed to be unambiguously feasible, as our primary goal is to evaluate agents' core competencies on macOS. But we appreciate the suggestion and plan to include a subset of infeasible tasks in future versions.

　

　

Q7: What percentage of failures are due to step exhaustion instead of actual capability limitations?

Answering this question requires rerunning the experiment with a 50-step limit. However, we lack the resources to do so within the rebuttal period. We will conduct these extended runs and provide the detailed failure breakdown in our next revision.

Thank you for your thoughtful and positive assessment, particularly for highlighting our paper's strengths. We welcome the opportunity to clarify the points you noted.

　

Q1: How were the annotators selected?

• Computer science graduate students
• Daily macOS users & Unix shell proficiency
• Specialized roles:
  • Advanced apps: the annotator has video editing & SwiftUI development experience
  • Safety subset: created by authors

　

　

Q2: What's the task curation/filtering process? How quality, naturalness, and diversity are ensured?

Step 1: High-level Categorization

• Prompted Claude to:
  1. Brainstorm macOS use-case categorizations
  2. Summarize from a full list of OSWorld main set tasks
  3. Merge 1 & 2 to form initial categorizations
• Discuss with annotators to validate and form the 7 final categories

　

Step 2: Task Prototype Creation

• Collected 115 raw task proposals from the public using questionaire (for reference only)
• Brainstormed among authors and with other GUI agent researchers
• Created 8 example tasks; one example as follows:

　

Step 3: Annotator Training

Conducted training sessions for annotators to let them:

• Understand our purpose and motivation
• Understand the above prototype
• Learn how to write and validate a new task (using a Mac/Macbook + SSH terminal + ChatGPT)

　

Step 4: Task Creation

• Annotators were instructed to:
  • Imitate the given prototypes to write new tasks based on (but not limited to) authors' ideas
  • Write tasks in their assigned category(s) (e.g. File Management/Advanced Apps)
  • For naturalness, annotators are encouraged to follow Mac User Guide, Apple Support, Apple Discussions, and App built-in templates
  • For determinism, annotators are required to write clear, evaluation-friendly tasks
• Each annotator first create 5-10 tasks; authors validate and provide feedback
• Then, annotators finalize the creation of all tasks
• Annotators use a Mac/Macbook + SSH terminal to repeatedly validate the preparation/evaluation scripts

　

Step 5: Quality Filtering

• All tasks were validated (with some edited) by authors, ensuring fair and deterministic evaluation
• The safety subset was created and validated by the authors

　

How are properties ensured?

• Diversity: questionnaire + brainstorming + OSWorld diversity + Claude knowledge
• Naturalness: Apple official guides & templates
• Quality: author-driven full validation 　

　

Q3: How were preparation/evaluation scripts annotated?

Also written by annotators. Annotators simultaneously do:

1. Write task instructions
2. Design preparation & evaluation scripts
3. Validate on-the-fly (Mac/Macbook + SSH terminal)

　

　

Q4: What were annotators instructed to do?

Please refer to Q2 Step 4.

　

　

Q5: Were there any quality-based filtering?

Please refer to Q2 Step 5.

　

　

Q6: Human performance on the benchmark?

Test subject: computer science undergraduate student with macOS & iWork experience, but no video-editing or SwiftUI background

• Overall 82.2%
• Lowest: Advanced Apps (32.3%), as these tasks require domain knowledge (iMovie, Xcode, SwiftUI)
• The relatively high human performance supports the naturalness of our dataset, indicating that the tasks are everyday, intuitive activities that ordinary users can generally understand and complete accurately

We sincerely appreciate your encouraging evaluation of macOSWorld's key contributions. We are pleased to address the insightful questions you raised.

　

Q1: Limited evaluation granularity. Extending to track fine-grained performance (e.g., partial completion of task)?

Although not written in the paper, 64 out of 202 macOSWorld tasks include 0-1 scoring based on partial completion. These are fully implemented and available in our open-source release. Following are two examples:

Why not reported in the paper? Multiple valid completion paths (e.g., GUI vs. Terminal) make it hard to define fair, path‑agnostic scores. We leave this as an open question for future research.

Why open-sourced? These annotations can facilitate future RL‑based reward functions or development of fairer fine-grained evaluations.

　

　

Q2: The scope of the safety subset is limited: English-only safety subset is moderately small.

We extended the 29 safety subset tasks to all 5 languages.

Takeaway: High, consistent deception rates across languages show this vulnerability is language‑agnostic and requires urgent mitigation.

　

　

Q3: Plans to make the benchmark less costly? (e.g., less dependent on cloud computing)

Yes, we already developed a VMware-based version for local deployment. Both versions will be released simultaneously. Comparison:

　

　

Q4: How is task representativeness ensured?

This is ensured by where the tasks are derived from. The majority of macOSWorld tasks are derived from sources that contain representative macOS use-cases:

• Built-in app templates (e.g., Numbers "Personal Budget" document)
• Mac User Guide
• Apple Support & Apple Discussions
• Apple official tutorials (e.g., SwiftUI)

　

　

Q5: How is metric validity ensured?

The primary metric is task success rate. Its validity is guaranteed by our task curation protocol:

• When creating a task, annotators iterate live on Mac + SSH terminal, running and refining each evaluation script until only the intended interaction passes, eliminating false positives/negatives.
• Authors also reviewed every task to enforce finite, unambiguous success condition(s), ensuring binary success reflects true completion.

(Please see Reviewer UN4t for the complete task curation process)