OS-ATLAS: Foundation Action Model for Generalist GUI Agents

Thanks for your questions. At the very beginning, we need to admit that we have a bug in the evaluation script of the 7B model. In particular, we forgot to skip certain special tokens when the 7B model is decoding. As a result, the 7B model's performance is much lower than expected. Note that this bug only happens to 7B models in Tables 4 and 5, but it does NOT influence other experiments. Along with the release of our model weights, we will also release the eval scripts to help researchers reproduce those results. You can find the new results in the updated paper. We also make a copy here for your convenience:

Results on Web and Desktop platforms under the OOD setting:

Results on Web and Desktop platforms under the supervised finetuning setting:

Results on Mobile platform under the OOD setting:

Results on Mobile platform under the supervised finetuning setting:

Q1: I would appreciate more details about the toolkit, particularly on its usability and how other researchers could potentially customize it for their own use cases.

A1: We will provide detailed documentation alongside the release of the toolkit. In brief, our goal is to make the toolkit as user-friendly as possible, allowing users to simply click and go.

• The web toolkit is the easiest to use. Researchers only need to prepare a list of web URLs they wish to crawl and then initiate the synthesis toolkit with a single click.
• For mobile and desktop toolkits, researchers must manually install the applications from which they want to collect data. They will also need to adjust the synthesis configuration accordingly before starting the script. Once initiated, the script will automate the remaining data synthesis process.

Additionally, we will include tutorials that demonstrate how to modify the scripts to collect specific data types (e.g., for those interested in gathering more grounding data for a search bar).

---

Q2: Any discussions on choosing a pertaining + fine-tuning over mixing grounding and action data?

A2: Thank you for the insightful question. First, we must acknowledge that we chose a two-stage training approach primarily due to GPU constraints. Given the large number of SFT experiments we conducted, which involved fine-tuning the pretrained grounding model on a single agent dataset, we could not afford to repeatedly retrain the model using grounding data combined with different action data splits.

Following your suggestion, we experimented with a mixed-training strategy that combines all agentic data from six benchmarks with all grounding data. The performance of the mixed-trained model is comparable to that of the two-stage training strategy. Interestingly, we found that the mixed-trained model is more robust to changes in the prompts. Due to the short rebuttal time and GPU constraints, we are experimenting with a 1B model now. But we will scale the experiments to 4B/7B, and add a section in the appendix to discuss those findings.

Q1: The study does not touch any GUI specific challenges, such as specific visual and text distribution, cross-system differences, long action planning, and so on. It also does not present new evaluations that are more suitable for GUI. For example, setting up an emulator-style benchmark with the data synthesis toolkit. Nonetheless, those aspects are beyond the claim of the paper, and future research on related topics could benefit from the data and model released by this study.

A1: We believe we have made efforts to address some key challenges in GUI interaction: GUI element grounding, single-step GUI problem solving (i.e., action grounding), and their cross-system generalization. We have conducted evaluations on OSWorld (Table 3), an emulator-style interactive benchmark. However, we agree with the reviewer that additional GUI-specific challenges, such as long-term action planning and developing more comprehensive benchmarks, are important directions that need to be addressed in future research.

---

Q2: There could be more analyses on the data and model scaling, to help future study understand the amount of extra data and compute needed if the application needs further model enhancement or finetuning. The existing scaling analyses in Figure 3 and Sec. 4.2. are somewhat noisy and fail to give clear directions.

A2: Thanks for pointing this out. We also find that downstream task performance is not an ideal metric to measure scaling law and can be noisy. This is because downstream datasets often cannot accurately reflect the true data distribution, and evaluation metrics are too coarse-grained – for instance, correctly clicking an element does not necessarily mean the predicted coordinates exactly match the ground truth.

To study the scaling effect more rigorously, we plot the loss curve and, following [1], fit a power law-based scaling curve, as shown in Figure 7 in the updated paper's appendix. Through our scaling law analysis, we estimate that increasing training data by 8 times could lead to a 40% relative reduction in loss. Moreover, scaling data by 64 times might potentially yield a 57% relative decrease in loss.

[1] Scaling laws for neural language models.

---

Q3: To better support the claim on being effective “universally across all GUIs,” there could be more discussion on the per-task and environment performance. It remains unclear why certain tasks perform better or worse on web/desktop/mobile and different operating systems.

A3: The primary reason behind varying performance across datasets is their inherent difficulty levels. For instance, in the desktop domain, we achieve over 90% success rate (SR) on OmniAct, while for OSWorld, even human performance is below 80%. Consequently, comparing absolute performance across benchmarks is not particularly insightful. However, we find that desktop tasks are generally more complex than mobile and web domain tasks, due to the inherent complexity of desktop screenshots.

Another reason behind varying performance across datasets is that different benchmarks employ varied experimental setups. To maintain our model's universality, we intentionally avoided over-optimizing our design for specific benchmarks. Instead, we adopted a unified evaluation setting to demonstrate our model's generalizability across GUI environments. Take GUI-Odyssey as an example: our best model, OS-Atlas-7B, initially achieves a 61% SR as in Table 5, compared to their SOTA model OdysseyAgent at 65% SR. The performance difference stems from their unique approach of using previous action sequences as historical context to inform predictions. By adopting their proposed best practice, we can improve our performance to 68% SR.

In essence, achieving optimal performance on certain tasks would require case-by-case considerations regarding model, data, and computational scaling, which is not the focus of our paper. But we are more than willing to continue this discussion to share our observations and learned lessons if you are interested in certain benchmarks.

Q1: The number of instances in Table 1 is not similar for each platform, especially for desktop (54K)

A1: The considerable difference in the volume of web data compared to desktop data arises from the inherent diversity of the data itself. Statistics show that the average person uses 9 mobile apps daily and 30 apps monthly in 2024.[1] In contrast, the average internet user has been visiting around 30 unique web pages each day in 2010.[2] As a result, the diversity of desktop data is significantly lower than that of web data.

Our dataset of 54K desktop entries includes system apps and the 10 most commonly used third-party apps. In contrast, our web data comes from millions of different websites. We initially gathered 330K desktop data samples and applied strict filtering criteria to ensure a high-quality dataset. The experimental results in ScreenSpot's desktop domain and the ablation analysis, validate the effectiveness of this data. Nonetheless, we are eager to collaborate with the community to further scale both desktop and mobile data following the release of our toolkit.

[1] https://buildfire.com/app-statistics/

[2] Differential Internet Behavior’s of Students from Gender Groups. IJCA 2010

---

Q2: There are some missing details: how to ensure the correctness of the synthetic data, the specifics of action tuning and the format of the inputs and outputs, and the precise meaning of "thought."

A2: Thanks for pointing this out, we will include a concrete example and the following clarification regarding action fine-tuning:

Inputs:
<system_prompt>
<image>
Task: In the Arts & Culture app, I want to create an art gallery with the title Self Art.
History: 
Step 1: Click on the Favorites 
Step 2: Click on the Galleries section

Outputs:
thoughts:
Click on the Create Gallery button
actions:
CLICK <point>[[255,439]]</point>

During action finetuning, as shown in the example, the model will take the following input:

• A system prompt, as detailed in Table 6 in the appendix.
• The task associated screenshot.
• Task instruction and action histories, which contain thoughts or subtask instructions from previous steps.

The model is trained to predict the following two fields:

• thoughts: the reasoning of what the model should do at the current step. You can also understand it as a model generated subtask instruction.
• actions: action type and parameters (e.g., coordinates)

---

Q3: How to ensure the correctness of the synthetic data

A3: We apply both rule-based and model-based methods to synthesize data. For rule-based methods, the data is considered correct as long as the human annotations in the annotated ally tree are accurate. We only need to filter out data with rendering errors or network errors, as detailed in lines 210-215.

For model-synthesized data, validating correctness remains a challenge even the entire LLM community is struggling with. We can only assess the correctness and effectiveness of the data through the model's performance after training. During our preliminary investigation, we conducted experiments to ensure that incorporating these model-synthesized data would not degrade the model's performance. In one controlled evaluation, we found that after pre-training on rule-synthesized data, continuous training on model-synthesized data slightly improves performance.

We also randomly sample 50 data entries from each split to manually validate their correctness before proceeding with final model training.