What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities

Thank you for appreciating our paper as comprehensive and identifing highly valuable insight, your constructive comments and suggestions are valuable to us. Below is our detailed response to clarify the points you raised.

Q1: The Effect of Task Intent on Planning.

A1: The task intent is to enhance the planning capability of the planner in a plug-and-play way. We further explore the generalizability of task intents on the open-source and closed-source models. (1) For open-source models, we fine-tune OS-Atlas-Base-4B and UGround-V1-7B in the same dataset with and without task intent, respectively. As shown in the following table, incorporating task intent in the training data significantly improves the model's planning performance on OmniBench. This indicates that task intent has the potential to serve as fine-tuning data to guide models in improving their planning capabilities.

(2) For closed-source models, we use prompts with and without task intent to Qwen-VL-Max, Gemini-2.0-Flash, and Claude-3.5-Sonnet for planners, with UGround-V1-7B serving as the grounding models.

 

Q2: Compare with Human Demonstration Trajectories.

A2: Thank you for nicely pointing out that Omnibench may obtain high-quality graph-based tasks. To address this, we compare human demonstration trajectories from the GUIAct dataset and synthetic trajectories from OmniBench. We conduct experiments on the OmniAct-Web dataset and apply the same fine-tuning settings for fair comparison.

The results in above table show that graph-based trajectories in OmniBench are of high quality and lead to better performance.

 

Q3: Performance per Complexity Dimension.

A3: Thanks for your constructive suggestions. We report the model's performance across different levels within each complexity dimension in the table.

where D means Dependency, B means Branching, I means Instruction, K means Knowledge, and H means Hierarchical. As can be seen, the model's performance gradually decreases as the complexity increases, indicating that the task complexity is reasonable and independent.

 

Q4: How to design predefined resource lists for determining subtask inputs and outputs?

A4: Thanks for your interest in the details of our subtask determination. To capture the specific state of the virtual environment, we manually design cold start resources based on the principles of subtask definition and task transition to ensure accuracy and facilitate seamless integration between subtasks. To further explain the implementation, we provide additional examples of input and output resources in Figure 1 via the anonymous link: https://anonymous.4open.science/r/OmniBench_rebuttal-A4C7/r4.md

Q5: How to ensure environmental consistency?

A5: Thanks for your comment, we also believe this verification is important. Since there is an inherent trade-off between environmental realism and consistency, when introducing web-based applications~(e.g., Skype, Mail, DeepL) to the OmniBench evaluation environment, we will use firewalls to block their outbound traffic and preload the necessary page state to provide an environment that ensures realism and consistency.

Q6: Writing issues.

A6: We sincerely thank you for constructive suggestions on expression, organization, figures, and citations. We will correct them in next version.

We sincerely thank you for professional comments and high appreciation of our work! We are encouraged that our research is recognized as laying the foundation for future advancements. We will address your concerns point by point.

1.Data Quality

Thank you for your suggestion, we conduct additional experiments on OmniAct and AndroidControl benchmarks. We follow the experimental setup in the OS-Atlas paper, using our high-quality graph-based trajectories to train our models: OS-Atlas-4B and UGround-7B-V1. The results are shown in the following tables. Our models achieve superior performance on both benchmarks compared to the baselines, which showcases the effectiveness of our graph-based trajectories.

Table:Evaluation on OmniAct

Table:Evaluation on AndroidControl

 

2.The Rigor of the Conclusions

We appreciate your insightful observations. We define the impact of textual order on the model as its instruction sensitivity, conducting experiments with standard deviation as the metric. We construct 10 specially designed test tasks, each associated with three task instructions that are semantically identical (based on the same task graph) but differ in textual order. As shown in the table below, the original MLLMs tend to be less sensitive to instruction variations, but perform poorly overall. Though fine-tuning them on navigation tasks enhances the performance, it also compromises the models' robustness to instructions, with OS-Atlas-Pro and Aguvis exhibiting significantly higher sensitivity. Moreover, after incorporating graph-structured task samples from OmniBench into fine-tuning, the models' performance is further improved while largely preserving their robustness, with Omni-OS-Atlas and Omni-Aguvis exhibiting reduced sensitivity. This indicates that the trajectory data from OmniBench can help models better recognize complex dependency structures embedded in task instructions.

 

3.Experimental Setup

Thank you for your valuable suggestions. Considering MLLMs without grounding-specific training may struggle to perceive fine-grained UI elements in screenshots, we adopt the setup of A11Y+Screen for them. We also conduct the evaluation with only Screenshot setup for fair comparison. Compared to Table4 and the table below, baseline MLLMs consistently achieved poorer performance in Screenshot setup.

 

4.Minor Weaknesses

For more detailed introduction to Cross-Verification, please refer to Appendix B.2. We present additional case studies in Fig1, specify the 20 scenarios in Fig2, and more statistics on OmniBench in Fig3 of the anonymous link: https://anonymous.4open.science/r/OmniBench_rebuttal-A4C7/r3.md

 

5.Questions For Authors

To Q1: In OmniBench, test tasks collected from the virtual environment are initially selected based on a joint constraint of five complexity dimensions. We also manually filter the tasks to improve quality. To Q2: Additional examples of input and output resources are provided in Fig1 via the anonymous link. These resource types, carefully designed by humans, capture the specific states of the virtual environment. Clearly defining these resources for subtasks effectively represents and constrains state transitions in complex virtual environments, facilitating seamless integration between subtasks. To Q3: We give examples of how APIs are combined in Fig4 of the anonymous link.

 

We will integrate these experiments and correct the grammatical errors in the next version of paper.

We sincerely appreciate your constructive and insightful comments. We will explain your concerns point by point.

Q1: More graph-based complexity metrics

A1:

Thank you for the valuable questions. First, we clarify that the current complexity metrics are based on five fundamental graph-based dimensions: dependency, branching, hierarchy, knowledge, and instruction. Despite their simplicity, these node degree-based metrics are effectively aligned with the definition of subtask capabilities. Additionally, following your suggestion, we also adopt feature correlation between nodes as the complexity metrics to conduct further evaluation. As the table below shows, both metrics exhibit high Pearson Correlation (ρ) with human evaluations, demonstrating their suitability for accurately reflecting model performance. We will include all these metrics in the revision.

 

Q2: Compared to goal-based reinforcement learning

A2:

Thanks for your insightful observation. Both our intent-based prompting and goal-based reinforcement learning leverage explicit goal representations to guide agents toward long-horizon objectives, thereby improving planning quality. In our case, the intent serves a role similar to goal conditioning in RL: it narrows the agent’s decision space, aligns actions with the final objective, and encourages completion of multi-step plans. This conceptual alignment highlights the value of goal-driven formulations across both reinforcement learning and prompt-based paradigms.

 

Q3: Detailed analysis of CR and LC

A3:

Thanks for your interest in our metric details. (1) Compared to the trajectory similarity: The trajectory similarity requires strict alignment with a reference path, while graph-structured tasks may have multiple feasible execution paths due to their inherent branching and parallelism. In contrast, CR and LC are path-agnostic, making them more robust to graph-structured tasks. (2) Compared to the success rate: The success rate only provides binary judgments of task completion and fails to evaluate the intermediate completion of the subtask. In contrast, CR and LC can fully leverage the intermediate feedback provided by the graph-based evaluator, allowing for fine-grained evaluation of agent behavior. (3) Further experiments. We assess Pearson Correlation between human ratings and our metrics as well as traditional metrics on 300 sampled trajectories. The table below shows a strong alignment between our metrics and human evaluations, indicating that CR and LC more faithfully reflect human judgment.

 

Q4: Generalization on less structured, real-world virtual agent applications

A4:

Thank you for the valuable question. In fact, we have carefully designed realistic tasks in OmniBench, which enable the proposed evaluation and results to effectively generalize to real-world scenarios. To validate this, we conduct experiments on OmniAct, which collects data from real devices. Compared to Table 4 and the table below, models that performed well on OmniBench also achieve relatively better performance on OmniAct. Furthermore, the trajectories from OmniBench also effectively enhance our models' performance in real-world scenarios. This indicates that the evaluation conclusions drawn from OmniBench can be generalized to real-world virtual agent applications.

 

Q5: Robustness of intent extraction process

A5:

Considering the handling of subtasks whose intents might be overlapped or ambiguously defined, we employ an LLM to conduct post-verification processing for the intents. It consists of two steps: First, we instruct the LLM to determine whether the extracted intents are clearly and explicitly expressed. If not, we re-extract the intents. Second, we maintain a pool of existing task intents to avoid overlap. For each newly extracted intent, the LLM assesses whether it overlaps with any existing intents in the pool. If an overlap is detected, the new intent is discarded.

Thanks again for your valuable suggestions. We will integrate the above content in next version.

We greatly appreciate your insightful feedback. To better demonstrate the diversity of automatically synthesized tasks in OmniBench, we restate that our approach first explores a range of various subtasks from the explorable environment and then iteratively synthesizes subtask trajectories and evaluation. Finally, the subtasks are composed into diverse tasks bottom-up. Next, we provide more detailed examples of the synthesized tasks.

Example of the OmniBench Synthesized Tasks

First, we present 7 subtasks discovered by the MLLM during environment exploration:

And we introduce 2 tasks constructed bottom-up from the above subtasks:

We also provide examples of corresponding subtask trajectories and the evaluation function.

Example of Subtask Trajectory

{
    "trajectory_id": "XXX",
    "instruction": "Using the file explorer, navigate to C:\\Users\\user\\Desktop\\images\\ and new a Text Document named introduction.txt",
    "observations": [
        obs1.png, obs2.png, ...
    ],
    "actions": [
        {
            "function": "click_input",
            "args": {
                "button": "left",
                "double": false
            },
            "rect": [
                124,
                1020,
                179,
                1080
            ],
            "description": "There are many application icons on the taskbar, and I need to select the File Explorer to complete the task.",
            "thought": "To fulfill 'Using the file explorer, navigate to C:\\Users\\user\\Desktop\\images\\ and new a Text Document named introduction.txt', I need to first click the 'File Explorer' button to open the corresponding application.",
            "control_text": "File Explorer"
        }, ...
    ],
    "subtask_id": "XXX"
}

Example of Subtask Evaluation Function

EvalResult = namedtuple('EvalResult', ['success', 'message', 'progress'])

def evaluate_agent_task_completion(dir_path: str, file_name: str) -> EvalResult:
    # Extract the last directory name
    dir_path = dir_path.rstrip('/\\')
    folder_name = os.path.basename(dir_path)

    # Check if navigation to the specified directory was successful
    if not (check_mouse_clicks(text=folder_name) or check_keyboard_types(text=dir_path)):
        return EvalResult(False, "Subtask execution fails because agent did not navigate to the specified directory.", 0/2)

    # Check if the new text document was created
    file_path = os.path.join(dir_path, file_name)
    if not check_file_exists(file_path=file_path):
        return EvalResult(False, f"Subtask execution fails because the file was not created in the directory.", 1/2)

    # All checks passed, subtask is considered complete
    return EvalResult(True, "Subtask completed successfully", 2/2)

More representative synthesized tasks are visualized in Fig1 of the anonymous link: https://anonymous.4open.science/r/OmniBench_rebuttal-A4C7/r1.md

Thank you again for your suggestion. We will integrate these examples into the next version.