UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction

R1: Failures cases

Claude 3.7 Sonnet was released on Feb 24, 24 days after the deadline, and Qwen 2.5-VL on Jan 28, just two days prior—making it infeasible to include them in the review version. However, we have now evaluated both models on our grounding benchmarks, and results are available at <link>.

We summarize key error patterns below and confirm that they hold for the above models and refer the reviewer to our responses to Reviewer 3NZB (R3) and Reviewer 7jPV (R1) for more details and examples:

Element Grounding: Models struggle with visually similar elements, platform-specific icons, and small elements in dense UIs. GUI agents perform better on functional tasks due to training alignment, while VLMs underperform, revealing task-specific gaps.

Layout Grounding: Frequent issues include overly large or loosely defined bounding boxes and failure to group related UI elements correctly.

Action Prediction: Models often fail to ground actions accurately, hallucinate elements/actions, and perform poorly on dense, complex interfaces.

R2: Concern about generalizing findings from open-source to proprietary desktop environments

We appreciate the reviewer’s concern. However, many open-source apps closely emulate proprietary counterparts, capturing similar complexity. For instance, LibreOffice mirrors Microsoft Office features and serves over 200 million users[1]. Given this overlap in functionality and interface design, we believe our benchmark remains meaningful and representative of real-world GUI scenarios.

[1]Wikipedia page for LibreOffice. https://en.wikipedia.org/wiki/LibreOffice

R3: Limited real-world applicability due to the exclusion of closed-source applications

We agree that automation is widely used in proprietary desktop environments; however, including such applications poses legal and licensing challenges, as many restrict redistribution of software interfaces, recordings, or interaction data, making it difficult to build a publicly shareable dataset around them. To ensure accessibility and reproducibility, we focus on open-source software.

More importantly, our benchmark includes widely-used open-source applications like LibreOffice, VSCode, GIMP, Firefox, Brave, and Shotcut, which serve as strong counterparts to commercial software. These applications support complex, real-world workflows, share high functional similarity with proprietary software and are backed by active communities. We believe this focus ensures UI-Vision remains both practical and widely applicable.

R4: Planner + VLM grounding analysis

We clarify that the planner selects the action and target element, while the grounding model only provides coordinates. Thus, the improvement is primarily due to better action selection. To analyze this further, we compared the decrease in error rates across platform categories (Fig. <link>) in this setup. Productivity tools show the largest improvement (26%), even though Entertainment tools have the highest baseline grounding accuracy—highlighting the planner’s effectiveness. In contrast, Creativity platforms show the smallest improvement (14%), reflecting the challenges of planning and grounding in functionally rich interfaces with small UI elements.

R5: Lack of multi-step planning evaluation

We agree that long-term planning is important, but current models still struggle with single-step actions (nearly 0% recall on drag and only 20% on click actions). Strengthening performance on these core tasks is a necessary first step, with multi-step planning as a valuable direction for future work.

R6: Missing support for hybrid or cross-platform GUI environments

We agree that hybrid environments are a valuable direction for future work. However, our current benchmark already presents significant challenges for existing models showing that even desktop settings require substantial progress. We believe UI-Vision offers ample opportunities to advance model capabilities.

R7: Anonymous repo with full benchmark

We apologize for the inconvenience but as per ICML policy, we are only permitted to share links to figures and tables. However, to give reviewers a clearer sense of the benchmark, we have included detailed failure cases in our response to Reviewer 3NZB (R3) and provided additional task examples at <link1> and <link2> (5s loading). All code, data, and evaluation scripts will be soon released.

R8: Low performance of InternVL-8B

The main reason is that InternVL2-8B fails to consistently generate meaningful bounding boxes, often producing arbitrary outputs like [0, 0, 50, 50] or [0, 0, 200, 200]. This suggests limited training on grounding tasks, particularly for small UI elements. Its poor performance is also reflected in the ScreenSpot benchmark [1] (Table 2).

[1] Wu, et al. OS-ATLAS: A foundation action model for generalist GUI agents.

We thank the reviewer for the comments and address them below.

R1: Fine-grained error analysis, ablations on task difficulty, and generalization across software categories

To deepen our understanding of model performance, we conducted an error analysis through a human study for element grounding and found that models often struggle with small UI elements and platforms with high functional density. To investigate further, we leveraged the dense bounding box annotations in our dataset and sampled a diverse subset of elements that were consistently challenging for top-performing models such as UI-TARS, UGround-v1, and Aria UI. We selected cases where one or more models failed, resulting in a subset of 5479 samples spanning basic, functional, and spatial categories. We report detailed results across software and categories in Table in <link>, and summarize key findings below:

Error Types: Models often confuse visually similar elements, miss platform-specific icons, and fail to detect small elements in dense layouts.

Task Difficulty: GUI agents perform better on functional grounding than basic grounding, likely due to alignment with their training data. In contrast, VLMs underperform functional tasks, revealing task-specific weaknesses.

Category Generalization: Performance varies significantly across software categories. Creativity tools like Blender and GIMP (112 elements/frame, 418 px/element) show the lowest accuracy, while simpler platforms like VLC (63 elements/frame, 875 px/element) perform best. Notably, screenshot resolution had little impact.

We refer the reviewer to our response to Reviewer 3NZB (R3) for qualitative study and analysis on Layout Grounding and Action Prediction and more details and examples on Element Grounding.

R2: cross-software generalization analysis

We compare model performance on common apps (e.g., VSCode) vs. less common ones (e.g., FreeCAD, QGIS) using the element grounding subset in R1. results are included in Table in <link>. While we cannot confirm exact training data used in several models, this serves as a proxy for generalization analysis. We observe all models show significant accuracy drops on less common apps, confirming consistent generalization challenges.

R3: Concern about dataset bias due to focus on open-source software

Many open-source applications closely emulate proprietary counterparts, capturing similar complexity. For instance, LibreOffice mirrors Microsoft Office features and serves over 200 million users[1]. Given this overlap in functionality and interface design, we believe our benchmark remains meaningful and representative of real-world GUI scenarios. We refer the reviewer to our response to Reviewer aEXy (R3) for more details on our choice to focus on open-source software

[1]Wikipedia page for LibreOffice. https://en.wikipedia.org/wiki/LibreOffice

R4: Evaluation of inference speed, latency and token efficiency

We perform a detailed analysis of the inference speed, token efficiency and latency for different models and different benchmark tasks and report the numbers in <link>.

R5: Ensuring generalisation vs memorisation

Since we do not have access to the training data recipe of the most of models, we are not able to carry out a comprehensive study on this point. However, the failure case of Element Grounding in Fig. 12 in <link> indicates merely memorization can not ensure accurate grounding. Models need good generalization ability to excel on the task.

R6: Analysis of spatial grounding and drag actions

In the spatial setting, models must first correctly identify the reference element and then reason about its spatial relation—both steps are required for success. Also, VLMs are known to struggle with spatial reasoning, limiting performance.

For drag actions, models are rarely trained on such interactions in web data, making them difficult to execute. Also, success depends on accurately predicting both start and end points, increasing the chance of error.

R7: Failure cases in layout grounding

As shown in Fig. 13(a) and 14(a) in <link>, Gemini-1.5-Pro often fails to return a minimal bounding box for the ground truth region, although the correct region is usually contained within the predicted box.

R8: Domain shift between open and closed software

Yes, we do. However, open-source systems are built to provide functionalities similar to those of closed-source counterparts, so evaluating model capabilities in these scenarios will directly correlate with those of closed-source ones.

R9: Potential for automated annotation

Yes, we do. We have applied LLMs in the annotation of layout grounding in Sec 3.2. However, the major challenge is that UI tasks are quite fine-grained, so it will be hard to control the quality during automatic annotation at scale.

We thank the reviewer for recognizing the value of our desktop-centric benchmark, its utility for comprehensive analysis of GUI agents, and the clarity of the paper. Below, we address the concerns raised.

R1: Clarification on “largest desktop-centric benchmark” claim

We appreciate the reviewer’s point and agree that “largest” can be interpreted in different ways. While OmniAct includes 9802 samples (7639 desktop-related), it focuses solely on action prediction. In contrast, our benchmark includes 6484 samples across 83 applications and supports three tasks—action prediction, layout and element grounding—with dense frame-level UI annotations. Considering the range of platforms, tasks, and annotation details, we believe UI-Vision is the most comprehensive desktop GUI benchmark, and we are happy to revise the claim for clarity.

R2: Clarification on 442 vs. 450 video count inconsistency

To clarify, our dataset contains 450 densely annotated videos across 83 applications. However, for the action prediction task, we use 442 videos, excluding 8 that involved complex actions (e.g., press and hold Ctrl + drag + release), which most models cannot yet handle reliably making evaluations difficult. This is noted in the Limitations section (L795–797), and we will ensure both this and the earlier typo are clarified in the camera-ready version.

R3: Qualitative Evaluation

Below we highlight qualitative error analysis for all three benchmark tasks:

Element Grounding: Figures are available at <link>. Observations are on SOTA UI-TARS model

Fine-grained ambiguity: The model fails to recognize the correct target among several visually similar candidates (Fig. 9), highlighting the need for improved disambiguation strategies.

Lack of domain knowledge: The model misinterprets platform-specific elements, such as “Fontwork” represented by “F” symbol (Fig. 10). Incorporating external knowledge could help improve performance.

Small element detection: The model struggles with small UI elements, particularly in high-resolution or dense interfaces (Fig. 11). Iterative zoom-in strategies may address this limitation.

Cross-platform generalization: The model incorrectly transfers layout assumptions across platforms—eg., predicting the minimize button's position on macOS as it would appear on Windows (Fig. 12). This suggests memorization and overfitting.

Layout Grounding: Example cases below are available at <link>

Inaccurate bounding box placement: Closed-source models often predict bounding boxes that loosely cover the correct region without precisely matching its boundaries (Fig. 13a, 14a). This suggests difficulty in precise layout partitioning

Poor functional grouping: Open-source models sometimes fail to group elements correctly, even when the query explicitly mentions them (Fig. 13b).

Superficial semantic matching: Models sometimes default to grounding smaller elements that share surface-level keywords with the query but are semantically unrelated (e.g., predicting a “Design” button for a different design-related query, Fig. 14b)

Action Prediction: Example cases below are available at <link>

Poor grounding: Models often predict the correct action type but fail to ground it to the appropriate UI element (Fig. 15). This reflects challenges in bridging perception and execution.

Lack of platform knowledge: We observe that models sometimes hallucinate actions (eg., refer to non-existent elements) or misinterpret platform-specific elements (Figs. 16, 17), likely due to limited training exposure to diverse desktop environments.

High interface complexity: Dense and feature-rich platforms pose greater challenges for accurate action prediction. UI-TARS exhibits the highest error rate (85%) on creativity platforms (112 elements/frame) while performing better (72% error) on simpler education platforms (62 elements/frame).

R4: Clarification on offline/static benchmark setting

Our benchmark is offline and static by design. The controlled setup allows for a fine-grained evaluation of how perception and grounding errors affect downstream actions. By structuring tasks from perception to action prediction, UI-Vision helps isolate failure modes and provides insights crucial for building more robust agents before deployment in dynamic environments.

R5: Annotator consistency during dataset creation

We partnered with a for-profit company that provided experienced annotators (L-604-605). All annotators underwent training on the software and were required to pass several assessments related to the tasks to proceed. Those who failed were excluded. Additionally, we followed a detailed annotation protocol, which was refined during a month-long pilot phase where annotators received detailed feedback to ensure high-quality and consistent data (L-676-677).

We thank the reviewer for recognizing the value of our desktop-focused benchmark, task diversity beyond web/mobile settings, dense annotations, and actionable insights. We address the concerns below.

R1: Failures cases

We summarize key error patterns below and refer the reviewer to our response to Reviewer 3NZB (R3) for more details and examples.

• Element Grounding: Models often confuse visually similar elements, fail to recognize platform-specific icons and struggle with small elements in dense layouts.

• Layout Grounding: Common issues include bounding boxes that are too large or loosely defined and failure to group related UI elements correctly

• Action Prediction: Models frequently fail to ground actions correctly, hallucinate UI elements, and perform poorly on complex interfaces with many interactive elements.

R2: LLAMA-3.3-70B layout grounding validation process

The authors conducted a manual validation process to ensure quality using three criteria: (i) bounding boxes must tightly enclose relevant UI elements without including unrelated regions; (ii) grouped elements must form a semantically meaningful and visually coherent unit; and (iii) labels and descriptions must accurately reflect the group's function. Groups failing any check were discarded. The authors had a detailed protocol and access to the platform and its documentation to ensure consistency. We will include these validation details in the camera-ready version.

R3: Latency metrics

We report latency per query, average output tokens, and GPU usage across all three tasks in Tables at <link>, using default Hugging Face implementations for consistency. Token efficiency was measured with GPT-4 tokenization. Models like UI-TARS, trained on action-heavy tasks, generate longer outputs due to detailed step-by-step reasoning.

R4: Recall@d metric and value selection

We choose the d values based on the average bounding box size across the dataset after resizing all screenshots to a standard resolution (800×700) yielding a base value of (25, 35). This base value is then rescaled based on the original resolution of each sample to ensure consistent evaluation across varying interface sizes.

R5: Ablation on annotation density and task design impact

We perform a detailed ablation analysis (Fig. 18 at <link>) to understand factors affecting performance on 3 tasks. Across all three tasks, we find that densely packed applications with smaller UI elements like GIMP (112 elements/frame, area of 418 px/element) show lower performance compared to entertainment platforms with simpler layouts like VLC (63 elements/frame, area of 875 px/element).

Regarding task design, GUI agents perform comparably or better on functional grounding than basic grounding, likely due to alignment with their training data. In contrast, both open- and closed-source VLMs perform worse on functional tasks. Spatial grounding is the most challenging, as it requires identifying the correct element and reasoning about its relative position—an area where VLMs generally struggle due to limited spatial reasoning.

For a more comprehensive analysis with detailed settings, we refer the reviewer to our detailed response to Reviewer 7jPV (R1).

R5: annotator qualifications and training process

Beyond academic background, annotators were selected through technical assessments, language tests, and task-specific bootcamps. Those who didn’t meet the criteria were excluded. We used a detailed annotation protocol refined during a month-long pilot with feedback to ensure consistency (L-676–677). Quality was further ensured through manual reviews and ongoing performance monitoring. We will clarify these details in the camera-ready version.

R6: Clarification on UI-Vision’s unique contribution vs. existing benchmarks

We appreciate the reviewer’s concern and would like to clarify the distinct contribution of our work. While recent efforts [1–4] have advanced GUI agent evaluation, they primarily focus on web environments or online interaction. Specifically, [1] includes limited desktop data and annotations (OmniAct: 5412 training samples across 38 platforms only for action prediction). [3] and [4] focus mostly on web-based tasks. While [2] (OSWorld) targets desktop platforms, it evaluates across a limited number of platforms in an online setting using broad task completion metrics.

In contrast, our benchmark supports offline evaluation across 450 real-world desktop tasks spanning 83 applications. It provides a complete pipeline of three benchmark tasks, along with detailed evaluation metrics. This setup allows models to be assessed from basic perception to planning and execution, all within a single structured benchmark. Unlike existing works, UI-Vision offers fine-grained insights into where and how models fail, making it a valuable tool for diagnosing and improving GUI agents.