GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.

W1: Comparing GUI-Actor-7B + verifier to baselines that predict actions in a single pass overstates its advantage, because the verifier injects an extra round of visual evidence and decision logic that competing methods never see. A like-for-like comparison should disable the verifier or add an equivalent post-filter to the baselines.

Thank you for the valuable comment. We reported the performance of GUI-Actor-7B both with and without the Verifier in our ablation studies (see Tables 4 and 5). To facilitate a more direct comparison, we reorganize key results and extend the evaluation to Qwen2.5-VL based models, as shown in Table H and Table I below. We can see that even without the Verifier, GUI-Actor consistently outperforms all SFT counterparts (OS-Atlas, Aguvis, and UGround). Notably, on the challenging OoD benchmark ScreenSpot-Pro, it also surpasses UI-TARS (trained on a substantially larger dataset and using a more complex 3-stage pipeline). This underscores the inherent strength of the GUI-Actor grounding model and its robustness to distribution shift. We will include this comparative table in the next revision.

Table H. Performance Comparison across benchmarks for various models using Qwen2-VL as backbone.

Table I. Performance Comparison across benchmarks for various models using Qwen2.5-VL as backbone.

W2: Running the GUI-Actor-7B + verifier increases GPU memory and adds an extra forward pass per click, introducing latency that is hard to hide in real-time GUI agents.

Thank you for the insightful comment. We agree that adding the verifier introduces additional computational overhead. Our motivation for integrating the verifier is to explore test-time scaling for GUI-Actor. Unlike typical GUI grounding models, GUI-Actor can naturally propose multiple diverse candidate regions in a single pass, making it well-suited for pairing with a verifier at test time.

To reduce inference time, we adopt three strategies: (1) using a lightweight 2B verifier, (2) applying early stopping when the verifier’s confidence score exceeds 0.95, effectively reducing the average number of verifier calls to 3.3 per sample on ScreenSpot-v2, and (3) leveraging vLLM for efficient inference.

Considering real-time GUI agents, we have some findings below to support this design. For example, GUI-Actor-2B combined with a 2B verifier outperforms standalone 7B grounding models on the ScreenSpot-Pro benchmark. This means that with proper parallelization, the overall runtime is approximately equivalent to two 2B model inferences—still notably more efficient than running a 7B model—while achieving superior performance. This makes our approach particularly promising for deployment in real-time GUI agents on resource-constrained devices such as phones, which may not support 7B models.

Table J. Performance comparison across different model configurations.

W3: The proposed method makes the “coordinate-free” pointer to land only at patch centers. The 28 × 28 patch grid becomes too large and coarse when the input resolution is small, making fine-grained targets (e.g., tiny icons or narrow scroll bars) unreachable.

Thank you for the thoughtful feedback. Theoretically, GUI-Actor is capable of locating elements smaller than 28x28 but larger than 14x14. The challenge of small element localization does exist for those smaller than 14x14, e.g., when they are located at the corner patch of a 28x28 grid. Fortunately, these extreme cases are relatively rare in typical desktop scenarios, as shown in Table K.

To mitigate this limitation, we explored a simple yet effective "Zoom-In" strategy for GUI-Actor, where GUI-Actor makes an initial coarse prediction, then zooms in 2× around the predicted location and performs a second round of grounding. Table L shows this significantly boosts performance on ScreenSpot-Pro, demonstrating its value in high-resolution settings with small UI elements.

Table K. Frequency of small UI elements (longest edge < 14 px) fully contained within a single patch.

Table L. Effectiveness of the "Zoom-In" strategy on ScreenSpot-Pro.

Q1: The proposed method sets a confidence threshold, but it is not clear how the threshold is defined, and what if no candidate is above the threshold.

Thank you for pointing out the confusion. The verifier evaluates candidates sequentially, and we use an intuitive confidence threshold for early stopping: the first candidate exceeding this threshold is selected as the final prediction. If no candidate meets the threshold, we fall back to the one with the highest verifier score. We will clarify this in the revision. This strategy effectively reduces the average number of verifier calls from 20 to just 3.3 per sample on ScreenSpot-v2, with only a negligible impact on performance.

Q2: The author demonstrated fine-tuning the Qwen-2-VL model only. Is it possible to apply the proposed coordinate-free training to an existing GUI model such as UI-TARS-7B?

Thank you for the insightful question. Here we further apply our coordinate-free GUI-Actor framework to UI-TARS-7B. As shown in Table M, when using UI-TARS-7B as the base model, GUI-Actor achieves comparable performance on the ScreenSpot-v2 benchmark and higher accuracy on the more challenging ScreenSpot-Pro benchmark. We attribute this improvement to UI-TARS-7B’s strong visual encoder, which is initialized from Qwen2VL and further trained on GUI-specific datasets, likely leading to richer and more semantically aligned representations. When combined with our action-head-based grounding approach, these enhanced features enable more precise localization. These findings suggest that our method extends beyond general-purpose VLMs and can effectively complement and enhance existing GUI models.

Table M. Evaluation of GUI-Actor applied to two different base VLMs: Qwen2VL-7B and UI-TARS-7B-SFT.

Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.

W1: The action head requires the model to output the < ACTOR > token. For GUI-Actor-7B with LiteTrain, the VLM parameters remain frozen—how is the model reliably induced to produce this new < ACTOR > token, and was this token newly added to the vocabulary?

Yes, the <ACTOR> token is newly added to the vocabulary. As noted, in the GUI-Actor-7B w/ LiteTrain setting, all original VLM parameters are frozen; only the <ACTOR> token embedding and the parameters introduced in the Action Head are trainable.

During inference, if the model is only performing the GUI grounding task, it is not necessary for it to generate the <ACTOR> token. Instead, in the LiteTrain setting, we directly include the <ACTOR>token in the input prompt, e.g., "pyautogui.click(<ACTOR>)", and extract its final-layer hidden representation. This representation is then passed to the action head for grounding.

W2: Tables 4 and 5 show that GUI-Actor-7B (no Verifier) outperforms GUI-Actor-7B w/ LiteTrain (no Verifier) by a substantial margin. Could you analyze which component drives this gain? Is it better <ACTOR> formatting, more accurate action-head predictions, or something else?

The <ACTOR> token formatting is the same in both settings, so the performance gap is not due to differences in prompt structure. We believe the performance gain of GUI-Actor-7B (without Verifier) over GUI-Actor-7B with LiteTrain (also without Verifier) mainly comes from the difference in how many parameters of the model are fine-tuned. In the full fine-tuning setting, the entire model—including the ViT backbone—is updated, allowing for deeper task-specific adaptation. In contrast, the LiteTrain variant freezes the backbone and only updates the <ACTOR> token embedding and the Action Head, which limits its ability to fully align multimodal representations for the grounding task. In summary, while LiteTrain offers a more efficient training approach, full fine-tuning leads to stronger task adaptation and better overall performance.

W3: Adding the Verifier (Tables 4 and 5) yields a 3–9% boost in performance. Why does the action head’s direct attention map exhibit such a significant bias? Although the action model (7B) is much larger than the Verifier (2B), the action head’s direct attention maps still show substantial bias. One would expect the stronger model to leverage its internal reasoning capacity without external checks—could this indicate that the action head was not trained well enough to fully elicit the VLM’s inherent reasoning knowledge?

Thanks for the insightful comments. While it's true that adding a smaller Verifier improves performance, we believe it's not sufficient to conclude that the action head was inadequately trained or unable to leverage the VLM’s reasoning capacity.

The Verifier is trained as a binary classifier to assess whether a given candidate position is plausible, whereas the grounding model learns to predict a full attention map over all input image patches. These two tasks, i.e., evaluation versus direct prediction, differ fundamentally in complexity. As noted in Line 176, selecting the correct option from a list (as the Verifier does) is generally easier than directly generating a precise location from an unconstrained space.

Importantly, the performance boost from the Verifier suggests that the grounding model often identifies the correct region but may not consistently assign it the highest score, especially in cases involving ambiguity or multiple plausible targets. In these situations, the Verifier helps resolve uncertainty by selecting the most semantically appropriate candidate. Rather than compensating for a weak grounding model, the Verifier serves as a complementary module that enhances decision-making in challenging scenarios. For instance, we observe greater improvement on the more difficult benchmark, ScreenSpot-Pro.

W4: In the setup on L215, the Verifier may be invoked up to K = 20 times, introducing extra runtime and compute cost. Please include an inference-cost comparison against baseline models and report the average number of Verifier calls per sample.

Thank you for your constructive suggestion. We acknowledge that the verifier introduces additional computational cost. However, we believe this cost is justified, as test-time scaling with a verifier provides valuable insights into the design of more powerful GUI models. Compared to prior grounding models such as UITARS and AGUVIS, GUI-Actor offers a key advantage: it can generate multiple candidate regions in a single forward pass, significantly reducing computation at this stage.

To mitigate the cost introduced by the verifier, we have introduced an early stopping mechanism in our work—terminating verification once the confidence exceeds 0.95. This reduces the number of verifier calls substantially, as shown in Table G. On average, only 3.3 verifier calls are needed per sample on ScreenSpot-v2. Thanks to the efficient vLLM setup, this adds just 1.3 seconds per sample on average—an acceptable overhead with further room for parallelization and optimization. We also compared this approach with a fully parallel solution that scores all candidate areas simultaneously without early stopping. While it achieves similar latency, it incurs higher computational cost.

Table G. Inference-time comparison of different grounding models with and without the Verifier.

Q1: Is the self-attention applied to image-patch embeddings in the attention-based action head causal or non-causal? Could this choice affect performance?

The self-attention mechanism in the action head is non-causal. This design choice is intentional, as the goal is to enhance the model's ability to capture spatial relationships among image-patch embeddings. A non-causal setup allows each patch to attend to all others, which aligns with the intuition that spatial reasoning in GUI layouts requires global context rather than sequential or directional constraints. We believe this non-causal design is better suited for aggregating layout and UI structure effectively.

Q2: During GUI-Actor-7B’s supervised fine-tuning, which VLM parameters were actually trained?

During supervised fine-tuning of GUI-Actor-7B, we adopt a two-stage training strategy. First, we warm up the model by training only the newly introduced token embeddings and the parameters in the action head, while keeping the backbone VLM frozen. After this warm-up phase, we fine-tune the entire model, including the backbone VLM parameters. We will clarify this more explicitly in Line 221–224 in the revised manuscript.

<think>To continue...homepage of Play Store.</think>
Action: Click the back arrow.                          
pyautogui.click(<ACTOR>)

Thank you for your thoughtful follow-up. Firstly, we performed a deeper analysis of the CAD examples and confirmed that this subset is particularly challenging. Most target UI elements in CAD are extremely small Chinese buttons, unlike other subsets that typically feature text-free icons or English text buttons. Given that our training data predominantly consists of English webpages, this domain gap may explain why the performance on the CAD subset is significantly lower than other subsets.

To investigate further, we examined the attention maps produced by the LiteTrain and FullTrain (GUI-Actor-7B) models. While the LiteTrain model typically shows lower top-1 grounding accuracy, its attention maps are notably more dispersed compared to those from the FullTrain model, likely due to the frozen backbone VLM (including the ViT) in its training setup. This dispersion leads to more diverse region proposals being passed to the Verifier. Crucially, this improves proposal recall, i.e., the likelihood that the ground-truth region is among the top-k proposals.

In contrast, the FullTrain model often produces sharper, more focused attention, which can be beneficial in general but may miss difficult or low-salience elements such as the small Chinese UI targets in CAD. As a result, LiteTrain—despite its weaker standalone grounding—can achieve superior final localization accuracy on CAD when combined with the Verifier.

Additionally, we note that the underlying VLM backbone, Qwen2VL, was pretrained on a substantial amount of Chinese visual-language data. By freezing this backbone in LiteTrain, we preserve its stronger native capability in handling Chinese content. In contrast, full fine-tuning may inadvertently degrade some of these specialized representations, particularly when training is skewed toward English-centric data. This further contributes to LiteTrain’s relative advantage in the CAD setting.

As suggested, we computed proposal recall (defined as the percentage of cases where the ground-truth region appears among the top-20 proposals) as shown in Table S. For comparison, we also include numbers from the Creative subset. The results further support our explanation:

Table S. Proposal Recall on CAD and Creative subsets.

We appreciate your suggestion and will include this analysis in a future revision to better characterize the interplay between attention diversity, language-specific capabilities, and Verifier effectiveness.

Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.

We address W1 and Q1 together, as they both raise points about the architecture and design of the attention-based action head.

W1: The paper doesn't really explain how the attention-based action head works. Key parts like the "GUI-aware" self-attention are mentioned but not detailed. We don't know how many attention heads are used or how they're configured. Same goes for the MLP projection designs. The exact architecture of the action head remains unclear. This makes it hard to understand or reproduce.

Q1: The attention-based action head design really needs more explanation. How does the self-attention layer actually pull together information from different patches? The paper talks about "GUI-aware contextual information" but doesn't say what makes it special for GUIs. Do they use custom positional encodings? Is there some way to model how GUI elements relate to each other spatially?

Thanks for pointing out the confusion. As shown in Figure 2, Eq (2), Eq (3) and described in Line 150-152, the action head comprises three main components: (1) a multi-head self-attention layer, followed by (2) an MLP to extract vision patch features, and (3) a [ACTOR] token-specific MLP to extract semantic features.

The self-attention layer operates over image patches, where each patch representation is computed as an attention-weighted sum of all other patch representations. This mechanism allows the model to explicitly aggregate spatial relationships among UI elements and layout structures beyond what the original ViT encoder captures, by leveraging multi-patch supervision.

Regarding positional encoding, we do not introduce additional positional embeddings in this self-attention layer, as the input features from the ViT backbone already incorporate positional information.

For the configuration details: The number of attention heads in the self-attention layer is set to 8; Both MLP components are two-layer feedforward networks with a GELU activation in between. As stated in Line 213–214, we use the same dimensionality as the backbone VLM for all dimension configuration within the action head.

We will revise the manuscripts and open source our code to ensure clarity and reproducibility.

W2: The experimental setup has some issues. The verifier training uses simple binary classification with only correct/incorrect labels. It relies on fake negative examples generated by random coordinate shifts. But real problems are usually messier than that. Users might click slightly off-target or have different interaction patterns.

Thank you for the feedback. We believe the comment may stem from a partial interpretation of our setup, and we would like to clarify the rationale and design behind our verifier training. (1) Use of Constructed Negatives: While the negative examples are indeed constructed rather than collected from real user interactions, this contrastive strategy is a widely used and effective practice in many learning paradigms such as contrastive learning and self-supervised learning. In our case, these constructed negatives help the verifier distinguish correct clicks from plausible but incorrect alternatives. (2) Diversity of Negative Samples: Our negative sampling is not limited to simple random coordinate shifts. As detailed in Line 185-187, we deliberately introduced varying levels of difficulty_** to better reflect real-world behavior. In addition to random offsets (which naturally include slightly off-target cases), we also select negative points located within the bounding boxes of other semantically meaningful UI elements. This design simulates realistic interaction ambiguities, such as users clicking on relevant but incorrect elements.

W3: There's also a lack of thorough error analysis. The authors don't dig into when and why things go wrong. What types of GUI elements cause failures? The verifier training uses those oversimplified artificial negatives.

Thank you for the valuable suggestion. We conducted a detailed error analysis across 40 examples from SS-Pro and SS-v2, respectively. Table E shows the details. We hope this breakdown offers insight into when and why failures occur and shows how dataset properties (e.g., ambiguity, element size) affect performance. We will include this analysis in the next version to guide future improvement.

Table E. Error Analysis on 40 examples from SS-pro and SS-v2, respectively.

Please refer to W2 for the concern on verifier training with artificial negatives.

Q2: The multi-patch supervision approach could use some ablation studies too. When they decide which patches count as "positive" based on overlap, how much does performance change if you tweak those thresholds? It seems like this could make a big difference but they don't test it.

The multi-patch supervision during training is determined objectively. As described in Line163-164 and illustrated in Figure 2b, our multi-patch supervision strategy labels all image patches that are partially or fully covered by the ground-truth bounding box as positive (label = 1), and all other patches as negative (label = 0). This binary supervision is applied objectively based on spatial overlap, without threshold tuning. While it is possible to apply soft labels to boundary patches (e.g., based on intersection ratios), we conjecture the impact would be limited, as only a narrow band of edge patches would be affected. We will consider this in future work.

Regarding the threshold used at inference time to aggregate multiple clustered patches into a final prediction, we conducted an ablation by varying the value from 0.1 to 1.0. As shown in Table F, performance remains stable within a moderate range (0.3 to 0.6). However, extreme values, i.e., either too permissive (0.1) or too restrictive (1.0, equivalent to using only the top-1 patch), lead to drops in accuracy. These results indicate that while the threshold choice does impact performance, the method is robust across a reasonable range. We will include this ablation in the final version to clarify this design choice.

Table F. Ablation study on the attention threshold for final prediction. Numbers from 0.3 is reported in the paper.

Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.

W: The primary weakness, which the authors acknowledge, is the dependency on the backbone VLM's fixed patch size. This makes it difficult to ground actions on very small UI elements that may be smaller than a single visual patch.

In our proposed method, the attention-weighted center of top-K patches can help mitigate this issue and localize small UI elements (<14×14 pixels) that overlap with 3-4 patches. However, current design of GUI-Actor may still struggle when such elements fall entirely within a single patch. Fortunately, these extreme cases are relatively rare in typical computer use scenarios, as shown in Table A.

To mitigate this limitation, we explored a simple yet effective "Zoom-In" strategy, where GUI-Actor makes an initial coarse prediction, then zooms in 2× around the predicted location and performs a second round of grounding. Table B shows this significantly boosts performance on ScreenSpot-Pro, demonstrating its effectiveness in mitigating grounding errors in high-resolution settings with small UI elements.

Table A. Frequency of small UI elements (longest edge < 14 px) fully contained within a single patch.

Table B. Effectiveness of the "Zoom-In" strategy on ScreenSpot-Pro.

Q1: The paper identifies the limitation regarding small UI elements due to fixed patch sizes. Could you provide a more quantitative analysis of this issue? For instance, what proportion of errors on ScreenSpot-Pro are attributable to this, and how effective is the described clustering refinement strategy at mitigating them? Have you considered architectural changes to address this more directly, such as incorporating multi-scale features from the vision encoder into the action head?

Thank you for the constructive suggestion. We provide a more detailed quantitative analysis in Table C, where ScreenSpot-Pro targets are grouped by bounding box area relative to the patch area (n = 14×14). As shown, accuracy for the smallest elements $[0,n)$ is significantly lower than for larger ones, underscoring the challenge of grounding small UI components. Notably, Table C also shows that our proposed refinement strategy (attention-weighted center) consistently improves accuracy across all size ranges, including the most difficult small-target cases.

We appreciate the suggestion regarding architectural improvements. Integrating multi-scale features, coarse-to-fine attention, or learning an offset is a promising direction for improving fine-grained localization. We leave this for future work.

Table C. Accuracy by target UI element size on ScreenSpot-Pro, measured by area, n=14*14, without Verifier.

Q2: The verifier is shown to be general enough to boost the performance of the AGUVIS baseline. This is a strong result. Given that your coordinate-free method excels at general localization and coordinate-based methods can offer high precision, have you considered a hybrid model? For example, one that uses the attention head to identify the correct UI element and a secondary mechanism to predict a precise click coordinate within that element's bounding box.

Thank you for the insightful question. Table D presents the results for a two-stage strategy in which the original screenshot is first cropped to half its size (H x W -> H/2 x W/2), centering the first-round prediction. Then the cropped image is used as input for a second-round prediction. This setup can be viewed as a hybrid model when different models are used across the two stages.

As shown in Table D, this two-stage approach consistently improves performance. It demonstrates that GUI-Actor not only offers more accurate coarse-grained localization but is also capable of producing finer-grained click coordinates. While small UI elements pose challenges for GUI-Actor in predicting precise coordinates, they similarly affect coordinate generation-based methods. In cases where UI elements are very small, coordinate-based models may struggle to capture and learn the correct semantics without explicit spatial alignment. In contrast, GUI-Actor benefits from such explicit alignment through its action head, which helps guide more precise localization.

Table D. Effectiveness of the hybrid strategy on ScreenSpot-Pro.

Q3: The training data for the verifier is constructed from the OS-Atlas dataset, while the main model uses a different collection of datasets. Could you discuss the potential impact of this domain shift? Would there be a benefit to training the verifier on the exact same data distribution that the main model is trained on?

Thank you for the question. Our Verifier was trained during an earlier stage of the project using the OS-Atlas dataset, before the UGround bounding-box dataset was released on May 1, 2025 (available on Hugging Face). Generating a full UGround-based verifier dataset requires extra time, so we decided not to create a new verifier dataset and train it again. Thanks to our diverse negative-sampling strategy and the inherent variety in OS-Atlas, the Verifier generalizes very well—even when paired with a grounding model trained on different datasets. While one could certainly retrain the Verifier on the exact same distribution as the main model, our experiments suggest that the existing setup already delivers strong performance with minimal domain-shift impact.