SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning

We sincerely thank Reviewer AHhf for recognizing the novelty of our work, particularly appreciating our innovative approach of self-evolutionary reinforcement fine-tuning and impressive performance on multiple benchmarks. Below, we provide detailed responses to each concern.

W1 & Q1. Data Filtering Reliability Concerns

Thanks for the comment. Difficulty is a relative measure that depends on the model. We apply data filtering using the specific model targeted for training to support our adaptive curriculum design as already-mastered samples provide no learning signal (zero gradient) during RL.

Additionally, we want to emphasize that our filtering process is not exclusively based on Qwen2.5-VL-7B. As described in Section 3.1, for Instruction Quality Filtering and Bounding Box Quality Filtering, we use GPT-4 and a fine-tuned variant of Qwen2.5-VL-7B, as the criteria for distinguishing good from poor examples are well-defined.

Q2. Scaling to Larger Models

Thanks for the comment. Unfortunately, due to constraints in both computational resources and time, we could not afford RL on a 32B/72B model. Nevertheless, our experiments demonstrate successful scaling from 3B to 7B models with consistent improvements, while concurrent works such as UI-R1 and GUI-R1 limited their experiments to 3B models.

From a theoretical perspective, larger models typically exhibit better attention quality and spatial reasoning capabilities, which should enhance our filtering effectiveness as attention maps become more precise with increased model capacity. Our core contributions—attention-based filtering and iterative data curation—operate at the architectural level through transformer attention mechanisms rather than model-specific features, suggesting natural scalability as model size increases.

Q3. Data Scaling

Thanks for your comment. We have systematically studied data scaling and observed encouraging trends. As shown below, increasing the number of high-quality filtered samples leads to steady improvements on ScreenSpot-Pro:

Moreover, our filtering strategy naturally balances data quality and quantity. Unlike traditional approaches that scale linearly with raw data volume, our method emphasizes useful data density—each filtered sample carries a higher learning signal. Through additional analysis, we find that data diversity plays a more critical role than sheer volume: adding 1K diverse samples leads to greater performance gains. We believe that incorporating more high-quality, diverse filtered samples has the potential to further enhance performance.

We sincerely thank reviewer iFgo for the acknowledgment that our high-quality filtered dataset could benefit the research community and that our task-specific method demonstrates reasonable design with significant performance gains on ScreenSpot-Pro. Below, we provide detailed responses to each concern.

Q1 & W1.Missing Ablation on Core Term

This ablation is embedded within our experimental design and results are reported in Figure 3(b). Our multi-stage training provides a natural ablation study:

• Stage 1: Standard GRPO without attention filtering ($f(\cdot) = 1$ for all samples)
• Stage 2+: GRPO with attention-based sample filtering ($f(\cdot)$ varies based on attention quality)

As shown in Figure 3(b), Stage 1 achieves 39.97% while Stage 2 reaches 43.0%, directly demonstrating the 3.03% improvement from attention-based filtering. This comparison isolates the effect of our core contribution while controlling for all other factors.

Additionally, we provide threshold sensitivity analysis ($\tau \in \{0.1, 0.2, 0.4\}$) in Appendix D, showing that $\tau=0.2$ yields optimal filtering performance.

Q2 & W2. Lack of Visual Evidence

Thanks for your suggestion. Additional figures are not permitted in the rebuttal, and we will include visualization results in the revision. However, based on our current analysis, we have observed that state-of-the-art (SOTA) methods often struggle in two key categories of scenarios. The first involves identifying UI icons and accurately grounding them to specific locations, such as in tasks like “close the location labels on the map” or “delete the 'article' tag for searching.” The second involves professional or domain-specific software operations, such as “use the select tool” or “adjust the style.”

Our method consistently performs better in these challenging scenarios, thanks to its stronger generalization capabilities and the use of an attention-based alignment mechanism that jointly models semantic intent and visual grounding. These advantages are also reflected in our results on the ScreenSpot-Pro benchmark, which is specifically designed to test such difficult cases.

On ScreenSpot-Pro, our model demonstrates significantly stronger icon grounding performance compared to SOTA baselines. Furthermore, for professional software-related queries, the accuracy of UI-TARS-72B drops below 10%, while our model achieves up to 40%, underscoring the robustness and practical utility of our approach in real-world, specialized UI tasks.

Q3 & W3. Selective Performance Reporting

Thanks for the comment. We provide the missing UI-TARS results below:

ScreenSpot

ScreenSpot-v2

ScreenSpot-Pro

Although UI-TARS-7B outperforms our SE-GUI-7B on ScreenSpot (89.5% vs. 88.2%) and ScreenSpot-v2 (91.6% vs. 90.3%), SE-GUI-7B matches the performance of the much larger UI-TARS-72B on ScreenSpot (88.2% vs. 88.4%) and ScreenSpot-v2 (90.3% vs. 90.3%). We argue that performance on these two datasets appears to be approaching saturation. Additionlly, UI-TARS benefits from training on 50 billion tokens including an undisclosed amount of proprietary annotated data. In contrast, our model is trained on only 3K curated samples. We demonstrate that with strategic data curation and iterative refinement, it is possible to achieve competitive performance using orders of magnitude less data.

Furthermore, UI-TARS requires extensive data collection and computational resources, which may not be accessible to most researchers. Our approach, relying solely on open-source data, achieves comparable results with significantly lower resource demands, further validating the effectiveness of our data-efficient methodology.

We deliberately focus on ScreenSpot-Pro, as its increased difficulty offers a more discriminative evaluation of model capabilities. The substantial performance gap observed on this benchmark (47.3% vs. 38.1% against UI-TARS-72B) highlights the clear advantage of our data-efficient approach in scenarios with limited supervision and higher task complexity.

Q4 & W4. Clarity, figure and type issues

Thanks for the detailed suggestion. We will clarify the $R_{\text{point}}$ notation, properly reference Appendix Figure 6 in the main text and improve quality of some figures. We will revise the manuscript accordingly.

We sincerely thank reviewer iFgo for acknowledging that our rebuttal has addressed your main concerns. We also appreciate the constructive suggestion to include more visual examples, and we will certainly incorporate them into the revised manuscript as suggested. By the way, If there are any additional points or clarifications that could assist in raising the rating, we would be pleased to provide them promptly. Thank you again.

We sincerely thank reviewer g7ps for recognizing the novelty of our work, particularly the exploration of attention maps to filter training samples. We are encouraged by the recognition of our impressive performance boost. Below, we address the specific questions raised.

Q1. Why use attention-map score as a constant coefficient rather than a component in the composite reward?

We designed attention as a gating mechanism rather than a reward component for a fundamental reason: data quality control vs. gradient weighting. As a gating mechanism: Poor-attention samples are completely filtered out, preventing the model from learning from potentially misleading supervision signals. As a reward component: $R = \alpha R_f + \beta R_p + \gamma R_{att}$ would still allow poor-attention samples to contribute to training with lower weights, potentially introducing noise. Integrating attention scores into the reward signal does not significantly alter the learning process. This is because samples with low attention values are typically the ones the model fails to predict correctly anyway, resulting in an inherently low task reward to begin with. Our goal is selective learning on high-quality samples rather than weighted learning on all samples. In GUI grounding, where spatial attention alignment is crucial, we found that complete filtering yields better results.

Q2.1. Performance Analysis Questions

We investigated these phenomenons:

1. Is there some special difficulties for Creative scenarios?: Creative categories (Photoshop, Blender, AI, etc.) involve specialized professional tools with: non-standard UI elements and custom icons, high-resolution interfaces requiring fine-grained localization and domain-specific visual vocabularies underrepresented in pre-training data. This creates a fundamental knowledge gap that RL cannot bridge-it optimizes existing capabilities rather than acquiring new domain expertise.

2. The performances on Creative and Scientific categories drop when using dense reward both with and without data filtering:: While dense rewards generally provide more granular feedback, reinforcement learning training is inherently stochastic. We conducted two separate runs of the initial training phase and observed that the overall performance remained stable at approximately 40.0, although there were slight fluctuations in performance across different sub-categories between the runs.:

3. The performance of icon grounding on Creative category is remarkably lower:: Icon recognition heavily depends on visual concepts learned during pre-training. Creative software icons are significantly underrepresented in typical training corpora, explaining why all models (including UI-TARS-7B 9.1% accuracy on creative icon grounding) consistently struggle in these categories—this is a dataset limitation, not a method-specific issue.

Q2.2. Missing Results

Thanks for the comment. We provide the missing UI-TARS-7B/72B and SE-GUI-3B results below:

ScreenSpot

ScreenSpot-v2

ScreenSpot-Pro

Although UI-TARS-7B outperforms our SE-GUI-7B on ScreenSpot (89.5% vs. 88.2%) and ScreenSpot-v2 (91.6% vs. 90.3%), SE-GUI-7B matches the performance of the much larger UI-TARS-72B on ScreenSpot (88.2% vs. 88.4%) and ScreenSpot-v2 (90.3% vs. 90.3%). We argue that performance on these two datasets appears to be approaching saturation. Additionlly, UI-TARS benefits from training on 50 billion tokens including an undisclosed amount of proprietary annotated data. In contrast, our model is trained on only 3K curated samples. We demonstrate that with strategic data curation and iterative refinement, it is possible to achieve competitive performance using orders of magnitude less data.

Furthermore, UI-TARS requires extensive data collection and computational resources, which may not be accessible to most researchers. Our approach, relying solely on open-source data, achieves comparable results with significantly lower resource demands, further validating the effectiveness of our data-efficient methodology.

We deliberately focus on ScreenSpot-Pro, as its increased difficulty offers a more discriminative evaluation of model capabilities. The substantial performance gap observed on this benchmark (47.3% vs. 38.1% against UI-TARS-72B) highlights the clear advantage of our data-efficient approach in scenarios with limited supervision and higher task complexity.

Q3. Clarity issues

Thanks for the detailed suggestion. The reward $r_i$ represents the sum of format reward and point reward: $r_i = \alpha R_f + \beta R_p$. "input text for scroll actions" should be "input text for type actions". We will revise the manuscript accordingly.

We sincerely thank reviewer z7zC for the acknowledgment of our strong performance on competitive benchmarks and the recognition that our SE-GUI-3k could benefit the broader community. Below, we provide detailed responses to each concern.

W1. The attention-based approach is less intuitively compelling than other aspects of the method.

Our attention-based method provides unique advantages: We deconstruct the grounding task into two fundamental stages: comprehension (interpreting the visual input and user query) and localization (outputting precise coordinates). We argue that most existing models are bottlenecked by accurate localization. Frequently, a model demonstrates a clear understanding of the target object yet errs in its reported coordinates. This discrepancy is empirically supported by experiments like the ground-truth vs. prediction analysis in SeeClick(ACL 2024) (many incorrect predictions are also close to the answer, suggesting the model recognizes the target but needs improvement in fine-grained localization). Consequently, attention mechanisms offer a direct window into the model's internal focus. They allow us to verify whether the model has successfully attended to the correct target, even when the subsequent localization step is erroneous.

In contrast, conventional threshold-based filtering is suboptimal as it cannot distinguish between two distinct types of low-scoring samples: (a) roughly identifies the correct region but fails precise localization vs. (b) completely misses the target area. A fixed threshold would indiscriminately discard both. Our attention-based approach, however, is designed to address this limitation. It selectively retains these valuable samples, allowing the model to gradually learn to refine its predictions on these challenging yet informative examples as training progresses. Since most current models use Transformer architectures, our method has broad applicability without requiring architectural modifications.

W2. Should the baseline be trained for the same total number of steps as all stages combined?

We understand your concern on the fairness of the ablations studies. We ensured all ablation trainings converged. Additionally, we found that the number of steps showed weak influence on model performance, as we will demonstrate below. We conducted experiments training baseline models for equivalent total steps (40 epochs), as shown in the table below.

Results demonstrate minimal improvement with prolonged training, eventually suffering from performance degradation due to overfitting. This confirms that our gains come from iterative attention-guided data curation rather than simply longer training duration. The key insight is that our method changes what the model learns from, not just how long it learns. It shows an important distinction between our multi-stage approach and extended single-stage training. Our method is fundamentally different from simply training longer—each stage uses attention analysis to curate a different subset of training data rather than continuing training on the same data. The Stage 1 model's attention maps guide Stage 2's data selection, creating an evolving curriculum that adapts based on the model's demonstrated focus patterns.

N1. Regarding concurrent work

While the ScreenSpot-Pro leaderboard shows some models with stronger performance, such as UI-TARS-1.5 and GTA. However, UI-TARS-1.5 is trained on private data, with their 7B model achieving 49.6% (marginally better than our 47.3%). GTA-1-7B reports 50.1% accuracy but is trained on million-scale datasets. In contrast, we achieve competitive results using only 3k open-source data points, demonstrating superior data efficiency while maintaining comparable performance.