AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs

Q4: Why does the scoring range from 0 to 3? More explanation is needed on how this scoring system handles errors, biases, and ambiguous cases

A: Thanks for raising this point. As explained in Section A.5, the 0-3 scoring range was chosen based on empirical findings that it strikes a balance between annotation efficiency and data quality. This range effectively filters out invalid samples while retaining most valid ones.

In our preliminary experiments, we observed that a binary score (0/1) for the LLM rejector was biased, often undesirably assigning a score of 1 to ambiguous samples. This was because the LLM tended to treat minor annotation errors as having negligible impact on overall quality. Similarly, the LLM-based verifier exhibited this behavior, resulting in a higher proportion of incorrect and ambiguous annotations.

(1) For the LLM rejector, to identify the optimal scoring range, we tested several thresholds: 0-2, 0-3, and 0-4. We conducted this evaluation on a set of 216 tasks, consisting of 147 valid and 69 invalid samples. We then plotted a rejection ratio curve for both valid and invalid samples at various thresholds (i.e., samples with scores below the threshold are rejected). The goal was to minimize the area under the curve (AUC) for valid samples, while maximizing it for invalid ones, ensuring that valid samples are ranked higher than invalid ones. Please refer to Figure F in the revised paper for a visual comparison.

The results showed that the 0-3 range provided the best separation: it had the largest AUC for invalid samples and a small AUC for valid ones, indicating a better balance. This range was therefore selected for the LLM rejector.

(2) For the LLM-based verifier, we tested the verification accuracy using 100 samples across the three scoring ranges: 0-2, 0-3, and 0-4. The verification accuracy is defined as the number of correctly classified annotations divided by the total (note that an annotation whose verification score is not full will be classified as incorrect). The 0-3 range achieved the highest accuracy at 96%, compared to 92% for 0-2 and 94% for 0-4. Further analysis revealed that the 0-2 range was prone to assigning high scores to ambiguous samples, reducing accuracy, while the 0-4 range was too stringent, misclassifying more valid annotations as invalid. Thus, the 0-3 range with higher accuracy was selected.

In conclusion, the 0-3 scoring range was chosen to mitigate LLM bias in binary scoring and to effectively balance the retention of valid samples with the rejection of invalid ones.

---

We sincerely appreciate your detailed feedback. We hope the above response can address all your concerns. If you have any questions, we are pleased to provide further clarification!

Q3: When fine-tuned using AutoGUI702k, there is significant performance fluctuation compared to SeeClick. Do the authors have further explanations for these variations?

We understand your concern and would like to further clarify it: the variations possibly originate from the discrepancy between the grounding tasks of the training data and benchmarks.

Qwen-VL-AutoGUI702k is fine-tuned with a focus on functionality semantics, which improves its performance on benchmarks that require locating elements based on functional referring expressions (e.g., FuncPred, VWB AG, and MOTIF). However, it may achieve marginal improvements on benchmarks that rely more on element properties such as relative positions or text (e.g., RefExp, ScreenSpot).

To make this explanation clearer, we inspect samples of each benchmark on which the performance gap between Qwen-VL-AutoGUI702k and SeeClick is noticeable:

(1) Inspecting 50 samples on FuncPred, SeeClick underperforms due to:

• (a) Heavily rely on element texts (64%): SeeClick can locate an element correctly when the referring expression (RE) includes the displayed text of the target element but struggles when the expression does not. For example, when the RE explicitly mentions “log-in” for a “Log-in” button, SeeClick can correctly locate the element; interestingly, when the referring expression says “This element triggers the display of a password recovery page”, SeeClick mistakenly locates a “Password” text field instead of the “Forget password” button below the text field.

• (b) Almost hit the target (16%): The predicted point of SeeClick is very close to the ground truth bounding box.

• (c) Others (20%): locate either wrong elements or non-interactable areas without any explicit reasons.

(2) Inspecting 20 samples on VWB AG, SeeClick is slightly worse due to:

• (a) Almost hitting the target (30%).

• (b) Heavily rely on element texts (30%). For example, given a referring expression “change the country/region of Calendar”, SeeClick mistakenly points to a calendar element instead of the “Country selection” menu.

• (c) Others (40%).

(3) Inspecting 30 samples on ScreenSpot, Qwen-VL-AutoGUI702k underperforms due to:

• (a) Incorrect text localization (46.7%). ScreenSpot contains many text localization tasks. For example, it misidentifies "number 5" as "number 7" in a calculator UI.

• (b) Almost hit the target (40.0%).

• (c) Others (13.3%).

(4) Inspecting 30 samples on RefExp, Qwen-VL-AutoGUI702k underperforms due to:

• (a) Difficulty with position-based REs (43.3%). The RefExp REs mostly are short phrases with position descriptions, such as “click on the third image on the right from the top”. As our AutoGUI dataset mainly focuses on element functionality, Qwen-VL-AutoGUI702k is not quite capable of understanding positional relationships. For example, a task says “select the symbol which is immediate to the left of keep me signed in”, but Qwen-VL-AutoGUI702k taps on the text “keep me signed in” instead of the checkbox left to this text.

• (b) Almost hit the target (36.7%).

• (c) Others (20%).

(5) Inspecting 30 samples on MOTIF, SeeClick underperforms due to:

• (a) Failed to understand functionality (40%), such as "set a daily reminder to write my diary," due to a lack of functional data in its training set.

• (b) Almost hit the target (16.7%).

• (c) Others (43.3%).

Hope this detailed analysis can address your concern.

Dear Reviewer vvNy:

We sincerely thank you for these constructive comments and evaluation of our paper. With the ICLR public discussion phase ending in two days, we kindly ask you to take a look at our responses. Our rebuttal provided more clarification of our framework and additional experiments in response to your concerns. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Q4: This paper mentions that the ultimate goal of the research is to enable next-generation software automation. However, the work presented here focuses solely on the GUI grounding task, lacking exploration of practical application scenarios.

A: Thank you for your feedback. We would like to clarify that:

• The mention of "software automation" in our paper serves as the broader context for our work. The primary focus of this paper is GUI grounding, which we consider a critical capability for vision-language models used in software automation.

• Furthermore, we explicitly state the goal of this research—namely, to provide a dataset rich in functionality semantics to enhance GUI grounding—in the paper title, as well as in the Abstract (lines 27-28) and Introduction (lines 43-44 & 90-94).

We hope these references help clarify the scope of our work and prevent any potential misunderstanding.

---

Q5: The experimental results indicate that the proposed AutoGUI dataset has not led to substantial advancements in GUI grounding tasks.

A: Thank you for your interest in the effectiveness of the AutoGUI dataset. We'd like to clarify that

• Qwen-VL fine-tuned with AutoGUI data achieves impressive grounding accuracy on FuncPred, VWB AG, and MOTIF (Table 4).

• Furthermore, the experiment in Q1 shows that fine-tuned with the AutoGUI dataset augmented by expanding data amount and reformatting, Qwen-VL shows more noticeable improvement. See Q1 for analysis.

• Moreover, the experiment in Q2 shows that AutoGUI is more significant compared with SeeClick on FuncPred, VWB EG, VWB AG, and MOTIF, although lags on RefExp due to referring expression format discrepancy.

In summary, we believe our AutoGUI dataset can enhance GUI grounding performance and we will continue expanding the data and utilize diverse functionality formats to reduce the gap between AutoGUI data and the benchmarks.

---

References:

[1] On the Effects of Data Scale on Computer Control Agents, NIPS 2024 Datasets & Benchmarks

We thank Reviewer ZWrk again for the insightful review and feedback and we hope that the above responses adequately address all concerns.

Thank you for a thorough review that will help us improve the work. Please see below for answers to your questions.

---

Q1:Qwen-VL-AutoGUI702k performed worse than SeeClick on VWB EG, RefExp, and ScreenSpot. The paper attributes this performance gap to the absence of data. More experiments are needed to support this argument.

A: We agree that a more comprehensive analysis, including the ScreenSpot subsets, would provide a clearer explanation. The performance metrics on these subsets are provided in the table below for reference.

To justify that the gap can be attributed to the absence of data, we annotated the trajectories from other sources. As no open-source large-scale trajectory data collected on Apple or desktop software was available, we used the AndroidControl dataset [1] to conduct this experiment. We annotated the trajectories from AndroidControl, expanding the dataset to 772k samples. Fine-tuning Qwen-VL on this augmented dataset led to improved performance on the ScreenSpot-Mobile and VWB EG subsets (Row 3). This suggests that dataset expansion can positively impact performance.

Another possible cause is referring expression format discrepancy. The RE formats used in the benchmarks differ from our functionality annotations, which could have constrained the benefits of the AutoGUI dataset. For example, our annotations describe functionality like "this element navigates users to the home page," while formats like those in MOTIF and RefExp are more user-intent-oriented, such as "select the text below the images.”

To fill this gap, we reformatted our functionality annotations to align with the user-intent format. For instance, "go to the link element that opens a pop-up window for login." These reformatted samples were incorporated into the AutoGUI-702k dataset for further fine-tuning, resulting in a noticeable improvement on RefExp (Row 4). This demonstrates that aligning the RE format can help reduce the performance gap.

In summary, the performance gap can be attributed to both the absence of certain data and discrepancies in referring expression formats. We will revise the experimental analysis in the main paper to incorporate these insights.

---

Q2: Is the role of AutoGUI data not as significant as SeeClick data?

A: We’d like to clarify that AutoGUI data is significant compared to the SeeClick-Web portion that is actually annotated by the SeeClick authors.

Only the web portion in the SeeClick training data (SeeClick-Web) is generated by the SeeClick authors while the other portions (i.e. Widget captioning, RICOSCA, screen summarization, and LLaVA Pre-train) originate from existing public data sources.

To directly compare the effectiveness of SeeClick-Web and our AutoGUI data, we conducted two experiments: 1) Comparing Qwen-VL fine-tuned on SeeClick-Web versus AutoGUI702k; 2) Comparing Qwen-VL fine-tuned on the full SeeClick dataset versus a combination of AutoGUI702k and the full SeeClick dataset, excluding SeeClick-Web.

The results show that AutoGUI data outperforms SeeClick-Web when used independently (r2 vs. r1). Furthermore, when combined with other data sources, AutoGUI still leads across the four benchmarks. The underperformance of AutoGUI on RefExp is likely due to a format discrepancy between the functionality annotations in AutoGUI and the referring expressions in RefExp.

---

Q3: Additional experiment on the Visual appearance and category annotation type.

A: We used GPT-4o-mini to annotate the visual appearance as well as the element category for 125k elements in AutoGUI702k. A description is like: A blue eye icon representing the "Show password" button.

Fine-tuning Qwen-VL with 25k and 125k subsets, we find that the visual appearance & category annotation type is inferior to the proposed functionality annotation type, especially on the FuncPred benchmark.

We thank the reviewer for the insightful comments. Below, we address the concerns:

---

Q1: The dataset would enjoy more advantages if it contained more platforms.

A: We fully agree that expanding our dataset to include more platforms would significantly enhance its diversity and relevance. To address this, we are planning to incorporate GUI data from additional platforms in the next iteration of AutoGUI. Specifically, we intend to collect data on Windows using pywinauto [1], and on iOS using the Xcode Accessibility Inspector [2]. We also hope that the community can extend our annotation pipeline to diverse platforms, further enriching the dataset's utility and applicability.

---

Q2: The performance of state-of-the-art VLMs (both open and closed source).

A: In response to your concern, we have expanded our evaluation to include three powerful models. The results show that GPT-4o, while strong in general tasks, does not perform as well on the GUI element grounding benchmarks. In contrast, Qwen2-VL-72B-Instruct, which has been specifically trained with extensive GUI data, shows superior grounding accuracy over GPT-4o. However, Llama-3.2-11B-Vision-Instruct exhibits suboptimal performance across all benchmarks.

---

Q3: Authors could have performed more fine-tuning experiments.

A: We fine-tuned Llava-1.5-13B with the three scales of AutoGUI data. The results show that this model also demonstrates improved grounding accuracy across all evaluated benchmarks, further validating the contribution of our dataset.

---

Q4: The benefits of the AutoGUI dataset compared with other datasets.

A: In response to your concern, we compared AutoGUI702k with two large datasets mentioned in Table 1, i.e., Wid. Cap.+RICOSCA (merged due to shared GUI screenshot sources), and SeeClick-Web. We did not compare with UI REC/REG and Ferret-UI as the former has been compared in Table 5 while the latter was not open-source.

The table below shows that the updated AutoGUI dataset (row 4) achieves improved grounding accuracy across all benchmarks, except RefExp. Upon further analysis, we observed that the referring expression (RE) formats in Wid. Cap.+RICOSCA and RefExp are similar, both incorporating user intents with positional descriptions, which may explain the higher performance on RefExp. For example, RefExp tasks are user intents with position descriptions, such as “click on the third image on the right from the top” (refer to https://huggingface.co/datasets/ivelin/rico_refexp_combined). RICOSCA samples also take on this format (https://huggingface.co/datasets/rootsautomation/RICO-SCA).

Additionally, the RE format of RefExp differs from that of our functionality annotation, which might have limited the perceived benefits of AutoGUI702k. For example, our functionality is phrased as “this element navigates users to the home page”, while RefExp includes user intent-focused REs.

To fill this format-related gap, we have reformatted our functionality annotations to also reflect user intents, such as “go to the link element that opens a pop-up window for log-in”. These reformatted samples were added to the AutoGUI702k dataset. Row 4 shows that aligning our functionality REs more closely with the RefExp format helps to boost performance on this benchmark.

In summary, the AutoGUI data demonstrates clear benefits compared with other datasets and we can adjust the functionality annotation format to reduce a potential domain gap.

Q5: Is it easy to scale AutoGUI to new platforms?

A: Yes, it is easy if we can obtain the source code of GUI layouts. For example, one can collect and annotate GUIs on Windows using pywinauto, and iOS using Xcode Accessibility Inspector.

---

Q6: Is it easy to scale AutoGUI to multiple languages?

A: Yes, scaling AutoGUI to multiple languages is straightforward when using LLMs with reliable multilingual capabilities. In our pipeline, we utilize Llama-3-70B, which supports several languages, including Chinese, German, Russian, and Spanish. For example, in the AutoGUI702k dataset, a GUI displayed a SAMSUNG login page in traditional Chinese before interaction, and a Google login page afterward. The LLM translated the Chinese text into English as part of its Chain-of-Thought reasoning, ultimately deducing that the element was used for logging in with Google.

---

Q7: Have the authors analyzed the overlap between train and test data to avoid any contamination?

A: Yes, we have. Since our focus is on annotating contextual functionality for GUI elements, we define two elements as distinct if they serve different functions within their respective contexts. For example, two "magnifier" buttons on the same GUI might have different roles—one for zooming in and the other for searching. To ensure no contamination, we investigated whether the test elements appeared in the training set by checking if bounding box overlapping occurred on the same GUIs. After this analysis, we found no such overlap.

---

Q8: How does the proposed dataset compare with the other datasets of Table 1 in terms of performance? Are results substantially better in comparison to the same models finetuned on other datasets?

A: The results are better on most of the benchmarks. We kindly ask the reviewer to refer to Q4 for analysis.

---

Q9: How does it affect the resolution of the input image when solving this task for VLMs?

A: We are sorry that we do not quite understand this question. We guess the reviewer is concerned about how input image resolution impacts GUI grounding performance. While this issue is more closely related to model design, we conducted an ablation study to investigate the effect of input image resolution using the SLiME and Qwen2-VL-72B models. Both models employ an image division strategy, where high-resolution screenshots are split into patches and encoded into a visual feature sequence for input to the LLM.

We evaluated three input resolution configurations:

(1) Vision Encoder’s Native Resolution: We disabled the division strategy and resized the input screenshot to the size of a single patch (336x336 for SLiME and 224x224 for Qwen2-VL).

(2) Resized Longest Dimension: We resized the input screenshot to a longest dimension of 644, maintaining the aspect ratio.

(3) No Resizing: The longest dimension of the GUI screenshot, typically greater than 1000, was retained.

The results below show that input resolution significantly affects GUI grounding performance, with higher resolutions yielding improved accuracy. This is expected, as GUI content is often dense and complex, requiring models to process fine details for accurate grounding.

---

References:

[1] https://github.com/pywinauto/pywinauto

[2] https://developer.apple.com/documentation/accessibility/accessibility-inspector

---

We thank the reviewer again for the insightful review and feedback! Hope we have addressed your concerns.

Thank you for the insightful review and feedback. Please see below for responses to your concerns.

---

Q1: Any analysis of the fine-tuned model’s generalization across diverse UI types and applications?

A: We calculate the grounding metric values for the three platforms used in the ScreenSpot (other benchmarks focus either exclusively on mobile or web scenarios and are not suitable for analysis). See the table below:

The fine-tuned models, as well as SeeClick, show lower scores on the desktop and web splits. After examining failure cases, we identified two main factors contributing to this:

1) Content richness: The web split contains samples with higher content richness than the mobile split. After downsizing in the preprocessing stage of the models (Qwen receives 448x448 and SLiME receives at most 1344 x 1344), the content becomes more obscure and distorted, likely leading to grounding difficulty.

2) Data absence: Collecting desktop data presents additional challenges due to the complexity of installing various software and capturing their UI states.

Despite these challenges, table 4 in the main paper shows that the fine-tuned models still demonstrate notable scalability across mobile-related benchmarks (MOTIF and RefExp) and web-related ones (VWB), suggesting a degree of generalization. We believe that the community can leverage our automated annotation pipeline to enrich our dataset, thereby enhancing the model's robustness and performance across various platforms.

---

Q2. Can LLM-based processes alone consistently maintain high-quality annotations as the dataset scales?

A: We recognize the importance of maintaining high-quality annotations and would like to clarify that the efficacy of LLM-based annotation and verification has been substantiated in several studies [1-5]. These studies either justify the high quality of LLM-generated annotations or the effectiveness of LLM as a judge.

In our work, the primary GUI data sources include the top 200 most visited domains (Line 946 in Sec. A.2) and commonly used Android built-in apps (Line 966), covering the GUIs frequently encountered in daily use. We develop the AutoGUI annotation pipeline towards these prevailing GUIs and guarantee data quality on these GUI distributions via human verification (Section 4).

However, we acknowledge that no single annotation method can perfectly generalize across all types of GUI data. Nevertheless, the AutoGUI pipeline is designed to be adaptable, allowing for the integration of domain-specific rules for annotation, rejection, and verification as needed. We believe the community can continue to extend our pipeline to enhance both scalability and quality.

---

References:

[1] CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation, EMNLP 2024

[2] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NIPS 2023

[3] TarGEN: Targeted Data Generation with Large Language Models, COLM 2024

[4] Is a large language model a good annotator for event extraction, AAAI 2024

[5] Making large language models better data creator, EMNLP 2023

---

We thank the reviewer again for the insightful review and feedback! Hope we have addressed your concerns.

Dear Reviewer dSda:

We sincerely thank you for these constructive comments and evaluation of our paper. With the ICLR public discussion phase ending in two days, we kindly ask you to take a look at our responses. Our rebuttal also provided additional experiments in response to your concerns. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.