Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms

Q4: Provide a comprehensive comparison to previous GUI datasets and analysis of your dataset is needed.

A4: We provide a comprehensive comparison to previous multi-platform datasets.

In addition, we discuss in detail and provide flowgram comparison on how the data pipeline of Ferret-UI is different from ours in Appendix C. Importantly, we highlight following key differences between Ferret-UI One and Ferret-UI:

• Multi-platform support: The training data of Ferret-UI One consists of 5 platforms compared to 2 platforms of Ferret-UI.

• Raw annotation qualities: The majority part of bounding boxes, labels and on-screen text annotations of Ferret-UI One is either extracted from source data or annotated by human, while these annotations of Ferret-UI are all generated from model detection.

• Bounding box prompting: When generating advanced tasks, Ferret-UI One uses GPT-4o+SoM visual prompting for bounding boxes, while Ferret-UI uses purely textual prompting containing coordinates, bbox labels and text annotations.

• Advanced task quality: Due to the constraint of textual prompting, Ferret-UI pipeline cannot perceive the visual content of UI elements, thus limiting the quality and content diversity of generated advanced tasks.

Q5: Case Studies and Failure Analysis: Include failure cases.

A5: We have added a section analyzing typical failure cases of Ferret-UI One in referring, grounding and navigation tasks in Appendix I. Typical referring and grounding failures include recognizing rendered text in images, wrongly recognizing widget-like objects contained in images, grounding widgets with blurry edges, requiring to performing too many navigation steps in one screenshot, and unseen navigation patterns in training set.

Q6: Discuss limitations

A6: (1) One major limitation of our current dataset is the simplified action space. Currently we mainly support some simple actions: tapping/clicking, typing and scrolling etc. More advanced actions, such as multi-finger swiping and zoom-in/out gestures are currently not supported. (2) Multi-lingual support: our model currently supports English-only GUI understanding. (3) Noise exists in advanced tasks generated by GPT-4o+SoM visual prompting when the bounding boxes are dense in the given screenshot, causing 14% average error rate when GPT-4o tries to refer to the correct bounding boxes. Finding efficient ways to eliminate noises in such large scale synthetic data is urgently needed.

Q7: A more detailed discussion of future direction is also suggested on topics of building more general agents across various platforms or how to improve the grounding or navigating capabilities of current models or agents.

A7: We discuss this in rebuttal to Q3 for reviewer #zXTq. To summarize, in order to make state-of-the-art UI agents useful in practice, extensive future work is necessary in (1). Improving the agents’ general completion rate. (2). Improving the agents' self-reflection and replanning capabilities. (3). A careful study of safety concerns.

We thank the reviewer for the detailed feedback and constructive suggestion. Below, we address key concerns and provide clarifications with additional experiments and analyses.

Q1: Extra Probing for Model's Potential: The experiment results are expected to be superior to Ferret-UI because Ferret-UI One is trained on more high-quality instruction tuning and task-specific data.

A1: Our contributions go beyond merely engineering improvements. While we acknowledge that Ferret-UI One builds upon its previous model, we introduce new scientific methodologies to overcome specific challenges in cross-platform UI understanding. Key advancements include:

• Adaptive N-gridding for enhanced high-resolution perception, which optimally balances local and global visual encoding across devices. This method allows Ferret-UI One to accurately perceive widgets on screens of various resolutions while minimizing distortion. Table 5 demonstrates the benefits of this technique.

• Closing the domain gap among different platforms: Another major challenge is the multi-platform integration of GUI understanding. We believe one of our contributions is finding that through curated data processing and uniformation, we can mitigate the domain gaps of GUI understanding across different platforms and successfully train a single model that well understands UI knowledge from different platforms.

Q2: Provide grounding test results on OS-world and VisualAgentBench.

A2: We want to clarify that our data pipeline is built based on Ferret-UI. We appreciate the suggestion to explore additional benchmarks such as OS-world and VisualAgentBench. However, our primary goal in this work was to demonstrate Ferret-UI One's effectiveness and generalizability across a diverse set of platforms and benchmarks, such as GUI-World, which is a comprehensive and challenging benchmark for multi-platform UI understanding tasks. The GUIDE is also a challenging benchmark for probing multi-step navigation potential under single-step settings by providing action history.
While OS-world and VisualAgentBench are valuable benchmarks, they focus on broader or slightly different aspects of UI interaction and may not align with the specific goals of our study. Specifically, VisualAgentBench emphasizes on real-time and multi-step navigation in simulation environments, while OS-world focuses on desktop which requires much broader knowledge of operating systems like Windows, Ubuntu and MacOS and is beyond our scope. We believe our chosen benchmarks adequately capture the core capabilities of Ferret-UI One, such as fine-grained grounding, referring, and robust cross-platform transfer.
We agree that the superior performance of Ferret-UI One compared to its predecessor, Ferret-UI, is partially due to high-quality instruction tuning and task-specific data. However, it is important to emphasize that Ferret-UI One also introduces significant innovations, such as adaptive high-resolution gridding and enhanced multi-platform training data generation, which contribute to its strong results. These advancements were specifically designed to address challenges inherent in multi-platform GUI understanding and generalization.
To further explore the model's potential, we have conducted additional analyses within the scope of our existing benchmarks. For instance, we include detailed case studies of grounding and navigation tasks, analyze failure cases, and evaluate cross-platform transferability. These analyses provide deeper insights into the model's performance and potential limitations, which we believe are more relevant and impactful for the current scope of the paper.

Q3: Evaluation and Analysis of the Synthetic Dataset: Provide quantitative analysis of GPT-4o's annotation quality and compare it with previous datasets.

A3: We conducted a quantitative evaluation of GPT-4o annotations by manually checking a randomly sampled subset consisting of 100 advanced task examples. The result shows that GPT-4o achieved an accuracy of 86% in terms of choosing correct bounding boxes from visual prompting, and achieved a 4.6/5 avg score on the quality of the textual answer.

We thank the reviewer for the thoughtful feedback and for acknowledging the strengths of our work. Below, we can address your concerns as follows:

Q1: A significant concern is the severe data imbalance across platforms, as evident in Table 1.

A1: We acknowledge the data imbalance, particularly between high-resource platforms (e.g., web, iPhone) and low-resource platforms (e.g., iPad, AppleTV). To mitigate this, we applied targeted approaches:

• Loss Weighting: During training, we assigned higher loss weights to lower-resource platforms like iPad and AppleTV based on their scale, encouraging the model to focus on these underrepresented data. We tried using balanced data to train the model, by cropping the data amount for platforms with excessive data, but found it not as effective as simply using the loss weighting to balance the training. Specifically, when balancing the example numbers from each platform to 66k, we find that the average performance is 1.5% lower than training with full scale data and loss weighting where the weight is inversely proportional to the task data scale.

• Advanced Task Generation: We supplemented low-resource platforms with additional advanced tasks generated from existing examples, using diverse prompts and varying visual contexts. In particular, for AppleTV and iPad data, for each screenshot, we generate data for all the 3 advanced tasks which equals to 3 training examples, while for other platforms we randomly generate data for only one advanced task for each screenshot equaling to 1 training example.

Q2: Can you provide detailed examples of failure cases, especially for low-resource platforms like AppleTV?

A2: We have added a section analyzing typical failure cases of Ferret-UI One in referring, grounding, and navigation tasks in Appendix I. Typical referring and grounding failures include recognizing rendered text in images, wrongly recognizing widget-like objects contained in images, grounding widgets with blurry edges, requiring performing too many navigation steps in one screenshot, and unseen navigation patterns in the training set.

Besides failure case study, we have also added new visual examples of our model across diverse platforms in the Appendix for visually understanding the capabilities of our model.

Q3: How does the quality of GPT-4o generated training data vary across platforms, and are there systematic differences in generation quality?

A3: We conducted a quantitative evaluation of GPT-4o annotations by manually checking a randomly sampled subset consisting of 100 examples, where each sample could be either advanced task we proposed. The result shows that GPT-4o achieved an accuracy of 85% in terms of choosing correct bounding boxes from visual prompting and achieved a 4.6 out of 5 human evaluation score on the quality of the textual answer. Since our cross-platform raw data is highly uniformed before advanced task generation with GPT-4o, we do not notice significant systematic differences in generation quality.

Q4: What are the inference time comparisons between platforms for Ferret-UI 1 ?

A4: We evaluate the inference speed of our model with different sizes in single-step inference setting(i.e. One screenshot and one QA each time), tested on one A100GPU with 500 samples. Result shows that our model with Gemma2B backbone uses 0.89s/frame, while the model with Llama8b backbone uses 1.29s/frame, and model with Vicuna-13B consumes 1.45s/frame. Since Ferret-UI One shares the same codebase with Ferret-UI, which only uses Vicuna-13B as backbone, we obtain a similar inference speed of 1.45s/frame for Ferret-UI.

Q5: The experimental design could be more rigorous. The cross-platform transfer results presented in Table 4 would benefit from error bars / error spread measures.

A5: We argue that it is not feasible to include error bars in Table 4 since it is a zero-shot test setting. In zero-shot testing, the model's performance is assessed without fine-tuning or training on the target task. As a result, there is no inherent variance from retraining or reinitializing the model, which typically contributes to error bars.

I appreciate the authors' detailed response addressing my concerns. The quantitative analysis showing the 1.5% performance trade-off with loss weighting versus balanced data sampling provides valuable insight into their design choices. The detailed inference timing benchmarks across different model sizes (0.89s/frame for Gemma2B to 1.45s/frame for Vicuna-13B) and the thorough failure case analysis in Appendix I add significant value to the study. I am raising my rating to 8 based on these clarifications.

Q3: What would you consider a "usable" performance of a model like Ferret-UI?

A3: We argue that all state-of-the-art UI agents show a low completion rate in UI navigation, which requires significant research and improvements to make them useful in practice.

To address your concern, we’ve built a UI agent on top of Ferret UI models for multi-step navigation, similar to CogAgent [2]. The agent can interact with UI screens to perform actions including tapping, swiping, and typing. In our preliminary study on multi-step navigation, we design a set of 100 online test episodes from across 20 first party apps and third party apps, all were evaluated on real iOS devices. Episodes covered a spectrum of use cases (e.g., managing notes in the Notes app, modifying a contact in the Contacts app, navigating in Maps, creating a calendar invite). To measure an agent's navigation performance accurately and repeatably, we build an online evaluation framework with careful staging and evaluation.

We evaluate our UI agent against the GPT4o agent with set of mark prompting [3] by task completion rate. A navigation result is rated by three independent annotators, and the averaged results are shown in the following table. Our UI agent achieves a 53.2% completion rate and outperform GPT4 agent by 10%. This shows the great potential of building UI agents on top of strong UI understanding models. On the other hand, we also acknowledge that the achieved 53.2% completion rate is not satisfactory enough. Similar low completion rate has also been found in WebArena (WebRL[4]), with only 40% completion rate.

Though out-of-the-scope of the Ferret UI One paper, we conducted an error analysis on several agent models on UI navigation tasks, including state-of-the-art CogAgent [2], GPT4-SoM agent [3], and the agent model built on top of Ferret UI. We identified that mistakes in planning, rather than action grounding, account for the majority of navigation failures in all models, a similar conclusion can be found in UIGround [5] as well. Specifically, all agents lack replanning and self-reflection capabilities in challenging scenarios, which require in-depth app exploration. Hallucinations may occur when finding hidden UI elements.

Besides, the agents may make dangerous mistakes, such as deleting contacts, or leaving unexpected comments. Developing a holistic guardrail to prevent agents from risky actions is an important future step for safe UI navigation, which has been only recently explored in a preliminary study [6].

To summarize, in order to make state-of-the-art UI agents useful in practice, extensive future work is necessary in (1) improving the agents’ general completion rate; (2) improving the agents' self-reflection and replanning capabilities; (3) a careful study of safety concerns. What’s more, Ferret-UI One is not a model specifically made for navigation but focuses on understanding and single-step interaction. Besides navigation, the model can also be used in other applications such as accessibility, automated testing and so on.

Q4: Ethics Concerns

A4: We would like to argue that our name choice is driven by technical claity rather than the affiliation. Furthermore, our paper stands out disregarding its original as discussed in the scientific contribution point. There are many papers built on previous work with the same conventions, such as LLaVA-1.5[7] and SAM 2[8].

References:

[1] You, Keen, et al. "Ferret-ui: Grounded mobile ui understanding with multimodal llms." European Conference on Computer Vision. Springer, Cham, 2025.

[2] Hong, Wenyi, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang et al. "Cogagent: A visual language model for gui agents." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290. 2024.

[3] Yan, An, et al. "Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation." arXiv preprint arXiv:2311.07562 (2023).

[4] Qi, Zehan, et al. "WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning." arXiv preprint arXiv:2411.02337 (2024).

[5] Gou, Boyu, et al. "Navigating the digital world as humans do: Universal visual grounding for gui agents." arXiv preprint arXiv:2410.05243 (2024).

[6] Zhang, Zhuohao Jerry, et al. "From interaction to impact: Towards safer ai agents through understanding and evaluating ui operation impacts." arXiv preprint arXiv:2410.09006 (2024).

[7] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[8] Ravi, Nikhila, et al. "Sam 2: Segment anything in images and videos." arXiv preprint arXiv:2408.00714 (2024).

We thank the reviewer for the thoughtful feedback and for acknowledging the strengths of our work, particularly the model's clarity, writing quality, and performance over prior work. Below, we respond to specific points raised in the review.

Q1: What do you think are the scientific contributions of this paper? How much of it is just good engineering of tested ideas?

A1: Our contributions go beyond merely engineering improvements. While we acknowledge that Ferret-UI One builds upon its previous model, we introduce new scientific methodologies to overcome specific challenges in cross-platform UI understanding. Key advancements include:

• Adaptive N-gridding: Enhancing high-resolution perception, which optimally balances local and global visual encoding across devices. This method allows Ferret-UI One to accurately perceive widgets on screens of various resolutions while minimizing distortion.

• Closing the domain gap among different platforms: Another major challenge is the multi-platform integration of GUI understanding. We believe one of our contributions is finding that through curated data processing and uniformation, we can mitigate the domain gaps of GUI understanding across different platforms and successfully train a single model that well understands UI knowledge from different platforms. Specifically,

  • Table 2 shows significant improvements over previous methods (89.73 GPT-4o score vs 45.81 for Ferret-UI)
  • Table 3 demonstrates strong zero-shot performance on GUI-World benchmark
  • Table 4 shows systematic analysis of zero-shot transfer capabilities between platforms. The results reveal important patterns about UI knowledge transfer: Strong transfer between iPhone/iPad/Android (>65% performance); Limited transfer from AppleTV to mobile platforms; Platform similarity in resolution and aspect ratio impacts transfer success.

While the paper does include significant engineering work, the core scientific contributions around adaptive N-gridding and cross-platform learning advance our theoretical understanding of UI comprehension, which hits a balance between novel insights and engineering implementation. We also argue that its predecessor model, namely the Ferret-UI, has been peer-reviewed and published in ECCV 2024 [1].

Q2: Since the authors fail to disclose whether the model is going to be available, and how, it is hard to evaluate the impact and relevance for the community of the work described here.

A2: We are actively exploring avenues for sharing Ferret-UI One with the research community while ensuring compliance with legal and organizational constraints. Particularly, its predecessor models, Ferret-UI, and relevant code have already been made available on Github, and we also plan to release our model.

Based on the answers from the authors, I still think that this work is mostly engineering improvement of the previous, non-reviewed work.

It seems that the authors agree with my assessment that the current accuracy is not enough to make it useful, what is not said in the paper. I am not sure whether the authors are planning to do so and, by doing it, lowering their claims.

It is good to know that they are planning to make it available.

Considering that it is mostly an engineering paper which provides incremental gains but not useful results, I believe my rating is appropriate.