GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding

Thank you very much for your valuable feedback. We apologize for any confusion caused by certain details in the paper. We will address each of your concerns and provide explanations to help you better understand the contributions of this paper step by step:

Q1: Could the authors provide further justification for the novelty of the model and training method?

A1: Thank you for your critique. Our innovation in our training methodology is the two-step training pipeline. As shown in Figure 8, this approach—first training on video captions before progressing to complex reasoning or conversation tasks—proves more effective than training directly on mixed data. We would like to emphasize that our primary contribution lies in proposing a GUI-oriented dataset and benchmark. We utilized existing models for training primarily to validate the effectiveness of our dataset, rather than modifying model architectures to demonstrate novelty. We selected VideoChat2 because it achieved SOTA performance at the time of our research; LLaVA-OV was released in August.

Q2: Could the authors include the missing results in Table 1, specifically"H" for Gemini-Pro-1.5 and "R" and "E" for GPT-4o?

A2: Thank you for this valuable feedback. We have expanded our experimental evaluation to include both Gemini and GPT-4, with detailed results presented in Tables 1, 2, and 3. In response to your comments, we have substantially revised this section of the manuscript. The keyframe identifier is now framed as an exploratory experiment, with all main results utilizing human evaluation to simulate realistic scenarios of model perception and operation. We have also incorporated two additional baselines for keyframe selection, termed 'model-based' approaches, drawing from the embodied AI domain. For clarity, we have renamed the 'Extract' component to 'Program.' For a comprehensive view of our keyframe selection methodology comparisons, we direct you to Tables 3 and 6 in our revised manuscript.

Table 1: Gemini-Pro-1.5 performance with human extracted keyframe.

Table 2: GPT-4o performance with randomly extracted keyframes.

Table 3: GPT-4o performance with programly (formerly "Extracted") extracted keyframes.

Thank you very much for your valuable feedback. We apologize for any confusion caused by certain details in the paper. We will address each of your concerns and provide explanations to help you better understand the contributions of this paper step by step:

Q1: Provide explicit statements that indicate the significance of each of your contributions.

A1: We thank the reviewers for the opportunity to clarify the significance of our contributions. Below, we address the question regarding the impact and novelty of each contribution in our work:

1. Development of GUI-WORLD Dataset:

   GUI-WORLD is the first dataset specifically designed for instruction-tuning in video-based GUI domains. It goes beyond static environments, addressing dynamic and sequential GUI tasks. Spanning 12,379 videos across six diverse scenarios (e.g., desktop, mobile, multi-window, and XR), it bridges a critical gap in GUI-oriented MLLM research, enabling the evaluation of dynamic tasks not covered in prior datasets like WebArena. This dataset can serve as both VideoLLM pretraining and instruction tuning for enhancing the GUI-related capability.

2. Benchmarking Multimodal LLMs:

   Our comprehensive benchmarks evaluate nine state-of-the-art MLLMs, revealing their limitations in handling dynamic and sequential GUI tasks. These results emphasize that even advanced models struggle with GUI comprehension without enhancements such as keyframe selection or textual context integration. By providing this benchmark, we establish a baseline for future research in GUI-oriented MLLMs.

3. Proposal and Implementation of GUI-Vid:

   GUI-Vid is the first fine-tuned video-based GUI agent tailored for complex GUI tasks. It achieves a 30% performance improvement over baseline models, surpassing open-source video LLMs and achieving results comparable to leading commercial vision LLMs like GPT-4V. GUI-Vid demonstrates significant improvements in dynamic and sequential GUI comprehension, showcasing the utility of GUI-WORLD.

4. Insights into GUI Challenges:

   Our analysis highlights actionable factors—such as keyframe resolution, the integration of textual information, and improved vision perception—that directly affect model performance. For example, we demonstrate that models leveraging higher-resolution frames perform better, suggesting clear pathways for researchers to optimize future systems.

5. Implications for Future Research:

   GUI-WORLD and our findings establish a foundation for further exploration of GUI-oriented MLLMs in emerging areas like multi-window and XR environments. By identifying existing challenges and proposing GUI-Vid as a prototype solution, our work invites the community to tackle these limitations and improve multimodal LLMs' real-world applicability.

Q2: Clarify whether the data collection pipeline itself is a contribution.

A2: At the time of our paper writing and submission, works like GUICourse and VideoGUI had not yet been published. Therefore, our claim of novelty is temporally valid in two aspects:

1. We were the first to develop a pipeline that crawls GUI tutorial videos from YouTube and annotates mouse and keyboard operations within them to create a GUI video dataset.
2. We developed specialized software tools and quality assessment metrics for recording and annotating human interactions with GUI videos. This systematic approach to data collection and validation represents a significant technical contribution.

Our data collection pipeline thus constitutes a meaningful contribution to the field, as it introduced novel methodologies and tools for creating high-quality GUI interaction datasets. The timing of our submission predates similar works in this space, establishing our pipeline's originality in the research timeline.

Q3: Add appropriate citations for relevant work that you are building on and explicitly state how the methodology presented here is different.

A3: Thank you for bringing this to our attention. We have updated both our tables and the related work section to include citations and discussions of all the relevant works you mentioned. Our key differentiation can be summarized as follows: we are the first to extend GUI understanding tasks into the video domain, and we demonstrate the critical importance of visual dynamic context contained within videos for comprehending GUI-related tasks.

Q4: Add additional commentary on the "smaller contributions" that as noted under "Weaknesses".

A4: Thank you for your valuable suggestions! We have thoroughly restructured our manuscript by providing deeper insights through expanded analysis of model failure cases. Additionally, following Reviewer iEU7's recommendation, we have reorganized the paper structure by presenting our proposed model after the benchmark section, which enhances the overall clarity and flow of the manuscript.

Thank you very much for your valuable feedback. During this interval, we tried our best to address all your concerns and revised our paper based on your advice. We will address each of your concerns and provide explanations to help you better our latest update of this paper step by step:

Q1.1: Uneven presentation of results/tasks in categorizations.

A1.1: Thank you for your suggestion. We will reorganize our evaluation framework in the main text into three categories: Caption, Complex Tasks, and Conversation. The Caption category includes both simple and detailed captions, testing the model's basic and comprehensive understanding of video content. Complex Tasks encompasses Static analysis, Dynamic analysis, and Reasoning, evaluating the model's ability to interpret GUI elements and make logical inferences based on video content. The Conversation category examines the model's capability to provide helpful responses and maintain coherent multi-turn dialogues with users - a crucial skill for any chatbot. This three-tier framework effectively covers fundamental comprehension, reasoning capabilities, and knowledge expression skills.

Q1.2: Add a table with definition for the major categorizations.

A1.2: Thank you for your suggestion. Given the diverse presentation formats in our dataset, we will include a table to clearly illustrate the categorization method used for each experimental section. Our modified classification in our latest manuscript follows two frameworks: when categorized by scenario, we divide tasks into MCQA and free-form; when categorized by task type, we separate them into caption, complex, and conversation tasks. We did not include MCQA in our final scope as our research primarily focuses on free-form questions.

We provide a clear summary of all tables in our experiments in Table 1. You can also view this in Table 10 in our revised manuscript.

Table 1: Summary of our table, objective and category (view axis).

Q1.3: Include an example metadata record (or at least the schema) for one or more videos.

A1.3: Thank you for your suggestion. We will create and upload demonstration videos to our project homepage, which will help the community better understand our dataset construction methodology.

Q1.4: Why does Table 5 not have results for MCQA tasks? Could the authors shed a bit more light on their choices for how to organize these results?

A1.4: Thank you for your suggestion. We need to address an error in Table 5 where we incorrectly labeled Sequential Task as Dynamic task. To clarify, Dynamic tasks actually encompass both Sequential and Prediction tasks, and we will average these two columns to present the correct Dynamic task results. Regarding the MCQA omission, the table specifically focuses on breaking down free-form questions to calculate their average performance. While this was our intended approach, we acknowledge that we failed to clearly communicate this rationale. We apologize for this lack of clarity and have already corrected this explanation in our latest manuscript.

Q2: Could the authors briefly elaborate on human-MLLM collaboration for generating QA pairs?

A2: Thank you for your question. Our process consists of three main phases. First, we gather comprehensive video information including keyframes, actions performed at each keyframe, and annotators' descriptions of action purposes (e.g., "scrolling down to view more content"). We also input global video context to GPT-4V, such as the operating platform (Mac, Windows) and an overall operation summary (e.g., "opening a browser, searching for neurips, visiting the official website, and checking submission deadlines"). We prompt GPT-4V to generate diverse QA pairs, captions, and conversation questions in batches, specifically instructing it to avoid repetitive questions to ensure variety. In the second phase, human annotators review and correct any inaccurate information in GPT-4V's generated content based on the video context. Annotators also enhance overly simple questions from GPT-4V to increase complexity. Finally, we conducted a quality assessment by sampling 1,000 data points for annotation. Our statistical analysis shows a 98% human satisfaction rate, confirming the high quality of our dataset.

Thank you very much for your prompt reply. We will address each of your concerns and provide explanations to help you better our latest update of this paper step by step:

A1.1/1.2: Thanks for attempting to address the issue of categorization, I recognize its challenging in a dataset that varies along a number of axes. However I should say that I don't think the addition of Table 10 to be helpful, it introduces another categorization scheme (Task vs Scenario—what is the difference) and from a flow perspective, its not that helpful to have a table in the appendix explaining the tables in the main paper.

Re-A1.1/1.2: Task and Scenario is two primary axis that we consider most important to show the evaluation results. The task-specific analysis shows performance across different capabilities such as image captioning, complex QA, and conversation. The scenario-specific analysis evaluates performance across various application domains such as XR, iOS, software, etc. We also update the caption of Table 10 to avoid ambiguity. Due to space constraints in the main paper, Table 10 has to be placed in the appendix. We plan to integrate this table into the main content in the camera-ready version. Thanks for your advice!

A1.3: That isn't what I meant, I meant you should include an example metadata record (text) in the paper/appendix.

Re-A1.3: Got it. The following metadata record for one sample has been included in Figure 9 in the appendix. Thanks for your advice!

{
    "system": "Windows",
    "app": [
        "edge, bing, steam"
    ],
    "region": "partial",
    "goal": "View the submission interface for the dataset and benchmark track of nips 2024.",
    "keyframes": [
        {
            "frame": 32,
            "sub_goal": "Click to start downloading, restart downloading lethal company.",
            "mouse": "click",
            "keyboard": "none",
            "keyboardOperation": ""
        },
        {
            "frame": 176,
            "sub_goal": "Click on edge, edge returns to the top of the screen.",
            "mouse": "click",
            "keyboard": "none",
            "keyboardOperation": ""
        },
        {
            "frame": 781,
            "sub_goal": "Click on the hyperlink for dataset and benchmark, preparing to jump.",
            "mouse": "click",
            "keyboard": "none",
            "keyboardOperation": ""
        },
        {
            "frame": 839,
            "sub_goal": "Jump to openreview, loading.",
            "mouse": "click",
            "keyboard": "none",
            "keyboardOperation": ""
        },
        {
            "frame": 1079,
            "sub_goal": "The webpage loaded the submission interface for dataset and benchmark track.",
            "mouse": "none",
            "keyboard": "none",
            "keyboardOperation": ""
        },
        {
            "frame": 1131,
            "sub_goal": "Place the mouse on \"add a submission\"",
            "mouse": "hover",
            "keyboard": "none",
            "keyboardOperation": ""
        }
    ],
}

A1.4: Inconsistency between table results and textual analysis.

Re-A1.4: Sorry for misleading you in L371-373. After we combine Prediction and Sequential, we forget to revise the result analysis part. We’ve now revised the result analysis that match to table results.

A3: Misleading name of Random keyframe selection.

Re-A3: Sorry for the misleading setting name. To avoid confusion with the existing 'sequential' task, we have replaced the term Random with Linspace, as it better reflects our methodology of uniform sampling (inspired by np.linspace() function).

A7: Thank you for running this experiment, could you say more about why you picked MVBench given the focus of this dataset on GUI tasks? My current interpretation of this is fine-tuning on GUI-world makes the base model worse at MVBench, but does MVBench contain a relevant set of tasks that you think GUI-world should help performance on? Why not one of the datasets you mention in the introduction (Table 1)?

Re-A7: Sorry we misunderstand your point, our formerly intent is to show the general performance of our model in other video VQA task to show that out model is still usable for other video tasks. We’re running experiment on Mind2Web to further prove our dataset enhance GUI understanding.

A14: Copyright of Youtube video and advice on containing link for video.

Re-A14: Thank you for raising this point. All videos in the dataset are protected under Creative Commons licenses. For videos sourced from YouTube, direct reference links will be provided upon dataset release.

Thank you once again for taking the time to review our revised manuscript. Your detailed and valuable feedback has helped us improve both the clarity and quality of our work much better.

Thank you for your detailed reply. However, I may find myself unable to fully agree with all the points raised.

The authors define GUI Agents as two categories: (1) GUI assistants that help users interact with interfaces, as demonstrated in CogAgent, (2) agents that execute GUI operations through coding.

Unfortunately, this definition seems to present two issues:

1. Based on the paper and the reply, the authors appear to emphasize that a chatbot with strong GUI understanding capabilities is itself an agent. This interpretation diverges from the broader and more widely accepted definition of GUI agents within the field.
2. The authors split agents into "GUI assistants" and "agents that execute GUI operations through coding." However, GUI agents do not necessarily require coding capabilities. Even without the ability to write code (e.g., using PyAutoGUI or Playwright), LMs can still perform agent tasks via zero-shot or few-shot learning on benchmarks such as Mind2Web or AndroidControl through multi-choice question (MCQ) formats. (Similarly for grounding tasks through SoM prompting.) (And even by generating plans and actions in natural language with a human evaluation. [1])

Even the example provided in the response, CogAgent, undergoes large-scale pretraining followed by fine-tuning on web and Android agent tasks, and it has been comprehensively evaluated on benchmarks like AITW and Mind2Web. Moreover, CogAgent is capable of coding PyAutoGUI (see its results on OSWorld)

While I understand that improving GUI understanding has the potential to enhance a model's performance on agent tasks, the paper's title, "A Dataset for Agents," implies a stronger focus on how this dataset can directly or indirectly improve agent task performance. As a researcher in this field, I would be more interested in learning how to use this dataset to effectively boost agent task capabilities.

A common paradigm involves two training stages: (1) training on GUI understanding or related data, followed by (2) fine-tuning on downstream agent tasks. The CogAgent mentioned in the reply itself is already a good example. There are also many recent ones (for example, [2]).

Given the current the discussion, I would recommend revising the title to something like "for GUI Understanding", "GUI Assistance" to more accurately reflect the dataset's focus and applications.

[1] Zheng et al. "Gpt-4v (ision) is a generalist web agent, if grounded." ICML 2024. [2] Liu, et al. "Harnessing Webpage UIs for Text-Rich Visual Understanding." arXiv

Thank you very much for your valuable feedback. We apologize for any confusion caused by certain details in the paper. We will address each of your concerns and provide explanations to help you better understand the contributions of this paper step by step:

Q1: The paper does not clearly demonstrate how much this dataset contributes to actual GUI agent tasks.

A1: Thank you for your valuable feedback. We would like to first clarify the concept of GUI agents as discussed in our paper. In the field of GUI agents, there are two primary categories: (1) GUI assistants that help users interact with interfaces, as demonstrated in [1], and (2) agents that execute GUI operations through coding. Our paper specifically focuses on the first category.

We would like to emphasize our key contributions: We have made significant progress in the domain of video-based GUI agents by introducing two pioneering contributions:

1. The first video GUI caption & instruction & multi-round conversation dataset across six GUI scenarios
2. A finetuned GUI VideoLLM that achieves SOTA in GUI understanding.

Furthermore, we have validated the effectiveness of our datasets by achieving state-of-the-art performance on open-source VideoLLM implementations. This accomplishment demonstrates the practical value and robustness of our approach.

Our ultimate goal is to empower the open-source community to make meaningful contributions in the GUI domain and help bridge the gap with closed-source developments.

Q2: Can you show more experiments to address the issue in Weakness? Is it possible to demonstrate the capability of the model on traditional GUI grounding and agent benchmarks?

A2: As explicitly stated in Section 4.2 and Appendix A1, our agent is not designed for GUI grounding or code writing/execution tasks. The VideoLLM framework we employed has not been fine-tuned for code generation tasks and functions primarily as a GUI chat assistant. Moreover, our dataset does not include code generation tasks for fine-tuning. Consequently, our model does not possess operational task capabilities - a limitation we clearly acknowledged in Section 4.2 (updated to Section 5.2) of our paper:

"However, in our explorative experiments, GUI-Vid still fails in GUI operating tasks via code generation like GPT-4 performing in Cradle [2], which is likely due to the baseline LLM's weaker performance and the challenges of code generation instruction fine-tuning."

We would like to emphasize that research value in the GUI agent domain extends beyond grounding and operational capabilities. Currently, open-source models still demonstrate limited understanding of GUI content, and this understanding is fundamental to developing agents capable of system operation. This precisely highlights the contribution of our dataset and benchmark, whose objective is to advance open-source VideoLLM development in the GUI domain to catch up on commercial models.

We appreciate your critical feedback and thank you for your thorough review of our paper.

[1] CogAgent: A Visual Language Model for GUI Agents

[2] Cradle: Empowering Foundation Agents Towards General Computer Control

Dear reviewer, we want to follow up regarding our response to your review. If there are any additional concerns or further clarifications needed, we’d be more than happy to provide additional information. We look forward to hearing from you.