Harnessing Webpage UIs for Text-Rich Visual Understanding

We appreciate the constructive feedback and have carefully reviewed the concerns raised about the quality assessment of the generated data and labels in the MultiUI dataset. Below, we provide clarifications and improvements to address these points.

Discussions of Data Quality

To ensure the quality of the generated data and labels in MultiUI, we conducted a human study on 100 randomly selected samples. The results indicate that:

• 97% of the samples are accurate, with only 3% containing position errors, which we believe do not significantly impact overall data quality.
• Potential issues such as hallucinations or mislabeling may exist; however, these are mitigated by the robust construction pipeline leveraging accessibility trees and LLM-generated instructions for filtering and labeling.

More importantly, the experimental results on 19 downstream tasks strongly demonstrate the dataset's effectiveness. Models trained on MultiUI achieved significant performance gains across diverse benchmarks. This highlights the reliability of MultiUI for downstream tasks and confirms its practical utility despite any minor imperfections.

Assurance of Data Quality in Construction

Our pipeline includes multiple safeguards to ensure the quality of data points:

1. Accessibility Trees: Leveraging accessibility trees significantly reduced noise compared to traditional rule-based or raw HTML parsing approaches.
2. Multi-Step Curation Process: Content filtering is conducted via the Llama-3-70B-Instruct model, ensuring only valid and contextually relevant data is included.
3. Task-Specific Heuristics: Heuristics applied during task extraction eliminated ambiguous or low-information samples.
4. Downstream Task Performance: To verify alignment, we tested a range of models fine-tuned on MultiUI and observed consistent performance improvements across benchmarks.

Thanks for your response. It resolves most of my concerns. As a result, I will keep my positive rating and increase my confidence score.

Thank you for your thoughtful review. We’re glad you found our dataset relevant, our methodology clear, and our results strong!

Construction Process of Grounding Task Samples

• Action grounding task:
  Given the accessibility tree of a webpage where each UI element is accompanied by its bounding box, Llama is instructed to:

  1. Select an element (e.g., a button to navigate to the next page) on the webpage.
  2. Generate an instruction (e.g., "change to next page") that targets the selected element.
     The resulting <instruction, bounding box> pair is collected as a sample for the action grounding task.
     The prompt used for Llama is detailed in Appendix C.

• Element grounding task:

  • For text elements (e.g., <p>, <h> tags), element descriptions are their inner text within the HTML.
  • For icon/button elements, descriptions are extracted from the “aria-text” attribute of the corresponding tag.

Adding GPT-4o Performance in Table 2

We have updated Table 2 to include GPT-4o.

• Notably, GPT-4o only shows a marginal improvement over GPT-4V on the GUI grounding benchmark (ScreenSpot).

View in Color

Thank you for the suggestion! We have updated all figures for better color visualization.

Ablation Study of Grounding Data

• Table 1 highlights that MultiUI includes 526K embedded image captions & QA data, where a significant portion of the embedded images are natural images (e.g., photos of landscapes, buildings, animals, and plants).
• This dataset enhances the model’s understanding of natural images, leading to improved performance on RefCOCO+.
• This analysis highlights the gap between GUI/Doc/Chart/OCR images and natural images, which we have explicitly discussed in the updated version.

ScreenSpot Results of SeeClick and CogAgent

Thank you for the reminder! We have added these results to Table 2:

• Our UIX-Qwen2-7B outperforms both SeeClick and CogAgent, despite being smaller in model size (7B vs. SeeClick: 9.6B, CogAgent: 18B).

Implementation of High-Resolution Strategy

• The "maximum resolution" column in Appendix E refers to the total resolution of multiple tiles.
• We follow the dynamic resolution strategy described in LLaVA-NeXT [1].

Test Splits

We have added the test splits for each benchmark in Appendix G of the revised version.

[1] LLaVA-NeXT Blog (2024-01-30)

Thank you for maintaining your positive assessment of our work! We sincerely express our gratitude for your constructive comments and suggestions!

Best, Authors

Thank you for recognizing MultiUI as a diverse dataset for enhancing multimodal understanding and its significant impact on improving performance across various text-rich visual tasks. We appreciate your acknowledgment of its importance in advancing this area!

Documentation and Code Guide

We have provided an anonymous GitHub repository linked on the first page, which includes:

• README
• Example data
• Training & evaluation code

In response to the reviewer's suggestion, we have added:

• More detailed documentation for the example data.
• A clearer code guide.

Thank you for the valuable suggestion!

Relying on Llama and LLaVA

• As shown in Figure 2, the construction pipeline of MultiUI does not rely on LLaVA.
• To evaluate the effectiveness of MultiUI, we conducted experiments across 19 diverse benchmarks, covering:
  • GUI-related
  • OCR/Doc/Chart
  • General domains

Detailed results are provided in: Table 2, Table 3, Table 4, and Table 5.

These comprehensive evaluations robustly demonstrate the effectiveness and versatility of models trained on MultiUI.

To further demonstrate the quality of our dataset:

• We conducted a human study on 100 randomly selected samples from MultiUI.
• A minor issue was identified: 3 samples contained position errors.
• However, this minor issue does not significantly affect the overall data quality.

Thank you again for your thoughtful feedback, which has helped us further improve the clarity and presentation of our work!

Fine-Tuning Experiments on Similar Datasets

We considered relevant datasets for specific scenarios like document understanding (e.g., DocVQA, ChartQA) and GUI-related tasks (e.g., SeeClick, CogAgent). However:

• To our knowledge, MultiUI is the first automatically created dataset constructed from web UIs, benefitting diverse scenarios.

• While we didn’t directly fine-tune models on these datasets due to limited computing costs, we compared our trained models with:

  1. Document understanding models (e.g., DocOwl-1.5, DocOwl-2 in Table 3).
  2. GUI grounding models (e.g., SeeClick, CogAgent in Table 2).

We found that UIX-Qwen2 outperformed specialized models on both document-related (e.g., DocVQA, ChartQA) and GUI-related (e.g., VisualWebBench, ScreenSpot) benchmarks.

Quality of Website Curation

The employed URLs are taken from FineWeb [1], which involves:

• Filtering out malicious/NSFW websites.
• Additional filtering based on the accessibility tree, which simplifies webpage content.

We conducted a human inspection on 300 randomly selected samples and found no objectionable content, such as adult material, gambling, or violence.

Data Decontamination

We considered data decontamination during the construction process. As described in Section 2.1, we:

• Removed all URLs appearing in downstream benchmarks (ratio: <1%).

Discussions

Impact of Image Resolution

Webpage images typically have higher resolutions (e.g., 1280 × 1280) compared to natural images (e.g., COCO). We hypothesize that higher resolutions yield better performance on web UIs. Our experiments with the LLaVA-1.5 architecture revealed:

• Smaller resolutions (336 × 336) hinder performance due to loss of detail.
• Doubling resolution (672 × 672) led to notable improvements while balancing training costs.

Comparison Among Different LLaVA Variants

As described in Section 3.1 and Appendix E, we conducted experiments on three LLaVA variants:

1. UIX-Vicuna
2. UIX-Llama3.1
3. UIX-Qwen2

Key findings:

• Training Llama3.1 was unstable.
• Qwen2 was used as the primary base LM.
• Results varied based on training data (e.g., improvements on DocVQA were more significant with LLaVA-1.5 data than with LLaVA-NeXT data).

References

[1] The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.

Thank you for your thoughtful review and for recognizing the strengths of our work, including the clarity of our paper, the validity of our data pipeline, and the value of the MultiUI dataset for the community. We appreciate your positive feedback on our contributions and experimental results.

Comparison with Other Datasets

We have added a table to compare the proposed MultiUI with some relevant datasets in Appendix K. We can see that MultiUI not only covers most task types, including QA, OCR, grounding, captioning, and action prediction, but also contains most training samples. This highlights the broad applicability of the proposed MultiUI data for advancing text-rich visual understanding and grounding across various scenarios.

Table: Comparison among GUI-related datasets.

More Baseline Results (e.g., GPT4o or Claude 3.5, MolMo, Gemini, Llama3.2)

We have added results of GPT4o and Gemini 1.5 Pro into Table 2 and Table 3 in the revision version. We will also include results for Molmo and Llama 3.2 in the revised version.

Experiment Results on Other Backbone Architectures

Even though we adopted the LLaVA architecture, we explored different variants of the LLaVA models. Specifically, we experimented with:

1. Various base language models including Vicuna, Llama, and Qwen.
2. Different vision encoders such as CLIP and SigLIP.
3. Two versions of general visual instruction-tuning data: LLaVA-1.5 data and LLaVA-NeXT data.

As shown in Table 2 and Table 3, the proposed MultiUI dataset consistently leads to significant improvements across all LLaVA model variants.

Novelty

The contribution of our work is not only about scaling up. We propose to harvest diverse multimodal instructions of multiple task types from webpage UIs with the aid of text-based LLMs, which is not without challenges, such as:

• Handling lengthy raw HTML.
• Defining generalizable tasks from the web domain to other domains.
• Ensuring the diversity of synthesized instructions.

Alleviating Hallucination

We neither claim nor aim to completely solve hallucination, but we aim to alleviate it. Unlike vision-only models that rely on images, our method leverages HTML, which is:

1. Carefully written by human engineers/designers and corresponds precisely to the rendered webpage.
2. Processed into an accessibility axtree, containing no hallucinated elements theoretically.

However, we acknowledge that advanced text models may still exhibit some hallucination phenomena during data construction.

Thank you for maintaining your positive rating and for appreciating the additional experiments. We are glad that these efforts have clarified your concerns. We will add your mentioned results in the revised version.

Best, authors