VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

We appreciate that the reviewer recognizes our benchmark as a valuable resource for evaluating multimodal language models in webpage understanding!

Bias Introduced by GPT-4V: We considered the potential bias introduced by GPT-4V during our data annotation process. Consequently, we have undertaken further data verification by human annotators as detailed in Appendix B. For the action grounding task, as described in Appendix A, annotators were encouraged to write their own instructions. Ultimately, 80% of annotated action descriptions are different from the ones initially generated by GPT-4V.

Unique Screenshot Numbers: The total number of unique screenshots is 332, instead of 139. While captioning, heading OCR, and action grounding tasks generate one sample per website, other subtasks construct multiple samples for each website.

Related Work: We discussed the related work that relies on the DOM/source code to carry out tasks on web pages in Introduction and Section 2.2 (e.g., web agent benchmarks like MiniWob++, WebShop, Mind2Web, and WebArena). The mentioned work DocLLM, focuses on documents and tables understanding rather than web pages, which clearly has different goals from our work. We will include the comparison between DocLLM and our WebBench in the revised version.

Automatically Extracted h1 Tag: As indicated in Figure 3, only 46 out of 139 samples are properly displayed in screenshots and can be considered good headings for web pages. This underscores the necessity of our human verification process.

Results of Gemini Pro in Figures 4 and 6: We have uploaded the updated figures to the anonymous link. (https://postimg.cc/gallery/F0yRZ0f).

Model-Based Metrics for Open-Ended Generation Tasks: Given that model-based metrics may introduce uninterpretable bias and additional computational cost, we prefer lightweight and rule-based evaluation metrics. For instance, for the WebQA task, annotators are instructed to propose objective questions with short answers (less than five words in most cases), making SQUAD-F1 the appropriate metric.

We appreciate the reviewer’s recognition of our efforts to break down the challenges in web understanding across a wide range of models. Below, we add the results of Gemini 1.5 Pro and Claude 3 Opus. Overall, Gemini 1.5 Pro performs the best on the majority of the subtasks and achieves the highest average score. Claude Opus shows exceptional performance on Element OCR. We will include the results and the analysis in our revision!

We appreciate that the reviewer finds our benchmark well-designed, the experiments comprehensive, and the analyses insightful.

Differences from Existing Datasets: 1) As described in the third paragraph of the introduction, web-agent benchmarks like Webshop [1] focus on end-to-end capabilities without a fine-grained assessment of specific skills such as OCR, semantic understanding, and grounding, which was also mentioned by Reviewer f5BP (thanks!). 2) The WikiWeb2M dataset [2] includes only three captioning tasks on a webpage, and the WIT dataset [3] focuses on image-text retrieval. Additionally, both datasets are limited to Wikipedia web pages and do not reflect the diverse scenarios web agents encounter. In contrast, WebBench spans 139 most popular websites, covering 87 sub-domains and encompassing seven different sub-tasks, including captioning, OCR, web QA, and element or action grounding.

Noise in Task Annotation: As detailed in Appendix B, the human annotators undertook meticulous efforts to curate and verify all subtasks carefully. For the element grounding subtask, we identified and removed the problematic cases, including those noted by the reviewer.

Not that many samples per task: Our dataset contains multiple examples of different subtasks within a single screenshot. We include more screenshots rather than annotating numerous examples within the same screenshot to ensure comprehensive coverage of diverse real-world scenarios. Consequently, our dataset spans 139 of the most popular websites across 87 sub-domains. Notably, several widely adopted multimodal datasets prioritize diversity over quantity, such as MMVet with 218 samples and MME with 2374 samples $4$

$5$ .

[4] Yu, Weihao, et al. "Mm-vet: Evaluating large multimodal models for integrated capabilities." International Conference on Machine Learning, 2024, PMLR.

[5] Fu, Chaoyou, et al. "MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models." 2023, arXiv, eprint 2306.13394.

Calculation Detail of Correlations Between Subtasks in Figure 5: From the results in Table 1, we have seven sets of metrics (columns), and the Pearson correlation coefficient is calculated between every two columns.

Typos: We will correct the typos in the revised version.

We appreciate the reviewer's recognition of our benchmark as high-quality, clearly defined, and well-motivated!

Metric for OCR Tasks: We reported the F1 value of ROUGE-L because the average ground-truth length in our dataset is longer than that of common scene OCR datasets (e.g., Element OCR averages 39.5 words). Consequently, the longest common subsequence may be more appropriate than exact match for evaluation. For reference, we also provide exact match (EM), word-level F1, and ROUGE-L results below. The EM metric is overly strict and does not accurately reflect OCR capabilities, while word-level F1 is similar to ROUGE-L. For simplicity, we will continue to use ROUGE-L as the evaluation metric for the two OCR tasks, and we will include the following table in the appendix.

Annotator Details: For the WebQA task, two co-authors carefully examined each screenshot and formulated questions along with corresponding answers. For the Action Grounding task, the annotation process was carried out by seven experienced researchers, including four co-authors. We have included the annotator IDs for each sample in the anonymous dataset (see the link below). Additionally, two co-authors conducted the final verification for all subtasks.

Annotation Pipeline for Other Subtasks: Thanks for the catch! All subtasks involved human annotation as described in Appendix B. Since the annotation processes for the other subtasks were relatively straightforward, we did not provide extensive details. We will expand Section B to clarify the annotation processes for all tasks.

Open-Source Raw Data: Yes! We plan to make all resources open-source. This includes URLs, access dates, raw crawled data, final data samples for different tasks, the code for rendering websites, and the inference code for evaluation. These resources have been uploaded to an anonymous repository for preview (https://anonymous.4open.science/r/Anonymous_WebBench-E7D4) and will be publicly released.