An Illusion of Progress? Assessing the Current State of Web Agents

We appreciate that the reviewer recognizes the thorough error analysis and the significance of the benchmark and WebJudge method in advancing research on GUI agents. We thank the reviewer for their detailed feedback and constructive suggestions. Here, we treasure the opportunity to address your concerns and improve the quality of our work.

W1: The results of open-source models like UI-TARS.

Thanks for pointing this out. The results for open-source models are indeed very important, as they help identify shortcomings and guide future improvements. We find that UI-TARS(UI-TARS-1.5-7B) has already reported its performance on Online-Mind2Web[1]. It achieves a 75.8% success rate evaluated by WebJudge (GPT-4o), which even surpasses Operator. It’s also worth noting that UI-TARS-1.5-7B has achieved new state-of-the-art performance on several other benchmarks, such as OSWorld and AndroidWorld, demonstrating that it is indeed a very strong open-source agent model. We have been in contact with the UI-TARS team to validate their reported results.

W2: It will be nice if the WebJudge method can based on open-source model.

Thank you for the great suggestion! We synthesized training data from multiple models and trained a lightweight yet reliable WebJudge-7B, based on Qwen2.5-VL-7B. This model achieves a high agreement rate with human judgments (87%) and exhibits a small success rate gap of just 3.9% compared to human evaluation. Moreover, it generalizes well to out-of-distribution (OOD) benchmarks. Please refer to our general response for more details.

W3 and Q1: The maintenance of Online-Mind2Web.

We acknowledge that maintaining the leaderboard is challenging, but we are committed to putting in significant effort to ensure its long-term value. Please see our general response for more details.

Q2: The selection of agents.

This is a great question. In general, we selected well-known and widely recognized agents, especially those that claim to achieve very high success rates on WebVoyager, such as Browser-Use, to rigorously evaluate whether their performance still holds in real-world scenarios. We conducted a thorough literature review [2-4] and excluded agents that could not be evaluated (e.g., those that are not open-source or lack API acess). We ended up with the five agents for evaluation. We also considered including Google DeepMind's Project Mariner, but as of the submission time, we were unable to access its internal API, so it could not be included.

We want to note that we also evaluated Claude Computer Use 3.7 recently, a stronger and highly anticipated agent from Anthropic. Compared to version 3.5, it shows notable improvements, with stronger exploration capabilities and a tendency to produce longer trajectories, similar to Operator. However, it still falls short of Operator, achieving only a 56.3% (under human evaluation) success rate and struggling with more complex tasks.

We sincerely appreciate the constructive comments and thoughtful questions. We greatly appreciate your recognition of our rigorous experimentation, comprehensive analysis and effective automatic evaluation method. We will address your questions in the following paragraphs.

W1: The discussion of WebJudge’s limitations.

Thank you for the great suggestion! While WebJudge achieves higher agreement compared to existing methods, we acknowledge that it still has certain limitations. We have conducted a thorough analysis of 15 cases where WebJudge's evaluations deviate from human judgments and provide several insights that may inform future research on developing more reliable automatic evaluation methods. Among the 15 error cases, 5 involve missing task requirements despite identifying them as key points, 6 contain incorrectly generated key points, and the remaining 4 fall into other categories (e.g., assigning low scores to important screenshots, thereby missing key information, or failing to capture certain details within screenshots).

• Missing task requirements despite identifying them as key points. In some cases, WebJudge overlooks specific task requirements in the final outcome judgment phase, even when these requirements were explicitly identified as key points. For instance, if a task specifies searching houses with exactly two beds but the applied filter is “2+” beds (i.e., 2, 3, 4, or more), WebJudge may still deem the result correct. Similarly, when a task requires finding a document from the year 2023, WebJudge may ignore this time constraint and hallucinate that all requirements have been met, despite having correctly highlighted it as a key point. We suppose that this behavior may be due to the large input context in the outcome judgment phase, which includes numerous image tokens and generated summaries, potentially causing the model to overlook previously identified key points. We will add the discussions to the revised paper.
• The generated key points are not always accurate. The key points generated by WebJudge may not be accurate and sometimes include unnecessary constraints, resulting in overly strict evaluations. For instance, in a task that involves browsing upcoming superbike events, the key points may specify the need to apply an “upcoming” filter. However, if the website displays upcoming events by default and does not provide such a filter, WebJudge may incorrectly judge it as a failure due to the perceived absence of this filter. This issue arises because the key point generation phase only depends on the task description, without considering the website. A promising direction for future work is to make key point generation dynamic, allowing key points to be refined as the web agent explores the environment. As the agent gains a deeper understanding of the website, this could lead to more accurate key points.

Q1.1: Is it possible to use WebJudge to conduct fine-grained error analysis?

This is a very interesting question! Based on our experience, WebJudge can accurately identify which key points are not satisfied. We randomly reviewed 10 tasks to evaluate the accuracy of key point assessment and found that WebJudge achieves a high accuracy of 90.3%. This suggests the potential of using WebJudge for fine-grained analysis. However, as we discussed in W1, there are still some cases (i.e., 6 out of 15 cases) where the generated key points are inaccurate.

Q1.2: Aggregating individual key point labels for WebJudge.

Thank you for bringing up this interesting question! In fact, in the final outcome judgment stage of WebJudge, we explicitly prompt the model to evaluate each key point before deciding whether the task is complete(i.e., “Thoughts: your thoughts and reasoning process based on double-checking each key point and the evaluation criteria”). Therefore, in a sense, we currently require the model to verify each key point in a chain-of-thought manner before outputting the final outcome judgment (success or failure).

We sincerely appreciate the reviewer’s recognition of our contributions and acknowledgment that our experiments are clear and detailed. We address your questions in the following texts:

W1: "The paper is globally clear except that it does not adequately presents the subject."

Thanks for sharing this concern. In Appendix Table G.1, we provide a variety of example tasks from Online-Mind2Web, which can help readers who do not know web agents better understand our benchmark. Overall, our tasks are more diverse and closely aligned with real-world user needs. Completing these tasks requires web agents to browse and interact with websites through a series of actions, such as clicking and typing. Our benchmark includes not only information-seeking tasks (i.e., where agents interact with the web to find specific answers) but also contains tasks that require agents to perform real operations, such as adding items to a cart or modifying a delivery address. Regarding screenshots, we refer to web page image screenshots, not HTML-based representations. Thank you for pointing this out. We will clarify this further in the revision.

W2: "The static dataset fundamentally always outdated".

Thanks for bringing up this important issue. Indeed, updating tasks over time is both important and challenging. We already have a plan for updating tasks and maintaining the benchmark to ensure it remains valuable in the long term. Please refer to our general response for more details.

Q1: Release code and dataset.

We will release all our results, models, datasets, and code under a free software license. We will add it in the revision to make it clearer.

Q2: No Google Search.

We acknowledge that employing search engines is a natural way to solve a task. To better understand its impact, we evaluate the Browser-Use agent without this restriction. Interestingly, even when it is allowed to use Google Search, its performance did not improve significantly(26% vs 31%), suggesting that current agents still struggle to handle real-world tasks on real-world websites. Please see our general response for more details.

Q3: "What is MLLM? What is a "frontier web agent? Fig 2 cannot be read in B&W."

We appreciate your feedback. We will further clarify these terms in the revision. "MLLM" refers to Multimodal Large Language Model. "Frontier web agent" refers to current state-of-the-art and well-known web agents, such as OpenAI's Operator, Anthropic's Claude Computer Use, and open-source agents like Browser-Use. We will revise Figure 2 to make it readable for black-and-white print.

We thank the reviewer inKQ for the constructive comments. We are glad you find Online-Mind2Web is diverse, grounded in real-world tasks, and recognize the effectiveness of WebJudge. We will address your questions in the following paragraphs.

W1 and Q1: The selection of websites.

For website selection, we followed a similar approach to Mind2Web, prioritizing selecting popular websites based on traffic volume reported by SimilarWeb(https://www.similarweb.com/). Specifically, for tasks from the Mind2Web and Mind2Web-Live datasets, we chose tasks from different websites as much as possible. (Notably, tasks from the original Mind2Web and Mind2Web-Live datasets were created based on website traffic volume in the U.S., as ranked by SimilarWeb. Please see more details in the paper [1].) For the newly constructed tasks by us, we similarly selected previously unpresented but popular websites, ranked by traffic volume, according to SimilarWeb. To further ensure diversity, we also included 14 tasks from niche websites. Based on this construction process, we ultimately obtained 300 tasks covering 136 unique websites. You can see the task popularity (traffic volume) distribution in Figure A.1 in the appendix. We will clarify the selection process of websites in our revision. Our primary selection criterion was traffic volume, as it most closely reflects real-world scenarios where users typically perform tasks on popular websites. At this stage, we didn't consider technical complexity factors. However, we still included websites from different regions, such as those in the UK, as well as a small number of niche websites in our task set to enhance diversity and representation.

Here are the statistics of the website category. We categorize them into 12 domains including Shopping & E-Commerce, Food & Recipes, Housing & Real Estate, Finance & Investment, Health & Medical, Travel & Transportation, Entertainment & Media, Technology, Education, Government & Services, Jobs & Careers and Other. The table below shows the number of websites for each domain.

[1] Deng, Xiang, et al. "Mind2web: Towards a generalist agent for the web." Advances in Neural Information Processing Systems 36 (2023): 28091-28114.

W2 and Q4: The "no Google Search" constraint.

We thank the reviewer for pointing out this important discussion! Please see our general response.

W3: The bias risks and dependency on proprietary models.

We understand the concern regarding bias and reliance on proprietary models. To address this, we have synthesized training data from multiple models and trained a lightweight WebJudge-7B, based on Qwen2.5-VL-7B. This model not only achieves a high agreement rate with human judgments (87%) but also shows a small success rate gap of just 3.9% compared to human evaluation. Furthermore, it generalizes well to out-of-distribution (OOD) benchmarks, even outperforming rule-based methods. Please see our general response for more details. We will add the details in our final manuscript.

Considering that WebJudge-7B may overfit on our benchmark and thus has limited applicability, we further test it on AgentRewardBench [1], which comprises 1302 trajectories spanning 5 popular benchmarks: WebArena (WA), VisualWebArena (VWA), AssistantBench (AB), WorkArena (Work), and WorkArena++ (Wk++).

GPT-4o (Ally Tree) represents the prior SOTA results in the paper[1], which is based on the accessibility tree. Rule-based denotes the official rule-based evaluation method. “*” indicates results are taken from the paper [1]. We follow previous settings and report the same metric of precision on AgentRewardBench [1], as shown in the following table.

The table below illustrates the success rate gaps of various methods compared to human evaluation across several benchmarks.

We can find that WebJudge-7B achieves both high precision (75.7%) and a smaller success rate gap (6.0%), making it a reliable and scalable evaluator for further research. Overall, these results further emphasize the effectiveness and reliability of WebJudge-7B as an automatic evaluator, making it a valuable tool for rapid iteration in agent development and evaluation.

[1] Lù, Xing Han, et al. "AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories." arXiv preprint arXiv:2504.08942 (2025).