WebArena: A Realistic Web Environment for Building Autonomous Agents

Thank you for taking the time to review our paper and for your helpful feedback. We are happy that you appreciated the realism and the complexity of the benchmark!

We think that all your questions are addressable within this discussion period. Please see our response below. We will be happy to address additional questions during the discussion period if anything is unclear.

Environment/datasets/resources do not fit the style of ICLR

We kindly note the primary area within ICLR that this paper is submitted to is datasets and benchmarks, and we believe our work is a good fit for it. Please see the ICLR 2024 call for papers (https://iclr.cc/Conferences/2024/CallForPapers) where “datasets and benchmarks” is listed as one of the 20 topics of interest.

Will using GPT-4 for evaluation limit the use of the WebArena environment?

Thanks for pointing this out. We argue that using GPT-4 will not prevent future research on WebArena. Only ~10% (82 out of 812) of the examples require using GPT-4 for evaluation, and the cost is less than 1 US dollar.

Is GPT-4 evaluation accurate?

This is a reasonable concern. To evaluate this, we manually checked 40 examples and found that 39 of them are identical to our human judgment. In addition, among the 82 examples that require using GPT-4 for evaluation, the answer of 49 (60%) examples is a date (e.g., 10/23/2022) or time duration (e.g., 15 minutes). In these cases, GPT-4 is only used to judge the different format of the answers. We quantitatively evaluate the correctness of GPT-4 in this case by generating different formats of a date and time duration programmatically. We randomly sample negative examples. For instance, Nov 3, 2022, November 3, 2022, 3rd November 2022, 3 Nov 2022, 2022-11-03, and 3rd of November, 2022 are all correct variances of 2022/11/03. The accuracy of GPT-4 is as follows:

We can see that GPT-4 and its recent turbo version are extremely accurate, both achieving 100% accuracy. We will add these experiments to our revision.

Where do humans fail?

As described in our paper (Section 3.2), we found that 50% of the failures are not due to human annotators’ inability to perform the tasks. The major errors are (1) misinterpreting the intent such as providing the travel distance when asked for travel time. (2) Incomplete answers such as providing only the customer's name when asked for name and email. And (3) incomplete executions such as partially filling the product information. The other 50% are more severe failures where the human performed the task in a wrong way.

Thank you for taking the time to review our paper, for your constructive feedback, and for your kind words regarding the high quality of the benchmark!

We think that all your questions are addressable within this discussion period. Please see our response below. We will be happy to address additional questions during the discussion period if anything is unclear.

Comparison with AndroidEnv

The key difference between AndroidEnv and WebArena is the reproducibility of the environment and tasks. The Android Apps inside AndroidEnv are live, meaning that they change over time rather than being “frozen”. Therefore, AndroidEnv faces the same technical challenges as a benchmark that was created based on live websites. For instance, CAPTCHAs, unexpected content modifications, and configuration changes. These obstacles can make it difficult to perform fair comparisons across various systems over time. As a result, creating and evaluating realistic tasks is challenging. In contrast, all these challenges are resolved in WebArena. We have updated the description of AndroidEnv in the related work section (revision in blue).

Additionally, AndroidEnv focuses on Android applications, while WebArena focuses on browser-based applications. While there are some similarities in the tasks, there are also differences, such as UI layouts, that make the two settings unique.

How do open-source models perform on WebArena?

We are delighted that many follow-up works (not by us) have evaluated open-source models on WebArena after our public release. Because we are not sure whether directly citing the works will break the anonymity policy, we only report the numbers here. We will cite these works in the camera-ready version.

Overall, WebArena presents a significant challenge for both closed and open models.

Do you evaluate agents other than CoT and Direct?

Yes, we performed preliminary experiments with agents with other prompting methods that perform planning and critique, similar to RCI [1] and Reflexion [2]. However, we did not proceed with the full experiment as the performance on a subset of examples was significantly lower.

Since the tasks in WebArena span multiple pages, planning and critiquing based on the current observation is usually inaccurate. A general-purpose LLM (like GPT-4) does not have enough knowledge about the specific websites included in the WebArena benchmark. For instance, given the task of “Checkout my pending orders”, the agent first plans to perform the action click [1283] to go to the "My Account" page., its critique component responds that It assumes that clicking on the "My Account" link ([1283]) will take you to the page where pending orders are displayed. However, there is no guarantee that this is the case. The "My Account" page could have a different purpose or may not even exist.. However, there is indeed a “My order” link inside “My Account”. Furthermore, the plan is usually vague and it does not give enough hints for the agent to perform the concrete next step. We found that the agent repeated the same action more frequently and failed to execute the given task.

We will add this discussion to the Appendix in our revision.

Implementation and evaluation details of the agents

As described at the beginning of Section 4, we performed evaluations under a few-shot in-context learning setting. We did not perform any finetuning which updates the parameters. The prompts with or without CoT are listed in Appendix A.5, but we agree that it may make the paper clearer if we move additional details to the main paper. We will consider this for the final version, depending on how much space we can make in the paper.

The evaluation process is described in Section 3.2 and concrete examples in Table 1.

The formulation of POMDP

We originally used the POMDP formulation because it is a standard way of modeling these problems in related works (e.g., WebShop [2]). However, we agree that this formulation can be confusing. Hence, we decided to remove the mention of POMDP and rephrase the task-solving process in a more simple way as follows:

Given a task described as a natural language intent $\mathbf{i}$, an agent issues an action $a_t \in \mathcal{A}$ based on intent $\mathbf{i}$, the current observation $o_t \in \mathcal{O}$, the action history and the observation history $a_{1}^{t-1}, o_{1}^{t-1}$

Section 2.1 has been updated accordingly.

Thank you very much for your time to review, your constructive comments, and the encouraging words! We believe that your questions, and especially the questions regarding experiments, are addressable within this discussion period, please see our response below.

Do you evaluate agents other than CoT and Direct?

Yes, we performed preliminary experiments with agents with other prompting methods that perform planning and critique, similar to RCI [1] and Reflexion [2]. However, we did not proceed with the full experiment as the performance on a subset of examples was significantly lower.

Since the tasks in WebArena span multiple pages, planning and critiquing based on the current observation is usually inaccurate. A general-purpose LLM (like GPT-4) does not have enough knowledge about the specific websites included in the WebArena benchmark. For instance, given the task of “Checkout my pending orders”, the agent first plans to perform the action click [1283] to go to the "My Account" page., its critique component responds that It assumes that clicking on the "My Account" link ([1283]) will take you to the page where pending orders are displayed. However, there is no guarantee that this is the case. The "My Account" page could have a different purpose or may not even exist.. However, there is indeed a “My order” link inside “My Account”. Furthermore, the plan is usually vague and it does not give enough hints for the agent to perform the concrete next step. We found that the agent repeated the same action more frequently and failed to execute the given task.

We will add this discussion to the Appendix in our revision.

How do accessibility trees encode structures without something similar to the tags in HTML?

The accessibility tree shares a similar structure to the YAML format, where indentations reflect the hierarchical structure.

Comparison between HTML observation and accessibility tree observation

We experimented with using HTML as the observation with gpt-3.5-turbo-16k-0613 and the results are as follows:

Empirically, we found that using accessibility trees yields better performance. Qualitatively, we found HTML is longer and contains a lot of redundancy. For instance, a single button can be nested inside several layers of HTML elements. This redundancy increases the burden on LLMs to localize and use the relevant information.

We will add the results and the discussion to the Appendix in our revision.

Do you have golden traces?

We completely agree that reference trajectories from humans could add value to the existing resources. We did make an initial attempt to provide such resources, but eventually we decided that it was not feasible within the scope of this paper due to the engineering effort involved in implementing a robust trajectory recording tool. We leave implementing such a tool and the collection of reference trajectories as our future work.

Is the success rate of CS graduate students higher than other evaluators?

We acknowledge that there may be a difference in human performance when annotators with different demographics are involved. In fact, many tasks in our dataset require domain-specific knowledge. For instance, an average user may not know what a git merge request is; or how to create a product in a complex content management system. We aim to design tasks that have easy-to-imagine outcomes (e.g., a new product page is created) rather than those that are easily performed by an average user without significant domain knowledge. We will add this discussion to Section 3.2 if space permits, or to the Appendix otherwise.

In addition, we attempted to resolve the potential ambiguities and ensure the annotation correctness with the process we described in Section 3.2 (Annotation process). We would love to look into the concrete examples where you and your colleagues found it difficult to perform the tasks if you don’t mind sharing the task_id.

Does the model use the tool websites as humans?

In this work, our primary goal is to provide the setup of having individual tool websites for future studies. In our prompts, we simply inform the model that the homepage lists other websites rather than more explicit instructions on using some sites as tools. In this case, GPT-4 did not use these tool websites effectively. For instance, it did not use the calculator when doing tasks that require arithmetic operations. We leave more explicit prompting as an interesting and important future work.

[1] Language Models can Solve Computer Tasks, Kim el at, 2023 [2] Reflexion: Language Agents with Verbal Reinforcement Learning, Shinn el at, 2023