Thank you for taking the time to review our paper, and we truly appreciate your recognition of the contributions of WebCanvas!

We believe that all your concerns can be addressed during the discussion phase. Please refer to our responses below. If there are any remaining unclear points, we would be more than happy to further clarify them during the discussion.

1. Clarification on Symbol Representation

We sincerely thank you for pointing out the issue with our symbol representation. We fully accept this suggestion. The original notation $a_{1}^{t-1}$ may not be clear. We will revise the notation to $a_{1:t-1}$ in the relevant sections (L136, L837) of the manuscript.

2. Response to Issues in Section 2 and Framework Description

On the problem formulation: In the problem formulation, we referenced the formulation used in WebArena[1]-the well-recognized paper in web agents. We intentionally avoided mentioning the original POMDP formulation because it might confuse some readers in our setting. WebCanvas uses a Key Node-based Evaluation approach and does not involve a traditional "Final Objective Function." This is because WebCanvas focuses on progress evaluation during the task execution. A task is deemed successful if the agent completes all the step scores associated with the task's key nodes. This approach is fundamentally different from traditional benchmarks, which rely on either Reference-based or Outcome-based evaluations. Figure 3 in the paper illustrates the advantages of Key Node-based Evaluation, particularly in its ability to handle the dynamic and diverse nature of real-world web environments.

On Appendix E.1 content placement: We agree that Appendix E.1 provides additional details about the experimental framework. However, we decided to include this content in the appendix for two primary reasons:

• Length considerations: Due to the page limit, we prioritized the core contributions and innovations of our work. For readers familiar with prior work, the concise description in Section 5 is sufficient to understand our framework and experimental methodology. For readers less familiar with the background, Appendix E.1 offers a clear and comprehensive explanation.

• Avoiding potential misunderstanding: Our agent framework is a baseline framework and does not introduce significant innovations compared to existing frameworks such as WebArena[1] or VisualWebArena[2]. By placing this content in the appendix, we aim to prevent any misinterpretation that our framework itself represents a major contribution. This paper is mainly innovating on introducing a define-and-evaluate scalable evaluation framework to connect web agents to live environment and we thoroughly jusify its effectiveness.

[1] WebArena: A Realistic Web Environment for Building Autonomous Agents, Zhou et al., 2023

[2] VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks, Koh et al., 2024

Dear Reviewer m7E8, We wanted to say in advance our heartfelt thank you's for the time and effort you've put into both the reviews that have passed as well as the upcoming current discussion period. We know that you all have your own papers that you have to deal with during this busy time, and sincerely appreciate the time you've taken to spend on ours.

We are so excited about this paper and its findings, so we are very much looking forward to the upcoming discussions with you! Please don't hesitate to ask us any questions big or small, and we are happy to provide any further clarifications.

Thank you for recognizing the value of our benchmark dataset, evaluation framework, and the openness of our data infrastructure. We understand that your concerns primarily revolve around the completeness of the dataset comparisons, the diversity and quantity of datasets, and our platform maintenance plan, along with some questions regarding specific details of the paper. We will address these issues one by one. If there are still any unclear points, we would be more than happy to clarify them further during the discussion.

1. Explanation of Dataset and Model Comparison Settings

We chose not to compare with other benchmarks for several reasons. The well-known web agent benchmarks previously published, such as WebArena[1] and VisualWebArena[2], are based on tasks defined in closed environments, making it difficult to replicate these tasks and their evaluation functions in live environment, thus preventing fair comparisons. While WebLinx[3] presents an important setting, it primarily focuses on multi-turn conversations, which differ from our current task setting. GAIA[4] is not specifically focused on web tasks, with relatively few tasks entirely operating within a web browser. The task definitions in AssistantBench[5] are insightful and clear, and we plan to incorporate this dataset in future iterations. However, many tasks in AssistantBench (e.g., "Which gyms near Tompkins Square Park (<200m) have fitness classes before 7am?") are more suited for parallel inference using multiple agents rather than a single agent. In these cases, each agent’s trajectory would be ideal for definition and evaluation under the current WebCanvas framework. We plan to support this kind of multi-agent system evaluation in the future, which we believe is essential for assessing truly complex web tasks. However, we do not believe this detracts from our contribution in this paper, which is the first to propose an online evaluation method for sequential web tasks. Our comparison with Mind2Web aimed to clearly demonstrate the limitations of reference-based evaluation in live environments, as explained in detail in Section 6.2.

To clarify the model benchmarking results, our paper primarily focuses on benchmarking the reasoning capabilities of LLMs. Therefore, we fixed a universal agent framework (Appendix E.1) and evaluated the performance of both open-source and closed-source base LLMs under this framework (Table 4). We also analyzed the value of different components within this LLM-based agent framework (Table 12). Exploring the optimal framework is not the main focus of this paper, and the requirement for long context length for web agents limited the number of models we could test. We respectfully argue that we have tested a sufficient number of base models under this setting and have clearly demonstrated the performance differences among these models.

[1] WebArena: A Realistic Web Environment for Building Autonomous Agents, Zhou et al., 2023

[2] VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks, Koh et al., 2024

[3] WebLINX: Real-World Website Navigation with Multi-Turn Dialogue, Lù et al., 2024

[4] GAIA: a benchmark for General AI Assistants, Mialon et al., 2023

[5] AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?, Yoran et al., 2024

2. Dataset Diversity and Quantity

Task Distribution and Domain Coverage: To ensure that Mind2Web-Live reflects the domain distribution of Mind2Web, we followed Mind2Web's categorization and selected tasks from the following domains: Travel, Shopping, and Entertainment. Please note that the Info and Service domains in Mind2Web only appear in cross-domain tasks in the test set, which is why they are not included in Mind2Web-Live. We have provided statistics on the task distribution in the training and test sets across these domains, which can be found in Table 2 of our response to reviewer mDNN. We have included these statistics in Appendix A.3 of the manuscript. Additionally, Figure 13 and Figure 14 of the paper provide detailed statistics on Completion Rate and Task Success Rate across different websites, allowing us to evaluate Web Agent performance across various domains.

Task Quantity in the Mind2Web-Live Benchmark: As mentioned in our response to reviewer mDNN, maintaining a large test set significantly increases the cost for community evaluation of web agents. For instance, testing 1,000 tasks per model with an average runtime of 1.5 minutes per task would take approximately one day and incur high API costs. Therefore, we decide to maintain a relatively small, high-quality, general-purpose foundation benchmark that retains sample diversity. Furthermore, the vision of our work is to enable community members to build evaluation sets in any vertical domain at a low cost in the future, rather than just centrally releasing a dataset. We also plan to incorporate additional high-quality benchmarks, such as AssistantBench, in future iterations, and work out better UI for community members to define their own evaluation sets more easily.

Thank you for taking the time to review our paper, and we truly appreciate your recognition of the contributions of WebCanvas!

We believe that all your concerns can be addressed during the discussion phase. Please refer to our responses below. If there are any remaining unclear points, we would be more than happy to further clarify them during the discussion.

1. On the Distribution of Data and Tasks in Mind2Web-Live

Data Annotation Process: Given the significant time and effort required to annotate tasks and key nodes in a real-world online environment, we had to select a subset of tasks from the original Mind2Web dataset, which contains over 2,000 tasks, to construct Mind2Web-Live. To retain the characteristics of Mind2Web as much as possible, we avoided making substantial modifications to the task descriptions.

Construction of the Training Set: From the original Mind2Web training set (comprising 1,009 tasks), we excluded all time-sensitive tasks. Subsequently, 75% of the remaining tasks were annotated with key nodes, resulting in the Mind2Web-Live training set, which includes a total of 438 tasks.

Construction of the Test Set: The test set is based on the cross-task subset of Mind2Web test set. After removing time-sensitive and unannotatable tasks, we re-annotated the remaining tasks to form the Mind2Web-Live test set. It is important to note that the Mind2Web test set includes cross-website and cross-domain tasks, totaling over 1,000 tasks. However, annotating these tasks involves considerable costs, and large-scale online testing imposes significant time and API usage expenses. For example, testing 1,000 tasks at an average runtime of 1.5 minutes per task would take approximately 1 day and result in high API costs, which are unaffordable for many researchers. As a result, we had to forgo the annotation of this portion of the data.

The selection ratio of tasks from the Mind2Web dataset for Mind2Web-Live is shown in Table 1.

Task Distribution and Domain Coverage: We followed the same task categorization used in the Mind2Web. Tasks were selected from the following domains: Travel, Shopping, and Entertainment. (Note: The Info and Service domains in Mind2Web only exist in the test set's cross-domain tasks and are therefore not included in Mind2Web-Live.) We analyzed the distribution of tasks across these domains for both the training and test sets, as detailed in Table 2. Furthermore, in Figures 12 and 13 of the paper, we provide a comprehensive analysis of the Completion Rate and Task Success Rate for tasks across different websites to evaluate the Web Agent's performance in various domains and subdomains.

Mind2Web-Live represents an innovative attempt to evaluate Web Agents in online environments. Although the data selection process involved certain limitations, we adhered to Mind2Web’s task categorization standards and preserved its domain distribution. The intuition behind this work is mainly the scalability of this define-then-evaluate framework for the community. Moving forward, we will collaborate with the research community to expand the live evaluation benchmarks for web agents further, bringing in brilliant data resources and thoughts like AssistantBench[1], and other vertical domains.

Table 1:

Table 2:

[1] AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?, Yoran et al., 2024

2. On the Maintenance Costs and Community Contributions

Maintenance Costs: It is true that as the dataset scales up, maintenance costs may increase accordingly. However, the motivation behind this project is to optimize the cost. As described in Section 4.2, we developed an automated maintenance system capable of detecting invalid workflows and key nodes to run each two months. Additionally, for issues that the system cannot detect, community members can report problems through our bug report module in our platform. This multi-channel maintenance mechanism is designed to reduce manual effort and minimizes the accumulation of errors.

As for the cost of corrections, four authors collectively spent less than one hour completing a round of maintenance(L258), demonstrating that our approach is efficient and feasible at the current scale. Even as the dataset continues to expand in the future, we are confident that maintenance will remain feasible through the appropriate optimization of workflows.

The Inevitability of Key Node Evaluation: We acknowledge that in a real-world online environment, data obsolescence is unavoidable. However, with our Key Node-based task maintenance approach, we can achieve relatively reliable and efficient management. In comparison, other common evaluation methods have notable limitations:

• Reference-based methods: These methods rely on fixed paths, which are incapable of handling the diversity of task paths in online environments. As shown in Figure 10, the agent and the annotated task take completely different paths, yet reference-based evaluation methods fail to account for such variations, leading to reduced reliability.

• Outcome-based methods: Evaluating tasks solely based on their final state through specific scripts execution is similarly sub-optimal for a scalable evaluation system. First, certain tasks cannot be assessed using only the final state alone. For example, consider the task of submitting a form. If we rely on a single state to determine whether the form content is valid and whether the submission is completed, there are two possible approaches: One approach is to monitor the API call for submitting the form (which may not be permitted by the website). Another approach is to check the feedback page indicating successful submission. However, if the website does not include such a feedback design, it becomes impossible to evaluate the task. Apart from this limitation, maintaining these scripts incurs high costs, requiring specialized expertise and significant time investment, as stated in OSWorld[1], where the scripts were annotated by the authors themselves. In contrast, our Key Node-based approach, relying on URL and element validation, can be conducted by annotators with basic computer knowledge. Furthermore, in-progress key nodes evaluation is dispensable. Our findings(L373) indicate that only 47% of tasks can be effectively evaluated using final key node alone.

Regarding Community Contributions and Extensibility: We actively encourage community participation in maintaining the dataset. Our code and community platform allow for flexibility to support community-driven contributions. Over the past few months, we have received 12 improvement suggestions for our agent code and 25 task modification requests from 5 groups through various channels. This highlights the community's recognition and engagement in our work. In the future, as the dataset scales beyond the capacity of the author team to maintain independently, we are open to hire paid annotators for data maintainance for the community to move forward.

[1] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, Xie et al., 2024

Dear Reviewer mDNN, We wanted to say in advance our heartfelt thank you's for the time and effort you've put into both the reviews that have passed as well as the upcoming current discussion period. We know that you all have your own papers that you have to deal with during this busy time, and sincerely appreciate the time you've taken to spend on ours.

We are so excited about this paper and its findings, so we are very much looking forward to the upcoming discussions with you! Please don't hesitate to ask us any questions big or small, and we are happy to provide any further clarifications.

Thank you for taking the time to review our paper and for acknowledging the motivation behind WebCanvas!

We believe that all your concerns can be addressed during the discussion phase. Please refer to our responses below. If there are any remaining unclear points, we would be more than happy to further clarify them during the discussion.

1. Comparison of Key Node Evaluation with Other Evaluation Methods in Online Environments

Comparison with Reference-Based (or Step-Based) Evaluation In Section L368-L371 and Figure 10, we demonstrated that reference-based evaluation does not work effectively in complex online environments with agent tasks. By visualizing both human-annotated and agent-executed trajectories, we found that even when all key nodes are satisfied, there can be significant differences in possible trajectories. However, reference-based evaluation methods fail to account for such variations, leading to reduced reliability. The experiments comparing Key Node-based evaluation with reference-based evaluation also show that results from step-based testing on offline datasets do not necessarily reflect model performance in live online environments (please see Table 3 and the description of the second part comment below).

Comparison with Outcome-Based Evaluation Thanks for your question to help us clarify this comparison further. Outcome-based evaluation is insufficient for online tasks where completion cannot be judged solely by the final outcome state. For instance, in a form submission task, relying on a single state to determine whether the form content is valid and whether the submission is completed has challenges. One approach is to monitor the API call for form submission, which may not be allowed by the website, while another is to check if a feedback page appears. However, if there is no feedback design, evaluation becomes impossible. Moreover, creating such scripts is time-consuming, requiring specialized expertise and significant time investment, as highlighted in OSWorld[1], where scripts were manually annotated by the authors. Additionally, the dynamic nature of online environments necessitates constant script maintenance, further increasing costs, thus it's suboptimal in a scalable evaluation system and complex domain like web environment. In contrast, our Key Node-based approach, which uses URL and element validation, can be conducted by annotators with basic computer knowledge. We will include this discussion in Section 6.2 of the paper. Evaluating tasks solely by the final key node is also suboptimal, as shown in our findings (L373), which indicate that only 47% of tasks can be effectively evaluated using the final key node alone. The Key Node-based evaluation also provides additional benefits, such as evaluating agent progress throughout the task, which facilitates analyzing and improving agent performance. This approach also enabled us to conduct experiments on planning with improved in-progress rewards, demonstrating the value of better reward signals for in-context reasoning.

We hope this illustration highlights the completeness and accuracy of key node-based evaluation, making it a superior method for assessing online agent performance.

[1] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, Xie et al., 2024

Dear Reviewer uzoM, We wanted to say in advance our heartfelt thank you's for the time and effort you've put into both the reviews that have passed as well as the upcoming current discussion period. We know that you all have your own papers that you have to deal with during this busy time, and sincerely appreciate the time you've taken to spend on ours.

We are so excited about this paper and its findings, so we are very much looking forward to the upcoming discussions with you! Please don't hesitate to ask us any questions big or small, and we are happy to provide any further clarifications.