A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

> Questions 6

We revised Section 3 to clarify our pre-training (Pre-Training with Mixture of Long-Span Denoising in Section 3.1) and fine-tuning (Section 3.2) corpora. We also summarize our revision below.

For the pre-training dataset, we collect 100 WARC files (April 2019) from the CommonCrawl corpus and remove the non-Unicode or alphanumeric-only HTML documents. We then extract subtrees around <label> elements that have a special attribute called “for” that associates the corresponding label with a unique input element in the same HTML document. This pre-processing step improves the quality of the pre-training corpus by focusing only on HTML that is relevant for instruction following and grounding. Our final dataset has 3.41M examples.

For the finetuning dataset, we sample instructions by randomly assigning values to placeholders in manually-curated templates. We employ a rule-based parser to decompose instructions into sequences of sub-instructions; corresponding reference IDs are retrieved from HTML using regular expressions. At each step of the process, Flan-U-PaLM is provided with the sub-instruction and the associated HTML snippets to generate navigation programs that are executed through Selenium WebDriver. HTML-T5 is fine-tuned using self-experience demonstrations gathered through instruction sampling, scripted planning, and prompted program synthesis, as detailed earlier. It utilizes task instructions (e.g. please search 2 bedroom and 2+ bathroom houses in new york, ny with a max price of $7500 on real estate website), sub-instruction histories (e.g. go to real estate website,type in new york into search,click on search, click on price, click on max rent), and raw HTML as inputs. Subsequently, it generates the next sub-instruction (e.g. type in 7500 into max rent) and extracts the relevant data-ref attributes used for retrieving HTML snippets.

> Questions 7

The agent navigates to the same home page but the underlying HTML document could be different. For example, home pages can show example listings that change between visits to the website.

> Questions 8

HTML-T5 is first pre-trained with the new HTML corpus that we curated from CommonCrawl (Section 3.1). Subsequently, it is finetuned on the entirety of MiniWoB's available demonstrations. Figure 4 compares the success rates on MiniWoB among finetuned methods based on different pre-trained models (LongT5, Flan-LongT5, HTML-T5, and Flan-T5). All the models in Table 4 are exclusively finetuned with 12K/347K MiniWoB demonstrations, rather than with multi-task corpora including other NLP tasks. We compare the efficacy of pre-trained models for simulated web automation tasks.

We appreciate your careful reading and detailed discussion of our paper. We address your concerns below and please let us know if there are remaining questions or unclear points.

> Questions 1

Pre-defined action spaces are a-priori defined sets of all effective actions that an agent can perform. Our goal is to show that programs offer superior alternatives to pre-defined action spaces: They are more flexible (open-ended), and adaptable, allowing them to benefit from advancements in program synthesis. By “canonical sub-instructions”, we refer to the rule-based parsing of instructions to sub-instructions. We revised the caption of Figure 1, deleted the “canonical” word from the abstract to reduce ambiguity, and revised Section 3 to clarify.

> Questions 2

Yes, few-shot prompting. We used generic examples, including selecting checkboxes, entering text into inputs, clicking on options, and scrolling, for few-shot prompting. These examples are independent of any domain in our study to ensure simplicity, robust generalization to unseen websites, and prevent any information leakage. We also clarified few-shot prompting in Section 3.3.

> Questions 3

Input is common across all tasks, encompassing user instructions, navigation history, and raw HTML documents. However, outputs vary depending on the specific task. In real-world website navigation (Section 4.1), outputs consist of paired sub-instructions and corresponding data-ref attributes associated with the relevant elements. Within MiniWoB, outputs are defined as pre-defined actions (e.g., "click(ref=X)") as outlined by the simulator. In Mind2Web, outputs manifest as multiple-choice questions (e.g., “B”) and operational labels (e.g., “Click XXX”). Section 4.1 focuses on interactive real-world evaluation, while Section 4.2 explores evaluation on the simulator and offline evaluation. We revised section 4.2 to clarify.

> Questions 4

Planning is predicting the next sub-instruction to be executed based on the current state. Rather than predicting all the sub-instructions in advance, HTML-T5 iteratively generates the next sub-instruction per step (i.e. closed-loop planning). Summarization is retrieving all the relevant HTML snippets by predicting their data-ref attributes, similar to extractive summarization. The combined sub-instruction and HTML snippets serve as the input for Flan-U-PaLM. We revised Section 3 to clarify.

> Questions 5

We added the following paragraph to the beginning of Section 3 to explain the high-level workflow of WebAgent:

“Users initiate natural language interactions with a clear intent, such as apartment searching. Upon receiving the initial request, HTML-T5 formulates a “go to<URL>” sub-instruction, triggering Flan-U-PaLM to generate a corresponding Python program that navigates to the specified website.The raw HTML content of the newly opened page is extracted and fed into HTML-T5 along with the user’s instruction and previous planning steps. This information is utilized to predict the next sub-instruction and identify relevant reference IDs for extractive HTML summarization. Flan-U-PaLM,in turn, generates a Python program based on these sub-instructions and the combined HTML snippet.This iterative process of planning, summarization, and program synthesis continues until a designated end-of-episode sub-instruction is predicted or the maximum number of iterations is reached.”

Dear Reviewer Jsxu,

Thank you again for your taking the time to review our paper. Do you have any further questions about the paper?

Please let us know if you have any further questions. We will try to address them before the discussion period ends.

Thank you!

The authors

We thank the reviewer for the thoughtful review and comments. Please let us know any remaining questions or concerns if you have.

> Weaknesses 1

We have added new results to Appendix K (Figure 9) that compare Flan-U-PaLM-540B, publicly available gpt-3.5-turbo, and smaller model-size variants of the Flan-PaLM family of models (8B and 62B sized models). These models were tested on the map website using the same set of instructions. The results demonstrate that:

• Flan-U-PaLM-540B and gpt-3.5-turbo exhibit comparable performance (80% success, 93.8% score).
• Flan-PaLM-62B (60% success, 86.3% score) falls short of Flan-U-PaLM-540B due to inferior program synthesis capabilities.
• Flan-PaLM-8B was unable to generate valid programs.

Our findings indicate that increasing model size enhances WebAgent performance, and that any LLM capable of generating python/selenium code can be integrated into WebAgent.

For Mind2Web, we included the results from a recent concurrent work (Synapse [1]) in Table 3. HTML-T5 still compares favorably and achieves the best result.

> Weaknesses 2 & Questions 1

We have made the following revisions to Section 3 to improve clarity:

Explanation of the WebAgent Workflow: We introduced a new paragraph at the beginning of Section 3 to explain the high-level workflow of WebAgent, including user-WebAgent interaction and planning, summarization, and program synthesis components.

Revised Section 3.1: We revised Section 3.1 to clarify architectural details of HTML-T5 as well as the pre-training corpus and objectives.

Revised Section 3.2: We elaborated on the self-experience supervision approach, which involves sampling new instructions using templates, curating navigation scripts to gather planning (sequence of sub-instructions) and summarization data (corresponding reference IDs), prompting Flan-U-PaLM to generate Python programs, and utilizing execution feedback to eliminate incorrect trajectories. Furthermore, we provided a detailed explanation of HTML-T5 fine-tuning, which employs demonstrations collected through scripted planning and prompted programming to train HTML-T5 for automated planning and summarization tasks.

We collect 260 episodes on real-estate, 230 episodes on social-media, and 410 episodes on map websites (explained in Evaluation Methodology in Section 4.1). Examples for different tasks are illustrated in Appendix D. Please let us know if you have further unclear points.

> Weaknesses 3 & Questions 2

We revised Section 3.2 to clarify fine-tuning of HTML-T5 and prompting of Flan-U-PaLM. HTML-T5 is fine-tuned to predict sub-instructions and corresponding data-ref attributes directly from raw HTML documents. HTML snippets that correspond to these “data-ref” attributes are extracted and merged, similar to how extractive summarization [2] or retrieval works. We feed merged HTML snippets to prompt Flan-U-PaLM to generate navigation programs as depicted in Figure 3. Please let us know if you have further unclear points.

> Weaknesses 4 & Questions 3

Our preliminary analysis has shown that approximately 90% of our pre-training HTML corpus has around 4K context length (see Figure 5). While raw HTML documents can be longer, our pre-processing methodology extracts useful subtrees for pre-training; substantially reducing context length (see Appendix C) while improving effectiveness (see Table 7 for an ablation). That is why we pre-train HTML-T5 using 4096 window size (Pre-Training with Mixture of Long-Span Denoising in Section 3.1), but finetune it with 16K window size (Section 4.2) to generalize to real-world HTML documents.

> Weaknesses 5 & Questions 4

In the context of real-world navigation, we constructed a single dataset of instructions. This dataset was carefully partitioned into training and testing sets to guarantee no overlap. All instructions included in the testing set are provided in Appendix F. For both MiniWoB and Mind2Web, we adopted the experimental setup established by their respective authors. In MiniWoB, instructions are randomly generated from a vast pool of instructions, and environments accept an argument for either "training" or "testing." For Min2Web, training and testing sets are maintained separately.

> Weaknesses 6

Thank you for pointing out the syntactic mistakes. We updated our manuscripts and fixed the issues we found.

[1] Deng et al., (2023) Mind2Web: Towards a Generalist Agent for the Web (https://arxiv.org/abs/2306.06070)
[2] Xiao and Carenini (2019) Extractive Summarization of Long Documents by Combining Global and Local Context  (https://arxiv.org/abs/1909.08089)
[3] Guo et al., (2021) LongT5: Efficient Text-To-Text Transformer for Long Sequences (https://arxiv.org/abs/2112.07916)

We appreciate the careful reading and thoughtful comments. We address your concerns below, and please let us know if there are remaining questions or unclear points.

> Weaknesses 1 & Questions 2

We have added new results to Appendix K (Figure 9) that compare Flan-U-PaLM-540B, publicly available gpt-3.5-turbo, and smaller model-size variants of the Flan-PaLM family of models (8B and 62B sized models). These models were tested on the map website using the same set of instructions. The results demonstrate that:

• Flan-U-PaLM-540B and gpt-3.5-turbo exhibit comparable performance (80% success, 93.8% score).
• Flan-PaLM-62B (60% success, 86.3% score) falls short of Flan-U-PaLM-540B due to inferior program synthesis capabilities.
• Flan-PaLM-8B was unable to generate valid programs.

Our findings indicate that increasing model size enhances WebAgent performance, and that any LLM capable of generating python/selenium code can be integrated into WebAgent.

> Weaknesses 2

We have made the following revisions to Section 3 to improve clarity:

Explanation of the WebAgent Workflow: We introduced a new paragraph at the beginning of Section 3 to explain the high-level workflow of WebAgent, including user-WebAgent interaction and planning, summarization, and program synthesis components.

Revised Section 3.1: We revised Section 3.1 to clarify architectural details of HTML-T5 as well as the pre-training corpus and objectives.

Revised Section 3.2: We elaborated on the self-experience supervision approach, which involves sampling new instructions using templates, curating navigation scripts to gather planning (sequence of sub-instructions) and summarization data (corresponding reference IDs), prompting Flan-U-PaLM to generate Python programs, and utilizing execution feedback to eliminate incorrect trajectories. Furthermore, we provided a detailed explanation of HTML-T5 fine-tuning, which employs demonstrations collected through scripted planning and prompted programming to train HTML-T5 for automated planning and summarization tasks.

Revised Section 3.3: We clarified the few-shot prompting.

We collect 260 episodes on real-estate, 230 episodes on social-media, and 410 episodes on map websites (explained in Evaluation Methodology in Section 4.1). Examples for different tasks are illustrated in Appendix D. Please let us know if you have further unclear points.

> Questions 1

To provide a more comprehensive comparison, we have added a state-of-the-art (SoTA) baseline utilizing GPT-3.5 (Synapse [2]) in Table 4. However, we emphasize that our primary focus is on comparing models that are readily accessible, disclose their training corpus, and adhere to the same training dataset size (12K or 347K). While GPT variants demonstrate impressive performance on MiniWoB, direct comparisons are hindered by the lack of transparency regarding their training corpora. In this context, HTML-T5 emerges as a promising step towards on-device and privacy-preserving deployment.

> Questions 3

We have explained the details of HTML-T5, including its architecture, training process, and data preprocessing steps. HTML-T5 utilizes publicly available T5 models and is trained on the CommonCrawl corpus. While PaLM models are not open-sourced, we intend to release HTML-T5 after thoroughly evaluating its potential societal implications.

[1] OpenAI (2023) GPT-4 Technical Report (https://arxiv.org/abs/2303.08774)
[2] Zheng et al., (2023) Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control (https://arxiv.org/abs/2306.07863)

We thank the reviewer for the careful reading and constructive feedback. We address your concerns below. Please let us know if you have further questions.

> Weaknesses 1

Following ICLR 2024 Author Guide (https://iclr.cc/Conferences/2024/AuthorGuide), we added the ethics statement section between the main text and reference. We discussed that a careful treatment of the subject from the researchers, policymakers and industries to form guidelines and regulations to prevent misuse of web automation from negatively impacting real users is needed. Please check page 10 in the revised paper.

> Weaknesses 2

We have added new results to Appendix K (Figure 9) that compare Flan-U-PaLM-540B, publicly available gpt-3.5-turbo, and smaller model-size variants of the Flan-PaLM family of models (8B and 62B sized models). These models were tested on the map website using the same set of instructions. The results demonstrate that:

• Flan-U-PaLM-540B and gpt-3.5-turbo exhibit comparable performance (80% success, 93.8% score).
• Flan-PaLM-62B (60% success, 86.3% score) falls short of Flan-U-PaLM-540B due to inferior program synthesis capabilities.
• Flan-PaLM-8B was unable to generate valid programs.

Our findings indicate that increasing model size enhances WebAgent performance, and that any LLM capable of generating python/selenium code can be integrated into WebAgent.

> Questions 1

Our planner, HTML-T5, is trained to automatically replan whenever a new state is observed. By doing so, the agent will react more quickly when an unusual transition is encountered. For example, in case an error causes the state to go to the home screen before finishing apartment search, the agent will immediately start replanning from the first step.

> Questions 2

Our results show that the performance of WebAgent would improve with scale. Model-size ablation of HTML-T5 shows that if we increase the number of parameters from 220M to 3B, the performance gets better (Table 8 in Appendix H). In Table 4, we also show that increasing the data size from 12K to 347K improves the performance. We also briefly explain these points in Section 5 (Broad Generalization across the Internet).

> Questions 3

We have explained the details of HTML-T5, including its architecture, training process, and data preprocessing steps. HTML-T5 utilizes publicly available T5 models and is trained on the CommonCrawl corpus. While PaLM models are not open-sourced, we intend to release HTML-T5 after thoroughly evaluating its potential societal implications.

Dear Reviewer Gbpj,

Thank you again for your taking the time to review our paper. Do you have any further questions about the paper?

Please let us know if you have any further questions. We will try to address them before the discussion period ends.

Thank you!

The authors

Dear Reviewers,

We would appreciate it if they could check our updates and feel free to raise further questions if you have. We are happy to clarify them. We also appreciate the response from some reviewers. Thank you so much for your time!

Sincerely,

Authors