HeaP: Hierarchical Policies for Web Actions using LLMs

We thank the reviewer for their feedback! Please find below our response:

The evaluation is based on too few samples: 45 tasks on MIniWob++, 125 examples of two domains on WebArena, 5 distinct tasks with 20 scenarios on Ariline CRM, and 3 website with 10 searches per site on Live Websites.

• We provide the most comprehensive evaluation on web tasks, i.e. 4 distinct environments including live websites and totaling ~2500 evaluations. Prior works in this domain [1-5] (in Table 1) evaluate on 1-3 datasets, and none evaluate on live websites.
• On MiniWoB++, we evaluate all tasks that do not require visual inputs (see Table 4 in Appendix B for detailed breakdown). Each task was evaluated on 50 seeds totalling 2250 (=45*50) episodes or trajectories lasting 2-10 timesteps.
• We evaluate two distinct domains on WebArena, a SOTA challenging dataset that came out only a couple months ago. We outperform the best performing LLM agent in their paper by a factor of 2x by leveraging hierarchy (see Fig. 4). None of the prior works in Table 1 evaluate on WebArena. In fact, since prior works use large amounts of demonstrations (e.g. 347k, 12k, 2.2M), it’s not possible to use those on WebArena tasks since they don’t have large-scale human demonstrations yet.
• We can certainly evaluate up to 100 scenarios on Airline CRM, but the confidence intervals are fairly tight.

Given the fact that human demonstration is collected to form the prompts to the LLM in HEAP, it should be actually evaluated on more diverse websites instead of fewer websites.

• We reiterate that to the best of our knowledge is this is the most diverse evaluation on web automation tasks compared to prior works [1-5] in this domain.
• Live websites are challenging since web pages are complex containing ~12k tokens, that we compress down to 2k tokens in text formats for an LLM without loss of performance (see Fig. 6). The purpose of showing live website evaluations is to show the real-world applicability of our method.
• It is not feasible to do extensive evaluation on the live websites (none of the prior works [1-5] in this domain do this either):
  • Bot detectors
  • Safety constraints
  • Evaluating success
  • Require visual reasoning that needs multi-modal models

Could the author elaborate on the demonstration collection process described in 4.2?

• Following is the excerpt from Appendix D.1 where we detail this
  • We collect demonstrations from a human user who is given a conversation context \phi and performs a sequence of actions in the browser to complete the task.
  • We design a browser plugin to record webpage DOM d and events such as clicks and types.
  • Each demonstration is parsed as text by converting the DOM tree into a list of salient web elements like links, buttons, inputs. Each element is represented as text, e.g. <button id=18 title="Travelers">1 Adult />. Action events are parsed as a CLICK <id> or TYPE <id> <val>.

What's the purpose of D_{label} on the end of page 4 (Sec 4.2)? How does this dataset utilized in the HEAP?

• D_label is used to generate examples for both high-level and low-level prompts.
• Following is the excerpt from Appendix D.2 where we detail this
  • Once we have a dataset of labeled demonstrations D_{label} we can use that to generate base prompts for both the high-level task planner and low-level policies.
  • For the task planner, we concatenate the following input–output pairs as in-context examples in the base prompt: {input: [context \phi, initial state s0]} and {output: sequence of web policy calls (\pi1, \psi1), (\pi2, \psi2) . . . , (\piT , \psiT )}. For each web policy, we search the dataset D_label for all instances of the policy and create a prompt by concatenating examples of {input: [instruction \psi_t, current state s_t, previous actions a_{t:t−k} ]} and {output: next action a_t}.

What is the trainable component in the HEAP? e.g. which function / parameters described in Algorithm 1 is learnable?

• Since HEAP uses in-context learning, the parameter that can be varied is the input prompt (P_task in Line 7, P_policy in Line 12), specifically the few-shot examples in the prompt.

Why the models are not evaluated on the entire benchmarks but only a set of them.

• On MiniWoB++, we evaluate all tasks that do not require visual inputs.
• WebArena is a recent and challenging benchmark, and we picked two domains that were easy to collect demonstrations and did not require visual inputs.

What does training size 21 in Table 1 last row mean? The HEAP was shown 21 samples, in which format, to train which part?

• Please see Table 3 which shows the breakup of the 21 examples in HeaP by examples per policy (see row MiniWob++).
• Each example is a tuple of (context, browser content, previous action, action). The examples are used as part of the prompt that is sent to an LLM to predict the action. See Appendix G.1, Listing 2 for the final prompt with examples.

We thank the reviewer for their feedback! Please find below our response:

I recommend moving some implementation details like prompts into the main body to help the reader better understand the work.

• We are happy to do so

Do you label the high-level task plans?

• We do not. The demonstrations collected from the human demonstrations are states and low-level actions. We then use “autolabeling” at each timestep to label the low-level policy that was being executed. Our scripts convert the labeled datasets to prompts for both high-level planners and low-level policies. Finally, we augment prompts with chain-of-thought reasoning.

We thank the reviewer for the feedback! Please find below our responses:

The idea of hierarchical planning with a high-level planner and low-level policies using LLM has been explored by many previous robotics works

There seems to be misunderstanding on the claimed contributions of this paper.

• We agree that hierarchical planning is a fundamental concept in AI and has been the bedrock of robotics and decision making at large.
• We don’t make any universal claims on hierarchical planning with LLMs
  • Although we note that at the time of writing this paper there is no work (to the best of our knowledge) that uses LLMs for both high and low-level planning
• Instead our focus is on enabling LLMs to solve tasks on the web
  • This is an increasingly important domain, where we present the first results from an LLM based approach that achieves SOTA results with orders of magnitude lower demonstrations (see Table 1). We do so by showing that hierarchical policies are essential for solving web tasks. We provide the most comprehensive evaluation on web tasks, i.e. 4 distinct environments including live websites and totaling ~2500 evaluations. Prior works in this domain [1-5] (in Table 1) evaluate on 1-3 datasets and none evaluate on live websites.

We are happy to update the writeup to clear any misunderstanding.

Considering these previous works, the novelty of this paper is limited to applying existing ideas to web datasets and potentially the technical details of autolabeling from human demonstrations.

Thank you for the related works, we will include the relevant ones in the paper. However, we note that:

• None of them propose hierarchical LLM policies (LLM is only used as the high-level planner)
• None of the them look at the web domain
• Some of them don’t even look at planning tasks (e.g. Q&A)
• None of them propose a method to auto-generate prompts by autolabeling demonstrations

Going into further details

• LLM-planner https://arxiv.org/pdf/2212.04088.pdf
  • Focuses exclusively on vision-language navigation, a very different domain
  • Moreover, only the high-level planner is LLM, low-level planner is a combination of engineered/learned components
  • The authors themselves mention that none of the cited works use LLMs for hierarchical planning in the VLN domain.
• Paap https://aclanthology.org/2022.suki-1.8.pdf
  • Focuses on defining a hierarchical grammar. Doesn’t use LLMs or train a model.
• Parsel https://arxiv.org/pdf/2212.10561.pdf
  • Uses a LLM to decompose a natural task into a bunch of subtasks that is then fed to a codeLM to generate a code
  • There is no policy / feedback from the real world
• Decomposed prompting https://openreview.net/pdf?id=_nGgzQjzaRy
  • Does not look at planning tasks but rather question answering
  • Requires users to manually annotate low-level policies while we autolabel

Why is the web simply not another dataset?

• Web is an increasingly important and challenging domain for decision making
• Many fundamentals challenges to creating web agents:
  • Countless ways to interact with the web, leading to a combinatorially large number of tasks.
  • Web interfaces differ from one website to another. It is intractable to cover all such variations in the prompt.
• We achieve SOTA results compared to prior web automation works and do this with orders of magnitude less data (see Table 1). To achieve this, we had to solve a number of problems:
  • Intractable to cover all tasks and web interface combinations in a single prompt, that we tackle with our hierarchical LLM approach, HeaP
  • Tedious for users to segment demonstrations into low-level policies, that we automate via autolabellng (see Appx D)
  • Complex web pages that can have up to 12k tokens, that we compress to 2k without loss of performance (see Fig. 6)

[1] Liu et al.. Reinforcement learning on web interfaces using workflow-guided exploration, ICLR 2018. https://arxiv.org/abs/1802.08802

[2] Humphreys et al., A data-driven approach for learning to control computers, ICML 2022. https://arxiv.org/abs/2202.08137

[3] Gur et al., Understanding HTML with large language models, arXiv 2023 https://arxiv.org/abs/2210.03945

[4] Furuta et al., Multimodal web navigation with instruction fine tuned foundation models. arXiv, 2023. https://arxiv.org/abs/2305.11854

[5] Yao et al.. React: Synergizing reasoning and acting in language models. ICLR 2023. https://arxiv.org/abs/2210.03629

We thank the reviewer for their feedback! Please find below our responses:

The tasks are not that challenging .. seems that the proposed method struggles with book-flight which is a basic constrained task .. method is very far from being deployed in the real world

Contextualizing work against existing baselines and benchmarks

• We rigorously compare our approach against baselines, including prior works and zero vs few-shot and flat vs hierarchy methods. We evaluate tasks across 4 distinct environments including live websites, totaling ~2500 evaluations.
• We achieve SOTA results compared to prior web automation works [1-5] and do this with two orders of magnitude less data (see Table 1).
• Live websites are challenging since web pages are complex containing ~12k tokens, that we compress down to 2k tokens in text formats for an LLM without loss of performance (see Fig. 6, Appx F).
• The purpose of showing live website evaluations is to show the real-world applicability of our method. However, it is not feasible to do extensive evaluation on the live websites (none of the prior works [1-5] do this either) due to:
  • Bot detectors
  • Safety constraints
  • Evaluating success
  • Require visual reasoning that needs multi-modal models

On MiniWob++ book-flight task

• While prior works scored 0.00, 0.87, 0.00, 0.48, 0.00 on this task, our approach gets a success rate of 0.9 (see task-wise performance breakdown in Table 4, Appx B)
• This task's complexity lies in navigating UI elements such as date grids and applying filters, e.g., "shortest," "one-way vs. round trip," "cheapest", over multiple steps with changing pages (see Fig. 8 for illustration)
• HeaP handles these challenges through dedicated low-level policy prompts like "choose-date", "fill-text," that get composed given a high-level task like "book-flight"

On airlinecrm book-flight task

• Our motivation for building this simulator that we plan to open-source was to simulate longer-horizon tasks with more complex webpages (see Appx E)
• The book-flight task is connected to a mock database and involves >20 steps (as opposed to ~10 steps in MiniWoB++).
• The complexity lies in solving more involved subtasks like adding passenger details, payment information, selecting flight schedule while interacting with diverse UI elements like text boxes, selection links, date boxes.
• We show benefits of both hierarchy and few-shot examples in solving these tasks

Limitations and Open Challenges

We acknowledge there are many limitations and open challenges that need to be solved before our approach can be safely and reliably deployed in the real-world on an open set of tasks. We list these in Limitations (Sec 6) and Broader Impacts (Appx A), in particular,

• Complex webpages. Parsing pages containing long tables, databases, etc needs advanced compression techniques such as learning dedicated saliency models.

• Multimodal observations. HeaP is currently unable to handle pages with visual only components since those observations don’t get parsed from the HTML DOM. Leveraging pretrained multi-modal models offer a promising avenue.

• Verification. Real-world applications would require a verification module to ensure the successful completion of actions, especially in cases where websites may crash or reset. Using LLMs to verify actions is another promising direction of future work.

• Error Recovery. HeaP may click on a wrong link sending it to a new webpage and must learn to recover from such errors. Learning from incorrect actions either via human feedback or self-verification are interesting directions of future work.

• Safety. Finally, action LLMs carry potential for misuse given their execution on open-domain environments, requiring careful security and verification solutions.

We are happy to include or expand out any points here that the reviewer thinks are important!

[1] Liu et al.. Reinforcement learning on web interfaces using workflow-guided exploration, ICLR 2018. https://arxiv.org/abs/1802.08802.

[2] Humphreys et al., A data-driven approach for learning to control computers, ICML 2022. https://arxiv.org/abs/2202.08137

[3] Gur et al., Understanding HTML with large language models, arXiv 2022 https://arxiv.org/abs/2210.03945

[4] Furuta et al., Multimodal web navigation with instruction fine tuned foundation models. arXiv, 2023. https://arxiv.org/abs/2305.11854

[5] Yao et al.. React: Synergizing reasoning and acting in language models. ICLR 2023. https://arxiv.org/abs/2210.03629

We ran evaluations with the llama2-13b model and include results along with common failure modes as Appendix G in the paper. Overall, we find the performance to be lower than gpt-{3,3.5,4}. The drop in performance could be due to a number of reasons such as the model size or the training data on which the models are trained. However, we find that HeaP still outperforms Flat for many tasks. We also observe that the few-shot examples do not improve performance. We think that fine-tuning the model on a larger dataset of demonstrations can help address some of these failure modes and leave that as interesting future work.

Thanks for the detailed response and your efforts for the extra Llama experiments. I have increased the score to 8.