Self-Guided Hierarchical Exploration for Generalist Foundation Model Web Agents

Thanks a lot for your insightful and inspiring comments! We provide the following clarifications in response to your concerns:

1. Human-annotated data

• We would like to clarify that our framework is largely self-guided and relies only on a very small amount of human-annotated data. Specifically, we collected 1,200 trajectories with human annotations to fine-tune the outcome evaluator in the final stage of training. These annotations do not form the core of the agent training pipeline.

• To demonstrate that SAGE remains effective even without any human annotations, we provide an ablation study. As shown in the table below (also refer to the table 2. In the appendix), the performance drop is minor when removing the human-annotated data, confirming that our method’s strong performance is driven by autonomous exploration rather than manual supervision:

• This result shows that even using raw model evaluators without human annotations, SAGE still performs competitively, and the additional fine-tuning only provides a modest improvement.

2. MCTS in open-web environments

• We agree with the reviewer that MCTS typically requires tight control over the environment state. In our framework, we address this by using Playwright to wrap the webpage state, allowing us to store and reload the web page. This enables the reset functionality required by MCTS. Regarding open-web WebVoyager versus the sandboxed WebArena, there is no fundamental difference in how MCTS operates: while some webpages in WebVoyager may occasionally change, they generally remain stable over the short time horizon of a single search process. If a specific branch of the search tree cannot be reliably replayed due to dynamic content, we simply discard that branch and continue the search. We acknowledge that our current approach is less robust for pure JavaScript single-page applications that heavily reuse the same URL, and we consider improving MCTS handling for such dynamic websites an interesting direction for future work.

3. Efficiency of low-level exploration

• We appreciate the reviewer’s question regarding the efficiency of our low-level exploration strategy. In our context, sample efficiency refers to the success trajectories achieved by the agent when generating trajectories under the same number of tasks tried. Compared to a standard rollout strategy, our low-level exploration prioritizes promising action sequences and avoids repeatedly sampling failed trajectories. This results in a higher proportion of successful trajectories for the same number of samples, which translates into a more effective training signal and faster policy improvement.

• To provide direct evidence, we conducted an additional ablation study on WebArena using the same initial SFT models. For each model, we sampled 10k trajectories using either standard rollouts or our low-level exploration. As shown in the table below, the low-level exploration strategy consistently produced more successful trajectories, which validates its superior sample efficiency:

• By enabling targeted search over the action space, our low-level module improves not only the learning speed but also the final task success rates, as confirmed by our main results and ablations in Table 5.

4. SFT claude data

• Different from PAE, which emphasizes demonstrating that the proposer–agent–evaluator framework can improve performance based on SFT models, our focus is on showing that SAGE provides a more effective training paradigm compared with PAE. Due to space constraints, we did not include the SFT baseline results in the main text. For completeness, we now report the SFT performance on WebArena, compared with PAE and SAGE in the table below.

5. Comparison with PAE

• The baseline method PAE establishes a general self-improving framework for web agents by combining three components: a proposer that generates diverse task instructions, an agent that attempts to solve these tasks, and an evaluator that judges task completion to provide rewards. This design successfully removes the reliance on manually curated datasets, enabling web agents to improve through autonomous interaction with web environments. However, PAE samples tasks from a static instruction distribution and does not explicitly address the progression from simple to complex tasks or the challenges of long-horizon exploration.

• Building on this foundation, our work introduces a hierarchical exploration strategy as the core of SAGE. Specifically, we propose 1) a self-guided task proposer that adaptively adjusts task difficulty by analyzing the agent’s performance and decomposing difficult tasks into simpler subtasks, and 2) a low-level exploration mechanism using MCTS-based planning and step-wise learning to improve trajectory quality and sample efficiency. These contributions transform the training process into a dynamic curriculum that allows agents to acquire fundamental skills first and then tackle increasingly complex tasks. As a result, SAGE significantly improves performance compared to PAE, while maintaining the fully autonomous training paradigm.

6. Complexity and reproducibility

• We acknowledge that SAGE involves multiple components, but each module is designed to address a specific challenge inherent to training autonomous web agents. The pre-exploration stage reduces the enormous task space by systematically mapping website structures; the top-level exploration module tackles the difficulty of long-horizon tasks by generating a self-evolving curriculum from easy to hard; and the low-level exploration module improves the agent’s ability to explore and solve complex tasks efficiently.

• To ensure reproducibility, we have already provided extensive technical details in both the main paper and the supplementary material, including full descriptions of the pre-exploration process, task generator prompts, pseudocode for the MCTS rollout, hyperparameter configurations, and evaluator training procedures. Furthermore, we will release the full codebase and trained models upon acceptance to facilitate reproducibility and further research.

Thanks a lot for your insightful and inspiring comments! We provide the following clarifications in response to your concerns:

1. Novelty compared with PAE

• We appreciate the reviewer’s comment on the novelty relative to PAE. While our work inherits the general proposer–agent–evaluator pipeline introduced by PAE, the core contributions and motivations of SAGE are fundamentally different. PAE’s primary goal is to provide a usable framework for web agents to train without relying on predefined task sets, showing that autonomous web training is feasible. In contrast, our focus is on introducing a novel technical approach that allows web agents to learn progressively from simple to complex tasks through hierarchical exploration. This is a key distinction: while PAE establishes the feasibility of self-training, it does not provide an effective mechanism for skill acquisition on long-horizon or complex tasks, often leading to stagnation when the agent fails to make progress on difficult instructions.

• To address this, SAGE introduces a hierarchical exploration strategy that bridges this gap. Our pre-exploration phase systematically builds structural knowledge of the website, enabling the task generator to propose relevant and feasible tasks. The top-level exploration module dynamically adjusts the curriculum, by decomposing tasks with low success rates into simpler subtasks and composing new, harder tasks from high-success ones, while the low-level exploration module, built on MCTS rollouts and step-wise learning, enables the agent to actively search for solutions under sparse rewards. This design allows the agent to bootstrap its skills, mastering simpler operations before tackling complex reasoning tasks.

• Empirically, this technical innovation translates into significant performance gains: SAGE improves over PAE by 7–15% on WebVoyager and 15–25% on WebArena, demonstrating that our hierarchical exploration framework substantially enhances both adaptability and final success rates beyond what PAE can achieve.

2. System efficiency without human instructions

• While fully autonomous learning may seem to enlarge the exploration space, SAGE mitigates this challenge through a structured pre-exploration phase, which systematically maps the interactive elements and navigational pathways of each website. This phase drastically reduces irrelevant exploration by building a compact structural representation of the environment before training begins. Moreover, our top-level exploration module dynamically adapts task difficulty, ensuring that the agent progresses from simple tasks to complex objectives in a guided manner rather than exploring blindly.
• Regarding the use of human annotations, our framework relies on extremely limited human input. The only human-labeled data we use is for fine-tuning the evaluator. Importantly, SAGE remains effective even without human annotations. As shown in the following table (also reported in Table 2 of the supplementary), SAGE trained with only model-generated evaluator signals performs strongly, achieving results close to or better than using human corrections.

3. Comparisons with other strong prompt-based agents.

• We appreciate the reviewer’s suggestion to include comparisons with other prompt-based agents such as GPT-4V. In our experiments, we have already compared against Claude 3.5 Sonnet, which is widely recognized to be on par with or even** stronger** than GPT-4V in web reasoning tasks. Our results show that SAGE-trained open-source models, such as Qwen2.5VL-32B, achieve comparable or even better results than Claude 3.5 Sonnet on both WebVoyager and WebArena benchmarks, demonstrating the strong effectiveness of our approach.
• To further address this point, we additionally provide GPT-4V results on WebArena. Even when compared to GPT-4V, SAGE-trained models achieve competitive or superior performance despite using significantly smaller base models without any manual prompt engineering.
• We also note that SAGE is fundamentally a self-improvement framework rather than a single static agent, and thus its contribution lies in the training methodology rather than the absolute performance of the base model. Our approach can, in principle, be applied to even larger closed-source models to achieve further gains. | Model | Map | Reddit | OSS | Gitlab | CMS | Avg | |------------------------|:----:|:------:|:----:|:------:|:----:|:----:| | GPT-4V | 38.3 | 45.6 | 42.5 | 25.0 | 24.6 | 34.8 | | Claude 3.5 Sonnet | 40.1 | 46.9 | 42.0 | 24.2 | 25.1 | 35.6 | | Qwen2.5VL-32B SAGE | 42.3 | 43.7 | 46.1 | 31.4 | 35.2 | 39.7 |

4. Quality of generated tasks

• Regarding the validity of generated tasks, we do not guarantee that every task is fully valid, as this issue also exists in the original WebVoyager and WebArena datasets. For tasks that turn out to be invalid or infeasible, the agent will output a response indicating that the task cannot be completed, and the evaluator will verify whether this judgment is correct. As a result, invalid tasks do not negatively impact the training process or the final policy quality.
• For the non-triviality of tasks, our top-level exploration strategy adaptively adjusts task difficulty based on the agent’s performance. Tasks with high success rates are merged into more complex ones, while failed tasks are decomposed into simpler subtasks. This approach ensures that tasks remain diverse, challenging, and effective for skill acquisition. Although the task generator is not intended to perfectly match the real-world task distribution, it successfully guides the agent to progress from simple to complex tasks, which is validated by our significant improvements on both WebVoyager and WebArena benchmarks.

Thanks a lot for your insightful and inspiring comments! We provide the following clarifications in response to your concerns:

1. Core contribution of SAGE and comparison with PAE

• SAGE builds upon the proposer–agent–evaluator (PAE) paradigm but addresses key limitations in curriculum learning and long-horizon task decomposition. While PAE establishes a general structure for autonomous training, it uses a static instruction distribution and lacks mechanisms for adjusting task difficulty or improving exploration efficiency.

• PAE’s self-improving loop, where a proposer generates tasks, the agent attempts them, and an evaluator provides feedback, demonstrates that web agents can be trained without human-written data. However, it does not support dynamic task adaptation or skill progression.

• SAGE introduces a hierarchical exploration framework that redefines task generation and learning. It includes a task generator that decomposes failed tasks and synthesizes harder ones from successful trials, enabling adaptive curricula. At the execution level, SAGE employs MCTS-based low-level planning and step-wise policy refinement to improve the quality and diversity of collected trajectories.

• While SAGE retains the core PAE infrastructure for autonomous interaction, it adds a new exploration strategy, including curriculum-based task generation and planning-driven trajectory selection. These components fundamentally reshape training, enabling agents to gradually acquire complex skills and achieve much stronger performance under the same autonomous framework.

2. Hierarchical explanation

• By hierarchy, we refer to the multi-level process that enables agents to solve complex web tasks. At the top level, the agent decomposes tasks into subtasks and learns transferable skills; at the low level, it executes precise interactions (e.g., button clicks or typing) to complete each subtask.

• These levels interact closely: pre-exploration provides structural knowledge for curriculum generation, and low-level rollouts feed back successful trajectories to refine future tasks for the top-level exploration. For example, in Fig 12, the agent fails at a complex task but succeeds after breaking it into simpler goals like locating a project and identifying its contributors. We will revise the paper to clarify this hierarchical structure and add more detailed explanations and cross-references.

3. MCTS algorithm details

• We apologize for the lack of clarity in explaining the details of our MCTS-based training pipeline. Below, we address the reviewer’s questions one by one, and we are happy to provide further clarifications if needed.
• Regarding which components of the pipeline rely on fixed neural networks versus gradient-based learning: In our proposer-agent-evaluator system, both the task generator and evaluator models along with the value remain fixed during the training phase and only adjust the context of their inputs dynamically. The agent model is the only component that undergoes gradient-based updates throughout training. Notably, MCTS serves purely as a planning and sampling strategy to improve the quality of rollouts, but it is not directly involved in optimizing the agent’s parameters.
• Regarding the training objectives: Since we train only the agent, we adopt step-level DPO as our training loss. Let $x$ denote the task prompt, and let $\tau_{\text{win}}$​ and $\tau_{\text{lose}}$​ be two trajectories with the same prefix steps $s_1, s_2, \dots , s_k$, where one trajectory is successful and the other one is fail. Our objective is:

$L(\theta) = - E_{(x, s_1, \dots, s_k,\tau_{win}, \tau_{lose})~D} [ \log ( \beta * \log ( \pi_{\theta}(\tau_{win} | x; s_1, \dots, s_k) / \pi_{ref}(\tau_{win} | x; s_1, \dots, s_k) ) - \beta * \log ( \pi_{\theta}(\tau_{lose} | x; s_1, \dots, s_k) / \pi_{ref}(\tau_{lose} | x; s_1, \dots, s_k) ) ) ].$

where $\pi_{\theta}$ denotes the agent policy and $\pi_{\text{ref}}$ is a fixed reference policy.

• Regarding how the overall pipeline operates: Before the first epoch, we perform a pre-exploration phase to analyze the website’s structure and collect environment-specific information. In each epoch, the task generator produces a set of tasks using the structural information as context. We then randomly sample tasks and use MCTS rollouts to generate successful and failed trajectories. After collecting a sufficient number of trajectories for the current epoch, the agent is updated using the step-DPO objective above. Finally, the success statistics of tasks are fed back into the task generator’s context for the next epoch, allowing it to adaptively refine task difficulty.

• To help clarify the interaction between the agent and MCTS, we have added the following pseudocode to illustrate our low-level exploration strategy:

Algorithm: Low-Level Exploration Algorithm

Input: Question Q, agent model $\pi$, initial page $p_0$, max iterations N, value model V, episode length L

1. $T \leftarrow$ Initialize($p_0$)
2. for $i = 1$ to $N$ do
3.     $s \leftarrow$ root($T$)
4.     while $s$ is not a leaf node do
5.         $s \leftarrow \arg\max_{s'} \left( v_{s'} + \epsilon \sqrt{ \frac{\log n_s}{n_{s'}} } \right)$
6.     end while
7.     if $s$ is not a terminal state then
8.         $a_1, \dots, a_k \leftarrow \pi(Q, s)$
9.         $s'_1, \dots, s'_k \leftarrow$ Apply actions to $s$
10.         $v_{s_1'}, \dots, v_{s_k'} \leftarrow V(s_1'), \dots, V(s_k')$
11.         $s' \leftarrow \arg\max_{s_j'} v_{s_j'}$
12.         $r \leftarrow$ Rollout from $s'$ up to depth $L - depth(s')$
13.         Update $v_{s'}$ and $n_{s'}$ with result $r$
14.     end if
15.     BackPropagate($s$)
16. end for

• We agree that these technical details should be presented more systematically. In the revised version of the paper, we will improve the organization of the methodology section, adding a pseudocode and additional illustrative figures to clearly show the interactions between MCTS, the agent, and the task generator.

4. Fragment presentation

• We appreciate the reviewer’s feedback. Due to space limitations, we initially placed some algorithmic details in the appendix; in the revised version, we will reorganize the content and incorporate key components into the main text to improve clarity and coherence.

5. Accessibility tree explanation

• We would like to clarify that our paper does not mention the term “hierarchical accessibility tree.” We only refer to the standard accessibility tree, which is a browser-defined structure. The accessibility tree is essentially a subset of the DOM tree that contains elements relevant to displaying the content of a webpage. Each element is represented by its role (e.g., link or button), text content, and properties. Compared to the raw DOM, the accessibility tree preserves the structural information of a webpage while being much more compact and directly interpretable.
• We appreciate the reviewer’s suggestion. In the updated version, we will move the figure and explanation currently placed in the appendix into the main text for better clarity.

6. Environment-specific augmentations

• Our approach is designed to work with most modern web environments that are based on standard HTML. The accessibility tree we use is automatically derived from the HTML DOM by the browser, meaning that for any HTML-based website, we can directly obtain both the DOM and the corresponding accessibility tree. Similarly, our BFS-based initial trajectory collection relies on the save and load functionality provided by Playwright, which is generalizable to the vast majority of websites that support session resets or navigation history.
• We acknowledge that our current method may not be fully applicable to certain highly dynamic websites, such as those built entirely with custom JavaScript logic or virtualized rendering where the DOM structure is not fully exposed. Handling these specialized environments remains an interesting direction for future work. In future extensions, we may consider more generalizable state representations—for example, relying solely on raw webpage screenshots (without set-of-marks) and using absolute coordinate-based actions for interaction—so that the framework can adapt to a broader range of web technologies.

7. Confidence training clarification

• In our implementation, the confidence score is represented as a scalar in the range [0,1]. For the initial model, we obtain confidence annotations by prompting Claude Sonnet 3.5 to generate action sequences where each action is accompanied by a confidence score. These annotated trajectories are then used during supervised fine-tuning to initialize the confidence prediction capability of our base model.
• During the subsequent SAGE training process, we further refine the agent model using the step-DPO framework. A refinement is necessary because the MCTS-based sampling process does not always follow the action with the highest predicted confidence from the agent’s policy. To correctly align confidence with actual successful decision paths, we manually adjust the confidence labels in successful trajectories: the executed action is assigned a confidence of 1. This approach ensures that the agent learns to calibrate its confidence scores according to actions that lead to successful task completion.

Thank you for the response. I still don’t fully understand how the task proposer in PAE differs from the one in SAGE once we set aside the MCTS component. Could you please explain the core differences again, ideally with side-by-side pseudocode for the two proposers?

I realise that SAGE first gathers experience with BFT, and PAE doesn't. The SAGE paper states:

SAGE introduces a dynamic feedback mechanism that adjusts the curriculum by decomposing failed tasks and composing harder ones from successful completions

It seems that PAE could make a similar claim. A concise pseudocode comparison would help clarify where the mechanisms diverge.

The term hierarchical is also confusing in this context.

Our hierarchical structure is implemented through the system, not just through prompting. At the top level, we control the space of agent behavior by generating tasks that span different website functionalities and difficulty levels.

PAE and many other task-generation systems could say the same, yet they usually avoid the label hierarchical.

At the low level, we control the agent’s fine-grained decision-making by enabling it to explore multiple concrete actions per state using MCTS.

I’m unsure whether you intend to call MCTS itself hierarchical. In most RL and HRL literature the label hierarchical is reserved for temporally-abstract policies (options, skills) that persist at deployment.

More generally, the term hierarchical is overloaded, especially in RL (e.g., HRL). It would help to clarify that, in your paper, hierarchy refers to the two nested loops that operate during training: the task proposer at the top level and the MCTS-augmented learner at the low level (if I understand correctly).

Thanks a lot for your insightful and inspiring comments! We provide the following clarifications in response to your concerns:

1. High computational resources requirements and error bar

• We thank the reviewer for raising concerns regarding the lack of error bars and statistical analysis. We fully agree that reporting variance is important. However, as in prior work on web agents [1,2], collecting multiple runs for all experiments is computationally prohibitive due to the high cost of real web interactions. This is an inherent challenge in training and evaluating real-world web agents at scale, where each run requires lots of browser-based operations with full model inference.

• To provide evidence of robustness, we conducted 5 independent runs on WebArena for our largest open-source model and the proprietary model. The observed standard deviation across different websites was consistently low, suggesting that performance differences are reliable rather than stochastic. Detailed results are provided below:

• The high computational requirements primarily arise from the intrinsic difficulty of web agent training: strong base models and sufficient task diversity are necessary for meaningful evaluation. While our current compute budget limits the scale of repeated evaluations, we emphasize that our results remain reproducible. To help the community build upon this work, we will release all code, trained models, and necessary configurations upon acceptance. We also believe that larger-scale evaluations can be conducted by organizations with greater compute resources, and we welcome such extensions in future work.

[1] Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents. ICML 2025.

[2] WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. ICLR 2024.

2. Broder Impact

• We acknowledge that our discussion of broader impacts in the original submission was limited, and we appreciate the reviewer’s suggestion to elaborate further. While SAGE focuses on training web agents, the interaction with real-world websites (e.g., during WebVoyager evaluation) could raise potential security concerns, such as accidental access to unsafe websites or unintended information leakage. In our experiments, we mitigated these risks by deploying the agents within a controlled network environment equipped with firewall-like mechanisms and strict network policies to block potentially harmful requests. Furthermore, the agents were not provided with any personal or sensitive information, and all tasks involving user data or authentication were restricted to sandboxed environments (e.g., WebArena).

• It is important to emphasize that our work primarily aims to improve the technical capabilities of web agents in a research context, rather than deploying them for open-ended, real-world usage. For future large-scale deployment, additional safeguards would be necessary, such as enhanced firewall protections, strict access control, human-in-the-loop validation, and anomaly detection mechanisms. We thank the reviewer for raising this point and will add a more detailed discussion of these considerations in the revised version.

3. Safety Issue

• We thank the reviewer for emphasizing the importance of safety concerns. We fully agree that ensuring the safe and responsible deployment of autonomous web agents is a critical issue—not only for our framework but for the field at large.

• In our current work, SAGE is purely a research framework designed to enhance the autonomous learning capabilities of web agents. All experiments are conducted in strictly controlled environments: WebArena is a fully sandboxed simulator, and WebVoyager provides access only to publicly available pages, without involving any private or sensitive user data. Moreover, all web access is routed through a secure firewall layer that restricts and monitors outbound traffic, ensuring that the agent cannot interact with external websites beyond the experimental scope or perform actions with real-world consequences. These safeguards ensure that the agent operates safely within research boundaries.

• That said, we recognize that general safety concerns surrounding web agents, such as unintended interaction with real-world systems, information leakage, or malicious misuse, are of paramount importance. We agree with the reviewer that these concerns warrant deeper discussion. In particular, the potential for LLM-based agents to autonomously explore the web raises novel safety risks that differ from traditional AI systems. These include challenges around authentication spoofing, unintentional automation of sensitive tasks, and reinforcement of harmful behaviors through reward hacking.

• We believe these are important open problems in the design and deployment of general-purpose web agents, and we are committed to addressing them in future work. In the revised version, we will expand our discussion of these issues in a dedicated section. This will include considerations such as: (1) decoupling policy outputs from execution via an intermediate control layer with safety filters; (2) adding a dedicated safety-monitor model to detect and intervene on risky behaviors; and (3) incorporating human-in-the-loop oversight and runtime constraints such as rate limiting and permission gating. We reiterate that our current research focuses on algorithmic capability in safe offline environments, but we strongly agree that future deployment must be accompanied by rigorous safety audits and infrastructure-level safeguards. We thank the reviewer again for highlighting this essential topic.