NNetscape Navigator: Complex Demonstrations for Web Agents Without a Demonstrator

We thank Ezws for their thoughtful feedback. We provide clarifications below:

Clarity on the reward model for the instruction-first baseline: Yes, we confirm that we use the same reward model described in Lines 199-200 for both the instruction-first baseline and NNetNav. While this is briefly mentioned in Lines 138-146, we apologize for any lack of clarity and will revise the draft to make this more explicit.

Using the reward model for filtering: Please see our response to Q1 below for a detailed response on how the reward model is used.

Performance gains minor for certain websites: We attribute this to limitations in the WebArena success rate metric. When we utilize a model-based evaluator, we observe more consistent performance improvements across all websites as can be seen in Table-2. Notably, such model-based evaluators are becoming more common in the community (Pan et al. 2024, Zhuge et al. 2024).

Comparison with previous methods that report results on WebArena:

Thank you for the suggestion. We are happy to provide comparisons with prior results on WebArena using the official leaderboard [https://shorturl.at/TFf5q]. Except for AutoWebGLM, which heavily leverages human demonstrations, all methods in the sub-10B category achieve below a 7% success rate. In contrast, our model achieves a 7.2% success rate, establishing a new state-of-the-art at this scale without relying on human demonstrations. Additionally, following feedback from other reviewers, we conducted a new experiment where we fine-tuned LLaMA-3-8b on NNetNav-6k. As detailed in Section 2, the 6,000 demonstrations are expanded into over 77,000 training instances due to multiple training instances per trajectory. Using the same hyperparameters as used in the paper, fine-tuning LLaMA-8b-instruct resulted in a WebArena success rate of 10.3%—a significant increase over the previously reported 7.2% (where we used a smaller dataset for fair comparisons with other baselines). This result surpasses previously published results for sub-10B models using synthetic datasets and, to our knowledge, establishes our model as the state-of-the-art among agents that do not use closed-source models (like GPT-4 or GPT-3.5), or human demonstrations:

• LLaMA-8b-NNetNav [ours]: 10.3% (6k demonstrations)
• AutoWebGLM-7b (S1): 2.5% with 240 demonstrations
• Synatra-CodeLLaMA-7b: 6.3% with 30k demonstrations
• Patel et al. (2024): 9.36% with Qwen-1.5-72B. Our results surpass those reported by Patel et al. (2024), a prior work mentioned by nEkA, who used a significantly larger Qwen-1.5-72B-Chat model.

These results highlight that our approach is the current state-of-the-art among methods relying solely on synthetic demonstrations, without utilizing closed-source models. We hope these updated comparisons offer a clearer perspective on the advancements made by our work.

Next, we answer questions from Ezws below:

Using the reward score: We appreciate the inquiry regarding the reward score. To clarify further, our reward model uses a prompted LLM that takes the trajectory summary and the instruction as input, and outputs a score between 1 and 5 (see the prompt in Listing 5 of the appendix). If the score is less than 4, the output is set to 0; otherwise, it is set to 1. This determines whether we should prune or continue exploring, as outlined in Algorithm-1. Notably, the threshold is not manually set; rather, we rely on the language model to generate an appropriate rating. While this method may introduce some noise, it performs well in practice.

Filtering for Instruction-first SFT: Yes, the same reward model is used for filtering demonstrations in the Instruction-first sampling. Here, a demonstration is kept only if the reward score is 1; otherwise, it is discarded. We will clarify this more explicitly in the manuscript (though it is briefly mentioned in Lines 138-145).

Using different LMs for NNetNav and training: Indeed, the same LLM can be used for data collection, training, and inference. We have already conducted this experiment and report the results in Table-3 (see Lines 405-411), using LLaMA-3-8b for both data collection and inference. Here, we find improvements of over 4 points in webarena success rate, entirely from self-training.

Improvements on Reddit and Gitlab: We do observe improvements on Reddit and Gitlab (refer to Table-2). However, due to inherent limitations in the WebArena success rate metric—which hard-codes certain solutions and does not provide partial credit—these improvements may not be fully reflected. For instance, even if a model's performance on a task improves from 0% to nearly 90%, the success rate remains 0% under the current metric. A graded evaluation, however, would capture these gains.

We hope these clarifications adequately address your concerns, and we are committed to revising the manuscript to enhance clarity based on your valuable feedback. Thank you once again for your insightful comments.

I thank the authors for their thoughtful responses and for clarifying the instruction-first baseline and the use of different LMs. The proposed approach of relabeling and rejection sampling is interesting, and I acknowledge the performance of the sub-10B models using synthetic datasets. However, I remain concerned about the method's limitations, given the lack of discussion and analysis of failure modes and scalability compared to other approaches. As such, I will maintain my rating marginally below the acceptance threshold.

We thank nEkA for their review and feedback. We address points raised by nEkA below:

Differences with BAGEL

We appreciate the opportunity to clarify the differences between BAGEL and our work. We would like to draw nEkA’s attention to Lines 510-515 of the manuscript, where we discuss these differences. Additionally, as mentioned in the abstract (Lines 015-017), prior interaction-first techniques, such as BAGEL, do not address efficient exploration in complex environments. For convenience, we provide a summary of the differences below:

• Algorithmic Differences: BAGEL employs a flat exploration strategy effective for simpler environments like MiniWoB++ but becomes inefficient on complex, real-world websites. In contrast, NNetNav introduces a novel exploration method that leverages inferred sub-tasks to prune exploration paths early, significantly improving efficiency. As demonstrated in Figure 2, this results in nearly halving the number of interactions with the environment.
• Experimental Differences: While BAGEL is limited to simpler environments, our experiments extend NNetNav’s application to complex, real-world websites. Additionally, we fine-tune large-scale models (8 billion parameters) for our experiments, whereas BAGEL uses a RAG-based approach. Further, we are releasing NNetNav6k, the largest dataset to date for demonstrations on WebArena. In contrast, BAGEL has not released a dataset. While both approaches share a high-level motivation centered around interaction-first data collection, the specific methodologies and experimental setups are notably distinct. We hope this strengthens nEkA's confidence in the novelty of our approach.

Performance compared to prior work

• We would like to clarify that the methods by Kim et al. and Furuta et al. are not unsupervised for MiniWoB++. Specifically, Kim et al. utilize domain-specific prompts, while Furuta et al. leverage the reward function. Similarly, Zheng et al. rely on human demonstrations for in-context learning. More importantly, these works use significantly larger closed-source models, such as GPT-3.5 and GPT-4, whereas we employ a much smaller, open-source LLaMA-8b model and achieve notable performance improvements of 20 points through fine-tuning with NNetNav demonstrations.
• Regarding WebArena, we respectfully note that prior works mentioned by nEkA rely on closed-source models like GPT-3.5/GPT-4. Additionally, Patel et al. (2024) employ Qwen-1.5-72B, which is ten times larger than our model. For results on models within the sub-10B scale, we direct nEkA to the WebArena leaderboard [https://shorturl.at/TFf5q]. Notably, except for AutoWebGLM, which uses extensive human demonstrations, all methods in the sub-10B category achieve less than a 7% success rate. In contrast, our model achieves a 7.2% success rate, establishing a new state-of-the-art at this scale without human demonstrations.

Effectiveness of NNetNav-6k

We respectfully clarify a misunderstanding regarding the NNetNav-6k dataset. While NNetNav-6k consists of 6,073 demonstrations, each demonstration (instruction, trajectory) generates multiple training instances based on the trajectory length. Thus, NNetNav-6k results in over 77,000 training instances, as described in Section 2 (Lines 138-145). In response to nEkA’s suggestion, we have now extended our experiments to utilize the full NNetNav-6k dataset. Using the same hyperparameters, we fine-tuned LLaMA-8b-instruct and achieved a 10.3% success rate on WebArena, a significant improvement over the 7.2% reported in Table 1, where we used a smaller dataset for fair comparison with other baselines. This result surpasses previously published results for sub-10B models using synthetic datasets and, to our knowledge, establishes our model as the state-of-the-art among agents that do not use closed-source models (like GPT-4 or GPT-3.5), or human demonstrations:

• LLaMA-8b-NNetNav [ours]: 10.3% (6k demonstrations)
• AutoWebGLM-7b (S1): 2.5% (240 demonstrations)
• Synatra-CodeLLaMA-7b: 6.3% (30k demonstrations)
• Patel et al. (2024): 9.36% (Qwen-1.5-72B). Moreover, our performance exceeds that of Patel et al. (2024), a prior work mentioned by nEka, who used a much larger Qwen-1.5-72B-Chat model:

We believe that these new results affirm the effectiveness of our approach, and we hope this addresses nEkA’s concerns and reinforces confidence in our contributions.

Thank you authors for the detailed response, and I really sorry for the late reply due to my personal matter.

> Re: W1

Thank you for listing the difference between BAGEL and NNetNav. However, I think current writing emphasizes that demonstration-first data collection method is what you newly introduced in this paper (Section 3), and you compare it against instruction first methods. I believe, instead of those high level concept, the difference you listed in the response is the novelty of this paper, and you should mention the relationship to BAGEL explicitly before presenting your method. Otherwise, the contribution of BAGEL is unfairly ignored. For the hirerachical exploration method, you mentioned it from the efficiency perspective. However, I think it is unclear whether this efficiency contribute to the performance improvement (maybe somewhat inefficient method may achieve better results).

> Re: W2

I think that Web Agent is a highly application-focused research area. Withouht very convincing reasons (here, this should be related to a performance-cost tradeoff), I dont't think the method with sub-optimal performance and lower cost is prioritized against the method with optimal performance and higher cost.

For the baselines, Patel et al. (2024) employ Qwen-1.5-72B, but also apply LoRA for the finetuning. In contrast, I guess you employ full-parameter finetung. Because both two are still tractable approach to some extent and LoRA v.s finetuning causes performance tradeoff, this discussion may not provide convincing evidence.

> Re: W3

Thank you for the clarification on the dataset. For clarity, you may benefit from chaning order of section 6 into forward.

We sincerely thank YpQW for their review. Below, we address some comments:

Using heuristics:

We would like to respectfully draw YpQW's attention to more mature literature on self-supervised exploration, where collecting synthetic demonstrations using exploration policies is a well-established practice. We agree that in particularly challenging environments, $\pi_\text{explore}$ may struggle to gather meaningful trajectories. We also acknowledge that our LM summarizer could potentially overlook important details when comparing differences. However, for websites, where the task is to summarize differences between two accessibility trees, we have found that this method works effectively.

Collecting data in a completely unsupervised manner, particularly in difficult sequential decision-making environments, remains an open problem. Our paper specifically addresses the collection of demonstrations on websites where our practical heuristics have proven effective. We do not claim generalization to other types of grounded environments.

Using human demonstrations:

We agree that incorporating human demonstrations is an interesting avenue but falls outside the current scope of this work. To date, there is no large-scale dataset of human demonstrations for complex web-based tasks. Additionally, collecting such demonstrations at the same scale as NNetNav would be significantly more resource-intensive, as it requires training annotators with domain-specific knowledge (e.g., GitLab requires understanding the basics of version control software).

Why not train on the larger set of demonstrations:

Thank you for this suggestion. We have now extended our training to include the full NNetNav-6k dataset. As detailed in Section 2, we convert the 6,000 demonstrations into over 77,000 training instances, as each demonstration generates multiple training examples based on trajectory length. Using the same hyperparameters, we fine-tuned LLaMA-8b-instruct, resulting in a WebArena success rate of 10.3%—a significant improvement over the 7.2% reported in Table 1 (where we used a smaller dataset for fair comparison with other baselines). This represents substantial progress over previously published results for sub-10B models using synthetic datasets and, to our knowledge, establishes our model as the state-of-the-art among agents that do not use closed source LLMs (like GPT-4 or GPT-3.5), and with no additional human supervision:

• LLaMA-8b-NNetNav [ours]: 10.3% (6k demonstrations)
• AutoWebGLM-7b (S1): 2.5% with 240 demonstrations
• Synatra-CodeLLaMA-7b: 6.3% with 30k demonstrations
• Furthermore, our results exceed those of Patel et al. (2024), a prior work mentioned by reviewer-nEkA, who used a significantly larger Qwen-1.5-72B-Chat model: Patel et al. (2024): 9.36% with Qwen-1.5-72B

We hope these updated results more convincingly highlight the downstream utility of NNetNav-6k, demonstrating not only its uniqueness as the first dataset of complex instructions on WebArena websites but also the effectiveness of our approach. We believe these findings will strengthen your confidence in the quality and impact of our dataset.

Comparison to NNetNav (baseline): To clarify, both SFT (NNetnav + distil.) and SFT (NNetnav) are methods proposed in our work, so we are neutral regarding which performs better. However, SFT (NNetnav + distil.) is more resource-intensive, requiring twice the environment interactions and an additional round of LLM sampling per instruction. We agree that further exploration of rollout differences between NNetNav trajectories and instruction-following ones would be insightful. We have included preliminary experiments on this in Lines 354-366 of Section 5. If YpQW has specific suggestions, we would be happy to conduct and report additional experiments.

We thank ezyY for their thoughtful review. We appreciate your positive feedback on the ideas presented in our paper. Below, we address your concerns with clarifications and updates:

Notation and clarity

Thank you for your comments on notation. In addition to fixing the draft, we would like to provide a few clarifications here:

• Defining Plausible: We appreciate the suggestion for a clearer definition. In our context, we use the term "plausible" to refer to instructions that align with the genre of a given website (e.g., a shopping instruction on an e-commerce website or a "comment on a post" instruction on a social media site).
• Quality of generated instructions: This is an excellent point. We use Figures 3 and 4 to illustrate aspects of diversity, while Table 5 provides specific examples of our generated instructions. Additionally, we have included new results obtained by fine-tuning LLaMA-8b on NNetNav data, which further supports quality claims. We agree that evaluating instruction quality comprehensively remains an open challenge, and we appreciate your acknowledgment of this complexity.
• About $\mathcal{A}$: We apologize for any confusion related to this notation. Based on your suggestion, we will simplify it in the revised version. The action space of the LLM is indeed finite but quite large, as many actions refer to an accessibility tree ID, with some observations containing up to 1,000 such IDs.

Reward model:

We apologize for the typo and any resulting confusion. To clarify, we employ two distinct reward models:

• The first is a binary reward model used for NNetNav data creation (referenced in Lines 199-200).
• The second is a model-based evaluation (described around Line 343) using GPT-4v, which assigns scores from 1-5 based on a different prompt. This model is solely used for evaluation. Notably, there is a typo in Line 344, where we mistakenly referred to Appendix A.2 instead of the correct Appendix B, which contains the prompt for this model-based evaluator.

Cross-website generalization:

We would like to clarify the context of our cross-website generalization experiment. Our intention here is not to demonstrate the effectiveness of NNetNav but rather to explore the generalization capabilities of the base LLM. Specifically, we split our data to train on NNetNav demonstrations from Shopping, Maps, and CMS, and then assess generalization to instructions on unseen websites. While NNetNav can be applied to any website, this section aims to evaluate how well current language models generalize from demonstrations on specific websites to new ones.

Using NNetnav-6k:

Thank you for your comments. We have now trained language models using the entire NNetNav-6k dataset. As outlined in Section 2, we convert the 6,000 demonstrations into over 77,000 training instances since each demonstration results in |trajectory| training examples. We fine-tuned LLaMA-8b-instruct with the same hyperparameters as presented in the paper and Appendix D, achieving a WebArena success rate of 10.3%. This represents a substantial improvement over the results in Table 1 (7.2%), which focused on a smaller dataset for fair comparison with other baselines. Notably, this model is the state-of-the-art among approaches that do not use GPT-4 / GPT-3.5, and use no additional human supervision:

• AutoWebGLM-7b (S1): 2.5% with 240 demonstrations
• Synatra-CodeLLaMA-7b: 6.3% with 30k demonstrations
• Patel et al. (2024): 9.36% with Qwen-1.5-72B. Furthermore, our results exceed those reported by Patel et al. (2024), which used a much larger Qwen-1.5-72B-Chat model.

We hope these updated results more effectively demonstrate the downstream utility of NNetNav-6k, beyond being the first and only dataset of complex instructions on WebArena websites.