AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

Thank you for taking the time to review our work and provide detailed feedback! We are grateful that you acknowledge how our work provides a scalable and cost-effective solution for web agent training compared to existing human annotation methods.

We also noticed you have some constructive questions about our work, and we're happy to elaborate further below!

W1: Potiential Overlap with Mind2Web Test Data

A: Thank you for your valuable suggestion regarding overlap analysis! We carefully considered this aspect during data generation and conducted a thorough analysis. Our findings demonstrate that Mind2Web remains a valid benchmark for out-of-domain evaluation even after training with our generated data:

• Mind2Web Dataset Structure:
  Mind2Web includes splits such as train and test_cross-{task, website, domain}, encompassing 137 mainstream websites. Among these splits:

  • The train and test_cross-task splits share the same websites, making it suitable for in-domain evaluation.
  • However, the train split and the test_cross-website and test_cross-domain splits have no overlap in terms of websites, ensuring that website and domain evaluations are genuinely out-of-domain.

• AgentTrek Data Generation:
  In our AgentTrek data generation, we prioritized collecting trajectories from mainstream websites with abundant web tutorials. To assess potential overlap, we analyzed the overlap between the websites targeted in our data and those in the Mind2Web test set. The results are summarized below:

These statistics highlight that the majority of websites targeted during our data generation process do not overlap with those in the Mind2Web test set. Even for the test_cross-task split, considered for in-domain evaluation, the overlap remains minimal. This ensures that Mind2Web can still serve as a robust out-of-domain evaluation benchmark, accurately reflecting the effectiveness of our generated data.

We appreciate your suggestion and have included these detailed statistics in the appendix to provide full clarity. Thank you again for your insightful feedback!

W2: Scale of Trajectories and Insufficient Effectiveness like Mind2Web:

A:

• Thank you for highlighting concerns about the data scale! We fully agree that AgentTrek is a scalable approach with great potential for further dataset expansion. We are actively working on generating more data and are excited to share that we have already verified and collected 10K agent trajectories, which, to the best of our knowledge, represents the largest web trajectory dataset currently available. We are committed to releasing these trajectories in multi-modal format, including screenshots, HTML/AXTREE, videos, and intermediate reasoning processes. We hope these contributions will provide valuable resources for advancing agent research.

• Effectiveness is indeed a critical factor, as you pointed out. Our experiments demonstrate that our data delivers significant performance improvements on Mind2Web. While the synthetic data we introduced is larger in scale than Mind2Web, it is important to note that, as shown in the W1 overlap analysis, our dataset is predominantly out-of-domain relative to Mind2Web. This inherently gives the in-domain Mind2Web-train data an advantage in terms of data efficiency. However, even with this difference, AgentTrek data consistently shows significant performance gains across all splits, whether used for standalone training or in combination with Mind2Web-train. We believe these results strongly validate the effectiveness and value of the AgentTrek dataset.

We sincerely thank you for your recognition of our work! We deeply appreciate your acknowledgment of our data generation process. Furthermore, we are grateful for your recognition of the cost-effectiveness of AgentTrek’s data collection approach. As you noted, the high cost of human annotation is indeed a significant bottleneck, and our method effectively reduces this barrier. Lastly, we are thankful for your recognition of the effectiveness of our collected data. Our data has demonstrated improvements in GUI agents' planning and grounding capabilities across various datasets, outperforming traditional datasets. The approach introduced by AgentTrek will continue to contribute to scalable training for GUI agents and address cost challenges in GUI agent data collection.

We also noticed you have some constructive questions about our work, and we're happy to elaborate further below!

W1: More comprehensive explanation for automatic labeling and tutorial filtering.

A: Thank you for your valuable feedback regarding the need for more detailed explanations of our automatic labeling and tutorial filtering processes.

For the tutorial filtering method, we implemented a systematic approach to extract relevant tutorials from web content using the following criteria:

• Keyword Density: The web content must contain a minimum of 20 common keywords.
• Keyword Diversity: The web context must incorporate at least 4 distinct common keywords.
• Essential Keyword Frequency: At least one critical keyword must appear multiple times (minimum twice) within the context.

Following this filtering process, we then truncate the selected web content to the first 32k characters to maintain consistent input size.

For the automatic labeling, we used prompt based LLM to automatically label the preliminarily filtered web contexts. We have added our prompt used to automatically label in Appendix G.2.

W2: Open-source models to assess the feasibility and effectiveness of data construction within this framework.

A: Thank you for your insightful suggestion about exploring open-source models to reduce costs. In this paper, we implemented several strategies to improve the success rate and lower the cost of generating effective samples, including:

1. Filtering up-to-date, high-quality webpages during the initial data collection phase.
2. Paraphrasing webpages into step-by-step tutorials to make them easier for the model to follow.
3. Optimizing prompts to enhance the evaluator’s recall of positive samples.

These efforts significantly improved cost efficiency, and we appreciate your recognition of this aspect.

We agree that adopting open-source models could further reduce costs and enhance controllability. However, our experiments reveal that current open-source models still lag significantly behind closed-source alternatives in success rates. For example, we conducted replay experiments with Qwen2.5-72B-Instruct, and the results are as follows:

The success rate for open-source models remains substantially lower, leading to a higher cost per positive sample. Consequently, we opted for closed-source models in this study to ensure data quality and scalability.

That said, open-source models hold promise for future exploration. Our WebArena results demonstrate that fine-tuning open-source models with AgentTrek data generated by GPT-4o significantly enhances their performance:

In future work, we plan to explore fine-tuning Qwen2.5-72B-Instruct iteratively to bootstrap the data generation loop. This iterative approach could gradually reduce our reliance on closed-source models while maintaining high data quality. Thank you again for your valuable feedback!

Bootstrap is an excellent approach to enhance open-source models for data construction. Additionally, I appreciate the results provided for OOD environments. My initial concerns were also related to tasks requiring precise control—specifically, coordinate-level control to execute actions effectively (it is very common and the tutorial may can not make effect for the fine-grained action). Given the time constraints for conducting additional experiments, do you have any thoughts or insights on addressing this aspect?

I would like to modify my score higher if this issue can be addressed.

Thank you for recognizing our work! We are pleased that you highlighted the cost-efficiency and scalability of our approach in AgentTrek compared to the existing human annotationn. Additionally, we are pleased that you highlighted the diversity and effectiveness of our collected data, encompassing various task types and web environments. While existing GUI agent datasets are often limited to specific subdomains, our method enables GUI agent data to cover a broader range of domains, making the training of more powerful GUI agents possible.

We also noticed you have some constructive questions about our work, and we're happy to elaborate further below!

W1: Contribution of Various Components (ScreenShots/DOM/HTML/Video, etc.) to Dataset Effectiveness

A: Thank you for highlighting the importance of evaluating the contributions of different modalities in AgentTrek. We fully agree that utilizing these components— screenshots, HTML/AXTree structures, videos —is crucial to comprehensively assessing the dataset’s effectiveness.

AgentTrek supports a variety of modalities, including HTML/AXTree, screenshots, and videos, and its effectiveness has been demonstrated in visual grounding (see Table 5, ScreenSpot) and offline vision-based agent planning with grounding (see Table 6, Multimodal-Mind2Web).

To provide further clarity, we conducted an additional experiment to evaluate the performance of a pure textual agent using textual AgentTrek trajectories. This allow us to study the contribution of textual modalities of AgentTrek.

We fine-tuned the Qwen2.5-7B-Instruct model using AgentTrek trajectories that included accessibility tree as observations and playwright actions as the agent's action space. We then evaluated the model on WebArena, an OOD web agent benchmark featuring self-hosted websites. These websites are entirely out-of-domain (OOD) from the AgentTrek dataset, ensuring that the evaluation reflects the model's generalization capability.

We fine-tuned Qwen2.5-7B-Instruct on AgentTrek’s textual data and achieved the following results:

We found that:

1. Textual Effectiveness: AgentTrek’s textual trajectories significantly boost performance, surpassing open-source baselines and nearing GPT-4o.
2. OOD Generalization: Strong results on WebArena confirm that AgentTrek’s data generalizes well to unseen domains.

While this shows the value of textual data, we recognize the importance of further quantifying contributions from other modalities (e.g. videos) and plan to explore this in future work. Thank you for your suggestion!

W2: Expand Evaluation Benchmarks

A: Thank you for your valuable suggestion regarding evaluation benchmarks. We understand the importance of broadening evaluations, and we hope the WebArena experiment presented in W1 addresses your concerns. WebArena is a challenging, online, out-of-domain (OOD) benchmark featuring self-hosted websites that are entirely unseen during training. Our results demonstrate that AgentTrek data not only improves web agent performance across multiple modalities but also generalizes effectively to WebArena OOD scenarios, highlighting its real-world applicability.

We also evaluate our model on MiniWoB++ as you mentioned, which also demonstrate the effectiveness of Qwen2.5-7B-Instruct w/ AgentTrek.

[1] Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale, Ou et al., 2024