Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale

Thank you for taking the time to review our paper! We are very happy to hear the encouraging words on our motivation, creative usage of data sources, and performance improvement. We believe the comments are resolvable during the rebuttal period. Please see our response below. We would be very happy to address additional questions during the discussion period if anything is unclear.

How could Synatra possibly help push SOTA?

Is synatra limited to distilling agent-capability from stronger models to weaker models?

Varying the size of the dataset.

This is a great question, we believe Synatra holds the potential of pushing SOTA when scaling up. We additionally train a CodeLLama-7b on 100$k$ synthetic data and the results on different size of synthetic data is below:

We can see that training on more synthetic data provides 1.48% improvement on WebArena, surpassing GPT-3.5. In addition, among the 51 tasks Synatra-CodeLLama-7b got correct, we found GPT-4-turbo failed on 57% of them. For instance, while GPT-4 suffers from severe errors such as repeatedly typing the same content to the same input box (26.85%), this only happens 6.9% of the time for Synatra-CodeLLama-7b. This indicates our approach could assist models in performing tasks the data generation model (e.g., GPT-4-turbo) cannot accomplish.

Can we compare the model trained on smaller set of human annotations on roughly the same task distribution as synatra?

We agree it will be interesting to have such experiment, however, as discussed in the introduction section, collecting human annotations of similar task distribution can be challenging. The main difficulty is on configuring the task prerequisites. For example, in Synatra, we have synthetic trajectories to cancel Amazon order. However, it requires an Amazon account with qualified order for cancellation for human trajectory collection. Such setup is instance specific. Hence, it is prohibitive to collect human annotations at scale as in Synatra. In the paper, we also note that “it is important to recognize that the quality of human demonstrations is presumably higher, but they require meticulous design and control during the data collection process.” Hopefully this could further clarify the challenges in human data collection, and highlight the simplicity of our proposed approach.

I'm not sure if you can claim in the abstract that "we show that the synthetic demonstrations can be more effective than a similar number of human demonstrations."

We will soften the claim to only compare with human demonstration collected in a dataset-agnostic way, meaning that the data collection process is not meticulous design and control.

What stopped you from finetuning GPT-4 or GPT-3.5 on synatra?

On the one hand, the cost of finetuning GPT-3.5 or GPT-4 on 89 millon tokens for a few epochs is prohibitive for us. On the other hand, we hope to push the frontier of open-weight models. We have spent more efforts on scaling up the data size as discussed above, and we are working on scaling up the model to stronger open-weight models.

6.3 rather ad hoc

Thank you for pointing this out. We will add more quantitative study in this section and move the case study to Appendix in our camera ready version.

Open-souce -> Open-weight

Thank you for the suggestion. We will update the paper to reflect the accurate wording.

Thank you for taking the time to review our paper! We are very happy to hear encouraging words on our domain agnostic data synthesis approach that fills in the current gap. We believe the comments are resolvable during the rebuttal period. Please see our response below. And we would be very happy to address additional questions during the discussion period if anything is unclear.

Comparison with baselines

Comparison with LLMs finetuned with direct demonstrations (human annotated)

Please Refer to the same section in global rebuttal due to character limit.

Comparison with approaches that achieves high performance on MiniWoB++ or Mind2Web

Thank you for bringing this up, we are aware of the many models and methods, we did not include them as baselines because of the following reasons:

• Many approaches incorporate dataset-specific designs, making them challenging to apply to generic web-based tasks like WebArena. For instance, MindAct trained only with Mind2Web training data, as we can see with Mind2Web-CodeLlama-7b, while the model achieves strong performance on Mind2Web test set, it has limited performance on all held-out datasets. Similarly, Synapse uses trajectories as examples in the context; its prompting schema heavily relies on in-domain examples for MiniWoB++ and Mind2Web. In these datasets, the trajectories are short and can fit within the context window, and they offer relatively large training data on similar tasks or websites. However, in WebArena, trajectories can easily exceed the context length, and there are no provided training trajectories.
• Many approaches are orthogonal to our data synthesis approach. The key component in MindAct is the ranking model that reduces a full HTML to the top-k potentially relevant elements, to remove irrelevant context. It is possible to apply a similar two-stage pipeline to our approach, instead of feeding the full AXtree into the model, we can first perform a reranking to select a subset of nodes in the AXtree instead. Alternatively, we can simply generate key elements rather than the full HTML as the observation.
• Some approaches do not have public access. At the time we wrote the response, HTML-T5 is not open-sourced yet. We will incorporate the additional experiments to our updated draft. We are happy to investigate other approaches if we missed any.

Can you further fine-tune on demonstrations from available benchmarks? How much does it improve when human demonstrations added?

Finetuning with both human demonstrations and our synthetic data is straightforward because our data design is general for web-based tasks. Further finetuning the model on high-quality human demonstrations can presumably bring significant gain. However, such data is scarce, and we hope our work can encourage the creation of more high-quality data. In addition, our experiment shows that adding human demonstrations that are collected without meticulous control may not improve the model’s performance on general web-based tasks. As shown in the table in global rebuttal, simply training the model on human demonstrations collected in Mind2Web results in very limited performance on the two held-out sets. We also experiment with adding Mind2Web data into our synthetic data, and we also see a similar overfitting trend.

Could assigning unique “id” result in overfitting when detecting salient elements?

We peformed a post processing to replace the unique ID with a random integer number in the final data. Hence, the model will not overfit to any specific ID.

You mention that you use viewports in Mind2Web benchmark to fit into the context window. But to extract viewports, you need to know where the ground truth element resides in the DOM (or accessibility tree) or you assume you have access to an oracle scrolling agent. I think both cases are different from your baselines and suggests leakage of test data.

The main reason of doing this processing is to make sure the web page observation can fit into the context window of a model. We use the same processed web page to all baselines and our approach. Hence it is a fair comparison.

When you use NL format rather than programs, how do you ground generated actions to executable environment actions?

The NL format we uses still requires the generated action to be in set formats. Here is an example:

Past actions:

1. click [4904] where 4904 is img calendar wheel with marker

For actions to be parsed accurately, they still need to be in the format such as click [xxx]

Performance of the model trained with Mind2Web demonstrations on Mind2Web test sets?

The performance of the model trained with Mind2Web demonstrations on Mind2Web test sets is as followed:

When tested on other benchmarks, how do you ground the action on the particular benchmark?

We follow WebArena to represent a web page as an accessibility tree with ID at the beginning of each tree node. This can be done by processing the HTML in the three benchmarks. In this way, we can refer to any element with its corresponding ID such as click(element_id=123).

It is not clear if the claim that collected demonstrations have a similar distribution as Mind2Web as they have fatter tails.

Sorry for the confusion. Our goal was to demonstrate that our approach allows us to freely fit our generated trajectories to arbitrary distributions, such as Mind2Web. We will update Section 4 to clarify this.

Text and legend in Figure-2 are not readable.

We will make sure to update the figures in the camera ready version.

Thank you for taking the time to review our paper! We are very happy to hear encouraging words about our motivation and effectiveness of our method. We believe the comments are resolvable during the rebuttal period. Please see our response below. We would be very happy to address additional questions during the discussion period if anything is unclear.

Comparison with RAG-based approach

Thanks for pointing out this. We agree that the RAG-based approach is an alternative method of utilizing existing knowledge. Below is our comparison with the RAG-based approach. Specifically, we used the same collection of tutorials as the retrieval pool and employed cosine similarity of semantic embeddings of task descriptions to find the three closest tutorials. These tutorials were then included as additional context when prompting LLama3.1-instruct-8B to complete the tasks. Note that many RAG methods rely on benchmark specific in-context examples, we do not use any in this case. The results on WebArena are shown below:

Comparison with LLMs finetuned with more direct demonstrations

Please Refer to the same section in global rebuttal due to character limit.

Details about the method, e.g., how is the LLM fine-tuned and the training hyper-parameters; the configurations about baselines and evaluation dataset

We apologize for the missing details. The details of the training are listed in Appendix I. The CodeLlama checkpoints are fine-tuned with A100 GPUs with deepspeed acceleration framework. We set the context length to 4096 tokens. To train on 50k dataset, we train with 6 x A100 80G GPUs for about 20 hours. We use a batch size of 48, and learning rate of 5e-5. We use cosine annealing and a warm-up ratio of 0.03. We use supervised fine-tune with full-parameters. We decided on this rather than lora from early experiment comparing the two’s performance. We only calculate loss on the response of the agent, while the observation, task title, past histories etc. are masked in loss calculation. We will update section 5 to properly refer to the corresponding section in appendix I. We will update the details of baseline and evaluation setup in section 5 as well. For WebArena, we evaluated with a temperature of 0 and default set of parameters. For MiniWoB++, we evaluated on a subset that is the same as what's used in AgentLM/CodeActAgen. We take the five run-average success rate. We re-ran all the baseline models with the same setting for fair comparison. We will update these details in section 5 and appendix. And we will also release our training/evaluation code.

How is success rate calculated in these web tasks?

Both benchmarks provide validators to programmatically check if a task is completed or not inside their corresponding environments. For instance, if a task is to put a certain item into the shopping cart, a program check would be to verify if the said item is in the cart by the end of the trajectory.

What is the source of performance improvement?

Our analysis shows that synthetic data teaches the model on multiple levels. First, it provides an understanding of basic task formats. Initially, the base model had issues such as repeating actions, interacting with nonexistent elements, or performing invalid actions not specified in the prompt. After training, these errors significantly decreased, approaching GPT-4 levels. The detailed breakdown is shown below:

Second, synthetic data enhances the model’s understanding of web pages. For instance, the base model correctly typed into a typable element (e.g, a input box) less than 75% of the time. Post fine-tuning, this accuracy improved to over 93%. Lastly, we qualitatively examined model’s planning on the task before and after training, we found that synthetic data yields more accurate task planning and hence improve task performance.

Does the improvement come from GPT-4 trained on the test tasks?

It is highly unlikely that GPT-4 was trained on the exact test tasks. Even if GPT-4 encountered some of the test data during training, our approach does not rely on its prior knowledge of them. Our data synthesis process is entirely independent of the evaluation datasets, including WebArena, MiniWob, and Mind2web. We did not utilize any specific test tasks or testing environments, such as the websites in WebArena, during data generation. Instead, we leveraged GPT-4 to transform randomly sampled wikiHow articles, targeting general web-based tasks, and randomly sampled web pages. Given the vast volume of web content, it is improbable that the sampled pages would overlap with those seen during the test.

Thank you for taking the time to review our paper! We are very happy to hear the encouraging words about the work being novel and addressing the lack of demonstrations in LLM finetuning for agentic tasks. We believe the comments are resolvable during the rebuttal period. Please see our response below. We would be very happy to address additional questions during the discussion period if anything is unclear.

The comparison with self-refine

The core difference between our approach and self-refine is that we apply LLM refinement for different purpose. Our proposal is a data synthesize approach that leverage LLM to transfer non-demonstrations into direct demonstrations and further finetune a model on the synthesized data. Our goal is to enhance a model’s performance on computer tasks. On the other hand, self-refine focuses on refining a model's response to a given prompt, it is a prompting mechanism. Self-refine does not involve generating training data and finetuning.

Comparison with models finetuned on their own dataset

Also included in global rebuttal

Direct demonstrations for web-based tasks are scarce, resulting in a limited number of LLMs fine-tuned with such data. We made our best effort to survey or implement approaches that are fine-tuned with direct demonstrations. We include three additional baselines and the results are shown below:

Here, AutoWebGLM-7b-42k is finetuned with their 42$k$ synthetic direct demonstrations. The result is taken from [1]. Mind2Web-CodeLLama-7b is finetuned with the Mind2Web training data of 9$k$ direct demonstrations. The finetuning is done by us with the same configurations as Synatra. For RAG-Llama3.1-Instruct-8b, we used the same collection of tutorials as the retrieval pool and employed cosine similarity of semantic embeddings of task descriptions to find the three closest tutorials. These tutorials were then included as additional context when prompting LLama3.1-instruct8B to complete the tasks. We aims to compare different ways of using existing knowledge with RAG-Llama3.1-Instruct-8b We can see Synatra achieves the best performance on all held-out datasets. We also note that finetuning datasets used in AgentLM, CodeActAgent include a reasonable amount of out-of-box web-based demonstrations. In fact, Mind2Web training set made up 66% of the examples in AgentLM’s training set. We will incorporate the additional experiments to our updated draft. We are happy to investigate other approaches if we missed any. [1] Lai et al. Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent.

Discussion on perpetual data machine

Creating perpetual data machine is an exciting and open research question. Within the scope of our work, we propose leveraging existing resources. Currently, massive text corpora are primarily used for next-token prediction during pretraining, yet each piece of text can embed more diverse information. As models continue to improve through iterative training, they may be able to extract additional information or transform knowledge to enhance their capabilities.