Learn-by-interact: A Data-Centric Framework For Self-Adaptive Agents in Realistic Environments

We thank the reviewer for recognizing the novelty of the backward construction in synthesizing data and are glad to hear that the reviewer finds the paper well-written. Below we address the concerns raised by the reviewer. We used purple text to clarify the parts that we modified in the revision.

Weakness 1: First, we would like to clarify that, instead of just in-domain documentation, Learn-by-interact leverages broader resources including software manuals, website FAQs, operation tutorials, and etc. This significantly widens the applications of the proposed approach compared to only considering the documentation. In general, we believe that this setup is a common scenario in realistic environments, as human users frequently use various resources to find solutions. For example, students may look up textbooks to understand a theorem, shopping website users may browse FAQs to check the procedure for returning an item, etc. In this paper, we demonstrate that such resources are available in tens of environments, which indicates the generalizability of Learn-by-interact in practical applications.

Additionally, we do not assume that a pre-trained LLM is familiar with the target environment. Instead, we use a general-purpose LLM and augment it with environment-related resources to generate task instructions. This indicates that a wide range of LLMs could be potentially leveraged to synthesize data following the procedures in Learn-by-interact.

Weakness 2: We thank the reviewer for pointing out two missing related works in the paper. We add the discussions and citations in section 5. AgentGen focuses on synthetic environments generated by LLMs, while AgentTuning leverages existing datasets and self-instruct to create task instructions. In contrast, Learn-by-interact targets at realistic and complex settings and synthesizes data from scratch based on diverse resources including tutorials, documentation and more.

Weakness 3: We agree with the reviewer that Learn-by-interact consumes a lot of LLM calls to generate data. However, we would like to clarify: (1). Although the number of LLM calls in backward construction has quadratic complexity in terms of the original trajectory length, it is only linear in terms of the number of generated examples, because each pass of backward construction will correspond to a new instance. (2). In Figure 2, we demonstrate that Learn-by-interact is much more efficient compared to other agentic approaches. It archives significantly higher performance with remarkably fewer LLM calls and consumed tokens. (3). Learn-by-interact is fully autonomous without using any existing datasets or human annotation, which could be even more costly compared to LLM calls in the paper.

Thank you very much for the insightful review! We are glad to hear that the reviewer recognizes the potential of the proposed method and finds that backward construction improves data collection efficiency. Below we address the concerns and questions raised in the review. We used purple text to clarify the parts that we modified in the revision.

Question 1: We agree with the reviewer that, in terms of the original trajectory length, the complexity is quadratic. However, each pass of backward construction will correspond to a newly generated example. In other words, for an original trajectory with length L=len(T), we need O(L^2) LLM calls to generate task instructions, which produce O(L^2) new instances. Therefore, in terms of the number of synthesized examples N, the complexity of backward construction is linear O(N).

Figure 2 applies to inference only, which illustrates the efficiency of various agentic approaches in downstream applications. We modify the caption to clarify this point.

Question 2: In Algorithm 1, line 11, the function L refers to LLM, the same LLM backbone. We correct this typo in the revision.

Question 3: We agree with the reviewer that the drop rate is high in OSWorld and Spider2-V. One explanation attributes this to the increased difficulty in these two benchmarks, as evidenced from the lower baseline performance in Table 2. With harder tasks, we would expect a lower percentage of trajectories to satisfy the criteria to be high-quality data, which leads to a higher drop rate.

Question 4: Following the reviewer’s suggestion, we re-filter the data in OSWorld and Spider2-V and consider a synthesized instance of high quality if any of LLMs in the committee finds the designed criteria satisfied. We denote this policy as relaxing policy and the one in the paper (require all LLMs in the committee to judge the criteria satisfied) strict policy. The number of synthesized examples before and after two filtering policies are shown below:

Using algorithm 2, but replace the data with the version filtered by relaxing policy, we obtain the following results (shown in percentage of the resolved rate):

We observe a notable performance decrease across two models in both OSWorld and Spider2-V. In particular, using the data filtered by relaxing policy, Claude-3.5-sonnet suffers from a 5.7% drop in OSWorld. This indicates that the additional examples retained by the relaxing policy are of low quality, which makes the overall performance significantly worse.

Question 5: The append means adding an element to the list, while the operator += means concatenating another list to the current list. For example in Algorithm 2:

• Before line 10, if R=[[i_1,t_1], [i_2, t_2]], and the new element is [i_3,t_3], then R=[[i_1,t_1], [i_2, t_2],[i_3,t_3]] after line 10.
• Before line 19, if H=[o_1,a_1,o_2,a_2], and the new list is [o_3,a_3], then H=[o_1,a_1,o_2,a_2,o_3,a_3] after line 19.

Questioin 6: In Section 3.4, we add a sentence to clarify that all percentage units are omitted for brevity.

Dear Reviewer,

We are glad to hear that most of your comments have been addressed! Thank you very much for your detailed review and constructive feedback on the paper!

Regards,
Authors of submission 11615

Thanks a lot for the review! We are glad to hear that the reviewer finds the proposed approach novel and effective across agentic scenarios. We also appreciate the reviewer’s recognition of our paper writing, comprehensive experiments and ablation studies. Below we address the concerns raised by the reviewer. We used purple text to clarify the parts that we modified in the revision.

Weakness & Question 1: Please refer to our response to the question 3 and 4 of the reviewer pZmd, where we include more discussions on the filtering of the synthesized data and conduct an ablation study to investigate the influence of a lower filtering rate.

In Appendix G, we include representative examples of trajectories that get filtered out.

Question 2: Yes, we implement LATS on our evaluated benchmarks by ourselves.

Dear Reviewer,

We are delighted to find that we have addressed your concerns and questions! Thanks once again for your insightful comments and suggestions on the paper! They are highly appreciated!

Regards,
Authors of submission 11615

Thanks a lot for the review! We are glad to hear that the reviewer finds the problems addressed in the paper of great significance and that the collected datasets might be a valuable resource to the community. We also thank the reviewer for the appreciation of the promising results and comprehensive analysis of the scaling law and trajectory length. Below we address the concerns raised by the reviewer. We used purple text to clarify the parts that we modified in the revision.

Weakness 1: First, we would like to clarify that, instead of just in-domain documentation, Learn-by-interact leverages broader resources including software manuals, website FAQs, operation tutorials, and etc. In general, we believe that this setup is a common scenario in realistic environments, as human users frequently use various resources to find solutions. For example, students may look up textbooks to understand a theorem, shopping website users may browse FAQs to check the procedure for returning an item, etc. In this paper, we demonstrate that such resources are available in tens of environments, which indicates the generalizability of the proposed approach in practical applications.

Weakness 2: We agree with the reviewer that based on the experimental setting in the paper, we observe diminishing returns as the synthesized data size increases. To demonstrate the effectiveness of the synthesized data in a larger scale, we believe that it will depend on joint efforts of model size, architectures, learning algorithms, and more. As shown in Figure 3, one signal indicates that training-based approaches usually achieve more significant improvement compared to the training-free ones, and larger models benefit more by training on synthesized data compared to their small counterparts. It is possible that, in some other settings, e.g., tuning larger models, we can expect more significant gains beyond 100k data as the model has more capacity to incorporate a larger amount of knowledge.

Question 1: We agree with the reviewer that it is possible to sample task instructions from LLMs based on the given environments. However, we note the following potential concerns regarding this approach: the distribution and the diversity of the generated data are hard to control. Without conditioning on prior documents, one will need intensive prompt engineering to guide LLMs in generating diverse task instructions. On the other hand, the related resources are usually crafted by experts or written by real users, which cover most important scenarios that people who interact with the environment are interested in.

Following the reviewer’s suggestion, we compare Learn-by-interact with the version without relying on existing resources in WebArena. Except for sampling task instructions from LLMs based on given environments, we follow the same procedures in Learn-by-interact to synthesize 10k examples. The results of in-context learning with Claude-3.5-sonnet are shown below:

As shown in the results, using the task instructions only based on given environments results in a performance drop compared to the version that leverages related resources. The gap becomes larger as more data is generated. This indicates the effectiveness of using existing resources to generate high-quality data.

Question 2 Yes, this is possible! The three examples in Table 31 and the first example in Table 32 (in the appendix) can be completed by sub-trajectories of the second example in Table 32. The reviewer may refer to Tables 24-30 for the corresponding visual demonstrations.

Question 3: In Learn-by-interact, we would like to demonstrate it to be a general approach that can be integrated with many existing methods. Importantly, it is interesting to see that our approach offers additional benefits on top of the state-of-the-art pipeline. This motivates us to choose Step as the baseline implementation at the time we started experiments.

Question 4: Thanks a lot for pointing out typos in the paper. We have fixed them and highlighted the modifications in purple text in the revised PDF.

Thank you very much! We are glad to learn that most of your concerns have been addressed, and we agree that it would be interesting to investigate the cross-website generalizability of the synthetic data. Motivated by this, we design the following experiments.

Across five websites in WebArena, we consider the content management systems (CMS) as a held-out test set and leverage the synthetic data from the remaining websites as the training data, which includes 81,346 examples. To ensure a fair comparison and avoid the influences of the discrepancies in the training set size, we downsample the original data that covers all the WebArena domains from 109,276 to 81,346 instances. Following the same evaluation pipelines in the paper, we assess the performance of in-context learning with Claude-3.5-sonnet and training with Codestral-22B. The results on the CMS subset are shown below.

From the results, we observe that, even without specifically using the data sampled from the CMS environment, Learn-by-interact demonstrates significant improvements over the baseline in both training and in-context learning. This indicates that the proposed approach holds the potential for cross-website generalization and is likely to achieve better performance when utilizing data from more websites.

We hope that this addresses your questions regarding the generalizability of the proposed approach! Thanks once again for your valuable feedback and insightful suggestions on the paper!