AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning

1- Fine-tune Smaller Models: We are unsure about the "generated data" referred to by the reviewer. Is this data generated through interactions between GPT-4 and the environment to create successful trajectories? This also seems to require the use of very capable LLMs. It is important to note that the goal of our work is to enable LLM Agents to quickly and autonomously adapt to unfamiliar new environments. These environments do not have sufficient high-quality human or generated data for fine-tuning smaller models. Our method requires only 36 turns of interactions with the environment to build a Manual, whereas accumulating enough data to fine-tune an LLM would require an order of magnitude more interactions.

2- Scalability to Larger, More Complex Environments: We currently place all rules in the context to ensure that the Planner can access all the rules, thus eliminating the impact of rule retrieval on our experimental outcomes. However, for more complex and larger environments, leveraging technologies like Retrieval-Augmented Generation (RAG) is indeed necessary to effectively address the challenges. To demonstrate our method's potential in handling complex environments, we conducted experiments on the Reddit site of WebArena. WebArena [1] is a high-difficulty benchmark simulating real website interactions. In this environment, we set a maximum of 15 rules for AutoManual. We used AutoManual paired with GPT-4-turbo to build and formulate tasks in the Reddit site environment, subsequently testing on 106 tasks. The test results and comparisons with other works are shown in the table below. We provide the code and trajectories at [our comment below]. We will also include these experimental results in our paper.

Notice: Although SWE-Bench is also a challenging benchmark, its interactivity is limited. Therefore, we chose a more interactive environment, WebArena, which aligns better with the "Interactive Environment" in our paper's title.

Table: Test on WebArena (Reddit) using GPT-4-turbo

3- Sensitivity Analysis of AutoManual on Examples and Initial Rules: In Appendix F.2, we analyze the sensitivity of AutoManual to the example and initial rules. We provided AutoManual with an example of the "enter-text" task from MiniWob++, which is the simplest task type in this suite, along with the corresponding initial rules. Our findings suggest that while simpler examples and initial rules do influence the outcomes, the impact is minimal.

Strategy for Generating Initial Rules and Examples: Generally, for a new environment, we first select one task to serve as the example task, with no specific criteria for selection (In the ALFWorld experiments, we chose the simplest "put" task and achieved significant results). Then, a human operator generates a trajectory that completes the task, thus providing the necessary example. Initial rules typically comprise a "special mechanism" and a "success process" type rule, which involves considering any counterintuitive aspects of the environment and summarizing strategies to accomplish the example task.

4- Task Similarity: We follow the vector similarity approach used by Voyager and Adaplanner for skill retrieval, utilizing OpenAI's embedding model to process task names into embeddings. These embeddings serve as queries and keys, with the vector similarity from the FAISS library used to retrieve the most similar skill.

5- How The Six Rule Types Were Created: In "Rule System" of Section 3.3, we have mentioned the selection of the six rule types used in our rule system. Specifically, we believe that knowledge about the environment necessitates recording the transition functions, $T(o'|o, a)$ for observable states and $T(s'|s, a)$ for system states, as well as understanding how trajectories $\tau_{\rho}$ lead to outcomes of r=0 or r=1. These insights correspond to the task types: "Special Phenomenon" for $T(o'|o, a)$, "Special Mechanism" for $T(s'|s, a)$, and "Success Process" for $r(\tau_{\rho})=1$, "Unsolved Error" $r(\tau_{\rho})=0$. Additionally, the analysis of task segments within trajectories helps in achieving overall task success by identifying successful or failed sub-goals. This leads to the rule types "Useful Helper Method" and "Corrected Error."

6- The terminology "9 task types with feedback" and "44 task types without feedback" used in our paper follow previous works, Adaplanner. The "44 task types without feedback" denotes tasks where the HTML of the webpage remains static and does not change in response to the actions performed. For example, in the "click-button-sequence" task type, the user is required to click buttons One and Two, yet no observable changes in the HTML occur during these actions; the reward is provided only at the end of the task. Conversely, the "9 task types with feedback" refer to tasks where the webpage reacts to the actions taken. An illustrative example is the "tic-tac-toe" task, where after the agent makes a move, the webpage (the opponent) responds by making its move based on the agent's actions. This makes the "9 task types with feedback" more challenging as they represent more complex tasks, and the agent does not have full visibility of the webpage's content at the start.

[1] Zhou, Shuyan, et al. "Webarena: A realistic web environment for building autonomous agents." arXiv preprint arXiv:2307.13854 (2023).

[2] Fu, Yao, et al. "Autoguide: Automated generation and selection of state-aware guidelines for large language model agents." arXiv preprint arXiv:2403.08978 (2024).

[3] Sodhi, Paloma et al. "Step: Stacked llm policies for web actions." arXiv preprint arXiv:2310.03720 (2024).

We provide the code and trajectories at https://anonymous.4open.science/r/automanual-3860.

Regarding the Exclusion of AdaPlanner's GPT-3 Results:

1- Reproducibility Issues with AdaPlanner's GPT-3: The high success rate claimed by AdaPlanner's GPT-3 is nearly impossible to replicate. We utilized the publicly available code from AdaPlanner and tested it multiple times using OpenAI's davinci-002 and gpt-3.5-turbo-instruct models, only achieving success rates between 50%-70%. During our trials, we identified and fixed several formatting bugs in the code, yet the highest success rate plateaued at 76%. Additionally, others have also reported difficulties in replicating these results in the issues section on AdaPlanner's GitHub [see our comments below]. We invite reviewers to personally test the code to see the reproducibility of the results.

Note: OpenAI discontinued text-davinci-002 and text-davinci-003 models as of January 4, 2024, leaving davinci-002 as a substitute and recommending users to switch to gpt-3.5-turbo-instruct which is purported to have equal or enhanced capabilities.

2- Lack of Implementation Details for "Skill Filtering": The AdaPlanner paper discusses a crucial "Skill Filtering" stage in Section 3.3, yet we found no implementation of this in the provided code, nor is it detailed in the paper. According to the paper, this filtering boosts success rates significantly, as shown in Figure 4. In addition, implementing such filtering would require testing a substantial subset, if not all, of the test set multiple times. This frequent validation using the test set could unfairly benefit AdaPlanner over other methods that do not use the test set in such a manner.

3- GPT-3 Tends to Copy Curated Examples: Adaplanner meticulously designed its human examples to the extent that they only require replication to fulfill the task. Adaplanner provided one example for each of the six task types in ALFWorld, each example comprehensively offering the code needed to solve the respective task. This resulted in a scenario where, faced with a specific task type, one only needs to change the item names and target container names in the human example to produce a working code solution. To validate this point, we conducted an experiment: For a new task, we replaced the item and target names in the human example with those from the new task description, and ran the resulting solution function. We discovered that this direct copying approach (simply replacing names) achieved a success rate as high as 70%. Furthermore, we found that 30% of the failures were mainly due to errors from the sub-function ask_LLM() rather than issues with the solution itself. The code and traces for this experiment are available at [our comments below].

Given such potent human examples, it becomes less about the LLM agent's planning abilities and more about who can best copy and make minor modifications. Compared to the high innovativeness of gpt-3.5-turbo, gpt-3 tends to replicate text from examples. This highlights the truth behind the authors' statement that "gpt-3.5-turbo (and one would assume later models) was their tendency to hallucinate": the newer models are more inclined to generate answers that differ in form from the perfect human examples. However, we believe that simply replicating these artificially designed solutions does not serve as a fair comparison. By contrast, our AutoManual system does not face this issue, as it was only provided with a single human example for the most basic task.

For these reasons, we have excluded AdaPlanner GPT-3's extraordinary result from our comparative analysis. We believe it is our duty to provide fair and reproducible experimental results as a standard for comparison for our readers and for future research.

On Exaggerated Claims:

We acknowledge that we did not conduct human subject testing on "readability or global understanding" for AutoManual. However, the enhanced readability of sorted and categorized Markdown Manual text over a list of unordered rules is both intuitive and supported by established research [1]. We have provided the final generated Markdown Manual in the appendix (Figures 1 and 2) so reviewers can assess its organization and readability themselves. Proper categorization and arrangement not only improve readability but are also known to enhance LLM performance, aligning with human experience and empirical evidence [2].

We appreciate your feedback, and we will adjust our language to moderate the claims made.

Additional Questions:

In Appendix D, we mentioned our choice of N_max as 12 to avoid overly long context issues. This section also includes detailed descriptions of our hyperparameters. We will make these more prominent.

[1]: Beck, I. L., McKeown, M. G., Sinatra, G. M., & Loxterman, J. A. (1991). Revising social studies text from a text-processing perspective: Evidence of improved comprehensibility. Reading Research Quarterly, 27, 25 1-276.

[2]: How I Won Singapore’s GPT-4 Prompt Engineering Competition

The publicly available code of AdaPlanner [https://github.com/haotiansun14/AdaPlanner]

The issue of reproducibility on AdaPlanner's GitHub [https://github.com/haotiansun14/AdaPlanner/issues/2]

The code and traces for the experiment in "3- GPT-3 Tends to Copy Curated Examples" are available at [https://anonymous.4open.science/r/adaplanner_alfworld-F4B7].

1- Significance of Selected Ablations:

Each row in the ablation study table in Section 4.2 was carefully chosen to demonstrate the contribution of different components of AutoManual to the overall results. Specifically:

• The first row, "Planner only," serves as our baseline, testing only our Planner Agent without any building process.
• The second row, "Planner + Lib.," involves building only the Skill & Reflect Library during the building process, highlighting that merely recording skills and reflections without constructing rules is insufficient.
• The third row shows the "Offline Learning" version of AutoManual, illustrating that using offline trajectories rather than online learning significantly reduces effectiveness, thereby supporting our claims made earlier in the paper.
• Rows four to six respectively demonstrate the impacts of removing the Skill & Reflect Lib., case-conditional prompting, and manual formulation components from AutoManual, underlining the necessity of these elements for optimal performance.
• The final row represents the full version of AutoManual, achieving the best results.

These meanings also correspond to the comparison of each paragraph in Section 4.2. Other combinations have no special meaning or comparative significance, so they are not included in the results. We will make the significance of each row more explicit in the revised manuscript.

2- GPT's Temperature:

We also noted that setting the temperature to 0 for GPT does not always produce identical results, but the impact on our experiments is minimal. This is evident from the standard deviation, approximately 2, shown in Figure 4. In fact, the three runs in each result were not completed on the same day; the experiments often differed by many days, and then the average was taken. To confirm the stability of our AutoManual, we have conducted additional experiments with varying temperatures and 3 runs below.

Table: Additional Experimental Results of AutoManual on ALFWorld

3- Task Generalization in AutoManual: You raised an excellent point regarding task generalization. But we have to say that each task type has its own unique solution, so the Manual built on some tasks may not have a high test result on the remaining tasks. For example, AutoManual only constructs a manual out of Put, but when testing on Put Two, it will require specific strategies and an understanding of the environment: For Put Two, the agent needs to know that it cannot take two things at the same time, and should employ a strategy of taking one and putting one, etc. However, for some similar task types, AutoManual does have task generalization. For instance, removing one of the clean, heat, and cool task types during building and testing across all tasks showed a success rate nearly identical to 97.6%. This is because strategies for these tasks can be easily generalized from one to another.

4- The Planner Can Ignore Rules: This issue, as noted in the Limitations of the Appendix, is indeed recurring and challenging. We may consider integrating an additional validator agent to re-confirm whether the Planner adheres to the set rules or commits errors, using techniques like self-refine, but this would significantly increase token consumption. We leave this challenge to future research.

5- Other Environments: It is important to note that almost no environment is both practical and unrelated to GPT’s training data. For famous games like Atari or Sokoban, relevant knowledge is likely included in GPT's training dataset. Given that Atari's Instruction Manual is a public document, GPT has likely seen and memorized the strategies for playing these games. In addition, even though home and web-based environments might overlap with GPT's training materials, our experiments show that relying solely on GPT's inherent knowledge (Planner Only) achieved only a 77.6% success rate, indicating the insufficiency of intrinsic knowledge for solving tasks in these environments. Our AutoManual method effectively supplements environmental knowledge through trial and error.

1- Clarification on Environment-Specific Information:

First, the environment-specific information provided by humans can be categorized into four types:

• (i) Brief description of the environment and roles (about 25 words)
• (ii) Actions that the agent can perform
• (iii) Examples of humans completing the task
• (iv) Potential environmental knowledge embedded during LLM training

Among these, Types (i) and (ii) are fundamental information for the agent to perform tasks in the environment, and can easily and justifiably be included in the prompts. Type (iv) is challenging to control and eliminate, but in our experiments, all methods used the same LLM and the same types (i) and (ii) prompts to balance the influence of this knowledge. Type (iii), the examples, are the most labor-intensive, each exceeding 200 words (ReAct and Reflexion require 12 such examples). To further illustrate the significance of examples in previous methods, we demonstrate in the table below that the performance of Reflexion (GPT-4-turbo) with only two 'put' task examples drops from 85.9% to 73.1%. Our AutoManual system requires just one human-provided example (and derived initial rules).

Additionally, as stated in Section 3.2, all content in the skill and reflection libraries is proposed and updated by the LLM Agent itself, not manually injected, which differs from "the examples."

2- Reduction of Human-designed Prompts:

As previously mentioned, types (i) and (ii) of the LLM Agent's input prompt are fundamental environmental information, while type (iv) is inherently uncontrollable during training. The main human effort is concentrated on type (iii), with prior works requiring multiple human-written environment-specific examples (each over 200 words), whereas AutoManual needs just one from humans. Furthermore, some previous Agents, such as Generative Agents [14] and Voyager [23], were specifically designed for their respective gaming environments within their frameworks, but we do not tailor our framework to any particular application environment. Therefore, we claimed that AutoManual significantly reduces the need for elaborate design and domain knowledge.

We appreciate your feedback and will tone down the claims in the Abstract accordingly.

3- Experimental Fairness with Expel:

Thank you for highlighting this. We followed the original settings of the Expel paper in our experiments with Expel (GPT-3.5-turbo), which indeed might not have been entirely fair compared to our methods (GPT-3.5-turbo). We have added the experiment settings you mentioned below, where both Builder and Offline Trajectory’s Actor and Reflector use GPT-4-turbo, while Task Inference uses GPT-3.5-turbo. The results show that although the new Expel setup with GPT-4-turbo improves performance, it still does not match AutoManual's results: 67.2% vs. 86.2%. As discussed in our paper, this is due to Expel’s reliance on offline learning rather than our online learning, its rule management is rudimentary, etc.

4- Clarification of Terminology:

The term "free-form code" is defined in Section 3.2, paragraph two, as: "This form simplifies planning by only generating code necessary for the current environmental situation and feedback without the overhead of integrating previously executed code." We use "free-form code" to distinguish from the "function-form code" used by Voyager and AdaPlanner. They require the output code to be a Python function, while our approach imposes no such restriction.

The term "natural programming capabilities" is described in the paper CodeAct [24] as "Code data is widely used in pre-training today’s LLMs. These models are already familiar with structured programming languages." We appreciate your suggestion and will clarify these definitions further.

5- Initial Rules:

'Initial rules' are extracted by humans from the single provided example, demonstrating how rules should be extracted to the Agent. In Appendix F.2, we investigated whether the provided initial rules and that one example impact results, finding AutoManual robust to such variations.

Table: Additional Results