AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents

We appreciate the reviewer for the positive evaluation of our paper and providing constructive comments. We have addressed individual comments below and conducted additional evaluations. We will also carefully incorporate your feedback into an updated version of our paper.

“... As the context-aware guideline and inter-task knowledge both contribute to the performance, does it imply that there're sharing contexts across different tasks? Then what's the percentage of inter-task sharing CONTEXT, and inter-task guideline retrieved by guideline selection?”

We hope to clarify that both contexts and guidelines are shared across different tasks. Below, for each context, we count the number of tasks where the same context is shared and then calculate the average percentage in ALFWorld. We apply the same calculation for guidelines:

“... But I can't find "ReAct+ExpeL" setting in Table 1 …”

Thank you for your feedback. The "ExpeL" baseline in Table 1 corresponds to the "ReAct + ExpeL" baseline. We will correct this confusion by changing the "ExpeL" baseline in Table 1 to "ReAct + ExpeL" in the paper.

“Lack of explanation about the setting "AUTOGUIDE + Reflexion" in section 4”

We appreciate your feedback. We will provide more detailed information about our implementation of Reflexion, including the maximum number of trials (3), the GPT version (GPT-4-Turbo), the temperature setting (0.0), and the number of shots (2 for ALFWorld and 0 for WebShop), in Section 4.

"Context Identification" module necessitates handcrafted context definitions per task. Since guideline selection relies on context matching, will the number of demonstrations affect task performance?

In response to your insightful comment, we introduce a new experiment that eliminates the use of handcrafted context definitions in our context identification module. Specifically, we devise a general prompt (please refer to below “Context Proposal Prompt for WebArena”), where GPT-4-Turbo proposes a possible set of context summaries. Then, we use GPT to replace the human-designed list of context summaries (e.g., “On the main page of reddit” in Figure 7) with the ones proposed by GPT (e.g., “On a forum main page with navigation links and user menu”; please refer to below “GPT Proposed Contexts for WebArena”). We show that employing AutoGuide with GPT-proposed contexts in the few-shot examples (rather than human-designed contexts as in Figure 7) achieves comparable results to the original AutoGuide on WebArena:

We additionally conduct a new experiment on WebArena, varying the number of demonstrations in the human-designed context identification prompt. The result shows that our context identification module is robust to the number of demonstrations:

‘A typo: line 110 Appendix C.1: -> C.1.”

Thank you for noting this. We will fix this typo in the paper.

---

Context Proposal Prompt for WebArena

(Instruction) You are an autonomous intelligent agent for general-purpose text-based interactive tasks. You will be provided with a few trajectories of tasks from an interactive environment, each trajectory includes a goal, and a sequence of observations and actions. For each observation, we want you to abstract the underlying context at that step into a short, concise, and general context summarization, which describes the context given the goal and target.
For example: 
1. On the main page of Google Scholar with a [search] box. 
2. Attempted to chop an object but failed, therefore getting an error message as observation. 
Notice: Your summarization must be general enough, so that similar situations, especially similar ones in consecutive steps, get summarized into the same context identification. Therefore, please do not include any task-specific information, for example, do not mention a specific user name, specific product category, or specific object type, instead conclude them to a broad and general category: a user, a product or an object. Please refer to attempted actions in the past tense and use the same words for observation and actions in the trajectories. Please list all the unique context summarizations in a python list: Answer: ['context_summarization1', 'context_summarization2', 'context_summarization3', ...]. Please don't put a context_summarization that doesn't appear in the provided trajectories. 

(Trajectory Format Specification) The goal is after "Intent:", observations are after "Observation: ", including an accessibility tree of a focused webpage and its corresponding URL, and actions are after "Action: ".

(Input) Here is the input: {Trajectories}

GPT Proposed Contexts for WebArena

On a forum main page with navigation links and user menu
On a forum list page with navigation links and sorting options
On a forum category page with navigation links and submission options
On a submission preview page with options to edit or submit
On an alphabetical list of forums with navigation options

Thank you for your constructive review and helpful questions. We have addressed each comment individually and conducted additional experiments based on your insightful feedback. We will also carefully incorporate your feedback into an updated version of our main paper and appendix.

“The author claims that their method and the in-context example selection module in ExpeL are orthogonal, so they only compared ExpeL with guidelines. This point requires experimental verification.”

In response to your insightful feedback, we conduct two additional experiments on ALFWorld: AutoGuide with the In-Context Example Selection Module and ExpeL with the In-Context Example Selection Module. The results are summarized in the table below, leading to the following conclusions:

• AutoGuide can be readily expanded with other techniques, such as in-context example selection.
• AutoGuide consistently outperforms ExpeL, both with and without the in-context example selection module.

“The author should provide the number of tokens used or the number of times the LLM is called (in training and inference), to more comprehensively compare the various methods.”

Please refer to the tables below for the number of tokens used during training and inference. We compute these statistics using a subset of training and testing tasks on WebArena. For training, we observe that AutoGuide requires more tokens than ExpeL due to the additional context generation. However, please note that this context generation enables the filtering of irrelevant guidelines during inference. As a result, AutoGuide uses a comparable number of tokens to ExpeL (which provides all guidelines at inference to an LLM agent) and achieves higher test performance than ExpeL.

“How is the number of training tasks determined for different domains?”

Generally, the more training tasks available, the better, as a larger offline dataset can enhance AutoGuide's performance. In practice, we determine the number of training tasks for each domain based on several factors: the number of available offline experiences/tasks and budget considerations.

We greatly appreciate your positive evaluations of our paper and insightful feedback. Below, we respond to your valuable comment. We will also carefully incorporate your feedback into an updated version of our main paper and appendix.

“For each different task, the guidelines have to be constructed, and that can introduce noise. This may hinder the effectiveness of the method.”

First, we hope to clarify how AutoGuide generates context-aware guidelines across different domains and tasks:

• For each domain (e.g., ALFWorld, WebArena, WebShop), a new set of contexts and guidelines is extracted.
• Within a domain, contexts and guidelines are extracted from training tasks and then used for test tasks.
• For each state of a test task, the context is identified and the most helpful guidelines are selected.

AutoGuide can experience noise from three sources: 1) noise in the context identification module, 2) noise in guideline selection module, and 3) noise in guideline extraction module. In this rebuttal, we conduct an additional analysis to demonstrate that the noise in each module is minimal.

For the noise sources of 1) and 2), we provide statistics on the rate at which the context identification module incorrectly identifies contexts and the guideline selection module selects incorrect guidelines on ALFWorld through manual evaluation. This analysis demonstrates that both the context identification and guideline selection modules produce results with low noise:

For the noise source of 3), it is challenging to determine the correctness of the guideline extraction module compared to the first two modules (i.e., context identification and guideline selection). Therefore, we qualitatively examine each extracted guideline in Figures 14-15. Our observations indicate that these guidelines contain helpful and effective knowledge, as demonstrated in our empirical experiments. Having a standardized metric for quantifying the quality of generated guidelines would be beneficial, and this is part of our planned future work, as noted in Appendix A (Limitation and Broader Impacts).

Lastly, AutoGuide employs principled methods to systematically address potential noise. Specifically, AutoGuide includes a context-matching procedure (Lines 128-131) to determine if the currently generated context matches any previously identified contexts. Additionally, AutoGuide uses the guideline selection module (Section 3.3) to choose up to $k$ relevant guidelines based on the current context. If no guidelines are deemed relevant, this guideline selection module can opt not to select any guideline.