Adaptive Environmental Modeling for Task-Oriented Language Agents

Reply to questions A

1.The self-reference method seems fundamentally flawed, given that the underlying Large Language Model (LLM) is not perfect.

It's important to highlight that Algorithm 1 is responsible for building the Action-Observation Tree, with the LLM being just a component of this process. We have taken into account your concern about the LLM's imperfections. The design of Algorithm 1 greatly lessens the dependency on the LLM's performance, thereby reducing the risk of potential flaws, such as hallucinations, influencing the outcomes. The role of the language model is simply to choose the most exploratory action from a given set of observations and executable actions. Thus, the language model is more than capable of performing its designated role within the algorithm. This is evidenced in our experiments, as described in Section 4.3.1, where it is noted that “agents developed with both GPT-3.5 and GPT-4 frameworks succeed in generating nearly 100% executable actions within both environments”.

2.There's a lack of study on how this might differ from a perfect model.

As detailed in Section 4.3.1 of our experiments, the language model consistently produces 100% executable actions when presented with observations and a set of executable actions. This indicates that the Action-Observation Trees constructed by Algorithm 1 are inherently accurate. The primary aspect of their imperfection is not incorrectness, but incompleteness. The Comprehensiveness score in Section 4.3.1 assesses how far these trees are from being complete Action-Observation Trees.

Reply to questions B,E：Comprehensiveness Score

For specific tasks like Webshop and ALFWorld-Eco, the scores are typically based on conventional accuracy or , in other words, success rates. Your query seems to center around the Score for Comprehensiveness. Let’s explore this in detail: Given your expertise in reinforcement learning, you can think of this Score as akin to a weighted State Coverage. The 'weight' in this context is indicative of the informational value of each action in enriching our understanding of the environment from a human perspective. Take the Webshop example: for the observation [result page 1], which includes [result 1] (a), [result 2] (b), [result 3] (c), [Back to Search] (d), and [Next >] (e). The letters in "( )" representing the potential informational value of each clickable action. Assuming resources allow the agent to explore two actions, an agent 1 opts to explore [result 1] and [Next >], its score would be calculated as $\frac{a + e}{a + b + c + d + e}$. Conversely, if another agent 2 explores [result 1] and [result 2], its score would be $\frac{a + b}{a + b + c + d + e}$.

In this context, for question E, absent a human-centric evaluation and assuming equal weights for all actions, both agents 1 and 2 would end up with the same score of $\frac{2}{5}$, thereby diminishing the significance of the comprehensiveness evaluation. To address the issue of potential human bias, we have meticulously developed a set of Scoring Criteria for Comprehensiveness.

• Information Relevance: Evaluate how relevant the information at each node is to the overall understanding of the Webshop environment. Irrelevant or overly detailed information about specific products should be scored lower.
• Depth of Environmental Insight: Score based on how deeply the node's information contributes to understanding the environment. Nodes providing unique or essential insights should receive higher scores.
• Clarity and Conciseness: Assess whether the information is clear and to the point. Unnecessarily verbose or unclear descriptions should be scored lower.

We also incorporate assessments from multiple crowd-sourced evaluators to ensure a balanced perspective, while paying close attention to the consistency of the scoring system.

Regarding consistency: To assess score distribution consistency, we first conducted an intuitive check and found that the score distribution was relatively consistent. We then proceeded with a quantitative statistical analysis. To minimize the influence of individual crowd workers' scoring preferences, we normalized the scores for each worker. This involved dividing each node's score by the total node scores and then multiplying by 100. Subsequently, for each node, we calculated the standard deviation of scores given by different crowd workers. Finally, we computed the average of these standard deviations across all nodes. The results were 0.76 for Webshop and 0.42 for ALFWorld-Eco, indicating a relative consistency in the crowd workers' scoring.

Thank you once again for your constructive feedback. If our responses have effectively addressed your concerns, we would deeply appreciate it if you could re-evaluate our work and grace us with a positive score. If you have any new suggestions, we will also incorporate them into the revised version.

Thank you very much for your detailed feedback. We suspect that you might be a reviewer with a background in reinforcement learning. Some differences in background might have created a significant gap between what we intended to express and what you understood. Firstly, we would like to offer some further elaboration and clarification regarding our motivation, and then we will address your concerns in detail.

Clarification about Motivation and Contribution

We deeply regret that our assumptions about the reader's background may have led to an underemphasis on the motivation of our work, causing some misunderstandings. We hope that the subsequent comparison table and explanatory text will provide you with a clearer understanding of the underlying motivation behind our study.

Clearly, environmental information is greatly beneficial for agents in solving tasks. Previously, agents based on large language models primarily utilized environmental feedback for decision-making. Our approach goes a step further by explicitly incorporating environmental modeling into the context, which is the core motivation of our work. As we continue to consider how to model environments using text, we find that external environments are continually fluctuating, therefore it is best to let the LLM-based agent adaptively model.

Starting from a perspective of resource and contextual limitations, we propose a relatively practical and adaptive method. This involves using the language model itself to explore the environment and employing textual descriptions for environmental modeling. Undoubtedly, the agent concepts in reinforcement learning are more mature than those in LLM-based methods, the design philosophy of this method is partly inspired by reinforcement learning such as greedy exploration. Our experiments demonstrate that explicitly introducing environmental modeling into the context indeed enhances the model's performance.

[1]Synergizing Reasoning and Acting in Language Models

[2]Reflexion: Language Agents with Verbal Reinforcement Learning

[3]https://github.com/Significant-Gravitas/AutoGPT

[4]Deliberate Problem Solving with Large Language Models

[5]Graph of Thoughts: Solving Elaborate Problems with Large Language Models

Below we provide responses to the concerns and questions you have raised:

Reply to questions A,C,D ：Environment Model / Environment Modeling

We sincerely appreciate your insightful review from the perspective of classical reinforcement learning, which has brought to our attention the potential for confusion in our terminological expressions. In agreement with your perspective, we acknowledge that the term "environment model" indeed possesses a very standard and established definition within the realm of reinforcement learning. Recognizing this concern, as you might have observed, we have carefully avoided using the term "environment model" to describe the static Action-Observation Tree in our work. This deliberate choice stems from our understanding that to align with the conventional definition of an environment model, one would need to integrate additional components, like the state/observation transition model. Our approach has been to employ the concept of "environment modeling" solely to describe the objective and process. In this process, we determined that the Action-Observation Tree we developed was sufficiently comprehensive for the language model's environmental understanding, thereby eliminating the necessity to construct the other aspects typically associated with a standard environment model.

7.Limitations

The primary aim of our approach is to model the environment within a context. However, real-world environments can be complex and may not be comprehensively described within the textual limitations of context length. Two developments could alleviate this potential limitation. First, as mentioned in the discussion section of the paper, is the introduction of a deletion mechanism. Second, research on expanding the context length of LLMs is advancing rapidly, and there are already some LLMs available that handle extended contexts.

Thank you once again for your constructive feedback. If our responses have effectively addressed your concerns, we would be immensely grateful if you could provide us with a positive score. If you have any new suggestions, we will also incorporate them into the new version.

2.Task Executor:

Activity: You are an agent and engaged in an online shopping task on the Webshop site.

Environment：Below are the interaction for the Webshop : [Action-Observation Tree](Similar to figure 3,4,5 in appendix)

Objective: Complete the purchasing task concerning 'the instruction' by iteratively performing the Think and Action steps outlined below:

Think: Given the observation [ the observation details ], think how you can accomplish the instruction task.

Action: Based on your formulated strategies, perform an action to interact with the environment.

The actions you take must be one of the following:
"think[strategy]" to accomplish the instruction task.
"search[product]" to look for specific products.
"click[button]" to interact with actionable elements. Only strings in "[ ]" are button.

Then you can set the instruction task.

3. Updating the Environment Tree with New Trajectories.

As the Agent carries out its tasks, it concurrently constructs an Action-Observation Tree, essentially forming new trajectories. Thus, the update as defined in Formula 3 fundamentally represents a fusion of two distinct trees. Notably, this phase proceeds without the language model. We meticulously engage in a breadth-first traversal of the nodes within the trajectory-representing tree. Whenever we encounter nodes absent in the tree that depicts the environment, we refine and expand the latter by integrating these new nodes. To illustrate, refer to the comparative analysis of the Action-Observation Trees in figures 4 and 5 in the appendix. Figure 5 vividly demonstrates the enrichment of the tree with novel nodes, such as [Buy Now] and [Attributes], introduced through the progression of task trajectories.

Reply to comments and questions 8：Evaluations and Results

1.More Test Cases

In the webshop dataset, we initially used only 100 instruction tasks. When we expanded it to 300 tasks, the agent's effectiveness (success rate) continued to show a slight increase.

2.More Benchmarks

More benchmarks are certainly beneficial. To our knowledge, most suitable LLM-based agent benchmarks are still in the process of being refined and their corresponding papers are under review. We are unsure about the research community's acceptance of the quality of these LLM-based agent benchmarks. If you have any mature benchmarks to recommend, we would be more than willing to further refine our work. In fact, the new environment ALFWorld-Eco that we constructed in our paper is also aimed at enabling more comprehensive benchmark assessments.

3.More metrics

Beyond success rate, we are currently unsure what other comprehensive and reasonable metrics might be. Could you specify any particular metrics? If you mean error bars and statistical errors, we assure you that these will be included in future updated versions.

4.The reason for the lower results

The reason for the lower results is that our approach is completely zero-shot, without any examples. Introducing a few-shot learning could indeed significantly improve performance, but it does not align with the problem scenario we are considering. We want language agents to adapt autonomously to new environments, especially those unexplored by humans, and thus we do not include few-shot examples with annotations and detailed task and environment instructions."

5.Abaltions

Section 4.3.2 and Figure 2 include the Ablation Studies. It seems you have missed them: The figure presents the ablation performance of the method, where “React” indicates the exclusive application of the React method without any environmental adaptation. “React+Pre-task” denotes that the examination only occurs in the pre-task phase, with no updates occurring in the in-task phase. A comparison of these results reveals that both exploration and updating are crucial for the agent’s performance in the ALFWorld-Eco environment, underscoring the importance of the two components.

6. Statistically Significant and Error Bars

As detailed in section 4.1, to ensure consistent results, we meticulously calibrated the temperature to zero. This precision calibration leads to remarkably consistent performance of the language model across several experiments on the same dataset, demonstrating minimal variability. Additionally, to reinforce the statistical robustness of Figure 2, we implemented a strategy of randomizing the dataset order and conducting repeated averages. Although it's not possible to present images here, rest assured that Error Bars will be prominently featured in the final version of the document .

Thank you very much for your detailed feedback. We suspect that you might be a reviewer with a background in reinforcement learning. Some differences in background might have created a significant gap between what we intended to express and what you understood. In this regard, we hope to further discuss this with you. Below we provide responses to the concerns and questions you have raised:

Reply to comments and questions 1, 2, 3, 4, 5: Unclear motivation

We deeply regret that our assumptions about the reader's background may have led to an underemphasis on the motivation of our work, causing some misunderstandings. Please understand that concepts like 'recursive greedy exploration and update' or 'using language agents' are not the primary drivers of our research, but rather elements of our proposed solution.

We hope that the subsequent comparison table and explanatory text will provide you with a clearer understanding of the underlying motivation behind our study.

Clearly, environmental information is greatly beneficial for agents in solving tasks. Previously, agents based on large language models primarily utilized environmental feedback for decision-making. Our approach goes a step further by explicitly incorporating environmental modeling into the context, which is the core motivation of our work. As we continue to consider how to model environments using text, we find that external environments are continually fluctuating, therefore it is best to let the LLM-based agent adaptively model.

Starting from a perspective of resource and contextual limitations, we propose a relatively practical and adaptive method. This involves using the language model itself to explore the environment and employing textual descriptions for environmental modeling. Undoubtedly, the agent concepts in reinforcement learning are more mature than those in LLM-based methods, the design philosophy of this method is partly inspired by reinforcement learning such as greedy exploration. Our experiments demonstrate that explicitly introducing environmental modeling into the context indeed enhances the model's performance.

We will follow your advice to update the abstract and introduction accordingly.

[1]Synergizing Reasoning and Acting in Language Models

[2]Reflexion: Language Agents with Verbal Reinforcement Learning

[3]https://github.com/Significant-Gravitas/AutoGPT

[4]Deliberate Problem Solving with Large Language Models

[5]Graph of Thoughts: Solving Elaborate Problems with Large Language Models

Reply to comments and questions 6,7：The use of greedy action policy and language.

Greedy Action Policy

During the initial stage of our environmental exploration, we deliberately employ a greedy action policy, enabling the model to choose from the top-k actions. This strategy is designed to maximize the model’s exploration of the environment by drawing upon the inherent knowledge of the LLM-based agent. However, in the second phase, our focus shifts to task resolution. We move away from a greedy approach in task execution. This is because the integration of environmental modeling is intended to facilitate a more globally informed action selection process. In the 'Think' phase of React, the agent can apply the environmental modeling insights for comprehensive reasoning and planning, leading to the selection of actions that are more suitable.

the Use of Language Model

The language model is primarily used as an Evaluator in the initial exploratory phase and Task Executor in the second stage.

1.Evaluator: As illustrated in Algorithm 1, in the step of actions_next ← A.SelectActions(node.observation, actions_available), given the observation and the available actions set, the action that maximizes the exploration of the environment is selected by the language model. The specific prompt is essentially as follows:

Activity: You are an agent engaged in exploring the Webshop site.

Objective: Maximize environmental exploration by selecting appropriate actions, through iteratively performing the Think and Action steps outlined below:

Think: Given the observation [ the observation details ], consider which actions would maximize the acquisition of environmental information.

Action: Based on your formulated strategies, choose the top-k appropriate actions to interact with the environment.

The actions you take must be one of the following:
"think[strategy]" to accomplish the Objective.
"search[product]" to look for specific products.
"click[button]" to interact with actionable elements. Only strings in "[ ]" are buttons.

4.Details on what different stages of Figure 2

In section 4.3.2, there are approximately six lines of text describing the details of different stages, starting from "Specifically, 60 tasks are randomly assorted into six distinct groups…" and ending with "the average accuracy of the six groups at diverse stages is once more averaged per round to denote the final accuracy." To simplify, the stages primarily emphasize the sequence of events. For example, stage 1 represents the first 10 tasks executed in that environment. As the agent performs more tasks, it becomes increasingly adapted to the environment, thus improving its success rate.

Thank you once again for your constructive feedback. If our responses have been helpful in addressing your concerns, we would greatly appreciate a positive score. And we look forward to hearing from you.

Best regards.

Thank you for taking the time to review this paper and provide detailed feedback. We sincerely appreciate the insightful comments and the critical examination of our work. Below we provide responses to the concerns and questions you have raised:

Reply to Weaknesses1: Provide details about the usage of the language model and the prompt.

The language model is primarily used as an Evaluator in the initial exploratory phase and Task Executor in the second stage.

1.Evaluator: As illustrated in Algorithm 1, in the step of actions_next ← A.SelectActions(node.observation, actions_available), given the observation and the available actions set, the action that maximizes the exploration of the environment is selected by the language model. The specific prompt is essentially as follows:

Activity: You are an agent engaged in exploring the Webshop site.

Objective: Maximize environmental exploration by selecting appropriate actions, through iteratively performing the Think and Action steps outlined below:

Think: Given the observation [ the observation details ], consider which actions would maximize the acquisition of environmental information.

Action: Based on your formulated strategies, choose the top-k appropriate actions to interact with the environment.

The actions you take must be one of the following:
"think[strategy]" to accomplish the Objective.
"search[product]" to look for specific products.
"click[button]" to interact with actionable elements. Only strings in "[ ]" are buttons.

2.Task Executor:

Activity: You are an agent and engaged in an online shopping task on the Webshop site. 

Environment：Below are the interaction for the Webshop : [Action-Observation Tree](Similar to figure 3,4,5 in appendix)

Objective: Complete the purchasing task concerning 'the instruction' by iteratively performing the Think and Action steps outlined below:

Think: Given the observation [ the observation details ], think how you can accomplish the instruction task.

Action: Based on your formulated strategies, perform an action to interact with the environment.
 
The actions you take must be one of the following:
"think[strategy]" to accomplish the instruction task.
"search[product]" to look for specific products.
"click[button]" to interact with actionable elements. Only strings in "[ ]" are button.

Then you can set the instruction task.

3. Updating the Environment Tree with New Trajectories.

As the Agent carries out its tasks, it concurrently constructs an Action-Observation Tree, essentially forming new trajectories. Thus, the update as defined in Formula 3 fundamentally represents a fusion of two distinct trees. Notably, this phase proceeds without the language model. We meticulously engage in a breadth-first traversal of the nodes within the trajectory-representing tree. Whenever we encounter nodes absent in the tree that depicts the environment, we refine and expand the latter by integrating these new nodes. To illustrate, refer to the comparative analysis of the Action-Observation Trees in figures 4 and 5 in the appendix. Figure 5 vividly demonstrates the enrichment of the tree with novel nodes, such as [Buy Now] and [Attributes], introduced through the progression of task trajectories.

Reply to Weaknesses2 : Provide details about experiment

1.Scoring Criteria for Comprehensiveness

• Information Relevance: Evaluate how relevant the information at each node is to the overall understanding of the Webshop environment. Irrelevant or overly detailed information about specific products should be scored lower.
• Depth of Environmental Insight: Score based on how deeply the node's information contributes to understanding the environment. Nodes providing unique or essential insights should receive higher scores.
• Clarity and Conciseness: Assess whether the information is clear and to the point. Unnecessarily verbose or unclear descriptions should be scored lower.

2.Regarding Consistency

To assess score distribution consistency, we conducted an intuitive check and found that the score distribution was relatively consistent. We then proceeded with a quantitative statistical analysis. To minimize the influence of individual crowd workers' scoring preferences, we normalized the scores for each worker. This involved dividing each node's score by the total node scores and then multiplying by 100. Subsequently, for each node, we calculated the standard deviation of scores given by different crowd workers. Finally, we computed the average of these standard deviations across all nodes. The results were 0.76 for Webshop and 0.42 for ALFWorld-Eco, indicating a relative consistency in the crowd workers' scoring.

3.Is the accuracy used in Table 2 and Figure 2 in ALFWorld Eco ？

Yes，accuracy or, in other words, success rate.