Enhancing Decision-Making of Large Language Models via Actor-Critic

Question 1: 151-152: "Due to the auto-regressive nature of LLM, it does not do reasoning and planning explicitly." This seems controversial. Chain-of-thought/o-1 are also auto-regressive decoding, but arguably they have some reasoning in them. Same with 152-154, "Accordingly, LLM with actor-only methods often struggles with complex tasks that require multiple steps of planning and reasoning".

A: There might be some misunderstandings. Our statement refers specifically to the actor-only method, which directly outputs selected actions ($a\sim \pi_{LLM}(\cdot|g,h)$ as defined in the Preliminary Section). This method does not involve explicit reasoning or planning. In contrast, chain-of-thought and o1 incorporate reasoning processes, which distinguishes them from the actor-only method.

Thanks again for your efforts and insightful comments! We hope our clarification addresses your concerns and sincerely appreciate it if you could re-evaluate our work. Any further feedback and discussions are much appreciated.

References

[1] Zhang, Bin, et al. "Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approach." ICLR 2024 Workshop on Large Language Model (LLM) Agents.

[2] Szot, Andrew, et al. "Large language models as generalizable policies for embodied tasks." The Twelfth International Conference on Learning Representations. 2023.

[3] Yao, Weiran, et al. "Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization." The Twelfth International Conference on Learning Representations.

[4] Zhou, Yifei, et al. "ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL." Forty-first International Conference on Machine Learning.

[5] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837.

[6] Mudgal, Sidharth, et al. "Controlled Decoding from Language Models." Forty-first International Conference on Machine Learning.

[7] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems 36 (2024).

[8] Carta, Thomas, et al. "Grounding large language models in interactive environments with online reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[9] Tan, Weihao, et al. "True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning." The Twelfth International Conference on Learning Representations.

[10] Yao, Shunyu, et al. "Webshop: Towards scalable real-world web interaction with grounded language agents." Advances in Neural Information Processing Systems 35 (2022): 20744-20757.

[11] Hao, Shibo, et al. "Reasoning with Language Model is Planning with World Model." NeurIPS 2023 Workshop on Generalization in Planning.

[12] Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." The Eleventh International Conference on Learning Representations.

Weakness 3: The writing of the paper, and in particular the motivation (see above) and the experiment section can be improved. Section 5.2 and 5.3 only state the the proposed methods are better than baselines and other ablations without investigating into the reason of the gap. E.g. Why is LAC so much better than ReAct/ RAP, and is there any finding of comparing the performance of LAC with different base models. The experiment section does not provide such necessary analysis information.

A: We thank the reviewer for the suggestions and we add more discussions to explain the performance gaps.

• LAC vs. ReAct/RAP: LAC's superior performance stems from its balanced integration of the strengths of both actor and critic. While actor-only (e.g., ReAct) methods excel in short-term actions, they often struggle with long-term reasoning. In contrast, critic-only (e.g., RAP) methods conduct explicit reasoning but might mispredict future trajectories and lead to even worse action selection occasionally compared with actor-only methods. LAC addresses these limitations by balancing the actor's action generation and the critic's evaluation. We have provided illustrative examples for AlfWorld and BabyAI in Figure 1 and Figure 13 respectively. In summary, actor-only and critic-only methods make mistakes at different time steps, our LAC can select the correct action.

• LAC with Different Base Models: Regarding the performance of LAC with different base models, we highlight two key findings: (1) Our method is general and can be adapted to various base models, and (2) stronger base models, such as Gemma-7B, demonstrate higher performance when integrated with our approach. However, due to the incomplete public availability of training details for these base models, further in-depth analysis will require additional investigation.

We will include these discussions in our paper to improve the overall clarity and rigor of the paper. Thank you for highlighting these areas for improvement.

Weakness 4 & Weakness 5: (W4:) It is unclear how generalizable and computationally efficient the proposed method is. In particular, it seems that the method can only be applied to tasks with a finite action space and it is unclear if the method can generalize to realistic tasks with unbounded action space such as web browsing. (W5:) The tasks used in this work are more on the simple side. It would be interesting to see if the proposed method can work in more challenging tasks such as web browsing, coding, minecraft etc.

A: We have systematically exhibited the computational cost of our method and baselines in Figure 5 of the paper. In brief, although our method does not have the lowest cost per task, when considering both the success rate and the cost, our approach is the most cost-effective.

To show the generalizability of our method, we have conducted new experiments using the WebShop benchmark [10], which simulates a web browsing task with a potentially infinite action space. In this scenario, an agent is required to purchase a product based on specific instructions (e.g. "I need a long clip-in hair extension which is natural looking, and price lower than 20.00 dollars") through web interactions (e.g. search "long clip-in hair extension", click buttons such as "[item ID]" or "back to search"). Within this context, the 'search' and 'click' action can indeed lead to an unbounded set of potential actions, as the agent can continuously refine its queries and selections based on dynamic web results.

We represent the detailed results in Figure 7 of the revised paper, and we also show some results in Table 1 and Table 2 below. We found that our method, LAC, consistently outperforms other baselines, in terms of both success rate and final reward across various base models. This demonstrates the robustness of our method in handling more complex and unbounded action spaces.

Table 1: Success rate comparison in WebShop benchmark

Table 2: Final reward comparison in WebShop benchmark

Weakness 2: The novelty of the method is limited. The language critic and value critic are two main proposals in this paper. However, the language critic is relatively simple and can be considered as a direct use of CoT [5] where the agent is asked to generate thoughts reflecting on the previous round actions before taking actions. The objective of the value critic is also similar to constrained decoding [6] that has been widely used in the alignment domain without the need of performing gradient updates on models.

A: We respectfully disagree with the assessment regarding the novelty of our contributions. Below we address the points raised and clarify the distinct contributions of our work relative to the cited methods.

• Novelty of the Language Critic: (1) While it is true that our language critic draws inspiration from techniques like Chain-of-Thought (CoT) prompting [5], our approach goes beyond merely generating intermediate reflections. Specifically, the language critic in our framework evaluates prior actions using natural language feedback and dynamically integrates this feedback into the actor’s decision-making process through in-context learning. This enables the agent to not only reflect on previous decisions but also refine its subsequent action space in a structured manner. (2) Unlike CoT, which focuses on reasoning for static problem-solving, our language critic is explicitly designed for sequential decision-making tasks, where the evaluation of actions must adapt to the evolving context of the task. This extension is critical in domains requiring iterative feedback loops and real-time adjustments, as demonstrated in our benchmarks (Section 5.2 and 5.3 of the paper).
• Novelty of the Value Critic: (1) While constrained decoding [6] provides a mechanism for aligning generated outputs with desired constraints, our value critic operates differently in both its objectives and implementation. Constrained decoding typically enforces constraints at the output level, often without evaluating long-term sequential outcomes. In contrast, our value critic explicitly estimates the expected cumulative rewards of actions by leveraging LLMs' internal belief probabilities, coupled with a gradient-free optimization framework. (2) The introduction of trajectory rollouts within our value critic further distinguishes it from constrained decoding. By simulating future outcomes of candidate actions, our approach provides a robust mechanism to estimate long-term action-value relationships, ensuring both contextual alignment and quantitative optimization. As shown in the ablation studies (Section 5.3), this integration leads to a significant performance boost over value-only or heuristic methods.
• Integration of Dual Critics with Policy Improvement: The most significant novelty of our framework lies in the synergistic integration of the language critic and value critic and enabling gradient-free policy improvement. While each critic independently enhances decision-making, their combined use enables a holistic evaluation of actions, balancing qualitative insights (language critic) with quantitative optimization (value critic). This dual-feedback mechanism is unique to our framework and has not been addressed in prior works such as CoT or constrained decoding.

We believe that these distinctions substantiate the novelty of our proposed methods and their significant contributions to LLM-based sequential decision-making. Thank you for pointing out this opportunity to clarify our contributions.

Thanks for the valuable comments and helpful suggestions. Here we provide additional experimental results and detailed explanations for clarifying your questions. The detailed experimental results are provided in the revised version of the paper.

Weakness 1: The motivation that "these methods typically adopt either an actor only or critic only approach" (line 42-43) misses many related works. The paper relies on [1] to be the only paper that discusses actor-critic methods for LLM agents but many important related works are missing. Even PPO is commonly considered as an actor-critic method where it has an actor that estimates the V function to reduce variance for policy gradient estimation. Thus, many prior works that use PPO for LLM agents should be considered as actor-critic methods (e.g. [2]). Retroformer [3] can also be considered to be an actor-critic method where the critic is a natural language-based critic. Other works also applied value-based actor-critic methods to LLM agent tasks (e.g. [4]).

A: Thank you for your insightful feedback. However, our paper primarily addresses enhancing decision-making performance in LLMs with few samples, a context that often contrasts with traditional actor-critic with LLM methods that require large datasets [2,3,4]. We have also recognized the importance of discussing traditional actor-critic with LLM methods and included several works [3,8,9] in the Related Work of our initial submission. We thank your suggestions and will incorporate citations [2,4] to provide a more comprehensive overview of actor-critic methods in the context of LLMs in the revised version of the paper.

Dear Reviewer,

Thank you for your time and effort in reviewing our work. We have provided detailed clarifications and experimental results in our rebuttal to address the issues and concerns raised in your comments.

If our response satisfactorily resolves your concerns, we kindly ask if you could reconsider your evaluation of our work. Should you have any additional questions or comments, we would be happy to engage in further discussions to ensure all aspects are addressed.

Thank you again for your thoughtful review and support.

Best regards,
The Authors

Question 1.2: For the value critic, it seems that the main difference from constrained decoding is that the value critic uses Monte Carlo estimations of the Q function instead of actually training a parametric Q function. This practice is also commonly adopted when it is unreliable to train a Q function (e.g. [1] https://arxiv.org/pdf/2406.14532).

A: We also respectfully disagree with the claim about value-critic.

(1) Our value-critic employs a unique Q-value estimation method described in Eq.3, which leverages the internal beliefs of LLMs about success and failure to compute the Q-value. This approach is distinct from the methodology presented in [1], which does not incorporate our specific design.

(2) Though the KL-regularized RL objective is a common practice for balancing conflicting goals (as seen in methods like DPO [5] and Constrained Decoding [4]), the critical aspect lies in how each term in the objective is defined. Our value-critic accounts for long-term consequences, resulting in more stable value-based evaluations derived from the LLMs' internal beliefs regarding success and failure.

To further demonstrate the strengths of our value-critic. We conducted additional experiments comparing the performance of LAC and LAC w/ direct evaluation on the WebShop benchmark.

• LAC: Our original method. It generates value-based evaluations by extracting LLMs' internal beliefs of success and failure as described in Eq.3.
• LAC w/ direct evaluation: We prompt the LLMs to directly output the probability of success $p(y=+1)$ while keeping all other components unchanged. The Q-value is then calculated as $\log\frac{p(y=+1)}{1-p(y=+1)}$.

The results, presented in Tables 7 and 8, show that our LAC method outperforms LAC w/ direct evaluation in terms of success rate and reward across most base models. Analysis of the resulting trajectories revealed that LAC w/ direct evaluation often produces a non-informative success probability (e.g., $p(y=+1)=0.5$), leading to ineffective improvements in policy.

Table 7: Success rate comparison of LAC and LAC w/ direct evaluation in WebShop benchmark

Table 8: Final reward comparison of LAC and LAC w/ direct evaluation in WebShop benchmark

Question 1.3: It is unclear to me what the challenges are to combine those two critics, and simply combining two common practices is not novel enough for a paper that claims to come up with a new method.

A: We would like to clarify that our approach is not a combination of the critics. Instead, LAC represents a novel integration of the actor and dual-critics in a cohesive manner. The primary challenge we address is the seamless integration of the actor with both critics. Specifically, the novelty lies in:

• LAC provides a synergistic framework where the critics are designed to complement each other—language critic captures contextual feedback from its history with interpretability and qualitative reasoning, while value critic evaluates predicted future trajectories and ensures quantitative alignment with long-term outcomes.

• LAC's novel gradient-free optimization strategy harmonizes their outputs and seamlessly leverages both historical performance and future predictions to improve the actor's policy effectively.

Question 3: How does ReAct, combined with a value critic, perform?

A: As indicated in Table 1 above, ReAct combined with a Value-Critic corresponds to LAC w/o lang-critic. The performance results for this configuration are presented in Figure 4 of the initial submission (Section 5.3). In brief, LAC w/o lang-critic consistently underperforms compared to the full LAC in both AlfWorld and BabyAI benchmarks across various base models. However, it generally outperforms ReAct, highlighting the effectiveness of our value-critic.

Question 4: How does ReAct, combined with a language critic, perform?

A: Similarly, ReAct combined with a Lang-Critic corresponds to LAC w/o value-critic. The results for this configuration are also shown in Figure 4 of the initial submission (Section 5.3). In summary, LAC w/o value-critic underperforms compared to the full LAC across various base models in the AlfWorld and BabyAI benchmarks. Nonetheless, it typically outperforms ReAct alone, indicating the effectiveness of our lang-critic.

Question 5: Is the prior \pi_{LLM} the same LLMs used for Q_{LLM} for computing the critic values?

A: No, they are not identical. $\pi_{LLM}$ is based on the original model, while $\mathcal{Q}_{LLM}$ utilizes a fine-tuned version of the same model, adapted using several trajectories. The fine-tuning details are provided in Appendix B.2.

Question 6: How does language critic + value critic perform? (Essentially LAC without update action distribution, instead using the critic values to choose an action)

A: According to Table 1 above, the configuration of the language critic combined with the value critic corresponds to Value-critic only. The performance results for this setup are detailed in Figure 4 of the initial submission. In summary, Value-critic only consistently underperforms full LAC, demonstrating the effectiveness of our gradient-free policy improvement component.

Weakness 1: The terminology used in the paper could be more precise, particularly in relation to terms commonly found in the reinforcement learning literature.

A: Thank you for your feedback. If the reviewer could specify which terms you find imprecise, it would greatly help us improve the clarity of our paper.

Thanks again for your efforts and insightful comments! We hope our clarification addresses your concerns and sincerely appreciate it if you could re-evaluate our work. Any further feedback and discussions are much appreciated.

Thanks for your comments and valuable suggestions. Here we provide detailed explanations for your questions. The detailed experimental results are provided in the revised version of our paper.

Question 1: Is it reasonable to summarize the algorithm differences in the following manner: ReAct includes reasoning + actor, Critic only includes future trajectory + actor, and Lang-critic includes evaluation + actor?

A: For better clarification, we first provide a summary of our method’s variants in Table 1 below. Our approach integrates the actor with dual-critics: the Lang-Critic, which provides language-based evaluations of actions, and the Value-Critic, which offers value-based assessments of candidate actions that will be used to refine the policy via gradient-free policy improvement as described in Eq.6.

Regarding the differences:

• ReAct includes reasoning + actor, but it does not include critics so we classify it into Actor-only methods.
• The term Critic-only is misleading if interpreted as including future trajectory + actor. Critic-only refers to the methods that solely rely on value-based evaluations of actions for decision-making while ignoring the actor's prior knowledge of action distribution, although they also use the actor to sample candidate actions.
• Similarly, characterizing Lang-critic as including evaluation + actor is inaccurate. The Lang-Critic is a component of our LAC that specifically provides language-based evaluations of actions.

Table 1: Summary of our method's variants.

Weakness 2 & Question 2: (W2:) The paper is missing important baselines needed to understand the performance gain claims. (Q2:) Why does the value critic require a future trajectory, and how does it perform without future trajectories?

A: The Value-Critic relies on future trajectory predictions to improve the accuracy of its evaluations. By predicting future trajectories, the critic considers long-term consequences and evaluates actions more effectively, which ultimately leads to better decision-making.

For a full comparison, we have conducted a new experiment for LAC w/o rollout, in which the Value-Critic generate value-based evaluations without future trajectory predictions. We represent the detailed result in Figure 10 and Figure 11 of the revised paper, and we also show some results in Table 2 and Table 3 below. The results show that LAC w/o rollout consistently underperforms compared to the full LAC across various base models. This finding emphasizes the importance of future trajectory predictions for accurate evaluations.

Table 2: More ablation studies in AlfWorld benchmark.

Table 3: More ablation studies in BabyAI benchmark.

Question 3: While the benchmarks are well-known, I think they are perhaps not realistic of tasks that people might actually want LLMs to accomplish. For example, it would be interesting to see the authors evaluate on tasks considered in the GDP-Zero paper such as donation solicitation [1].

A: Thank you for suggesting an evaluation on more realistic tasks. While we were unable to include results on donation solicitation as discussed in [3] due to time constraints, we did conduct experiments on WebShop [2], which we believe represents a more realistic task that LLMs might be expected to perform. A brief description of this benchmark and the corresponding results are provided in the responses above. Our proposed method, LAC, consistently outperforms other baselines in terms of both success rate and final reward across various base models. These results highlight the effectiveness of our approach in addressing more complex and realistic tasks. We hope the experiments on WebShop [2] address your concerns.

We hope these revisions and clarifications adequately address your concerns. Thank you for your valuable feedback and insightful suggestions.

References

[1] Yu, Xiao, Maximillian Chen, and Zhou Yu. "Prompt-Based Monte-Carlo Tree Search for Goal-oriented Dialogue Policy Planning." The 2023 Conference on Empirical Methods in Natural Language Processing.

[2] Yao, Shunyu, et al. "Webshop: Towards scalable real-world web interaction with grounded language agents." Advances in Neural Information Processing Systems 35 (2022): 20744-20757.

[3] Yu, Xiao, Maximillian Chen, and Zhou Yu. "Prompt-Based Monte-Carlo Tree Search for Goal-oriented Dialogue Policy Planning." arXiv preprint arXiv:2305.13660 (2023).

Thanks for the valuable comments and helpful suggestions. Here we provide additional experimental results and detailed explanations for your questions. The detailed experimental results are provided in the revised version of the paper.

Weakness 1 & Question 1: (W1:) Namely, the authors only consider tasks with small action spaces, where evaluating each action individually is tractable. In many more realistic tasks such as dialogue, I imagine that the action space would be more open-ended and I am unsure how to adapt the proposed method. (Q1:) How would the method change when the task has a potentially infinite action space (such as in dialogue)?

A: Thanks for the insightful suggestion. We have conducted new experiments using the WebShop benchmark [2], which presents a scenario with a potentially infinite action space. This benchmark requires an agent to purchase a product based on specific instructions (e.g. "I need a long clip-in hair extension which is natural looking, and price lower than 20.00 dollars") through web interactions (e.g. search "long clip-in hair extension", click buttons such as "[item ID]" or "back to search"). Within this context, the 'search' and 'click' actions can indeed lead to an unbounded set of potential actions, as the agent can continuously refine its queries and selections based on dynamic web results.

We represent the detailed results in Figure 7 of the revised paper, and we also show some results in Table 1 and Table 2 below. We found that our method, LAC, consistently outperforms other baselines, in terms of both success rate and final reward across various base models. This demonstrates the robustness of our method in handling more complex and open-ended action spaces.

Table 1: Success rate comparison in WebShop benchmark

Table 2: Final reward comparison in WebShop benchmark

Weakness 2: In addition, the tasks considered rely on being able to simulate future trajectories with high fidelity, which may be harder to do in more complex environments. Specifically, it is likely much harder to faithfully predict trajectories when the agent is engaging with another human rather than simply moving around in a static environment.

A: We acknowledge the challenge of accurately predicting future trajectories in complex environments. To mitigate this issue, our method incorporates a KL-divergence constraint in Equation 4, balancing the actor's prior knowledge with the critic's evaluations, rather than solely relying on the critic's evaluations. This approach reduces reliance on potentially inaccurate future predictions.

Weakness 3 & Question 2: (W1:) Finally, the value critic currently only works for tasks with binary outcomes (success or failure). (Q2:) Have the authors experimented with tasks with more expressive outcomes/rewards? I am curious if both critics can still behave well when there are more nuanced outcomes than just success or failure.

A: Thank you for your valuable feedback. The WebShop benchmark discussed above also includes expressive rewards in the range of [0, 1]. In scenarios where the purchased product only partially satisfies requirements, the final reward is a value between 0 and 1. The results of the final reward in Table 2 above show that LAC outperforms other baselines across various base models, demonstrating that it can effectively adapt to more nuanced outcomes beyond binary success or failure.

Thank you for conducting additional experiments on the WebShop domain. Could the authors clarify how actions were chosen to be "judged" when the action space is large? Were a subset of permissible actions sampled at the beginning?

Overall, I still believe that the paper makes a positive contribution and maintain my score. I do find the results between LAC and ReAct to be rather close for more sophisticated models though. They also appear to be weaker than the results obtained by LATS, which to my knowledge, currently achieves state-of-the-art in this domain [1]. If the authors could also show a substantial benefit of LAC over LATS, I would consider further raising my score.

[1] https://arxiv.org/pdf/2310.04406

Dear Reviewer,

Thank you for your time and effort in reviewing our work. We have provided detailed clarifications and experimental results in our rebuttal to address the issues and concerns raised in your comments.

If our response satisfactorily resolves your concerns, we kindly ask if you could reconsider your evaluation of our work. Should you have any additional questions or comments, we would be happy to engage in further discussions to ensure all aspects are addressed.

Thank you again for your thoughtful review and support.

Best regards,
The Authors

Thank you for your timely feedback! Here we provide new experimental results and extra clarifications to your questions.

Question 1: Could the authors clarify how actions were chosen to be "judged" when the action space is large? Were a subset of permissible actions sampled at the beginning?

A: We do not manually sample a subset of permissible actions at the beginning. Instead, we allow the actor to sample a subset of candidate actions with higher probabilities and then apply gradient-free policy improvement to refine the action distribution. This strategy enables us to focus on actions with the highest potential, allowing us to efficiently navigate the large action space while ensuring effective decision-making.

Question 2: I do find the results between LAC and ReAct to be rather close for more sophisticated models though.

A: We appreciate your observation. However, it's important to note that the performance of LAC and ReAct is influenced by a variety of factors of LLMs beyond model complexity, including reasoning, modeling, and planning capabilities. For instance, as shown in the results for WebShop and AlfWorld, the final reward or success rate gap between LAC+Llama-3-8B and ReAct+Llama-3-8B is more significant than the gap between LAC+Mistral-7B and ReAct+Mistral-7B, even though Llama-3-8B is more sophisticated than Mistral-7B. This suggests that the disparity in performance is not solely a function of the sophistication of the models used.

Question 3: They also appear to be weaker than the results obtained by LATS, which to my knowledge, currently achieves state-of-the-art in this domain [1]. If the authors could also show a substantial benefit of LAC over LATS, I would consider further raising my score.

A: Thanks for your constructive suggestion. We present a detailed performance comparison between LAC and LATS in Table 3 and Table 4 below. Our results demonstrate that LAC consistently outperforms LATS across all evaluated base models on the WebShop benchmark. Notably, LAC combined with Gemma-7B surpasses the performance of LATS with GPT-3.5 in terms of Success Rate and achieves comparable performance in terms of Reward. These findings underscore the effectiveness of LAC in enhancing the decision-making capabilities of large language models.

Table 3: Success rate comparison between LAC and LATS in WebShop

Table 4: Final reward comparison between LAC and LATS in WebShop

Thanks again for your prompt feedback. We hope our extra explanations and experimental results address your concerns and we would appreciate it if you could re-evaluate our paper. If you have any further questions or concerns, please feel free to let us know.

References

[1] Zhou, Andy, et al. "Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models." Forty-first International Conference on Machine Learning.

Thanks for the valuable comments and helpful suggestions. Here we provide detailed explanations for your questions.

Weakness 1: The first kind is due to the nature of academic work, which is resource constrained (small teams; limited compute). This induces a set of "easy" criticisms such as "insufficient experimental validation" or "excessive focus on the small sample regime".

A: Thank you for your understanding regarding the constraints of academic work. While we acknowledge the limitations of resources, our study demonstrates that even with few samples and small models, it is possible to achieve performance that outperforms state-of-the-art LLMs.

To further validate the generalizability of our method, we have conducted new experiments using the WebShop benchmark [1], which simulates a web browsing task with more complex and infinite action spaces. In this scenario, an agent is required to purchase a product based on specific instructions (e.g. "I need a long clip-in hair extension which is natural looking, and price lower than 20.00 dollars") through web interactions (e.g. search "long clip-in hair extension", click buttons such as "[item ID]" or "back to search"). Within this context, the 'search' and 'click' actions can indeed lead to an unbounded set of potential actions, as the agent can continuously refine its queries and selections based on dynamic web results.

We represent the detailed results in Figure 7 of the revised paper, and we also show some results in the following Table 1 and Table 2. We found that our method, LAC, consistently outperforms other baselines, in terms of both success rate and final reward across various base models. This demonstrates the generalizability of our method in handling more complex and infinite action spaces.

Table 1: Success rate comparison in WebShop benchmark

Table 2: Final reward comparison in WebShop benchmark

Question 1: Authors mention fine-tuning under a limited budget (e.g., 18 trajectories), following the specification on lines 1048-1057. However what is the source these trajectories? Are they human annotated trajectories? Trajectories sampled from the zero-shot architecture, perhaps conditioned on task success? How are the special tokens which indicate positive/negative judgment produced for the fine-tuning data?

A: The 18 trajectories used for fine-tuning are human-annotated and sourced from the training split of our benchmarks, including both successful and failed trajectories to ensure balance. Each can be annotated in about 10-20 minutes, allowing us to maintain a manageable budget.

The special tokens indicating positive and negative judgments for the fine-tuning data are also human-annotated. Specifically, we assigned the following labels based on the relevance of actions to the final goal:

• Positive judgment like "GOOD": assigned to actions deemed necessary for achieving the final goal.
• Negative judgment like "BAD": assigned to actions that were determined to be useless or incorrect in the context of reaching the goal.
• Other judgments like "UNKNOWN": assigned to actions that could not be evaluated as either good or bad based on the trajectory history.

We have included this information in Appendix C.3 of our paper for further reference, where we detail the annotation process and the criteria used for assigning these labels.

Question 1.1: Under what policy (action distribution) is the future simulator generating? If the reweighted action distribution becomes divergent from the prior, will the future simulator be invalidated?

A: The future trajectories are generated based on the original actor $\pi_{LLM}$. While this may lead to some divergence of the reweighted action distribution from the prior, our method mitigates this risk by incorporating a KL-divergence constraint in Equation 4 in our manuscript, which helps to prevent the new actor from deviating too far from the original actor.

References

[1] Yao, Shunyu, et al. "Webshop: Towards scalable real-world web interaction with grounded language agents." Advances in Neural Information Processing Systems 35 (2022): 20744-20757.

Question 2: Authors argue equation (6) on line 375 is a more sample efficient way to incorporate labeled trajectory information than conventional policy gradient via the equation on line 833. However the value-critic on line 833 does not take simulated rollouts as a parameter. Perhaps this is a typo? Otherwise the comparison is invalid.

A: We thank the reviewer for pointing this out. The value-critic on line 833 also takes the simulated rollouts as input in practice. We have corrected this typo in the revised version of our paper.

Question 3: In figure 8, it is unclear why LAC+fine-tuned-actor underperforms LAC. The lack of commentary raises reader suspicion.

A: Compared to LAC, the underperformance of LAC w/ fine-tuned-actor arises from its tendency to overfit the training trajectories. This overfitting causes the actor to favor actions that are more frequent in the dataset, potentially leading to suboptimal action selection.

For example, in the AlfWorld training dataset, the action "take an apple from X" occurs frequently. After fine-tuning, the actor may disproportionately generate this action, even when it is irrelevant to the current goal. One case is that the current goal is to "heat some egg and put it in the garbage can". When the agent sees an "apple 2" in "fridge 1", it generates and selects an irrelevant action "take apple 2 from fridge 1", which does not align with the task.

This tendency towards overfitting arises because the complexity of the policy function, which maps states $s$ to actions $a$, often exceeds that of the critic. The policy often has to capture a wide variety of potential actions for each state, particularly in complex environments. However, the quite limited training dataset in our setting restricts its ability to generalize effectively, resulting in memorization of specific actions rather than flexible decision-making. In contrast, our critic, which includes a world model for rollout and an evaluation function, focuses on capturing more predictable dynamics of the environment and simpler evaluation criteria. This typically requires simpler mappings than those needed for the policy, thus avoiding overfitting.

We have added this clarification in the revised version of our paper to address any ambiguity.

We hope these revisions and clarifications adequately address your concerns. Thank you for your valuable feedback.