Enhancing Decision-Making of Large Language Models via Actor-Critic

Q1: Lack of intuitive explanations and in-depth analysis.

A: We appreciate the reviewer's emphasis on intuitive explanations and analyses. In Fig.10 and 11 (Appx.A.7), we have provided detailed illustrative analyses using representative examples from ALFWorld and BabyAI-Text, which demonstrates into how each component of our actor-critic framework (LAC) influences action selection.

Fig.10 presents a concrete scenario where the agent's goal is to "put a saltshaker in the drawer." At a critical decision step (Step 4), we observe the following intuitive distinction between components.

• LLM-based prior policy alone mistakenly suggests "go to drawer 1" because the base LLM model overlooks that the agent has already found the correct object ("saltshaker 1") in cabinet 2. This error exemplifies the common hallucination problem in LLMs, which occurs when the model disregards previous states and incorrectly recommends irrelevant actions.
• In contrast, the critic suggests "take saltshaker 1 from cabinet 2" because it evaluates potential actions by predicting future trajectories and determines that this action will successfully pick up the correct object.
• Our method leverages these distinct insights by optimizing the prior policy's action distribution based on the critic's evaluation (see Lines 4-8 in Algo.1). It effectively corrects the errors introduced by the prior policy, balancing the strengths of prior policy (flexible but sometimes inaccurate) and critic evaluations (accurate but computationally intensive).

This illustrative example explicitly reveals why integrating actor (prior policy) and critic leads to substantial performance improvement.

Q2: The difference of the actor-critic framework between the decision-making scenario and the general LLM reasoning scenario.

A: We respectfully disagree with this claim.

Compared to classic RL actor-critic algorithm, our method is distinguished by two key contributions that specifically address significant challenges in enhancing LLMs' decision-making capabilities: (1) To effectively extract action evaluation information from LLMs, we propose a novel Q-function estimation approach (Sec.4.1) that leverages LLMs' internal belief about success or failure of the current task; (2) To efficiently utilize these action evaluation insights for policy optimization, we formulate the policy improvement problem as a KL-divergence constrained optimization and derive a closed-form solution (Sec.4.2), allowing us to optimize the policy in a gradient-free way.

Compared to the general LLM reasoning scenario, our method introduces two novel features that are specially designed for sequential decision-making problems: (1) To support long-term planning, our critic module first predicts possible future trajectories for action evaluation, which significantly improves the accuracy of action-value estimation. (2) To improve LLM's decision-making ability, our method introduces a deliberate and principled integration of LLMs as both prior policy (actor) and action evaluation (critic). Rather than simply prioritizing one over the other as in prior work, our approach synergistically combines the two, achieving a balanced framework optimized explicitly for sequential decision-making scenarios.

Q3: LATS's performance in other benchmarks.

A: LATS cannot be directly applied to the other benchmarks we used for two main reasons. (1) LATS requires the ability to revert the agent to earlier states in the environment, which ALFWorld and BabyAI-Text do not support. LATS relies on model-free MCTS, using environment simulator as a world model, reverting simulators to earlier states during tree search. This limitation is also noted in their original paper (Page 9). (2) While it might be possible to modify these environments to make them reversible, it would create an unfair comparison. Our method and other baselines do not rely on simulators during reasoning in ALFWorld and BabyAI-Text, whereas LATS would gain an advantage from this modification.

Nevertheless, we still attempted to adapt LATS for ALFWorld by using LLMs as world models, similar to our method, for a fair comparison. The results, presented in the table below, show that LATS fails in almost all tasks. This is because its tree search severely depends on the environment simulator for precise state transitions. With only LLM-based world models, the state transitions often deviate from the actual environments, due to LLMs' inherent hallucinations and partial observability of ALFWorld.

Table 1: Performance comparison of LAC (Ours) and LATS in ALFWorld.

Thanks for your efforts and insightful comments! We hope our clarification addresses your concerns and sincerely appreciate it if you could re-evaluate our work. Any further feedback is much appreciated.

Thanks for your comments and valuable suggestions. Here we provide detailed explanations to address your questions.

Q1: Lack of discussion on failure cases. It would be nice to have a small section on the failure cases of using Q_llm (are there any specific kinds of tasks/trajectories where Q_llm doesn't generalize?)

A: We have provided illustrative examples from ALFWorld and BabyAI-Text in Figures 10 and 11, respectively. In these cases, the critic ($Q_{LLM}$) alone may occasionally fail to select the correct actions. This is because the estimation of Q-values relies on the LLM's ability to predict future trajectories. When the LLM experiences significant hallucinations, the Q-values can deviate, leading to suboptimal action selection.

For example, in Figure 10, the agent needs to find and take a "saltshaker". In step 2, the critic suggests "go to drawer 1", because it incorrectly predicts that the saltshaker is there, resulting in a suboptimal action. In contrast, the prior policy suggests a more systematic search, starting with "cabinet 1" and then moving to "cabinet 2", where the target saltshaker is actually located.

Similarly, there are failure cases where incorrect action evaluations of critic lead to unsuccessful task completion. For instance, in the task "find two soapbars and put them in the cabinet", the critic mistakenly identifies a "soapbottle" as a "soapbar", causing the agent to take the wrong object and fail the task.

In summary, the critic may struggle to generalize in cases where the LLM suffers from significant hallucinations, such as mis-predicting future trajectories due to partial observability or incorrectly identifying target objects due to the base LLM's inherent hallucinations. Thank you for the suggestion, and we will expand the discussion on failure cases in the revised version.

Q2: In related works, it would be nice to cite works like: Tree search for language model agents. arXiv preprint arXiv:2407.01476.

A: Thank you for pointing out this relevant work. We will include this work in the revised version of our paper.

Q3: [Minor] The approach mostly combines existing ideas rather than introducing fundamentally new concepts. This is a minor concern of mine.

A: While our method draws inspiration from the actor-critic algorithm in classic RL, it is distinguished by two key contributions that address significant challenges in enhancing LLMs' decision-making capabilities: (1) We propose a novel Q-function estimation approach (Sec. 4.1) to extract action evaluation information from LLMs that leverages LLMs' internal belief about success or failure of the current task; (2) We formulate the policy improvement problem as a KL-divergence constrained optimization and derive a closed-form solution (Sec. 4.2), allowing us to optimize the policy in a gradient-free manner using the action evaluation.

Q4: How does the method perform with reasoning-LLMs like DeepSeek-R1 Distill Llama or Qwen?

A: We have conducted preliminary experiments with reasoning LLMs like DeepSeek-R1-Distill-Qwen-7B. However, we observed that they often tend to overthink rather than output direct environmental actions in both our method and the baseline ReAct. For instance, even when we explicitly prompt the reasoning LLMs to output actions (e.g., "Please make sure your response is in the following format:\n> {The action you choose}"), the models still generated detailed explanations but avoided selecting the next action. A typical response might be: "I need to find a key to open the safe or locate the pencil in the drawers. Since I can’t (…), I’m unable (…), I must (…)". This issue has also been noted in prior work [1]. We believe that using reasoning LLMs for decision-making tasks requires deeper exploration to balance internal reasoning with effective environmental interaction.

Q5: Can LAC be applied to more challenging environments (WebArena or BALROG), and multimodal decision-making tasks (VisualWebArena)?

A: Thank you for the suggestion. Due to the limited time and computing resources available during the rebuttal period, we were unable to conduct experiments on these benchmarks, which generally require larger base LLMs for better performance. Given that our method has already demonstrated effectiveness in various complex sequential decision-making tasks (e.g., ALFWorld, BabyAI-Text and Webshop), we believe it can be extended to these benchmarks as well. We will explore this direction in future work.

Thanks again for your efforts and insightful comments! We hope our clarification addresses your concerns. Any further feedback and discussions are much appreciated.

---

[1] Cuadron, Alejandro, et al. "The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks." arXiv preprint arXiv:2502.08235 (2025).

Thanks for your comments and valuable suggestions. Here we provide detailed explanations and experimental results to address your questions.

Q1: Questions regarding gradient-free policy updates: (1) How many actions are evaluated per state? (2) Is the policy updated after each action, or in batch updates? (3) How are policy parameters updated, particularly in the case of an infinite action space?

A: (1) The number of candidate actions is a hyperparameter, set to 5 in our experiments, that is, we evaluate the top 5 candidate actions per state. Empirically, evaluating 3-5 top actions is generally sufficient for the benchmarks used, as these actions are sampled by leveraging LLMs' prior knowledge. In most cases, effective actions are included within this set.

(2) The policy is improved by reweighting with Eq.5 after evaluating all sampled candidate actions for the current state.

(3) We do not update policy parameters directly. Instead, we derive a new policy from the original one using a closed-form solution (Eq.5-6) of the policy improvement objective (Eq.4). For infinite action spaces, we evaluate the policy by assessing a set of top candidate actions, similar to how we handle finite action spaces.

Q2: Report performance without finetuning for comparison and show in a separate figure that the performance can be further enhanced.

A: Thank you for the suggestion. We will include comparisons without finetuning in the revised version of the paper. The results, presented in the table below, show that our method outperforms the baseline both with and without finetuning, and the performance can be further improved with finetuning. With finetuing, our method provides more consistent performance with different LLMs.

Table 1: Ablation studies on the impact of finetuning.

Q3: How can a proportional policy be updated without gradients?

A: We update the distribution by adjusting the probabilities of the top candidate actions, while leaving the probabilities of other actions unchanged. Specifically, we formulate the policy improvement objective as a KL-constrained optimization problem (Eq.4). We then derive a closed-form solution (Eq.5) to this problem, which is a weighted combination of the original policy and the action evaluation values.

Q4: What does maximizing Q mean if it does not represent cumulative rewards? Why optimize Q-values instead of maximizing the prediction probability?

A: In our method, maximizing the Q-function corresponds to maximizing the success probability for the current task. The benchmarks simulate realistic scenarios where there is no immediate reward during execution, and the environment only provides feedback on task completion (success or failure) at the end of each episode. To model this, we design the Q-function to be positively correlated with success probability using a Sigmoid function (Eq.1), ensuring that maximizing Q effectively maximizes the success probability.

While directly maximizing the success probability is also valid and could lead to a different Q-function formulation, we compared our approach against it in Appendix A.2. Our method outperformed it in most tasks and models. We speculate this is because our method leverages more internal information from LLMs, utilizing both success and failure probabilities (as derived in Eq.2), resulting in more accurate and stable action evaluations. While there could be other formulations that use additional internal information, our approach remains both simple and effective.

Q5: In Equation 4, what is the distribution of the history h?

A: The history $h_t$ in Eq.4 is not a distribution, but rather a given context for the policy and critic.

Q6: What task is depicted in Figure 5? Why does LAC require fewer steps?

A: The task depicted in Figure 5 is from ALFWorld. The metric "Steps per task" is calculated as an average over both successful and failed tasks. LAC requires fewer steps due to its higher success rate, enabling it to complete tasks within the maximum step limit, while other baselines often reach this limit without completing the tasks. If we only consider successful tasks, the step cost is similar across methods: Ours: 15.32 steps, ReAct: 17.75 steps, and RAP: 16.36 steps.

Thanks again for your efforts and insightful comments! We hope our clarification addresses your concerns. Any further feedback and discussions are much appreciated.