Efficient Reinforcement Learning with Large Language Model Priors

Q2: About querying LLM for each observation encountered by the agent (which is computationally expensive).

Our work leverages LLMs' domain knowledge to enhance RL sample efficiency, crucial for real-world applications where online sampling is costly. Our methods significantly outperform baselines in both online and offline settings.

Most studies of LLMs as agents for decision-making tasks [1, 2, 3, 4, 5] also face the challenge of querying the LLM for each observation encountered by the agent. In particular, RL fine-tuning approaches [3, 4, 5], which use large LLMs (up to 7B parameters) as the action policy, require querying the LLM at each step to collect data for online training. Reflect_RL [6] distills GPT-4's prior knowledge into a smaller LLM (GPT-2xl) but still relies on extensive querying of GPT-4 to generate a sufficiently comprehensive dataset for each specific task. Since the LLM prior is fixed, we can resort to the caching technique within our framework to alleviate the need for querying the LLM at every step, as per your suggestion.

We introduce a variant of the DQN-Prior algorithm, called DQN-Prior (Cache), which uses cached action candidates for previously seen observations and queries the LLM prior, storing its outputs only for new observations. We present the time and computational costs of our DQN-Prior(query LLM each time) and DQN-Prior (Cache), as well as the RLFT baseline, TWOSOME [4], for learning to converge on Overcooked (Tomato) in the table below. Detailed results are illustrated in Appendix 9.4 on page 21. The time-consuming of our DQN-Prior(Cache) is significantly lower than the TWOSOME, while maintaining comparable performance.

The caching technique is well-suited to our framework, as the 7B LLM is capable of providing a reliable sub-action space. DQN-Prior (Cache) significantly reduces query time costs but also achieves superior performance compared to DQN-Prior (querying the LLM at each step), as frequent queries introduce more uncertainty.

[1] Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." The Eleventh International Conference on Learning Representations. (2023)

[2] Shinn, Noah, et al. "Reflexion: Language Agents with Verbal Reinforcement Learning." Advances in Neural Information Processing Systems 36 (2023).

[3]Wen, Muning, et al. "Reinforcing LLM Agents via Policy Optimization with Action Decomposition." The Thirty-eighth Annual Conference on Neural Information Processing Systems.2024

[4]Tan, Weihao, et al. "True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning." The Twelfth International Conference on Learning Representations.2024

[5]Christianos, Filippos, et al. "Pangu-agent: A fine-tunable generalist agent with structured reasoning." arXiv preprint arXiv:2312.14878 (2023). 2023

[6] Zhou, Runlong, Simon S. Du, and Beibin Li. "Reflect-RL: Two-Player Online RL Fine-Tuning for LMs." arXiv preprint arXiv:2402.12621 (2024).

We thank the reviewer for their time and effort in reviewing our paper, we are glad that you appreciate 1) the intuitive and simple method, 2) the novelty of our approach and 3) the clear writing of our paper. We have provided detailed explanations and clarifications to resolve your concerns and made several updates that have improved the paper based on your recommendation. We respectfully hope you can reconsider your score to reflect our updates.

Q1 The method is limited to the setting of text observation. Can it be applied to image environments

Our primary goal is to explore how incorporating LLMs' prior knowledge can enhance RL training in sequential decision-making tasks. A number of standard RL benchmarks with discrete action spaces, such as StarCraft[1], Overcooked[2], Card games[3], and BabyAI[4], feature symbolic observation or grid-based image observation that can be converted into text using rule-based methods. Previous studies [1,2,3,4] have verified the feasibility of LLMs on solve these games. These textualized RL benchmarks are also suitable for our approach. In this work, we have demonstrated the effectiveness of our approach on several prominent decision-making tasks, including ALFWorld, Overcooked, and Frozen Lake.

In addition, Atari-GPT [5] has demonstrated that multi-modal LLMs, such as GPT-4 and GPT-4V, can achieve sub-optimal performance on Atari tasks with image obervations. Building on their attempts, our framework could be readily adapted to image-based observation settings by utilizing multi-modal LLMs to generate action priors. We believe that leveraging our framework to further enhance the decision-making capabilities of multi-modal LLMs would be a worthwhile direction for future work.

[1] Ma, Weiyu, et al. "Large language models play starcraft ii: Benchmarks and a chain of summarization approach." arXiv preprint arXiv:2312.11865 (2023).

[2] Zhang, Ceyao, et al. "ProAgent: building proactive cooperative agents with large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 16. 2024.

[3] Costarelli, Anthony, et al. "Gamebench: Evaluating strategic reasoning abilities of llm agents." arXiv preprint arXiv:2406.06613 (2024).

[4] Carta, Thomas, et al. "Grounding large language models in interactive environments with online reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[5] Waytowich, Nicholas R., et al. "Atari-GPT: Investigating the Capabilities of Multimodal Large Language Models as Low-Level Policies for Atari Games." arXiv preprint arXiv:2408.15950 (2024).

Q3: incorporate LLM priors into RL systems through the reward function rather than the action distribution.

Thank you for your recommendations. Our work leverages the reasoning and high-level planning capabilities of LLMs to facilitate RL training by generating sub-optimal action priors, while the studies you recommended utilize LLMs to generate intrinsic reward functions through their powerful coding or in-context learning capabilities. Both our approach and the studies on LLM-based reward functions avoid the costly fine-tuning of LLMs. We have added a discussion of the recommended works to the related work section in the Appendix on page 23. Additionally, we have conducted experiments comparing our method with the LLM-based reward function approach [6] you recommended. Please refer to the response box below for details.

[6] Kwon, Minae, et al. "Reward Design with Language Models." The Eleventh International Conference on Learning Representations. 2023

Q4: Give some more detail on the environments

Thank you for your suggestion. We have included detailed reward shaping descriptions in Section 4.1 and added examples of observation and action representations for all environments in Appendix 9.1 on page 18.

Q5: About references missing dates and Fig 1.

Thank you for carefully reviewing our paper and pointing out these issues. We have updated Fig. 1 and the references according to your suggestions.

Dear reviewer,

We would like to add an additional experiment comparing our approach with using LLMs to generate rewards to further address your concerns. Our work leverages an LLM to generate sub-optimal prior actions for each state, thereby reducing the large action space into a manageable one, thus eliminating the need to explore meaningless actions and improving sample efficiency. Following your suggestion, we include a comparison with [1], which uses LLMs to generate reward signals using in-context examples for RL training.

The ALFWorld environment lacks dense reward signals, providing only a reward of 1 for successfully completing the task, and features large action spaces with a maximum of up to 50 actions, making it challenging to solve. We implement the baseline [1] of using an LLM to generate auxiliary rewards on ALFWorld, called DQN-LLM2R (DQN with LLM-generated Rewards) for simplicity. The comparisons of baselines are shown in the table below. Specifically, we use LLM (Qwen-1.5 7B for a fair comparison) to generate an auxiliary reward for each state, and the prompt we used is:

Task Description: {instruction}
Current Observation: {state_text}
Based on the current observation and its alignment with task completion, please select an appropriate reward. 
For example:
1. The task is 'Place the spray bottle on the toilet.' Rewards are: 0.2 for finding the spray bottle, 0.4 for picking it up and holding it, 0.6 for moving toward the toilet with it, 0.8 for reaching the toilet with it, and 1.0 for placing it on the toilet.
2. The task is 'Put a candle on the toilet.' Rewards are: 0.2 for locating the candle, 0.4 for picking it up and holding it, 0.6 for moving toward the toilet with it, 0.8 for reaching the toilet with it, and 1.0 for placing it on the toilet.
Choose a reward from these options: [0.0, 0.2, 0.4, 0.6, 0.8, 1.0].
Reward: 

The complete failure of DQN-LLM2R may stem from two primary factors: 1. It's quite likely that the 7B LLMs assign incorrect reward signals, leading the Q-function to learn in the wrong direction, and 2. the large action space, which makes it hard for DQN to explore subgoals. However, our DQN-Prior can effectively solve the task by leveraging the high-level action reasoning and planning capabilities of LLMs to simplify the exploration challenge. In addition, while the LLM may be suboptimal in determining the correct action, achieving a win rate of only 0.2 when directly taking actions, our algorithm performs RL within a suboptimal sub-action space, effectively leveraging the prior knowledge of LLMs and improving error tolerance on action generation from LLM.

Examples of LLM generated reward are given as:

• Example 1:

Task: put a soapbottle in toilet. Observation: You arrive at loc 13. On the countertop 1, you see a candle 2, a sink 1, a soapbar 1, a toiletpaper 3, and a toiletpaper 1.

LLM output ['1.0\n\nGreat task! Here are some'] Incorrect

• Example 2:

Task: put a candle in toilet. Observation : You pick up the candle 1 from the countertop 1.

LLM output ['0.2\nExplanation: The task is to'] Correct

• Example 3:

Task: put a handtowel in garbagecan. Observation: You arrive at loc 18. The drawer 1 is closed.

LLM output [ '0.8 for reaching the toilet with the hand'] Incorrect

• Example 4:

Task: put a candle in toilet. Observation: You arrive at loc 10. On the garbagecan 1, you see a spraybottle 1, and a toiletpaper 2..

LLM output ['0.2\nExplanation: The task is to'] Incorrect

[1] Kwon, Minae, et al. "Reward Design with Language Models." The Eleventh International Conference on Learning Representations. 2023

We sincerely appreciate the valuable comments on improving our paper. We hope that our responses sufficiently address your concerns. We are eager to participate in ongoing discussions throughout the reviewer-author discussion period if you have further comments or concerns. Please let us know if you have any further questions or concerns and we are very happy to address them.

Dear reviewer uGBu,

We sincerely thank you for your valuable comments on improving our work. As per your suggestion, we have added an experiment to validate the feasibility of our method in image-based environments by integrating our algorithm with a Vision-language LLM.

We conducted experiments on Frozen Lake, a grid-based game with image observations. In this setup, we used Qwen2-VL-7B to generate action candidates based on the image observations and then perform RL within this prior subspace.

As shown in the table below, our DQN-Prior (Qwen2-VL-7B) achieves superior performance compared to both the standard DQN (when trained with the same number of episodes) and Qwen2-VL-7B. This demonstrates that by performing exploration within the prior action space provided by Qwen2-VL-7B, our method significantly enhances the sample efficiency of DQN and delivers a performance boost over Qwen2-VL-7B.

We thank the reviewer for their time and effort in reviewing our paper, we are glad that you appreciate the important topic and impressive performance improvements of our work. We have provided detailed explanations and clarifications to resolve your concerns and made several updates that have improved the paper based on your recommendation. We respectfully hope you can reconsider your score to reflect our updates.

Q1: Value-based version: (1) requires both the frozen LLM and the trained Q-network, which is quite memory-intensive; (2) includes a baseline that discards the frozen LLM.

In our paper, we set the Q-network to learn an adapter (a 3-layer MLP) on top of the BERT embedding, with the BERT model (which has only 110M parameters) fixed. As a result, due to the small size of the BERT model and the fact that only the adapter (a 3-layer MLP) is being learned, the computational memory and cost of the Q-network remain manageable. We have clarified the setting of Q-network in lines 224-229 on page 5.

Our DQN-Prior explores and updates within the prior action space, sampling $k$ suboptimal action candidates from the frozen 7B LLM. Indeed, the DQN baseline discards the frozen LLM and explores the entire action space. As shown in Fig. 3, our DQN-Prior significantly improves sample efficiency compared to DQN, thanks to the manageable yet reliable prior action space, especially in ALFWorld, where DQN fails to learn due to the large action space and sparse reward signal.

Q2: Comparison on BabyAI Gridworld Environment

Thank you for your suggestion. We will include a comparison on the BabyAI environment in the reversion.

Q3: Any intuitions on why GFlan is failing completely in ALFWorld (TextWorld) while it maintains some performance in Overcooked?

There are two main reasons for this phenomenon: 1) ALFWorld has a large action space, with up to 50 possible actions, compared to Overcooked, which has a maximum of only 8 possible actions. 2) The Overcooked environment provides dense reward shaping signals, while ALFWorld offers sparse reward signals—1 for successfully completing a task and 0 otherwise. Please refer to section 4.1 on page 6 for detailed reward settings.

Q4: About Clarity and typos

We sincerely thank you for your valuable suggestions. We have updated the draft point by point according to your suggestions.

Specifically, about the question of "k = 1 indicates the performance of the LLM on its own, without involving the Q-function" --> Why calling this scenario k=1 and not k=0 ? you could theoretically use your frozen LLM to create a single action subset."

In this scenario, the frozen LLM generates a single action, resulting in an action subset containing only that action, thus we call this scenario as k=1.

Q5: About the clarity of the policy-based version:

Yes, you are correct. We use the frozen Qwen-1.5 7B (or 14B for the more complex Overcooked Salad task) LLM to generate prior actions and train a Flan-T5 small LLM (with fewer than 1B parameters) as the action policy. We have added an illustration of the policy-based RL with LLM prior, as shown in Fig. 2 on page 5.

Q6: Value-based methods designed for offline learning makes more sense than DQN due to the overestimation risk.

[1] has verified that off-policy DQN can be applied to offline, discrete action space settings but tends to underperform compared to its variants that use multiple Q-estimations to mitigate overestimation. The CQL we tested, a value-based offline RL algorithm, is specifically designed to address the overestimation of out-of-distribution state-action pairs. This work introduces CQL-Prior and validates that optimizing within the prior action space significantly enhances the sample efficiency compared to CQL.

[1] Agarwal, Rishabh, Dale Schuurmans, and Mohammad Norouzi. "An optimistic perspective on offline reinforcement learning." International conference on machine learning. PMLR, 2020.

Q7: Why does CQL underperform Behavioral Cloning?

Thank you for your question. The main reason for this result may be the insufficient data volume, which limits CQL's ability to effectively discriminate the optimal path. Since the offline dataset contains many successful trajectories, detailed in Table 11 of the appendix, BC can learn from these successful trajectories and attain moderate performance with fewer examples. Additionally, we add a group of results on the offline setting of Overcooked(Salad) in Table 1, which are also briefly illustrated below. These results indicate that CQL is limited by the data volume, BC is constrained by the ratio of good to bad data. In contrast, our CQL-Prior demonstrates greater robustness to both sample volume and data distribution.

Q8: "We plot the rewards averaged over the final third of the training processes for trainable baselines" --> Evaluation of a model performance is not performed by conducting periodic performance measurements over a test set of tasks ? Could you elaborate on how performance is measured ?

Yes, you are correct. The performance of online training is assessed through periodic evaluations of the latest model. The statement "We plot the rewards averaged over the final third of the training processes for trainable baselines" is incorrect and should be removed. Really thank you for pointing that out.

Dear reviewer 8Lkv,

We have added additional experiments on BabyAI as you suggested. We compare our policy-based algorithm, GFlan-Prior, with GFlan in the BabyAI environment. We note that this task is challenging due to the partially observed nature of the environment and the random initialization of the map for each episode. We focus on the BabyAI task 'Go to the red ball', setting the maximum horizon at 30 and training both GFlan and GFlan-Prior with 500 episodes. The comparison of baselines in BabyAI is presented in the table below, and a detailed results illustration can be found in Appendix 9.4 Fig.7 on page 21. Our GFlan-Prior, which integrates both LLM prior knowledge and environmental interaction experience, also accommodates the partially observed BabyAI and outperforms both GFlan and the LLM-prior.

We sincerely appreciate the valuable comments on improving our paper. We hope that our responses sufficiently address your concerns. If you have further comments or concerns, we are eager to participate in ongoing discussions throughout the reviewer-author discussion period. Please let us know if you have any further questions or concerns and we are very happy to address them.

Dear Reviewer,

We are delighted that our response addressed many of your concerns and that you recommend acceptance. We would greatly appreciate it if you could kindly update the score in the openreview comment box.

We thank you for your further detailed review and questions, our responses are as follows:

1. Experiment Details on BabyAI:
   Yes, we conducted experiments on the task "Go to the red ball" based on the codebase provided by Carta et al GFlan. The main performance differences from Carta et al paper may result from the following factors:

   • The original BabyAI environment sets the maximum horizon to 100, but we reduced it to 30 to avoid situations where random navigation could achieve the goal and to more effectively evaluate the algorithm's decision-making ability.
   • In addition, the original GFlan paper (Figure 5) reported results after 1.5 million training steps using Flan-T5 large (780M parameters). In contrast, our results were obtained with fewer than 15K training steps (using 1% of the training examples) and Flan-T5 small (80M parameters), for the limited training time and computational resource considerations. Furthermore, our GFlan-prior also performed better than GFlan under the same training settings (including the same horizon, training steps, and action policy LLM).

   Moreover, we ran GFlan on the task "Go to the red ball" with a maximum horizon of 100, fewer than 150K training steps, and Flan-T5 small as the action policy. Under these settings, GFlan achieved a winning rate of 0.75, and our GFlan-Prior achieved a winning rate of 0.9. The winning rate of LLM-prior is 0.45.

2. Compare to a DQN-Based Approach with Larger LLM for Q-Value Estimation:
   Thank you for your valuable suggestion. We agree that this comparison could verify whether leveraging prior knowledge from LLMs for action priors is more effective than improving state-action embeddings by using a larger model than BERT. We plan to include a comparison using larger LLMs (such as Qwen-1.5 7B for a fair comparison) for the Q-network to explore this idea.

Best Regards,

Dear reviewer,

We really thank you for your valuable comments on improving our work. We would like to include an additional experiment to compare the effectiveness of using a large LLM for generating action priors versus improving state-action embeddings.

We compared three baselines on Overcooked(Tomato):

1. DQN-Prior: Uses Qwen-1.5 7B to generate a sub-action space and BERT to encode (s, a) pairs for Q-value estimation.
2. DQN (BERT): Uses BERT to encode (s, a) pairs and performs DQN across the full action space.
3. DQN (Qwen-1.5 7B): Uses Qwen-1.5 7B to encode (s, a) pairs and performs DQN across the full action space.

To reduce the overhead of querying the 7B LLM for every state, we store its outputs: For DQN-Prior, we store action proposals. For DQN (Qwen-1.5 7B), we store (s, a) embeddings.

As shown in the table below and Fig 8 in Appendix on page 22, DQN-Prior outperforms both DQN (Qwen-1.5 7B) and DQN (BERT). By leveraging the LLM’s high-level reasoning and planning ability, DQN-Prior narrows the exploration space to a reliable action sub-space, improving sample efficiency. While DQN (Qwen-1.5 7B) benefits from the 7B LLM’s superior semantic understanding and outperforms DQN (BERT), The (s,a) embeddings from the 7B LLM also cannot be directly used for action selection. Instead, a mapping from embeddings to Q-values must be learned from scratch to correctly rank the whole action space, thereby increasing sample complexity.

In addition, DQN-Prior requires less time and CPU memory compared to DQN (Qwen-1.5 7B). The higher memory and time consumption of DQN (Qwen-1.5 7B) are primarily due to its large embedding size (4096 dimensions), which significantly increases the cost of training a mapping (MLP adaptor) from state-action embeddings to Q-values, as well as the memory required to store these embeddings.

We hope that our responses have solved your concerns and respectfully hope you can consider the final decision accordingly. As the author-reviewer discussion period is coming to an end, we are respectfully looking forward to further discussing with you. Please let us know if you have any additional questions or concerns, we would be more than happy to address them.

Best regards,

The authors

Dear reviewer,

We are glad that you recognize the additional experiments with a larger LLM, and that “the comparative study with Carta et al.'s work is good.”

We use three random seeds for training and present the training curve in Figure 7(b) in appendix on page 21. Due to limited rebuttal time, it was challenging to fine-tune the 780M Flan-T5 Large with a vast number of training steps, so we opted for simpler setups. Despite this, our method consistently outperforms GFlan on the same training episodes. These results already offer compelling evidence for our main claim that incorporating prior knowledge of LLMs enhances the sample efficiency of RL training.

Best regards

We thank the reviewer for their time and effort in reviewing our paper, we appreciate your recognition of 1) the clear writing and logical flow, 2) the neat idea of using LLMs' knowledge to guide RL training, and 3) the appropriate experimental setups of our work. We have provided detailed explanations and clarifications to resolve your concerns and made several updates that have improved the paper based on your recommendation. We respectfully hope you can reconsider your score to reflect our updates.

Q1 Practicality of value based approaches: ."....Noticeably, these environments are designed to accept actions in a particular format or use rigorous action decoding, etc. to deal with the combinatoric complexity. Such environments are rare in reality."

Our primary goal is to explore how incorporating LLMs' prior knowledge can enhance RL training in textual sequential decision-making tasks. Several textual decision-making benchmarks such as ALFWorld, Science World, and Webshop have been proposed to evaluate LLMs as agents. Additionally, many standard RL benchmarks with discrete action spaces, such as StarCraft[1], Overcooked[2], and Card games[3], feature symbolic observation and action representations that can be converted into text using rule-based methods. These text-based environments have well-structured action spaces, making them suitable for our value-based approach. In this work, we demonstrate the effectiveness of our value-based approach on several prominent decision-making tasks, including ALFWorld, Overcooked, and Frozen Lake.

[1] Ma, Weiyu, et al. "Large language models play starcraft ii: Benchmarks and a chain of summarization approach." arXiv preprint arXiv:2312.11865 (2023).

[2] Zhang, Ceyao, et al. "ProAgent: building proactive cooperative agents with large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 16. 2024.

[3] Costarelli, Anthony, et al. "Gamebench: Evaluating strategic reasoning abilities of llm agents." arXiv preprint arXiv:2406.06613 (2024).

Q2 " Problems with choice of prior inducing LLMs: Two LLMs might produce semantically same action but different text. I am curious to know whether the Q-learning based approaches would assign similar values to both. "

Thank you for your question. During implementation, we prompt the LLMs with the textual state and admissible actions. The LLMs then generate a free-form text output, which is mapped to a textual executable action using a simple rule-based method, i.e., directly extracting the action appearing in the LLM output. Through this process, the Q-network processes only the textual executable action, ignoring other irrelevant information from the LLMs. Please refer to lines 102–107 for more details and lines 972-987 on page 19 in the appendix for an example of mapping LLM outputs to actions.

Q3 Novelty of the additional KL penalty in policy-based RL: Compared to ELBO-based RL approaches and supervised fine-tuned (SFT) policy in single-step decision-making tasks

From a variational inference perspective, our algorithm falls under the category of ELBO-based RL approaches. This work presents an effective pathway to integrate LLMs' rich domain knowledge into policy-based RL by utilizing it as an informative action prior within the traditional ELBO-based RL framework.

The objective of our policy-based RL is similar to the formulation in RLHF settings, where a language model is fine-tuned with KL regularization to preserve its original language capabilities. In our work, LLMs are used for providing suboptimal action priors for challenging sequential decision-making tasks, while the actual belief (posterior) is derived by training a smaller language model to distill both task-relevant LLM priors and environmental experience.

Q4 "Should the proportionality sign on line 147 be equal to? I understand the sentence is correct in its current form"

We use the proportionality sign because we omit the term $p(\mathcal{O}=1|s_t)$, which is irrelevant to action and not always equal to 1.

Q5 "The is overloaded on lines 112-113, especially, please change the " as the action prior " sentence ......"

Thank you for your suggestion. We have revised the sentence for clarity: "Here, we specify $p_{LLM}(a|s)$ as the action prior $p(a|s)$." In our work, $p$ denotes any general probability distribution, while $p_{LLM}(a|s)$ is specified as a concrete distribution for $p(a|s)$ as the action prior within this framework.

We sincerely thank you for carefully reviewing our paper and providing valuable suggestions. We have revised the draft point-by-point following your review.

We thank the reviewer for their time and effort in reviewing our paper, we are glad that you appreciate the clarity and the interesting topic of our work. We have provided detailed explanations and clarifications to resolve your concerns and made several updates that have improved the paper based on your recommendation. We respectfully hope you can reconsider your score to reflect our updates.

Q1: About the convergence guarantees of sampling using LLM as prior for Exploration and Q-function update

Thank you for your question. This work presents an effective approach to incorporating LLM prior knowledge to enhance value-based RL by exploring and optimizing within the prior action space. Proposition 1 on page 4 shows that our posterior sampling strategy over the prior action subspace approximates soft sampling over the entire action space, guided by the LLM prior distribution and Q-values. The reviewer makes an interesting point. We think convergence guarantees for Q-learning and CQL within the prior space are worth investigating in future theoretical analysis.

Empirical results in Fig. 3 and Table 1 demonstrate that DQN-Prior and CQL-Prior based on the LLM prior space significantly outperform baselines and improve sample efficiency. Specifically, we present a set of offline results on Overcooked (Salad) in the table below, where CQL-Prior significantly reduces data requirements compared to CQL for successfully completing a task, while achieving comparable performance when sufficient data is available. These results empirically demonstrate that optimization within the prior action space barely hinders CQL convergence while improving sample efficiency.

Q2: "If the LLM can not provide a good prior, constraining the learned policy as shown in Equation (8) will result in sub-optimal policies"

Thank you for your question. We agree that a poor LLM may not provide helpful information, and the incorporation of the LLM prior, as shown in Eq. (8), may not directly assist in learning the optimal policy. The results in Fig. 4(a) and (c) also demonstrate this phenomenon: the LLM prior itself may be suboptimal for decision-making tasks, and a large coefficient for the KL constraint (between the learned action policy and the LLM prior) negatively impacts the performance of the learned action policy. In fact, LLMs only provide priors, the actual belief (posterior) about the scenario is derived from our training within the environment. This approach, to some extent, alleviates the issue of biased priors. As shown in Fig. 4(a), although the LLM prior itself may be suboptimal for decision-making tasks, the variational posterior approximation in Eq. (8) successfully learns a proficient action policy. In addition, we can reduce the reliance on the LLM prior by using a smaller KL coefficient, or by implementing a parameter decay mechanism.

Q3: About time-consuming and computation-consuming for querying the LLM each time.

Our work leverages LLMs' domain knowledge to enhance RL sample efficiency, crucial for real-world applications where online sampling is costly. Our methods significantly outperform baselines in both online and offline settings.

Previous RL fine-tuning approaches [1, 2, 3], which use large LLMs (up to 7B parameters) as the action policy, also require querying the LLM at each step to collect data for online training. Since the LLM prior is fixed, we can resort to the caching technique within our framework to alleviate the need for querying the LLM at every step.

We introduce a variant of the DQN-Prior algorithm, called DQN-Prior (Cache), in which we use cached action candidates for seen states and query the LLM prior, storing its outputs only for unseen states. We present the time and computational costs of our DQN-Prior(query LLM each step) and DQN-Prior (Cache), as well as the RLFT baseline, TWOSOME [2], for learning to converge on Overcooked (Tomato) in the table below. Detailed results are illustrated in Appendix 9.4 on page 20-21. The time-consuming of our DQN-Prior(Cache) is significantly lower than the TWOSOME, while maintaining comparable performance.

The caching technique is well-suited to our framework, as the 7B LLM is capable of providing a reliable sub-action space. DQN-Prior (Cache) significantly reduces query time costs but also achieves superior performance compared to DQN-Prior (query LLM each step), as frequent queries introduce more uncertainty.

[1]Wen, Muning, et al. "Reinforcing LLM Agents via Policy Optimization with Action Decomposition." The Thirty-eighth Annual Conference on Neural Information Processing Systems.2024

[2]Tan, Weihao, et al. "True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning." The Twelfth International Conference on Learning Representations.2024

[3]Christianos, Filippos, et al. "Pangu-agent: A fine-tunable generalist agent with structured reasoning." arXiv preprint arXiv:2312.14878 (2023). 2023

Q4: About continuous action space

Thank you for your question. As you mentioned, this work focuses on investigating the incorporation of LLM background knowledge into RL by letting the LLM generate reliable action candidates. However, it is challenging for LLMs to directly generate continuous actions [6]. Previous attempts to use LLMs for solving continuous RL tasks have also avoided outputting continuous actions. For instance, [4] uses LLMs to plan over pre-defined high-level skills (e.g., "pick up an object"), which then form a discrete action space, similar to the setting in our paper. Other studies use LLMs to generate code for enhancing state representation [5] or designing reward functions [6]. Our work leverages the LLM's high-level action reasoning and planning abilities; therefore, this work focuses on discrete action spaces. It’s important to note that the action space size in ALFWorld is not as small as imagined, with up to 50 possible actions, making exploration still challenging.

[4] Lin, Kevin, et al. "Text2motion: From natural language instructions to feasible plans." Autonomous Robots 47.8 (2023): 1345-1365.

[5] Wang, Boyuan, et al. "LLM-Empowered State Representation for Reinforcement Learning." ICML (2024).

[6] Yu, Wenhao, et al. "Language to Rewards for Robotic Skill Synthesis." Conference on Robot Learning. PMLR, 2023.

Dear Reviewer h6X1,

We sincerely appreciate the valuable comments on improving our paper. We hope that our responses sufficiently address your concerns. If you have further comments or concerns, we are eager to participate in ongoing discussions throughout the reviewer-author discussion period. Please let us know if you have any further questions or concerns and we are very happy to address them.

Dear Reviewer,

We really thank you for your valuable comments on improving our work. We are glad that our response has addressed your concerns regarding computational cost and the potential for sub-optimal policy results due to a poor LLM prior. We will now address your concerns about action sample number k and continuous actions.

1. Proposition 1 requires that $k \to \infty$. However, in your experiments (as shown in Table 2), the maximum $k$ you choose is 15.

Thank you for your question. In Proposition 1, in the limit of $k \to \infty$, the sampled actions can exactly approximate the LLM prior action distribution, $p_{\text{llm}}(a|s)$, by the law of large numbers. Empirically, we sample k actions to approximate the LLM prior distribution and set $k=5$ for most training scenarios to balance the computational cost and approximation accuracy. Even with a small $k = 5$, our method outperforms other baselines, as shown in Figures 3 and 4 and Table 1, demonstrating that k = 5 provides a sufficient approximation for LLM prior distribution. This is further demonstrated in Figure 4(b) (page 9) and Figure 5 in the appendix (page 20), where $k = 5$ already significantly outperforms $k = 1$.

2. I understand that handling continuous action spaces may not be trivial. However, I believe that investigating continuous action spaces is essential for current RL research, both from a theoretical and practical perspective. There are also some emerging works, such as [1].

Thank you for your recommendation. Our main contribution lies in investigating how to leverage the rich prior knowledge of LLMs to enhance the sample efficiency of RL training. We achieve this by developing an efficient approach that uses the fixed LLM as a suboptimal action prior. The work you referenced, [1], also highlights that “LLMs are inherently text-based and may be incompatible with numerical environmental data.” Thus, this work fine-tunes the LLM to adapt it to continuous action space environments.

Enabling text-based LLMs to directly output continuous actions is beyond the scope of this work's contribution. Instead, we prioritize discrete action spaces, where most general-purpose LLMs are already capable of functioning without fine-tuning, to effectively validate our idea that rich knowledge in LLMs can assist in RL training. We thank the reviewer's recommendation, and we plan to discuss related works on LLMs for continuous spaces in the related work section.

[1] Pang, J. C., Yang, S. H., Li, K., Zhang, J., Chen, X. H., Tang, N., & Yu, Y. KALM: Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

We hope that our responses have solved your concerns and respectfully hope you can reconsider your final decision accordingly. As the author-reviewer discussion period is coming to an end, we are respectfully looking forward to further discussing with you. Please let us know if you have any further questions or concerns and we are very happy to address them.

Best regards,