RL-GPT: Integrating Reinforcement Learning and Code-as-policy

Dear Reviewer ep8Y,

Despite the negative score, we really appreciate your detailed review. We address your questions below.

Q1. Related works.

A1. Thanks for mentioning these papers. We will add some of them to the related work part in the revision.

• RL-VLM-F introduces visual feedback to the reward function design.
• Plan-Seq-Learn combines motion planning and RL skills to solve long-horizon tasks, using LLM as the high-level planner.
• BOSS chains short-term skills to construct long-term skills with LLM supervision.
• SPRINT uses LLMs to relabel instructions.
• DIAL uses foundation models for dataset augmentation.
• LiFT uses foundation models for task instruction proposal and reward design.

The motivation, technical novelty, and experimental results of our work distinctively set it apart from these works:

• Our motivation is to equip LLMs with RL capabilities to explore environments and autonomously decide which actions to learn using RL.
• Our technical novelty involves inserting high-level actions coded by LLMs into the RL action space.
• Our main results demonstrate that our agent can successfully determine which actions to learn and complete long-horizon targets in Minecraft.

We will modify Sections 2.2 and 2.3 following your suggestions.

Q2. Code-as-Policy.

A2. Sorry for the confusion. Yes, the definition of "Code-as-Policy" in this paper refers to LLMs writing code that is then executed in the environment to control the agent. We will reference this paper to clarify this concept.

Q3. Writing issues.

A3. Thanks for these valuable suggestions!

• L133: It means our agent will not focus on reward design. Instead, default reward functions, such as sparse rewards and distance rewards, are used for agent training in the environment.
• L136: It means that one of the output dimensions of the RL network will represent the coded high-level action. This high-level action will be executed when the network selects that option.
• L181: You can find the outputs in lines 185 to 187. The inputs are simply observations from the environment, as shown in Tab. 10.

We will clarify our method based on your suggestions in future versions.

Q4. Experiment issues.

A4. Thanks for these valuable suggestions!

• L230: For these tasks, the maximum exploration steps are capped at 3K. We will compare the success rate for these tasks. More details can be found in Tab. 16.
• L232: It is linked to Section C.
• Seeds: For quantitative ablation studies on harvest tasks, we used 3 seeds. Due to the large number of tasks, the rest were tested on 1 seed. This approach aligns with practices in the field. Given the low simulation speed of the Minecraft game, other significant works, such as VPT [1], also report outcomes based on a single RL training seed.

[1] Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos, 2022

Works like Voyager focus solely on high-level planning, using human-coded skill libraries to bypass the need for low-level control. Our method integrates both high-level planning and learning low-level skills, directly addressing the challenge of playing the Minecraft game using keyboard and mouse actions. RL is necessary to acquire low-level policies autonomously.

• We are the first agent that utilizes an RL training pipeline as a tool and decides which actions to be learned by RL.
• We are the first to incorporate GPT-coded actions into the RL action space, enhancing the sample efficiency for RL.
• The critic agent is similar to Voyager, but Voyager doesn't have a two-level hierarchical framework since it only contains high-level planning.

Here is a system-level comparison table:

Dear Reviewer efny,

Thank you for appreciating our work with valuable suggestions. We address your questions below.

Q1. Generalization to other environments.

A1. We acknowledged this concern and addressed it to some extent in Appendix Section D. It is difficult to find real-world environments like Minecraft, which require both high-level planning and low-level control. We applied our methods to some robotic tasks that demand both long-horizon planning and precise motor execution. Some results are shown in Fig. 7 and Fig. 8 in the attached pdf.

• Kitchen Environment Training: Fig.7 illustrates the RL training process in the Kitchen environment[1]. The vertical axis represents the success rate, and the horizontal axis represents the number of training steps. Inserting coded motion planning into the action space accelerates learning. Our method learns faster compared to the baseline.
• Furniture Environment Demonstration: In Fig.8, we present a qualitative demonstration of the Furniture environment[2]. The motion planning action effectively aids in hole-finding tasks during table and chair assembly. The baseline struggles to find the correct location at the same training step.

[1] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, 2019

[2] Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks, 2021

Additionally, VLMs can be integrated into our critic agent, enabling future expansion to vision-based environments like driving, as shown in Fig. 9 of the attached pdf. Simple MuJoCo simulated environments may not be ideal for our method due to their focus on low-level controls and lack of high-level planning or decision-making. However, our method performs well in some cases, as illustrated in Fig. 10. For instance, GPT-4 can code an action to reverse the car and then move it forward based on the topography. Modifications are needed for different domains, such as adjusting the task descriptions in the prompts. The powerful zero-shot capability of GPT should ensure generalization ability.

Q2. Smaller / open-source LLMs.

A2. Thanks for this good question! Here is the comparison on the ''harvest a log'' task. The performance of Claude is similar to GPT-4. Vicuna-13b has lower performance due to its poor coding ability. Mixtral-8x7B works much better than Vicuna-13b. Open-sourced methods are continuously improving, making them promising for future agent development.

Q3. When the fast agent fails to code up a sub-action.

A3. Thanks for this good question! When the fast agent is unable to code an action, it implies that at least part of the action requires RL training. The slow agent will decompose this action, which is a process of gradually analyzing which specific sub-action needs RL. For the steps that can be coded, the code is written. If decomposition fails to resolve the issue after a certain number of iterations, the task will be handed over to RL.

Q4. Keeping an archive of solutions.

A4. Yes, both successful coded actions and well-trained RL networks will be preserved during agent optimization. They will be executed as skills with specific names, similar to Voyager's vector database.

Dear Reviewer m13r,

Thank you for appreciating our work with valuable suggestions. We address your questions below.

Q1. Manual design is needed for each specific environment.

A1. Yes, we acknowledge that some manual design is necessary. However, compared to existing agents like Voyager, which require humans to write low-level action code, our RL training reduces manual design by 90%. As LLMs' zero-shot capabilities continue to advance, the need for manual design will further decrease.

Q2. Expensive API call.

A2. Thanks for this good question! We agree that calling API has a cost. Here are the statistics. We count the average tokens on different tasks for the first 3 iterations.

GPT-4-32k will cost $0.12 per 1,000 tokens. The newly released GPT-4o Mini will be more cost-effective.

Here is the comparison on the ''harvest a log'' task. Vicuna-13b has lower performance due to its poor coding ability. Mixtral-8x7B works much better than Vicuna-13b. Open-sourced methods are continuously improving, making them promising for future agent development.

Q3. How VLMs could be leveraged in other environments?

A3. Thanks for this good question! VLMs can function as more effective critic agents in our framework. While LLMs can only indicate whether the agent succeeded with its coded actions, VLMs can explain why it failed in the environment. As shown in Fig. 9 of the attached pdf, GPT-4V provides more detailed feedback across different environments. For example, in Minecraft, it can identify that the agent keeps attacking the ground instead of finding the cow. In the driving simulation environment, it can note that the vehicle is gradually drifting off the road. This feedback can be used by both our fast and slow agents for self-improvement.

Q4. Generalization to other environments.

A4. We acknowledged this concern and addressed it to some extent in Appendix Section D. It is difficult to find real-world environments like Minecraft, which require both high-level planning and low-level control. We applied our methods to some robotic tasks that demand both long-horizon planning and precise motor execution. Some results are shown in Fig. 7 and Fig. 8 in the attached pdf.

• Kitchen Environment Training: Fig.7 illustrates the RL training process in the Kitchen environment[1]. The vertical axis represents the success rate, and the horizontal axis represents the number of training steps. Inserting coded motion planning into the action space accelerates learning. Our method learns faster compared to the baseline.
• Furniture Environment Demonstration: In Fig.8, we present a qualitative demonstration of the Furniture environment[2]. The motion planning action effectively aids in hole-finding tasks during table and chair assembly. The baseline struggles to find the correct location at the same training step.

[1] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, 2019

[2] Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks, 2021

Modifications are needed for different domains, such as adjusting the task descriptions in the prompts. The powerful zero-shot capability of GPT should ensure generalization ability.

Dear Reviewer LWBp,

Thank you for appreciating our work with valuable suggestions. We address your questions below.

Q1. Generalization to other environments.

A1. Thanks for this suggestion! We acknowledged this concern and addressed it to some extent in Appendix Section D. It is difficult to find real-world environments like Minecraft, which require both high-level planning and low-level control. We applied our methods to some robotic tasks that demand both long-horizon planning and precise motor execution. Some results are shown in Fig. 7 and Fig. 8 in the attached pdf.

• Kitchen Environment Training: Fig.7 illustrates the RL training process in the Kitchen environment[1]. The vertical axis represents the success rate, and the horizontal axis represents the number of training steps. Inserting coded motion planning into the action space accelerates learning. Our method learns faster compared to the baseline.
• Furniture Environment Demonstration: In Fig.8, we present a qualitative demonstration of the Furniture environment[2]. The motion planning action effectively aids in hole-finding tasks during table and chair assembly. The baseline struggles to find the correct location at the same training step.

[1] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, 2019

[2] Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks, 2021

Yes, modifications are needed for different domains, such as adjusting the task descriptions in the prompts. The powerful zero-shot capability of GPT should ensure generalization ability.

Q2. How are these components synergistically combined?

A2. Thanks for this suggestion! The key technical novelty is illustrated in Fig.2 of the paper. Incorporating high-level actions generated by LLMs into the RL action space captures the core ideas presented. This core design allows the agent to choose between RL and code-as-policy.

Q3. More detailed explanations or examples.

A3. Thanks for this suggestion!

• Slow Agent: This agent will decide which parts can be coded and which parts should be learned. The first iteration loop is designed to correct decision-making errors.
• Fast Agent: This agent will code the coding tasks from the slow agent. The second iteration loop is designed to correct coding errors.
• Critic Agent: The critic agent provides feedback from the environment for each code execution. The output from the critic agent after one step serves as feedback for the fast agent, while outputs from a sequence of steps serve as feedback for the slow agent.

Q4. Can the fast agent also call the RL agent inside its loop?

A4. Sorry for the confusion. The fast agent cannot perform RL training. Its role is to generate code based on the requirements provided by the slow agent.

Q5. The architecture and inputs of the PPO RL agent.

A5. Sorry for the confusion. Yes, we will learn the weights for each RL sub-action (marked orange in Fig. 3).

1. The "Harvest log" is sent from the slow agent to the fast agent in text format.
2. The fast agent generates high-level codes for the action.
3. Coded actions are inserted into the RL action space.
4. RL training is performed after this insertion.

Q6. Action space vs reward function.

A6. Thanks for this good question! We have conducted ablation studies, as shown in Tab. 5. Using an action space with higher-level actions can shorten the task horizon and reduce reward sparsity. This design is more efficient as it leverages the coding ability and common sense of LLMs. For instance, LLMs understand that it takes 10 attacks to break a tree. When designing a reward function, even though "10 attacks" yields a high reward, randomly sampling those 10 individual attacks is a time-consuming process. In contrast, with action space design, an action like "attack 10 times" can be directly generated, resulting in an immediate high reward.

Q7. Typos.

A7. Thank you for pointing that out. The correct values are 34% and 65%. We will correct this in the revision.