Text2Reward: Reward Shaping with Language Models for Reinforcement Learning

2. L2R can only use a sum of reward terms v.s. T2R can write stage and dense rewards such as if-else statements:
   1. Our use of dense rewards, as opposed to sparse rewards, allows for a more nuanced and continuous evaluation of agent behaviors, crucial for complex tasks that involve many steps.
   2. For example, the stack cube task needs "pick up cubeA, place it on cubeB, release cubeA and make sure cubeA is static after releasing". However, L2R will lead to a contradiction between the reward term that encourages "pick up cubeA" and "release cubeA" when writing them in a simple summation expression.
   3. To show the necessity of dense rewards, we trained RL with oracle-sparse-reward on ManiSkill2 as an additional experiment in Appendix E. This baseline adapts the oracle expert-written reward codes to mirror the L2R's format (the sum of a set of individual terms) by removing all if-else statements and only keeping functional reward terms, while disregarding the original L2R's inability to utilize point clouds for distance calculations. While "oracle-sparse-reward + RL" yielded results comparable to the zero-shot setting for 3 simpler tasks, it completely can not work (i.e. success rate ≈ 0) on the remaining 3 more challenging tasks, underscoring the limitations of sparse rewards.
3. L2R is model-based and needs additional expert effort v.s. T2R is model-free and can easily be transferred to new environments and real-world application:
   1. The efficacy of the L2R approach is significantly contingent on the performance of Model Predictive Control (MPC). While it is a well-accepted notion that model-based methods, like MPC, outperform model-free methods when an accurate world model is accessible, the practical deployment of MPC is constrained by the requirement for such a model. Crafting this world model demands additional expertise and is not straightforwardly translatable to real-world applications due to this dependency.
   2. The selection of reward term, design reward term template, and even finding appropriate reward term weight all needs an extra large amount of expert effort.

---

W2: Use of open-source LLMs for reproducibility and accessibility.

A: Thanks for your suggestion! As you mentioned, it is a common practice to use the advanced GPT models from OpenAI or close source model PaLM from Google for novel applications and studies in previous work (Chain-of-Thought[1], Zero-shot planner[2], SayCan[3], ReAct[4]). For the reproducibility, we used open-source algorithms and environments and provided full reproducible prompts and code. Following your suggestion, we also add experiments in the application of our method using open-source or publicly available models on Llama-2 and Code-Llama (Llama-2 further pretrained on code corpus). We include these findings in our revised manuscript and provide a more comprehensive evaluation of our method's capabilities across different models in Appendix F. The results show that the open-source models still have a huge gap with the most advanced models in the difficult task of reward function generation and reflect the necessity of using the most advanced one to demonstrate the possibility of our method.

[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, et al,. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://arxiv.org/abs/2201.11903

[2] Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. https://arxiv.org/abs/2201.07207

[3] Michael Ahn, Anthony Brohan, Noah Brown, et al., Do As I Can, Not As I Say: Grounding Language in Robotic Affordances https://arxiv.org/abs/2204.01691

[4] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. https://arxiv.org/abs/2210.03629

---

Minor Issues 1: Typographical errors and suggestions to include reward function examples in the main paper.

A: We appreciate your suggestions. The typographical errors have been corrected in our revised manuscript. While we acknowledge the value of including reward function examples for clarity, space constraints necessitate a concise presentation. Therefore, only a key example is showcased in Figure 1. We intend to include more comprehensive examples in the camera-ready version, where space permits.

---

Q1: Contribution compared to concurrent work, such as Eureka.

A: Eureka represents an important concurrent development in the field, we all address the important problem of dense reward specification in reinforcement learning, using a method based on zero-shot and few-shot prompting of a language model, as Reviewer#4 pointed out. We believe that our work, alongside Eureka, contributes to the broader understanding and capabilities of LLMs in policy training and reinforcement learning.

Dear Reviewer VQaY,

Thank you for your appreciation and detailed evaluation of our work. We are glad you think the evaluation of our method is comprehensive and your appreciation of the real-robot experiments and human-feedback findings. We have revised our manuscript and addressed several points you mentioned.

We have revised our manuscript and are addressing the specific questions in the following reply, including:

1. discussion on technical contributions
2. incorporation of 2 baselines on 6 tasks of ManiSkill2 to justify technical contributions
3. incorporation of experiments of LLama-2 and Code Llama on 10 tasks of MetaWorld; add discussion and improvement on reproducibility

Please let us know if you have any further questions and we can provide any additional clarifications to help finalize your assessment and rating of our paper.

---

W1: Technical contribution compared to Yu et al. 2023

A: Thanks for your question and advice! We understand your primary concern pertains to the comparison with the L2R paper by Yu et al., 2023, in terms of technical contributions. We wish to clarify there are fundamental differences between L2R and our method (T2R).

Our method's technical contribution extends beyond L2R in three key aspects:

1. L2R can only use limited reward terms v.s. T2R can use any reward term
2. L2R can only use a sum of reward terms v.s. T2R can write stage and dense rewards such as if-else statements and can incorporate any other Python feature
3. L2R is model-based and needs huge expert effort (for building the MPC model and writing reward term templates and even weights) v.s. T2R is model-free and can easily be transferred to new environments and real-world application

To demonstrate these, we have incorporated two new experiments:

1. We introduce a new baseline of "L2R prompt + RL" in the main body of our paper and test the baseline on 6 ManiSkill2 tasks. "L2R prompt + RL" was only deployable on 2 tasks due to the limitation of its reward terms (contribution#1), falling short in the remaining 4 tasks, underscoring the limitations of the task coverage of L2R.
2. We introduce an upgraded version of L2R --- "oracle-sparse-reward + RL" --- in Appendix E and test this baseline on 6 ManiSkill2 tasks to further demonstrate the importance of stage and dense reward. While "oracle-sparse-reward + RL" yielded results comparable to the zero-shot setting for 3 simpler tasks, it completely can not work (i.e. success rate ≈ 0) on the remaining 3 more challenging tasks, underscoring the limitations of sparse rewards (contribution#2).

For more detailed explanation of these three differences:

1. L2R can only use limited reward terms v.s. T2R can use any reward term:
   1. The use of Model Predictive Control (MPC) within the L2R framework restricts the reward term to a twice-differentiable norm with its minimum at zero, typically the L2 distance.
   2. This presents a limitation when dealing with the ManiSkill2 benchmark, which uses a point cloud to depict complex, articulated rigid bodies with intricate surfaces, offering a more realistic representation of real-world scenarios. L2R’s inability to go beyond the L2 distance measure between vectors restricts its representation of such complex geometries.
   3. In contrast, our method's flexibility supports a broader spectrum of coding packages and styles, thus enhancing its adaptability across various environments and tasks.
   4. Our experimental results further highlight this distinction: when testing the baseline on 6 ManiSkill2 tasks, “L2R prompt + RL” was only deployable in 2 tasks due to the limitation of its reward terms, falling short in the remaining 4 tasks.

Dear Reviewer bPzM,

Thank you for your kind feedback and constructive comments. And thanks for your recognition of our work in addressing the important problem of dense reward specification in reinforcement learning, using language model prompting.

We have revised our manuscript and are addressing the specific questions in the following reply, including:

• discussion on data leakage and generalization ability
• incorporation of experiments on Llama-2 and Code-Llama on 10 tasks of MetaWorld; discussion and improvement on reproducibility

Please let us know if you have any further questions and we can provide any additional clarifications to help finalize your assessment and rating of our paper.

---

W1: The significance of our work

A: We appreciate the reviewer's perspective. However, we respectfully disagree it is limited to the significance of the method this work proposed. To clarify, we want to emphasize the promise of this work for different communities (RL, Robotics, and NLP) as follows:

For the reinforcement learning community, it is promising to use LLMs to automate policy learning by generating reward functions that are difficult to write down by hand. It is not only an important application scenario for replacing human experts in writing rewards functions but also a novel way that has never been shown before to pour LLMs’ knowledge and abilities into the RL for robotics and control, as well as games. Our work shows promising initial results for further exploration in the future (as shown in the main experiments of Figure 2).

For the robotics community, one of the most important things in the field of robotics is how to learn low-level motor skills at large scales efficiently, scalably, and automatically (or in a self-supervised manner) [1][2]. We believe that by leveraging LLMs, our method (automatic text-to-reward) takes a solid step toward automatically learning large-scale motor skills.

For the natural language processing community, our work proposes text-to-reward as a hard task for code generation and shows a huge gap in performance between the most advanced model and ones from the open-source community (as shown in the newly added experiments on Llama-2 and Code-Llama in Appendix F), showing the potential of improving in this aspect for people in the NLP community.

Overall, our method could be an important direction in the future as a novel way to pour the knowledge learned from LLMs into general RL, low-level robotics skills learning and control, which are not the focus of previous studies that focus on high-level planning, and the method could get more effective with the growth of LLMs.

[1] Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Zackory Erickson, David Held, Chuang Gan, RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation https://arxiv.org/abs/2311.01455

[2] Open X-Embodiment: Robotic Learning Datasets and RT-X Models, https://arxiv.org/abs/2310.08864

---

W1 & Q1: Generalization to environments not included in GPT-4's training data.

A: Thanks for your question! We understand your concern regarding the potential data leakage of environment and reward code in GPT-4 and other LLMs, which might lead one to assume that the model's ability to generate high-quality code stems from exposure to "golden environments" and "golden examples."

However, we would like to clarify three sources of evidence from our paper and previous work:

1. Evidence from the input side (environment and prompts) and output side (generated reward function) of our proposed method for reward function generation;
2. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper;
3. Evidence from previous research showed the generalization capability on code generation problems of new domains that are not included in the training set.

1. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation.

As much as you suggest using a simple, purpose-designed simulated environment, we would like to clarify that the Python classes and properties for our prompts were crafted by our team and differ significantly from the environment's original code. We then map the generated code to the environment's original code using a one-to-one deterministic mapping.

import numpy as np

def compute_dense_reward(self, action): 
    # Normalize action 
    action = np.clip(action, -1, 1) 

    # Calculate distance between gripper and cube 
    gripper_pos = self.robot.ee_pose.p 
    cube_pos = self.cubeA.pose.p 
    dist_gripper_cube = np.linalg.norm(gripper_pos - cube_pos) 

    # Calculate distance between cube and goal 
    goal_pos = self.goal_position dist_cube_goal = np.linalg.norm(goal_pos - cube_pos) 

    # Check if the robot is grasping the cube 
    grasping_cube = self.robot.check_grasp(self.cubeA) 

    # Define reward components 
    reward_dist_gripper_cube = -1.0 * dist_gripper_cube 
    reward_dist_cube_goal = -1.0 * dist_cube_goal 
    reward_grasping_cube = 1.0 if grasping_cube else -1.0 

    # Define weights for reward components 
    weight_dist_gripper_cube = 0.3 
    weight_dist_cube_goal = 0.5 
    weight_grasping_cube = 0.2 

    # Calculate total reward 
    reward = weight_dist_gripper_cube * reward_dist_gripper_cube \ 
                + weight_dist_cube_goal * reward_dist_cube_goal \ 
                + weight_grasping_cube * reward_grasping_cube 

    # Regularization on action 
    reward -= 0.01 * (action ** 2).sum() 
    return reward

import numpy as np

def compute_dense_reward(self, action):
    reward = 0.0

    is_obj_placed = np.linalg.norm(self.goal_position - self.cubeA.pose.p) <= 0.025
    is_robot_static = np.max(np.abs(self.robot.qvel)) <= 0.2

    success = is_obj_placed and is_robot_static

    if success:
        reward += 5
        return reward

    tcp_to_obj_pos = self.cubeA.pose.p - self.robot.ee_pose.p
    tcp_to_obj_dist = np.linalg.norm(tcp_to_obj_pos)
    reaching_reward = 1 - np.tanh(5 * tcp_to_obj_dist)
    reward += reaching_reward

    is_grasped = self.robot.check_grasp(self.cubeA, max_angle=30)
    reward += 1 if is_grasped else 0.0

    if is_grasped:
        obj_to_goal_dist = np.linalg.norm(self.goal_position - self.cubeA.pose.p)
        place_reward = 1 - np.tanh(5 * obj_to_goal_dist)
        reward += place_reward

    return reward

Dear Reviewer XwRM,

Thank you for your thoughtful review and appreciation of our work! We are glad to hear that you acknowledge our contribution to expanding the research of Large Language Models (LLMs) to low-level robotic control. Text2Reward (T2R) avoids the need for extensive manual templating for each robotic form or task and is adept at creating versatile, free-form reward functions.

We have revised our manuscript and are addressing the specific questions in the following reply, including:

• clarifications on details
• discussion on data leakage and generality.

Please let us know if you have any further questions and we can provide any additional clarifications to help finalize your assessment and rating of our paper.

---

W 1,2,3: Data leakage in GPT-4

A: Thanks for your question! We understand your concern regarding the potential data leakage of reward code in GPT-4 and other LLMs, which might lead one to assume that the model's ability to generate high-quality code stems from exposure to "golden examples."

However, we would like to clarify three sources of evidence from our paper and previous work:

1. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper;
2. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation;
3. Evidence from previous research showed the generalization capability on code generation problems of new domains that are not included in the training set.

1. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper

We did experiments on learning novel skills in MuJoCo, as shown in Table 1 and Figure 4, T2R generates reward codes under our created new instructions for novel skills. We also show the generated reward function codes in Appendix D.2.

And for your concern about Christiano et al., 2017, as far as we know, although a hopper backflip was demonstrated in their work, this behavior is learned from human preferences using a neural network reward model, rather than using a symbolic reward code. Furthermore, our generated reward function (shown in Appendix D.2) is very different from what they showed as an expert-written case in their blog (https://openai.com/research/learning-from-human-preferences) of this work.

2. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation.

From the input end, one piece of evidence, as shown in Appendix C, is that the prompts we provided to LLMs like GPT-4 for our T2R method are derived from expertly abstracted class Python representations, with specific attributes and function names coined by us, not exact snippets from the ManiSkill2 or MetaWorld GitHub repositories. This rules out the possibility of the language model plagiarizing directly through context memory since previous work [1] has attempted to perturb instructions used for code generation and found significant performance drops.

On the other hand, we also have preliminary results that entering direct commands like "generate a reward function for crawler" or "pick the cube" into GPT-4, or inputting the ManiSkill2 environment code context into GPT-4 to let it complete the compute_dense_reward section, none of which were successful. This not only demonstrates the inadequacy of mere memorization but also validates the effectiveness of our method.

import numpy as np

def compute_dense_reward(self, action): 
    # Normalize action 
    action = np.clip(action, -1, 1) 

    # Calculate distance between gripper and cube 
    gripper_pos = self.robot.ee_pose.p 
    cube_pos = self.cubeA.pose.p 
    dist_gripper_cube = np.linalg.norm(gripper_pos - cube_pos) 

    # Calculate distance between cube and goal 
    goal_pos = self.goal_position dist_cube_goal = np.linalg.norm(goal_pos - cube_pos) 

    # Check if the robot is grasping the cube 
    grasping_cube = self.robot.check_grasp(self.cubeA) 

    # Define reward components 
    reward_dist_gripper_cube = -1.0 * dist_gripper_cube 
    reward_dist_cube_goal = -1.0 * dist_cube_goal 
    reward_grasping_cube = 1.0 if grasping_cube else -1.0 

    # Define weights for reward components 
    weight_dist_gripper_cube = 0.3 
    weight_dist_cube_goal = 0.5 
    weight_grasping_cube = 0.2 

    # Calculate total reward 
    reward = weight_dist_gripper_cube * reward_dist_gripper_cube \ 
                + weight_dist_cube_goal * reward_dist_cube_goal \ 
                + weight_grasping_cube * reward_grasping_cube 

    # Regularization on action 
    reward -= 0.01 * (action ** 2).sum() 
    return reward

import numpy as np

def compute_dense_reward(self, action):
    reward = 0.0

    is_obj_placed = np.linalg.norm(self.goal_position - self.cubeA.pose.p) <= 0.025
    is_robot_static = np.max(np.abs(self.robot.qvel)) <= 0.2

    success = is_obj_placed and is_robot_static

    if success:
        reward += 5
        return reward

    tcp_to_obj_pos = self.cubeA.pose.p - self.robot.ee_pose.p
    tcp_to_obj_dist = np.linalg.norm(tcp_to_obj_pos)
    reaching_reward = 1 - np.tanh(5 * tcp_to_obj_dist)
    reward += reaching_reward

    is_grasped = self.robot.check_grasp(self.cubeA, max_angle=30)
    reward += 1 if is_grasped else 0.0

    if is_grasped:
        obj_to_goal_dist = np.linalg.norm(self.goal_position - self.cubeA.pose.p)
        place_reward = 1 - np.tanh(5 * obj_to_goal_dist)
        reward += place_reward

    return reward

Dear Reviewer fmuH,

Thank you for your careful review and constructive comments. We are glad you describe our paper as an interesting application with nice results across environments.

At a very high level, we would like to clarify several terms to make sure that we are on the same page. By human oracle we mean the human-written dense rewards provided by the environment (tuned by the authors of the environment paper). By few-shot, we mean using related human-written dense rewards as in-context reward examples, so few-shot itself does not include any human feedback or interactive generation. Furthermore, in the few-shot setting, for each task, we remove its oracle code from the retrieval pool to make sure that the LM does not cheat.

We have revised our manuscript and are addressing the specific questions in the following reply, including:

• Clarification on details
• Additional examples provided in the paper

Please let us know if you have any further questions and we can provide any additional clarifications to help finalize your assessment and rating of our paper.

---

W1: There is no qualitative analysis/discussion of what the source of the improvement is:

W1.1 & W1.2: Why does zero-shot sometimes outperform few-shot? Why does few-shot fail to outperform oracle even though oracle is in context?

A: Thanks for your question! We observed that the LM tends to copy code snippets from the few-shot code when provided. If the human oracle is not optimal, the few-shot human oracles might inadvertently introduce biases or constraints, limiting the LM's creative reward generation.

W1.3: In the cases that few-shot improves on zero-shot, what is the source of this improvement?

A: Robotic tasks are related, especially the dense stages rewards. When few-shot improves on zero-shot, the few-shot examples likely provide relevant, task-specific stage rewards based on which the LM can rewrite.

---

W2: There are missing details in the experiments:

W2.1: Clarification on the human feedback process and its trace in the experiments.

A: In the revised manuscript Appendix D.3, we added a detailed example showing the traces of the whole round to show how human feedback improves the reward code step by step.

W2.2: Human evaluation for locomotion

A: The authors evaluated each rollout. We add this detail to the revised manuscript.

W2.3: Clarification on the setting of Figure 6

A: The locomotion example in Figure 6 is zero-shot + human feedback, but not few-shot (because we do not have a human oracle for all locomotion tasks).

W2.4: The choice of $k$ for the number of neighbors in few-shot context.

A: We set $k$ as 1 in all experiments of few-shot settings.

W2.5 & W2.6 & W2.7: Few-shot and oracle are sometimes non-distinguishable. Few-shot sometimes underperform oracle. Is it always the case that the Oracle code for the task is put into the context for Few-shot?

A: We design the experiment so that the oracle code for the task is never in the context or few-shot. For each task, we remove its oracle code from the retrieval pool to make sure that the LM does not cheat. This explains why few-shot is not necessarily better than oracle because it cannot copy the oracle. When few-shot and oracle are similar, one potential reason is that there is a similar task in the few-shot example, and the LM can adapt that oracle code to fit into this task.

---

W3: The importance of code errors section vs. generation/feedback process details.

A: Thank you for your suggestion! We wish to clarify that when using GPT-4, syntax errors are generally not encountered, as indicated in Table 2, where direct generation has only a 10% error rate. Furthermore, we found that by feeding the error messages from the cases with syntax errors back to GPT-4, all such errors can be rectified. Given that it has become commonplace to use error messages to guide GPT-4 in correcting syntax issues, especially considering that reducing syntax error rates is not the focus of our paper, we choose to not add this as a section.

---

W4: Duration of the iteration loop and consideration of early evaluation.

A: We provide information on the time taken for each iteration loop with the hardware we used and discuss the potential for early evaluation strategies to improve efficiency in Appendix A.

---

W5: Inclusion of code examples for novel behavior tasks.

A: Just as you mentioned, to enhance clarity and understanding, we include reward code examples generated for novel behavior tasks in Appendix D.2. This will provide a clearer picture of how the LM approaches these tasks and the nature of the generated rewards.

For your concerns regarding the qualitative analysis...

I much prefer this change. I do think that this section should refer to both precise performance curves in Figure 2 and examples of generated code in the Appendix (where included, like with Pick Cube) as this makes the information location job much easier on the reader. I don't think this section answers my previous concern that oracle code was being leaked in the case of Pick Cube, and I still don't have a response to this...

Detail clarification

I appreciate these changes.

The importance of code errors...

I think the structure is more informative now.

I think given the many changes I would currently support changing my score from 6 to 8. I would prefer seeing the minor change above before the camera ready, though I realize discussion closes soon.