Eureka: Human-Level Reward Design via Coding Large Language Models

Dear reviewer JeMn,

Thank you reviewer for your thoughtful comments and suggestions! Here, we respond to the specific questions and comments the reviewer raises. Please let us know if you have lingering questions and whether we can provide any additional clarifications during the discussion period to improve your rating of our paper.

Question/Comment 1: What is the computational cost of Eureka?

We have added the computation resources as well as run time of Eureka in Appendix D.4. In short, a single run of Eureka, with the default hyperparameters in our experiments, can be executed on 4 32GB memory GPUs (e.g. NVIDIA V100) and take less than one day of wall clock time even for the most complex environments (e.g. bimanual manipulation). Here, we also address some related questions from the reviewer:

(a) For each sampled reward, how many environments do you create to train?

Response: The number of environments created for training is determined by the task-specific hyperparameters set in the original benchmarks. For example, the number of environments is 2048 for Humanoid. We find the default number to work well for all our rewards, demonstrating the robustness of our Eureka approach.

(b) How do you decide when to terminate each running experiment?

Response: We do not perform early termination for RL runs. Fixing an environment, each RL run will execute for a fixed number of episodes according to the default task-specific hyperparameters provided in the benchmarks. That said, we believe that it is possible to use early termination to filter out unpromising reward candidates. This is an effective strategy for reducing the overall computational cost, and we leave it to future work for investigating this as computational efficiency is orthogonal and complementary to our core contribution of automated reward design.

(c) Is it possible that RL training is not long enough so it will miss an effective reward sample?

Response: This is certainly possible, but in practice, we find the benchmark default training hyperparameters are sufficient for discovering good reward functions across our tasks. With additional compute budget, we can also increase the training time to reduce this kind of false negative.

(d) What will be the total time of finding a good reward based on what kind of device?

Response: All our experiments took less than one day of wall clock time, and each individual experiment can be done on 4 V100 GPUs.

Question/Comment 2: Can this method be used for tasks with diverse objects?

Response 2: Yes, Eureka is a general reward design method that is not specific to any particular object type. As an example, we have demonstrated that Eureka can design effective reward functions for the novel task of pen re-orientation and spinning. The same procedure also discovers effective rewards for re-orienting a cube on a different robot hand (i.e., AllegroHand). In addition to existing evidence in the paper, we note that prior works on in-hand object re-orientation often uses the same reward function for diverse objects [1,2], and it is reasonable to assume that Eureka rewards, which work better than one such prior reward, can be applied across objects as well.

Question/Comment 3: Can this method still work with tasks that require images as input?

Response 3: Yes, we can infer state variables relevant to reward definition from vision. State estimation has been effectively applied in many prior works for vision-based robotic learning in real world scenarios [3,4]. While state estimation techniques may not always be accurate in complicated scenes, we believe that this is a challenge facing all real-world policy learning methods, and research progress on these fronts can readily benefit Eureka’s application to real-world tasks. On the other hand, if it is only the final policy that needs to act from visual inputs, we can use Eureka to first design the most effective reward function in simulation and then distill the state-based policy to a vision-based policy; this strategy has been effectively applied in many prior Sim2Real approaches [5].

Question/Comment 4: Can the success/fitness function serve as initialization for Eureka reward search?

Response 4: Yes, this is indeed possible. We have conducted a similar experiment in which we use a human-supplied reward function to initialize Eureka (Section 4.4). Note that supplying the fitness function, which is also the optimization objective for Eureka reward search, is likely to lead to reward functions that can better optimize it. For example, Eureka can choose to use the fitness function to create a success bonus in its reward generations.

We thank the reviewer again for their time and effort helping us improve our paper! Please let us know if we can provide additional clarifications to improve our score.

From my perspective, tasks in these environments, usually require a human to take a lot of time and effort to shape a good reward, and when the human is designing the reward, the human has ultimate access to every part of the environment code, and even discuss with other people. Thus, Eureka can design a reward that can get such results is indeed useful. Compared to baseline, eureka also does not require task-specific prompts, which can be used in different tasks, this has already improved a lot for "generality". Even for a human to design a reward, humans need learn some specific task knowledge for different environments. And with more rigorous experiments have been provided, I would like to raise my score.

Dear reviewer XD65,

Thank the reviewer for your thoughtful comments and suggestions! Here, we respond to the specific questions and comments the reviewer raises. Please let us know if you have lingering questions and whether we can provide any additional clarifications during the discussion period to improve your rating of our paper.

Question/Comment 1: Unrealistic assumption of access to the environment source codes; reinforcement learning setups only require access to a black-box simulation.

Response 1: We would like to clarify that the problem setting of our work is not reinforcement learning, but rather reward design. This is traditionally done by humans who write and refine reward function codes in trial-and-error fashion in order to induce the desired behavior; Eureka automates this at scale. Its outer loop is a code generation problem, and it is standard to assume access to parts of the source code in order to generate/complete other parts of the code [1]. Furthermore, as we have clarified in our updated manuscript, Eureka’s “environment as context” procedure requires only the environment’s state and action variables to be exposed. This information can be supplied, for example, via an API without requiring the full environment code; see Example 1 in Appendix E, where we showed that a documentation-style description of the environment variables is sufficient for Eureka to generate effective reward functions. Eureka’s inner-loop does use RL to evaluate its reward candidates, and this inner-loop indeed only accesses the environment with Eureka generated reward function as black-box simulation. We hope that this explanation helps clarify the misunderstanding regarding Eureka’s use of environment as context and the problem setting it tackles.

Question/Comment 2: Strong Assumptions on fitness function F(); This limitation implies that the method is applicable only to tasks that come with known ground-truth reward functions.

Response 2: We would like to clarify that the ground-truth reward functions are simply sparse task success criteria, which are easy to define in practice. On all our 29 benchmark tasks, they can be written in one line of code (see Appendix B), which is much shorter and simpler than the human-written shaped reward functions that we compare Eureka to. Therefore, we believe that “tasks that come with known ground-truth reward functions” are not difficult to satisfy in practice, and Eureka readily applies to all such tasks with programmatic success conditions. This is a large space of tasks, and we have extensively evaluated a broad spectrum of robotic tasks in our experiments. Therefore, we believe that our approach’s applicability is already broad.

Furthermore, we do even show results where no fitness function is analytically expressed. In Section 4.4, we have studied cases where the desired behaviors are difficult to express via a mathematical function, such as running in a stable gait. In these cases, we have introduced Eureka from Human Feedback, which allows humans to express the “fitness” feedback in textual form, bypassing an analytical fitness function. We have shown that Eureka is indeed compatible with such a notion of fitness and can effectively generate reward functions that better align with human intention. In our limitation section in the updated manuscript (Section 5), we have also discussed the possibility of using a vision-language model (VLM) to construct automated textual feedback to improve the reward functions, providing a scalable alternative to human-based fitness feedback.

Please let us know whether our response has sufficiently addressed your concern about the fitness function; we are happy to provide more clarification as we believe that this is an important point that we’d like the reviewer to reach an agreement with.

Question/Comment 3: Clarification on how the pre-trained policy was obtained.

Response 3: The pre-training stage is identical to the original Shadow Hand environment in the Isaac Gym benchmark suite we study, except that we swapped out the cube object to the pen object. The application of Eureka for this stage is identical to the benchmark tasks, in which Eureka iteratively discovers better reward functions. The best Eureka reward is then used to train a policy to solve the task; the resulting converged policy from this training is the “pre-trained” policy.

Question/Comment 4: Is the training process – both pre-training and fine-tuning stages – guided by the reward functions derived using the LLM-powered reward design method proposed in this paper? Are these reward functions identical?

Response 4: Yes, during both stages, the policies are trained using the best Eureka-discovered reward function from the pre-training stages. These reward functions are hence identical.

I am actually curious about the doubts of assumptions. From my perspective, for example, when a human decides to tackle a new task following a sim2real pipeline based on RL algorithm[1], the human might first establish an environment in simulation, in such case, the human also needs to write a reward function. And it seems reasonable for me to assume knowledge of the environment and sparse rewards.

[1]Chen, Yuanpei, et al. "Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation." 7th Annual Conference on Robot Learning. 2023.

Dear reviewer hi7d,

Thank you for your thoughtful comments and suggestions! Here, we respond to the specific questions and comments the reviewer raises. Please let us know if you have lingering questions and whether we can provide any additional clarifications during the discussion period to improve your rating of our paper.

Question/Comment 1: The main part of the paper lacks technical details. For example, in the subsection on evolutionary search, what is the exact input, outputs, or about the specific method being used?

Response 1: We have revised our manuscript to clarify the technical approach in Section 3 and added pointers to the Appendix for specific prompts and examples when relevant. Please let us know whether there are additional places you’d like us to add details.

In the context of evolutionary search iteration, the inputs are (a) the best reward function from the previous iteration, (b) its reward reflection (Section 3.3), and (c) the prompt instruction for how to mutate the provided reward function given its reward reflection feedback. These inputs will be the prompt to the LLM, which then generates $K$ i.i.d. outputs, corresponding to $K$ reward samples.

Question/Comment 2: Clarifications on reward reflection. For example, the algorithm specifies the reward functions and their evaluations are provided as inputs, but the text also talks about the choices of RL method that are important for this part.

Response 2: Thanks for this suggestion! We have updated our manuscript to point to concrete reward reflection examples in App. G.1. The reviewer is correct in that reward reflection is a textual evaluation of reward function (e.g., summarizing the training dynamics of its components). The subsequent discussion about the choices of RL method is meant to provide intuitive justification as to why reward reflection is helpful. We have improved the writing to more cleanly separate the procedure itself from its motivations. Please let us know if there is any remaining clarification we should include.

Question/Comment 3: For “environment as context”, what level of detail does the environment need to be described. For simulators different from Isaac, what information do you expect you need to provide for this to work.

Response 3: “Environment as context” requires only the state and action variables in the environment to be exposed in the source code; in practice, as described in Appendix D, we have an automatic script to extract just the observation function from all raw environment source code. Note that our prompts and the observation source code do not reveal identity of the simulator, so Eureka is already general and we expect the same level of information, namely the names of state and action variables in the environment, to be needed for other simulators. To empirically validate this claim, we have also added an experiment testing Eureka on the Mujoco Humanoid environment; its observation code is included in Appendix E. As seen, the Mujoco variant exposes variables in a comment block whereas the Isaac Gym variant exposes them as class attributes, but since both contain sufficient state and action information, Eureka is able to competently generate effective reward functions in both cases, showcasing its generality.

Question/Comment 4: A limitation section should be included. Do you expect the approach to be useful when we do not have full access to the code of the environment, or if the environment is the real world.

Response 4: Thanks for this suggestion! We have included a limitation section in our updated manuscript that details Eureka’s current limitations and several future work directions to overcome them.

We do expect the approach to be useful when we do not have full access to the code of the environment. As discussed in the previous question, Eureka only needs to know the state and action variables of the environment in its context to generate reward functions; this information could be automatically extracted (as in our current approach) or accessed via an API (as the reviewer suggests) without access to the full environment code.

If the environment is real world, as we expanded in our new limitation section, we believe that Sim2Real approaches are particularly promising. This approach allows Eureka to readily combine with mature Sim2Real techniques to enable real-world transfer of policies learned in simulation. Another approach is to use state-of-art state estimation techniques to construct a symbolic representation of the real-world environment to allow Eureka to define reward functions. State-estimation based solutions have been attempted by many other works targeted at learning and deploying in the real world [1,2].

Dear reviewer DCBe,

Thank you for your thoughtful comments and suggestions! Here, we respond to the specific questions and comments the reviewer raises. Please let us know if you have lingering questions and whether we can provide any additional clarifications during the discussion period to improve your assessment and rating of our paper.

Question/Comment 1: Human rewards are zero-shot and not tuned for many RL trials to further improve the performance. The comparison to human rewards may be unfair.

Response 1:

First, there are good reasons to believe that the human reward functions are representative of expert-level human reward functions and are not “zero-shot”. Given that they are written by RL researchers who designed the benchmarks, we hypothesize that they are likely carefully tuned to ensure that they are optimizable by their choice of reinforcement learning algorithms – so that the benchmarks are solvable; we discuss this aspect in Section 4.1. In particular, the authors of the bi-manual Dexterity benchmark wrote the following excerpt in Appendix 2.1 of their paper, confirming judicious and iterative reward shaping, as per common practice:

“Designing a reward function is very important for an RL task… because the scenarios of each task are different, the hyperparameters of the reward function will inevitably be different. We have tried our best to avoid manual reward shaping for each task provided that RL can be successfully trained.”

This excerpt confirms that their reward functions are not zero-shot. Furthermore, their codebase also has commented-out reward snippets (e.g., https://github.com/PKU-MARL/DexterousHands/bidexhands/tasks/shadow_hand_block_stack.py#L1456, https://github.com/PKU-MARL/DexterousHands/bidexhands/tasks/shadow_hand_kettle.py#L1488), which are likely prior attempted reward functions/components that did not work as well.

Given this and the fact that we cannot control for how the original human reward engineering took place, we believe that our experiments are a valuable and meaningful comparison of Eureka’s reward generation capability against expert humans.

Second, we note that the purpose of comparing against the official human reward functions is to be able to ground the quality of Eureka reward functions against some pre-existing reward functions that we know to be competent. It is certainly possible to come up with better reward functions to compare against, but we instead believe that the interesting scientific finding is rather the fact that Eureka is capable of improving reward functions over multiple iterations without manual prescription. It is this improvement capability that enables Eureka reward functions to eventually surpass human reward functions for various tasks despite the inferior quality of Eureka’s initial reward generations (Iteration 1 in Figure 1).

Question/Comment 2: A baseline that searches the weights for different reward terms in the human rewards should be considered.

Response 2: Thank you for this suggestion. We have implemented this baseline for Humanoid in the Isaac suite. More specifically, we choose 4 components from the respective human reward functions and grid search the best weight combination from having 3 possible values for each weight: 0.1x, 1x, 10x the original value. Therefore, we search over 81 tuned human reward functions (80 new ones), making the comparison fair to Eureka in terms of the total reward candidates searched. The results are below:

As shown, while Human (Tuned) does improve, the relative ordering between Eureka and Human does not change. The relatively small gap between Human and Human (Tuned) also suggests that the official human reward function is reasonably well-tuned and competent. Finally, we note that this is an unfairly advantaged baseline to Eureka because Eureka is meant to design a reward function from scratch (hence a reward design algorithm) whereas this baseline tunes an existing reward function. Even then, given the same reward sample budget, Eureka’s best reward generated from scratch outperforms the best tuned human reward function.

Question/Comment 3: Discussion on whether the proposed approach handles the difference between optimizing for the internals of the simulation vs its sensing/acting interface.

Response 3: Thank you for this suggestion. We have confirmed that our observation code does not leak details about the simulation internals; an example of extracted observation code is in Appendix D (Example 1). In general, we have also revised our manuscript to include a more concrete description of our “environment as context” procedure to clarify that observation code needs to expose only usable state variables in the environment for our proposed approach to work well.

Dear reader,

Thank you for reading our paper and providing thoughtful comments! We are grateful for your suggestions and have added several experimental results in our updated manuscript addressing your concerns over our experimental protocols. Here, we respond to the specific questions the reader raises and believe that there are several misunderstandings regarding our paper that we would like to address. Please let us know if you have lingering questions and we are happy to answer any additional inquiries.

Question/Comment: Section 4.2 implies considering the maximum performance achieved by any iteration of PPO in any of the runs.

Response: This is incorrect, and we believe that there is a misunderstanding regarding our evaluation protocol. Our evaluation protocol is as follows. We first obtain a reward function using each approach (ours and the baselines). Then, for each of these reward functions, we perform 5 PPO runs. For each PPO run, we consider the best checkpoint out of 10 checkpoints taken at fixed training intervals, and evaluate the performance of each checkpoint by averaging the performance over 1000+ test rollouts that randomize over initial environment states. Then, we average the performance over the best checkpoints from all the PPO runs.

Importantly, this same procedure is applied to all final reward functions, including the ones from our baselines. In particular, the maximum is taken over the same number of checkpoints for each approach, so there is no bias towards our approach. Thus, this protocol is a fair procedure that does not advantage our method. Furthermore, we believe that this is a reasonable method for reporting a reward function’s performance as the performance of the checkpoint is evaluated using a large number of rollouts, and an average is taken over all individual RL runs to mitigate outliers.

We acknowledge that these details may not have been conveyed in sufficient detail in our paper, and we have updated our manuscript to leave no ambiguity. Furthermore, we have added several additional evaluation metrics, as the reader suggested, to complement our core evaluations in the paper, offering a holistic picture into Eureka’s efficacy in reward design.

Question/Comment: Additional evaluation metrics, such as inter-quantile mean (IQM), should be considered.

Response: We agree and thank you for this suggestion. We have computed Mean, Median, IQM, and probability of improvement metrics with 95% stratified bootstrapped intervals in Appendix F, Figure 11 and 12, using the official code from Agrawal et al., 2021. These new results demonstrate that Eureka is consistently effective and outperforms baselines over different evaluation metrics, supporting that it indeed is capable of generating well-behaved reward functions across tasks.

Question/Comment: Error bars should be included to contextualize the reported average success rate in Figure 4.

Response: We agree and thank you for this suggestion. In our updated manuscript, the main result of Figure 4 as well as Figure 10 in Appendix have now included error bars, which compute standard error over tasks and distinct Eureka and baseline runs. As shown, Eureka’s reward functions are effective across tasks and are equally robust to variances within RL runs as human-written reward functions. We thank the reader for suggesting these additions as we believe they do strengthen our claim.

Question/Comment: Does Eureka use orders of magnitudes of additional samples compared to the baseline? RL Training curves should be included.

Response: For the final training runs, Eureka uses the exact same number of environment samples as the baseline final reward functions (e.g., L2R and Human) for training. As stated in Section 4.2 of our paper, “for each task, all final reward functions are optimized using the same RL algorithm with the same set of hyperparameters”; this fixed set of hyperparameters includes the predetermined number of environment samples used for RL training. Given that the hyperparameters are chosen by the original benchmark designers to ensure the Human reward function is effective, we believe that our evaluation setting in fact disadvantages Eureka’s reward functions as Eureka does not tune the RL algorithm (PPO) to work well with its reward functions.

We have included the aggregate RL training curves on the 20 Dexterity tasks in Figure 10, Appendix F. As shown, Eureka reward functions exhibit better sample efficiency compared to the baselines, but given that the RL algorithm is not tuned to Eureka rewards, the learning curves do exhibit slightly more variance throughout training.

Just want to double-check about the one final reward for each task, do the author just run once for each task using Eureka (with different seeds or the same seeds for different tasks?) and then use the final reward to run the subsequent 5 runs with different seeds?