R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models

We thank the reviewer for the constructive feedback. For the reviewer's questions, we will respond to them one by one as follows.

1. [How are the parents selected, and how is it determined which reward module to insert into another parent?]

We maintain a buffer of reward functions with a fixed size of 5 (Only the top-5 best-performing ones are retained), where each reward function is associated with the success rate of the RL policy it guides. We apply a softmax over these success rates and sample two reward functions based on the resulting probabilities.

We then randomly select reward modules for crossover. The main advantage of this approach is that it does not require any API calls. A potential improvement would be to leverage an LLM to guide the module selection process; however, this typically incurs additional LLM API usage.

2. [Which reward functions are used for training in Figure 2?]

The reward function that achieves the highest RL success rate across all generations is selected for final evaluation, which is consistent with the setting used in Eureka.

3. [Are the trajectories for parameter alignment optimization sampled using the initially randomized policy?]

The trajectories are collected using the trained policies guided by different reward functions within the population.

4. [Are the generated reward functions only suitable for PPO?]

The generated reward functions are applicable to various RL algorithms. To demonstrate this, we conducted experiments on Metaworld, using the generated rewards to guide SAC for policy learning (max success rate and the required steps). The results are as follows:

We observe that R* achieves a 100% success rate on these tasks, matching the performance of the expert-designed rewards.

5. [I am curious whether the authors have considered strategies to address this issue, or at the very least, whether these limitations should be explicitly discussed in the manuscript.]

For complex visual tasks, we believe a feasible direction is to obtain the relative positions of targets through object detection and prediction, and associate them with variables—this typically requires additional training. However, our current method does share the limitation mentioned by the reviewer: it cannot be applied to tasks where low-level information is inaccessible.

6. [The reported Eureka performance seems significantly lower than that reported in the original Eureka paper.]

The main reason lies in our environment configuration, which limits the number of parallel environments. In the original Eureka paper, some tasks use a large number of parallel environments, requiring at least 4 RTX 4090 GPUs to run. We reduce the number of parallel environments to ensure that the program can run with only 40GB of GPU memory. However, this reduction significantly increases the learning difficulty for the algorithm. Besides, we use GPT-4o for our experiments, which may also introduce some differences.

However, to ensure a fair comparison, all experiments are conducted under the same configuration.

The specific configuration of the number of parallel environments is shown in the table below.

7. [typos: In line 6 of the pseudocode]

Thank you to the reviewer for pointing out this typo. We will correct it in the revised revision.

---

We would appreciate it if the reviewer can confirm that the concerns had been addressed and, if so, reconsider the assessment. We’d be happy to engage in further discussions.

We thank the reviewer for the constructive feedback. For the reviewer's questions, we will respond to them one by one as follows.

---

1. [The performance of PPO with oracle reward seems very low to me -- it's much lower than the performance reported in Eureka tasks. Maybe I'm missing something here but I'm not sure why this happens.]

The main reason lies in our environment configuration, which limits the number of parallel environments. In the original Eureka paper, some tasks use a large number of parallel environments, requiring at least 4 RTX 4090 GPUs to run. We reduce the number of parallel environments to ensure that the program can run with only 40GB of GPU memory. However, this reduction significantly increases the learning difficulty for the algorithm.

However, to ensure a fair comparison, all experiments are conducted under the same configuration.

The specific configuration of the number of parallel environments is shown in the table below.

2. [One weakness of this paper is that writing is very verbose and some paragraphs are really long. In particular the ones in the introduction.]

Thank you for the valuable suggestions. We will work on improving the overall clarity and expression throughout the manuscript in the revised revision.

3. [What's the fundamental limitation of Eureka and how is the proposed idea addressing the limitations? Why should the proposed algorithm work better than Eureka?]

The limitations of Eureka are mainly reflected in two aspects:

1. Insufficient utilization of existing knowledge: Eureka relies on the best-performing individual during each reward function update. However, the individual may represent suboptimal solution, and continuously optimizing based on it can lead to poor results. The introduction of a crossover mechanism aims to fully leverage existing knowledge without introducing any API calls, enabling integrated exploration across different modules of the superior reward functions and ensuring more efficient exploitation.
2. Inefficient parameter optimization: Reward functions often involve numerous parameters, such as weights within and between reward components. Iterative optimization of these parameters using LLM is inefficient. To address this, we propose a preference learning-based approach, which makes the gradients of the reward function parameters differentiable. By collecting preference data from the critic population, we optimize these parameters more effectively.

To solve above problems, we propose the reward structure evolution and parameter alignment Optimization. The former focuses on efficient structure search to avoid getting trapped in suboptimal designs, while the latter performs effective parameter optimization. By combining both components, our method enables more efficient reward design compared to Eureka.

---

We would appreciate it if the reviewer can confirm that the concerns had been addressed and, if so, reconsider the assessment. We’d be happy to engage in further discussions.

We thank the reviewer for the constructive feedback and for a positive assessment of our work. For the reviewer's questions, we will respond to them one by one as follows.

1. [they do not discuss Eureka in depth, so it is really hard to estimate the novelty and contribution of this work. ]

Eureka maintains a population of reward functions and iteratively improves them using an LLM. The improvement process involves feeding the LLM with the best-performing reward function discovered in the current population, along with feedback from its RL training process. Based on this information, the LLM adjusts both the reward function logic and its parameters to enhance performance.

The limitations of Eureka are mainly reflected in two aspects:

1. Insufficient utilization of existing knowledge: Eureka relies on the best-performing individual during each reward function update. However, the individual may represent suboptimal solution, and continuously optimizing based on it can lead to poor results.
2. Inefficient parameter optimization: Reward functions often involve numerous parameters, such as weights within and between reward components. Iterative optimization of these parameters using LLM is inefficient.

To address the two issues mentioned above, we propose reward structure evolution and parameter alignment optimization. The former fully leverages high-quality reward functions without introducing any API calls, enabling thorough exploration through crossover among components within superior reward functions. The latter adjusts parameters via preference learning, where a key challenge lies in constructing a reliable preference dataset. To tackle this, we utilize the LLM to generate critic functions—a process that does not involve human participation. Moreover, to ensure the accuracy of preference labels, we build a population of critics and annotate preferences between states through a voting mechanism. Finally, we optimize the reward function parameters based on the learned preferences, providing more accurate reward signals.

1. [The writing is not that well structured. For example paragraphs 2 and 3 are very long and hard to read.]

Thank you for the reviewer’s suggestion. We will improve the presentation in the revised version.

---

We would appreciate it if the reviewer can confirm that the concerns had been addressed and, if so, reconsider the assessment. We’d be happy to engage in further discussions.

We thank the reviewer for the constructive feedback and for a positive assessment of our work. For the reviewer's questions, we will respond to them one by one as follows.

1. [The novelty of the proposed ideas is low, and seem to be a collection of multiple ad-hoc steps]

Our work focuses on reward function construction and enhancement from two key perspectives: reward structure evolution and parameter alignment optimization. The former aims to design meaningful reward components using LLMs and evolutionary principles, while the latter focuses on optimizing the parameters within those reward components.

In reward structure evolution, we introduce code-level crossover operations to fully exploit existing reward function implementations and address potential suboptimality issues.

In parameter alignment optimization, we first use LLMs to construct state-level critic functions. To mitigate potential errors from relying on a single critic, we build a critic population and use it to generate a preference dataset. We then make the numerical values in the reward function code differentiable and apply preference learning to efficiently optimize the reward function.

Our method is primarily designed and optimized based on the challenges associated with reward generation, aiming to construct more efficient reward functions.

2. [the paper is focused mainly of robotic hand manipulation tasks, and there is no mention of this until the experiments section. The paper should do a better job of providing context and motivation.]

Thank you for the reviewer’s suggestion. We will improve the presentation in the revised version.

3. [It seems to me that the critic population annotation is more important to the success of the method than crossover-based reward function generation. Is this correct?]

The goal of crossover-based reward function generation is to ensure the effective utilization of existing reward functions, which can facilitate the discovery of well-structured designs. Since parameter optimization does not modify the underlying structure, it is generally most efficient to apply parameter tuning only after a reasonable structure has been identified.

In many tasks, the initial parameter settings provided by the LLM are already fairly reasonable. In such cases, parameter alignment may have limited impact, whereas crossover can continuously explore and recombine high-quality existing reward functions to uncover improved designs. However, when the LLM provides suboptimal parameter settings, parameter optimization becomes crucial for constructing more effective reward guidance.

Based on our experimental results, parameter alignment is generally more efficient in most cases.

4. [What is the intuition behind the crossover-based design? I am unable to get a feel for why it leads to such a big improvement in performance, and why the LLMs cannot generate such functions in the first place.]

Eureka continuously improves the reward function by providing the best-performing individual to the LLM, while other reward functions are directly discarded. When the best individual is actually suboptimal, reflecting and improving based on it can lead the entire population to converge toward a suboptimal solution, ultimately resulting in low-quality outcomes.

In contrast, the crossover-based design aims to fully leverage existing superior reward functions without introducing any additional LLM overhead. By performing crossovers between different reward modules in superior reward functions, it enables thorough exploration and helps avoid being trapped in suboptimal solutions.

As shown in the results of Figure 4, the reward functions discovered by EA have a significantly higher probability of being the best in the population (the probability of best policies originating from crossover exceeds 50% in most tasks,with some tasks surpassing 80%), which further demonstrates EA's ability to effectively explore existing rewards and discover better reward functions.

---

We would appreciate it if the reviewer can confirm that the concerns had been addressed and, if so, reconsider the assessment. We’d be happy to engage in further discussions.