LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search

We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.

We will try our best to address each of the concerns and questions raised by the reviewer below:
1. [Re: How is the "elite" reward function/policy formally defined (e.g., success rate threshold, moving average performance)? Does the definition vary across tasks or remain consistent?)]

The criterion for defining an "elite" is the success rate. We evaluate each policy over 20 episodes to assess the quality of the corresponding reward function. This definition is applied consistently across all manipulation tasks.

2. [Re: Dependence on Human-Designed Rewards: While LaRes starts from human-designed rewards and retains them in the population, the paper lacks experiments exploring fully LLM-generated rewards from scratch (without human input). Comparisons to Eureka under such conditions—where reward functions are initialized without human guidance—would strengthen the claim of LLM-driven optimization. （Comparison with Eureka and Roska on their benchmarks is suggested. This would validate the necessity of human guidance.）]

We thank the reviewer for valuable suggestions. LaRes can also be applied in settings where no human-designed reward functions are available.

To address the reviewer’s concern, we evaluated LaRes on several tasks used in Text2Reward, where no expert rewards are provided. The results—measured in terms of success rate and sample cost—are shown below:

We observe that our method still significantly outperforms other methods.

3. [Re: Comparison with Eureka and Roska on their benchmarks is suggested. This would validate the necessity of human guidance.]

We thank the reviewer for the valuable suggestion. LaRes primarily emphasizes sample efficiency, which requires development based on off-policy RL. However, modifying existing algorithms within IsaacGym to support efficient off-policy learning is quite challenging.

To address the reviewer’s concern, we instead conducted experiments on OpenAI Humanoid and Ant tasks based on MUJOCO, which are commonly used in both Eureka and Roska. These experiments are performed without any human-designed rewards, and each is limited to 1 million environment steps. The results are shown below, we have currently completed experiments up to 500K steps, and we will update the results for 1M steps before the discussion deadline.

We observe that LaRes significantly outperforms both Eureka and Roska on the Ant and Humanoid tasks.

4. [Re: LLM Generation Dynamics: The process of LLM-generated reward function updates (e.g., how many new functions are created per iteration, which ones are replaced) is not clearly defined. For example, the paper states the population size is 5, but it is unclear whether LLMs generate all 5 functions or replace a subset.]

In our experiments, we set the population size to 5 and maintained 3 elites. If the RL policy guided by the human-design reward function is selected as an elite, we generate 3 new reward functions; otherwise, 2 new reward functions are generated. The individuals to be replaced are selected based on their success rate, in ascending order.

5. [Re: Notation and Clarity in Formulation: The description of the Thompson sampling mechanism (e.g., "i-th action" referring to policy selection) is ambiguous, as it could be misinterpreted as action-space elements rather than policy indices. Clarifying this notation earlier in the text would improve readability.]

We thank the reviewer for the valuable suggestion. We improve the presentation in the revised version.

6. [Re: Continual Learning and Plasticity Concerns: The paper mentions using the best policy’s parameters for initialization but does not address potential plasticity loss (e.g., catastrophic forgetting), a known issue in continual learning [1]. Compared to methods like RoSKA [2], which explicitly handle this, the framework could be enhanced in its theoretical robustness.]

We thank the reviewer for the valuable input. As shown in our previous responses, we provide comparative experiments; here, we would like to offer some clarification.

ROSKA is based on PPO and primarily emphasizes policy reuse, whereas LaRes is built upon off-policy RL and focuses on sample reuse from a finer-grained perspective. Moreover, the notion of continual learning in LaRes emphasizes maintaining learning stability and consistency when the reward function changes, thereby reducing performance fluctuations. This differs from conventional continual learning, which typically focuses on network plasticity.

We appreciate the reviewer’s suggestion and will include this discussion in the revised version.

7. [Re: Normalization vs. Scaling: Why use reward scaling with elite statistics instead of normalizing all rewards to the Normal distribution? Does the current approach introduce unnecessary complexity?]

In practice, the dataset is constructed using a learn-from-scratch approach. While using a normal distribution is a possible direction, it poses a challenge: during the first generation, the population must train for a long time to collect sufficient data to characterize the current reward distribution. This makes it difficult to directly model the initial reward functions using a normal distribution.

If we leave the first-generation reward functions unchanged and only map subsequent rewards to a normal distribution, this would lead to a distribution shift problem.

Therefore, we analyze the reward distribution of elite individuals from the previous generation, and re-map the outputs of newly generated reward functions to match this distribution.

8. [Re: Hyperparameter Sensitivity of Parameter Constraints: Has the weight of the parameter constraint loss (Equation 4) been ablated to assess its impact on stability and performance?]

We thank the reviewer for the helpful suggestion. LaRes does not tune this hyperparameter; it is set to the default value of 1.0 across all tasks. While further tuning may lead to improved performance, we find that 1.0 is generally sufficient to achieve strong results.

We hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.

We are pleased to have addressed several of the concerns raised. Below, we provide our replies to the remaining issues.

1. [Re: could you explain what each of these values represents? Why are some presented as percentages and others as integers?]

We sincerely apologize, we have just discovered a misalignment issue in the previously provided table. The correct results are as follows: the first value denotes the success rate, and the second value indicates the sample cost required to achieve that success rate. As shown, LaRes demonstrates superior sample efficiency.

2. [Re: Second, I’m quite surprised that your proposed method can achieve a 100% success rate without expert rewards. Assuming the experimental results are accurate, I would appreciate more details, such as the number of training iterations and a further analysis of this outcome.]

This results is expected, as these tasks are less challenging than those presented in the main body of our paper. Most of them are derived from the Text2Reward paper, where even a standard SAC algorithm can achieve a 100% success rate. In contrast, SAC performs poorly on the tasks proposed in our main experiments, often achieving very low or even zero success rates.

The experimental settings are as follows: a population size of 5, a sample cost of 200,000 per iteration, and five iterations in total. All methods are based on the same SAC backbone, use the same prompt design with the same set of provided variables, ensuring that performance differences are not attributed to other confounding factors.

3. [Re: while the current results are already promising, you mentioned that updated results with 1M will be provided, and I look forward to seeing those. I also wonder how you obtained the values for Eureka and Roska in the table.]

We sincerely appreciate the reviewers’ patience. Below, we present the results with 1M samples.

Both Eureka and Roska are reproduced within the same framework as ours to ensure a fair comparison. As Roska has not released its source code, our implementation is based primarily on the configurations and pseudocode provided in its original paper and appendix. All algorithms are implemented on top of SAC. Overall, Eureka fails to ensure sustained performance improvement, whereas both ROSKA and LaRes consistently achieve such improvement, with LaRes exhibiting superior sample efficiency.

4. [Re: For Q6, please directly address how your method enables the network to learn new rewards during the continuous evolution of the reward function while avoiding the issue of diminished network plasticity.]

We thank the reviewer for raising this question. We believe it is a valuable point. In our current work, we do not explicitly address the issue of plasticity. This is primarily based on two considerations:

• As noted in paper [1], "Loss of plasticity is most relevant for systems that use small or no replay buffers, as large buffers can hide the effect of new data." Both Eureka and Roska are built on PPO, where the impact of plasticity loss tends to be more significant. In contrast, LaRes maintains a large replay buffer, which alleviates the severity of plasticity challenges [2].
• When the reward function changes, LaRes relabels all experiences in replay buffer, which in turn changes the data distribution. Although we align reward distributions, data-level differences remain inevitable. Such changes in data can help mitigate plasticity degradation to some extent from a data-centric perspective. Some works have shown that modifying the replay buffer can help mitigate the loss of plasticity [3].

We fully agree that explicitly addressing plasticity, as Roska does, is valuable and meaningful. However, it is beyond the current scope of our work. LaRes primarily focuses on improving sample efficiency at the sample level and reducing instability during the learning process.

We will incorporate a discussion of plasticity-related works, particularly in connection with Roska, in the revised version. If the reviewer has any suggestions or recommendations, we would be grateful to receive them.

[1]. Loss of plasticity in deep continual learning

[2]. Experience Replay Addresses Loss of Plasticity in Continual Learning

[3]. The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning

We greatly appreciate your thoughtful and detailed comments. Below, we provide clarifications regarding several of the concerns.

1. [Re: I’m puzzled as to why the standard SAC was able to achieve a 100% success rate on this task without the provision of an expert reward. My original question was intended to inquire about how the authors’ method would perform without this reward function. However, the response indicated that even the standard SAC could achieve 100% success without it]

There seems to be a misunderstanding here. The claim that SAC can achieve a 100% success rate is based on the human-designed reward function, specifically the expert reward used in MetaWorld.

Below are the success rates and sample cost of SAC based on the human-designed reward in these tasks.

These tasks are relatively simple, so SAC with expert-designed rewards can already achieve strong performance.

We will further discuss the task difficulty levels in MetaWorld in our third response below.

2. [Re: The inputs utilized by different methods, and an analysis of the experimental results]

The results below do not rely on human-designed rewards.

All three methods, including LaRes, generate reward functions directly based on task descriptions, prompts, and available environment variables—without using any human-designed rewards. All three methods are based on exactly the same input for fair comparison.

LLM zero-shot refers to generating the reward function using the LLM only once, without any further tuning or feedback. Based on the results, we draw the following three conclusions:

• Since the tasks belong to the simpler category, LLM zero-shot can typically achieve a certain level of success.
• An evolutionary mechanism is essential—Eureka generally outperforms LLM zero-shot.
• LaRes places greater emphasis on sample-level reuse, making it more sample-efficient than other methods—an aspect of primary concern in the DRL community.
• LaRes w/o expert-designed rewards achieves even greater sample efficiency than SAC w/ expert-designed rewards in some tasks.

3. [Re: Could the authors explain why their implementation of SAC performed poorly? If SAC exhibits such significant reproducibility issues, I am concerned whether the authors’ method might also be susceptible to similar problems.]

Our SAC implementation directly adopts the open-source code provided in EvoRainbow [1], specifically the MetaWorld SAC implementation. The experimental results we obtained are consistent with those reported in the original paper, ensuring reproducibility.

The SAC implementation used in our work demonstrates superior performance. A comparison between our SAC results and those reported in Text2Reward reveals that our SAC version consistently achieves better performance.

As for the tasks presented in the main text, the relatively poor performance of SAC with expert-designed rewards is primarily due to the high difficulty of these tasks.

In MetaWorld, tasks can generally be categorized into three levels: easy, medium, and hard.

• The tasks mentioned above—where reward functions are constructed directly without human-provided rewards—mostly fall into the easy category. In such tasks, SAC can achieve a 100% success rate using a human-designed reward function with very limited sample cost. Many reward generation works, such as Text2Reward[2], evaluate their methods under this category using zero-shot reward generation.
• The medium tasks represent a more challenging group, where SAC with expert-designed rewards typically achieves less than 50% success rate after 1 million steps.
• The hard tasks are the most difficult, where the success rate of SAC after 1 million steps is usually below 10%.

All the tasks included in the main text of our paper are selected from the medium and hard categories.

Thank you for your detailed response, which has clarified my doubts. Based on our discussion and the original manuscript, I believe this work demonstrates a certain degree of innovation, and the experimental results and conclusions hold some reference value. However, it appears that due to time constraints, the authors have not conducted supplementary experiments on tasks of medium or hard difficulty. Therefore, these experimental conclusions should be accompanied by rigorous descriptions of their prerequisite conditions. On the premise that the reviewers commit to incorporating the content discussed into the revised version, I will revise my rating upwards.

We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.

We will try our best to address each of the concerns and questions raised by the reviewer below:
1.[Re: The relation between high-level reward evolution and low-level policy learning is not clear. Furthermore, the relation between the framework and techniques like continuous learning, Thompson sampling, and reward scaling should be clarified.]

LaRes is primarily focused on sample-efficient task solving. To address this challenge, we decouple the problem into two levels: high-level reward function optimization and low-level policy learning. At the upper level, an LLM is used to construct a population of reward functions, which guide the learning of policies at the lower level. To further improve sample efficiency, we adopt a sample-level sharing strategy, where trajectories collected by different RL agents are shared and relabeled based on their respective reward functions.

The introduction of Thompson Sampling also aims to enhance sample efficiency by allowing stronger policies within the RL population to interact more frequently with the environment, thereby reducing waste from weaker policies.

It is important to note that our notion of continual learning differs from conventional definitions, which typically focus on network plasticity. In contrast, LaRes is designed to maintain policy stability in the face of evolving reward functions, ensuring that performance does not degrade or collapse when the reward function changes.

To achieve this, we introduce reward scaling and a consistency loss. The former ensures distributional alignment of rewards before and after changes, while the latter constrains policy updates to prevent performance drops.

2. [Re: The authors claimed that LaRes can cooperate with other RL methods, which is not demonstrated in the evaluation. ]

LaRes is compatible with any off-policy RL algorithm. To address the reviewer’s concern, we integrated LaRes with DQN and evaluated it on MinAtar, a set of discrete control tasks based on image inputs. The experimental results are shown below:

We observe that LaRes consistently leads to performance improvements.

3. [Re: the current design requires multiple Reward functions to fine-tune the model, some experiments on computation cost should be involved.]

LaRes requires optimizing multiple RL policies in parallel, each guided by a different reward function. Executing this process serially would indeed incur significant computational overhead. To address this, we implemented a distributed framework using a server–client architecture: sampling is performed on the server side, while each client independently optimizes its own policy. Overall, due to scheduling and communication overhead, LaRes incurs approximately 20% additional runtime.

Regarding computational resources, all experiments on MetaWorld are conducted using CPU only. Each run consumes (pop size + 1) CPU cores—one for the server process and the remaining for individual worker processes corresponding to each policy in the population.

4. [Re: The main comparison in evaluation adopts SAC, which was proposed in 2018; more comparisons with SOTA methods are recommended. Furthermore, the baselines in ERL-ralated are out-of-date except ERL-Re and EvoRainbo. ]

We thank the reviewer for the insightful comment. LaRes is inherently flexible and can be built upon different underlying algorithms. In general, the stronger the backbone algorithm, the better the overall performance of LaRes.

To address the reviewer’s concern, we conducted a comparison using two advanced RL algorithms: TD7 and CrossQ. The results are shown below.

We observe that although LaRes uses the relatively weaker SAC as its backbone, it still outperforms the advanced RL algorithms TD7 and CrossQ. This highlights the critical role of reward function optimization in achieving strong performance.

5. [Re: In Fig2,3,4, several results of LaRes exhibit larger variance than baseline; more discussion on it is required. Why LaRes exhibits larger variance compared to the baseline]

We thank the reviewer for the helpful suggestion. The performance of LaRes is indeed influenced by the inherent randomness of the LLM. In some runs, suboptimal reward functions are generated, leading to reduced policy learning efficiency and increased variance in certain tasks. We believe this issue can be mitigated by either increasing the number of reward function optimization iterations or enlarging the reward function population, both of which allow for a more thorough search and make the framework more robust to the stochastic nature of the LLM.

6. [Re: No appendix, and the code is not open-sourced.]

We provide a detailed appendix, including Limitations and future work (Appendix A), Implementation details (Appendix B), Reward function examples (Appendix C)

We commit to releasing the code as soon as the paper is made public, which we consider an important contribution of this work.

7. [Re: Further experiments on Locomotion tasks]

To further demonstrate the effectiveness of LaRes, we evaluate it on the two most challenging tasks in OpenAI MuJoCo: Ant and Humanoid. Notably, this version does not incorporate any human-designed reward function.

We have currently completed experiments up to 500K steps, and we will update the results for 1M steps before the discussion deadline.

We observe that LaRes also demonstrates superior performance on these tasks, further highlighting its effectiveness and generality.

We hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.

Dear Reviewer XhCm,

The author-reviewer discussion period will end soon on Aug. 8. Please read the authors' rebuttal and engage actively in discussion with the authors.

AC

We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.

We will try our best to address each of the concerns and questions raised by the reviewer below:
1. [Re: The paper acknowledges that LLM-generated functions may lack diversity, which could limit the exploration potential of the evolutionary process]

Yes, we promote diversity by constructing an LLM-based population and employing a relatively high temperature during generation. While we acknowledge that more effective mechanisms for ensuring diversity may exist, this is not the primary focus of LaRes at the current stage.

2. [Re: While the authors also explore scratch generation, the method generally relies on human-designed reward functions as a base, which may limit applicability in settings where expert knowledge is unavailable.]

We thank the reviewer for the helpful suggestion. We conducted experiments on the following two domains where no human-designed reward function is required:

• 5 Tasks from Metaworld (win rate & sample cost):

• Openai Humanoid & Ant tasks: We have currently completed experiments up to 500K steps, and we will update the results for 1M steps before the discussion deadline.

We observe that LaRes still demonstrates a significant advantage.

3. [Re: Could the authors provide any theoretical insights into convergence properties or bounds for LaRes? How can we trust the stability of LLM-driven reward evolution in the long run?]

The results generated by LaRes are often superior to those produced by standard RL algorithms. This is primarily because LaRes follows the ERL framework and always maintains the expert reward function within the reward population. When LLM-generated reward functions perform poorly, the policy guided by the original reward function dominates the process, ensuring a performance lower bound. In addition, LaRes constructs an RL policy population, which further augments sample diversity and enhances overall performance.

Our work primarily focuses on empirical validation of the proposed method's effectiveness, supported by extensive experiments across 12 MetaWorld tasks, 4 ManiSkill tasks, 5 human-reward-defined tasks, as well as Humanoid and Ant tasks. Regarding theoretical support, we acknowledge in the Limitations section that the current work lacks formal theoretical guarantees, which we consider an important direction for future research.

4. [Re: How does LaRes scale with more complex or high-dimensional tasks beyond the 16 robotic benchmarks tested?]

To address the reviewer’s concern, we primarily extended our experiments in two directions:

• MinAtar, a benchmark based on image inputs.
• Ant & Humanoid, two of the most challenging control tasks from OpenAI's suite.

The results on MinAtar:

The results on Humanoid and Ant without human-designed reward functions, we have currently completed experiments up to 500K steps, and we will update the results for 1M steps before the discussion deadline.

We observe that LaRes also demonstrates superior performance on these tasks, further highlighting its effectiveness and generality.

We hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.

Dear Reviewer 4fMG,

The author-reviewer discussion period will end soon on Aug. 8. Please read the authors' rebuttal and engage actively in discussion with the authors.

AC

Dear Reviewer 4fMG

Thank you very much for your valuable time and constructive feedback.

As there is only one day remaining before the author–reviewer discussion closes, we would like to kindly confirm whether our responses have addressed your concerns, or if you have any further suggestions or remaining questions. Your additional comments will be greatly appreciated and will help us ensure the manuscript meets the highest possible standard.

We sincerely appreciate your input and look forward to your reply.

Best regards,

Authors Paper 27186

We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.

We will try our best to address each of the concerns and questions raised by the reviewer below:

1. [Re: It would be insightful if the authors can discuss the relationships between the LLM's generative priors and performance on such tasks. Specifically, why might an LLM-guided search still fail on a hard-exploration task even if it can generate dense rewards? Does the LLM's common sense reasoning create a bias that prevents it from discovering the unconventional reward components needed to solve these problems?]

First, task decomposition is essential for a given task, as it can significantly reduce the difficulty of reward generation. Directly using LLMs to generate long and complex reward functions is often impractical.

We fully agree with the reviewer’s observation that even with dense rewards, learning a successful policy may still fail. This is mainly due to a potential misalignment problem—dense rewards are not necessarily positively correlated with task success, leading to reward hacking. This issue arises from multiple factors. On one hand, individual reward components may be poorly designed—for example, critical reward terms may be underweighted and dominated by less important ones, or inappropriate activation functions may be used.

Moreover, reward design can be viewed as a black-box optimization problem, where some reward components that seem intuitively useful may not be necessary for successful policy learning. This highlights the need for RL–driven evaluation of reward functions. It also implies that when a reward function fails to guide learning effectively, the capability of the RL algorithm itself is a non-negligible factor. For instance, the performance of SAC and AC can differ significantly under the same reward design.

Therefore, we argue that reward functions generated by LLMs, like those manually designed by human experts, are inevitably prone to being suboptimal or even ineffective. Thus, iterative refinement and reflective optimization are often necessary to achieve reliable and aligned reward signals.

2. [Re: When the reward function population is updated, all experiences in the shared replay pool need to be relabeled. How much computational overhead is this process for a replay pool of samples of different sizes?.]

The computational overhead of this process is minimal. We directly relabel the replay buffer through code, without involving any gradient operations, resulting in very low complexity.

We measure the relabeling time and find that relabeling 200,000 samples takes only 5.89 seconds, and the cost scales linearly with the number of samples.

3.[Re: LaRes contains a series of hyperparameters of the genetic algorithm, such as the population size of the reward function, the number of evolutions, the elite size, etc. ]

We thank the reviewer for the valuable feedback. Regarding the population size, we followed the common settings used in prior ERL works, setting it to 5, with 5 evolution number( same with Eureka). The elite size is set to 3.

To address the reviewer’s concern, we conducted the following additional hyperparameter experiments:

We observe that a population size of 5 generally yields the best performance. A larger population may introduce potential out-of-distribution issues, while a smaller population can limit the algorithm’s exploration capacity.

For the evolution number, we define it in the number of environment interaction steps between two evolutionary updates.

We observe that a frequency of 200k generally performs well. A smaller evolution frequency may lead to insufficient training of the lower-level policy, while a larger frequency can result in under-exploration of the reward function search space.

For the elite size, we found that 3 generally yields the best results. An elite size that is too small tends to increase the risk of falling into suboptimal solutions.

4. [Re: I would strongly recommend that the authors conduct additional experiments with alternative LLM to demonstrate the robustness and generality of their approach.]

We thank the reviewer for the valuable suggestion. We conducted evaluations across a diverse set of benchmarks, including Qwen-plus, GPT-4o-mini, and DeepSeek-V3. The experimental results are shown below.

We observe that although different models exhibit some variation in performance across tasks, LaRes consistently achieves a high success rate regardless of the underlying LLM framework.

5. [Re: In the framework, the elite reward functions are not replaced, and the RL agent guided by the reward function designed by humans does not participate in Thompson sampling to ensure the stability of learning. However, does this design also bring the risk of prematurely converging to local optima? If these strategies are not globally optimal, will the mechanism continuously guide the search process, thereby hindering the exploration of new and potentially better reward paradigms? Have you considered other mechanisms to balance stability and the need for long-term exploration?]

We agree with the reviewer's perspective that exploration versus exploitation is a fundamental challenge in any learning or optimization problem. We did consider using UCB to address this issue more effectively; however, UCB introduces several hyperparameters that require careful tuning. While better performance might be achievable through fine-tuning, we chose to adopt Thompson Sampling for the sake of simplicity.

As for long-term exploration, one viable approach is to increase the number of environment interaction steps per generation for each population member. However, this presents a trade-off: too many steps may lead to sample inefficiency, while too few may result in suboptimal performance.

Finally, regarding the decision to exclude the RL reward function from evolutionary optimization, this follows prior ERL settings. This design ensures that when the population fails to discover better solutions, the RL reward function can still provide a performance lower bound.

6. [Re: what is the nature of this evolutionary trajectory? Do the reward functions converge towards a specific structure? Do they tend to become more complex over generations, or do they become simpler as the LLM distills the problem's essence?]

In practice, we observe that the structure of the reward function typically remains relatively stable from the early to later stages of training. For example, in a pick-and-place task, the reward function usually consists of components such as approach reward, grasp reward, and reward for moving toward the target. The main changes over time tend to lie in the tuning of hyperparameters—both within and between reward components—as well as in the implementation details of each module, such as different transformation functions. Overall, the reward function does not become increasingly complex; instead, once the general structure is established, subsequent efforts focus on refining and optimizing specific details.

7. [Re: Have you considered adding explicit instructions in the Prompt to guide the LLM to generate more diverse reward functions? Will this further enhance the performance of LaRes? ]

We thank the reviewer for the valuable suggestion. In our implementation, we set a relatively high temperature for the LLM (i.e., 1.0) to promote population diversity.

To address the reviewer’s concern, we experiment with designing diversity-enhancing prompts. Specifically, in each generation step, we feed the previously generated result back into the LLM as an additional input, encouraging the model to produce more diverse outputs. The results are shown below.

We observe that this approach leads to a decline in performance. Upon closer inspection of the generated reward functions, we identify a key issue: while the method increases diversity, it often results in reward functions that omit essential modules. For example, after the first reward function is generated with a human-aligned structure, subsequent ones tend to introduce entirely different reward components for the sake of diversity.

As previously discussed, good reward functions tend to share a common structure, differing mainly in their finer details. We therefore believe that enhancing diversity requires more principled mechanism design. Simply relying on prompt-based guidance is insufficient to ensure both high diversity and strong performance.

We hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.

Dear Reviewer bxvD,

The author-reviewer discussion period will end soon on Aug. 8. Please read the authors' rebuttal and engage actively in discussion with the authors.

AC

Dear Reviewers,

Please note “Mandatory Acknowledgement” button is to be submitted only when you fulfill all conditions below (conditions in the acknowledgment form):

• read the author rebuttal
• engage in discussions (reviewers must talk to authors, and optionally to other reviewers and AC - ask questions, listen to answers, and respond to authors)
• fill in "Final Justification" text box and update “Rating” accordingly (this can be done upon convergence - reviewer must communicate with authors first)

AC