Response to Reviewer qnw7

We sincerely thank Reviewer qnw7 for the detailed and comprehensive review. We particularly appreciate the acknowledgment that framing reward design as MCTS over LLM decisions is novel and for recognizing our strong empirical results. In response to the mentioned weaknesses and questions, we have organized our detailed replies into the following sections:

---

Question 1: OOD tasks performance

The reviewer's concern regarding generalization to novel tasks is indeed valid. To address this directly, we design and conduct two entirely new experiments using the Isaac Gym Ant environment:

1. 'Ant Lie Down': This task tests for generalization to a novel objective. The instruction provided to the LLM is:

   "To make the ant lie down on its back (belly-up) as quickly and stably as possible. Its four feet should point towards the sky."

2. 'Ant Patrol and Return': This is a compositional task designed to test the LLM's ability to generate reward functions for unforeseen, multi-stage problems. The instruction is:

   "To make the ant first run forward to the checkpoint position, then turn around and return to its starting origin position."

The evaluation score for each task is based on a success condition, defined by an indicator function 1[·].

• For 'Ant Lie Down', success is 1[orientation_reward > 0.9], corresponding to a stable belly-up pose.
• For 'Ant Patrol and Return', success is 1[task_phase == 1 and dist_to_start < 0.5], signifying the completion of the entire two-stage trip.

Using the same experimental configuration as in our main Isaac Gym experiments and employing the GPT-4o-mini model, we obtained the following results:

In these experiments, the 'Sparse' condition refers to using only the evaluation score as the reward signal. While it is not feasible to provide a handcrafted expert baseline for such novel tasks, the experimental results sufficiently demonstrate two key points:

1. The paradigm of LLM-based reward generation is capable of generalizing to these specialized, novel tasks.
2. Our RF-Agent continues to outperform the existing baseline methods on these new challenges.

We will add this section to the manuscript and thank you again for this insightful question.

Question 2: About Feedback Insight and Ablation

We thank the reviewer for this insightful question, which highlights a critical and valid concern about the potential fragility of an LLM's numerical reasoning capabilities.

We completely agree that LLMs are not inherently designed as numerical signal processors. As detailed in Appendix B.2 (line 567), this feedback is composed of three elements: nums, correspongding statistics, and analysis tip. We conducted the following ablation studies to verify the feedback effectiveness:

• w/o full feedback: The baseline where l_feedback is entirely removed, and the search relies solely on the final success score, F.
• w/o nums: Removes the numerical progression from the feedback.
• w/o statistics: Removes the pre-computed statistical summary.
• w/o analysis tip: Removes the analytical heuristic/tip.

The results are as follows:

Removing l_feedback entirely (w/o full feedback) causes a significant performance drop. More importantly, the ablations show that removing either the pre-computed statistics (w/o statistics) or the analytical tip (w/o analysis tip) individually degrades performance.

We speculate that analysis tip helps LLM to a certain extent how to analyze sequences and further exert effects in subsequent diver actions. More refined optimizations can still be made regarding feedback, which can be discussed in depth in subsequent research, thank you again for your insights.

Question 3: Computational Cost

We tallied the input_token (prompt), output_token (completion), and the actual monetary cost for RF-Agent, Eureka, and Revolve on the Ant and Humanoid tasks. It is important to note that input_tokens can be processed in a single, efficient forward pass (a 'prefill'), whereas output_tokens are generated autoregressively (a 'decode' process), which is more computationally intensive. This efficiency difference is reflected in API pricing (e.g., Input: $0.6 per 1M tokens vs. Output: $2.4 per 1M tokens).

The data in the tables leads to the following conclusions:

1. RF-Agent utilizes 2-3 times more prefill tokens than the baselines, while the number of decode tokens is only moderately higher (approx. 1.2-1.4 times). Given that the prefill stage is computationally cheaper than the decode stage, the overall increase in resource consumption, as reflected by the total cost, is moderate and remains within an acceptable range.

2. Furthermore, we find that this asymmetric token consumption aligns perfectly with the design philosophy of RF-Agent. The higher usage of prefill tokens is a direct result of leveraging historical information (i.e., previously generated thoughts and reward functions). As the MCTS tree grows, these artifacts are flexibly combined and included in the prompt, forming a multi-step reasoning process for the LLM. This process enables the LLM to generate more sophisticated and effective reward functions, which in turn leads to the significant performance growth observed in our experiments.

Other Questions

Re l. 50-52: On one hand, our ablation studies demonstrate the importance of balancing exploration and exploitation. On the other hand, a key reference is "The 2014 General Video Game Playing Competition," a benchmark for complex decision-making games. In this article, methods based on an exploration-exploitation balance, such as MCTS, consistently achieved top ranks (e.g., 1st, 2nd, 3rd), while approaches like genetic and greedy algorithms ranked significantly lower (e.g., 8th, 9th, 12th).

Re l. 112: Yes. To maintain consistency with prior work, we assume the reward R is a function of the current state and action (S x A).

Re l. 113: Thank you for the suggestion. We will change the notation for the action space to A for better readability.

Re l. 114 (π): We will clarify in the text that π is a policy sampled from the policy space Π.

Re l. 114 (F): Yes. F is a general evaluation score. It can represent the total sum of rewards or, for long-horizon tasks, a final binary signal (0/1) indicating task success.

Re l. 115: Due to space constraints, we did not detail the full MDP definition in the main text. The discount factor γ is indeed part of the standard MDP formulation. We will add a detailed explanation to the appendix in the revised version.

Re l. 117-118: Your understanding of the formulation is correct.

Re l. 120: Thank you. We will add a definition clarifying that ℒ is the space of possible language sequences.

Re l. 121: Thank you. We chose to retain the parameter θ to explicitly represent that different LLMs could be used in our framework, but we agree that it can be simplified for clarity.

Re l. 124-125: l_feedback consists of a predefined text template combined with numerical values returned during the training process, as detailed in Appendix B.2 (line 567). l_task is determined by the benchmark itself, such as the brief text descriptions provided for each task in the Isaac Gym repository.

Re l. 125-126: This is a very thoughtful point. In the revised manuscript, we will more formally define the high-level decision-making problem to ensure it strictly aligns with our proposed solution process.

Re Eq. (1): This notation describes a single generation process where the LLM's response contains both the thought z and the reward function R. The model first decodes z, and then subsequently decodes R.

Re l. 205 and l. 218: These terms refer to sampling nodes from the elite set and from the current MCTS tree, respectively. The information from these sampled nodes is then used for the next generation step.

Re Appendix I: Thank you for this suggestion. For readability, we initially provided a one-sentence summary comparing the reward functions. We acknowledge that this may lack sufficient detail. In the revised manuscript, we will highlight the specific differences in the code using color.

Response to Reviewer veGp

We sincerely thank Reviewer veGp for the clear and detailed review. We particularly appreciate the positive feedback on our work, especially regarding the innovative framework combining MCTS and LLMs and the clear figures and thorough experimental comparisons. In response to the mentioned weaknesses and questions, we have organized our detailed replies into the following sections:

---

Question 1: Token Consumption Comparison

We tallied the input_token (prompt), output_token (completion), and the actual monetary cost for RF-Agent, Eureka, and Revolve on the Ant and Humanoid tasks. It is important to note that input_tokens can be processed in a single, efficient forward pass (a 'prefill'), whereas output_tokens are generated autoregressively (a 'decode' process), which is more computationally intensive. This efficiency difference is reflected in API pricing (e.g., Input: $0.6 per 1M tokens vs. Output: $2.4 per 1M tokens).

The data in the tables leads to the following conclusions:

1. RF-Agent utilizes 2-3 times more prefill tokens than the baselines, while the number of decode tokens is only moderately higher (approx. 1.2-1.4 times). Given that the prefill stage is computationally cheaper than the decode stage, the overall increase in resource consumption, as reflected by the total cost, is moderate and remains within an acceptable range.

2. Furthermore, we find that this asymmetric token consumption aligns perfectly with the design philosophy of RF-Agent. The higher usage of prefill tokens is a direct result of leveraging historical information (i.e., previously generated thoughts and reward functions). As the MCTS tree grows, these artifacts are flexibly combined and included in the prompt, forming a multi-step reasoning process for the LLM. This process enables the LLM to generate more sophisticated and effective reward functions, which in turn leads to the significant performance growth observed in our experiments.

Question 2: Comparison with Roska and Progress Function(LLMCount)

We thank the reviewer for suggesting these relevant works. We will cite and discuss the differences between these methods and our own in the revised manuscript.

Due to the time constraints and the lack of publicly method code for reproduction, our comparison is presented in a table based on the differences and the results reported in their respective papers. For ROSKA, we noted that the number of training steps used for the Isaac Gym tasks differed from our setup. We retrained the reward function under the same steps for comparison.

Based on the reviewer's valuable suggestions and our further analysis, we believe the following points are promising directions for future exploration:

1. LLMCount focuses on achieving competitive results non-iteratively through reward transformation. This makes it highly efficient under very limited sample budgets, but it relies on the construction of external knowledge. For iterative optimization processes like ours, we believe the approach from LLMCount could serve as an excellent initialization method.

2. Regarding the use of historical information, ROSKA implicitly fuses it at the policy level to assist its evolutionary process. In contrast, our method explicitly leverages history at the LLM level, using a rich set of actions and tree search to construct new reward functions. Each approach has its own advantages and is driven by different motivations. Our work, in particular, has been validated with high performance in more complex manipulation environments. Future research could potentially combine these two distinct optimization paradigms.

Q3 & Q4: About Detailed Ablation

We thank the reviewer for the valuable suggestion to conduct a more detailed ablation study. Given the rebuttal time constraints , we include several key ablations as follows.

Part1: More Ablation on Feedback

Due to the rebuttal character limits, please check our reply to R2 (zhzV), located in part2 of Q2 for detailed ablation part of the feedback.

Part2: More Ablation on Action Combination

We further conducted a combinatorial study of action, which supplemented the ablation in the paper(line 324):

By analyzing the ablation results, we can categorize our actions from the perspective of information utilization. The mutation action can be seen as a local operator, as it modifies a single parent node. In contrast, crossover and reasoning act as global operators, as they synthesize information from multiple nodes or entire ancestral paths.

The results clearly indicate that relying exclusively on one type of operator leads to a significant performance drop. For instance, using only a local action (w/ mutation) or only global actions (w/ crossover + reasoning) both result in considerable performance degradation. Therefore, a balanced combination of both local and global operators is crucial for achieving the best performance.

Part3: More Ablation on Thought-Align

Finally, we present an ablation study on the thought align mechanism. In our current design, this process consists of two steps:

1. First Design Thought: Following a Chain-of-Thought (CoT) like process, the LLM first generates a high-level 'design thought' before composing the actual reward function code.
2. Second Thought Align: To mitigate potential hallucinations from a single generation step, after the reward function is generated, we prompt the LLM to re-examine the code's content and then revise the original 'design thought' to ensure it accurately reflects the code's logic.

To quantitatively validate the effectiveness of this mechanism, we conducted further ablations on these intermediate components:

The results indicate that the design thought mechanism is crucial across all tasks. Notably, the second thought align step has a particularly significant impact on the more complex manipulation task (Allegro Hand).

Regarding a direct percentage validation of this alignment's impact, it is challenging to isolate. This is because the second step corrects the thought, not the reward function code itself. The benefit of this corrected thought is realized later, when it is used to generate subsequent reward functions for child nodes. However, we are confident that this alignment process positively influences the quality of future rewards generated during the RF-Agent tree search, as empirically demonstrated by the overall performance gains shown in the table above.

Response to Reviewer zhzV

We sincerely thank Reviewer zhzV for the insightful and detailed review. We particularly appreciate the acknowledgment that RF-Agent, through rigorous experiments, provides promising support for solving this problem from a sequential decision-making perspective. In response to the mentioned weaknesses and questions, we organize our detailed replies into the following sections:

---

Question 1: OOD tasks performance

The reviewer's concern regarding generalization to novel tasks is indeed valid. To address this directly, we design and conduct two entirely new experiments using the Isaac Gym Ant environment:

1. 'Ant Lie Down': This task tests for generalization to a novel objective. The instruction provided to the LLM is:

   "To make the ant lie down on its back (belly-up) as quickly and stably as possible. Its four feet should point towards the sky."

2. 'Ant Patrol and Return': This is a compositional task designed to test the LLM's ability to generate reward functions for unforeseen, multi-stage problems. The instruction is:

   "To make the ant first run forward to the checkpoint position, then turn around and return to its starting origin position."

The evaluation score for each task is based on a success condition, defined by an indicator function 1[·].

• For 'Ant Lie Down', success is 1[orientation_reward > 0.9], corresponding to a stable belly-up pose.
• For 'Ant Patrol and Return', success is 1[task_phase == 1 and dist_to_start < 0.5], signifying the completion of the entire two-stage trip.

Using the same experimental configuration as in our main Isaac Gym experiments and employing the GPT-4o-mini model, we obtained the following results:

In these experiments, the 'Sparse' condition refers to using only the evaluation score (i.e., the 1[...] indicator function) as the reward signal. While it is not feasible to provide a handcrafted expert baseline for such novel tasks, the experimental results sufficiently demonstrate two key points:

1. The paradigm of LLM-based reward generation is capable of generalizing to these specialized, novel tasks.
2. Our RF-Agent continues to outperform the existing baseline methods on these new challenges.

We will add this section to the manuscript and thank you again for this insightful question.

Question 2: About Feedback Insight and Ablation

Part1：Illustration on Feedback

Regarding l_feedback, in current design, it serves as auxiliary information that summarizes the dynamics of the variables within a given reward function. As detailed in Appendix B.2 (line 567), this feedback is composed of three parts:

1. The change trend of each reward component over the training epochs.
2. Their corresponding statistics (e.g., max, mean, min).
3. Potential analytical suggestions.

This design is intended to provide a moderate, not excessive, amount of information. In our framework, the ideal optimization signal is the final task success rate, F. The l_feedback is treated as a secondary, auxiliary factor to help analyze the reward's behavior. This choice is motivated by practical deployment considerations, where a full state feedback might not be available. Consequently, we deliberately limit the auxiliary information to only include the dynamics of the reward components themselves.

To give a concrete example of how l_feedback influences the search process: if a reward component exhibits very little variance during training, l_feedback suggests that this component might be ineffective. This information then guides the node's action expansion. For instance, during a crossover operation between two high-performing reward functions (both with high F), the l_feedback allows the agent to identify and discard ineffective components from both parents. By combining only the proven, meaningful components, the crossover is more likely to produce a superior child reward function.

Therefore, our current framework places a greater emphasis on the design of the search actions . We believe that well-designed actions are the primary mechanism for intelligently utilizing the insights from l_feedback.

Part2: Ablation on Feedback

Furthermore, the current design of l_feedback component has been experimentally validated to demonstrate its effectiveness. We conducted a set of ablation studies on its constituent parts:

• w/o full feedback: A baseline condition where l_feedback is removed entirely. The MCTS optimization, as well as the information provided to the search actions, relies solely on the final evaluation score, F.
• w/o nums: In this ablation, we remove the part of the feedback corresponding to the numerical progression of the reward components over the training epochs.
• w/o statistics: This condition removes the corresponding statistical summary (e.g., max, mean, min) of the reward components.
• w/o analysis tip: This condition removes the analytical "tip" which is intended to help the LLM interpret the feedback.

We use the same ablation setting as in the original text and conducted the experiment, and the results are as follows:

The ablation study demonstrates that removing the l_feedback component entirely causes a significant performance degradation. However, it is noteworthy that even without this detailed feedback, the combination of MCTS and LLM is robust enough to achieve reasonable performance by optimizing solely on the final evaluation score.

Furthermore, the results indicate that each of the three auxiliary information components within our feedback design contributes positively to the final performance.

We will include a more in-depth discussion of these findings in our revised manuscript.

Question 3: About Reward Design Problem Motivation

We thank the reviewer for this insightful question, as it allows us to further elaborate on the motivation behind our reward design problem setting. This approach originates from the principles of reward engineering. The evaluation score, F, often serves as a sparse and monolithic signal. Using F directly as the reward R for policy learning is often inefficient and makes the learning process difficult, especially for complex manipulation and compositional tasks.

This challenge is particularly evident in our experimental results.

• As shown by the Sparse results for Bi-DexHands in Figure 3(after line 296), when a task only provides a binary 0/1 success signal at the end of an episode, a policy trained directly on this sparse reward can barely learn within the given step limit.
• Similarly, as shown in Table 1(after line 248), even for tasks like Ant and Humanoid that provide a dense metric F at every step, using F directly as the reward is still significantly less sample-efficient compared to using a shaped reward function.

Therefore, the significance of our problem setting is to explore how to better construct dense reward signals to efficiently guide policy learning. From this perspective, a reward function is an interpretable, easily deployable, and modifiable way to provide this guidance. Furthermore, our method investigates how to leverage LLMs to generate these better reward functions. This approach enables automated optimization while reducing the laborious manual effort required from human experts in reward engineering.

Question 4: About Action Design Discussion

The design of our current action framework is intentionally high-level and conceptual. It does not operate on concrete environmental assets or objects (e.g., it does not propose adding a specific 'velocity penalty'). Instead, our action set can be applied to any reward function that exhibits fundamental characteristics, such as using a selection of environment observations, containing tunable hyperparameters, and being composed of multiple components. This makes our framework inherently generalizable.

In designing this framework, we integrated several key ideas: common heuristic operations (like mutation and crossover), an action that simulates human-like step-by-step thinking (reasoning), and an action that emphasizes exploration (different thought). Together, these form a comprehensive set of high-level, conceptual actions designed to effectively navigate the complex space of reward functions.

Response to Reviewer 87TL

We sincerely thank Reviewer 87TL for the thorough and constructive review. We particularly appreciate the recognition that the motivation for introducing MCTS is natural and novel and that the experimental section sufficiently demonstrates the effectiveness of our approach. In response to your mentioned weaknesses and questions, we consolidate our detailed replies into the following sections:

---

Question 1: About MCTS Iterations

We thank the reviewer for acknowledging the novelty of our MCTS-based approach to reward function optimization.

To ensure a fair comparison, as stated on line 278, all methods were allocated an identical number of total samples per experiment. Each sample constitutes a full cycle of LLM-based reward generation and a subsequent RL training run. Consequently, since the number of samples is identical for all methods in each experiment, the upper bound on total training time is also consistent across all approaches.

To further analyze efficiency, we can compare the training cost (i.e., number of samples) required to achieve stable performance on the more challenging Bi-DexHands tasks. As shown in Figure 5 (after line 318), which plots success rate versus the number of samples, we can establish performance thresholds. Defining "stable success" at 0.6 and 0.8 success rates, the number of samples required by each method to reach these levels is as follows:

In this table, Easy/Hard denote the BiDex-Expert-Easy/Hard groups. ~512 means a baseline was approaching the target success rate by the 512-sample limit, whereas 512+ means it failed to do so.

The results clearly show that our method is more sample-efficient. On the Easy tasks, for example, it reaches a 0.6 success rate nearly five times faster than the baselines, which fail to reach the 0.8 success rate within the entire budget. Regarding the precise wall-clock duration, we refer the reviewer to Appendix C (line 620). While the full training budget for each Bi-DexHands task is approximately 40 hours, our method consistently converges to a high success rate in under 20 hours.

We hope these details resolve any concerns regarding training efficiency and time, and we conclude that RF-Agent is a more efficient method that achieves superior performance in less total training time.

Question 2: About Aciton Expansion

Your understanding is comprehensive. The transition from s_1^0 to s_2^0 and s_2^1 is also an action expansion. The reason it is not marked with an "orange" arrow in Figure 2(after line 126) is to differentiate the stages of the search process.

The non-orange arrows indicate expansions that were previously completed and whose resulting nodes (like s_2^0 and s_2^1) are already stored in the tree. The orange arrow, however, is used to specifically denote the expansion (to node s_5^1) that is currently active. In this way, the figure illustrates a dynamic process where arrow colors distinguish past expansions from the present one.

Question 3: About Selection Stage and Depth

Your description about the selection phase is basically accurate. We would like to add an implementation detail for clarification. For ease of deployment, we perform action expansion in parallel from the selected leaf node.

For instance, when the selection phase reaches a leaf node s_1^0 (i.e., a node with no children), we simultaneously expand n new child nodes (s_2^0, s_2^1, ...), each corresponding to a different action. Once this batch of new nodes completes their simulations, their values are backpropagated up the tree (e.g., s_2^0 -> s_1^0 -> root and s_2^1 -> s_1^0 -> root), just as the reviewer described. The backpropagations are processed sequentially in the order that the simulations finish.

In the subsequent selection phase, the search might proceed to a different branch (e.g., to sibling node s_1^1) if its UCT score is higher. Alternatively, if the backpropagated values have made the s_1^0 branch more promising, the search may continue deeper into one of its newly expanded children (s_2^0, s_2^1, ...). This creates an efficient and flexible iterative optimization process.

Regarding tree depth, our maximum depth setting is correlated with the total simulation budget (e.g., a depth limit of 16 for 512 total nodes). Empirically, we observed that the search tree's effective depth naturally stabilized between 8 and 12, which was sufficient to achieve high success rates. Constraining the depth to be shallower (<8) confirmed its importance, as it caused a significant performance degradation on some tasks (e.g., the success rate on Bidex-CatchAbreast dropped from 0.7 to 0.5).

Question 4: About Design Thought

We are grateful to the reviewer for this crucial point. In our experiments, the "design thought" serves as a high-level plan, generated by the LLM itself, for composing the reward function. We require this thought to explain how different components are combined into a whole, which is an approach analogous to Chain-of-Thought (CoT).

We agree that the "design thought" process could be made more effective with a more structured framework. As the reviewer insightfully suggests with reference [1], utilizing a state memory to track which state variables are used for the reward function would indeed be a more effective approach.

Inspired by this, we conducted a brief preliminary experiment with three conditions: (1) without "design thought" (w/o thought), (2) with our original self-generated "design thought" (w/ self thought), and (3) with a "state-structured thought" (w/ state-structured thought). The "state-structured thought" condition implements the Reward State Execution mechanism mentioned in [1]. In this setup, we track the historical usage of variables and prompt the LLM to generate a "thought" based on this auxiliary information, ensuring the thought specifies how the selected variables are to be combined.

The experimental results show that the "design thought" process is necessary, as its absence degrades reward design performance. Furthermore, a more fine-grained thought process, as implemented in the "state-structured thought" condition, yields additional performance improvements. The Reward State Execution mechanism is indeed analogous to an external memory module in LLM agent design. We are grateful for the reviewer's profound insight and will add a discussion of the performance gains from this structured approach to our revised manuscript.

Regarding CoT reasoning, our original "self thought" method is, in fact, an implementation of this process. As formalized in Equation 1(after line 165), our method first generates a "self thought"—the high-level reasoning plan—before composing the corresponding reward function. Therefore, the performance improvement observed when comparing the w/ self thought and w/o thought conditions already demonstrates the effectiveness of CoT reasoning in our framework.