Reinforcement Learning with Adaptive Reward Modeling for Expensive-to-Evaluate Systems

Q1: Experiments with more seeds should be reported.

A1: In our submission, we reported results based on 3 seeds. To address your concern, we have now conducted experiments with 10 seeds. The results (see table in comments for details) confirm that our method remains robust across a larger number of seeds and consistently outperforms the SOTA baselines across all evaluated scenarios. Notably, the improvements is more evident with 10 seeds, as the reduced standard deviations indicate greater statistical reliability.

We will update the results in Section 5 to further validates the effectiveness and robustness of our approach.

Q2: More ablation study.

A2: We have conducted additional ablation studies on the module described in Section 4.4 and the results are as follows:

Removing any component results in a performance decline. Notably, excluding model warm-up leads to random rewards in the initial iterations, causing suboptimal policy optimization and a 7% drop in performance. Similarly, removing parallel computation reduces training efficiency by limiting the reward model’s access to sufficient samples for accurate fine-tuning, resulting in an approximate 10% performance decrease. These results show that each component contributes to the final performance. We will include these results into a new ablation study section (Section 5.6) of the revised version.

Q3: Only provides result for the selected hyperparameter, which is unreliable.

A3: Besides the grid search results in Figure 8, we would like to provide practical guidelines that enable users to efficiently estimate an approximate range for the optimal hyperparameters, ensuring the method’s reliability and reproducibility without exhaustive tuning.

First, the optimal number of fine-tuning interval can be approximated by the complexity of the reward function, which is positively correlated with its evaluation time. For example, Table 2 indicates that reward computation in pandemic control is more time-consuming than molecular design, and grid search identified an optimal fine-tuning interval of 9 and 7 (Figure 8a) for the two scenarios, respectively, suggesting that a more complex reward function requires additional samples for effective RM fine-tuning.

Second, the optimal number of fine-tuning epoch should balance the fine-tuning duration with the policy optimization (see Section 5.5). Specifically, we can measure the time $t_1$ required for one sampling and policy optimization iteration. Then, based on the previously determined fine-tuning interval and the reward function’s computation time, we can estimate the number of samples needed for fine-tuning and compute the time $t_2$ per fine-tuning epoch. The theoretically optimal number of epochs is $N \approx \frac {t_1}{t_2}$, providing a robust starting point. Search within a small neighborhood of this estimated value typically yields the optimal setting, as validated in our experiments.

We will add the above guidelines on hyper-parameters as Section 4.5 of the revised version.

Q4: The RM (reward model) at late-stage becomes worse at predicting the early rewards, using early reward to fine-tune may help.

A4: Thanks for your suggestion. We would like to first clarify that the reward prediction error you have observed does not undermine the performance of our method, as the RM’s role is to provide accurate evaluations for the current policy rather than historical ones. Moreover, it is worth emphasizing that Figure 7 showcases our AdaReMo enables the RM to adapt to the current policy’s outputs, as evidenced by near-zero error values along the diagonal elements.

To further address this, we include samples from previous fine-tuning cycles into RM updates. The updated results for Figure 7 are shown below:

The first value uses only current-cycle samples; the second includes previous-cycle samples. Though off-diagonal errors decreased after using historical samples, diagonal errors increased significantly, reducing the RM’s ability to accurately evaluate the current policy and resulting in a 5% performance drop. This confirms that focusing RM updates on current samples enhances overall optimization effectiveness. We have included the above experiments into Section 5.4 of the revised manuscript.

Q5: Equation expressions.

A5: Please see our response to Reviewer pywE’s Q3.

Dear Reviewer joeE,

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of the core idea behind AdaReMo, particularly the decoupling of the fast decision-making loop from the slow, expensive reward evaluations. We hope the following responses can address your concerns.

Q1: Hyperparameters settings.

A1: Thanks for your constructive comments. Besides the grid search results in Figure 8, we would like to provide practical guidelines that enable users to efficiently estimate an approximate range for the optimal hyperparameters, ensuring the method’s reliability and reproducibility without exhaustive tuning.

First, the optimal number of fine-tuning interval can be approximated based on the complexity of the reward function, which is positively correlated with its evaluation time. For example, Table 2 indicates that reward computation in pandemic control is more time-consuming than molecular design, and grid search identified an optimal fine-tuning interval of 9 and 7 (Figure 8a) for the two scenarios, respectively, suggesting that a more complex reward function requires additional samples for effective RM fine-tuning.

Second, the optimal number of fine-tuning epoch should balance the fine-tuning duration with the policy optimization (Section 5.5). Specifically, we can measure the time $t_1$ required for one policy optimization iteration. Then, based on the previously determined fine-tuning interval and the reward function’s computation time, we can estimate the number of samples needed for fine-tuning and compute the time $t_2$ per fine-tuning epoch. The theoretically optimal number of epochs is $N \approx \frac {t_1}{t_2}$, providing a robust starting point. Search within a small neighborhood of this estimated value typically yields the optimal setting, as validated in our experiments.

We sincerely appreciate your constructive comment, and we have added the above guidelines on hyperparameters in Section 4.5 of the revised manuscript.

Q2: Are the improvements robust across a wider variety of expensive-to-evaluate systems?

A2: Thank you for your insightful question. The proposed AdaReMo is designed as a general solution for expensive-to-evaluate systems. To demonstrate its versatility, we conducted extensive experiments across three quite diverse scenarios: molecular design, pandemic control, and urban planning. The choice of tasks has also been acknowledged by Reviewer kWsg. We sincerely appreciate your suggestion and consider the inclusion of a broader range of expensive-to-evaluate tasks an important direction for extending our approach. In particular, we plan to investigate the effectiveness of AdaReMo in additional domains such as robotics and autonomous driving.

Thank you once again for your valuable comments, please let us know if you have additional questions.

Dear Reviewer pywE,

We express our sincere gratitude for your thorough review and valuable feedback on our paper. Regarding the potential alternative algorithms and the issue of online and offline splits you mentioned, we hope the following replies can address your concerns.

Q1: REINFORCE/REINFORCE++, RLOO and GRPO serve as substitutes for the proposed AdaReMo.

A1: Thanks for your constructive comments. We would like to provide more clarification on the necessity of the proposed AdaReMo compared to substitutes like RLOO and GRPO with additional experiments. In fact, although the introduction of the reward model in AdaReMo draws inspiration from RLHF, its primary focus diverges from the RL optimization algorithms mentioned, such as REINFORCE, RLOO, and GRPO. For instance, GRPO avoids using a critic model to compute advantages due to the typically large size of value networks in RLHF. In contrast, the value network in AdaReMo is significantly smaller, rendering the removal of the critic model less critical. Instead, AdaReMo targets the efficiency imbalance caused by a computationally expensive reward function, which sets it apart from these alternatives.

To evaluate the optimization efficiency of various algorithms, we conducted experiments by replacing PPO with RLOO and GRPO. The results are summarized in the table below:

As demonstrated, AdaReMo combined with PPO consistently outperforms RLOO and GRPO across multiple metrics and domains. While RLOO and GRPO eliminate the need for a critic model, they require multiple sampling, which increases computational complexity and hinders training efficiency. Furthermore, the training time per iteration for RLOO and GRPO does not significantly differ from that of PPO, suggesting that optimizing the critic model is not the primary bottleneck in training efficiency. Additionally, GRPO’s method of calculating advantages can amplify minor differences, potentially destabilizing the policy optimization process.

In summary, these results validate the effectiveness of AdaReMo in addressing reward function complexity. Thank you again for your valuable feedback, and we will incorporate the above discussion and experiments into the appendix of the revised manuscript.

Q2: Why detach the reward modeling to online/offline splits.

A2: Thank you for your comment. The online/offline splits are proposed to address the computationally expensive reward functions, rather than reducing the computational cost of solution generation. Specifically, the online system leverages the reward model to deliver rapid reward estimations, enabling the agent to receive real-time feedback. Without this mechanism, the agent would face significant slowdowns due to the time-intensive reward calculations, as exemplified by applications like RLGN (Meirom et al., 2021) for pandemic control. Conversely, the offline system continuously refines the reward model using the latest exploratory samples to ensure its accuracy. In the absence of this offline phase, the reward model would fail to provide reliable evaluations for samples generated by the evolving policy, leading to significant errors throughout the optimization process, as illustrated in Figure 6a. In summary, the online/offline splits effectively balance real-time efficiency with long-term accuracy.

Thank you again for your valuable comment, and we will include the above discussion in Sections 4.3 and 5.4 of the revised manuscript.

Q3: Typo in Eq4 and Eq7.

A3: Thanks for pointing out this issue. We have corrected it in the revised manuscript. Specifically, Eq4 should be $s_i = \text{MLP}_p(\bf{a}_i)$, where $\bf{a}_i$ denotes the embeddings of action ${a}_i$. We have checked the entire paper and added following corrections as Reviewer kWsg suggested,

• Eq1. $\max_{\Theta} E_{\pi_\Theta} \left[ \sum_{t=0}^{T} \gamma^t r(s_t, a_t) \right]$
• Eq5. $\nabla_{\Theta} J(\Theta) = E \left[ \min \left( r_t(\pi_\Theta) \hat A_t, \text{clip}(r_t(\pi_\Theta), 1 - \epsilon, 1 + \epsilon) \hat A_t \right) \right]$

We hope these responses fully address your concerns and welcome any further questions or suggestions you may have.

Dear Reviewer Q9Xy,

We would like to express our sincere gratitude for your thorough review and constructive feedback on our paper. We are particularly pleased that you have recognized the effectiveness of our proposed approach, AdaReMo, in solving the efficiency bottleneck between policy optimization and computationally expensive reward calculation. We hope the following responses address your concerns.

Q1: Is the warm-up cost included in Figure 6a? A brief discussion of the warm-up cost tradeoff might be valuable.

A1: Thank you for your insightful comments. Figure 6a illustrates the training process from when policy optimization begins, and the warm-up cost is not included. Specifically, the model warm-up takes only one fine-tuning interval, accounting for only 3% of the total optimization time. We also conducted additional ablation experiments by removing the warm-up phase. Without model warm-up, we observed a performance decrease of 7% across the evaluated metrics. This decline arises because the policy is initially trained with randomly generated rewards, which introduces additional noise and hinders optimization. These results underscore the critical role of model warm-up in enhancing training efficiency, leveraging a modest (3%) time investment to yield substantial improvements in model performance. We will add the above results and further discussion into a new ablation study section (Section 5.6) of the revised version.

Q2: Typos and grammatical errors.

A2: We sincerely thank the reviewer for their thorough and careful scrutiny. In response to your feedback, we have thoroughly reviewed the manuscript, correcting all identified typos and grammatical mistakes.

We hope the above replies resolve your concerns and welcome any further questions.