Modification-Considering Value Learning for Reward Hacking Mitigation in RL

Thank you for your detailed review and insightful questions. We appreciate the chance to clarify MCVL's mechanism, assumptions, and evaluation. We will revise the paper to make these points clearer.

For the method, I couldn't really get why the proposal would avoid reward hacking. [...] how would it know that it would result in a "worse policy"? [...] I cannot get what is the reasoning behind checking if a sample will improve the policy [...]

The core idea is that the agent has a utility function $U$ which can judge how good the trajectory is according to prior experience. When the agent encounters a new transition, it uses this utility function to check if learning from this transition will result in a behavior aligned with prior experiences. It checks that the new policy would not contradict all previous experiences without reward hacking. It doesn’t mean that the policy is “improved” in the standard RL sense.

When MCVL forecasts that learning from a transition will make the agent execute behaviors that it does not prefer now, it rejects the update. We describe this in L.157(left)-L.131(right) in the Method. We will clarify the description and add another one in Introduction.

The reader has to read in details the method to only understand [...] that the method requires as an INPUT an already trained "safe" policy [...], the users of their method should be prepared to invest an obscene amount of extra compute...

We apologize for the lack of clarity and will make assumptions and costs more prominent.

• Input: MCVL requires an initial utility function, not a safe policy. This function captures initial preferences and can be learned from non-hacking data (e.g., Safe env, or random rollouts as in Reacher experiment). The utility function needs to prefer trajectories from current policy over reward hacking policy by the time reward hacking sequence is triggered. If reward hacking is hard to discover, the utility function has time to learn from transitions before it happens, without relying on the initial one. This is mentioned in the Abstract/Introduction, but we will make it more prominent. The RL policy can train from scratch.

• Cost: MCVL adds computational overhead. We discuss this and potential ways to mitigate it in the first paragraph of the Limitations section. With one of the proposed solutions, using a threshold, we observe a moderate ~1.8x slowdown vs. TD3 in Reacher environment. The goal of the paper was to show that reward hacking can be mitigated by avoiding inconsistent utility updates; we are leaving further optimizations to the future work. We will add more information on this topic in the paper.

I am not even sure if the gridworld domains actually represent a realistic situation [...]. I would expect that reward hacking wouldn't be identified by the user at all. [...] how the gridworlds are simulation a realistic reward hacking situation?

To study reward hacking and measure its mitigation, we need environments where we can detect and measure it. Our experiments show that in several environments used by prior work to illustrate the problem of reward hacking, it can be avoided by preventing inconsistent utility updates. We expect this principle to generalize to situations where reward hacking is hard to detect by the user. Our experiments do not assume humans watching the policy learned and identifying reward hacking.

THe Mujoco environment [...] how avoiding the reward hacking was incrporated in the metrics [...] report [...] the amount of times the reward hacking sequence was triggered.

• Metric: Performance tracks the intended task reward (reaching the target), excluding the hacking reward. When the baseline hacks, it neglects the target, causing performance to drop. MCVL improves performance throughout the training, showing it learns the intended task successfully. This distinguishes it from simply failing to learn. We will highlight this in the paper.

• Hacking Frequency: Figure 3e (bottom) implicitly shows this. Returns above 0 for the baseline require hacking. MC-TD3's returns show it rarely triggers the sequence (legitimate triggers are possible when the intended goal is nearby). We will add text clarifying this.

the method did not show any comparison against another method developed explicitly for avoiding reward hacking.

Direct comparisons are hard due to differing assumptions. MCVL's requirement (initial utility function) is often less restrictive than requirements of other work. The only prior work applicable to deep RL is ORPO [1] and it requires a safe policy. Our response to Reviewer dDMw includes new experiments showing an ORPO-like approach would struggle in our setting.

We hope this clarifies MCVL's rationale and addresses your concerns.

[1] Cassidy Laidlaw, Shivam Singhal, Anca Dragan - Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking, In ICLR 2025

Thank you for your review and valuable suggestions. We address your points below and will incorporate the responses into the paper.

Learned Modifications: [...] But wouldn't the new updated policy with fresh T trajectory samples always be better? [...] comparing policy discrepancies is [not] a systematic way of modifying utility...

Your intuition holds for standard RL agents optimizing a fixed target ($U_{RL}$). However, MCVL agents optimize their current utility ($U_{VL_t}$) and consider the consequences of changing that utility. As explained in the Method section (Lines 157 left-132 right), an update might contain new information that leads to future behavior deemed undesirable by the agent's current values ($U_{VL_t}$).

MCVL evaluates if incorporating new data (leading to $U_{VL_{t+1}}$) would result in behavior with lower expected utility according to the current $U_{VL_t}$. If so, the update is rejected. We are preventing the agent from learning to prefer trajectories (like reward-hacking trajectories) that its current self evaluates negatively. It's akin to considering long-term consequences before changing one's preferences based on short-term gains.

Performance Metric: [...] What does performance signify here? How is it quantified? [...] How does one know this intended behavior beforehand?

The performance metric, standard in prior work such as AI Safety Gridworlds [1] (as mentioned in L. 192), is the discounted sum of true rewards reflecting the intended task goal. This contrasts with the observed reward, which might be flawed and exploitable (leading to hacking).

The intended behavior (and thus true reward) is defined by the environment designer [1]; we use the standard definitions for these benchmark tasks. We describe the performance metric of each environment in Sec 4.1. Crucially, MCVL does not require the true reward/performance metric for training – it's used purely for evaluation to demonstrate that MCVL successfully avoids reward hacking (i.e., maintains high performance on the intended task even when the observed reward is misleading). Declining performance alongside increasing observed returns signals hacking; our results show MCVL prevents this decline.

Ablation Study: paper [...] does not evaluate the efficacy of the proposed method in mitigating reward hacking. [...] compare ablations between trajectory-level and per-step-level learning [...], what if instead of selectively using modify we conduct random modifications [...]?

Our experiments directly evaluate efficacy by tracking both observed returns and the true performance metric. As noted above, the divergence (or lack thereof) between these is the measure of reward hacking [2].

Regarding specific ablations:

• Trajectory vs. Step-level: MCVL uses standard step-level RL (DDQN/TD3 policies map states to actions) but incorporates trajectory-level context when deciding whether to perform the utility update (modify) for a given transition. This decision is the only difference to the baselines we compare to.
• Random modify: Randomly discarding transitions wouldn't remove all transitions with misleading rewards. MCVL's selective rejection is specifically designed to prevent updates predicted to lower current utility. A more relevant baseline, rejecting updates based on reward prediction error, is included in Figure 4a and performs worse.
• Other Baselines: Please also see our response to Reviewer dDMw, where new experiments show that, unlike MCVL, occupancy measure regularization methods, such as ORPO, would struggle to learn the optimal policy while avoiding reward hacking. Section 4.4 and Appendix C also contain further ablations.

Contribution and Novelty: [...] struggling to understand the novel contribution [...] How does the algorithm benefit RL algorithms, [...] recent RL algorithms [...] operate on trajectory-level samples.

MCVL's core contribution is a novel mechanism to mitigate reward hacking within existing RL frameworks by ensuring utility function updates are consistent with the agent's current values.

Its benefit is enhancing the safety and reliability of RL agents by preventing them from learning unintended, potentially harmful behaviors when reward functions are imperfect. This is a critical AI Safety problem.

While Decision Transformer and Diffusion RL use trajectories for sequence modeling or planning, MCVL uses trajectory information to validate potential updates during standard RL training. It modifies existing algorithms (DDQN, TD3) to make them safer, rather than being a new trajectory-based learning paradigm itself. After training, the policy remains a standard state-to-action map.

Thank you again for your review and please let us know if you have any further questions or suggestions.

[1] Leike, J., et al. AI safety gridworlds. 2017.

[2] Skalse, J., et al. Defining and characterizing reward gaming. NeurIPS, 2022.

Thank you for your thorough review and valuable suggestions. We would like to clarify several points and will incorporate these clarifications in the paper.

The paper claims that MCVL iteratively refines the initial coarse utility function, but it seems that the experiment results cannot reflect how the utility function is updated.

Refining the utility function means the agent continuously updates it using transitions not judged as reward hacking. Our results demonstrate MCVL robustly detects hacking while successfully learning improved non-hacking policies across diverse environments. We welcome specific suggestions for additional experiments or metrics if needed.

I think the authors should compare with other methods to mitigate reward hacking [...] Currently, the main results only compare MCVL with ordinary RL, and the results seem unsurprising.

The only prior work that is applicable to regular RL environments is ORPO [1]. Direct comparison with it is challenging due to different requirements (it requires known safe policy, uses policy gradient methods, works only with stochastic policies, and requires tricky discriminator tuning).

To make a fair comparison, we evaluated if an ORPO-like objective could succeed in our setting. We trained Q-functions for Initial (Safe env), Hacking (Full env, observed reward), and Oracle (Full env, true reward) settings using DDQN. We then checked if any regularization weight $\lambda > 0$ exists s.t. the ORPO objective $F(\pi, \pi_{ref}) = J(\pi, \tilde{R}) - \lambda \cdot D(\mu_{\pi}||\mu_{\pi_{ref}})$ satisfies both $F(\pi_{Oracle}, \pi_{Initial}) > F(\pi_{Initial}, \pi_{Initial})$ and $F(\pi_{Oracle}, \pi_{Initial}) > F(\pi_{Hacking}, \pi_{Initial})$. Note that just avoiding reward hacking would be trivial assuming a known safe policy. We tested two ways of obtaining stochastic policies required for ORPO: softmax of Q-values and $\epsilon$-greedy ($\epsilon=0.05$), with $\chi^2$ and KL divergences. Occupancy measures were computed with a 1000 policy rollouts.

The table presents percentage of runs such $\lambda$ exists (10 seeds):

Our results show that frequently no such $\lambda$ exists, indicating ORPO's occupancy measure regularization would likely fail to learn optimal policy without reward hacking for any choice of hyperparameters. Occupancy regularization may struggle when:

1. Oracle policy differs significantly from Initial one (comparable to difference between Hacking and Initial). The experiments in [1] use a modified Tomato Watering environment where the bucket was moved further away. It increases the difference of occupancy between Hacking policy and safe policy.

or

2. High hacking rewards require large $\lambda$, but large $\lambda$ prevents Oracle policy learning (e.g., in Rocks and Diamonds).

In contrast, MCVL consistently achieves the Oracle policy performance in all these environments.

We will add full details of the experiment and additional metrics to the paper.

The proposed method generally makes sense if an initial aligned utility function is available. This work assumes access to a safe environment and a safe reward function. [...] I think the assumption in this setting is strong [...]

Crucially, our core requirement is an initial, reasonably aligned utility function, not necessarily a fully specified Safe environment or reward function. A Safe environment can be used to learn the initial utility function, but it can also be learned from other sources like non-hacking random rollouts (as in our Reacher experiment). We use Safe environments in our gridworld experiments because triggering reward hacking in the original environments is too easy. A Safe reward function is not required for training MCVL; we use it purely for evaluation.

I am not sure if assuming the knowledge of a Safe version is reasonable for practical applications. [...] explain how the setting [...] relates to real-world scenarios.

As discussed (Lines 203-219), the Safe/Full setup models real-world scenarios like transferring from simulation to real-world and restricted lab environment to not restricted. We also mention that Safe is not required if reward hacking is hard to discover. We will clarify this paragraph and add additional examples including

• Training on simpler tasks with simpler reward design.
• Monitoring agent and removing trajectories with reward hacking.
• Using human demonstrations for initialization.

Thank you again for your valuable feedback. Please let us know if our responses address your concerns and if you have further suggestions.

[1] Cassidy Laidlaw, Shivam Singhal, Anca Dragan - Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking, ICLR 2025

Thank you very much for your thorough review and valuable suggestions. We are happy to answer your questions and will incorporate all answers in the final manuscript.

How robust is MCVL to inaccuracies in the learned transition model? For example, if the learned model deviates slightly from the true environment dynamics, can the forecasting mechanism still reliably detect utility inconsistencies?

Our method is robust to noisy transition models. We only use the transition model to compare trajectories produced by two policies. The only requirement for our method to work correctly is that rollouts where the policy executes reward hacking behavior have lower utility than the rollouts of the policy that does not hack rewards. To verify this empirically we are now running an additional experiment. There we add noise sampled from $\mathcal{N}(0,1)$ to each one-hot encoded observation produced by the transition model to simulate a situation where the transition model is inaccurate. We ran this experiment in the Box Moving environment and our method still obtains optimal true reward while avoiding reward hacking. We will include this experiment in the paper.

Is the choice of rollout length (h) for policy forecasting critical? Could the authors elaborate on how sensitive the performance is to different values of h? Is there a principled method to set this parameter, or does it require extensive tuning for each environment?

We provide descriptions of all hyperparameters and discuss how they can be chosen in Appendix G. Here is what we write about rollout length (h):

“This parameter controls the length of the trajectories used to compare two predicted policies. The trajectory length must be adequate to reveal behavioral differences between the policies. In this paper, we used a fixed, sufficiently large number. In episodic tasks, a safe choice is the maximum episode length; in continuing tasks, a truncation horizon typically used in training may be suitable. Computational costs can be reduced by choosing a smaller value based on domain knowledge.“

The performance of the algorithm is not sensitive to $h$, as long as reward hacking occurs within $h$ steps. Extensive tuning is not required as this parameter can be set to maximum episode length.

Would it be possible to test one or two additional MuJoCo tasks to further showcase the performance of MC-TD3?

Unfortunately, existing MuJoCo tasks are not directly applicable because of our evaluation protocol. It requires each environment to have different observed and true rewards to measure episode returns and performance metric respectfully. To make the experiment meaningful, both rewards need to be carefully curated, plausible and explainable. Unfortunately, designing new environments is not simple and beyond the scope of this work. We establish new state-of-the-art performance on existing environments and we agree that designing more complex environments for evaluation of reward hacking is an important direction of future work. Please note that our method is applicable to standard RL environments, and special environments are merely required to evaluate the performance of any algorithm that mitigates reward hacking.

Thank you again for your valuable feedback! Please let us know if you find our responses satisfactory and if you have any further suggestions.