Goodhart's Law in Reinforcement Learning

We kindly thank you for your review.

• A minor weakness is that the proposed early stopping algorithm might not perform well due to large reward losses from stopping early, which is somewhat expected due to its pessimistic nature. It is true that the pessimistic nature of the algorithm leads to the loss of reward. However, as we point out in Section 5 (Figure 5b), if the proxy is not too far from the true reward, then one can probably expect the loss of reward to be fairly small. However, we think that In practice, early stopping algorithm will not be used on its own, but rather as part of the iterative improvement algorithm (described in Appendix F). This is how our work might lead to some sort of improved RLHF training regime that provably improves Goodharting. (We can then make calls to the true reward function through human feedback to estimate theta, and conservative stopping just translates to collecting human feedback throughout the training process, rather than all at once - which is also how RLHF training processes currently look).

• The algorithm is also fairly impractical because it assumes prior knowledge of θ. We argue that knowing the upper bound of the distance between the true and proxy reward functions is not an extremely unrealistic assumption. For example, for a given reward learning algorithm, we might have certain bounds or rates of convergence to the true reward. Inverse reinforcement learning algorithms often come with proofs of their convergence rates - see e.g. "Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees”, Zeng et al., 2022. Using those convergence rates, we could get an approximation of the distance to the true reward. Note that we can always make the bound less tight, and therefore trade off the accuracy of the input to the early stopping and the optimality of the stopping point, which means that even in the presence of a weak signal our approach might still be useful.

• Your work seems to be tailored to the specific choice of difference metric between two reward functions (their angle). I guess that the main reason for choosing this distance metric is that it is a STARC metric. However, can you provide further justification for why the "angle" is a good choice or even the right choice? We have actually developed the angle metric independently, and only later realised that it is indeed a STARC metric. When working in the occupancy measure space, it is natural to consider the angle between the (projected) rewards - the dot product arises naturally when considering the distances between (normalised) vectors in the Euclidian space. We also note that Skalse et al. 2023, “Invariance in Policy Optimisation and Partial Identifiability in Reward Learning” (indirectly) show that many reward learning algorithms will converge to a reward with STARC-distance 0 to the true reward.

  • What could another reasonable metric be? Our angle metric can be understood as an expectation $E_{S \sim U, A \sim U}[R(S, A) R(S, A)]$ , where both states and actions come from the uniform distribution. It would be interesting to develop a better understanding of the metric of the shape $E_{S,A \sim D}[R(S,A), R[S, A])$ for a larger class of distributions $D$ over states and actions. Particularly interesting are the distributions induced by a particular policy $D_\pi$. Note that this is, in a way, no longer independent of where we are on the polytope $\Omega$, which presents more conceptual difficulties.

  • And, how would choosing a different metric impact your results? As long as we work with a STARC metric, we can always rescale the by a strictly monotonic function and get back to the proper angle. If we use a different distance, our geometric intuition might no longer hold.

Thank you very much for your review. Regarding the weaknesses you have pointed out:

• My concern is on the approximation of θ, which is a relatively new concept and requires some knowledge of the true reward feedback or true reward samples. When such estimation is not accurate, the stopping method could exhibit negative performance. It would be better if the author could show empirical results with approximated θ. The approximation of the distance between the true and proxy reward functions is not necessarily unprecedented. For example, for a given reward learning algorithm, we might have certain bounds or rates of convergence to the true reward. Inverse reinforcement learning algorithms often come with proofs of their convergence rates - see e.g. "Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees”, Zeng et al., 2022. Using those convergence rates, we could get an approximation of the distance to the true reward.
  We think that in practice, the early stopping algorithm will not be used on its own, but rather as a component of the iterative improvement algorithm (discussed in Appendix F). That algorithm provably converges to the optimal policy only if we keep the invariant of never falling into Goodharting. Therefore, using a sufficiently large number of real reward evaluations, in the end we do not lose any reward. In this way, we see our work potentially leading to some sort of improved RLHF training regime. (We can then make calls to the true reward function through human feedback to estimate theta, and conservative stopping just translates to collecting human feedback throughout the training process, rather than all at once - which is also how RLHF training processes currently look).
  We note that our work is meant to be the first step in the investigation of Goodhart’s law in RL: we feel that a proper empirical test of the effects of approximated θ on the performance of the algorithm would benefit from deferring to future work. It would require implementing and testing different reward learning algorithms, which unfortunately lies outside of our scope here.
• This paper is preliminary because it only considers finite state and action space. Extending our theory to the countable or continuous state spaces is an interesting problem. It seems that Theorem 1 holds for countably-infinite state and action spaces, the only change is in the proof of Proposition 1 and Lemma 1. In a continuous setting, the theory becomes more involved, and might require more complex tools from functional analysis. (Note most of our results just rely on separating the policy from the reward, which the above shows we can still do in the general setting.)
• The empirical results are also only on small grid world environments. It is only clear whether such a phenomenon exists in more broad continuous settings and what would be the practical way to solve it in these settings. We note that we use three fundamentally different types of MDPs (densely-connected, tree-shaped, and some variations of grid worlds). Our key results being similar across the classes gives us some evidence that they also generalise to more complex environments. Grid worlds constitute a standard evaluation tool for RL algorithms, densely connected MDPs are used in place of “uninformatively-random” environment, while tree-like environments are good models for many game or search problems. We also note that our Goodharting curves match the ones found in Gao et al. 2022, who use much larger state spaces (a LLM model) and much more complex reward models. However, working out how to apply the early stopping algorithm in those large or infinite spaces is indeed an open problem.

We hope that this helps to clarify our paper, and that the reviewer might consider increasing their score.

We are grateful for your detailed comments and questions. We give a detailed answer point-by-point below:

• Why should we not define it [the optimisation pressure] as simply the distance from the optimal policy? The optimisation pressure is intended to be an independent variable that we can vary freely, such that increasing the pressure leads to policies progressively closer to optimal (i.e. a parametrisation of some curve between the initial policy and the optimal policy). It is unclear how directly varying the distance $\epsilon$ (such that $J(\pi) - J(\pi') \leq \epsilon$) could work in practice. The contour lines (inverse images of $\epsilon$) in this case would result in a sequence of n-dimensional spheres in the space of policies, not a linearly ordered sequence of policies.
• The environments that have been used to establish that Goodharting is pervasive (Section 3) are somewhat simple. It is true that we mostly used small environments with at most a couple hundred states, which are far from real-life conditions. However, we do use three fundamentally different types of MDPs (densely-connected, tree-shaped, and some variations of grid worlds). Our key results being similar across the classes gives us some evidence that they also generalise to more complex environments. Grid worlds constitute a standard evaluation tool for RL algorithms, densely connected MDPs are used in place of “uninformatively-random” environment, while tree-like environments are good models for many game or search problems. We also note that our Goodharting curves match the ones found in Gao et al. 2022, who use much larger state spaces (a LLM model) and much more complex reward models.
• I understand that it is difficult to measure the NDH metric in environments where we cannot solve for the optimal policy, but it would have been nice to understand how pervasive this is in "real" problems, or at least in popular RL benchmark environments. As a side note, the fact that the NDH metric is inherently difficult to measure can be considered as a drawback of the proposed methodology -- can the authors comment? We agree that the fact that the NDH metric requires knowledge of the true reward is a drawback. At the same time, we feel that every “complete” measure of Goodhart’s law would require it (”complete” meaning a measure that is non-zero for any non-optimal policy, w.r.t. the true reward). Therefore, we still think it useful to have it defined - and supported by experiments, which we discuss in Appendix C. We expect that in cases where computing true NDH is computationally prohibitive, some approximate methods can be used. In our setup, we did not need to develop such approximations, so we leave this topic for future work. Another point of view might be that even though we used NDH, our reason for choosing it was simplicity (as we note in the paper), since all the measures we investigated were significantly correlated. Therefore, you could read the contribution here as the fact about the correlation of all “sensible” measures, rather than about the primacy of NDH. That gives us a reason to expect that approximating NDH is achievable in practice.
  As a side note, we want to point out that even though the measure is predicated on knowing the true reward, the point of our work is developing a better understanding of how to optimise misspecified reward without knowing the true reward. Therefore, the NDH metric is primarily useful for analysis, and its weak applicability in the deployment setting would not be an issue.
• It would also have been interesting to more systematically study which properties of environments imply that Goodharting is more likely to take place, do the dynamics of the MDP (e.g. a bottleneck structure) have any role? We agree that this is an interesting question! We have conducted the preliminary investigation, but due to a lack of space, we have delegated it to Appendix G. Specifically, we look at how different hyper-parameters of the experiment (the kind of the environment, the number of states |S|, number of actions |A|, temporal discount factor $\gamma$, sparsity of the environment, determinism of the environment, and more) correlate with the severity of Goodharting. We agree that investigating more systematically the key properties of the environment that would potentially give a list of “sufficient and necessary” conditions for significant Goodharting to occur (such as the bottleneck structure on an MDP you mentioned) would be very interesting, but we feel is outside of the scope of this work.

Thank you for your comments!

• The simulation lemma is indeed somewhat related to our work - specifically regarding the regret bound for our early stopping algorithm. However, we disagree that it already captures our theoretical contribution. The lemma provides the bound on the maximal reward obtained, while we are more concerned about the dynamics of the training, which the lemma does not take into account.
• It is a fair point that our work does not exhaust all relevant facets of Goodhart's law. Still, we note that the point of view we take already has precedence in the literature, and, for example, informs and explains some of the prior results on optimising LLMs with RLHF (e.g. Gao et al, 2022). Manheim and Gabarrant, 2019 provide a classification of different aspects of Goodhart’s law in ML: our work is best understood as (mostly) addressing the Regressional Goodharting, and, in principle, potentially extending to Extremal Goodharting.
• Regarding your specific point about the vagueness of the statement ‘a Goodhart drop occurs for x% of all experiments'. Without figures or tables, it’s difficult to understand what this means, such as what the criteria of a ‘Goodhart drop’ is (any non-zero drop, some negligible drop, etc)". We would like to point out that immediately after writing this we clarify: "meaning that the NDH > 0".
  As you noted, we are studying the problem from a conceptual, formal, and empirical point of view, which creates tradeoffs in what to include in the main text. We chose the high-level logic of the paper to be the four-step reasoning “What is Goodharting? → Does it occur in practice? → Why does it occur? → How can we prevent it?”.
  Separately, we did investigate many other questions - in particular, the relationship between kinds of environments/ hyperparameters and the severity of Goodharting, and provided the conclusions and all relevant tables and figures Appendix G. It was difficult to fit all this in the main text without sacrificing other parts, which we felt were more important to understand the “critical path” of the argument outlined above.
• The early stopping proposal is natural, but also seems very conservative. It is true that the conservative nature of the early algorithm might lead to a significant loss of reward, when used naively. But even then, as we point out in Section 5 (Figure 5b), if the proxy is not too far from the true reward, then one can probably expect the loss of reward to be fairly small. We have reasons to suspect this will be true in practice: we note that Skalse et al. 2023, “Invariance in Policy Optimisation and Partial Identifiability in Reward Learning” (indirectly) show that many reward learning algorithms will converge to a reward with STARC-distance 0 to the true reward.
  However, potentially an even more important point is that we think in practice early stopping algorithm will not be used on its own, but rather as part of the iterative improvement algorithm (described in Appendix F). This is how our work might lead to some sort of improved RLHF training regime. We can then make calls to the true reward function through human feedback to estimate theta, and conservative stopping just translates to collecting human feedback throughout the training process, rather than all at once - which is also how RLHF training processes currently look.
• Furthermore it requires knowledge of θ, which is just assumed to be known. We argue that knowing the upper bound of the distance between the true and proxy reward functions is not an unrealistic assumption. For example, for a given reward learning algorithm, we might have certain bounds or rates of convergence to the true reward (e.g. "Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees”, Zeng et al., 2022). We also note that the angle distance is a STARC metric, and it is similar to the EPIC metric. There has been a considerable amount of literature on using EPIC in practice, demonstrating how to compute the angle alignment of true and proxy reward in concrete settings (e.g. "Skill-Based Reinforcement Learning with Intrinsic Reward Matching”, Adeniji et al. 2023, where they apply EPIC-based learning for robotic control, or "Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning” Rocamonde et al., 2023, who describe computing EPIC for CLIP-based rewards). The last thing to note is that we can always make the bound less tight, and therefore trade off the accuracy of the input to the early-stopping and the optimality of the stopping point, which means that even in the presence of a weak signal our approach might still be useful.