Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification

Thanks for the thoughtful criticism. We considered writing more about how asymptotic results relate to real life, and will do so in the next version of the paper. We additionally hope that this addresses your concerns:

• Asymptotic vs bounded/clipped: With a bounded RM the asymptotic results would indeed not apply. In practice, the same pattern of rare failures producing most of the reward and resulting in near zero utility would likely apply, with sufficiently regularized policies having, say, a 0.1% chance of 10000 reward and otherwise matching the base policy. This could change if error and utility are nonindependent and the base policy is close to optimal so that it actually encounters lots of clipped rewards. (See also the next paragraph, and the last bullet point of our rebuttal to SFxG.) Though it would be possible to prove things about the bounded case, we presented the asymptotic results because we thought they were much cleaner.
• Take-home message: We think there is an interesting take-home message. It is true that KL divergence penalties work sufficiently well to control overoptimization in current language models, though overoptimization has been demonstrated in practice by the results of Gao et al. But our theoretical results show that light-tailed + independence prevents overoptimization. Therefore in such settings we conclude that overoptimization is most likely a result of independence being violated, perhaps as textual features that increase error also decreasing utility. We have improved section 7.4, which discusses this, in the latest version.
• Deterministic RM: Our theorems still apply when the reward model is deterministic. The randomness in our theorems comes from the sentences generated by the LLM (which are random) being passed through a the deterministic reward model, which results in a joint distribution over $V$ and $X$. One of the contributions of our paper is the framing that, to study the final achieved reward, it is sufficient to examine this 2d distribution. Did we understand your question correctly?

We welcome any more comments you might have. Thank you for the review!

Thanks for the positive feedback. To address the weaknesses and questions:

• Implicit regularization: we acknowledge this as a weakness, and have some ideas for how implicit regularization can prevent Goodharting. But there is not really any evidence yet, and we would be excited to see follow-up work like this.
• Typo on line 34: we have fixed this in the latest version. Thank you for pointing it out!
• Question about theorem 4: Thank you for pointing out that we need to assume unbounded light-tailed $V$, we inadvertently dropped it and will explicitly state it for the final version. For the result to hold, the proxy reward should be unbounded so that $\lim _{\beta \rightarrow 0^{+}} \frac{1}{2} E[V \mid \pi] = \infty$ in the proof (for reference $\beta$ is the regularization coefficient of the KL term).
• When proxy reward is bounded: The optimal policy is Boltzmann rational. So how much utility we get will be determined by whether, for state-actions which are close to optimal, the reward is from utility or error. We roughly think that if you clip at large enough values, the asymptotic results in this paper will basically transfer. With more aggressive clipping, you tend to get both error and utility components when the reward is optimized, so neither of the scenarios in our theorems holds (neither wholly error in heavy-tailed nor arbitrarily high utility in light-tailed). It is possible we can characterize how much utility or error the solution will have based on their ratios for the largest $V$, but we’ll leave that to future work.

Clipping can also distort the reward (see response to reviewer G3eM), so while it prevents overoptimization somewhat it also rewards incorrect behavior.

We would welcome any more questions or feedback you have. Thank you for your review!

We wanted to add some information about the bounded proxy reward case:

• A correction: above, we said the optimal policy was Boltzmann rational, but in addition to the exponential term it still has the base likelihood.
• The best-of-N results from the comment to ngHC look very similar when reward is clipped to [-100,100], though in that case utility will increase again for very large N, and also when noise is Levy-distributed. When reward is clipped to [-10, 10], we no longer see overoptimization if utility is heavy-tailed.

Thank you for your review, which gives several intriguing suggestions for experiments, as well as presentation improvements.

As for readability and clarity, we have made various edits in the latest revision of the paper, including streamlining the background section for readers unfamiliar with AI alignment, clarifying the relationship of DMRMDP to RLHF, and improving presentation of graphs. Also see several changes we have already or plan to address in the rebuttal to reviewer G3eM.

We agree that further experimental evidence could be valuable, with some caveats.

• Wider variety of reward functions / real-world experiments: We think work here would be valuable, but it could risk being redundant with the wealth of examples of overoptimization in the existing literature (see concluding paragraph).
• Best-of-N experiments: This would complement our theorems 5 and 6 on optimization by conditioning, which describe the best-of-N distribution. We have already done the experiment to show that best-of-N succeeds in a toy light-tailed regime but fails in a toy heavy-tailed regime, and we will include this in the appendix of the final version.
• Additional experiments comparing KL divergence with other regularization or penalty methods: We will add a synthetic experiment with KL divergence, in response to this review and that of reviewer G3eM. The experiment demonstrates whether a reward function with artificially heavy-tailed error causes catastrophic Goodhart in the KL divergence setting. We are not sure which other regularization schemes we should try: the RLHF literature uses KL divergence, and Best-of-N is an alternative proposed precisely to avoid overoptimization. Perhaps an integral probability metric like the Wasserstein distance, or quantilizing optimization, which only goes up to a percentile of the misspecified reward? It would be good if you could clarify which schemes would be important to try here.
• LoRA: We think studying LoRA would be valuable to nail down exactly when inductive bias prevents catastrophic Goodhart, but considering that a search over LoRA hyperparameters and environments (see paragraph 4 of rebuttal to G3eM for why a single environment is as yet insufficient) would increase the compute requirements by orders of magnitude, this would be valuable follow-up work.

As for why we do not plan to implement all experiments suggested, we see this work as primarily providing a theoretical framework for experimental work already done by others, especially Gao et al [1] and specification gaming [2, 3]. The latter gives 60 examples of real-world utility loss due to specification gaming, several of which we think are examples of "catastrophic Goodhart". The link to previous specification gaming work will be clarified in the latest version.

We thank reviewer ngHC for their feedback. We hope that the theorems, existing experiments, and additional experiments listed above (demonstration of catastrophic Goodhart, and best-of-N study) address the given concerns. Please let us know what weaknesses remain and what we can do to address them, and consider revising your review if we have addressed your concerns to your satisfaction. We would welcome any further feedback and comments. Thank you!

[1] Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10835–10866. PMLR, 23–29 Jul 2023. URL https:// proceedings.mlr.press/v202/gao23h.html

We thank the reviewer for raising several questions and highlighting where the paper lacks clarity.

1. On reparameterizing rewards to change whether they are heavy-tailed:

   Reward can be reparameterized; however, in settings where the true reward is heavy-tailed, making reward artificially light-tailed or bounded can reward behavior that is unintended.

   • For example, a stock-trading agent should be rewarded by profit, but stock returns are known to be heavy-tailed. If we cap or otherwise transform rewards into $[-1, 1]$, it will have no incentive to take into account huge gains or losses. Since RLHF rewards as implemented in Ziegler et al are unbounded, clipping or transforming rewards could itself cause reward misspecification. We think this also applies when rewards are bounded but potentially very large.
   • In some cases, mostly when the reward is not the true intended one, it is possible to reparameterize the reward without adverse effects. In the RL literature for Atari games, rewards are changes in score clipped to $[-1, 1]$ [1].

Thank you for highlighting this, we will discuss when reward can or cannot be reparameterized in Section 7.

2. On precedent for DMRMDP

   • As for precedent, deterministic MDPs are common in the literature e.g. in [2], but we have not seen the Markovian returns property in this exact form. With our definition, we are not trying to introduce a new setting, but rather to list the properties of RLHF that imply our theorems, so the relevance of our results is clear.
   • We intend that trajectories must terminate in a sink state. Thank you for pointing out it is unclear, we will clarify the definition.

3. Is the converse of theorem 1 also true?

   Theorem 3 is effectively a converse to Theorem 1, and shows that if Q is light-tailed, the maximum achievable mean is no higher than the mean of Q in the limit as $\epsilon \to 0$. We will restate this theorem as an equivalence statement. Very good point!

4. Synthetic experimental setup

   We agree that a synthetic experimental setup would add value, and plan to have one in the camera ready. One difficulty is in the modeling choices to make for a realistic setup, especially if we want to relax independence between utility V and error X. (A big takeaway for us was that, because independence + light-tailed error precludes overoptimization in theory, but we observed RMs to have light tails, the overoptimization observed in experiments like Gao et al is likely from broken independence, e.g. a negative correlation between error and utility. This broken independence could take many forms and different models could easily result in different outcomes.)

5. Precise meaning of heavy/light tail under a policy

   We are taking actions as latent variables that determine the reward, but taking the expectation over randomness in the reward assigned by the environment for each possible trajectory (note in Theorem 2, $g(\tau)$ is the average reward of trajectory $\tau$). So if a certain policy in a certain environment has a 50/50 chance between producing trajectories $\tau_a$ and $\tau_b$, and the final state of $\tau_a$ gets reward with a normal distribution with mean 10, and the final state of $\tau_b$ gets reward with a Student-t distribution with mean 20, the relevant distribution of $U$ will be discrete with a 50/50 chance between 10 and 20.

All these questions as well as the two nitpicks will be, or have been already, addressed in the next version of the paper. We welcome any more comments or questions you might have, as these are good feedback that has improved the paper. In addition, we invite you to revise your score if your concerns have been answered to your satisfaction. Thank you for a thoughtful review!

[1]: https://arxiv.org/abs/1709.06009 Machado, Marlos C., Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. "Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents." Journal of Artificial Intelligence Research 61 (2018): 523-562.

[2]: Ronald Ortner, Online regret bounds for Markov decision processes with deterministic transitions, in: Algorithmic Learning Theory, 19th International Conference, ALT 2008, Proceedings, 2008, pp. 123–137.