Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking

Thank you for the time spent reviewing our work and your positive feedback. We have updated our manuscript to reflect your feedback and respond to your questions and comments below.

Both types of reward hacking issues arise from insufficient data coverage, a well-explored topic in bandits and reinforcement learning. Therefore, to me, the two new definitions of reward hacking don’t feel entirely novel to me. Although I admit that most prior work hasn’t focused on learning-from-preference settings. This consideration actually leads to some questions that I show in the Question section.

We clarify that the existing literature on insufficient data coverage in offline RL is only related to I reward hacking, and the notion of type II reward hacking and our classification are new. We highlight the differences between the setting we considered in this paper with offline RL, explaining the usefulness and novelty in defining the two types.

In the practice of RLHF finetuning of LLMs, we typically have access an initial model, which already has a decent performance on many downstream tasks, and a previously-collected preference data, which may not have been sampled from initial model (Wang et al. 2024). In this setting, we face two sources of distribution shift: one between the final model and data distribution, and the other between the initial model and data distribution. This setting is different from conventional offline RL, that considers access to an offline dataset (with possibly known data collection policy) and is only concerned with distribution shift between final model and data distribution (Levine et al. 2020). Due to the existence of two sources of distribution shift, we find it useful to define Type I and Type II reward hacking. These definitions motivate the design of our algorithm that achieves theoretical guarantees and a strong empirical performance.

It might be beneficial to merge Theorem 2 with Theorem 1 to provide a unified theoretical result for the proposed algorithm. Currently, they are presented separately, so I wonder how the use of dynamic labels will affect the suboptimality bound shown in Theorem 1.

We agree that it would be beneficial to merge the two results; however, such a merge is challenging as we now explain. Theoretical guarantee for POWER in Theorem 1 hold for general function classes and under a weak form of single-policy concentrability. However, in Theorem 2, we analyze convergence properties of a system of coupled differential equations, reflecting the learning dynamics of preference optimization combined with dynamic labels. Theorem 2 focuses on the softmax policy instead of general function class, as such learning dynamic analysis for general function classes is challenging. Additionally, merging Theorems 1 and 2 may need a new problem formulation that extends the notion of single-policy concentrability so that it depends on both the initial model and data coverage. We think these two challenges are nontrivial and an important direction for future work, perhaps drawing connections to recent results in analyzing policy gradient methods.

Compared to Zhan et al., 2023a, what is the advantage of your method? I think the settings are pretty similar and the concentrability coefficient is defined in the same way (please correct me if I missed any subtitles). Since you claim to address both reward hacking issues by proving a finite-sample bound, as in Theorem 1, do Zhan et al. (2023a) address them as well?

We indeed use the concentrability coefficient definition by Zhan et al. 2023a. However, our method and guarantees have important differences that we highlight below.

1. Our algorithm is highly practical with strong empirical performance whereas the algorithm by Zhan et al. 2023a algorithm is computationally intractable. Zhan et al. 2023a presents an algorithm that, although comes with sample complexity guarantees against type I reward hacking, involves computationally intractable components and cannot be implemented. In contrast, our approach POWER comes with sample complexity guarantees and is highly practical and achieves strong empirical performance. One key element is the weighted entropy which allows for simplifying the maximin objective into a single-step optimization problem.

2. Our method and analysis considers access to an initial model while Zhan et al. 2023a does not. Our approach mitigates type II reward hacking while their approach suffers from type II reward hacking. The work of Zhan et al. 2023a focuses on purely learning a policy from the preference dataset and does not consider access to an initial model. The setting considered by Zhan et al. 2023a is different from LLM practice where the RLHF step is applied to the initial model. As we explained earlier, we define type I and type II reward hacking as we are dealing with two sources of distribution shift, one between the initial model and data, and the other between the final model and data.

Due to the setting considered by Zhan et al. 2023a, their guarantees only hold for type I reward hacking and since their algorithm does not incorporate the initial model, it remains susceptible to type II reward hacking. In contrast, dynamic label procedure in our work mitigates type II reward hacking. (Minor note that reference policy/distribution in Zhan et al. 2023a is not the same as initial model, e.g., they suggest setting it based on the data distribution, and is related to our baseline policy $\pi'$.)

If I simply set $w(.)=0$, what would go wrong in Theorem 1? I believe that you have hidden some w-dependence term in the bound, and I am curious to know what that is.

That is an interesting question. If one intends to solve the maximin objective in equation (6), it is indeed possible to set $w(.) = 0$. Note that Theorem 1 provides guarantee on the maximin objective (6) and does not require any additional assumption on $w$ except for $w(x)>=0$ as defined in the Weighted Entropy Definition 1. We highlight that the dependency on $w$ indeed directly appears in the term $H_w(\pi)$ in the first inequality in Theorem 1 and the second inequality sets $w(y) = 1/|y|$.

In addition to maximin objective, guarantee of Theorem 1 also extends to the POWER objective in light of Proposition 3, where we prove equivalence of the POWER objective to the maximin objective (6) under certain regularity conditions (Assumption 1) and $w(y)> 0$. Intuitively, $w(y)>0$ and $\beta>0$ introduce a strict concavity property which allow for swapping order of max and min.

Can you provide some intuitive explanations on why choosing $w(y)=1/|y|$ leads to $\log V$ in Theorem 1?

The first bound for general $w$ in Theorem 1 depends on the weighted entropy. We show that for $w(y) = 1/|y|$, the bound on weighted entropy leads to $\log V$. For simplicity, consider the case where all responses have length $Y$. Shannon entropy is maximized at uniform distribution which assigns probability of $1/|V|^Y$, reflecting the uniform probability of $1/|V|$ for selecting a token at step 1 up to step $Y$. Maximum of Shannon entropy in this case is $- \log 1/|V|^Y = Y \log V$. However, considering the weights $1/Y$, the entropy becomes $1/Y \times - \log 1/|V|^Y = \log V$.

Minor problem: I don't see how "Sail into the Headwind" is informative in the title

Thank you for your feedback. We consider revising for a better title.

---

Thank you again for your feedback on our work. We hope that we have answered your questions and clarified the differences with Zhan et al. 2023. We are happy to answer any more questions and discuss further in case you believe there are any missing details.

References.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.

Thank you for the time spent reviewing our work and your positive feedback. We have provided our response below.

I am not sure the two components of your method are defending against each hacking type separately ( based on your paper construction, you are designing the approach this way). It seems that dynamic labelling simply gives low trust to low-coverage regions. Then it should help both stop the less-desired outcomes from being pushed to a highly-desired position; AND stop highly-desired outcomes from being pushed to a lowly-desired position where the pair has low coverage. Is that correct? Then should it be a technique for both hacking types?

We clarify that since POWER solves type I, backed by strong guarantees in Theorem 1, but does not mitigate type II reward hacking (e.g. Proposition 4), we design dynamic labels. We agree with you that dynamic labels do not solely mitigate type II; yet, we provide intuition on why we think both techniques are needed to mitigate reward hacking. If we only use dynamic labels: In the high-coverage regions, we converge to empirical rewards $\approx$ robust rewards due to high coverage. In the low-coverage regions, we remain close to the initial model. In regions with medium coverage, using dynamic labels alone interpolate between initial model and empirical rewards. Empirical rewards remain susceptible to statistical noise in medium coverage and thus susceptible to reward hacking. However, if we combine POWER with dynamic labels, we interpolate between the initial model and robust rewards, resulting in better mitigation of reward hacking.

We verify that empirically, the combined method POWER-DL works better than POWER alone and works better then POWER-DL with $\eta = 0$. In Appendix I.3 of the revised paper, we added empirical evaluation showing that if we only use dynamic labels but remove the robustness component with $\eta=0$, test performance degrades. This suggests both components help achieving the best performance.

As a side note, we highlight the difference between divergence-minimization and dynamic labels. Divergence-based methods aim at keeping the learned model close to the initial model wherever the learned model has a decent probability, regardless of data coverage. Therefore, they may be too pessimistic. However, the dynamic label procedure aims at keeping the learned model close to the initialization only in the untrustworthy, low-coverage region and allows the model to learn from the data in high coverage region.

It is nice that you have theoretical guarantee for POWER itself. And empirically, POWER-DL seems to work even better. But I feel the two moves could also interfere with each other due to the fact that highly-covered pairs will still influence the poorly-covered pairs and ultimately many outcomes are competing against each other --- hence even though your set the gradient to zero for poorly covered pairs, the final distribution of the actions can still be changed by other pairs. What are your thoughts on this?}

We agree that the poorly-covered and highly-covered pairs interact in the language space; however, we think POWER and dynamic label procedure can automatically operate on the abstract representation space, i.e., they handle poor coverage and high coverage regions in the representation space rather than language space. Coverage across the representation space is impacted by the aggregated information from all relevant samples, e.g. similar to the spectrum of feature covariance matrix that reveals which directions have higher coverage. For POWER, the general function approximation guarantee shows that it should correctly compute robust rewards in the representation space.

For dynamic labels, we presented our analysis focusing on low and high coverage pairs for better clarity; however, we think a similar argument can be extended to the feature space. Intuitively, the preference dataset implies a collection of preferences over features in the abstract space, e.g., if $\theta(y)$ represents response features. In this case, dynamic labels in principle can diminish gradients for untrustworthy regions in the representation space, considering the aggregate impact of preferences implied by the poorly-covered and highly-covered pairs, combined.

Intuitively, the weighted entropy tends to yield a more uniform distributed policy. Therefore, this is a natural question about how to balance the exploration and exploitation tradeoff.

Indeed using a weighted entropy indeed encourages a more stochastic distribution. An potential benefit of our weighted entropy approach is mitigating the notorious overconfidence challenge, in which RLHF-finetuned models are overconfident and have sharpened output probability (Leng et al. 2024, Kadavath et al. 2022). It would be interesting to study exploration/exploitation trade-off of our weighted-entropy approach in the online setting. However, the focus of our paper is on offline setting and this question is nontrivial and out of scope of this paper.

A more pronounced discussion on ``favoring unbiased policies'' is needed.

By favoring unbiased policies, we are referring to the main premise of The Principle of Maximum Entropy, which asserts that one should ``select the distribution the leaves the largest remaining uncertainty (i.e., the maximum entropy) consistent with your constraints. That way you have not introduced any additional assumptions or biases into your calculations.'' Source: https://mtlsites.mit.edu/Courses/6.050/2003/notes/chapter10.pdf ; also see Guiasu and Shenitzer 1985.

The $\beta$ controls the strength of the explicit pessimism and $\eta$ controls the strength of implicit pessimism. The two factors make the control over the pessimism complicated. In practice, how to balance the two types of pessimism, and in theory, how to mitigate over-pessimism?

Summary of hyperparameters. We first summarize our hyperparameters with theoretical values and experimental values to mitigate over-pessimism.

Clarification on the role of hyperparameters. We clarify that in our objective, $\beta$ is the weighted entropy coefficient that mainly impact the level of policy stochasticity and optimization landscape. This is different from $\beta$ in DPO that is the KL coefficient and impacts pessimism. In our algorithms, parameters against reward hacking are $\eta$ and $\gamma$, where $\eta$ leads to robust rewards and $\gamma$ keeps the learned model close to the initial model in untrustworthy regions. These two parameters allow preventing over-pessimism better than divergence-based methods. First, our framework allows to turn pessimism off completely by $\eta = \gamma = 0$, while it's not possible to set $\beta = 0$ in DPO. Second, $\eta$ and $\gamma$ provide data-dependent pessimism, leading to pessimism in regions with low data coverage, whereas divergence-based methods remain close to initial model everywhere that final model has a decent likelihood. Finally, two parameters $\eta$ and $\gamma$ allow us to better control type I and type II reward hacking, leading to a better final performance as suggested by our experiments. Intuitively, these parameters can reflect our beliefs about the quality of initial model relative to statistical errors in the data. For example, a larger $\gamma$ suggests more trust in the initial model, and a larger $\eta$ suggests noisier data.

How to select pessimism parameters and mitigate over-pessimism in theory. Our Theorems 1 and 2 set the hyperparameters to mitigate over-pessimism as achieve optimal statistical rate. Theorem 1 sets $\eta \asymp 1/\sqrt{N}$ by optimizing inequality (40), avoiding over-pessimism in the worst case. Theorem 1 also finds $\beta \asymp 1/\sqrt{N}$ and in light of Theorem 2, $\gamma$ can be set to $1/N^c$.

How to balance the two types of pessimism in practice. In practice, we treat these parameters as hyperparameters and select the ones that perform best in ranking implicit rewards on validation set. We also tested the robustness of our approach to hyperparameters and added the results to Appendix I.3, showing the performance of our approach is robust to hyperparameters. We note that several other RLHF objectives also have multiple hyperparameters. For example, DPO+SFT has two pessimism parameters $\beta$ is the KL coefficient and $\eta$ is the SFT coefficient, which are selected in a similar manner.

As shown in Huang, Audrey, et al. "Correcting the mythos of kl-regularization: Direct alignment without overparameterization via chi-squared preference optimization." Arxiv, the KL divergence is not enough to induce pessimism in order to mitigate the reward-hacking. Following this argument, the weighted entropy may also encounter similar problems.

Theorem 1 guarantees that our robust maximin objective (6) mitigates reward hacking for general function classes and it provably cannot fail in the example constructed in Huang et al. 2024. For our practical algorithm, we make regularity assumptions on the function class (Assumption 1 e.g. satisfied for Lipschitz continuous classes), allowing us to convert it to a minimax objective leading to the simple objective of POWER (Proposition 3). Huang et al. 2024 provides a construction that relies on (1) a function class that is a discrete set rather than a continuous function, (2) updating parameters despite receiving samples. Hence, the failure argument in Huang et al. 2024 breaks in continuous function classes or gradient-based optimization. In summary, our maximin objective provably cannot fail and our simplified POWER objective provably cannot fail for function classes that satisfy regularity conditions.

On the other hand, we proved (Propositions 1 and 2) that $\chi$PO (Huang et al. 2024) suffers from reward hacking in a reasonable setting with a continuous function class, softmax policy, and bounded rewards. The reason that theoretical guarantees of $\chi$PO do not extend is that it assumes data distribution on responses is equal to the initial model. This assumption is not satisfied in neither of the two common RLHF settings in practice: (1) using previously-collected data from other models, (2) collecting multiple responses from the model and using rankings to keep a subset of them.

The choice of the weights in the weighted entropy needs more details to elaborate; otherwise, the length-normalized technique has been utilized in the existing RLHF algorithms.

1. Our choice of weight in rooted in the sample complexity analysis of Theorem 1. We explained our choice of weight in the paragraph "Benefits of weighted entropy and choice of weights". Our theory shows that this choice of weights lead to removing a linear dependency to the response length in the sample complexity, which is a new theoretical justification for length-normalization.

2. Our Proposition (3) offers a theoretically-sound way of incorporating weights such as length-normalization in preference optimization, which is different from previous methods. Existing methods such as SimPO and RRHF apply length-normalization in the following way: $g(1/|y^+| \log \pi(y^+|x) - 1/|y^-| \log \pi(y^-|x)) + \eta \log \pi(y^+|x)$. However, our derivation suggest a theoretically-sound way of adding weights such as length-normalization, leads to $g(w(y^+) \log \pi(y^+|x) - w(y^-) \log \pi(y^-|x) + w(y^+) - w(y^-)) + \eta w(y^+) \log \pi(y^+|x)$. This leads to two differences to prior work: a weight coefficient in the SFT term and a weight gap in the PO objective.

3. Other choices for weights. The general weight-based objective in Proposition 3 provides a flexible framework for exploring different choices of weight that reflect preferences and other properties of responses. Here, we present other several alternatives for weights. One choice for weights are preference scores from datasets (such as those included in the Helpsteer2 dataset) or those obtained from the reward models. We did not include such weights for a fair comparison with the baselines since they do not readily use such information. Another possibility is a metric that evaluates undesirable characteristics of responses pertaining to reward hacking, such as repetition, hedging, etc. Lastly, inspired by the literature in contrastive learning, another interesting choice views the entropy weights as importance weights designed based on gradients, which aim at improving optimization and training and allow for adaptively focusing on more important preference pairs in the style of $\alpha$CL framework (Tian, 2022).

The examples used in propositions 1 and 2 are very simple, which deviated from the real applications in the fine-tuning of the LLMs, e.g., the definition of the reward function and policy classes in the examples. Therefore, from the results of propositions 1 and 2, it is only sufficient to obtain that the discussed popular preference optimization algorithms do not work well in such restrictive settings. However, the conclusion cannot be extended to the general settings in practice.

1. Our examples are reasonable special case of practical setting and closer to practice than prior theoretical works. The examples we construct in Propositions 1 and 2 are natural and reasonable, special cases of the general setting used in practice. In particular, we analyze the continuous softmax policy class applied to tabular features, and assume that the rewards are bounded. In contrast, previous work by Huang et al. 2024 shows insufficiency of KL divergence and failure of DPO by constructing a reward function class that is a discrete set and applying updates to the model despite receiving no samples. As a result, their argument breaks for continuous function classes or gradient-based optimization.

2. We analyze simpler examples to obtain a better understanding and intuition of design flaws in prior methods, which turns out to be that they overfit poorly-covered samples, leading to reward hacking. This motivate the design of our methods that are both theoretically sound and empirically strong. Furthermore, we prove algorithms such as DPO, IPO, and SimPO suffer from reward hacking in the basic softmax policy with bounded rewards, thus, they provably cannot achieve theoretical guarantees like Theorem 1 achieved by our approach under general function approximation.

3. Conclusion can be extended to more general theoretical settings and is supported by experiments. Theoretically, the conclusion that these methods suffer from reward hacking can be extended in a straightforward manner to more general setting such as linear parameterization and considering the spectrum of feature covariance matrix. Furthermore, our experiments corroborate this finding that methods like DPO and SimPO suffer from reward hacking, as evidenced by degradation of performance on downstream tasks like mathematical reasoning, which is a manifestation of reward hacking (Xu et al. 2024). Empirical study by Rafailov et al. 2024 also shows DPO suffers from reward hacking.

4. Analyzing simple examples of this kind are standard practice in statistical learning theory, and take the necessary first step to understand the more general setting. It is a challenging open problem to analyze the non-convex optimization problem of the transformer model. Therefore, we take the first steps toward understanding the algorithms in special case of softmax policy. This is a standard practice on theory of machine learning/RL and statistical learning theory; see e.g., Jin et al. 2021; Zhu et al. 2023; Lee et al. 2016.

The principle of maximum entropy (principle of the weighted entropy in the paper) has been widely studied in the RL literature and this is not a new idea. In particular, the work by Zhu, Banghua, Michael Jordan, and Jiantao Jiao. "Principled reinforcement learning with human feedback from pairwise or k-wise comparisons." ICML 2023 has theoretically studied the maximum entropy in RLHF/IRL settings.

We clarify that Zhu et. al. 2023 does not study maximum entropy (in a similar sense to us) in RLHF. We have confirmed with the authors Zhu et al. 2023 that they show MaxEnt IRL is similar to the maximum likelihood under Bradley-Terry/Plackett-Luce models, i.e. the standard reward objective $L_{BT}(r)$. Intuitively, the maximum entropy in MaxEnt IRL leads to a softmax similar to BT/PT models. Page 16 of their paper: “the algorithm of max entropy IRL also reduces to the MLE”.

Our work and our use of entropy are completely different from Zhu et al. 2023 and MaxEnt IRL. First, MaxEnt IRL does not have the policy entropy that we have in equation (5). Second, Zhu et al. 2023 propose an RLHF algorithm that leverages uncertainty estimation in linear models. Our approach uses weighted entropy, robust objective, and dynamic labels, and achieves guarantees under general function classes and is completely different from both MaxEnt IRL and Zhu et al. 2023.

Additionally, we do not claim that maximum entropy RL is a new idea and we cite several sources that maximum entropy in RL is well-established and widely used. We simply build our approach, that includes robust formulation and dynamic labels, upon its extension, maximum weighted entropy reward maximization. Also, in the context of RLHF, KL minimization is widely used with the goal of mitigating reward hacking. We prove that algorithms with KL minimization remain susceptible to reward hacking so our proposal is to use weighted entropy instead, which also helps with underoptimization.

As claimed in the paper, the type 2 reward-hacking is due to the decent choices appearing less desirable. It is reasonable to think about that if the over-pessimism in solving the type 1 reward-hacking is the root cause of not well-leveraging the decent samples. However, in this work, there is not a sufficient discussion on this point. See, Chen, Lichang, et al. "ODIN: Disentangled Reward Mitigates Hacking in RLHF." ICML 2024''

The root cause of type II reward hacking. We clarify that the root cause of Type II reward hacking is statistical fluctuations in the low coverage region, which can cause poorly covered, decent choices to appear less desirable, leading to degradation of initial model. While it is possible that if an algorithm for mitigating Type I is overly pessimistic it can exacerbate Type II reward hacking, importantly over-pessimism is not the root cause of Type II reward hacking. For example, Proposition 2 shows that SimPO, which does not incorporate any form of pessimism, still suffers from Type II reward hacking. Intuitively, even if we do not use any pessimism, learned reward of poorly covered choices might still underestimate the true rewards by mere chance and due to high statistical fluctuations in the low coverage region.

POWER is a pessimistic method and the level of pessimism can be adjusted through $\eta$. POWER-DL incorporates dynamic labels to keep the learned model close to the initial model in untrustworthy regions, and it enables a tradeoff between robust rewards and remaining close to the model in untrustworthy regions. This can lead to a better balance and less overall pessimism, as supported by our empirical evaluations in Section 6. We also added hyperparameter robustness evaluations to Appendix I.3 that shows both components achieve the best performance.

Our method avoids over-pessimism of divergence-based methods. We highlight a benefit of our approach over divergence-minimization in terms of avoiding over-pessimism. Divergence-based methods aim at keeping the learned model close to the initial model everywhere that the learned model has a decent probability, regardless of data coverage. Therefore, they may be too pessimistic by keeping the model close to initialization even in places with high data coverage. However, our dynamic label procedure aims at keeping the learned model close to initialization only in the untrustworthy, low-coverage region and allows the model to learn from the data in high coverage region.

Thank you for the time spent reviewing our work and your feedback on our work. We have updated our manuscript to reflect your feedback, extending our related work discussion (1.5 pages additional discussion in Appendix B) and clarifying our contributions. We have added many of the experiments that you suggested including MT-Bench evaluation as well as training and evaluation on the Mistral family. We have multi-iteration extension of our algorithm currently running and we will update the manuscript with those results before the end of discussion phase. We respond to your comments and questions below, and clarify our contributions.

There are many important and highly related literature that have not been reviewed in the paper. For example, a. Chen, Zixiang, et al. "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models." ICML 2024 b. Rame, Alexandre, et al. "WARM: On the Benefits of Weight Averaged Reward Models." ICML 2024 c. Pan, Alexander, Kush Bhatia, and Jacob Steinhardt. "The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models." ICLR 2022 The insufficient review of the related literature on reward-hacking and model alignment limits the credibility of the two types of reward-hacking in the paper.

Thank you for pointing out these papers. We have revised our paper, adding a thorough discussion and comparison with these papers and other relevant papers to Appendix B. We are happy to add more papers in case anything is missing. We present a comparison with the papers you mentioned here as well for completeness.

• Chen et al. 2024 propose an iterative self-play mechanism to improve LLM performance without additional preference data, that outperforms DPO. This work focuses on improving LLMs in an online, iterative setting with LLM feedback and does not study preference optimization from offline data, reward hacking, robustness to partial data coverage, and statistical learning properties. The focus of our work is the study of reward hacking in offline preference optimization and designing empirically strong methods to mitigate reward hacking that come with strong statistical learning guarantees. The offline setting is considerably different from the online setting in Chen et al. 2024, as the offline setting is concerned with handling partial data coverage, whereas the online is concerned with effective exploration and data collection.

• Rame et al. 2024 consider reward hacking from an empirical perspective, attribute reward hacking to distribution shift and human preference inconsistencies, and propose training multiple reward models and averaging them in the weight space. We study reward hacking from a statistical learning theory perspective and identify two types of reward hacking, both related to partial data coverage---we do not consider human preference inconsistencies in this work. We then propose theoretically founded methods to mitigate reward hacking. Our approach is much simpler and faster as we directly finetune the policy, and do not require training multiple reward models as in Rame et al. 2024.

• Pan et al. 2024 consider reward hacking due to human misspecification of the (proxy) reward model from a purely empirical perspective. They construct synthetic examples of misspecified rewards across various environments. They study impact of model size, optimization, and training on reward hacking given human misspecification. In contrast, we conduct a study of reward hacking in preference optimization that arise due to statistical fluctuations (and not human misspecification) that stem from partial data coverage. We conduct theoretical analysis, showing failure of existing methods, and design robust algorithms with theoretical guarantees. We also conduct experiments using real datasets---as opposed to synthetic designs of reward hacking in Pan et al. 2024---in LLMs showing our robust methods lead to significant improvement in both alignment benchmarks and maintaining performance on academic benchmarks such as mathematical reasoning.

Thank you for the time spent reviewing our work and your positive feedback. We have updated our manuscript to reflect your feedback and respond to your questions and comments below.

Wu et al. (2024) introduce a dynamic beta-based approach that could be a relevant baseline here, especially given its emphasis on preference data quality.

Thank you for suggesting this work. We have added a comparison of our work to Wu et al. 2024 to Appendix B.2. We are currently training $\beta$DPO to add it as a baseline to our experiments and will update the paper with the results before the end of the discussion period.

In Proposition 4 (I), POWER-DL is shown to recover the best-in-class policy for the constructive example given in Proposition 1. Does this imply that POWER-DL fully mitigates type 1 reward hacking? Or could there still be instances where type 1 reward hacking persists?

Guarantees in Theorem 1 show that POWER fully mitigates type I reward hacking, as long as the competing policy (such as the best in class policy) is covered in the dataset. Example in Proposition 1 satisfies assumptions of Theorem 1 such as the coverage assumption. Proposition 4 does not imply full mitigation of type I reward hacking by POWER, but servers as a sanity check and proof details provide some intuition on why POWER objective mitigates this type of reward hacking.

In Table 2, POWER-DL seems to perform better in the base setup (where it uses an existing preference dataset) than in the instruct setup (where a preference dataset is constructed by sampling from the initial model). Does this suggest that POWER-DL is more effective under a distribution shift between the initial model and the model(s) used to collect preference data?

Yes, we observe the improvement of POWER-DL over other methods is more pronounced in the base setup. This is expected because as you said, the distribution shift between data and the initial model is more significant in the base setup, and this increases the risk for reward hacking---also observed in the empirical study by Rame et al. 2024. POWER-DL mitigates reward hacking and therefore, we observe more gains in the base setup compared to other methods that remain susceptible to reward hacking. This suggests that the improvements by POWER-DL are due to mitigation of reward hacking.

---

Thank you again for your feedback on our work. We hope that we have integrated your feedback in the revised draft and answered your questions. We will revise the manuscript with the $\beta$DPO numbers before the end of discussion phase. We are happy to answer any more questions and discuss further in case you believe there are any missing details.

References.

Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In Forty-first International Conference on Machine Learning, 2024.