Data Refinement: Mitigating Reward Over-Optimization in Reinforcement Learning with Human Feedback

Reinforcement Learning with Human Feedback (RLHF) is a pivotal technique that ensures language models are closely aligned with human-centric values. The initial phase of RLHF involves learning human values using a reward model based on pairwise or $K$-wise comparisons. However, a study by Gao et al. (2022) showed that the performance of the reward model degrades after one training epoch, and optimizing too much against such reward model eventually hinders the true objective. This paper delves into these issues, using the theoretical insights to introduce improved reward learning algorithms termed "data refinement". The core idea is that during each training epoch, we not only update the model with the data but also refine the data using the model, eliminating noisy entries. Our empirical findings highlight the superior performance of this approach over the traditional methods.