Robust RLHF with Noisy Rewards

Thank you for all your valuable questions and comments. Below, we provide detailed responses and clarifications to address the points raised.

Theoretical Supplement

Clarifying Table 1 and Equation [4]

Table 1 shows the reward model's (RM) high error rate, indicating significant noise affecting its accuracy. Addressing this noise during RL training is crucial for improving policy performance. We analyze the theoretical impact of reward noise and propose robust mitigation methods.

Error in Equation [4]: The second expectation should indeed be $\Psi$.

$D_{RL}$ represents the set of prompts used during RL, where prompts $($x$)$ query the actor model to generate responses $($y$)$. While our practical algorithm includes KL regularization, $\Psi$'s simplified formulation highlights its role in improving RM robustness. Our focus is on $\Psi$'s theoretical properties, addressing reward noise issues.

---

Formal Definition of the Confusion Matrix

The confusion matrix $C$ is defined as:

$$C = \begin{bmatrix} c_{1,1} & c_{1,2} & \cdots & c_{1,K} \\ c_{2,1} & c_{2,2} & \cdots & c_{2,K} \\ \vdots & \vdots & \ddots & \vdots \\ c_{K,1} & c_{K,2} & \cdots & c_{K,K} \end{bmatrix},$$

where $K$ is the number of reward levels, and each element $c_{i,j}$ is:

$$c_{i,j} = \mathbb{P}(r_{\psi} = j \mid r^* = i),$$

the probability of the noisy reward model $r_{\psi}$ predicting $j$ when the true reward is $i$.

For the binary case ($K = 2$):

$$C = \begin{bmatrix} c_{0,0} & c_{0,1} \\ c_{1,0} & c_{1,1} \end{bmatrix},$$

where $c_{0,1}$ (error when $r^* = 0$ but $r_{\psi} = 1$) is $c_0$, and $c_{1,0}$ (error when $r^* = 1$ but $r_{\psi} = 0$) is $c_1$ in the paper.

The original manuscript omitted the confusion matrix to reduce notation complexity. However, we now include it for completeness.

---

Limitation to Discrete Random Variables

Our analysis focuses on discrete random variables, limiting its generalizability to continuous settings. To address this:

1. Discretize continuous variables into fine-grained levels for approximation.
2. Increase granularity to converge to the continuous case.

This provides a practical, though not fully rigorous, framework for continuous variables.

---

Definition of Robustness

Robustness is defined as:

$$\pi^*_{r_{\psi}}(\Psi) \rightarrow \pi^*_{r^*}(\Psi),$$

where $\pi^*_{r_{\psi}}(\Psi)$ is the optimal solution using the noisy reward model $r_{\psi}$, and $\pi^*_{r^*}(\Psi)$ is the optimal solution with the true reward model $r^*$. Robustness ensures that noise in $r_{\psi}$ does not affect optimization results. This definition will be added to the revised manuscript for clarity.

---

Addressing Novelty Concerns

Our work offers a novel approach to mitigating reward noise in RL:

1. Distinct Method:
   Rather than subtracting baselines or calibrating offline rewards, we derive robustness properties under noise theoretically.

2. Theoretical Contributions:

   • Affine Transformation Robustness (Theorem 1): Demonstrates robustness to reward noise.
   • Contrastive Penalty Reward (Theorem 2): Proposes a framework with theoretical advantages in noise reduction.

3. Experimental Results:
   Besides main experiement, the synthetic experiments also show significant performance improvement under high noise, providing actionable insights for RLHF practitioners.

---

On Missing References

We acknowledge missing the MaxMin-RLHF paper, which uses EM algorithms to address reward mismatch, closely related to our work. This will be cited in the revised manuscript. Other relevant works are already discussed in the "Related Work" section.

---

AlpacaEval Results

Our Llama3.1-ours model achieved a 23.9% LC Win Rate on AlpacaEval, surpassing Mixtral 8x7B v0.1 (leaderboard) and Llama3.1-instruct (20.9%). These results validate our method's real-world effectiveness.

Key Differences and Contributions

1. Variance Reduction Through Aggregation

• The algorithm in the linked paper significantly differs from ours in how it handles reference answers:

  • Subtracts only one reference answer, leading to high variance due to fluctuations in individual responses.
  • Our Method Aggregates multiple responses, reducing variance and yielding more stable and consistent rewards.

• Empirical Evidence:
  To highlight this difference, we conducted an experiment using our in-house test set with GPT-4 Turbo as the judge. The results demonstrate the superiority of our approach over the baseline:

  Comparison	Win Rate
  Single Response vs. Average Reward	37% : 10% : 53%

  • Interpretation: Subtracting only one reference answer introduces instability, often resulting in overly high or low rewards. In contrast, our aggregation method ensures more robust and consistent performance.

---

2. Theoretical Justification

• The linked algorithm is presented empirically, but it lacks both a theoretical foundation and a clear underlying mechanism.

• Our Contributions:
  We provide the first theoretical analysis in this domain, distinguishing our method from the baseline:

  • Affine Transformation Robustness (Theorem 1):
    We derive our formula through an affine transformation of the true preference, inherently enhancing robustness to noise.
  • Contrastive Penalty Reward (Theorem 2):
    We establish the theoretical advantages of our contrastive penalty reward, further validating its effectiveness.

---

Insights on Iterative Recalibration of Rewards

• Experimentation and Challenges:
  We tested iterative recalibration of the reward, as suggested, but encountered the following issues:

  • Reward Hacking: The model began exploiting the entire valid reward range, leading to extreme outputs with excessively high rewards.
  • Prolonged Training Effects: Extended training exacerbated this issue, producing unusual outputs similar to challenges observed in vanilla PPO.

• Mitigation Strategies:
  To address reward hacking, we implemented early stopping, which:

  • Delayed the onset of reward hacking.
  • Improved overall performance compared to vanilla PPO.

• While early stopping proved effective, further experimentation is required to fully quantify its impact. Unfortunately, due to computational resource constraints, we are unable to provide conclusive results at this time.

We appreciate your valuable suggestion regarding iterative recalibration and will explore this direction further in future work. Our findings, both theoretical and empirical, aim to contribute meaningfully to advancements in this area. Thank you for your insights, which have been instrumental in driving this research forward.

Key Differences Between Our Work and VinePPO

1. Theoretical Perspective

• Universal Framework for Reward Robustness:
  Our work introduces a universal framework to improve reward robustness during reward model (RM) inference. This framework is grounded in strong theoretical foundations:

  • Affine Transformation of True Preference:
    The contrastive reward is rigorously derived through an affine transformation of the true preference, inherently ensuring robustness to noise (Theorem 1).
  • Contrastive Penalty Reward:
    We establish significant theoretical advantages for the proposed contrastive penalty reward, demonstrating its effectiveness and reliability in practice (Theorem 2).

• Lack of Theoretical Foundation in VinePPO:
  VinePPO does not provide a comparable theoretical understanding or a framework to ensure reward robustness.

---

2. Empirical Comparison

• Performance in Math Tasks:

  • VinePPO struggles to compete with value function-free methods like RLOO and GRPO.
  • Our method demonstrates consistent advantages across general tasks, including code-related ones.

• Fundamental Motivations:

  • VinePPO relies on Monte Carlo (MC)-based estimates to bypass the value function.
  • Our method directly addresses the challenges of imperfect reward models, leading to improved robustness in the Proximal Policy Optimization (PPO) algorithm.

• Efficiency and Latency:

  • Unlike VinePPO, which requires sampling nine trajectories—an impractical budget for training large language models—our approach is computationally more efficient.
  • By pre-collecting baseline responses, we reduce latency and consistently improve performance across all prompt dimensions.

• Value Function-Free vs. Value Function-Based Methods:

  • Under the same computational budget, VinePPO and other value function-free methods fail to outperform methods that use value functions.

---

3. Experimental Results

• Higher Win Rate Against VinePPO:
  We implemented VinePPO for a fair comparison. Under the same sampling budget (5) and after 500 steps, our method demonstrates a higher win rate:

  Evaluation Metric	Ours vs. VinePPO
  GPT-4-as-a-judge	46.4% : 22.6% : 31%

---

4. Reward Normalization Comparisons

• Baseline Comparisons:
  We compared our approach against static normalization and hard clipping methods that subtract the average reward.

• Strength of Our Method:
  Our baseline reward and dynamic reward scaling methods significantly outperform vanilla reward normalization approaches:

  Evaluation Metric	Ours vs. Reward Normalization
  GPT-4-as-a-judge	36.1% : 34.2% : 29.7%

1. Limited Baseline Comparisons

We appreciate the reviewer’s observation regarding the limited baseline comparisons. Our method fundamentally differs from previous approaches like Offset [1], which aim to reduce reward noise before training by minimizing reward bias in the dataset.

• Key Difference:
  Unlike Offset and similar methods, our approach focuses on optimizing the LLM model using the noisy reward model during RL training. This distinction highlights the unique nature of our method and its contribution to improving robustness in reinforcement learning with imperfect reward signals.

---

2. Oversimplification of Complex Noise Distributions

We thank the reviewer for their perceptive insight into the complexity of noise distributions in real-world data.

• Current Work:
  We provide a brief analysis for multi-level reward settings in Appendix C, demonstrating that our framework can generalize to multiple reward models, provided the reward signals can be discretized.

• Future Plans:
  We recognize that further exploration of complex noise distributions is a promising direction and are excited to investigate this topic in greater depth in future work.

---

3. Plans to Address Other Types of Noise

We sincerely thank the reviewer for bringing up this intriguing direction.

• Future Work:
  We find this suggestion highly interesting and are eager to extend our framework to address other types of noise in future studies. We believe this line of work will further enhance the robustness and applicability of our method in diverse real-world scenarios.

---

Reference

[1] Park, Junsoo, et al. "Offsetbias: Leveraging debiased data for tuning evaluators." arXiv preprint arXiv:2407.06551 (2024).