InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

Thank you for acknowledging our strong empirical results. We will address each of your comments below and also in our revised manuscript.

---

W1: Some typos and missing assumptions in variational lower bound derivation.

RW1: Thank you for carefully checking our derivation process and pointing out the typos and missing assumptions. Based on your comments, we will modify the variational lower bound derivation process as follows:

• First, we will correct the typo in the lower bound of $I(Y; S)$ to $\mathbb{E} _{(x,y)} \left[ \int p _{\phi}(s \mid x) \log q _{\psi}(y \mid s) ds \right]$ in the revised version.

• Second, we will add the assumption about the hidden state distribution for Equation (29) in Appendix A of our paper: “The distribution of the hidden state $p_{\phi}(s \mid x)$ follows a Gaussian distribution with the mean and variance determined by the output of the encoder $f_{\phi}(\cdot)$”

• Third, we will remove the proof of $I(X; S \mid Y) \leq I(X; S)$ in Appendix A of our paper and directly use it as a well-known inequality.

It is worth noting that although there are some local issues in our variational lower bound derivation, such as typographical errors and missing assumptions, they are primarily located in Appendix A and do not affect the final derivation results. Therefore, they barely impact other parts of the paper, and our overall work remains valid.

W2: Question about $KL[f_{\phi}(x_w), r(S)]$.

RW2: Thanks for pointing this out. Based on the hidden state distribution assumption in RW1, this term should be corrected to $KL[p_{\phi}(s \mid x), r(s)]$. In the revised version, we will double-check our paper to avoid similar errors.

W3: Some unclear notations.

RW3: Thanks for your careful suggestion. In the revised version, we will correct the position of the footnotes, further clarify the definition of $X$ and $Y$, and replace $I(X,Y)$ by $I(X;Y)$.

Continuing from the previous block, i.e., Response to Reviewer YUjH [Part II], we now complete our derivation.

As established in the previous block, we have derived that our objective can be estimated as follows:

$L \approx \frac{1}{N} \sum_{n=1}^{N} \left[ \log q_{\psi}(y_n | h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon)) - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n), r(\boldsymbol S) \right]\right]$ .

According to the Bradley-Terry Model, the human preference distribution $p(y_n)$ can be formulated as:

$p(y_n) = p(\boldsymbol x_{n}^w \succ \boldsymbol x_{n}^l)= \sigma(r(\boldsymbol x_{n}^w)-r(\boldsymbol x_{n}^l))$,

where $\sigma(\cdot)$ is the logistic function, and $r(\cdot)$ is the reward model. Notably, in this work, reward model $r(\cdot)$ consists of the previously mentioned encoder $f_{\phi}(\cdot)$ and decoder $g_{\psi}(\cdot)$ and can be expressed as follows:

$r(\boldsymbol x) = g_{\psi}(h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon))= g_{\psi}(f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)+ f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)\boldsymbol \epsilon)$.

Combining the two equations, we obtain:

$\log q_{\psi}(y_n | h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon)) = \text{log}\ \sigma(g_{\psi}(h_{\phi}(\boldsymbol x_n^{w}, \boldsymbol \epsilon)) - g_{\psi}(h_{\phi}(\boldsymbol x_n^{l}, \boldsymbol \epsilon)))$ .

Now, our estimation of the objective becomes:

$L \approx \frac{1}{N} \sum_{n=1}^{N} \left[ \text{log}\ \sigma(g_{\psi}(h_{\phi}(\boldsymbol x_n^{w}, \boldsymbol \epsilon)) - g_{\psi}(h_{\phi}(\boldsymbol x_n^{l}, \boldsymbol \epsilon))) - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^w), r(\boldsymbol S) \right] - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^l), r(\boldsymbol S) \right]\right]$ ,

in which $\text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n), r(\boldsymbol S) \right]$ is replaced by $\text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^w), r(\boldsymbol S) \right] + \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^l), r(\boldsymbol S) \right]$.

Recalling that $h_{\phi}(\boldsymbol x,\boldsymbol \epsilon)=f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)+ f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)\boldsymbol \epsilon$, we abbreviate $g_{\psi}(h_{\phi}(\cdot, \boldsymbol \epsilon)))$ as $g_{\boldsymbol \psi} \circ f_{\boldsymbol \phi} (\cdot)$ for clarity and ease of understanding, leading to the final objective in our paper:

$L \approx \frac{1}{N} \sum_{n=1}^{N} \left[ \text{log}\ \sigma(g_{\boldsymbol \psi} \circ f_{\boldsymbol \phi}(\boldsymbol x_n^{w}) - g_{\boldsymbol \psi} \circ f_{\boldsymbol \phi}(\boldsymbol x_n^{l})) - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^w), r(\boldsymbol S) \right] - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n^l), r(\boldsymbol S) \right]\right]$,

where $\sigma(\cdot)$ is the logistic function.

In the revised version, we will include this detailed derivation in the Appendix A

Q2: Full derivation of Equation (29) based on Gaussian assumption.

Response: The full derivation of Equation (29) in our paper is as follows:

Let $\boldsymbol X$, $\boldsymbol S$, and $Y$ denote the random variable of reward model input, latent representation, and human preference ranking respectively. The variational lower bound of our IB objective can be formulated as follows:

$J(\boldsymbol{\theta})=I(\boldsymbol S;Y)-\beta I(\boldsymbol X;\boldsymbol S) \geq \mathbb{E} _{(\boldsymbol x,y)}\left[\int p _\phi(\boldsymbol s|\boldsymbol x) \log q _\psi(y | \boldsymbol s) d\boldsymbol s \right] - \beta\ \mathbb{E} _{\boldsymbol x}\left[KL(p _{\phi}(\boldsymbol S|\boldsymbol x)||r(\boldsymbol S))\right]=L$,

where $r(\boldsymbol s)=\mathcal{N}(\boldsymbol{s};\mathbf{0},\mathbf{I})$ is the variational approximation of the marginal distribution $p(\boldsymbol s)$. Notably, $p_{\phi}(\boldsymbol{s}|\boldsymbol{x})$ is modeled as a multivariate Gaussian with a diagonal covariance structure, where the mean and covariance are both determined by the output of the encoder $f_{\phi}(\boldsymbol{x})$, specifically $f_{\phi}^{\boldsymbol{\mu}}(\boldsymbol{x})$ for the mean and $f_{\phi}^{\boldsymbol{\sigma}}(\boldsymbol{x})$ for the covariance; please see the response to Q1 for their detailed relationship. Then, given a latent representation $\boldsymbol s$ drawn from $p_{\phi}(\boldsymbol s|\boldsymbol x)$, the decoder $g_{\psi}(\boldsymbol s)$ estimates the human preference ranking $y$ based on the distribution $q_{\psi}(y|\boldsymbol s)$.

By estimating the expectation on $(\boldsymbol x, y)$ using the sample estimate based on the preference dataset $\mathcal{D}=[\boldsymbol x_n,y_n] _ {n=1}^N$, where $\boldsymbol x_{n}$ comprises a human-chosen sample $\boldsymbol x_{n}^w$ and a human-rejected sample $\boldsymbol x_{n}^l$, with $y_n$ representing the corresponding human preference ranking, the variational lower bound of our IB objective can be approximated as follows:

$L \approx \frac{1}{N} \sum_{n=1}^{N} \left[ \int p_{\phi}(\boldsymbol s|\boldsymbol x_n) \log q_{\psi}(y_n|\boldsymbol s)d\mathbf s - \beta \ KL(p_{\phi}(\boldsymbol S|\boldsymbol x_n) ||r(\boldsymbol S)) \right].$

Based on the Gaussian distribution assumption on $p_{\phi}(\boldsymbol s|\boldsymbol x)$, we can use the reparameterization trick to write $p(\boldsymbol s|\boldsymbol x)d\boldsymbol s = p(\boldsymbol \epsilon)d\boldsymbol \epsilon$, where $\boldsymbol \epsilon$ is an auxiliary Gaussian random variable with independent marginal $p(\boldsymbol \epsilon)$. In this way, $\boldsymbol s$ can be expressed by a deterministic function $\boldsymbol s = h_{\phi}(\boldsymbol x,\boldsymbol \epsilon)=f _ {\phi}^{\boldsymbol \mu}(\boldsymbol x)+ f _ {\phi}^{\boldsymbol \sigma}(\boldsymbol x)\boldsymbol \epsilon$. Hence, we can get the following objective function:

$L \approx \frac{1}{N} \sum _ {n=1}^{N} \left[ \mathbb{E} _ {\boldsymbol \epsilon \sim p(\boldsymbol \epsilon)} \left[\log q _ {\psi}(y _ n | h _ {\phi}(\boldsymbol x _ n, \boldsymbol \epsilon)) \right] - \beta \ \text{KL} \left[ p _ {\phi}(\boldsymbol S|\boldsymbol x_n), r(\boldsymbol S) \right]\right]$

In our experiments, we further employ a sample estimate to determine $\mathbb{E} _ {\boldsymbol \epsilon \sim p _ (\boldsymbol \epsilon)} \left[\log q_{\psi}(y_n | h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon)) \right]$, by sampling a $\boldsymbol \epsilon$ from $p(\boldsymbol \epsilon)$, balancing computational complexity. Thus our objective can be estimated as follows:

$L \approx \frac{1}{N} \sum_{n=1}^{N} \left[ \log q_{\psi}(y_n | h_{\phi}(\boldsymbol x_n, \boldsymbol \epsilon)) - \beta \ \text{KL} \left[ p_{\phi}(\boldsymbol S|\boldsymbol x_n), r(\boldsymbol S) \right]\right]$ .

Due to word count limitations, the derivation continues in the next block, i.e., Response to Reviewer YUjH [Part III].

Q1: The relationship between $f_{\phi}$ and $p_{\phi}$.

Response: By assuming latent representation distribution $p_{\phi}(\boldsymbol s| \boldsymbol x)$ follows a multivariate Gaussian with a diagonal covariance structure, whose mean and covariance are determined by the output of the encoder $f_{\phi}(\boldsymbol x)$, the relationship between $f_{\phi}(\boldsymbol x)$ and $p_{\phi}(\boldsymbol s|\boldsymbol x)$ can be formulated as follows:

$p_{\phi}(\boldsymbol s \mid\boldsymbol x) = \mathcal{N}(\boldsymbol s \mid f_{\phi}^{\boldsymbol \mu}(\boldsymbol x), f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x))=\frac{1}{\sqrt{(2\pi)^k |\boldsymbol \Sigma|}} \exp\left( -\frac{1}{2} (\boldsymbol s - f_{\phi}^{\boldsymbol \mu}(\boldsymbol x))^\top \boldsymbol \Sigma^{-1} (\boldsymbol x - f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)) \right)$,

where $\boldsymbol x$ is the network input, $\boldsymbol s$ is the latent representation, $f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)$ is the mean of $\boldsymbol s$, and $\boldsymbol \Sigma$ is the diagonal covariance matrix determined by $f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)$. Specifically, the encoder $f_{\phi}(\boldsymbol x)$ generates two outputs: $f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)$ and $f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)$. The first output, $f_{\phi}^{\boldsymbol \mu}(\boldsymbol x)$, represents the $K$-dimensional mean of the latent representation $\boldsymbol s$. The second output, $f_{\phi}^{\boldsymbol \sigma}(\boldsymbol x)$ is squared to form the diagonal elements of the $K \times K$ diagonal covariance matrix $\boldsymbol \Sigma$.

In the revised version, We will further clarify the relationship between $f_{\phi}$ and $p_{\phi}$.

Q1: Some clarification questions about $\boldsymbol S$.

Response: Thank you for your valuable suggestion. You are indeed correct, and we recognize that the notation for $\boldsymbol S$ has caused some confusion. In the revised version of our manuscript, we intend to make the following clarifications to address the points you raised:

1. We will clarify that the notation $\boldsymbol S$ represents the tuple $(\boldsymbol S^w, \boldsymbol S^l)$, where $\boldsymbol S^w$ and $\boldsymbol S^l$ denote the latent representation corresponding to the accepted and rejected samples, respectively.
2. We will replace $p_{\phi}(\boldsymbol s|\boldsymbol x_n^w)$ with $p_{\phi}(\boldsymbol s^w|\boldsymbol x_n^w)$, and similarly, $p_{\phi}(\boldsymbol s|\boldsymbol x_n^l)$ will be replaced with $p_{\phi}(\boldsymbol s^l|\boldsymbol x_n^l)$.
3. We will clarify that $p_{\phi}(\boldsymbol s|\boldsymbol x_n)$ is equivalent to $p_{\phi}(\boldsymbol s^w|\boldsymbol x_n^w) \cdot p_{\phi}(\boldsymbol s^l|\boldsymbol x_n^l) .$
4. We will further clarify that $r(\boldsymbol S)$ is the independent product distribution over the tuple $(\boldsymbol S^w, \boldsymbol S^l)$, i.e., for an instance $(\boldsymbol s^w, \boldsymbol s^l)$, $r(\boldsymbol s)= r(\boldsymbol s^w) \cdot r(\boldsymbol s^l)$, where $r(\boldsymbol s^w)$ and $r(\boldsymbol s^l)$ each represent a single standard Gaussian distribution $\mathcal{N}(\mathbf{0}, \mathbf{I})$.

Furthermore, we will thoroughly review our entire paper to prevent similar questions. We appreciate your valuable suggestion, which has significantly helped us enhance the clarity and precision of our derivations.

Reviewer YUjH,

We sincerely appreciate your recognition of our work and your timely responses during the rebuttal process. Additionally, the reparameterization trick has indeed been used in our implementation. The related evidence can be found in our submitted code from Line 331 to Line 338 in InfoRM_code/utils/model/reward_model.py for your kind reference.

Best regards.

Thank you for acknowledging the novelty, reasonability, and performance improvements of our method. We will address each of your comments below and also in our revised manuscript.

---

W1: Comparison with UWO or WARM.

RW1: Thanks for your valuable comment. We would like to clarify that we have compared our method with Ensemble RM, i.e., UWO proposed in [1], in our simulated experiments. However, we did not include this comparison in our real-world experiments focusing on larger RMs due to computational constraints, as UWO requires loading multiple RMs during the RL process.

Following your suggestion, we further include WARM [2] in our real-world experiments. The results, reported in Table 1 of the submitted PDF, demonstrate that our InfoRM achieves better RLHF performance as compared with WARM.

Additionally, we would like to highlight that our InfoRM is a fundamental and flexible framework that can easily integrate with other techniques to complement each other. The results in Table 1 of the submitted PDF indicate that integrating InfoRM with WARM further enhances RLHF performance.

W2: Systematic hyperparameters selection for compared methods.

RW2: Thanks for your suggestion. For all methods in our real-world experiments, we directly use the recommended hyperparameters from the widely recognized technical reports [3,4], i.e., a learning rate of 5e-7 and a KL penalty of 0.001 where applicable.

To further address your concern, we report the performance of each compared method with different hyperparameter settings in Table 2 of the submitted PDF. The results show that a learning rate of 5e-7 and a KL penalty of 0.001 are indeed the optimal settings for all methods, validating the fairness and reliability of our experiments.

W3: The choice of initializing critic model from SFT model.

RW3: Thanks for your careful feedback. We would like to clarify that while we modify the network structure of the reward model, we do not alter the structure of the critic model to control the variables. Due to these structural differences, we cannot initialize the critic model from our modified reward model. To ensure fair comparisons, all critic models in our experiments are initialized from the SFT model.

Additionally, we would like to kindly argue that initializing the critic model using the SFT model is also a viable option. As stated in [4]: "Initializing the critic model with a reward or SFT model will converge to similar results."

W4: More datasets and reward/policy model sizes in our experiment.

RW4: We would like to kindly argue that our experiments are not limited to one dataset, one policy model, and a small range of reward model sizes. We clarify our experimental settings in the table below:

Following your suggestion, we also add the TL;DR summarization dataset to our real experiments. The results, reported in Table 1 of the submitted PDF, show that our InfoRM outperforms the compared methods on this task as well.

W5: Running multiple seeds is important.

RW5: Following your suggestion, we conduct our real experiments with a new seed (100) and report the results on the AlpacaFarm dataset in the table below. Due to time and resource constraints, we will include the results from more seeds in the revised version to ensure a robust evaluation.

W6: More neutral descriptions.

RW6: Thank you for your suggestion. We will carefully review the entire paper and revise any descriptions that lack neutrality to enhance the readability of our paper.

W7: Why existing methods of increasing model size or using ensemble models cannot effectively solve misgeneralization？

RW7: The underlying principle behind these existing methods is to implicitly remove spurious features by increasing the reward model's capability, which fails to directly address this issue and results in an inefficient solution.

In contrast, our method explicitly identifies and eliminates spurious features more efficiently. Specifically, our InfoRM achieves this by maximizing the utility of the latent representation for reward prediction while minimizing irrelevant human preferences. Our experimental results show InfoRM's superiority over existing methods with the same model size.

W8: Effectiveness of our CSI as an early stopping method.

RW8: Based on your suggestion, we report the comparison results of our InfoRM with and without early stopping in Table 3 of the submitted PDF. As shown, the early stopping strategy according to our CSI metric indeed enhances RLHF performance, particularly on the Anthropic-Harmless dataset.

Q1: The accuracy comparison between our InfoRM and Standard RM.

RQ1: Thanks for your valuable feedback. To address your concern, we report the accuracy of InfoRM and Standard RM on in-distribution reward model benchmarks (Anthropic-Helpful and Anthropic-Harmless) and out-of-distribution reward model benchmarks (AlpacaEval and Truthful QA) in Table 4 of the submitted PDF. The results demonstrate that our InfoRM achieves better RM accuracy and generalization, leading to improved RLHF performance.

[1] Coste, Thomas, et al. "Reward Model Ensembles Help Mitigate Overoptimization." ICLR 2024.

[2] Rame, Alexandre, et al. "WARM: On the Benefits of Weight Averaged Reward Models." ICML 2024.

[3] Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback." arXiv preprint (2022).

[4] Zheng, Rui, et al. "Secrets of RLHF in large language models part i: PPO." arXiv preprint (2023).

Dear Reviewer XuFQ,

Thank you for your positive feedback and for raising your score. We appreciate your detailed review and valuable comments. We hope our latest responses further address your concerns.

Best regards.

Q1: To clarify, when you write "Ensemble RM" you mean UWO? It would be better if this was made clearer in the paper, as I would take "Ensemble RM" to just describe using a mean ensemble rather than UWO. Given you are using UWO, how do you select the variance coefficient hyperparameter?

RQ1: Thank you for your valuable suggestion. In the revised version of our paper, we will use "Ensemble RM (UWO)" to refer to the UWO method. Additionally, for the two new baselines that we plan to add, namely Mean and WCO, we will use "Ensemble RM (Mean)" and "Ensemble RM (WCO)" to refer to them, respectively. The experimental results for the new baselines are reported in the response to Q2.

Regarding the selection of the variance coefficient hyperparameter for Ensemble RM (UWO) in our simulated experiments, we directly use the recommended value (i.e.,$\lambda$ = 0.1) for PPO from [1], as our simulated experimental setup aligns closely with that in [1]. To ensure fairness and reliability, we report the simulated RLHF performance of Ensemble RM (UWO) with different variance coefficients in the table below. The results demonstrate that $\lambda$ = 0.1 is indeed the optimal choice in our simulated experiments.

Q2: It would also be beneficial to compare to WCO and mean ensemble from Coste et al. for a more thorough set of baselines. & I would still like to see comparisons with ensemble methods in the realistic setting

RQ2: Following your suggestion, we further include the Ensemble RM (UWO), Ensemble RM (WCO), and Ensemble RM (Mean) [1] in our real-world experiments. The comparison results are presented in the following table. To ensure fairness and reliability, we also report the performance of Ensemble RM (UWO) with different variance coefficients. The optimal settings, selected based on the highest win ratio, are highlighted in bold. The results show that our InfoRM using a single RM achieves better RLHF performance compared to the ensemble methods, and integrating InfoRM with the ensemble methods further enhances RLHF performance. We will also include the Ensemble RM (WCO) and Ensemble RM (Mean) in our simulated experiments in the revised version.

[1] Coste, Thomas, et al. "Reward Model Ensembles Help Mitigate Overoptimization." ICLR 2024.

Q3: How exactly do you do early stopping with the CSI metric? Could you describe the algorithm here.

RQ3: We will first provide the details of our previous early stopping validation experiments, explaining how we use the CSI metric to select the stopping point during model training. Following this, we will provide an automated early-stopping algorithm based on our CSI metric for executing early stopping.

Early Stopping Validation Experimental Details: In this experiment, we implemented early stopping by saving multiple checkpoints, visually inspecting their CSI values, and selecting the one before a significant increase in the CSI metric as the final checkpoint. This process is validated by our observations that overoptimization correlates with a significant increase in the CSI metric, making visual inspection effective, as demonstrated in Section 5 of our paper. However, we acknowledge that automating this process by quantifying CSI metric changes would be more cost-effective. Below, we provide an automated early-stopping algorithm based on the CSI metric.

Automated Early Stopping Algorithm Based on the CSI Metric: The CSI-based early stopping algorithm is detailed as follows:

1. Set a maximum tolerable CSI change rate, $\epsilon_{\text{max}}$, which is empirically set to a relatively large value of 10. Let $C_t$ represent the CSI value at the $t$-th evaluation step. The change in CSI at this step is given by $\Delta_t = |C_t - C_{t-1}|$.
2. Calculate the ratio of the CSI change at the $t$-th evaluation step, $\Delta_t$, to the average change across all previous steps, $\frac{1}{t-1} \sum_{i=1}^{t-1} \Delta_i$. This ratio is denoted as $\epsilon_t=\Delta_t / (\frac{1}{t-1} \sum_{i=1}^{t-1} \Delta_i)$.
3. If $\epsilon_t > \epsilon_{\text{max}}$, trigger early stopping and exit the iteration. Otherwise, continue training.

To facilitate understanding, we summarize this algorithm as follows:

---

Input: Maximum tolerable CSI change rate $\epsilon_{\text{max}}$, initial CSI value $C_0$, maximum steps $T$

Initialize: $C_{\text{prev}} \gets C_0$

1. For $t \gets 1$ to $T$ do:
   1. Update model parameters.
   2. $C_t \gets$ evaluate_CSI(model)
   3. $\Delta_t \gets |C_t - C_{\text{prev}}|$
   4. $\epsilon_t=\Delta_t / (\frac{1}{t-1} \sum_{i=1}^{t-1} \Delta_i)$
   5. If $\epsilon_t > \epsilon_{\text{max}}$ then:
      1. Trigger early stopping and exit loop.
      2. Break
   6. $C_{\text{prev}} \gets C_t$

Output: Final model before early stopping.

---

Thank you for acknowledging the clarity of our paper and recognizing the potential of using IB to address reward misgeneralization in RLHF. We appreciate your positive feedback on the introduction of CSI as a valuable contribution. We will address each of your comments and concerns below and also in our revised manuscript.

---

W1: Direct results to support reward misgeneralization/ hacking mitigation.

RW1: Thanks for your feedback. Notably, there is no direct metric to assess reward misgeneralization/hacking. Currently, the only way to demonstrate reward misgeneralization/hacking is by observing the different trends of gold and proxy scores during the RLHF process in the simulated experiments, which we have conducted in our paper and the results demonstrate the effectiveness of our method in reward misgeneralization/hacking mitigation.

In addition, we also propose CSI as an auxiliary metric to measure reward hacking from the perspective of latent embeddings. Measuring hacking more effectively remains an open research question.

W2: Whether IB loss would affect the reward modeling abilities.

RW2: Thanks for your feedback. To address your concern, we report the accuracy of InfoRM and Standard RM on both in-distribution reward model benchmarks (Anthropic-Helpful and Anthropic-Harmless) and out-of-distribution reward model benchmarks (AlpacaEval and Truthful QA) in Table 4 of the submitted PDF. Our results show that IB loss can also enhances reward modeling abilities. Furthermore, the notable improvement in RLHF performance achieved by our InfoRM, as demonstrated in our paper, further substantiates this claim.

Q1: Does IB loss total address the misgeneralization issue? If not, when is IB inefficient?

RQ1: Our InfoRM specifically addresses the issue of reward misgeneralization from an optimization perspective. While our method significantly reduces misgeneralization, we acknowledge that completely addressing the issue is challenging due to the uncontrollable quality of preference datasets in practical applications. Specifically, IB loss may be less effective in scenarios where the dataset quality is poor or highly variable. Under such conditions, the balance parameters of our method must be carefully adjusted to optimize the trade-off between accurate reward modeling and effective mitigation of reward misgeneralization. We will add these analysis in the revised version.

Q2: Could you provide the Upper bound of the generalization error?

RQ2: The upper bound of the generalization error for our method is provided in Theorem 1 below, with the proof available in [1]. Theorem 1 demonstrates that the mutual information between the latent representation and observations, as well as the latent space dimensionality, upper bound the expected generalization error of our InfoRM method.

Theorem 1: Let $|S|$ be the cardinality of the latent representation space of InfoRM, $l(\cdot)$ be the loss function following sub-$\sigma$-Gaussian distribution, $X$ be the reward model input, $S$ be the latent representation of InfoRM, and $\Theta$ be the network parameters, we have the following upper bound for the expected generalization error of our InfoRM:

$E[R(\Theta) - R_T(\Theta)] \leq \exp \left( -\frac{L}{2} \log \frac{1}{\eta} \right) \sqrt{\frac{2\sigma^2}{n} \log I(X,S)}\leq \exp \left( -\frac{L}{2} \log \frac{1}{\eta} \right) \sqrt{\frac{2\sigma^2}{n} \log |S|},$

where $L$, $\eta$, and $n$ are the effective number of layers causing information loss, a constant smaller than 1, and the sample size, respectively. $R(\Theta) = \mathbb{E}_{X \sim D}[l(X, \Theta)]$ is the expected loss value given $\Theta$ and $R_T(\Theta) = \frac{1}{n} \sum _{i=1}^{n} l(X_i, \Theta)$ is a sample estimate of $R(\Theta)$ from the training data.

Q3: How much GPT-4 align with human in reward hacking identifications?

RQ3: We follow your valuable suggestion and conduct a human evaluation to validate GPT-4 as the hacking annotator. Specifically, we randomly sample 100 cases each from Anthropic Helpful, Anthropic Harmless, and AlpacaFarm. Then we engage two expert annotators proficient in alignment studies of LLMs and fluent in English. We ask them to evaluate these cases’ hacking phenomenon based on our pre-given descriptions of the hacking phenomena and the inter-annotator agreement rate is 96%. For cases where the annotators disagreed, we requested that both annotators reassess their evaluations to reach a consensus. The annotation serves as the reference to calculate the accuracy of the GPT-4-based evaluator in reward hacking identification. We find that the human-GPT agreement rate avengingly achieves a remarkable 96.7%, indicating the enhanced reliability of GPT-4 annotations in hacking detection. We will include these results in the revised version.

Q4: What is the architecture of the RM encoder?

RQ4: In our experiments, the RM encoder is derived from the standard RM, with modification to the final layer.

L1: The additional computational overhead of training InfoRM.

RL1: Thanks for your feedback. In fact, the primary consumption of time in RLHF occurs during the RL process. Although the introduction of IB loss does indeed introduce some additional computational overhead for InfoRM training, its impact on the overall RLHF process is minimal. To demonstrate this, we report the empirical estimates of the time costs for reward model training and RL process using Standard RM and our InfoRM in the table below.

[1] Zhang, Sen, Jing Zhang, and Dacheng Tao. "Information-Theoretic Odometry Learning." IJCV 2022.

We appreciate your positive feedback on the use of an information bottleneck objective and your recognition of the insightful derivation of the computable objective. We will address each of your comments and concerns below and also in our revised manuscript.

---

W1: Evidence of solving reward overgeneralization, such as length bias.

RW1: Thanks for your comment. In fact, we have discussed the role of our method in solving reward overgeneralization in Appendix C and cited it in Line 64, Introduction of our paper. In this section, we demonstrate the significant impact of our method in mitigating length bias across twelve datasets. Furthermore, we also discuss other reward overgeneralization phenomena that our method can effectively mitigate, such as excessive caution.

W2: Why does our method reduce the spurious correlation?

RW2: As stated in the Methodology section of our paper, by using variational information bottleneck, our InfoRM is trained to maximize the utility of the latent representation for reward prediction while minimizing the irrelevance of human preferences within it. This process eliminates the irrelevant information (i.e., spurious correlations), resulting in superior reward modeling as compared with the standard LLM-based RM, especially in alleviating reward overgeneralization, as verified in Appendix C of our paper. We will clarify this point further in the revised version.

W3: Comparison with a weakened baseline in Figures 7-13.

RW3: Thanks for your feedback. Due to space constraints, we illustrate the CSI values across various RMs (including RM+KL with differing KL penalties) during the RLHF processes using the AlpacaFarm, Anthropic-Harmless, and Anthropic-Helpful datasets in Figure 2 of the submitted PDF. Additionally, the corresponding RLHF performance, as evaluated by GPT-4, is listed in Table 2 of the submitted PDF. Our results reveal the following:

1. As the KL penalty increases, the growth trend of CSI values is gradually suppressed, indicating a less pronounced reward over-optimization phenomenon. This observation aligns with our intuition and further demonstrates the effectiveness of our CSI metric in detecting reward over-optimization.
2. Although RM+KL achieves comparable performance to InfoRM in mitigating over-optimization, InfoRM consistently outperforms RM+KL in final RLHF performance. This demonstrates that our method can significantly suppress the reward over-optimization phenomenon without largely compromising the final RLHF performance.

We will include relevant results on all testing datasets in the revised version.

Q1: Whether a KL penalty larger than 0.01 improve the performance of RM+KL in the simulated experiments?

RQ1: To address your concern, we provide the simulated RLHF results for RM+KL with different KL penalty values, which can be found in Figure 1(a) of the submitted PDF. Our findings indicate that a KL penalty value of 0.001 yields the best performance. When the KL penalty exceeds 0.01, the RLHF performance significantly degrades. We will include relevant discussion in the revised version.

Q2: It would be a great addition to the paper if RM+KL are compared in Figures 7-13.

RQ2: Please see the response to W3.

Q3: Does the proposed method solve the length bias problem?

RQ3: Please see the response to W1.

Q4: How is ensemble RM implemented?

RQ4: Ensemble RM in our experiments is implemented by combining the average reward across all models in the ensemble with the intra-ensemble variance, strictly following the UWO implementation in [1]. We will include this detail in the revised version.

Q5: Question about “IB dimensionality is set to 3." in Line 716.

RQ5: We apologize for this typo. As analyzed in Appendix D of our paper, the IB dimensionality in our experiments is set to 128, indicating that the final reward can be represented by a vector of this length. We will correct this typo and double-check our paper.

Q6: Report the win rate considering ties in Figure 15.

RQ6: Thanks for your feedback. In Figure 15 of our paper, our calculation of the win rate closely follows [2]. Following your suggestion, we also report the win rate considering ties in Figures 1(b) and 1(c) in the submitted PDF. We will include these results in the revised version.

[1] Coste, Thomas, et al. "Reward Model Ensembles Help Mitigate Overoptimization." ICLR 2024.

[2] Li, Yuhui, et al. "RAIN: Your Language Models Can Align Themselves without Finetuning." ICLR 2024.

Dear Reviewer rGMr,

Thank you very much for your positive feedback. We appreciate your recognition of the paper's contributions and the significance of our empirical results to a broad audience.

Best regards.