Explainable Reinforcement Learning from Human Feedback to Improve Alignment

Thanks for your constructive reviews. We address your comments in a point-to-point manner.

Answer to Weakness 1: Thanks for mentioning the explanation. Please note that the primary focus of this paper is to improve LM performance, and the explanation module is used as a tool to achieve this goal. Therefore, we evaluate the effectiveness of our method via downstream win rates.
Moreover, we provide a mathematically grounded explanation module and evaluate the explanation in subsection 6.3. First, our explanation follows the definition of post-hoc example-based explanation in [D1,D2] where a model’s decision is explained by identifying relevant examples from training data. In our setting, we find a set of training data to explain why a test-time response $\bar{y}$ is generated by explaining why $(\bar{x},\bar{y})$ has a high reward. This reformulation is mathematically grounded: the identified examples contribute to the high value of $r_\theta(\bar{x},\bar{y})$, and since RL optimizes the LM to maximize reward, a high reward $r_\theta(\bar{x},\bar{y})$ increases the likelihood that $\bar{y}$ is generated. Therefore, the identified examples (indirectly) contribute to the generation of $\bar{y}$. Second, we explicitly evaluate the fidelity (a widely used metric in XRL) of the explanation in subsection 6.3. In the case of explaining why the model generates unsatisfactory responses, fidelity in XRL [D3,D4] is defined as performance improvement from acting on the explanation. In our case, it means whether the unlearned model generates better responses. We measure this improvement using win rates of the unlearned model over the original RLHF model. Since the win rate is larger than 50%, it means that the unlearned model generates better responses. We also provide qualitative explanation examples in Appendix D.2, which is human-interpretable.
Following your suggestion, we add a qualitative evaluation based on human plausibility, which is a widely-used metric in XRL [D3,D4] that directly evaluates the explanation. We recruited 15 participants, and for each test-time response, provided them with our explanation and a set of randomly selected training examples of the same size. Participants were asked to choose which set better helps them understand why the model generates the response. The percentages favoring our explanation in the four tasks (opt+full-hh-rlhf, pythia+full-hh-rlhf, pythia+tldr, llama+tldr) are: 87%, 93%, 80%, 80%.

Answer to Weakness 2: We would like to clarify that the goal of the explanation module is to find "a cause" (lines 2 and 40) instead of the sole cause of unsatisfactory responses. We agree that other sources, including SFT and RL, can also contribute to unsatisfactory responses. However, it is infeasible to enumerate all such causes, and this paper focuses on the reward data for explanation. In our experiments, the three stages use disjoint partitions of the dataset (line 278). Thus, the LM does not directly observe the data used to train the reward model. However, focusing on the reward data remains well-justified: although the LM does not directly access this data, this data is used to train the reward model, and RL optimizes the LM to maximize the reward. Therefore, the reward data indirectly but causally influences the LM’s responses. We will include this discussion to clarify this causal pathway and acknowledge other possible sources of causes.

Answer to Weakness 3: Following your suggestion, we will clarify the causal reasoning path in our setting: reward training data influences the reward model, which determines reward values for LM outputs, which in turn shapes the LM policy via RL optimization. Therefore, examples that contribute to a high reward for a response indirectly but causally increase the likelihood that the LM generates that response. We will rephrase line 25 to emphasize "many responses are unsatisfactory after RLHF". We will add brief description on RM (which learns a reward signal to approximate human preference) and RL (which learns an LM to optimize reward). We have additional complete related works in Appendix A. We will include the limitation that this paper does not address other sources of causes, and one future work is to find other sources of causes.

Answer to Weakness 4: Following your suggestion, we add the following results:

1. We conduct an additional experiment using llama-2-13b model and imdb dataset. We report the win rates over the SFT model. The win rates are evaluated by Skywork-Reward-V2-Llama-3.1-8B (ranked top in RewardBench 2) and GPT-4 with the order of the candidate responses randomized:

The above results demonstrate that our method (XRLHF) can further improve the base RLHF algorithms (PPO and ReMax) on a large base model (llama-2-13b) and a new dataset (imdb).

2. We evalute the models on new and unseen datasets. Recall that the experiment section has two tasks: dialogue and summarization. The dialogue task includes opt-1.3B and pythia-2.8B models trained on the full-hh-rlhf dataset. The summarization task includes pythia-2.8B and llama-2-7B models trained on the Reddit TL;DR dataset. Here we evaluate the models trained in the dialogue task on a new dataset PKU-SafeRLHF, and evaluate models trained in the summarization task on a new dataset webis/tldr-17.
   (1) opt dialogue task

(2) pythia dialogue task

(3) pythia summarization task

(4) llama summarization task

The above results demonstrate that our method can also improve RLHF performance on new and unseen datasets.

Answer to Question 1: In our setting, we define a cause as a set of reward training data that indirectly contributes to the LM’s generation of $\bar{y}$ by contributing to the high value of $r_\theta(\bar{x},\bar{y})$. As discussed in our answer to Weakness 1, this causal path is mathematically grounded. Moreover, our explanation follows the definition of post-hoc example-based explanation [D1,D2] where a model's decision is explained by influential training examples. In our case, we identify the training examples to explain why the reward model assigns a high value to $(\bar{x},\bar{y})$, and this further explains why the LM generates $\bar{y}$ through the causal path.

Answer to Question 2: Please refer to our answers to Weaknesses 1 and 2.

Answer to Question 3: We agree that different prompts can affect the magnitude of the raw reward output by the reward model. Therefore, we compare calibrated rewards, rather than raw rewards, to the reward threshold. Specifically, the calibrated reward of a response is its raw reward subtracting the raw reward of the SFT model response to the same prompt. This calibration reduces prompt-specific bias in reward values. We then apply a fixed threshold to these calibrated rewards to classify satisfactory and unsatisfactory responses. We will clarify this calibration to avoid confusion.

Answer to Question 4: We evaluate the reward model’s performance before and after unlearning by measuring its agreement with GPT-4 preferences. Specifically, we randomly sample 500 examples from the test set, where each example consists of a prompt and two responses. For each pair, we compare the reward model's ranking with GPT-4’s judgment and compute the percentage of cases where the reward model assigns a higher score to the response preferred by GPT-4.

The above results show that unlearned reward has better performance than the original reward.

Answer to Limitation: Following your suggestion, we have addressed all your comments in our answers to the weaknesses and questions.

[D1] J.Crabbé et al, "Explaining latent representations with a corpus of examples", NeurIPS, 2021.

[D2] H Sun et al, "Accountability in offline reinforcement learning: Explaining decisions with a corpus of examples", NeurIPS, 2024

[D3] W Guo et al, "Edge: Explaining deep reinforcement learning policies", NeurIPS, 2021.

[D4] Z Cheng et al, "Rice: Breaking through the training bottlenecks of reinforcement learning with explanation", ICML, 2024.

We sincerely appreciate your time and effort in reviewing our paper. Thank you for recognizing the contributions of our work and for raising the score.

Following your suggestion, we will revise the title and abstract to emphasize that this paper uses explanation as a tool to improvement alignment, rather than focusing on the explanation itself. We will highlight the causal path to calrify why focusing on the reward training data is reasonable to explain the LM's response. Additionally, we will include the evaluations on the reward model performance in the experiment section.

Thank you again for acknowledging our efforts in addressing your concerns.

Thanks for your constructive review. We address your comments in a point-to-point manner.

Weakness: The base model is too weak, and models from two years ago are not sufficiently convincing given the current progress in large language models. Since policy unlearning based on negative gradient updates has already been widely used, the minimum-distance-based iterative training data selection is the main contribution of this paper. However, its effectiveness needs to be demonstrated on more recent and advanced models through broader and more up-to-date experiments.
Answer: Following your suggestion, we conduct an additional experiment where we use Llama-3.1-8B as the base model and full-hh-rlhf dataset as the training dataset. Same as the main paper, we partition the training data into three parts: 20% for supervised finetuning, 40% for reward learning, and 40% for reinforcement learning. We consider two base RLHF algorithms: PPO and ReMax. We report the GPT-4 win rates of the base RLHF algorithms and our method (XRLHF) over the SFT model on the test set of full-hh-rlhf:

The above results demonstrate that our method can further improve RLHF (e.g., PPO and ReMax) using the recent model Llama-3.1-8B.

Question 1: It is necessary to conduct experiments on more recent models (at least LLaMA-3.1 or Qwen-2.5).
Answer: Please refer to our answer to Weakness.

Question 2: This method should also get a better reward model, so its effectiveness need to be validated on reward model benchmarks such as RewardBench, PPE-Preference, RMB, RM-Bench, and JudgeBench.
Answer: We evaluate our reward models on RewardBench, specifically using the "allenai/reward-bench" dataset. This dataset has two splits: "raw" and "filtered", and we use the "filtered" split. Each data point in this split includes a prompt, a chosen response, and a rejected response. For evaluation, we score both the chosen and rejected responses using our reward models. If the reward assigned to the chosen response is higher than that of the rejected response, the reward model is considered correct at this data example. The overall score is calculated as the percentage of examples where the reward model is correct.
We provide scores of our reward models (before and after unlearning) across the five experiments: opt-1.3B+full-hh-rlhf, pythia-2.8B+full-hh-rlhf, pythia-2.8B+TL;DR, llama-2-7B+TL;DR, llama-3.1-8B+full-hh-rlhf. Note that we use two different base RLHF algorithms: PPO and Remax. Although these two RLHF algorithms use the same reward model before unlearning, they can generate different unsatisfactory responses and thus lead to different unlearned reward models. Therefore, we report two versions of the unlearned reward models corresponding to PPO and ReMax.

The above results show that the reward models become better after unlearning.

Thanks for your detailed reviews. We address your comments in a point-by-point manner. Due to space limitation, we cannot include our answers to the N comments.

Answer to W1: We add three baselines using the opt-1.3B model and full-hh-rlhf dataset: (1) DU: we directly unlearn $D_u$ from the RLHF policy $\pi_0$ by minimizing the log-likelihood of $D_u$; (2) RLHF500: we use the last 500 preference data in the test split of full-hh-rlhf, add this 500 data to the original train split, and run RLHF on the new train split. (3) RL500: we use the original reward model and original RLHF policy $\pi_0$ to run a second-round RL on the 500 data. Note that we exclude the 500 data when we evaluate performance on the test set. Therefore, the test set we use here is 500 examples smaller than the one used in the main paper. We report GPT-4 win rates of the original RLHF, our method, and the three new baselines over the SFT model on both the validation set $D$ and the new test set.

The above results show that our method (XRLHF) significantly improves RLHF on both validation and test sets, while RLHF500 and RL500 only slightly improve RLHF. The method DU also improves RLHF, however, the improvement is smaller than XRLHF. The reason is that unlearning unsatisfactory responses can degrade satisfactory responses (explained in lines 237-239). In contrast, XRLHF not only unlearns the unsatisfactory responses but also retains the satisfactory responses (explained in lines 257-260).

Answer to W2: We compare our method (XRLHF) to full retraining (FR) using the opt-1.3B model and full-hh-rlhf dataset. Specifically, we exclude the explanation (bad preference data) from the original reward training data, retrain a reward model on the new reward training data, and use the retrained reward to run RL. We consider two variants of FR: (1) naive FR: we follow the standard RLHF pipeline and use the retrained reward to run RL on the SFT model. (2) retain FR: we follow lines 257-263 where we use the retrained reward to tune the original XRLHF policy only on the unsatisfactory responses but restricts KL divergence from the original XRLHF policy on the satisfactory responses. We report GPT-4 win rates on both validation and test sets. The test set is the one used in the answer to W1.

The above results show that our method outperforms naive FR because naive FR can degrade satisfactory responses (explained in lines 237-239). The retain FR performs slightly better than our method because it completely removes the effect of the explanation and retains the satisfactory responses (lines 257-260). However, our reward unlearning achieves similar performance while requiring around 8% of the time for reward retraining.

Answer to W3: To quantify the degradation on $D\setminus D_u$, we provide the percentages of prompts of $D\setminus D_u$ where XRLHF response is worse than RLHF response (evaluated by GPT-4).

Answer to W4: We compare our method to PerpCorrect based on opt-1.3B model and full-hh-rlhf dataset. Since PerpCorrect does not have ReMax implementation, we only compare to it using PPO as the base RLHF algorithm. We provide the GPT-4 win rates of our method over PerpCorrect on both the validation and test sets: 60.5% (validation set), 56.2% (test set). Since the win rates are above 50%, it shows that our method outperforms PerpCorrect in general.

Answer to W5: We use a python package "cvxpy" to solve this quadratic program. First, we need to specify the variable, objection function, and constraint in equation 2. Second, we can formulate the quadratic program using "problem=cvxpy.Problem(objective,constraint)" and solve this problem using "problem.solve()".

Answer to W6: The weights of the convex combination are not ignored in the unlearning phase. Since a convex combination can include many training data with negligible weights, in practice, we only pick the top 30% data with highest weights as explanation and only unlearn these 30% data.

Answer to W7: Your understanding is correct. During reward unlearning, we only update the parameters of the final linear layer, while keeping the representation part frozen. This design has two benefits: (1) interpretability: since the representation part is fixed, it is easy for humans to interpret how a test-time example can be decomposed by training examples and how much each training example contributes to the high reward of the test-time example; (2) efficiency: only updating the final linear layer during unlearning is much more efficient than updating the parameters of both the transformer and final linear layer.

Answer to W8: We will add a discussion that if a sufficiently good reward model is available, a practical approach is to directly train a policy on it. In our case, we do not assume the access to such a good reward model during training. Instead, this kind of reward model is only used as a proxy for human judgments during evaluation.

Answer to W9: We agree that finding the smallest possible decomposition can also be a valid solution. However, this approach can be computationally intractable in practice, because it requires searching over all possible subsets of the training data (which is a space that grows exponentially with the training data size) to find the smallest decomposition. In contrast, our method finds the closest decomposition, which enables an efficient search strategy: we start from the nearest neighbor and iteratively add new data until a decomposition is found. This strategy is much more efficient because it avoids searching over all the possible decompositions (explained in lines 201-207), and we prove that this method's computation complexity is only polynomial (Theorem 1). Empirically, we find that it is typical to add no more than 100 nearest data points until a valid decomposition is found, while the total number of reward training data is more than 35,000.

Answer to W10: The goal of reward unlearning is to eliminate the effect of the example in Table 8 on the reward model, not to encourage the rejected response of this example. Since the original reward model is learned by maximizing the log-likelihood of this example, we use negative gradient to unlearn this effect. We agree that fully minimizing the log-likelihood of this example can encourage the rejected response. However, we do not run the negative gradient until convergence, instead, we only run several steps of negative gradient to unlearn this example.

Answer to M1: In the worst case where the entire dataset needs to be considered, starting from the closest data point does not have additional benefit. However, it is beneficial in practice because empirically, we find that a valid decomposition is typically found by considering no more than 100 nearest data points, while the total number of the data points is more than 35,000.

Answer to M2: We will add responses generated by PPO/ReMax. In D.4, we only include examples of XRLHF because we want to demonstrate the responses generated by our algorithm, and the comparison of XRLHF and RLHF is already evaluated using win rates.

Answer to M3: The goal of unlearning is not to invert the influence but to reduce the influence. We agree that fully minimizing the likelihood can invert influence, however, we only deploy several steps of negative gradient to reduce the influence.

Answer to M5: We will add a discussion to revise this claim and clarify that this claim considers a simplified case, while in practice, the policy may not maximize the reward because of the insufficient RL optimization and the KL constraint.

Answer to M6: We will add a clarification to the text that the closest convex combination means the combination where the sum of the distance of each member to the projected target is minimum.

Answer to M7: We will clarify that multiple decompositions may exist to influence the LM's responses and our method just finds one of them. We will also discuss that our explanation provides a plausible attribution of influence on the LM's responses, but there could be other sources that also influence the LM's responses.

Answer to W11: We will add the limitation that for larger models with high capability, in order to get enough unsatisfactory examples to explain, our algorithm may need a larger validation set.

Thanks for your constructive reviews. We address your comments in a point-to-point manner.

Answer to Weakness 1: We add the following ablation study on the proportion of data used for unlearning. Specifically, for each unsatisfactory response, our method identifies a convex decomposition over a subset of reward training data, assigning a weight to each data in this subset. In our original setup, we selected the top 30% training data with the highest weights as the explanation and unlearn these data. Here we vary this proportion and unlearn the top 0%, 10%, 30%, 50%, 70%, and 100% data. We report the GPT-4 win rates over the SFT model on the test set. Recall that our method improves over base RLHF algorithms and we conduct this ablation for two base RLHF algorithms, PPO and ReMax.

(1) opt-1.3B + full-hh-rlhf

(2) pythia-2.8B + full-hh-rlhf

(3) pythia-2.8B + TL;DR

(4) llama-2-7B + TL;DR

The above results show that the best proportion to unlearn is around 30%-50%. Unlearning too little ($\leq$10%) is not sufficient enough to enhance alignment, while unlearning too much ($\geq$70%) can degrade performance.

Answer to Weakness 2: We agree that a reward model may introduce bias as it cannot 100% capture human preference. To mitigate this, we use human evaluation to calibrate the filtering threshold (discussed in Appendix D.1). Specifically, we randomly sample 50 prompt-response pairs from D and use human evaluation to classify satisfactory and unsatisfactory responses. For each response, we compute the calibrated reward, which is the raw reward of this response output by the reward model subtracting the raw reward of the SFT model response to the same prompt. This calibration reduces the prompt-dependent bias. We then determine the filtering reward threshold as the upper bound of the calibrated rewards of the unsatisfactory group. Responses with calibrated rewards under this threshold are labeled as unsatisfactory. This approach ensures that the filtering threshold is grounded in human judgment.

Answer to Weakness 3: The samples with performance degradation are the samples that are far from the unsatisfactory prompt-response pairs in the representation space. Specifically, let us denote degrading samples as the samples with performance degradation and denote improving samples as the samples with performance improvement. In the experiment of opt-1.3B model and full-hh-rlhf dataset, the average distance between the improving samples and unsatisfactory prompt-response pairs is around 25, while the average distance between the degrading samples and unsatisfactory prompt-response pairs is around 80.
This improvement-degradation is trade-off controllable. In particular, our policy fine-tuning objective (5) has a KL term to restrict the divergence of the new policy from the original RLHF policy on the satisfactory responses. The coefficient of the KL divergence is $\bar{\beta}$. A larger $\bar{\beta}$ will result in less degradation on the satisfactory responses but also less improvement on the unsatisfactory responses. We conduct an ablation on this coefficient $\bar{\beta}$ where we use opt-1.3B model, full-hh-rlhf dataset, and ReMax base algorithm. We vary the value of $\bar{\beta}$: 0, 0.05, 0.1, 0.5, 1.0, and report GPT win rates over ReMax on both the unsatisfactory set $D_u$ and satisfactory set $D\setminus D_u$.

The above results show that a larger $\bar{\beta}$ results in less degradation on the satisfactory responses but also less improvement on the unsatisfactory responses.

Answer to Weakness 4: We evaluate our models on ApacaEval, specifically using the "tatsu-lab/alpaca_eval" dataset. For each prompt (i.e., "instruction") in this dataset, we use our models to generate responses. Recall that we have four experiments: opt-1.3B+full-hh-rlhf, pythia-2.8B+full-hh-rlhf, pythia-2.8B+TL;DR, llama-2-7B+TL;DR, and we consider two base RLHF algorithms: PPO and ReMax. We report the GPT-4 win rates of RLHF (PPO and ReMax) and our method (XRLHF) over the SFT model:

The above results demonstrate that our method (XRLHF) can also improve RLHF on AlpacaEval.

Answer to Weakness 5: In our case, we identify a set of training data to explain the unsatisfactory samples. Here, we provide analysis on the chosen and rejected responses of the identified training data. In particular, we use Beaver-7b-v3.0-reward for the full-hh-rlhf dataset and use Deberta-v3-large-v2 for the TL;DR dataset. We provide the average reward of chosen and rejected responses of identified training data in our four experiments (opt+full-hh-rlhf, pythia+full-hh-rlhf, pythia+TL;DR, llama+TL;DR) where the base RL algorithm is ReMax.

The above results show that the average reward of rejected responses is very close to chosen responses for identified training data in the full-hh-rlhf dataset, and is even larger than the chosen responses for identified data in the TL;DR dataset. To further inspect the reward distribution of the chosen and rejected responses of the identified training data, we provide the percentage of the identified training data where the rejected response has a higher reward than the chosen response:

The above results indicate that the identified training data include a large amount of misleading signal where the rejected response is actually better than the chosen response.

Answer to Question 1: Please refer to our answer to Weakness 3.

Answer to Question 2: Please refer to our answer to Weakness 4 where we evaluate our models on AlpacaEval. We conduct an additional experiment using Llama-3.1-8B model and full-hh-rlhf dataset for training, and we provide GPT-4 win rates over the SFT model on the alpaca_eval dataset.

The above results demonstrate that XRLHF can improve RLHF using Llama-3.1-8B base model.

Answer to Question 3: Please refer to our answer to Weakness 5.

Answer to Question 4: Our method includes two parts: explanation and improvement. Algorithm 1 (XRLHF) only includes the explanation and thus do not include the improvement part (i.e., unlearning and finetuning). We will add an overall algorithm that includes both parts.

I will carefully read the reviewers’ responses and reply to them today. I’ve noticed that the responses are quite extensive.

---

Update: I appreciate the author's efforts in providing more experiments and more discussion on my concerns. I will revise my scores, conditioned on the authors adding these experimental results and discussion to the final version of the paper.