Reliability-Aware Preference Learning for LLM Reward Models

This work aims to make preference learning more robust to noise in human preference labeling, specifically focussing on length bias.

As a first step, the authors collect a dataset called LIE (Length Incentivized Evaluations) that contains queries with 4 types of answers: (1) Short Correct, (2) Short Incorrect, (3) Long Correct, and (4) Long Incorrect. Next, 20 annotators are asked to provide preferences for answer pairs in the dataset, and it is confirmed that annotators prefer longer answers over correct answers (as found in earlier work as well). Then the authors train a standard reward model on the collected preferences, which has the same length bias.

Finally, the authors try to mitigate the bias by “reliability aware preference learning” (RAPL). The idea is to either adjust the reward or the bolzmann probability based on the difficulty of the questions. The authors experiment with 2 approaches to set the values: (1) Annotator self-reported confidence, and (2) an LLM-based autograder. Calibration is done through another collected dataset (TRUE), that contains annotator answers for questions with known answers.

Based on the experiments, the effectiveness of either method is somewhat unclear. Adjusting the bolzmann probability based on models fine-tuned on the TRUE dataset seems to work best, although the authors mention that “this strategy doesn’t work for just any value as the model trained using the average confidence value doesn’t perform too well” (line 511-512).

This paper argues that a source of error in current reward models is that the collected preference judgments used to train them can reflect biases in annotator preferences during data collection. To verify this hypothesis, the authors collect a dataset (based on TruthfulQA), Length Incentivized Evaluation (LIE), that consists of prompts paired with 4 responses that are {short + (factually) correct, short + incorrect, long + correct, long + incorrect} and train a reward model ($R_{LIE}$) on collected human preferences on examples where the length and correctness are negatively correlated. They then evaluate the reward scores assigned by $R_{LIE}$ (and another reward model trained on a held-out dataset HelpSteer2, $R_{HS2}$), on each of the 4 kinds of responses. They observe that both $R_{LIE}$ and $R_{HS2}$ assign higher scores to the longer responses, regardless of correctness due to the unreliability of the human annotations (both collected judgments for this paper, as well as those in help-steer).

In order to mitigate this issue, the authors propose to augment the traditional learning objective for reward models with a penalty for the annotator's reliability on that particular example. This is incorporated either with the $\beta$ parameter or as a corrective regularization term in the contemporary loss function.

In order to obtain reliability scores for examples, the authors experiment with three different ways: (1) self-reported annotator confidence, (2) a synthetic LLM-assigned grade with CoT prompting, (3) a learned classifier score that involves (a) collecting a dataset of 1000 new preference judgments on examples with a correct answer from various benchmarks - MMLU, BigBench, etc. (b) fitting a transformer-based classifier on a binary label {0/1} based on whether the annotator selected the correct label (c) the probability assigned by this classifier is used as a reliability score for inference.

Empirical results show that incorporating reliability into model training improves reward model predictions in some cases, with uneven trends between the particular ways of assigning reliability scores and incorporation methods.

The work aims to improve the reliability of reward modeling for better model alignment. First, they construct the LIE dataset where incorrect responses tend to be longer. The initial experiment shows that a reward model can be successfully misguided to favor longer incorrect responses. Based on this observation, they propose two methods, Reward Adjustment and Probability Adjustment, to account for the unreliability of preference for reward modeling. Both methods work by deemphasizing unreliable preference pairs in the training loss. The unreliability is estimated by either human or model-generated scores. They also construct the TRUE dataset to further calibrate the reliability measures. Experimental results show that Probability Adjustment with calibrated scores yields the best reward model on both their LIE dataset and in-the-wild dataset HelpSteer2.

This paper tackles the problem of learning from unreliable human annotations in RLHF. Specifically, it is assumed that there is some chance that the human label is incorrect / has low confidence. The reliability is then incorporated into the reward learning objective through 1) temperature scaling of the reward and 2) interpolation with a random guessing distribution. To evaluate the proposed approach, a new dataset is also built to measure reliance on length when evaluating answer correctness.

I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.