Post-hoc Reward Calibration: A Case Study on Length Bias

Thanks for your comments and your questions! Please see our pointwise responses to see if we could address your concerns.

Assumptions: The proposed method depends heavily on several assumptions, such as (1) decomposition of the biased reward, (2) independence of the bias characteristic, (3) sufficient density, and (4) Lipschitz continuity. These assumptions may not hold in all scenarios.

Yes, we agree that these assumptions may not hold in all scenarios. However, we have verified the effectiveness of our method across various reward models and scenarios where these assumptions do not perfectly hold. For example, in RewardBench, simply selecting the shorter responses would yield 60% accuracy; thus, the ideal correlation between reward and length should be slightly negative. On the contrary, in the RM-as-LLM-evaluator setting, the ideal correlation should be positive, according to the ChatbotArena.

Furthermore, these assumptions are not that difficult to deal with/satisfy in practice.

1. For the independence assumption, we can introduce $\gamma$ to control the calibration effect (please see the common concern in General Comment and line 1029 - 1049 in our paper).
2. Secondly, the sufficient density assumption is proposed because we may require sufficient data points for estimation: in the appendix, we perform a stress test to measure how much data we need for estimation (Fig. 12-14) and demonstrate that our proposed method can provide stable calibration even with only a few hundreds of data points.
3. Thirdly, the Lipschitz continuity just assumes that the reward value is a slow-varying function of $c$, i.e., it will not change dramatically with $c$. For example, for calibrating length bias, the function is not even continuous because we define length as the character length of the output. Still, the introduced method is effective regarding length bias.

We will add more discussions to clarify the essence of these assumptions.

Difficulty in determining d: In some scenarios, determining dd is difficult. While LWR can estimate Equation 5, linear approximation may often be inaccurate, necessitating further analysis.

Yes, determining d in equation 6 is challenging in practice, as explained in our submission (lines 192-198). Therefore, we introduce LWR to estimate the equation 5. For LWR, the only hyper-parameter is the bandwidth $f$ that determines the portion of the dataset used for locally weighted regression. Working with a local neighbourhood, LWR is broadly considered a good smoothing method for functions with unknown shapes. Moreover, our analysis in the appendix reveals that the proposed method is robust to different bandwidths $f$ of LWR.

Demonstrations Needed: Additional examples should be provided to showcase the method's effectiveness in various situations.

First, we want to emphasise that the effectiveness of our proposed method is demonstrated across three different settings, covering dozens of reward models (RewardBench setting), hundreds of LLMs (LLM evaluator setting) , and various LLM-RM combinations.

Then, we provide extra figures illustrating how the distribution of reward w.r.t. length changes before and after calibration to offer more insight into the reward calibration's result. The supplementary figures are in the appendix of our revised paper (Fig. 17--26).

Bias and Spurious Correlation: How does the characteristic bias lead to spurious correlations? Examples would facilitate a better understanding for the reader.

The spurious correlation in our submission indicates that the reward model, trained on the human preference dataset, may overly rely on some shortcuts / superficial features (e.g., the length of the output) to determine the preference.

Hyperparameter Selection: What guidelines are there for selecting hyperparameters in different scenarios, such as $\beta_0$ and $beta\1$?

The only hyper-parameter for the proposed RC-LWR method is bandwidth $f$, which determines the portion of the dataset we used for regression given a point of interest. The bandwidth was set as the default as in the statemodel.api implementation for all reward models and tasks. We also study the sensitivity of the proposed method to the bandwidth choice (Fig. 7) and verify that the calibration improvement is robust to the different bandwidths. We are sorry that algorithm 1 in the appendix may be misleading. $\beta_1$ and $\beta_0$ are not hyper-parameters but are fitted slope and intercept of the LWR,

Fig. 5 x-axis: What does the x-axis represent in Figure 5? Providing clarity would help readers interpret the results more effectively.

As mentioned in the caption of Fig. 5, the x-axis shows the original performance for different reward models on the reward bench, and the y-axis shows the calibrated performance. Thanks for your suggestion. We have revised the caption for clarity.

There are several methods that can mitigate length bias, particularly in DPO. Have you considered simple baselines such as simPO?

As discussed in the related work section in our submission, existing works that mitigate (length) bias are mainly from the perspective of (a) data handling, (b) reward model engineering, and (c) alignment algorithms. (a) and (b) usually require extra data collection and reward model training and is not post-hoc. And (c) is specifically designed for alignment procedure, thus having limited application scenarios. For example, using the length regularization term in SimPO for the RM-as-LLM-evaluator setting is not straightforward.

Regarding comparison with SimPO, our method achieves comparable results to simPO while using a weaker reward model.

Specifically, we use the same LLM (Meta-Llama-3-8B-Instruct and gemma2-9b-it), the same prompt dataset (Ultrafeedback), and the same sampled responses from the model. The only difference is that we use weaker reward models (FsfairX-LLaMA3-RM-v0.1 with 84.4 on rewardbench, GRM-llama3-8B-sftreg with 87.0 on reward bench) to annotate the preference, and simPO employs a stronger one (ArmoRM-Llama3-8B-v0.1, 90.4 on RewardBench). Detailed results are shown in the Table below:

We hope our responses could address your questions. Please let us know if you have further concerns or want to discuss more!

Simple baselines: The baselines used are relatively simple. Comparisons with more complex baselines, such as reward model engineering in reference, are needed.

To the best of our knowledge, the length penalty is the only baseline comparable to our method (i.e., it does not require extra data, extra models, or extra training and is applicable in various scenarios). Besides the length penalty baseline, in RMs as LLMs evaluator setting, we also compare with length-controlled regression (LC) proposed by Dubois et al. (2024) and was utilized to produce the AlpacaEval 2.0 leaderboard. LC is specifically designed for the AlpacaEval leaderboard and thus needs instruction_id and model_id for regression. Our method achieves comparable calibration performance without such information with LC on RM and GPT4 judges.

Because in our related work section, we mainly mention relevant works that ensemble reward models to mitigate bias, we supplement a baseline that ensemble rewards from different reward models. Specifically, we select the Top 9 models from the leaderboard as of Aug. and enumerate all possible ensembles between the two models. Their ensemble results are shown in the following Table. The cell $i,j$ denotes the performance by ensembling the model_i and model_j, where i and j are their ranks on the RewardBench leaderboard.

The final row shows the performance after our method's calibration.

The calibration surpasses most ensemble results except for ensembles with a much more robust model (The ensemble results that outperform the calibration method are bold, primarily the results that ensemble with the top-1 model). Intuitively, ensemble-based methods may not address common biases across models. For example, if these models overly prefer lengthy outputs, their ensemble model will also prefer lengthy outputs.

We have also supplemented this in our revised paper (lines 1222 - 1231).

Bias Exploration: It would be beneficial to explore other biases beyond 'length' and 'style', given that significant improvements are mostly seen in 'length bias'.

Other Types of Bias: Are there experiments demonstrating bias beyond length and style?

So far, we have mainly explored the two widespread biases of RMs: the length and style of the output. We respectfully believe that addressing length bias in various scenarios is already an essential contribution to the LLM community since length bias is considered one of the most prevalent biases in LLM evaluation, RM training, and RLHF. This phenomenon arises from the confluence of multiple contributing factors in the output. For example, the preference for the model's use of listing, markdown style, URL, or complex tokens can all contribute to length bias. Thus, calibrating the length bias can simultaneously mitigate other more basic biases to some extent.

Existing works focusing on mitigating length bias often require extra data [1], models [2], and training [3]. Or they are limited in their focus on the alignment algorithms [4]. Compared to them, our proposed method can be seamlessly applied to different scenarios and is very efficient regarding computation and data.

[1] Offsetbias: Leveraging debiased data for tuning evaluators

[2] Reward model ensembles help mitigate overoptimization. ICLR 2024

[3] Safe RLHF: safe reinforcement learning from human feedback. ICLR 2024

[4] Disentangling length from quality in direct preference optimization. ACL 2024

Dear reviewer fLWP,

We have posted some experiment results (e.g., a comparison with the RM engineering method) and clarifications to answer your questions. We have also updated our paper according to your suggestions. We would appreciate it if you could let us know whether our responses properly address your concerns.

Look forward to your reply.

Best regards,

Authors

Thanks for recommending our paper! Your comments are highly aligned with our paper. We have adjusted our paper in this revision according to your suggestions. We hope the following comments can answer your questions.

A slight weakness is the majority focus on length bias. The experiments pretty much all focus on length bias, with one generalisation task to markdown features.

Thanks for the comments. We would still like to highlight that length bias is one of the most prevalent biases for LLM evaluation, RM training, and RLHF. The phenomenon arises from the confluence of multiple contributing factors in the output. For example, the model's use of listing, markdown style, URL, or complex tokens can all correlate to length bias.

Compared with existing works focusing on length bias, our method can also effectively address the length bias issue without extra training or data and be readily applicable to a wide range of scenarios. We extend the proposed method to the md feature to demonstrate its generality. And robustly extending it to other features is one critical part of our future work!

Another slight weakness is that only the combination of length penalty + RC-LWR was tested. A more interesting direction would be to test a combination of two biases and determine whether you can apply the method multiple times to remove multiple biases.

Thanks for your suggestion! We provided the relevant results in our submission (Fig. 11). Specifically, we extend the proposed method to calibrate more than one characteristic by employing multidimensional Locally Weighted Regression, with every dimension representing different biases. However, our results show that simultaneously calibrating two features (length and md features) does not lead to stronger improvements. One primary reason could be that different biases are not independent and may correlate (The same concern also holds when removing different biases sequentially). And modelling their correlation is non-straightforward.

Overall, we agree with your idea to extend the current method to calibrate other/multiple biases. We think the current version, working well for length bias, already contributes well to the LLM community and we would love to explore your proposed directions in our future work!

Thanks for your invaluable suggestions. We hope that our revised paper and our pointwise responses address your concerns.

Presentation

Thanks for your suggestions. The presentation is dense because we tried to evaluate our method comprehensively in at least three scenarios, which requires compressing a lot of information into our paper. To make it more readable, we have paraphrased and revised Section 4.2 of our paper and will further clarify the presentation of our experiment sections.

Lack of insight into the result of reward calibration

Thanks for your valuable suggestion. To provide more insights into the results of our calibration, we supplement figures about reward w.r.t. length before and after calibration in the appendix of our revised paper.

Overall, the Spearman correlation decreases to nearly 0 after RC-LWR calibration. Moreover, the correlation between reward and length is non-linear, making the locally weighted regression a more robust and general form for estimating and removing the bias of interest. After length-penalty calibration, the overall correlation of the entire dataset will decrease, but locally, the bias may be amplified.

The authors do not consider the extent to which the correlation between length and reward may not be entirely spurious—there may actually be some amount of weight that the reward models should place on length.

Please refer to the common concern in the General Comment. We found that introducing a hyper-parameter $\gamma$ to determine how much of the estimated bias term to subtract from the original bias reward could smoothly control the calibration effect. What is more, one can tune this $\alpha$ to achieve better rewarding performance if some prior knowledge about the dataset is known/estimated.

With regards to the second weakness above, it would be helpful if the authors could plot the learned $\hat{r}\_\theta(c)$ function for several reward models to see what kind of effect subtracting it has compared to the linear length bias $\alpha$ c.

Thanks for your suggestion! We have supplemented the relevant figures (Fig. 17 -- 26) in the appendix of our revised paper. Please see the descriptions above and in the revised paper.

Also, it would be helpful to compare to a baseline of a linear length bias where the factor $\alpha$ is tuned per reward model.

Thanks for the suggestion. We supplement an ablation comparison in a setting where hyper-parameters could be tuned for every reward model. Specifically, we select the optimal $\alpha$ for every RM in length-penalty baseline and optimal $\gamma$ for every RM of our method. In this setting, the length-penalty baseline achieves 3.27 performance gain, and RC-LWR achieves 3.67. Please see our response to reviewer PP1Zs' weakness 2 for more ablations about $\alpha$ values.

Robustness to hyper-parameters is a highly desirable property in practice. We want to highlight that length-penalty is sensitive to $\alpha$, and our method is more robust to hyper-parameter choices (Fig. 7 and Fig. 8). And because $\alpha$ is not only reward-model-dependent but is also task-dependent, it is non-trivial to select optimal $\alpha$ per RM and task.

Please let us know what else we can do to help you to recommend our paper. Thanks for your time.

Thanks for your valuable comments and questions! Please see our responses below point by point.

1. Assumption: The assumption of “Independence of the biased characteristic” may be overly restrictive in practical applications, especially given that this study focuses on text length as the biased characteristic. My understanding of this assumption (line 166) is that even when x1 and x2 are fixed at any arbitrary lengths c1 and c2, respectively, the true reward difference between x1 and x2 would be expected to be zero, implying no correlation between reward and text length. However, in practical applications, this assumption will often not hold. For example, in cases where prompts explicitly request brevity or detail, such as “respond concisely” or “provide a detailed explanation,” reward often depends on the response length. This assumption could thus limit the generalizability of the proposed method, especially for tasks where length naturally influences quality. A more flexible approach that adjusts for prompt-specific length expectations could enhance applicability.

Thanks for your question; please see our general comments for the common concern. Basically, we highlight that the proposed method achieves good performance gains across various reward models and settings, where the assumption may not perfectly hold (e.g., for RewardBench, the ideal correlation between reward and length should be slightly negative). Then, we emphasise that a hyper-parameter $\gamma$ can be introduced to control the calibration effect, which helps users preserve or even inject certain biases into the reward model.

We also consider investigating prompt-dependent calibration an exciting research direction (though modelling the correlation between prompt and bias characteristics may require more data and computation, thus introducing a trade-off between efficiency and effectiveness). We would like to explore it in our future work.

Dear reviewer PP1Z,

We have posted some experiment results to answer your questions about Assumption and Baseline Settings. We have also updated our paper to include extra results. We would appreciate it if you could let us know whether our responses address your concerns.

Look forward to your reply.

Best regards,

Authors