A Long Way to Go: Investigating Length Correlations in RLHF

Thanks for the thoughtful review! We will add discussion of those citations you mention in any future version.

Thanks for the thoughtful comments! We’ll try to address concerns here.

more capable models

We conducted additional experiments with scaling up our RM experiments up to LLaMA-2 13B. We report intrinsic reward modeling accuracy results below (compare with Table 4 in the paper):

On these datasets, reward modeling accuracy is only marginally better than it was before, suggesting that increasing model scale isn’t necessarily the main bottleneck on these tasks. While the settings are a bit different, we will definitely consider extending our study to evaluate other models in future work.

length-unrelated tasks such as math, coding, length-constrained instructions…

This is a great point. It’s worth noting that we actually already have such a setting (the Stack setting) which we chose because it represents a realistic setting where math, coding and more complex reasoning may be used. Indeed, we do note this setting to demonstrate most of the discussed length-correlated behavior throughout the RLHF pipeline, though to a lesser degree.

Would the length bias issue correlate with tasks?” “In my opinion, before intervening in the RLHF process to mitigate the lengthy outputs, it is more important to recognize whether human annotators and LLM judges commonly prefer lengthy outputs

We agree with this point and discuss it in the paper (introduction, 3.1): length may be a valid feature corresponding to quality for humans. Two of the datasets we run experiments on, Stack and WebGPT, use human preference labels and these labels do show length biases (Table 5). However, these are fairly small, suggesting that length may not be a dominant feature for human annotation. These small length biases are not proportional to the dominance of length we find experimentally after RLHF on these same settings, which is particularly concerning given how commonplace these techniques / evaluations are. Regardless of whether length is a good or bad feature, we find it concerning if it comes at the cost of other potential good features.

Hi WB4w, can you check the authors' response and update your review if it addressed your concern (or participate in discussion with the authors if it did not)?

Thanks for the thoughtful comments! We address the main concerns here:

larger models

We conducted additional experiments with scaling up our RM experiments up to LLaMA-2 13B. We report intrinsic reward modeling accuracy results below (compare with Table 4 in the paper):

On these datasets, reward modeling accuracy is only marginally better than it was before, suggesting that increasing model scale isn’t necessarily the main bottleneck on these tasks.

We will add these experiments in any future version.

The chosen evaluation datasets are limited and more results on general benchmarks like MT-bench or AlpacaEval 2.0 can make the results more convincing

This is a fair point. We aim to cover a fairly diverse set of realistic tasks (multi-turn dialogue QA, single-turn long-form QA, and technical question answering) to demonstrate our findings on publicly available preference datasets at the time the work was conducted. We do note that the pairwise preference metric we report (setting specific) is based on GPT-4 evaluation which is used on MT-Bench and alpaca-eval and thus should give a reasonable approximation to what one may find in those settings, the main difference being the prompt set.

As for MMLU, we think this is a nice suggestion, but the fine-tuning recipes (datasets used, SFT, etc.) necessary to induce good chain-of-thought-based question answering are a bit different than the recipes for the kind of LLM alignment we study here.

Hi Bn2p, can you check the authors' response and update your review if it addressed your concern (or participate in discussion with the authors if it did not)?

Thanks for the thoughtful comments!

the last sentence of 2.1…

Thanks for pointing this out! We mainly mentioned this to qualify the win-rate preference results, since these may themselves have an undesired length-bias (e.g. STACK on SFT-LONG in Table 2), which has been supported in other recent work as well [1]. We will clarify this more in future versions.

[1] Yann Dubois, Balazs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. LengthCorrected AlpacaEval: A Simple Debiasing of Automatic Evaluators. https://github. com/tatsu-lab/alpaca_eval, 2024.

Thanks for the thoughtful comments! We’ll try to address the questions below:

I enjoy reading this paper but the only confusing section for me is Sec 3.1… most intuitive way is to just calculate ∆𝑅/∆𝐿

This is a great question! We’ll try to clarify here.

Intuitively, we wanted to compute reward increase $\Delta R$ when $\Delta L$ = 0. Because these cases are quite rare, our approximation of this is to compute the $\Delta R$ within each length-stratified bucket, as $\Delta L$ within buckets is quite small. We take an average of this across buckets, weighted by the number of instances (SFT + PPO combined) in each bucket.

We like the suggestion of reporting $\Delta R$/$\Delta L$ or correlation between $\Delta R$ and $\Delta L$! One slight limitation of $\Delta R$/$\Delta L$ is that it changes the units and is therefore not comparable to $\Delta R$ in the table. Furthermore, if we computed $\Delta R$/$\Delta L$ (assuming you meant $\sum_i (\Delta R) / \sum_i (\Delta L)$), it could be the case that length increases and reward increases, but these are unconnected. Controlling for length with our approach helps disentangle this. Note that we already report corr(𝑅, 𝐿) within a batch in Table 4.

For experiment setup in Section 3.1, I wonder is sft output or ppo output you would stratify based on the length, or combined?

We stratify based on length here, where we compare all the outputs from SFT that are in some length range (e.g. 20-40 tokens) with all the PPO outputs in the same length range (20-40 tokens). Note that the overall prompt set is the same, but each bin may not have comparable prompts between SFT and PPO.

For Table 2, Could you please win/tie/lose rate in addition to win rate only? I believe this could strengthen the performance comparison between models.

Our prompts (same as AlpacaFarm) ask GPT-4 to return one model name as output, corresponding to the better output. Because of this we rarely observe ties, thus the lose rate is just the reverse of win-rate.

For Table 3, why std ppo has failed reward optimization? As it has length and reward, I think it should be accessed by sim perf also?

Thanks for pointing this out! Standard PPO doesn’t have failed reward optimization; we used a - in the sim pref column for it because it is the reference point. We can replace those numbers with 50% for better clarity.