Removing Length Bias in RLHF is not Enough

Thank you for recognizing the impact of our research direction.

W1

As your suggestion, we have compared our method with other baselines on MT-bench. The results have been exhibited in the following.

From the results, we can find that our developed LPBC method still ourperforms other baselines on the evaluation of MT-bench.

W2&Q4

Thanks for your suggestion, we have reported error bars in Tables 2 and Fig. 4.

W3

Sorry for the inconvenience caused in reading. We will improve the writing quality to ensure better understanding.

The brief statement of the motivation is that the prompt-template bias learned by RM will result in LLMs preferring to generate responses in a specific format after RLHF fine-tuning, regardless of the format requested in the prompt. Thus, we develop a method to estimate the prompt-template bias so that we can remove it in the following RLHF process.

W4&Q1

Thank you for giving us the opportunity to clarify the novelty of our work.

The main contribution of our work is revealing the cause of prompt-template bias in reward modeling through theoretical analysis in Section 3.1 and propose PBC method to address this issue in Section 3.2.

In Section 3.3, we propose a novel method to simultaneously estimate prompt-template bias with PBC method and length bias with existing method, e.g. ODIN. We did not simply stack these two methods, that is, we did not just remove the prompt-template bias on an RM that had already been adjusted for length bias (ODIN+PBC in our paper). We chose a more refined modeling approach by decomposing the prompt-template bias into quality and length components for separate estimation, as shown in Eq.(14) (LPBC in our paper). Note that we have inlcuded the comparison beween ODIN+PBC and LPBC in our experiments.

For novelty in the aspect of technical, we admit the bias estimation and combination method is a little straightforward, but the benefit is that it won’t bring too much computation burdens and can be easily deployed on the original implementation of RLHF, which is particularly important given the significant computational resources required for LLM training today.

Q2

As mentioned in W3, our work mainly focuses on addressing the issue of prompt-template bias. As shown in Fig.3, the reward distributions on different categories show that the RM trained with vanilla preference loss tend to assign higher reward scores on the response in a specific format, e.g. tech article.

The quantitative experimental results shown in Table 2 show that our developed PBC method for removing prompt-template bias can lead to significant performance improvements compared to the original implementation of RLHF.

Q3

The prompt we used for GPT-4 evaluation is listed below, following the same experimental settings as in ODIN for a fair comparison.

"[System Prompt] You are a helpful and precise assistant for checking the quality of the answers.

[User Prompt] {prompt}

[The Start of Assistant1’s Answer] {response_a} [The End of Assistant1’s Answer]

[The Start of Assistant2’s Answer] {response_b} [The End of Assistant2’s Answer]

We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment."

We wish the reivewer prioritize RLHF techniques that are truly practical and implementable, rather than those that may appear fancy but remain confined to papers. Our work is driven by real-world problems, identified the underlying causes through theoretical analysis, and effectively addressed the issue in practice.

Dear reviewer

Sorry to bother you, as the discussion period is nearing its end, we hope that our response has adequately addressed your concerns regarding our paper.

If not, we kindly ask you to list your remaining concerns, so that we can improve the quality of our paper in the next round.

Your comments on our paper will be extremely important to us. Thanks.

Best wishes

Thank you for recognizing the impact of our research direction.

W1

In tems of notation, we assume it is correct to perform the averaging operation on the variable $y$ rather than the function symbol $C$.

W2

Thanks for your suggestion. We admit that Fig.1 is used to illustrate the difference between our method and vanilla RM training, where the word highlighted in red color indicates the template request in the prompt.

The statement of the problem is that the prompt-template bias learned by RM will result in LLMs preferring to generate responses in a specific format after RLHF fine-tuning, regardless of the format requested in the prompt. Thus, we develop a method to estimate the prompt-template bias so that we can remove it in the following RLHF process.

The cause of prompt-template bias is that the datasets for RM training usually only collect responses that satisfy the template/format requests in the prompt. Because it will be time-consuming and expensive to construct responses in various formats for each prompt in practice.

Q1

As stated in W2, the main cause of prompt-template bias issue is that the datasets for RM training usually only collect responses that satisfy the template/format request in the prompt. Our method developed in this paper can allevaite this issue.

The comparison in Eq.(6) is used to illustate that there is a chance that $C(x_a, \overline{y}_a)>>C(x_b, \overline{y}_b)$ will lead to $C(x_b, y_{ab}) > C(x_b, y_b)$, leading to the issue in Eq.(7).

Q2

Actually, we have compares the accuracy of various RMs on Fig.5(a). From the results, we can find that the constraint terms introduced by our method won't significantly affect the RM's accuracy.

Moreover, we have included a comparison of other quantitative performance metrics on the RM benchmark, which is likely the most popular benchmark for evaluating trained RMs.

Based on the RM benchmark comparison, our method (LPBC) does not significantly impact RM performance and even enhances it in certain areas, such as Chat and Chat Hard, where the focus is on the quality of generated responses.

Q3

Due to the page limitation, we have attached showcases of reward hacking in the pdf file, sorry for the inconvenience.

Q4

Thanks. Actually, Fig.5 mainly aims to illustrate that the performance of our method is not sensitive to the selection of hyperparameters, as there is no clear trend in the heat maps, as you mentioned.

As you suggested, we also report the standard deviation of each evaluation in the table below

Q5

Thanks for your suggestion, we have also included error bars in Table.2.

Dear reviewer

Sorry to bother you, as the discussion period is nearing its end, we hope that our response has adequately addressed your concerns regarding our paper.

If not, we kindly ask you to list your remaining concerns, so that we can improve the quality of our paper in the next round.

Your comments on our paper will be extremely important to us. Thanks.

Best wishes

First of all, we greatly appreciate your responsible review of our theoretical analysis on the issue of “prompt-template bias” and also thanks for acknowledging the performance of our method. We assume that our greatest disagreement lies in the theoretical analysis part, so we will try to address your concerns one by one in the following.

Q1

The training of reward modeling is not related to RL, nor is it related to the Bellman process. $C(x_a, \overline{y}_a)$ and $C(x_b, \overline{y}_b)$ exhibits gaps due to the absence of constraints addressing prompt-template bias in the original preference loss, typically the value of $C(x_a, \overline{y}_a)$ won't affect the preference order within the set of prompt-response pairs with the same prompt $x_a$ and response format $\overline{y}_a$.

We assume that you argee with the point that $C(x_a, \overline{y}_a)$ and $C(x_b, \overline{y}_b)$ will exhibit some gaps, but believe it should be within a reasonable range.

However, it is very difficult to define what is a reasonable range in practice. From the experimental results shown in Fig.3(a), the RM trained with original preference loss does assign higher scores on responses formatted as Tech Article, leading to the misordering of prompt-response pairs exhibited in Table. 1. As long as this gap causes the RM to assign higher reward scores to certain marginal samples (possibly OOD samples), like $r_\theta(x_b,y_{ab}) > r_\theta(x_b, y_b)$ in Eq.(7), it will guide the LLM to generate responses in a specific format after RLHF fine-tuning.

Q2.1

Thank you for your insightful comments on Eq.(7). However, we believe there may be significant misunderstandings regarding the assumptions underlying Eq.(7).

To assumption 1, we only claim $r_{\theta'}(x_b, y_{ab}) \approx r_{\theta'}(x_b, y_b)$ in our paper rather than strictly constraining $r_{\theta'}(x_b, y_{ab}) = r_{\theta'}(x_b, y_{b})$. We absolutely agree with your comment that the reward function should depend on the whole response $y_{ab}$ rather than only its template $\overline{y}_b$.

It is also the reason why we only assume $r_{\theta'}(x_b, y_{ab}) \approx r_{\theta'}(x_b, y_b)$ because it can be possibly achieved on the reward distribuion $r_{\theta'}(x,y)$ learned to approximate "gold standard" reward model $r_{\theta*}(x,y)$

To assumption 2, we didn't assume $C(x_b, \overline{y}_{a}) \approx C(x_a, \overline{y}_a)$, but only demonstrate there is a chance that

$C(x_a, \overline{y}_a) >> C(x_b, \overline{y}_b)$ leads to $C(x_b, y_ab) > C(x_b, y_b)$ .

Please notice the difference. $C(x_a, \overline{y}_a) >> C(x_b, \overline{y}_b)$ is achievable because the original preference loss does not impose any constraints on prompt-template bias and this gap can cause trained RM to prefer either prompt $x_a$ over $x_b$ or format $\overline{y}_a$ over $\overline{y}_b$.

As long as there is a tendency that RM prefers format $\overline{y}_a$ over $\overline{y}_b$, we can have

$C(x_b, y_{ab}) > C(x_b, y_b)$ to lead to $r_\theta(x_b, y_{ab}) > r_\theta(x_b, y_b)$ in Eq.(7)

Q2.2

Firstly, there are two templates in your comment: 1) template request in the prompt, denoted as $x_1$; 2) response templates, denoted as $\overline{y}$. In our understanding, your question is why the generated response $y$ will be unrelated to the template request $x_1$ during RM training.

Actually, it is the shortcoming of the original preference loss that we want to highlight. Assume there is a set of prompt-response pairs with the sample prompt $x_a$ and responses in the same template $\overline{y}_a$. Notably, during training RM with the preference loss, these characteristics (or elements) shared across all prompt-responses pairs won't affect the preference order, e.g. prompt $x_a$, response template $\overline{y}_a$ and the correlation between $x_a$ and $\overline{y}_a$. The most effective way to address this issue is to construct responses in various formats for the prompt $x_a$, helping the reward model to distinguish whether the response template meet the requirement of the prompt $x_a$. However, constructing responses in various formats for each prompt will be time-consuming and expensive in practice.

For OOD sample, it seems that we have reached a consensus on this point that reward estimation should exhibit high bias on OOD samples, no matter it is $x_{ab}$ or $y_{ab}$. So, we wonder if you agree with the fact that Eq.(7) can be probably achieved if $(x_b, y_{ab})$ is an OOD sample?

Summary

At last, we thank you for your effort in improving the quality of our paper, even if it is a rejected score. This issue actually stems from the process of deploying RLHF technology in our text creation product, so there is no doubt that it will occur in practice (even if we used nearly 190k human-annotated preference pairs for RM training). In this paper, we aim to analyze and explain this phenomenon, and propose a solution that can be practically deployed in the product, rather than a flashy but impractical algorithm.

Thank you for your responsible comments and your willingness to discuss with us.

First, please allow us to emphasize the consensus we have reached:

1. There is indeed hacking of the current RM through specific response formats, which causes the LLM after RLHF finetuning to tend to generate responses in a specific format (we have indeed encountered this issue in product implementation).

2. The theoretical analysis in this paper is only intended to confirm that prompt-template bias in some cases of preference pairs may also lead to reward hacking, but not all preference pairs will result in this.

3. As long as the prompt-template bias in certain preference pairs leads to reward hacking, the LLM aligned with this reward model may overlook the template requirements in prompts and tend to generate responses in a few specific formats.

Q1

Thanks for your patience in carefully reading our response. We are pleased to see that we have reached a consensus that there is a possibility of achieving $C(x_a, \overline{y}_a) > C(x_b, \overline{y}_b)$ in certain prompt-response pairs.

The reason why we strictly assume $C(x_a, \overline{y}_a) >> C(x_b, \overline{y}_b)$ is because we wish the prompt-template bias term to play a dominant role in the comparison of prompt-response pairs from different sets, leading to

$r_\theta(x_a, y_a) >> r_\theta(x_b, y_b)$ where

$r_\theta(x_a, y_a) = r_{\theta'}(x_a, y_a) + C(x_a, \overline{y}_a)$,

$r_\theta(x_b, y_b) = r_{\theta'}(x_b, y_b) + C(x_b, \overline{y}_b)$

We agree with your statement that $C(x_a, \overline{y}_a) >> C(x_b, \overline{y}_b)$ is a stronger condition, but you cannot deny that there are prompt-response pairs that satisfy this condition, which could lead to reward hacking.

Assumption 1

Sorry for not explaining it clearly in the first response. This time, we will provide an explanation based on the examples you provided.

Given a response $y_{ab}$ where $a$ indicates the template of code language and $b$ denots the theme on neural networks, the trained reward model will assign a reward score to the concatenation of the prompt $x_b$ and the response $y_{ab}$ as $r_\theta(x_b, y_{ab}) = r_{\theta'}(x_b, y_{ab}) + C(x_b, \overline{y}_a)$.

We fully agree with your comment that the reward function $r_\theta(x_b, y_{ab})$ should depend on the entire response $y_{ab}$, not just on elements $a$ or $b$ individually. However, please note that we have split $r_\theta(x_b, y_{ab})$ into two components:

1. $C(x_b, \overline{y}_a)$, which models the RM's scoring of a prompt $x_b$ when the response follows format $a$,

2. $r_{\theta’}(x_b, y_{ab})$, which is intended to model the remaining part of the reward score, such as whether the response’s theme satisfies with the prompt's request.

Thus, we can assume $r_{\theta'}(x_b, y_{ab}) \approx r_{\theta'}(x_b, y_b)$ because both $y_{ab}$ and $y_b$ follow the same theme $b$ after discarding the impact of response template. For a response $y_{ab}$ that heavily depends on $a$, its reward score will be dominated by the term $C(x_b, \overline{y}_a)$.

Moreover, as stated in our reply to Q.1, we assume the prompt-template bias term to play a dominant role in the comparsion, and thus the gap between $r_{\theta}'(x_b, y_{ab})$ and $r_{\theta}'(x_b, y_b)$ will be relative small when compared to the gap between $C(x_b, \overline{y}_a)$ and $C(x_b, \overline{y}_b)$.

Assumption 2

Actually, there is no need to assume $C(x_b, \overline{y}_a) \approx C(x_a, \overline{y}_a)$.

Given $C(x_a, \overline{y}_a) >> C(x_b, \overline{y}_b)$, we may have $C(x_a, \overline{y}_a) >>/> C(x_b, \overline{y}_a) > C(x_b, \overline{y}_b)$.

This can be achieved when there is a tendency that RM perfers format $\overline{y}_a$ over $\overline{y}_b$,

we can have $C(x_b, \overline{y}_a) > C(x_b, \overline{y}_b)$ to lead to

$r_\theta(x_b, y_{ab}) > r_\theta(x_b, y_b)$ in Eq.(7)

Q2.2

The reason for the prompt-template bias lies in what you mentioned: the responses collected for each prompt are too closely aligned with the prompt's requirements. As a result, the RM trained these preference pairs has not encountered responses with different templates and cannot distinguish whether a response's template meets the requirement in the prompt.

Yeah, we are trying to explain why an OOD sample, such as $(x_b, y_{ab})$, might receive an overly high bias estimate, e.g. Eq.(7).

Additional Comments

We still wish our responsible reviewer to prioritize RLHF techniques that are truly practical and implementable, rather than those that may appear fancy but remain confined to papers. Our work is driven by real-world problems, identified the underlying causes through theoretical analysis, and effectively addressed the issue in practice. Moreover, this technique has been proven effective in our industry and hope you can provide a fair judgment.

Thank you for recognizing our work. We believe that you are a reviewer with genuine experience in implementing RLHF and are well aware of the current shortcomings of RLHF.

W1

Thanks for your suggestion. The orginal title aims to emphasize that existing RLHF research mainly focuses only on length bias, while overlooking other potential biases, e.g. prompt-template bias in our paper. We also argee with your point and will try to come up with a new title for our article.

Q1

Thanks for your insightful questions.

Actually, we have evaluated our method on an LLM with 13B parameters, which we pretrained ourselves and intend to deploy in text creation products, and it consistently outperforms the original RLHF pipeline. Notably, the evaluation metrics on the industrial side are more stringent, but our developed method still performs well.

For generalization, the developed PBC is not limited to removing template bias in responses, but can address any characteristics shared among responses to a specific prompt, e.g. the language of the response (english or chinese). We constructed the concept of prompt-template bias in the text creation scenario purely for the ease of understanding. In real-world applications, the situation is more complex, but this does not undermine the effectiveness of our method, as it remains challenging to achieve diversity in responses to each prompt.

Overall, we have great respect for you because you are a researcher who genuinely focuses on the practical implementation of RLHF, regardless of whether this paper is accepted or not.

Dear reviewer

Sorry to bother you, as the discussion period is nearing its end, we hope that our response has adequately addressed your concerns regarding our paper.

If not, we kindly ask you to list your remaining concerns, so that we can improve the quality of our paper in the next round.

Your comments on our paper will be extremely important to us. Thanks.

Best wishes