Debiasing Online Preference Learning via Preference Feature Preservation

[W1-5] Importantly, the method proposed involves many modeling choices that are not properly ablated, which makes it tricky to know whether all are needed. Some of the following modeling steps are ablated but most are not thoroughly evaluated: (1) Quality of the feature classifier, (2) Performance of the distribution matching step, (3) Synthesizing system prompt from preference features (see positioning wrt related works below), (4) Double system prompt sampling (see Q above), (5) Curriculum learning via temperature scheduling

We clarify that most of the modeling steps mentioned, except for (1) the quality of the feature classifier, have been evaluated in our experiments. Specifically: (2) Performance of the distribution matching step and (3) synthesizing system prompts from preference features are analyzed in Table 2 and Figure 5(b). (4) Double system prompt sampling and (5) curriculum learning via temperature scheduling are evaluated in Table 3 and Figure 5(c).

For (1), we have conducted additional experiments to measure the accuracy of the trained feature classifiers on a separate test dataset. The results show moderate accuracy, which is limited by the small amount of training data and the long-tailed nature of preference features (see Figure 15 in the revised draft). These findings highlight the importance of the proposed distribution preservation step to address the feature classifier’s limitations. We have included these results and their discussion in Appendix F of the revised draft.

\begin{array}{c|ccccc} \hline \text{Metric} & \text{background} & \text{harmlessness} & \text{informativeness} & \text{style} & \text{tone} \newline \hline \text{accuracy} & 0.535 & 0.512 & 0.688 & 0.496 & 0.507 \newline \text{f1} & 0.532 & 0.513 & 0.663 & 0.489 & 0.426 \newline \hline \end{array}

---

[W2-1] Weak positioning wrt. rest of the literature. For example, why not compare to prompt optimization strategies such as OPRO and RLCD?

We clarify that our work is distinct from prompt optimization research like OPRO and RLCD. Prompt optimization aims to maximize output quality by finding the most effective prompts through systematic tuning. In contrast, our approach focuses on ensuring that system prompts reflect specific preference features during response generation or preference labeling, for preserving nuanced human preferences rather than simply maximizing output performance.

---

[W2-2] Since the motivation is to avoid biases, this method requires rigorous evaluation on a task with specific biases... Here we only measure for length bias. Could authors not consider an experimental setting with a clearer bias? Does your approach then accentuate biases present in the preference data?

Thank you for the opportunity to clarify. Beyond addressing length bias, we conducted experiments to examine biases in broader preference features, as shown in Figures 4 and 5. Specifically, we measured the KL divergence between the preference feature distributions assigned to new input prompts and the human preference feature distribution from the seed dataset. An increasing KL divergence across iterations indicates growing biases as the assigned features deviate from the original distribution. The results in Figures 4 and 5 show that the proposed PFP effectively mitigates these biases (i.e., maintaining low KL divergence), while baselines such as iterative DPO and SELFEE fail to do so. This demonstrates PFP’s efficacy in addressing biases across multiple preference feature aspects during online preference learning.

---

[W3] Typo

Thanks for your careful reading. We have corrected these typos in the revised draft.

---

[1] Pitis et al., Improving context-aware preference modeling for language models., arXiv:2407
[2] Lee et al., Aligning to thousands of preferences via system message generalization., NeurIPS 2024

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

Dear Reviewer VhEv,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1-1] On what basis can we assume to distill the full complexity of human preferences down to discrete unsupervised feature dimensions?

Thank you for raising this important point about the complexity of human preferences. We agree that human preferences are multifaceted and not easily reduced to discrete feature dimensions. However, our approach does not aim to capture the entirety of human preference complexity. Instead, we focus on mitigating the unintended biases that arise during online preference learning by preserving the distribution of the key preference features that can be extracted in existing datasets.

The feasibility of extracting and utilizing preference features has been supported by prior research. For example, the approach of deriving preference features from two labeled responses has been validated experimentally [1]. Additionally, studies like Janus [2] have shown that even predefined discrete features can effectively capture and represent a wide range of human preferences when used for supervised fine-tuning. Building on this foundation, we designed our framework to use a more extensive set of predefined features, enhancing its capacity to reflect diverse human preferences. These prior successes underscore the practicality and validity of our method in leveraging discrete feature dimensions to mitigate biases in online preference learning.

---

[W1-2] What is the motivation behind the process of distribution preservation (paragraph starting L254)?

The distribution preservation process is introduced to ensure that the distribution of preference features assigned to a new input prompt set aligns with the human preference feature distribution observed in the initial seed dataset. This is necessary because directly predicting preference features using the trained feature classifier (lines 247-252) can be insufficient due to the classifier’s limitations (see Table 7 of the revised draft), such as being trained on a small dataset and the long-tailed nature of preference features (see Figure 15 of the revised draft).

We also note that the effectiveness of this process is demonstrated in Figure 5(b), where the KL divergence between the human preference feature distribution and the assigned distribution is significantly reduced when using distribution preservation compared to naive predictions. This alignment leads to improved performance, including better alignment scores and reduced length bias, as shown in Table 2.

---

[W1-3] Paragraph starting L173: i am very familiar with this literature but find this explanation confusing. Missing related works here: mention SLiC-HF (Zhao et al., 2023) and online DPO (Calandriello et al., 2023)

We appreciate the constructive comments to improve the clarity of the draft and the suggestion to include additional related work. First, we have improved the paragraph to resolve the confusion in the revised draft (lines 192-196 of the revised draft). Next, we have added the references to SLiC-HF and online DPO in Section 2 of the revised draft.

---

[W1-4] L307: how can you change the system prompt to be only one of (s1,s2)? Surely the response which does not correspond to the chosen system prompt is going to be worse?

We randomly sample the one system prompt between $(s_1,s_2)$. After that, we consider the response from the sampled system prompt as a preferred response $y_w$ under this system prompt, and the other response which does not correspond to this system prompt as a dispreferred response $y_l$.

[Q1] The paper assumes a predefined set of preference features based on certain definitions and classifications. These assumptions may not cover the full spectrum of human preferences, and the preference features of math or coding tasks may be different. How to choose the most proper preference features for different tasks?

Our approach does not aim to choose the proper preference features for different tasks. Instead, we focus on mitigating the unintended biases that arise during online preference learning by preserving the distribution of the key preference features that can be extracted in existing datasets. Therefore, we used the fixed set of preference features through all the experiments, and the other categories of preference features can be also considerable.

The feasibility of the current way to extract and utilize preference features has been supported by prior research. For example, the approach of deriving preference features from two labeled responses has been validated experimentally [7]. Additionally, studies like Janus [8] have shown that even predefined discrete features can effectively capture and represent a wide range of human preferences when used for supervised fine-tuning. Building on this foundation, we designed our framework to use a more extensive set of predefined features, enhancing its capacity to reflect diverse human preferences. These prior successes underscore the practicality and validity of our method in leveraging discrete feature dimensions to mitigate biases in online preference learning.

---

[1] Yuan et al., Advancing llm reasoning generalists with preference trees., arXiv:2404
[2] Mitra et al., Orca-math: Unlocking the potential of slms in grade school math., arXiv:2402
[3] Li, et al., From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline., arXiv:2406
[4] Meng et al., SimPO: Simple Preference Optimization with a Reference-Free Reward., NeurIPS 2024
[5] Dong et al., RLHF Workflow: From Reward Modeling to Online RLHF., arXiv:24.05
[6] Park et al., Disentangling Length from Quality in Direct Preference Optimization., Findings of ACL 2024
[7] Pitis et al., Improving context-aware preference modeling for language models., arXiv:2407
[8] Lee et al., Aligning to thousands of preferences via system message generalization., NeurIPS 2024

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

Dear Reviewer sywp,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1] Diversity of tasks: Use of additional or more diverse datasets could strengthen the claims of the framework's effectiveness.

We clarify that our framework primarily targets modeling common human preferences in general language tasks. While math and coding tasks are important domains, they require specialized preference features that differ fundamentally from common human preferences. These specialized domains often use distinct data collection methods and training approaches, as demonstrated in recent works [1,2]. Given our framework’s specific focus on general human preference alignment, including these specialized domains would fall outside our intended scope.

Nevertheless, we agree that evaluating our framework on more diverse datasets can provide a stronger assessment of our framework. Following your suggestion, we have additionally evaluated our framework and key baseline methods on the "Arena-Hard-Auto" benchmark [3], which is another well-established benchmark to evaluate LLM alignment [4]. We present the results on the below table. Here, one can observe that while PFP performed slightly below SELFEE, it still outperforms the iterative DPO model by a significant margin. Notably, PFP consistently produced shorter responses compared to both SELFEE and iterative DPO, indicating effective bias mitigation in response length. We have added these new results and the discussion in Appendix F of the revised draft.

\begin{array}{l|cc} \hline \text{Methods} & \text{Score} & \text{Avg. len} \newline \hline \text{SFT} & 2.2 & 507 \newline \text{Iterative DPO} & 4.4 & 671 \newline \text{SELFEE} & 9.0 & 839 \newline \hline \text{PFP (Ours)} & 7.0 & 463 \newline \hline \end{array}

---

[W2] Comparative Analysis with State-of-the-Art Methods: The paper compares PFP with SFT, DPO and Iterative DPO but does not include a comparison with the latest state-of-the-art methods in bias mitigation for LLMs. Including comparisons with cutting-edge preference learning methods would provide a clearer picture of PFP's performance relative to the most advanced techniques.

We first clarify that our work is the first work that identifies the preference feature bias during the iterative preference learning; therefore, there are no comparable baselines that can be directly applied to mitigate this problem.

Next, in the perspective of length bias, we remark that the comparison with the latest state-of-the-art methods is already included in the original draft; in Tables 4 and 5, we observed that the proposed PFP is significantly better to mitigate the length bias compared to two different state-of-the-art methods based on the length penalty in reward model [5] and modified DPO objective [6].

Lastly, to respond to your suggestion to include cutting-edge preference learning methods, we have additionally conducted the experiments with SimPO [4], the latest offline preference alignment algorithm, as our new baseline. This baseline is trained using the same seed data size as PFP, which is also the same amount of data used by DPO. Only the batch size was set to 32, the same as our setup, and the remaining hyperparameters from Simpo's official github were used as is. The results are presented below, and one can observe that PFP yields better alignment performance with less length bias.

\begin{array}{l|ccc} \hline \text{Methods} & \text{LC Win Rate} & \text{Win Rate} & \text{Avg. len} \newline \hline \text{DPO} & 9.93 & 8.02 & 1409 \newline \text{SimPO} & 9.29 & 10.94 & 2192 \newline \hline \text{PFP (Ours)} & 15.24 & 10.18 & 1187 \newline \hline \end{array}

Dear Reviewer PYLd,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1] Were any experiments conducted to examine these additional preference aspects?

Thank you for the opportunity to clarify. Beyond addressing length bias, we conducted experiments to examine biases in broader preference features, as shown in Figures 4 and 5. Specifically, we measured the KL divergence between the preference feature distributions assigned to new input prompts and the human preference feature distribution from the seed dataset. An increasing KL divergence across iterations indicates growing biases as the assigned features deviate from the original distribution. The results in Figures 4 and 5 show that the proposed PFP effectively mitigates these biases (i.e., maintaining low KL divergence), while baselines such as iterative DPO and SELFEE fail to do so. This demonstrates PFP’s efficacy in addressing biases across multiple preference feature aspects during online preference learning.

---

[W2] Potential variability in the results due to using GPT-4 for evaluation.

We acknowledge the reviewer’s concerns regarding the use of GPT-4 for evaluation and the variance it may introduce. To address this, we conducted two additional rounds of experiments. In these new experiments, we fixed the initial DPO model and repeated subsequent evaluations to assess consistency. We conducted an evaluation on AlpacaEval 2.0, and obtained the results below. While the differences in LC Win Rate may appear modest (e.g., 1.46% improvement), the variance for these evaluations (e.g., 0.29 for PFP) supports the statistical significance of these results. Moreover, we remark that the primary contribution of our method lies in feature debiasing. As shown in Figure 4, PFP demonstrates substantial improvements over other baselines in mitigating preference feature bias. We have added these new results in Appendix F of the revised draft.

\begin{array}{l|ccc|cc} \hline \text{Methods} & \text{1st Seed Data} & \text{2nd Seed Data} & \text{3rd Seed Data} & \text{Average} & \text{Variance} \newline \hline \text{PFP (Ours): LC Win Rate} & 15.24 & 14.38 & 14.22 & 14.61 & 0.29 \newline \text{PFP (Ours): Win Rate} & 10.18 & 10.16 & 9.97 & 10.10 & 0.01 \newline \hline \text{SELFEE: LC Win Rate} & 14.23 & 12.58 & 12.64 & 13.15 & 0.84 \newline \text{SELFEE: Win Rate} & 17.49 & 15.59 & 17.06 & 16.71 & 0.99 \newline \hline \end{array}

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

Dear Reviewer PYLd,

We once again thank you for your review of our manuscript. We sincerely appreciate your valuable insights and feedback.

As the discussion period is drawing to a close, we kindly remind you that there are two days remaining for further comments or questions. We would be grateful for the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you for your time and consideration.

Warm regards,
Authors

Dear Reviewer 639N,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1] Notations in this paper can be heavily improved, in particular for the FE part

Thanks for your constructive comments! Following your suggestion, we have modified the corresponding parts in the revised draft (lines 233-235), as follows:

• (Original) Specifically, for the input instruction $x$ and the two responses $y_{w}$ and $y_{l}$, the feature extractor is defined as $\mathbf{p} = LLM_{FE}(x,y_{l},y_{w})$ where $\mathbf{p}=[p_{1},...,p_{5}], p_i \in C_i$.
• (Revised) Specifically, for the input instruction $x$ and the two responses $y_{w}$ and $y_{l}$, the feature extractor is defined as $\mathbf{p} = LLM_{\text{FE}}(x,y_l,y_w)$ where $\mathbf{p}=[p_{1},...,p_{5}]$, where each $p_i$ represents a probability distribution over the 5 sub-features of class $C_i$ (i.e., $p_i \in [0,1]^5$ and $\sum_{j=1}^5 p_i^j = 1$).

---

[W2] I am still not fully sure about the motivation of this work… In my belief, as long as the answer is correct, and if the system prompt doesn't have instruction to be precise (succinct), then there is nothing wrong in long generations (other than the computation aspect)

First, we would like to clarify that our main motivation is to eliminate biases at the preference feature level, rather than simply addressing length bias. Indeed, the reducing length bias is a side effect from debiasing at the preference feature level.

Next, regarding the problem with lengthy responses, length bias is a form of reward over-optimization that has been persistently noted as an issue [1,2,3,4]. While response length does show some correlation with human preference, current reward modeling tends to favor overly lengthy responses, leading to a divergence from actual human preferences. As a result, these models often generate responses that stray away from true human preferences, and therefore it is undesirable. But, it’s true that the generation of lengthy responses itself can be not problematic if the superiority between responses could be objectively defined like coding or math tasks using the correctness. However, we note that the superiority of responses cannot be explicitly measured depending on the situation, such as in open-ended tasks or subjective domains.

---

[W3] How is length-controlled generation done?

We did not separately generate the length-controlled responses to measure length-controlled (LC) win rate. Indeed, LC win-rate and normal win-rate are simultaneously evaluated for the same responses using GPT-4 as evaluator, while LC win-rate is measured by neutralizing the effect of response length to focus on quality, using a separately trained regression model [1].

---

[W4] Why should SELFEE work? … If the model is not good enough to begin with, then wouldn't it exacerbate the biases (or if it is incorrect for certain prompts)?

We first remark that SELFEE begins with an LLM fine-tuned using DPO on the initial seed preference dataset; therefore, depending on the size of the seed dataset and the degree of distribution shift in new prompts for each iteration, the effectiveness of SELFEE can vary. However, our experiments (Table 1) show that it yields significant improvements in alignment performance, demonstrating its suitability for our setup. Additionally, as observed in Table 1 and Figure 4, even online preference learning with an external reward model (Iterative DPO) experiences biased preference features and increased response lengths. This indicates that the increased response length in SELFEE is not merely a result of exacerbating biases from an insufficiently strong starting model but reflects broader challenges inherent in current online preference learning methods. Lastly, we note that the proposed PFP is also based on SELFEE framework which is more vulnerable for the bias, but it successfully reduce the bias in preference features and response length.

[W5] I am not sure why the output adjustment is needed. Can authors perform a study showing its use?

The output adjustment process (lines 252-269 of the revised draft) is introduced to ensure that the distribution of preference features assigned to a new input prompt set aligns with the human preference feature distribution observed in the initial seed dataset. This is necessary because directly predicting preference features using the trained feature classifier (lines 247-252 of the revised draft) can be insufficient due to the classifier’s limitations (see Table 7 of the revised draft), such as being trained on a small dataset and the long-tailed nature of preference features. We also note that the effectiveness of this process is demonstrated in Figure 5(b), where the KL divergence between the human preference feature distribution and the assigned distribution is significantly reduced when using output adjustment compared to naive predictions. This alignment leads to improved performance, including better alignment scores and reduced length bias, as shown in Table 2.

---

[W6] In Fig 4, why is it bad that the distribution for the preference feature is changing (that is KL divergence increasing). I am not fully sure why should that be linked with longer-generation

An increasing KL divergence across iterations indicates growing biases in the preference feature distribution, as the assigned features deviate further from the original human preference distribution in the seed dataset. Since our goal is to minimize such biases, a higher KL divergence is undesirable. On the other hand, it’s true that this increased KL is NOT directly linked with longer-generation; however, we hypothesize length bias may arise because certain features correlated with response length, such as the depth of informativeness or user’s background knowledge. Therefore, the biases on these features across the iterations (see Figures 16-18 in Appendix F of the revised draft), potentially contributes to longer responses.

---

[W7] Can authors run an experiment that just adds "being concise" in the system prompt, as another baseline?

Thank you for the suggestion. Following your suggestion, we have conducted this experiment; we found that it (SELFEE + Concise) led to some reduction in response length, but it also resulted in decreased overall performance. We have added these results in Table 11 in Appendix F of the revised draft.

\begin{array}{c|ccc} \hline \text{Methods} & \text{LC Win Rate} &\text{Win Rate} &\text{Avg. len} \newline \hline \text{SELFEE} & 14.23 & 17.49 & 2412 \newline \text{SELFEE + Concise} & 13.40 & 15.50 & 2218 \newline
\text{PFP (Ours)} & 15.24 & 10.18 & 1187 \newline \hline \end{array}

---

[1] Dubois et al., Length-controlled alpacaeval: A simple way to debias automatic evaluators., arXiv:2404
[2] Singhal et al., A long way to go: Investigating length correlations in rlhf., arXiv:2310
[3] Park et al., Disentangling length from quality in direct preference optimization., arXiv:2403
[4] Yuan et al., Following length constraints in instructions., arXiv:2406

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors