HPS: Hard Preference Sampling for Human Preference Alignment

Thank you for the insightful comments! We provide our response and hope our response addresses your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.

1) Our revision will discuss alignment methods like SLiC-HF (arXiv:2305.10425) and LiPO (arXiv:2402.01878) based on list responses. SLiC-HF is an alternative to RLHF-PPO by integrating the sequence-level contrastive method SLiC (arXiv:2210.00045) with human preference rankings:

$$\mathcal{L}(\theta)=\max(0,\delta-\log(\pi\_{\theta}(y^{+}|x))+\log(\pi\_{\theta}(y^{-}|x))-\lambda\log(\pi\_{\theta}(y\_{ref}|x))).$$

$y^{+}$, $y^{-}$, and $y_{ref}$ denote the positive, negative, and reference sequences, respectively. $\delta$ is a margin hyperparameter and $\lambda$ is a regularization weight. In contrast, our HPS framework focuses on rejecting all potentially harmful responses while leveraging the varying informativeness of dispreferred responses.

In LiPO-$\lambda$, it employs a listwise ranking objective with a Lambda weight $\Delta_{i,j}$. Given a list of responses $\boldsymbol{y}=(y_1,\dots,y_K)$,$\mathcal{L}\_{LiPO}=\mathbb{E}\_{(x,\boldsymbol{y},\psi)\sim\mathcal{D}}\left[\sum\_{\psi\_{i}>\psi\_{j}}\Delta\_{i,j}\log(1+e^{-(s\_i-s\_j)})\right],$where $\Delta_{i,j}=|2^{\psi_i}-2^{\psi_j}|\cdot|\frac{1}{\log(1+\tau(i))}-\frac{1}{\log(1+\tau(j))}|.$ Here, $\psi_{i}$ is the true reward score of response $y_i$, and $s_i=\beta\log\frac{\pi_{\theta}(y_i|x)}{\pi_{{ref}}(y_i|x)}$ is the implicit DPO reward. The rank position of $y_i$ in the ordering induced by $\mathbf{s}=(s_1,\dots,s_K)$ is denoted as $\tau(i)$. The Lambda weight assigns greater importance to response pairs with larger preference gaps, i.e., $\psi_i-\psi_j$. However, our HPS prioritizes hard dispreferred responses—those that closely resemble the correct output but remain incorrect.

2) Our HPS can be extended to the setting where multiple top responses are valid. Please see our response to Reviewer 674W.

3) Sampling a single importance-weighted dispreferred response DOES NOT reduce HPS to standard BT, since HPS designs an importance-weighted sampling strategy, unlike BT’s deterministic selection.

In HPS, a dispreferred response is sampled based on the importance-weighted distribution (L244):q(x,y)=\frac{e^{\gamma\cdot r_{est}(x,y)}}{\sum_{i=2}^{n}e^{\gamma\cdot r_{est}(x,y_{\tau(i)})}},\tag{1}where $(y_{\tau(i)})_{i=2}^{n}$ denote the dispreferred responses of a prompt $x$. This ensures that harder dispreferred responses—those more challenging to distinguish from preferred ones—are sampled more frequently and penalized more during training.

In contrast, BT always selects a fixed pair: the most preferred and the most dispreferred response, ignoring all other dispreferred responses. However, the most dispreferred response is often the easiest to reject. HPS addresses this by prioritizing harder responses, enabling the model to refine its distinctions between preferred and dispreferred outputs.

Empirically, BT struggles to capture the preference gap effectively. Table 2 shows that compared to BT, HPS significantly improves reward margins, reducing detrimental responses.

4) While our experiments focus on $n\leq 100$, Thm. 1 analytically characterizes how error bounds $\Psi_{1}$ and $\Psi_{2}$ scale with $n$, providing insights for larger settings. We limit $n$ to 100 due to limited GPU resources.

Moreover, Table 5 shows that as $n$ increases, the reward margin metrics consistently improve, aligning with the theoretical scaling behavior.

5) In Thm. 1, $d$ is defined in Assumption 1 (L239), and denotes the parameter dimension of the reward model. For implicit reward modeling, it corresponds to the trainable LLM’s parameter dimension. We will carefully refine our manuscript.

6) HPS samples a single dispreferred response via the importance-weighted distribution $q(x,y)$ (see Eq. 1 in response 3))

$q(x,y)$ defines a probability distribution over these responses, so sampling strictly follows this distribution. Deviating from it would introduce a different approach, compromising the theoretical guarantees in Sec. 5, particularly HPS’s improved sample efficiency and reward margin maximization over PL methods.

7) We use Qwen2.5-72B-Instruct for evaluation. In Sec. 7, we acknowledge that budget constraints limit us to open-source LLMs for estimating win rates. To strengthen our evaluation, we conducted a user study with human participants. See Tab.2 in our response 2) to Reviewer v4s5 for details.

8) Thanks. Since we mainly analyze PL and BT, we use these methods to investigate theoretical implications and empirical performance. To address your concern, we compare HPS with LiPO-$\lambda$ on HH-RLHF and find that HPS significantly improves the Reward Margin, limiting harmful responses.

Thank you for the insightful and valuable comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.

1) PL simplifies to BT when $n=2$, but differs when $n \geq 3$, explaining their different results in Table 2. In this experiment, each prompt has 100 responses ($n=100$). BT selects the most preferred and dispreferred responses for training, whereas PL uses all 100 responses to compute its loss (Eq. 4 and 5), leading to different performance outcomes.

2) Regarding the sensitivity of DPO, to address your concern, we have conducted experiments with $\beta=(0.1,0.25,0.5,0.75,1)$ and report the KL divergence $\mathbb{D}\_{KL}[\pi_{\theta}(y_w|x)||\pi_{ref}(y_w|x)]$ across these values, where $x$ is the prompt and $y_w$ is the winning response in the test set. The results in Tab.1 demonstrate the superiority of our HPS: it achieves the highest $RM_{R-DPO}$ for all KL values, confirming that HPS leads to stronger rejection of harmful responses.

Tab.1 Ablation results with $\beta$ on HH-RLHF under fine-tuning setting.

Moreover, we conducted a user study by selecting 15 prompt questions from HH-RLHF and 15 from PKU-SafeRLHF. For each question, four responses generated by SFT, DPO-BT, DPO-PL, and DPO-HPS are rated by 20 human evaluators on a 1–5 scale. To avoid bias, models were anonymized, and the response order was randomized. As shown in Tab.2, HPS achieves the highest quality score among all methods.

Tab.2 Human evaluation on user study dataset.

3 For experiments), Tables 2–4 in our paper demonstrate that HPS-based methods consistently outperform other methods. For each prompt, DPO-BT only selects the most preferred and dispreferred responses among $n=100$ responses for training, while DPO-PL and DPO-HPS use all $n=100$ responses.

Moreover, Table 5 presents an ablation study analyzing the impact of varying the number of responses $n=5,20,50,100$ on preference optimization. The results indicate that DPO-HPS scales better and achieves superior preference optimization with larger response sizes.

Specifically, we follow (arXiv:2306.17492) and expand response data by generating 100 responses using RLHFlow/Llama3-v2-DPO (arXiv:2405.07863) per prompt. The corresponding rewards are computed via Skywork-Reward (arXiv:2410.18451). Then we use methods like DPO-BT, DPO-PL, and DPO-HPS to fine-tune the language model on these data. Our preference fine-tuning methods explicitly leverage this broader response set rather than being constrained to the $n=2$ case.

4), Existing LLMs generate a single response autoregressively, which means that there is no inherent mechanism to ensure whether the generated content is harmful or harmless. This limitation raises concerns in scenarios that require adherence to safety and ethical standards.

Additionally, if we generate $K$ responses using the LLM, the computational cost becomes substantial due to the inference overhead. Even with multiple generated responses, LLMs without refined preference alignment cannot rank these responses autonomously, making it difficult to identify the top-ranked response based on quality.

To select the best response, a well-trained reward model is needed to rank the generated responses based on their quality and choose the highest-reward one. However, this approach introduces two key limitations:

• Inference Cost: Generating $K$ responses incurs significant computational overhead.
• Safety Concerns: While the top-ranked response may be of high quality, it is not guaranteed to be harmless, as both the LLM and reward model may fail to capture all potential risks.

To address these issues, we propose HPS, which ensures that lower-ranked responses with undesirable content are minimized, prioritizing the reduction of false negatives (L138). This consideration is particularly crucial in applications requiring high-quality and safe content generation, such as healthcare and education.

Furthermore, our HPS can be extended to the setting where multiple top responses are valid. Please see our response to Reviewer 674W.

5) For the missing subscript, we will correct it, and also carefully review the entire manuscript. Many thanks for your thorough proofreading and effort!

Thank you for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.

Our HPS method can also be extended to the setting where multiple top responses are all valid. As stated in Sec 4 (L138), our primary objective is to ensure that models generate helpful and harmless responses while avoiding harmful or dispreferred outputs. In our setup, we assume $y\_{\tau(1)}$ is the preferred harmless response, while we cannot guarantee that $(y\_{\tau(2)},\dots,y\_{\tau(n)})$ are entirely free from undesired content. Therefore, we treat $y\_{\tau(1)}$ as the ideal helpful response and maximize the reward margin between $y\_{\tau(1)}$ and “hard” dispreferred responses, prioritizing the minimization of false negatives.

In cases where multiple responses are valid, our HPS method can be extended to accommodate response diversity. Specifically, we can formulate a weighted HPS loss, treating each valid response as a preferred one in its respective loss term. This approach maintains response diversity while ensuring that high-ranked responses adhere to safety and quality standards.

For instance, given a training sample $d=(x,y\_{\tau(1)},y\_{\tau(2)},\dots,y\_{\tau(n)})\sim\mathcal{D}$, if both $y\_{\tau(1)}$ and $y\_{\tau(2)}$ are helpful responses, we can redefine the objective to train the model to reject all dispreferred and potentially harmful responses $(y\_{\tau(i)})\_{i=3}^n$, ensuring that it generates only the preferred responses $y\_{\tau(1)}$ and $y\_{\tau(2)}$ for a given prompt $x$. The modified loss function is defined as a weighted sum of two HPS losses:$\mathcal{L}\_{\boldsymbol{\theta}}=\mathcal{L}\_1+\lambda\cdot\mathcal{L}\_2$where $\lambda$ is a weighting hyperparameter, and\mathcal{L}\_{1}=\mathbb{E}\_{d\sim\mathcal{D}}-\log\left(\frac{e^{{r\_{\theta}(x,y\_{\tau(1)})}}}{e^{{r\_{\theta}(x,y\_{\tau(1)})}}+ N\_{1}\cdot\mathbb{E}\_{y\sim p(y)}[e^{{r\_{\theta}(x,y)}}q\_{1}(x,y)]}\right),$$$$\mathcal{L}\_{2}=\mathbb{E}\_{d\sim\mathcal{D}}-\log\left(\frac{e^{{r\_{\theta}(x,y\_{\tau(2)})}}}{e^{{r\_{\theta}(x,y\_{\tau(2)})}}+ N\_{2}\cdot\mathbb{E}\_{y\sim p(y)}[e^{{r\_{\theta}(x,y)}}q\_{2}(x,y)]}\right),with$$q_{1}(x,y)=\frac{e^{\gamma\cdot r_{est}\left(x,y\right)}}{\sum_{i=2}^{n}e^{\gamma\cdot r_{est}\left(x,y_{\tau(i)}\right)}},$$$$q_{2}(x,y)=\frac{e^{\gamma\cdot r_{est}\left(x,y\right)}}{\sum_{i=3}^{n}e^{\gamma\cdot r_{est}\left(x,y_{\tau(i)}\right)}},$$$N_{1}=n-1$,$N_{2}=n-2$, and$p(y)$is the probability distribution of the dispreferred response$y$. By optimizing the weighted HPS loss$\mathcal{L}_{\boldsymbol{\theta}}$, the model is encouraged to rank$y_{\tau(1)}$and$y_{\tau(2)}$above all dispreferred and potentially harmful responses$(y_{\tau(i)})_{i=3}^n$, thereby maintaining both helpfulness and response diversity.

We will include this discussion in the revision.

Thank you for the insightful and positive comments! We provide our response and hope our response addresses your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.

1) For quality-weighted BT, we could not find prior work directly related to it but identified two relevant methods: WPO (arXiv:2406.11827) and LiPO-$\lambda$ (arXiv:2402.01878). We will discuss them first, and will provide further discussion if you can provide specific references.

In WPO, weights are assigned to response pairs based on their occurrence probability: $\mathcal{L}_{WPO}=-\mathbb{E}\_{(x,y_w,y_l)\sim\mathcal{D}}[w(x,y_w)w(x,y_l)\log p(y_w\succ y_l|x)],$where $w(x,y)=\exp{\frac{1}{|y|}\sum^{|y|}\_{t=1}\log\pi\_{\theta}(y_t|x,y\_{<t})}$ and $|y|$ is the number of tokens in the output. So WPO modifies response weights to better align with on-policy data while following DPO to consider only the most preferred and most dispreferred responses. In contrast, our HPS accounts for multiple responses and focuses on leveraging the varying informativeness of all dispreferred responses.

Discussion of LiPO-$\lambda$ can be found in our response 1) to Reviewer 85yJ.

2) Regarding importance-weighted dispreferred response, we directly sample one dispreferred response according to the importance-weighted distribution (L244):$q(x,y)=\frac{e^{\gamma\cdot r_{est}(x,y)}}{\sum_{i=2}^{n}e^{\gamma\cdot r_{est}(x,y_{\tau(i)})}},$where $(y_{\tau(i)})_{i=2}^{n}$ denote the dispreferred responses of a prompt $x$. Thus, harder dispreferred responses will be sampled with higher probability and contribute more to the loss due to their higher probability $q(x,y)$. Then, we can incorporate the sampled dispreferred response $y$ into the loss function Eq. 9 for training.

3) For response similarity, it refers to two responses having comparable rewards, indicating shared semantics (key information). For example, Fig. 1 in submission shows that for a given prompt $x$, response $y_{\tau(1)}$ is more similar to $y_{\tau(2)}$ than to $y_{\tau(3)}$ since their content is closer. Consequently, $y_{\tau(1)}$ and $y_{\tau(2)}$ receive similar rewards, reinforcing their similarity.

However, we may not fully understand the question—please clarify if needed.

4) For human evaluation, we conducted a user study with human participants. See Tab.2 in our response 2) to Reviewer v4s5 for details.

5) Regarding reward $r_{est}$ of each ranked response $y_{\tau(i)}$, we either use the given reward $r_{est}$ or estimate it with a pretrained preference-aligned reward model (L230). In our experiments, we use Skywork-Reward-Llama-3 (arXiv:2410.18451). The scaling factor $\gamma$ is linearly increased from -5 to 5 at every 20% interval of the training process.

6) For experimental setting, your understanding is correct. In Table 2, DPO-PL and IPO-HPS independently fine-tune Llama3-8B on HH-RLHF using DPO with a PL preference model and IPO assuming an HPS preference model.

7) For experiments, we follow (arXiv:2306.17492) and expand response data by generating 100 responses using RLHFlow/Llama3-v2-DPO per prompt. The corresponding rewards are computed via Skywork-Reward-Llama-3. Then we use methods like DPO-BT, DPO-PL, and DPO-HPS to fine-tune the language model on these data. For each prompt, DPO-BT only selects the most preferred and dispreferred responses for training, while DPO-PL and DPO-HPS use all 100 responses.

To evaluate alignment, we measure Reward Margins (RM) in Table 1, where higher RM scores indicate better preference alignment with minimal harmful or biased outputs.

For PL methods like DPO-PL, directly using 100 responses per prompt (n=100) incurs excessive GPU memory costs (see Eq. 9 $\mathcal{L}_{PL}$). To mitigate this, we reformulate the PL sub-loss $\mathcal{L}\_{j}(d)$ using Monte Carlo sampling:

$$\mathcal{L}\_{j}(d)=-\log\left(\frac{e^{r\_{\boldsymbol{\theta}} (x,y\_{\tau(j)})}}{e^{r\_{\boldsymbol{\theta}}(x,y\_{\tau(j)})}+N \cdot\mathbb{E}\_{y\sim u\_{j}(x,y)}[e^{r\_{\boldsymbol{\theta}}(x,y)}]}\right),$$

where $N=n-j$ and $u\_{j}(x,y)$ is a uniform distribution over dispreferred responses. Instead of using all $N$ dispreferred responses, we sample 5 per loss term $\mathcal{L}\_{j}$, which is the maximum our 4×L40S GPUs can accommodate.

This sampling-based PL formulation is theoretically equivalent to vanilla PL and does not impact performance. The table below confirms that randomly sampling 5 or 1 dispreferred response from 100 yields similar performance on HH-RLHF and PKU-SafeRLHF. Since the strategy is developed in this work and used by HPS, it ensures a fair comparison between PL and HPS.

8) We have briefly discussed explicit Preference Fine-Tuning methods (L155). We will discuss them more in the Appendix.

Thank you for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.

1) For hard negative sampling, this work is the first to extend hard negative sampling to preference alignment, addressing new task-specific challenges with novel and effective solutions:

a) Handling Varying Informativeness of Dispreferred Responses. In metric and contrastive learning (arXiv:2010.04592, arXiv:2108.09335), hard negatives are typically selected based on representation similarity to a positive anchor. However, in RLHF, where responses are generated autoregressively, obtaining effective sentence embeddings is impractical. Instead, we define "hardness" in the reward space, where dispreferred responses similar to preferred ones (i.e., with close reward scores) are considered harder (L252). Furthermore, selecting hard negatives using $q(x,y)$ presents an intractable distribution challenge (L234), which we address via Monte Carlo importance sampling (Eq. 9).

b) Improving Sampling Efficiency. In previous work on metric and contrastive learning, backbone models such as ResNet or GoogLeNet were employed in vision tasks, where the model size is approximately $1\%$ that of Llama-3-8B in the RLHF setting. To ensure computational efficiency, we reformulate HPS into an efficient sampling approach, using a single Monte Carlo sampling to select a single dispreferred response per training sample.

c) Theoretical Analysis of Hard Negative Sampling in Preference Alignment. Our work is the first to provide a theoretical analysis of hard negative sampling in this context, offering new insights into alignment. We compare sample efficiency between the preference loss (PL) and HPS loss, demonstrating that HPS improves sample efficiency, particularly in data-scarce settings or when rapid convergence is needed. Additionally, we analyze how training with HPS maximizes the reward margin between preferred and hard dispreferred responses, ensuring a robust distinction between them. This strengthens alignment performance while minimizing undesired outputs. We will further elaborate on these distinctions in Sec. 1 and 4.

2) Regarding the differences in sample complexity bounds of our HPS and the PL, intuitively, they stem from the structural distinction of two losses. The PL loss is composed of a summation of $n$ piece of one-to-$N$ contrast losses $\mathcal{L}\_{j}(d)$:\mathcal{L}\_{PL}=\mathbb{E}\_{d\sim\mathcal{D}}\sum\_{j=1}^{n}\mathcal{L}\_{j}(d)=\mathbb{E}\_{d\sim\mathcal{D}}\sum\_{j=1}^{n}-\log({e^{r\_{\theta} (x,y\_{\tau(j)})}/\sum\_{k=j}^{n}e^{r\_{\theta}(x,y\_{\tau(k)})}}),\tag{1}In contrast, our proposed HPS loss focuses on encouraging the model to rank the most preferred response $y\_{\tau(1)}$ against all other dispreferred responses $(y\_{\tau(i)})\_{i=2}^n$:\mathcal{L}\_{\theta}=\mathbb{E}\_{d\sim\mathcal{D}}-\log(\frac{e^{{r\_{\theta}(x,y\_{\tau(1)})}}}{e^{{r\_{\theta}(x,y\_{\tau(1)})}}+N\cdot\mathbb{E}\_{y\sim p(y)}[e^{{r\_{\theta}(x,y)}}q(x,y)]})\tag{2}withq(x,y)=\frac{e^{\gamma\cdot r\_{est}\left(x,y\right)}}{\sum\_{i=2}^{n}e^{\gamma\cdot r\_{est}\left(x,y\_{\tau(i)}\right)}}.\tag{3}The HPS loss $\mathcal{L}\_{\theta}$ only uses one component, $\mathcal{L}\_1$, from the full summation in $\mathcal{L}\_{PL}$. Thus, the structural distinction between the two loss functions leads to the $n$-factor discrepancy in the asymptotic error bound.

More specifically, as shown in Appendix B.1, the difference between the HPS loss $\mathcal{L}\_{\theta}$ and the PL loss $\mathcal{L}\_{PL}$ has a direct impact on their gradients. We follow the mathematical notations in our paper. Specifically:

• For HPS-based loss:$\|\nabla\mathcal{L}\_{\theta}(\theta^{*})\|^{2}\_{\Sigma^{-1}\_{\mathcal{D}}}\leq C\cdot\frac{d+\ln(\frac{1}{\delta})}{m}$
• For PL-based loss:$\|\nabla\mathcal{L}\_{PL}(\theta^{*})\|^{2}\_{\Sigma^{-1}\_{\mathcal{D}}}\leq C n^{4} \cdot \frac{d+\ln(\frac{1}{\delta})}{m}$

The error bound for the HPS-based method $\|\Delta\_{HPS}\|\_{\Sigma\_{\mathcal{D}}}$, where $\Delta\_{HPS}:=\theta\_{HPS}-\theta^{*}$, is bounded by

$\frac{\|\nabla\mathcal{L}\_{\theta}(\theta^{*})\|\_{\Sigma^{-1}\_{\mathcal{D}}}}{\zeta}$

with$\zeta = \frac{1}{2+\exp(2\alpha\_{0}+\ln(n-1))+\exp(-2\alpha\_{0})}$as stated in Thm. 1. Similarly, the error bound for the PL-based method $\|\Delta\_{PL}\|\_{\Sigma\_{\mathcal{D}}}$, where $\Delta\_{PL}:=\theta\_{PL}-\theta^{*}$, is bounded by

$\|\nabla\mathcal{L}\_{PL}(\theta^{*})\|\_{\Sigma^{-1}\_{\mathcal{D}}}.$

From this, we observe that $\Delta\_{HPS}$ exhibits an error bound of $\mathcal{O}(n)$, whereas $\Delta\_{PL}$ has an error bound of $\mathcal{O}(n^2)$, differing by a factor of $n$. We will integrate this intuitive explanation into Thm. 1 for clarity.