More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment

We appreciate the reviewer's thoughtful comments and the opportunity to clarify several points. Below, we address each concern:

W1&Q1: Are the authors' main claims true for the LLMs that are aligned with RLHF or other direct methods (e.g., IPO, SimPO)?

We thank the reviewer for raising this important question and we address the two parts of this question separately: 

1. For other direct preference optimization methods (e.g., IPO, SimPO):

Yes, our main claims extend to other direct preference optimization methods beyond DPO. We conducted IPO experiments on Llama3.1-8B, employing the same data generation strategies as in our DPO experiments. The results are as follows:

Consistent with DPO, Self + RM achieves the lowest ASR (0.01), while preference data from stronger models or humans leads to significantly higher ASRs. This confirms that our core conclusion—self-generated preference data filtered by an RM yields the most robust alignment—also holds for IPO.

Furthermore, methods like SimPO introduce adjustments such as length-normalized reward scoring to address length bias, as discussed in the Discussion section of our paper. While such methods may help mitigate specific artifacts (e.g., response length), they do not fully resolve the vulnerabilities tied to multi-model preference data. As shown in Table 2 and Figure 3, length control alone is insufficient to improve safety when using externally sourced preferences.

2. For RLHF methods : We do not analyze RLHF methods (e.g., PPO, GRPO) in this work because they do not allow for clean isolation of the effects of different preference data sources. Unlike direct preference optimization methods such as DPO, which operate offline on fixed preference datasets, RLHF involves on-policy rollouts where the model generates its own responses and updates based on reward model scores. While the reward model in RLHF may be trained using different data sources, the policy is always optimized using rewards assigned to self-generated rollouts, making it difficult to disentangle the influence of the data used to train the reward model from that of the policy's own outputs. Our focus is on direct methods like DPO, where the setup enables controlled comparisons of data generation strategies and their impact on safety alignment.

Q2: Do the authors think that DPO is particularly vulnerable to the phenomenon the authors observed?

Yes, we think DPO and its variants are indeed more vulnerable when the preference data come from multiple, heterogeneous models. Unlike online RL methods (e.g., PPO) that continuously generate new rollouts (self-generated responses), DPO is trained on the fixed preference dataset. Any stylistic artifacts that differentiate one source model’s answers from another’s are therefore “baked in,” giving the policy stable shortcuts to exploit. Our work highlights the need to scrutinize multi-model preference data when using direct preference optimization methods.

W2&Q3: Does reward hacking discussed in 3.4 results only from data generation process? Does this phenomenon occur for other DPO parameters, such as β or learning rate?

To isolate the cause of reward hacking, we ran controlled experiments on the GPT-4o + Self dataset using Llama3.1-8B, varying $\beta$ from 0.01 to 0.9 and testing two learning rates: 5e-7 and 1e-7.

As shown in the table below, ASR remained consistently high and stable across all settings. This suggests that hyperparameter tuning does not alleviate reward hacking, reinforcing our conclusion that the issue mostly stems from the data.

W4: Minor comments & missing citations

We appreciate the reminder of references and math typos. We will fix the typos ($\pi_{rel}$ is $\pi_{ref}$ and $\pi_\rho$ is $\pi_\theta$), describe sigmoid function $\sigma(\cdot)$ and add the following references in our updated version.

• Wachi et al., “Stepwise alignment for constrained language model policy optimization.” NeurIPS 37, 2024.
• Huang et al., “One-shot safety alignment for large language models via optimal dualization.” NeurIPS 37, 2024.

With only a few days left in the discussion period, we would be most grateful if you could review our responses and share any further feedback at your convenience so that we may move closer to consensus on this paper.

Thank you for providing the authors' feedback. I have read other reviews and rebuttals. The responses mostly addressed my concerns, so I updated the score accordingly

We thank the reviewers for their careful consideration. We greatly appreciate the positive comments and address major concerns below.

W1: The notion of "safety" in this study is predominantly operationalized in terms of jailbreak resistance. However, critical dimensions such as toxicity mitigation and fairness remain unexplored, which constrains the broader applicability of the findings.

To further validate our findings, we evaluated toxicity mitigation and fairness using TOXIGEN [1], which covers over 135k toxic and 135k benign statements across 13 minority identity groups (e.g., African Americans, women, LGBTQ+ folks, etc.). We scored model outputs on a 1–5 Toxic-Score using GPT-4o. As shown in the below table, the toxic evaluation results from Llama3.1-8B align with our main conclusion—Self+RM achieves the lowest toxicity, while multi-model methods, such as using GPT-4o responses as chosen examples, result in much higher Toxic-Scores.

These results indicate that Self+RM not only strengthens jailbreak resistance but also reduces toxic outputs, underscoring its broader effectiveness for safety, including toxicity mitigation and fairness considerations.

Reference

[1] Hartvigsen, Thomas, et al. “Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.” ACL (2022).

W2: The experimental results are primarily derived from 7B-scale models. The study does not investigate whether the observed patterns persist in significantly larger models, raising questions about the scalability validity of the conclusions.

We agree that it is valuable to investigate the generalizability of our findings for the larger models. Thus, we conducted experiments on larger models: Llama-2-13B and Qwen2.5-14B.

As shown in the table below, both larger models demonstrate the same relative trend across data sourcing methods: Self+RM consistently yields the lowest ASR, while HC+Self and GPT-4o / Stronger+Self produce higher ASR values. 

These results suggest that our core findings on the relative safety effectiveness of different data curation strategies are robust across model scales, further strengthening our conclusion. We will include these larger model results in the updated version of the paper.

With only a few days left in the discussion period, we would be most grateful if you could review our responses and share any further feedback at your convenience so that we may move closer to consensus on this paper.

We thank the reviewer for their positive and constructive feedback. We address the questions raised below:

Q1: In the first SFT stage, does it only use the winning samples for SFT?

Yes, we follow the SPIN [1] paper setting and use only the winning samples for the first-stage SFT.

[1] Chen, Zixiang, et al. "Self-play fine-tuning converts weak language models to strong language models." arXiv preprint arXiv:2401.01335 (2024).

Q2: In the online DPO settings, we still find that self-generated RM can significantly boost the performance. Does this phenomenon also apply to the online setting in terms of safety?

Yes, our further experiments confirm that the benefits of self-generated preference data extend to the online DPO setting in terms of safety. We conducted experiments using Llama-3.1-8B under an online DPO setup for three iterations. In each iteration, we generated a 10k-sized preference dataset (self-gen + RM) using the current model. As shown in the table below, we found self-gen + RM continuously improves safety across iterations in the online DPO setting.

We thank the reviewers for acknowledging the strong conclusion of this work and the quality of the presentation. We address the concerns as follows: 

W1. An important component in both general tasks and safety tasks experiments is the reward model, but this paper lacks a description of this component. Is the same reward model used in both experiments? If so, what is its source? Additionally, does the reward model perform at the same level across different types of tasks?

We thank the reviewer for pointing out the omission of details about our reward model. We will include the following information in the revised version of the paper.

In all our experiments, we use the same reward model: OpenAssistant/reward-model-deberta-v3-large-v2. This model has been thoroughly evaluated in RewardBench (Lambert et al., 2024), a comprehensive benchmark designed to assess reward models across multiple dimensions, including general helpfulness, safety, and reasoning. We believe this model is a good choice for our study, as it offers a well-balanced performance across both general and safety-critical tasks. 

Overall Performance on RewardBench：

Below, we provide detailed model’s performance on general and safety-specific evaluation subsets from RewardBench:

 1. General Task Benchmarks Performance on two widely-used open-ended instruction-following benchmarks:

 2. Safety Benchmarks These benchmarks evaluate the model’s ability to (1) appropriately refuse harmful content and (2) avoid overly conservative refusals due to lexical similarity. Prompts and chosen/rejected pairs are adapted from modified versions of XSTest [Röttger et al., 2023] and Do-Not-Answer [Wang et al., 2023].

References

• Lambert, Nathan, et al. "RewardBench: Evaluating Reward Models for Language Modeling." arXiv preprint arXiv:2403.13787 (2024).
• Röttger, Paul, et al. "XSTest: A Test Suite for Identifying Exaggerated Safety Behaviors in Large Language Models." arXiv preprint arXiv:2308.01263 (2023).
• Wang, Yuxia, et al. "Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs." arXiv preprint arXiv:2308.13387 (2023).

W2: Although the authors tested the effectiveness of various model families, the sizes of these models were concentrated around 7 billion parameters. This may introduce a bias related to model scale in the conclusions. Would the conclusion hold for larger or smaller models?

We agree that it is valuable to investigate the generalizability of our findings for the larger models. Thus, we conducted experiments on two larger models: Llama-2-13B and Qwen2.5-14B.

As shown in the table below, both larger models demonstrate the same relative trend across data sourcing methods: Self+RM consistently yields the lowest ASR, while HC+Self and GPT-4o / Stronger+Self produce higher ASR. 

These results show that our core findings on the safety impact of data curation hold across model scales, further reinforcing our conclusion. We will include these larger model results in the updated paper.

With only a few days left in the discussion period, we would be most grateful if you could review our responses and share any further feedback at your convenience so that we may move closer to consensus on this paper.