Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples

We thank the reviewer for their careful reading and insightful comments. We address the concerns below:

Q1) Claims about hallucinations

We revised the statement “reduces undesired hallucinations” to “generates policies that have lower NLLs”.

---

Q2) Application to other DPO variants

Our study evaluates data selection on two datasets and four LLMs, showing significant gains. We agree that testing additional DPO variants is valuable. However, extending our method is non-trivial, as the curriculum relies on model, dataset, and loss function. To address your concerns, we include comparisons with other data selection baselines (see Q3).

Q3) Comparison with more data selection baselines.

The mentioned confidence-based selection method selects data using reward margins from golden reward functions. We reproduce their idea using GPT-4 ratings: $r^*(x,y_w) - r^*(x,y_l)$ and sort samples by this reward margin. Results are reported in this link (https://selective-dpo.pages.dev/), labeled with Reward Margin (Descending, GPT-4) .

We observe no consistent performance benefit, suggesting that reward margin is not a reliable selection metric in our setting. This may stem from our focus on real datasets, in contrast to the mentioned work using manually corrupted labels.

---

Q4) Evaluation on MMLU

We report MMLU and related results in Table 8 (page 17). Performance is broadly consistent across DPO variants. One exception is GSM8K, where SelectiveDPO-Mistral-7B often produces correct answers in dialogue form rather than the strict format ### <The Answer>. This issue is resolved by including ~10% more math-style examples during training.

---

Q5) Definition of the learned step

We defined the learned step as the step after which the implicit reward model can distinguish the preferred from rejected answers with a large probability: $P(y_w > y_l | x) = \sigma(\beta\frac{\pi_{\theta}(y_w |x)}{\pi_{\text{ref}}(y_w|x)} - \beta\frac{\pi_{\theta}(y_l |x)}{\pi_{\text{ref}}(y_l |x)}) > \sigma(\delta)$. This follows the beautiful DPO formulation and reflects the intuition that the LLM is secretly a reward model. Defining the learned step using only $\pi_\theta$ is indeed cleaner and worth further exploration.

---

Q6) Missing related work

We appreciate the reviewer’s reminder. The mentioned work observes similar trends in SFT training. However, its definition of “unfamiliar knowledge” does not directly apply to alignment tasks. We agree the work is very relevant and have now cited it in our Related Work section.

---

Q7) Scope of the data selection strategy

We thank the reviewer for recognizing our contribution. The proposed selecting strategy is tailored for alignment data (prompt–preferred–rejected). It does not transfer directly to pre-training or SFT tasks, which use corpus or prompt–completion pairs and would require adapted difficulty metrics. However, we note that many SFT data selection papers (including the one mentioned) are not tested on alignment or pre-training tasks. This is not typically viewed as a limitation.

---

We sincerely thank the reviewer once again. We have revised the paper to include additional baselines and discussions addressing the raised concerns. With these updates, we hope the reviewer may find the work strengthened and reconsider their evaluation.

Thank you for the insightful suggestions regarding comparisons and related work. We address each comment in detail below:

Q1) Comparison with other difficulty metrics.

Prior work [0] has examined prompt length and attention scores, finding limited benefits for alignment. Building on this, we conducted experiments with: completion length, perplexity, perplexity gap, and reward margin. Full results are available in this link (https://selective-dpo.pages.dev/).

None of these metrics consistently outperformed our validation-loss-based approach. Notably, sorting by completion length (ascending) led to model collapse: the model overfit to short completions and failed to recover, highlighting the potential risks of overly simplistic heuristics.

Q2) Comparison with additional data selection baselines.

The suggested baseline [1] targets multimodal LLMs and introduces perplexity to select high quality samples for SFT. Following their idea, we implemented two variants in the DPO setting:

• Perplexity of chosen
• Perplexity gap

To avoid arbitrary thresholding, we followed a consistent protocol: (1) sort examples by the metric, (2) train with fixed hyperparameters (Table 3, page 15), and (3) evaluate performance across data percentages. Result is available at the shared link. Key findings:

• Perplexity of chosen improves over random sampling, suggesting it is a viable scoring function for curriculum learning.
• However, when used as a selection filter, it does not clearly distinguish “useful” from “harmful” examples—all data partitions appear beneficial.

---

(Q3) Missing related work.

Thank you for pointing out relevant papers. We have reviewed and will include them in our revised Related Work section:

• [1] emphasizes diversity and complexity for SFT data selection.
• [2] introduces a tree-based measure of data difficulty and finds complex SFT data contributes the most.
• [3] proposes LLM-rated quality scores and emphasizes the role of data quality in SFT.
• [4] explores a general data selection framework for pre-training and fine-tuning.

---

(Q4) Can curriculum learning benefits SFT?

Our work centers on data selection for alignment, not SFT. We discuss curriculum learning primarily as a tool to investigate example difficulty for alignment tasks. While we are encouraged by positive signs (e.g., Figure 3), we acknowledge that the following discussion is preliminary and may lack the nuance of dedicated studies.

• CL in alignment: We observe modest gains in our ablation (Figure 3) and expect greater benefits with refined pacing functions. However, designing these strategies is outside the scope of this work.
• CL in SFT: This remains a promising area. Prior studies ([5], [6]) show benefits for learning robustness and reasoning tasks.
• Difficulty-aware selection in SFT: Although we focus on alignment, the core idea—that overly difficult examples may hurt small models—may extend to SFT. In particular, [7] reports similar challenges in distillation, where small models underperform when exposed to overly complex teacher outputs. However, we don’t think borrowing our finding to SFT is an easy thing since it needs to redesign the learned step and validation loss (SFT data has different format).

[5] YODA: Teacher-Student Progressive Learning for Language Models, Arxiv 2024

[6] Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond, Arxiv 2025

[7] Small Models Struggle to Learn from Strong Reasoners, Arxiv 2025

---

Once again, we thank for the insightful comments and references. We appreciate the positive comments regarding our experiment design and we respectfully hope that the reviewer could reevaluate our work given the responses to your concerns.

We sincerely thank the reviewer for the thoughtful feedback and for pointing us to highly relevant related works. Your kind comments have helped us better position our contribution. Below we address the concerns in detail:

Q1) Missing related work.

Thank you for highlighting arxiv 2410.08847. This work focuses on identifying and filtering out training examples that cause likelihood displacement in DPO. We believe our work carries different motivations: we study how example difficulty impacts alignment while they analyze which sample cause likelihood displacement in DPO. We now include this paper in the Related Work section and highlight the distinction.

Regarding arxiv 2009.10795, we appreciate your mention of this important work on training dynamics. Their findings—easy samples have little value, ambiguous ones aid generalization, and hard samples is helpful despite may contain noise—are insightful. However, our conclusions differ in several ways:

• In alignment, hard samples consistently degrade performance.
• Simple data errors cannot fully explain this degradation—we introduced a series of experiments to support this.
• Larger models benefit from difficult examples, as shown in Figure 5.

We believe this difference stems from the models under study: our work focuses on LLM alignment, while theirs analyzes small models and classification tasks.

---

Q2) Comparison with complicated data selection

We agree that comparing with more advanced data selection methods (e.g., CHES scores from Arxiv 2410.08847) would strengthen our work. However, conducting this comparison is non-trivial and requires:

• Calculating model-specific CHES scores on our datasets.
• Establishing principled thresholding strategies for selection.

We are actively working on this and will update our results once we complete a fair and thorough comparison.

Alternatively, we add comparisons against other intuitive data selection metrics including perplexity gap, reward margin, and completion length as suggested by other reviewers. The results are available at this link (https://selective-dpo.pages.dev/). We hope these comparisons would alleviate your concern.

---

Q3) Evolution of validation loss

To better illustrate our intuition, we visualize the preference probability metric: $P(y_w >y_l |x)=\sigma(r(x,y_w) - r(x,y_l))$, which closely aligns with validation loss: $VL = -\log P(y_w > y_l|x)$.

As shown in the new results (Figure 13), many easy samples are learned early. Roughly 40% of the samples remain difficult throughout training, indicated by consistently low preference probabilities.

In response to your question: yes, we do observe a small subset of samples that become “correct” first and “incorrect” after. These samples tend to lie between the easiest and hardest, suggesting the model intermittently understands them—consistent with the “ambiguous instances” concept in the training dynamics work you cited.

---

We appreciate your encouragement to consider easy and ambiguous examples more deeply. While our current work emphasizes the negative impact of overly difficult samples, we agree that a fuller picture—including the role of ambiguous instances—would benefit the field.

Once again, we thank the reviewer for the valuable suggestions and for pointing us to relevant research. Your comments have significantly strengthened the scope and clarity of our revision.

We thank the reviewer for the thoughtful and constructive feedback, particularly regarding the label flipping experiment and the perplexity gap baseline. These comments help clarify and reinforce our central contribution: that alignment performance is critically influenced by the mismatch between model capacity and example difficulty. Please check our new results at this link (https://selective-dpo.pages.dev/) and the following response:

Q1) Label flipping experiment (Figure 4a)

Figure 4a tests whether label noise is the primary cause of performance degradation on difficult examples. Flipping all difficult samples did not improve performance, suggesting that label noise alone is unlikely to explain the difficulty. While partial noise (e.g., 30%) may exist, we do not claim the data is noise-free—only that noise is not the dominant factor.

To address your concern, we flipped only those examples identified as both difficult and potentially mislabeled by a reward model (Skywork/Skywork-Reward-Gemma-2-27B-v0.2, 1,414 examples in Qwen2.5). This targeted flipping (Label Flipping (Skywork)) also showed no consistent benefit across four models, reinforcing our conclusion. Notably, the original labels are from GPT-4.

Q2) Reporting variance

All figures already include standard error bars across runs. In addition, we report standard error (over 3 runs) in the result tables for completeness. Here are the results on Alpaca Eval 2:

Q3) Curriculum transfer

Figure 9 compares a model’s own curriculum with that of a smaller model trained on the same (pre-training and SFT) data. Results show that a model benefits more from its own curriculum. To test cross-model transfer, we trained Qwen2.5-7B using curricula from Mistral-7B and Qwen2.5-32B. The results confirm our conclusion: a model’s own curriculum is most effective.

Q4) Comparison with perplexity gap.

While various data selection methods exist for SFT, such as perplexity-based filtering, they are not directly applicable to DPO-style preference data due to fundamental differences in format: SFT data consists of (prompt, completion) pairs, while DPO uses (prompt, preferred, rejected) triplets. This mismatch makes direct application of SFT techniques unsuitable for alignment. To address this gap, we implemented perplexity of chosen, perplexity gap, etc.

To assess their effectiveness without relying on arbitrary thresholding, we adopted a consistent evaluation protocol: (1) sort examples by each metric, (2) train DPO using the hyper-parameter set in Table 3 (page 15), and (3) evaluate whether performance drops after removing a portion of the data.

None of these alternatives yielded consistent improvements over our validation-loss-based selection strategy.

Q5) Length bias in the selected data.

As shown in Figure 10 (page 18), selected (easier) examples tend to have slightly shorter responses and smaller length gaps (preferred minus rejected). This pattern arises because DPO loss often assigns higher loss to longer examples. Since our strategy filters out high-loss (difficult) samples, it indirectly favors examples with smaller length gaps. While this introduces a mild length bias, we agree that it warrants future investigation.

We sincerely appreciate your insightful suggestions. These have helped clarify the scope and robustness of our findings. We will incorporate the new experiments and discussions into the revised manuscript. Please let us know if this response sufficiently addresses your concerns.