What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning

Thank you for the time and comments, we have enriched our evaluation with additional strong results on more benchmarks with different backbone models, please refer to the General Response for details. We address your separate concerns below.

Q1: How much “research gap” remains in instruction mining or SFT data selection? What is the research gap that this work contributes to?

We respectfully disagree with the reviewer on these comments about this research line in general, as well as our work’s contribution and significance. While we have no doubt that the reviewer understands these “simple” methods from previous works and our study, we believe there may be an oversight from the reviewer regarding the overall progress and current state of this research direction.

To provide context, we would like to briefly summarize recent advancements in SFT data selection, which will illustrate the "research gap" and the broader picture. Initial studies like Self-instruct [1] and Alpaca [2] showed that medium-sized datasets (ranging from 50k to 100k samples) could endow pretrained models with effective instruction-following capabilities, though Alpaca was later identified as a weak baseline upon more rigorous evaluations. LIMA [3] highlighted the significance of data selection in SFT by using just 1,000 carefully chosen or human-crafted samples to surpass Alpaca's performance. However, these samples were not selected automatically.

Alpagasus [4] and Instruction Mining [5], as the reviewer mentioned, began exploring automatic data selection in SFT. Yet, their comparisons were limited to random baselines or the already weak model Alpaca at the time. They did not conduct standard evaluations on challenging benchmarks like MT-bench/AlpacaEval nor did they compete with strong models such as Vicuna and WizardLM. This indicates a clear gap: no study in automatic data selection has demonstrated practical effectiveness through evaluations on challenging benchmarks and against SOTA open-source SFT models -- we do not know whether [4] and [5] really work or not from their weak evaluations. This gap is significant as it leaves unanswered questions about the potential and practicality of automatic data selection and whether it can lead to SOTA open-source SFT models.

We note that [4] and [5] have updated their papers with more evaluations and baselines after the ICLR submission deadline, which we hope the reviewer realizes. However, even in their latest versions, [4] does not compare with any other state-of-the-art open-source SFT model. Our approach has significantly outperformed it (as shown in Table 5 of our paper). Meanwhile, [5] does not report standard MT-bench and AlpacaEval scores.

Q2: Our contribution and significance over previous/recent works on instruction mining.

Following the response above, our research stands out as the first to propose an automatic data selection approach that consistently demonstrates strong performance across multiple challenging benchmarks, including MT-bench, AlpacaEval, and the Open LLM Leaderboard. Our approach outperforms other data selection methods by a large margin (Table 5), yielding results that are comparable to, or even surpass, those of SOTA open-source SFT models such as Vicuna, WizardLM, and Mistral-Instruct. These empirical results – being the first to achieve SOTA open-source performance with 10x less data selected automatically – is particularly noteworthy because it suggests that our approach, the datasets selected, and the data selection tools we have developed can be practically and beneficially applied by a broad spectrum of researchers and practitioners.

In terms of technical innovation, our approach diverges from previous methodologies. We utilize an evolving strategy to elicit more faithful complexity and quality scores from ChatGPT, as opposed to a direct scoring method, which our results (shown in Tables 2 and 3) indicate is less effective. While we recognize that the combination of metrics and reliance on GPT models are not novel, we have no reason to complicate things when we find such simple or trivial combinations of our proposed metrics are highly effective. Methodologically, exploring alternatives to GPT reliance is an intriguing future direction, yet it was not deemed essential for the current scope of this paper, particularly given that our data selection tool does not require ChatGPT at the inference stage.

Furthermore, we would like to remind the reviewer that both Alpagasus and InstructionMining were first arxived within 3 months prior to the ICLR submission deadline. According to the ICLR official reviewer guide (https://iclr.cc/Conferences/2024/ReviewerGuide#Reviewing%20instructions), such works are regarded as concurrent. While we still believe our work represents significant advancements beyond these studies as described above, we urge the reviewer to consider this timing factor as well when evaluating our research.

Q3: Reproducibility: Code, data, model checkpoints, or data selection tools are not provided during the reviewing phase.

We have promised to release these resources after the review period in the footnote of the first page, we surely will do when we deanonymize this paper – it is not common practice to release model checkpoints in an anonymous way during the review period as far as we know.

Q4: Appendix is not cut from the main paper. Appendix should not be submitted under the main paper.

Authors may use as many pages of appendices (after the bibliography) as they wish

quoted from ICLR 2024 Call for Papers (https://iclr.cc/Conferences/2024/CallForPapers).

Q: Should the appendices be added as a separate PDF or in the same PDF as the main paper?
Either is allowed: you can include the appendices at the end of the main pdf after the references, or you can include it as a separate file for the supplementary materials.

quoted from ICLR 2024 Author Guide (https://iclr.cc/Conferences/2024/AuthorGuide)

[1] Wang et al. Self-Instruct: Aligning Language Models with Self-Generated Instructions. https://aclanthology.org/2023.acl-long.754.pdf ACL 2023.
[2] Taori et al. Stanford Alpaca: An Instruction-following LLaMA model. https://crfm.stanford.edu/2023/03/13/alpaca.html 2023.
[3] Zhou et al. LIMA: Less Is More for Alignment. https://arxiv.org/abs/2305.11206. 2023.
[4] Chen et al. AlpaGasus: Training A Better Alpaca with Fewer Data. https://arxiv.org/abs/2307.08701. 2023.
[5] Cao et al. Instruction Mining: When Data Mining Meets Large Language Model Finetuning. https://arxiv.org/pdf/2307.06290.pdf 2023.

Dear Reviewer RtKh,

We have revised the paper and included additional strong results on more benchmarks. In the response above, we have also tried to clarify progresses on this research direction and our contributions over previous (concurrent) works. Since the rebuttal period is closing very soon, could you please check our response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you,
The authors

Thanks for your helpful comments! Below we address your concerns.

Q1: Limited evaluation

Thanks for the comments. We have added more evaluation results on the Open LLM Leaderboard benchmark [1] that consists of four tasks, ARC, HellaSwag, MMLU, and TruthfulQA. Additionally, we have enriched our evaluations by training more pre-trained LLMs on our selected data, such as LLaMA-2-13B and Mistral-7B besides LLaMA-1-13B in the original submission. We have summarized these results in the table of the general response, while the detailed results are in Table 3,4,5 of the paper. On the Open LLM Leaderboard, our approach outperforms the random baseline and other strong models across three different backbone models. However, we would like to note that the performance on benchmarks such as MMLU is largely determined by the underlying base model, thus the gains after alignment tuning is typically not large, unless the training data is specifically crafted for the benchmark. This is why recent works on alignment put more emphasis on MT-bench and AlpacaEval [2].

Notably, on MT-bench and AlpacaEval, our added results show that our method with the LLaMA-2-13B model surpasses Vicuna-13B-1.5 that trains on 10x more data, and LLaMA-2-13B Chat that has undergone extensive RLHF training. With Mistral-7B, our approach attains an MT-bench score of 7.29 with only 10k training data, marking a state-of-the-art performance on MT-bench for all open-source SFT models at 7B and 13B sizes. We also observe that, while the gains of the Mistral DEITA model on AlpacaEval are slightly lower than those on MT-bench, a deeper analysis (as depicted in Figure 2) indicates that the Mistral DEITA model excels particularly in mathematical and complex reasoning tasks, areas not primarily covered by AlpacaEval.

Q2: Marginal performance improvement. DEITA is worse than Vicuna-13B on AlpacaEval dataset. With human evaluation, Vicuna performs almost similar to DEITA.

We emphasize that the purpose of this work is not to set up a new record on the performance, but to achieve top results with much less data, for example, the training of DEITA is far more efficient than Vicuna by using 10x less training data. Furthermore, we note that Vicuna is a SOTA SFT model, and DEITA outperforms it on most of the cases across three different benchmarks and various backbone models (see the table in General Response), except for just one setting on AlpacaEval – DEITA is the first automatic data selection approach that achieves comparable performance to SOTA models such as Vicuna and WizardLM with 10x less data. Therefore, we view comparable performance to Vicuna as the strength of our approach, rather than weaknesses.

Q3: Missing comparison with instruction selection works

Thanks for the suggestion! In the original submission, we have shown that “direct scoring” from Alpagasus [3] does not work well in Table 2 and 3, and using ChatGPT to score all the samples in the data pool following their paper is too expensive, thus we did not explicitly include them in the main tables. Following the reviewer’s advice, we have added the Alpagasus and LIMA baselines, the comparison of different instruction selection approaches is shown below (based on LLaMA-1-13B):

We see that DEITA outperforms other instruction selection baselines significantly. We have included the table above as Table 5 in the paper.

Q4: Why the complexity measurement is obtained from “instruction”, and the quality measurement is obtained from “response”?

We measure complexity from instruction only since the evolution procedure of complexity directly follows WizardLM, which takes only the instruction as input and evolves it step by step to increase the instruction complexity. For quality, our intuition is to measure how effectively a response addresses the query, thus both the instruction and the response are inputs to the scorer. We recognize that in Figure 1, the instruction input was inadvertently omitted in the depiction of the quality scorer. This was our oversight, we have slightly revised Figure 1 to be more clear on this.

[1] Beeching et al. Open LLM Leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard. 2023.
[2] Tunstall et al. Zephyr: Direct Distillation of LM Alignment. https://arxiv.org/abs/2310.16944. 2023

Q4: Are these scores comparable across evolved sets from different seed examples?

This is a good point. To answer this question, we conduct human evaluation on randomly selected 50 sample pairs from the ChatGPT complexity score pool – any example pair is not from the same evolved set. The authors then score each pair of examples from five categories: “A is significantly more complex than B”, “A is slightly more complex than B”, “neither is more complex”, etc. Then we compute the agreement score between the authors and ChatGPT – first, we transform the ChatGPT scores to pairwise judgment. We count it as “significantly better/worse” when the difference between ChatGPT scores is larger than 2, “slightly better/worse” for difference of 1, and tie in the case of 0. If ChatGPT has a high agreement score with the authors, that means the ChatGPT scores are comparable across evolved sets.

In the “significantly better/worse” sample pairs, the ChatGPT scores are highly consistent with the authors, producing 83% agreement rate, which means the ChatGPT scores are comparable across evolved sets when the scores are not very close. In the “slightly better/worse” or tie categories, the authors also assign “slightly better/worse” or tie labels to 96% of such samples, suggesting high coarse-grained consistency. However, the fine-grained agreement on the three labels (slightly better/worse, tie) is only 43% – we find that it is hard even for humans to be consistent on comparing the slightly different complexity of instructions, especially when the two instructions are not relevant.

We tried to repeat the above experiments for quality scores as well, but we found that it is really difficult for humans to compare the quality of two responses that correspond to two very different query instructions.

Q5: Is it possible for the diversity criterion to end up selecting outliers in the datasets? How can this issue be fixed?

We agree with the reviewer that the diversity criterion may select outliers in the dataset. . However, we think the outliers may not be always bad – being diverse and infrequent, the outliers may actually increase the diversity of the selected SFT dataset. Although outliers in a dataset are often considered as low-quality samples, our score-first data selection approach should have filtered out the low-quality samples in the first step.

Q6: Can you use a different (smaller?) model for computing distances than the one you instruction-tune?

We have performed ablation analysis on the embedding model in Appendix C of the original submission. Figure 4 shows that a smaller model, E5-Large-v2 (a SOTA sentence embedding model), is more sensitive to the threshold $\tau$, and its performance exhibits a significant decline compared to LLaMA..

Q7: DEITA-6K loses more to the random selection baseline than to the Vicuna model according to the human evaluation

We agree that this is a bit surprising, and we think this may be attributed to the simplicity of the LIMA test set, which fails to effectively distinguish between models that generally perform well. It is worth noting that the random baseline is not weak, as it is derived from a SOTA data pool. This situation highlights a broader challenge in human evaluation: for complex queries like those in MT-bench, human evaluation becomes difficult and costly, even for expert annotators, due to the complex examples which often include extensive mathematical and coding content. Conversely, with simpler queries like the LIMA test, human evaluators struggle to differentiate between strong models that excel in simple scenarios. To provide more clarity, we have included a detailed breakdown of the MT-bench results (based on GPT-4 evaluations) in Figure 4 of our paper. This breakdown clearly shows that the random baseline based on Mistral-7B lag significantly in areas such as coding, mathematics, and reasoning, aspects that are less prominent in the LIMA test and challenging for human evaluators to assess.

[1] Beeching et al. Open LLM Leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard. 2023. [2] Tunstall et al. Zephyr: Direct Distillation of LM Alignment. https://arxiv.org/abs/2310.16944. 2023.
[3] Zhou et al. LIMA: Less Is More for Alignment. https://arxiv.org/abs/2305.11206. 2023.
[4] Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. https://arxiv.org/abs/2306.05685. 2023.

Thanks for your helpful comments and we are glad that you acknowledged the strengths of our work! Below we will address your concerns point by point:

Q1: Limited evaluation and more results on target benchmarks

Thanks for the comments. We have added more evaluation results on the Open LLM Leaderboard benchmark [1] that consists of four tasks, ARC, HellaSwag, MMLU, and TruthfulQA. Additionally, we have enriched our evaluations by training more pre-trained LLMs on our selected data, such as LLaMA-2-13B and Mistral-7B besides LLaMA-1-13B in the original submission. We have summarized these results in the table of the general response, while the detailed results are in Table 5,6,7 of the paper. On the Open LLM Leaderboard, our approach outperforms the random baseline and other strong SOTA models across three different backbone models in terms of the average results. In the meanwhile, we would like to note that the performance on benchmarks such as MMLU is largely determined by the underlying base model, thus the gains after alignment tuning is typically not large, unless the training data is specifically crafted for the benchmark. This is why recent works on alignment put more emphasis on MT-bench and AlpacaEval [2].

Notably, on MT-bench and AlpacaEval, our added results show that our method with the LLaMA-2-13B model surpasses Vicuna-13B-1.5 that trains on 10x more data, and LLaMA-2-13B Chat that has undergone extensive RLHF training. With Mistral-7B, our approach attains an MT-bench score of 7.29 with only 10k training data, marking a state-of-the-art performance on MT-bench for all open-source SFT models at 7B and 13B sizes. We also observe that, while the gains of the Mistral DEITA model on AlpacaEval are slightly lower than those on MT-bench, a deeper analysis (as depicted in Figure 4) indicates that the Mistral DEITA model excels particularly in mathematical and complex reasoning tasks, areas not primarily covered by AlpacaEval.

Q2: GPT-4 evaluation bias

This is a good point. As the reviewer cited, Wang et al. 2023 points out that win-rate scores given by GPT-4 correlate with length of responses. However, in section 5.4 of their paper, they also admit that human evaluation results largely correlate with the AlpacaFarm and benchmark-based evaluation due to the same trend in the comparisons among their Tulu-65B model and other models. On the other hand, several previous works have demonstrated that GPT4 annotator greatly correlates with human annotator [3, 4] with agreement ranging from 78%~85%. Based on these findings, while we recognize the limitations of GPT-4-based evaluations, we believe that GPT4-based evaluation is still valuable and able to provide useful insights of the models. In our experience, these evaluations often yield more reliable outcomes than those derived from average human annotators.

Q3: Inter-annotator agreement of human evaluation

Thanks for the advice! We would like to note that we tried hard to control the quality of human evaluation in the original submission – as described in Appendix D, we adopted MTurk in the beginning but abandoned their annotations due to low agreement scores among them. To justify our human evaluation, we follow LIMA [3] to randomly sample 50 generation pairs from our human evaluation dataset and compute the author-annotator agreement score – the resulting agreement score is 77%, which is reasonably high and aligns closely with the agreement levels reported in [3]. We have added the inter-annotator agreement score to Appendix D of the paper.

Dear Reviewer NSy4,

We have revised the paper and added many additional results to address your comments. Since the rebuttal period is closing very soon, can you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you,
The authors