Rethinking Data Selection at Scale: Random Selection is Almost All You Need

Thanks for your helpful comments, below we will address your concerns point by point:

W1 & Q1: We find that previous data selection methods were carried out on small-scale data sources, such as the alpaca data (with 5,200 pieces of data). However, the open-source models, like Llama3 or Qwen2, can not complete the SFT process with just tens of thousands of data instances. When training instruct models, a much larger scale of data (hundreds of thousands or even millions) is often required. For example, Qwen2 utilized over 500,000 pieces of data during the Supervised Fine-Tuning (SFT) process.[1] . Therefore, conducting data selection on only tens of thousands or a hundred thousand data sources is a rather toy scenario. In order to endow the model with the ability to effectively follow instructions and improve the reasoning abilities of LLMs, it is often necessary to train the model with large-scale data. Thanks to the reviewer's suggestions. We will make corresponding modifications in the subsequent versions.

W2: When the data source is single, the homogenization of the data is rather serious. Especially when the scale of the dataset is small, in this case, there is actually no big difference between diversity-based selection and random selection. Therefore, selecting data by data quality can pick out higher-quality data.

W3: Our findings were obtained step by step through our experimental analysis. Firstly, our motivation is that we want to explore the differences in effects between different methods and random selection, and we found that random selection is actually better when facing large-scale data sources (taking into account effectiveness and efficiency). Then, we arrived at our findings2 by comparing the effects between the quality-based method and the diversity-based method. Finally，at the beginning of section 5.4, when we compared the results of the openhermes data source and the wildchat data source, we found that although the data quality of openhermes was higher, the effect of wildchat was better instead. Therefore, we discovered that long texts have a more obvious benefit for the model. So we further proposed a method of selecting data by token length. Later, from the experimental results, compared with Qwen2-7B (a strong LLM), the gain was more obvious for llama3-8B (a weak LLM). Thus, we concluded that long text training is more suitable for weak LLMs. (In summary, our empirical findings are yield step-by-step, well-motivated, and well-connected). Since random selection may not ensure the best fine-tuning results, we believe that screening by token length can avoid the uncertainties brought by random selection and at the same time obtain a relatively good training result.

[1] Yang A, Yang B, Hui B, et al. Qwen2 technical report[J]. arXiv preprint arXiv:2407.10671, 2024.

W4: We are grateful to the reviewers for their criticisms and corrections. We will revise the relevant modifiers to make them more rigorous. For the comparison of the effects between different baseline methods and the random method, we will add the corresponding significance test results in the subsequent versions.

• We conducted the Mann - Whitney U test for each method against the results of 5 rounds of random selection. We adopted the right-tailed test approach, with the testing hypothesis being that the scores of each baseline method on different test tasks are greater than those of the random method. We also reported the p - value for each method being significantly better than that of the random method. We found that the p - value of all methods is higher than 0.05, which indicates that the results of all baseline methods are not greater than those of the random method.

Clarification&Typos： Thanks to the reviewer's suggestions. We will make corresponding modifications in the subsequent versions.

Q2: For the classification criteria based on quality and diversity, we referred to Paper [2] and provided an explanation in lines 132 - 135 and line 218-219. The quality-based approach means that the method focuses more on designing an algorithm and evaluation metrics to calculate the score of each piece of data, and finally conducts screening according to the data scores. The diversity-based method, on the other hand, focus more emphasis on the diversity of the dataset. Thank you for your suggestions. We will add this part of the explanation in the Intro or Related Work section in the subsequent versions.

Q3: On the one hand, our experimental results indicate that long texts bring higher gains to Llama3. Moreover, the performance of the Llama3 series of models is somewhat inferior to that of the Qwen2 series[1]. On the other hand, in terms of the token length during the pre-training stage of the models, the pre-training token length of Llama3-8b is 8,192 [2], while that of Qwen2-7B is 32,768 [1]. Therefore, training Llama3 with long texts yields higher benefits. Besides, a longer token length means an increase in the overall training volume. Moreover, most of the response parts in the current SFT datasets are written by GPT-4. When the gap between the two models（GPT-4 and Llama3）is large, when we use the output of a more powerful model for training, the less capable the model is, the higher the gains it can obtain.

[1] Yang A, Yang B, Hui B, et al. Qwen2 technical report[J]. arXiv preprint arXiv:2407.10671, 2024.

[2] Qin Y, Yang Y, Guo P, et al. Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models[J]. arXiv preprint arXiv:2408.02085, 2024.

[3] Dubey A, Jauhri A, Pandey A, et al. The llama 3 herd of models[J]. arXiv preprint arXiv:2407.21783, 2024.

We have already submitted the revised paper and made corresponding modifications in response to the comments of all the reviewers.

Thanks for your helpful comments, below we will address your concerns point by point:

W1: We acknowledge that the effectiveness of random selection on Qwen2 - 7B is not the best. However, we hold the view that the criterion for determining the applicability of a method does not necessarily depend on the model's training outcomes. Although the final outcome of DiverseEvol is approximately 1.5% higher than the average effect of random selection (58.11), it takes 5 to 7 days for DiverseEvol to finish the entire data selection process. Here, a tradeoff exists between the effectiveness and time efficiency. The significant increase in time cost for a relatively small improvement in effectiveness makes such a time investment in DiverseEvol perhaps not worthwhile. Therefore, after comprehensively considering the balance between effectiveness and efficiency, we concluded that when dealing with large - scale data sources, random selection is the most advantageous option.

Besides, We conducted the Mann - Whitney U test for each method against the results of 5 rounds of random selection. We adopted the right-tailed test approach, with the testing hypothesis being that the scores of each baseline method on different test tasks are greater than those of the random method. We also reported the p - value for each method being significantly better than that of the random method. We found that the p - value of all methods is higher than 0.05, which indicates that the results of all baseline methods are not greater than those of the random method.

W2: Most of the previous work has been tested on LLMs with a scale of 7B or 14B. Considering the amount of our data and the time cost of the baseline methods, we finally only conducted experiments on two 7B models. In fact, during the data selection stage, we carried out data processing work simultaneously with 8 A100 GPUs. If we want to select 50K samples, IFD takes 1.5 days to complete the data processing of one model on one dataset, LESS needs 1 day, Select requires 1 day, cross entropy takes 0.5 days, ZIP needs about 7 days, and DiverseEvol on average takes 6 days. The whole process takes about 17 days (on the premise that there is no task queuing waiting involved). In the case of two models and two datasets, we need a longer time to reproduce all the baseline methods. Therefore, we did not conduct experiments on the scale of 14B in the main text.

Considering the time cost, we added some experiments on selectIT and ZIP based on Qwen2.5-3B and Qwen2.5-14B, as shown in the following table. Judging from the results, the performance of ZIP is about 0.8% higher than that of random（avg）, and is not significantly higher than random.

Q1: At the beginning of section 5.4, when we compared the results of the openhermes data source and the wildchat data source, we found that although the data quality of openhermes was higher, the effect of wildchat was better instead. Therefore, we discovered that long texts have a more obvious benefit for the LLM SFT. So we further proposed a method of selecting data by token length. Later, from the experimental results, compared with Qwen2-7B (a strong LLM), the gain was more obvious for llama3-8B (a weak LLM). Thus, (we conclude that) long text training is more suitable for weak LLMs. Since random selection may not ensure the best fine-tuning results, we believe that screening by token length can avoid the uncertainties brought by random selection and at the same time obtain a relatively good training result.

We have already submitted the revised paper and made corresponding modifications in response to the comments of all the reviewers.

Thanks for your helpful comments, below we will address your concerns point by point:

W1: We randomly selected 100k pieces of data on OpenHermes. Based on this, we further selected 10k pieces from this 100k data using different baseline methods as the training set for SFT. The results are shown in the following table. Compared with selecting from the full set, when selecting from 100k pieces of data, the quality-based method can achieve a certain improvement in performance. However, the performance of the diversity-based method has slightly declined. We believe this is because changing our data source from 1M to 100k has affected the data distribution of the data source to some extent, resulting in a slight decline in the effectiveness of the diversity-based method on this subset. We found that when the scale of the dataset becomes smaller, some of our baseline methods are only slightly better than the results of random selection. This might be due to the fact that the dataset we used is more diverse than the datasets used by these methods in their respective papers.

W2:

• Even if some methods perform well on a certain model, as you mentioned (Diverse and ZIP each score 1% higher than the max over the random runs in Tables 5 and 6 respectively), the time cost they require is still extremely high compared to random selection. When selecting 50k pieces of data, both of these methods take about seven days (using 8 A100 GPUs). Therefore, we believe that the random method is more suitable for data screening of large-scale data sources.

• We presented the results of five rounds of random selection in the hope of demonstrating that the effectiveness of most of the current baseline methods is actually within the range of random selection. Combined with the results of the significance tests we will mention next, none of the current baseline methods are significantly better than random. Therefore, we believe that it is not necessary to conduct multiple random selections in practice.

• We conducted the Mann - Whitney U test for each method against the results of 5 rounds of random selection. We adopted the right-tailed test approach, with the testing hypothesis being that the scores of each baseline method on different test tasks are greater than those of the random method. We also reported the p - value for each method being significantly better than that of the random method. We found that the p - value of all methods is higher than 0.05, which indicates that the results of all baseline methods are not greater than those of the random method.

W3: In LESS, $\tilde{\Gamma}(\boldsymbol{z},\theta)$ represents the gradient values of different data $\boldsymbol{z}$ under different optimization states $\theta$. In ZIP, $\mathcal{C}$ means the compression ratio. We are sorry to make you confused, and we will make adjustments in the subsequent versions according to your suggestions. Thank you for your suggestions.

Miscellaneous Notes 1: yes， it means "second highest score"， we will make adjustments in the subsequent version.

Miscellaneous Notes 2: In the original paper, LESS involves selecting a part of the data for each task to train an LLM specifically targeted at a particular task. In this paper, our aim is to select a subset that can be adapted to any tasks. Therefore, we have modified a part of the code. On the one hand, we believe it does not conform to the application scenarios of LLMs at the current stage, the goal of Supervised Fine-Tuning (SFT) should be to enable the model to unlock different capabilities to adapt to different downstream tasks. The original intention of LLM SFT is to improve the generalization ability of the model. Previous works [1, 2] involved selecting a subset of data from the source dataset and then conducting evaluations on different tasks. On the other hand, considering the time cost and resource consumption of data selection, we did not train a separate model for each task.

Q1: In Figure 1, "Data size 10K - 300K" refers to the data sources used in the original texts of different methods. Among them, LESS uses the collection of FLAN V2 COT, DOLLY, and OPEN ASSISTANT 1 datasets, with a total of over 270,000 pieces of data. IFD and diverseEVol use the alpaca dataset with 5.2k data. SelectIT used alpaca-gpt4 datasets with 5.2k data. ZIP used 300K samples obtained from WizardLM, ShareGPT and UltraChat. "Data size 1M" refers to Openhermes2.5-1M dataset.

Q2: During the fine-tuning stage, we conduct full-parameter fine-tuning using 8 A100 GPUs. If the number of fine-tuning data is 10,000 pieces, it will take approximately 30 minutes. If the number of fine-tuning data is 50,000 pieces, it will take about 2.5 hours. In fact, the time cost in the fine-tuning stage is actually quite low. The most significant one is the time cost in the data selection stage. During the data selection stage, if we want to select 50K samples, apart from the random method, the fastest method is cross entropy, which takes approximately 15 hours to process one million pieces of data simultaneously using 8 A100 GPUs. IFD takes 1.5 days to complete the data processing of one model on one dataset, LESS needs about 1 day, SelectIT requires about 20 hours, ZIP needs about 7 days, and DiverseEvol on average takes 6 days.

[1] Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685, 2023

[2] Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032, 2023b

We have already submitted the revised paper and made corresponding modifications in response to the comments of all the reviewers.

Thanks for your helpful comments, below we will address your concerns point by point:

W1: Our paper focuses more on the comparison between different methods. The main conclusion finally drawn is that when faced with large-scale data sources, the effectiveness of most data selection methods does not actually significantly exceed the results of random selection. The first reference paper you mentioned proposed the QDIT method, but it only compared with random selection and did not compare with any state-of-the-art (SOTA) methods (and the first reference only talks about data diversity-based method, while our paper include discussions on both data diversity-based and data quality-based methods). Similarly, the second reference paper also did not compare with any SOTA methods and only conducted experiments on small-scale datasets. In summary, our paper focus more on the experimental analysis of different methods based on large-scale data source, and thus draws conclusions. Although the conclusions are similar, our focus is different from that of these two articles.

W2: In Section 5.3, we analyzed the unavailability of different methods from multiple aspects. Some aspects involve the inherent flaws of the methods themselves. For example, as pointed out in lines 414 to 415 of our article, the baseline method IFD tends to preferentially select data with shorter queries, which is consistent with the research findings reported in Paper [1]. In addition, some baseline methods are not suitable for data screening of large-scale data sources. For instance, as described in lines 429 to 430, the DiverseEvol method has a relatively high time cost (It needed average 6 days to finish the selection stage). Our experimental analysis comprehensively validates our claim that the random selection method is more suitable for selecting data from large-scale data sources.

W3: We consider that large-scale data sources refer to datasets with a data volume at the million level and diverse data sources (For example, annotated by dedicated workers, sourced from real users, and synthesized with models) as well as rich data types (For example, code data, math data, conversations, knowledge Q&A, etc.). Thanks to the reviewers' suggestions, we will include this definition in the subsequent versions.

[1] Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685, 2023

We have already submitted the revised paper and made corresponding modifications in response to the comments of all the reviewers.

Thank you for your response and the additional clarification—it has helped me gain a better understanding of the paper. However, I still feel that the contribution of this work is somewhat limited. The analyses presented appear to be similar to those in existing works, offering limited novel insights or deep analysis. Additionally, there is no formal exploration of the relationship between the optimal strategy and the amount of data, which I believe is an important aspect to address. I choose to keep my score at this time, but encourage the authors to further improve the current manuscript.

Thank you for your comment. We'd like to add some more points.

We believe that when judging the novelty of an article, it is not appropriate to take it out of context and it is not fair to assess the novelty of an article solely based on its partial contributions, but rather from the perspective of the full contributions.

Regarding the evaluation tasks, the first paper and our paper mainly focus on eliciting LLMs' reasoning and code generation abilities via SFT, while the second article mainly focuses on the alignment task. Therefore, it is not reasonable to predict that the length-based data selection method can yield better results in SFT downstream tasks as there exists a length-based method that performs well in alignment tasks. In general, the differences between reasoning and alignment tasks are significant.

From the perspective of motivation, the first article mainly seeks better data selection methods from the perspective of effectiveness, and the second article is inspired by the fact that GPT-4 prefers giving text with longer lengths higher scores in alignment evaluation. However, our paper is looking for data selection methods with better comprehensive performance from the perspective of the balance between effectiveness and efficiency.

In terms of the analysis part, the first article only compared the data-diversity part and data-quality part of its own method on large datasets, but we compared several different diversity-based methods and quality-based methods on million-level SFT datasets, our analyses are more comprehensive. The second article only verified the performance of the proposed length-based data selection method on small-scale datasets by data slicing, but our proposed method was inspired through analyzing the characteristics of the WildChat and OpenHermes datasets in combination with the subsequent experimental analysis of the token length part. We simultaneously compared and analyzed the performance advantages and disadvantages of the data quality-based, data diversity-based, and random selection methods on large-scale datasets. We believe that in the era of large models, it is more meaningful to conduct experimental analysis on large-scale datasets.

From the perspective of conclusion, the main contributions of these three papers are different. We obtained our three findings step by step through experimental analysis, and the way we conduct experiments is different from those used in the two references.

The paper [1]‘s conclusion is: There is a natural tradeoff between data diversity and quality.

The paper [2]'s conclusion is: A lightweight refinement of long instructions can further improve the alignment abilities of the LLMs.

Our conclusions are:

1. Most self-scoring data selection techniques do not significantly outperform random selection on large-scale datasets.
2. Data diversity holds more significance than data quality during the SFT phase.
3. It is useful to utilize token length as a criterion to conduct data filtering, yielding stable and efficient results for SFT when dealing with large-scale IT data.

In summary, our conclusions are more comprehensive.

[1] Bukharin, Alexander, and Tuo Zhao. "Data diversity matters for robust instruction tuning." arXiv preprint arXiv:2311.14736 (2023).

[2] Zhao, Hao, et al. "Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning." Forty-first International Conference on Machine Learning.

We are very grateful for the suggestions put forward by the reviewer 2efs. In the research direction of SFT data selection, neither our method nor some previous works have paid attention to the issue of data overlap between the training set and the test set. We agree with your point of view and will focus on the data overlap issue in our subsequent research.

Regarding the issue of token overlap you mentioned, we believe that it is inappropriate to count the overlap just between tokens. Because individual tokens don't have any practical meaning on their own. We calculated the n-gram overlap degree (n = 15) between the data subsets screened by different methods and the test data of different tasks. If the same n-gram tokens appear between two pieces of data, we consider this part of the data to be the same. The results are shown as follows, the numbers in the table represent the proportion of token-overlapping samples in each test set, (overlap samples/total test samples). We found that the proportion of overlapping samples between the training set and the test set is very low. Therefore, the impact of overlapping samples on the test tasks is actually very weak.

We have already submitted the revised paper and made corresponding modifications in response to the comments of all the reviewers.