SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection

Thank you for your insightful review.

Q1: Figure 2 is not clear to me. I suggest giving a brief description of each selection stage in the caption for easy understanding.

Thank you for your advice. Following your suggestion, we will revise the caption of Figure 2 to: Overall Framework of Our Proposed SelectIT Method. In the Token-level Self-Reflection, we employ the foundation model to rate the IT data from 1 to $K$. For Sentence-level Self-Reflection, we leverage the uncertainty of varied prompts on LLMs to enhance the rating process. In the Model-level Self-Reflection, we harness uncertainty among different LLMs to facilitate a collaborative decision-making process in selecting IT data. Finally, different levels of self-reflection are reasonably combined into SelectIT, which can effectively select high-quality instruction tuning data without relying on additional resources.

Q2: It seems like the foundational LLM's capacity becomes the key for this data selection. In this case, the selection method may not be as effective as other external resource methods.

• Comparison with other external resource methods: In Table 1, we compare SelectIT with various other external resource methods such as AlpaGasus [1], Q2Q [2], and Instruction Mining [3]. SelectIT demonstrates superiority in the abundant experiments including six different tasks and varying model parameter sizes. Furthermore, as shown in Table 7, SelectIT requires less time and fewer resources compared to AlpaGasus or its variant. Therefore, we have grounds to believe that leveraging the capacity of LLMs without external resources is sufficient for selecting high-quality samples.

• Emergence of more powerful LLMs: Based on the main result of Table 1, the LLaMA-2 series base models we used have exhibited sufficient capability in selecting higher-quality samples. With the continuous development of our LLM community, more powerful open-source LLMs are emerging, such as LLaMA-3.1, Mistral-7B-v0.2, and Qwen2 etc. Given that the performance of SelectIT is intrinsically connected to the capacities of foundation LLMs, its efficiency is not only unrestricted but also anticipated to enhance concurrently with the advancement of LLMs in the future.

[1] AlpaGasus: Training A Better Alpaca with Fewer Data. ICLR, 2024

[2] From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning. NAACL, 2024

[3] Instruction Mining: Instruction Data Selection for Tuning Large Language Models. arXiv, 2023.

Q3: How does SelectIT perform when applied to instruction tuning datasets other than Alpaca-GPT4?

In Section 5.1 of "Various Domain-specific Tasks", SelectIT is also applied to different datasets such as machine translation datasets. As illustrated from Lines 267-272 in the paper, following the configuration of ALMA[1], we opt for a total of 30K training examples for representative language pairs and utilize SelectIT to select a subset of 6K data for comparative training. As Table 6 demonstrates, SelectIT can invariably boost the translation performance of ALMA [1] using fewer yet higher-quality training data.

To further alleviate concerns, we apply SelectIT to other instruction tuning datasets such as WizardLM and larger Orca-GPT4, while keeping the hyper-parameters consistent. For more details, please refer to the Q1 of General Response.

[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ICLR, 2024.

Q4 : Can the authors provide more detailed examples or case studies illustrating the differences between data selected by SelectIT and other methods? And how these different influence the final model performance?

In Figure 5, we have comprehensively discussed the differences in the samples selected by SelectIT and other methods such as AlpaGasus.

• Length: Relative to AlpaGasus and the full set of Alpaca, the average sample length of SelectIT is the largest, reaching 241.85. Length is a crucial dimension in assessing sample quality [1]. Longer sentences typically contain more information or procedural steps, which are beneficial in enhancing model capabilities.

• Calculation: As shown in Lines 61-62 and Figures 5-6, one of the primary contributions of SelectIT is that it prefers additional calculation-type IT data which enhances the LLMs' reasoning capabilities, offering a new perspective on the characteristics of optimal IT data. As demonstrated in Table 1, SelectIT achieved optimal results on benchmarks assessing computational and mathematical capabilities. The enhancement in model capabilities further improved the LLMs' coding abilities, ultimately leading to an overall enhancement in comprehensive capabilities.

Furthermore, in Appendix Section A.4, we have presented relevant case studies contrasting SelecIT and AlpaGasus. In samples 1 to 4 of Figure 8, SelectIT demonstrates a preference for instruction-tuning data featuring complex mathematical problems, aligning with the findings of Section 5.4. In contrast, AlpacaGasus usually selects IT data predominantly providing solutions to queries or lacking coherent reasoning in samples 5 to 7, potentially hindering its effectiveness.

[1] Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning. arXiv, 2024.

Q5: How does the performance of SelectIT change with different values of the uncertainty factor (α) ?

Please refer to the Q3 of General Response.

Dear Reviewer 4rCk,

We greatly appreciate your valuable feedback and suggestions! To address your concerns, we have run extensive experiments on various datasets and made additional insightful observations. The results will be a nice plus to improve the clarity and useability of our method.

As the discussion period is coming to an end, we kindly request that you review these results. If our results and explanations help address your concerns, we would be grateful if you could acknowledge our rebuttal and consider adjusting your score accordingly.

Best wishes,
The Authors

Dear Reviewer 4rCk,

Thank you for your time reviewing the paper. Your constructive feedback will help improve the quality of our paper.

We have also addressed all your concerns in our rebuttal point-by-point. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be more than happy to address any more concerns you have.

We thank you again for your time!

Best,
The Authors

Thank you for your response.

We will give a clearer explanation of SelectIT's preference for longer instruction data.

As shown in the "Introduction" section and "Insights of Selective Data Curation" section, the core feature of Selective Alpaca is the increase in the proportion of computational data, which is different from other methods. Computational data is often longer than other types of data because it requires more reasoning steps, which further leads to the longer overall length of Selective Alpca. So, "Length" may still be the important and easy-to-get feature for evaluating the quality of sample data [1-4].

Secondly, SelectIT does not simply prefer longer data. As shown in Table 8 of our paper (also the table below), we compared the SelectIT with the previous work [1], which selects samples only by length. The results show that SelectIT can achieve better results with the same amount of data.

Finally, we further compared the output lengths of models trained on Selective Alpaca and Alpace-GPT4 on the Alpaca-eval benchmark. As shown in the following table, with the output lengths being similar, Selective Alpaca achieves a 1.5% improvement, which effectively eliminates the bias in sample length and explanation of the doubts about the preference of LLMs for samples with longer length.

Thank you again for taking the time to read our response! We are more than happy to continue the discussion if you are still confused about the paper.

[1] Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning. arXiv, 2024.
[2] Zero-Shot Generalization during Instruction Tuning: Insights from Similarity and Granularity. arXiv, 2024.
[3] LIMA: Less Is More for Alignment. NeurIPS, 2023.
[4] AlpaGasus: Training A Better Alpaca with Fewer Data. ICLR, 2024.

Thank you for your insightful review.

Q1: From the table 3, it seems Sentece-R works the best so what if we trained model on the data just from the Sentece-R? BTW, what is the ID 6, 7, 8 means in that Table?

Thank you for your suggestion. Actually, in Table 1, we have provided the experiment that trained the model on the data from Sentece-R, referring to System 7 and System 16. The results in Table 1 indicate that Sentece-R achieved the best outcomes among three different level methods (Token-R, Sentece-R, Model-R) of SelectIT, which aligns with the conclusions in Table 3 and further dissects the key aspects of SelectIT performance.

Additionally, IDs 6, 7, and 8 in Table 3 correspond to the system of the same IDs in Table 1, which means using the same methods to select data. We will provide clearer explanations for Table 3 in the next version.

Q2: My main question is that do we really need all three types selection models (method), can one or two of them already achieve great performance?

There might be an ambiguity that the three types are either the three different levels of self-reflection strategies or the three different bases models, we will address both aspects as follows.

• Different Levels of Self-Reflection Strategies: Based on Table 1 and Table 3, Sentence-R performs better than other select methods, such as Token-R and Model-R, which is attributed to the use of different prompts to quantify the uncertainty of the LLMs. So the single method, Sentence-R is good enough to select data. Furthermore, we integrate various levels of self-reflection: token, sentence, and model, to form the final SelectIT, achieving the best performance.

• Different Types of Bases Models: As you expect, SelectIT still performs well when there are only one or two base models. Based on Table 1 and Table 3, Sentence-R plays the most important role in SelectIT, which only uses the single base model. Model-R, as an extension of SelectIT, allocates weights to the result of each model based on the number of model parameters for data selection, mainly enhancing the robustness of the algorithm.

Additionally, we will supplement the experiments which using varying numbers of models to select the instruction data, with the results presented below:

Based on the above analysis and experimental results, it is evident that SelectIT still works well when there is a single base model, referred to as the Sentece-R. More base models are also beneficial for selecting high-quality data, which can effectively enhance the robustness of SelectIT.

Q3: What is the overhead of this data selection model? Like how long does it take to finish the process on 10K dataset.

In Section 5.3, we have elaborated on and compared the efficiency of SelectIT with other methods. Based on the Alpaca dataset, we deployed 4 A800 80G GPUs to select high-quality IT data, which ultimately required 5.8 hours. In contrast, utilizing the ChatGPT API required significantly more time, 19.07 hours. This issue is even more pronounced with the more powerful GPT-4, demonstrating the practical utility of SelectIT. For the 10K dataset, keeping the same GPU devices as above, it would necessitate 1.2 hours.

Dear Reviewer CKC2,

We greatly appreciate your valuable feedback and suggestions! To address your concerns, we have run extensive experiments on various datasets and made additional insightful observations. The results provide more evidence for our method justification.

As the discussion period is coming to an end, we kindly request that you review these results. If our results and explanations help address your concerns, we would be grateful if you could acknowledge our rebuttal and consider adjusting your score accordingly.

Best wishes,
The Authors

Dear Reviewer CKC2,

Thank you for your time reviewing the paper. Your constructive feedback will help improve the quality of our paper.

We have also addressed all your concerns in our rebuttal point-by-point. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be more than happy to address any more concerns you have.

We thank you again for your time!

Best,
The Authors

Thank you for your response.

For the Model-R of SelectIT, we used models to jointly determine high-quality data and reasonably weigh the sample quality according to the model parameters, which is similar to ensemble learning. As shown in "Insights of High-Quality Data in SelectIT" section and Figure 5, SelectIT which used multiple models is more conducive to screening data with biased calculation types. So, SelectIT can improve model reasoning ability and further improve the encoding ability of LLM, which ultimately leads to an overall improvement in comprehensive ability.

Secondly, MMLU consists of a set of questions about 57 subjects and its multiple-choice format makes it suitable for probing models’ knowledge [1-2]. So it is often used to evaluate the foundation model competence, as LLMs mainly learn world knowledge in the pre-training stage, not the instruction tuning stage (only containing a few instances). So there is a slight improvement on MMLU which is relatively reasonable in SelectIT, and also consistent with the results of AlpaGasus [3].

Thank you again for taking the time to read our response! We are more than happy to continue the discussion if you are still confused about the paper.

[1] Measuring Massive Multitask Language Understanding. ICLR, 2021.
[2] How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. NeurIPS, 2023.
[3] AlpaGasus: Training A Better Alpaca with Fewer Data. ICLR, 2024.

Thank you for your insightful review.

Q1: The paper states initially that it does not use extra resources for its new method. However, it mentions later that they used different sizes of LLaMA 2 models to help choose the best instruction data. This contradicts their first claim about not needing extra resources.

The 'Main Results' and 'Effect of Different Reflection Strategy' sections highlight the effectiveness and importance of Sentence-R in SelectIT. Serving as a highly adaptable method, SelectIT can handle varying quantities of base models. In instances where there is a single base model, SelectIT leverages Sentence-R for data selection without requiring additional models or data. However, when multiple base models are present, SelectIT appropriately allocates weights to the result of each model based on the number of model parameters for data selection, thereby boosting the algorithm's robustness.

In summary, SelectIT possesses the ability to use the base model directly for data selection without needing extraneous resources and it could achieve comparable performance with only using one base model.

Q2: Could you consider applying the SelectIT method to larger datasets, such as the Orca dataset for ChatGPT with 5 million samples and the GPT-4 dataset with 1 million samples? This would help to evaluate whether your method scales effectively with increased data volume.

In Section 5.1 of "Various Domain-specific Tasks", SelectIT has applied to different datasets such as machine translation datasets. As illustrated from Lines 267-272 in the paper, following the configuration of ALMA [1], we opt for a total of 30K training examples for representative language pairs and utilize SelectIT to select a subset of 6K data for comparative training. As Table 6 demonstrates, SelectIT can invariably boost the translation performance of ALMA [1] using fewer yet higher-quality training data.

To further alleviate concerns, we apply SelectIT to other instruction datasets such as WizardLM and larger Orca-GPT4, while keeping the hyperparameters consistent. The results are shown in the following table.

Based on various task types, including instruction tuning (IT), machine translation (MT), and diverse instruction datasets such as Alpaca, WizardLM, and the larger Orca-GPT4, extensive experimental results have robustly validated the general applicability and robustness of SelectIT.

[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ICLR, 2024.

Q3: The paper lacks a separate Section on the limitations of the approach.

Sorry for the misunderstanding. We have already provided the "Limitations" Section in Appendix A.1 of our original submission. Later, we aim to integrate this section into the main text to provide a clearer understanding of SelectIT to audiences.

Thank you for your insightful review.

Q1: Reliance on Tuning K: The method depends on tuning the parameter ( K ), which can significantly affect model performance. Will optimal K be training dataset dependent?

The performance of SelectIT is not critically dependent on the parameter $K$, based on the conclusions of Section 5.1 "Effect of $K$ ". As illustrated in Figure 2, when $K$ is sufficiently large, such as 5, 7, or 9, the impact of parameter $K$ on LLMs performance is negligible, with performance fluctuations around just 0.3. This is due to the multi-level self-reflection mechanism, which effectively enhances the accuracy and stability of sample selection.

Furthermore, based on Section 5.2, we also apply the SelectIT to the representative domain-specific task, machine translation (MT), which reuses the same hyperparameters $K$ and $\alpha$. As shown in Table 6, we present outcomes for two distinct language pairs in four distinct directions and, SelectIT can consistently enhance the translation performance of ALMA [1], which is widely recognized as a leading machine translation LLM.

Finally, referring to the response to Q1 of General Response, we also apply the same hyper-parameters of $K$ on the other datasets such as the WizardLM and larger Orca. The results demonstrate that the same $K$ hyper-parameters of SelectIT still yield consistently strong performance across different datasets. Therefore, extensive experimentation is not required to adjust K hyper-parameters, and it is independent of the training dataset.

[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ICLR, 2024.

Q2: Use of Multiple Base Models: Although the paper claims not to use external resources, it actually relies on multiple different base models to obtain the model-level score. This reliance contradicts the claim of being purely self-reflective and can be confusing, as the title suggests the method is entirely self-contained.

Please refer to the Q2 of General Response.

Q3: Sentence-Level Quality Evaluation: The sentence-level quality evaluation requires multiple rating prompts. It is unclear whether other methods, such as AlpaGasus, would achieve similar performance if they also used multiple rating prompts. This comparison is missing and could be critical for understanding the true advantage of the proposed method.

Thank you for your advice. We will provide a clearer explanation of Sentence-Level Quality Evaluation. To the best of our knowledge, we are the first one to use the effect of different prompts on LLMs to select high-quality data. Following Figure 2 and Equation 4, we do more than use K times prompting and averaging the rating score. More importantly, we calculated the standard deviation of the rating score to quantify the model's uncertainty, which is not only the first attempt in our work but also an important contribution to our work.

For the previous techniques, we compare Selective Alpaca with the official open-source dataset [1] in Table 1, which we think is fair enough. Additionally, based on the resource consumption we reported in Table 7 of the efficiency of SelectIT, using ChatGPT to K times-ly selects IT data is significant consumption and challenging to implement, which is also proved by recent research [2].

[1] https://github.com/gpt4life/alpagasus

[2] Liu et al., What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning. ICLR 2024.

Q4: Single Dataset Evaluation: All evaluations are conducted on a single training dataset, making it questionable whether this method can be generalized to other datasets, such as WizardLM, or multi-turn data, like the ShareGPT dataset. Testing on a more diverse set of datasets would strengthen the paper's claims of robustness and general applicability.

Sorry for the misunderstanding. In Section 5.1 of "Various Domain-specific Tasks", SelectIT has applied to different datasets such as MT datasets. As illustrated from Lines 267-272 in the paper, following the configuration of ALMA [1], we opt for a total of 30K training examples for representative language pairs and utilize SelectIT to select a subset of 6K data for comparative training. As Table 6 demonstrates, SelectIT can invariably boost the translation performance of ALMA [1] using fewer yet higher-quality training data.

To further alleviate concerns, we apply SelectIT to other instruction datasets such as WizardLM and larger Orca-GPT4. For more details, please refer to the Q1 of General Response.

[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ICLR, 2024.

Q5: How the $\alpha$ = 0.2 in the experiments is selected?

The value of $\alpha$ is determined through our analytical research on Sentence-level Self-Reflection. To further justify the appropriateness of $\alpha$ = 0.2, we will conduct supplementary experiments on varying α values of SelectIT on the Alpaca datasets. For more details, please refer to the Q3 of General Response.

Dear Reviewer KKrK,

We greatly appreciate your valuable feedback and suggestions! To address your concerns, we have run extensive experiments on various datasets and made additional insightful observations. The results further confirm the generality of our proposed method.

As the discussion period is coming to an end, we kindly request that you review these results. If our results and explanations help address your concerns, we would be grateful if you could acknowledge our rebuttal and consider adjusting your score accordingly.

Best wishes,
The Authors

Dear Reviewer KKrK,

Thank you for your time reviewing the paper. Your constructive feedback will help improve the quality of our paper.

We have also addressed all your concerns in our rebuttal point-by-point. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be more than happy to address any more concerns you have.

We thank you again for your time!

Best,
The Authors