Instruction Mining: Instruction Data Selection for Tuning Large Language Models

Hi, thanks for your response and the additional experiments! Thank you for the clarifications, they are useful.

• Correlation between loss & downstream performance: Thanks for these results! I guess your strong MT-Bench results in figure 3 suggest that optimising for loss works for improving 'real' MT-Bench results. It still would be nice to get some idea of how large of a +/- change in loss you need to see a difference in performance on MT-Bench. I guess this also suggests your derived rule is focusing on conversational ability, rather than other model capabilities - you would have to test if learning a new rule would work for improving e.g. multiple-choice ability (MMLU).

• Different seeds: I was especially curious about the variance in results over downstream metrics, either the gpt-annotation-based version of MT-Bench or open-llm benchmark scores (although, as you noted, if your method works better for general conversation ability, then just examining mt-bench variation is fine).

• Full Dolly results: Thank you for these - examining Table 3 in the paper, it seems like your method does generally better in mt-bench loss than training on the entire set, which is interesting!

I have carefully read the other reviews and responses and am keeping my score.

[Continued]

• W.3 & Q.4 Models trained on full dataset.

Orca full is too large for us to train at this moment, here we provide our updated result on full dolly(15k) dataset.

• W.1 The correlation between LLM-as-a-judge scores and loss.

In table4, we bring loss and LLM-Leaderboard together:

We would like to kindly point out that, since our optimization goal is to minimize the distance between the finetuned model and the target model, hence, GPT-4 in our paper, it is possible that MMLU does not correlate much with the loss. A straightforward intuition is that, MMLU is more like a question answering task, while our loss minimization goal aims to improve the model’s ability in conversation. In this case, we choose to relate loss with mt-bench, which is also LLM-as-a-judge scores to prove that loss can serve as a suitable indicator for model performance.

We are still downloading model weights and running experiments on 7B open-source models. We will provide the experiment results as soon as it is available.

• W.4 Name explanation.

Thanks for pointing out the ambiguity in our paper. In Table 4, both InstructionMining-Selected and InstructionMining-Random are over Orca.

Thanks again for your constructive feedback. We feel truly encouraged that you like the idea and find it insightful.

Reference:

[1] Zeng, Zhiyuan, et al. "Evaluating large language models at evaluating instruction following." arXiv preprint arXiv:2310.07641 (2023).

We apologize for the delayed response. Please note that some of our experiments and evaluations require several days to complete, which may result in a brief wait time before we are able to provide a reply. Thank you for your patience and we appreciate your time and effort in reviewing our paper and providing valuable feedback. We are pleased to hear your positive remarks regarding the novelty and practical applicability of our method for selecting high-quality instruction following data. In response to your concerns, we have outlined our comments below:

• Q.1 Difference in loss:

It’s hard to say the numerical relevance between loss and performance difference, here is the OpenLLM leaderboard result for random and selected 10k data of dolly.

However, we would like to point out that, during our experimentation, we feel that loss on mt-bench, also LLM-as-a-judge, is only able to evaluate a model’s conversational ability, while a language model still needs other abilities, e.g., question answering. And this is why we add OpenLLM benchmarks. From our point of view, Mt-bench and MMLU are two separate evaluation benchmarks, which does not correlate much. For example, a response below could yield high LLM-as-a-judge score and low loss, however, it is not a correct answer to this question[1].

Instruction: Sort the following list into alphabetical order. apple, banana, orange, grape.

Wrong Answer: No problem! Here’s the sorted list. Grape, apple, banana, orange.

Correct Answer: apple, banana, grape, orange.

• Q.2 Results on random selection experiments.

Thanks for pointing out. We will update more random results. These results and analysis will be added to our paper. Below is the detail for the appended random experiment results.

We summarize this table in to the analysis below:

• Q.3 Section 5.2, the usage of instruction rule.

Yes, we use data with rule estimated by 7B model to train 13B model.

We thank Reviewer TXJx for the thoughtful feedback. Your positive comments on our writing and our method for selecting data of superior quality is truly encouraging. For your concerns and questions for our work, please find response below.

• Q.1.1 Choice of D-eval.

Thanks for pointing this out. Indeed, we recognize that we didn’t explain in detail why we selected another D-eval, instead of just choosing an evaluation subset from the training data. When selecting D-eval, We want

1. D-eval to have high-quality instruction following examples in it.
2. D-eval to be diverse enough, hence, it should have: (a). various instructions to test the model’s generalizability; (b). High-quality responses.
3. D-eval to be relatively small, so that we can conduct the experiments efficiently.

At the time of submission, we only found two suitable, human labeled instruction datasets, self-instruct and mt-bench. To make sure that our selected D-eval satisfies these requirements, we also re-annotated self-instruct dataset using GPT-4 to get high-quality responses.

• Q.1.2 Distribution difference between D-eval, target tasks, and D

We agree that if we directly apply this rule to select data of other target tasks, our rule will probably not work as expected. However, we would like to point out that our paper mainly focuses on instruction tuning, hence, teaching the model to follow instructions. The datasets we choose to train the model and fitting the rule and mostly instruction-following data, and we mainly focus on tuning the model to make it better at following instructions. In this case, our task is general instruction following, which aims to tune an LLM to follow instructions. So there should not be much difference in the distribution of D-eval and target tasks.

• Q.1.2 How to select and estimate the effect of instructions with a biased/smaller dataset.

We agree that if the D-eval is biased or too small, our method might not work under these circumstances. We want to kindly point out that our method is highly dependent on a suitable, high-quality D-eval dataset. A strong hypothesis in this paper is that D-eval is of relatively high quality. In this case, selecting and estimating instruction quality using a biased D-eval is out of our current scope. However, selecting data when D-eval is biased or too small is of course an important research topic, and hopefully in our future research, we can keep on improving our method to make it more robust, towards different D-eval sets.

• Q.2 The correlation between selected D and D-eval.

We acknowledge that using D-eval for fitting the rule and then using the rule to select data will possibly result in overfitting on the D-eval. This could harm performance if D-eval is of low-quality. However, if D-eval is of high-quality, the selected D should be of similar quality. Hence, our method is strongly dependent on the hypothesis that D-eval is of high-quality. To ensure its quality, which is the self-instruct dataset in our paper, we re-annotated self-instruct dataset using GPT-4, to ensure that the label of self-instruct is of high-quality.

• Q.3 Training loss, data quality and output length.

Instead of using training loss, we use model inference loss on D-eval, so D-eval is an unseen dataset to the model. We recognize that inference loss cannot accurately represent a generative model’s performance. However, inference loss is still an indicator of the textual similarity between responses generated by GPT-4 and the trained model, and it is able to indicate data quality to some extent, as shown in Table 8. Choosing a suitable metric to evaluate a model’s performance or data quality requires us to consider both representability and efficiency, which still remains an unsolved problem in the research area. Inference loss is like a “compromise” that it is able to represent a model’s ability in following instructions to some extent and also save inference time.

Also, we would like to clarify that our analysis in appendix section D is only a descriptive analysis over the linear similarity. For example, output length in appendix D shows a positive linear correlation with loss, however, this does not mean that output length is strongly correlated with loss in other spaces. During rule fitting, under the combined influence of multiple indicators, output length becomes less significant, which results in its absence in the final estimated law.

We also would like to note that, during experimentation, we found finding a suitable metric quite hard since a generative language model cannot be simply evaluated only from its conversation ability or from its question answering ability. Recent works also discover that even GPT-4 could be “tricked” by superficial expressions[1]. However, expanding our optimization objective to multiple metrics is out of our current scope.

Dear Reviewer TXJx,

We just got the result of LIMA-7b model (LLaMA-2-7B finetuned with LIMA data). Its OpenLLM benchmark scores are shown in the table below.

We also compared its performance with our finetuned InstructMining model using LLM-as-a-judge. Results are listed in the table below.

This result will also be updated into our paper . If you have further questions, please feel free to reply. Thank you again for your review and kind advice!

Dear Reviewer,

We would like to extend our sincere gratitude for your assistance and support. As time is limited, we wanted to confirm that our responses have addressed any concerns you may have had. If there are still any outstanding issues, please do not hesitate to let us know. We eagerly await your further feedback and hope that if all of your main concerns have been resolved, you would consider raising your score.

Thank you once again for taking the time to review our paper.

Best regards, The Authors

Thank you for your review and for your thoughtful feedback! We are glad for your positive comments on the novelty and feasibility of our approach to select high-quality instruction-following data. Below is our response for your concerns and questions.

• Q.1 Why learned Eq.4 can work well on unseen instruction datasets? Is there any theory guarantee of the proposed method?

Thanks for pointing out the weakness in our paper. We don’t have a theory guarantee of our method currently, which is a little bit out of our scope. However, a possible reason of the effectiveness of our method is that, based on our current hypothesis, which are:

1. The rule-fitting test set is a high-quality dataset at the time of experiments.
2. Inference loss can indicate a model’s performance to some extent.

Our method would possibly select data examples which are of similar quality to the selected rule-fitting set. Hence, for an unseen dataset, our proposed framework can:

1. Label data quality. Our proposed rule can help quantify and label example quality.
2. Search in an effective way and converge towards a certain direction. After we get the labels, we will rank the data according to their quality scores, and set a quantile point to choose the top high-quality examples. To do this, we use Flaml blendsearch to ensure that the final chosen dataset converges on a relatively low inference loss.

We also would like to point out that the effectiveness of our method is based on the superficial alignment hypothesis[2] and mainly focuses on instruction tuning LLMs. We believe that base LLMs already have the knowledge to answer certain problems, while instruction tuning teaches the LLM to learn to understand and follow the instructions. Hence, we believe instruction tuning does not always require large scale training data. Through small amount of high-quality examples, an LLM should be able to learn to follow instructions.[1,2]

• Q.2 Other data selection methods?

Currently, instruction-following natural language data selection methods can be divided into three types:

1. Human selection. This method requires human efforts to label and select examples from datasets, which always results in large expenses.
2. Strong LLM labeling. This method requires serving or connecting very strong LLMs through apis, e.g., GPT-4. This method might lack explainability and also result in relatively large expense if the LLM api is not free and have high rate limits.
3. Machine selection. This method requires a model e.g., reward model, to mimic human or strong LLM to label data and set selection quantiles. This method can save efforts and work under very low expense.

At the time of our submission, only (a) and (c) is available. For (b), since our resource is limited at this point, leveraging strong LLMs like GPT-4 to label over 100k data examples is too expensive for us. We will try our best to find other ways to implement (b). For (a), we are currently conducting experiments using LIMA data. We will provide the result as soon as possible and will update it into our paper and comment here. And for (c), please refer to our ablation study section.

Reference:

[1] Kung, Po-Nien, and Nanyun Peng. "Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning." arXiv preprint arXiv:2305.11383 (2023).

[2] Zhou, Chunting, et al. "Lima: Less is more for alignment." arXiv preprint arXiv:2305.11206 (2023).

Dear Reviewer 9wuG,

We just got the result of LIMA-7b model (LLaMA-2-7B finetuned with LIMA data). Its OpenLLM benchmark scores are shown in the table below.

We also compared its performance with our finetuned InstructMining model using LLM-as-a-judge. Results are listed in the table below.

This result will also be updated into our paper . If you have further questions, please feel free to reply. Thank you again for your review and kind advice!

Thank you for your review, and for recognizing our contributions! We are encouraged that you find our proposed framework feasible for selecting useful instruction following data. Below is our response to your questions and concerns:

• Q.1 & W.1: Expense of training.

In implementing regression with ten variables, it is imperative to maintain an Event Per Variable (EPV) ratio exceeding 10. In pursuit of analytical robustness, we intentionally surpassed this threshold by collecting over 100 experiments. Each subset comprising 1,000 samples was trained with 8 A100 GPUs within 15 minutes. As a result, the cumulative time for these 129 experiments totals approximately 32 hours.

• Q.1 & W.2: generalization of the rule

In table 6 we also show the generalization ability for our rule across model (LLaMA-1-7B), size(LLaMA-2-13B) and LoRA(LLaMA-2-7B with LoRA).

• Q.1 & W.2: coefficients sensitivity for rule-fitting set

Certainly, the coefficient has sensitivity to the test set for rule-fitting. However, it's crucial to note that such changes do not compromise the validity of our methodology. Selecting an ideal set of rule fits presents a challenge, and for our study, we employed a dataset generated through human-labeled GPT-4, representing the best available option at the time of publication.

• Q.2: Reference.

Thank you for pointing this reference out, As this reference link is not openable from our side. If you are referring to Active Instruction Tuning [1], we will cite this paper in our later version.

• Q.3: Why is there an increase in inference loss when data selection is more important than data quality?

Could you please provide more detail regarding “data selection is more important than data quality”?

• Q.4.1: Why are other baselines worse than random selection?

If you are referring to Table 4, there are many other results in the open LLM leaderboard. The OpenLLM leaderboard results only show one aspect of the model ability, hence its ability in question answering. We also would like to point out that, during data quality evaluation and quantile searching, we mainly use inference loss as the indicator of data quality, which means that our optimization goal is to find a training set which results in relatively low inference loss. This optimization goal is highly correlated with a model’s ability in conversation. However, OpenLLM leaderboard benchmarks are more focused on question answering. In this case, we cannot promise that optimizing using inference loss would result in better performance in question answering tasks.

• Q.4.2: Are there other baseline methods?

For other possible baselines, currently, instruction-following natural language data selection methods can be divided into three types:

1. Human selection. This method requires human efforts to label and select examples from datasets, which always results in large expenses.
2. Strong LLM labeling. This method requires serving or connecting very strong LLMs through apis, e.g., GPT-4. This method might lack explainability and also result in relatively large expense if the LLM api is not free and have high rate limits.
3. Machine selection. This method requires a model e.g., reward model, to mimic human or strong LLM to label data and set selection quantiles. This method can save efforts and work under very low expense.

At the time of our submission, only (1) and (3) is available. For (2), since our resource is limited at this point, leveraging strong LLMs like GPT-4 to label over 100k data examples is too expensive for us. We will try our best to find other ways to implement (2). For (1), we are currently conducting experiments using LIMA data. We will provide the result as soon as possible and will update it into our paper and comment here. And for (3), please refer to our ablation study section.

• W.3: loss in double descent

Since loss is the main optimization objective in our paper, we use loss in y-axis in figure 1. We believe that loss is highly correlated with PPL, which used to be a commonly used metric for evaluating traditional generative language models. Hence, inference loss could be an indicator of a model’s performance. We’ve also done some experiments on other metrics. Please refer to Figure 4 for more details.

References:

[1] Kung, Po-Nien, et al. "Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks." arXiv preprint arXiv:2311.00288 (2023).

Dear Reviewer,

We would like to extend our sincere gratitude for your assistance and support. As time is limited, we wanted to confirm that our responses have addressed any concerns you may have had. If there are still any outstanding issues, please do not hesitate to let us know. We eagerly await your further feedback and hope that if all of your main concerns have been resolved, you would consider raising your score.

Thank you once again for taking the time to review our paper.

Best regards, The Authors

[Continued]

• Data size is more important than data quality

This shift could come from randomness and we have added some experiments for further reference. We apologize that we can only provide experiment results on datasets of relatively smaller size because of the limited time we have at this moment.

In this table, we can see that a significant change in loss should be larger than ~0.005 to ~0.01, while the difference between Select-loss and Random-loss are too small. Generally, if we focus on the general trend of the two lines in Figure 1, we can see that after around 40,000 to 50,000, the difference between Select-loss and Random-loss becomes smaller, which means that the importance of data selection decreases.

Thanks for your response again. Your review is of great help for improving our paper. If you feel most of your concerns have been addressed, it would be very nice of you to raise the score.

Reference:

[1] Roziere, Baptiste, et al. "Code llama: Open foundation models for code." arXiv preprint arXiv:2308.12950 (2023).

[2] Vicuna-v1.5 model card: https://huggingface.co/lmsys/vicuna-7b-v1.5

[3] Sharegpt unfiltered dataset: https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered

[4] OpenLLM Leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

[5] Nakkiran, Preetum, et al. "Deep double descent: Where bigger models and more data hurt." Journal of Statistical Mechanics: Theory and Experiment 2021.12 (2021): 124003.