Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning

The type of queries and topics could be relevant to coverage of data. I think it might be worth to do ablation study on the query and topic types.

We definitely agree with the reviewer. Part of the motivation behind the final version of the pipeline was that in Instruct-SkillMix-D, we noticed limited diversity in the types of Q&A produced (in particular, most Q&A pairs seemed to be of query type “help-seeking”). Thus, queries and topics (and extrapolated skills) would be important to coverage of data.

That being said, while past open efforts invested significant human effort in ensuring high coverage of topics and scenarios to sufficiently equip the LLM for scenarios it might encounter at deployment time, we take a subtly different tack: accepting that pre-training is the dominant source of the LLM’s “inner knowledge,” we focus our instruction tuning on merely teaching the LLM skills to drawing upon that inner knowledge and present it nicely during conversations. Recent work by Zhao et al. (https://arxiv.org/abs/2409.19808) corroborates the effectiveness of our approach: in particular, they show that fine-tuning on k skills yields notable improvements when applied to both out-of-distribution combinations of k skills, and as well as more than k skills.

We hope that most of your concerns have been addressed. If so, as the discussion period nears the end, we would appreciate it if you could reconsider your assessment. We’d be happy to engage in further discussions. We would be very happy to engage further in discussions to clarify any remaining concerns that you might havez

Response to Weaknesses

The baseline methods are not fair: the main comparison is to Alpaca 52K, which is really old and known to be a low quality dataset. I think the authors should try comparing their dataset to stronger datasets such as ShareGPT with the responses regenerated by GPT4-Turbo.

Please see the following portion in our meta-comment: Gains from pipeline vs. strong teacher model, Part 1: Rewriting responses to ShareGPT and UltraChat.

In my opinion, section 1.1 is somewhat misleading. The authors (in line 70-75) say it is a mystery why public instruction tuning does not match the performance of proprietary instruct models. However, these proprietary models are trained in a variety of stages, and with distillation and RL techniques. It is not expected that instruction tuning alone can match the performance of proprietary models.

Our paper does show that SFT on a tiny dataset can lead to better performance (on some evaluations) than well-regarded frontier models. However, the reviewer is absolutely correct that the phrasing is inaccurate. Llama-3-8b-Instruct was developed with a combination of distillation and RL techniques. However, Mistral-7B-Instruct-v0.2 was developed only with SFT with datasets openly available on Huggingface. A correct statement would be: Unlike prior open efforts to SFT Mistral-7B-Base-v0.2, our pipeline is the only method (to our knowledge) that can result in high performance via SFT Mistral-7B-Base-v0.2 and even surpass Mistral-7B-Instruct-v0.2 on some common benchmarks. We will rewrite this more clearly in the final version of the paper.

Table 4 suggests that the main reason for the good performance of Instruct-SkillMix is that a stronger model is used for distillation compared to previous IFT datasets. With the same judge, Alpaca-1K longest performs similarly to Instruct-SkillMix (although Alpaca 1K longest is a weak dataset in my opinion: instructions are created using text-davinci-003). Alpaca-1K longest does perform worse on the length-controlled benchmark, but this is not a fair comparison since Alpaca-1K longest is specifically biased to encourage longer responses.

We refer the reviewer to the following section in the meta-comment for additional benchmarks: Gains from pipeline vs. strong teacher model, Part 1: Rewriting responses to ShareGPT and UltraChat.

The authors claim that InstructSkillMix is a more performant data generation method than UltraChat and Vicuna (line 511). Although InstructSkillMix seems to be a strong method, my guess is that the primary reason that InstructSkillMix outperforms UltraChat and Vicuna is due to a stronger teacher model. If the authors want to make this claim, I think they should regenerate responses from UltraChat and ShareGPT using the same teacher model InstructSkillMix uses.

We refer the reviewer to the following sections in the meta-comment:

1. Gains from pipeline vs. strong teacher model, Part 1: Rewriting responses to ShareGPT and UltraChat
2. Gains from pipeline vs. strong teacher model, Part 2: Generalizability of pipeline with different teacher model

In my opinion relying only on AlpacaEval 2.0 and MTBench is a bit limited. It would be beneficial for the authors to evaluate their models on other tasks, including mathematical reasoning (MATH, GSM8K), instruction following (IFEval), and knowledge benchmarks (MMLU).

We kindly refer the reviewer to Table 8, where we report performance on GSM8K, MMLU (and other benchmarks). On these benchmarks, we do not see a drop in performance.

We would like to highlight that we reported these results in the appendix (instead of the main paper) because we wanted to evaluate our method on general instruction-following abilities. IFEval primarily tests formatting related skills (number of lines, paras etc.), and our current pipeline does not teach those skills either.

Response to Questions

Can you compare performance to ShareGPT with the responses regenerate with GPT4-Turbo?

We refer the reviewer to the following section in the meta-comment: Gains from pipeline vs. strong teacher model, Part 1: Rewriting responses to ShareGPT and UltraChat

We set Alpaca 52K as our baseline because it is the most commonly studied dataset in previous literature. Also, there are some technical difficulties in comparing against ShareGPT:

1. The quality of the dataset is uneven and it is known that a lot of effort is needed to filter the dataset. The Vicuna team does not disclose the exact filtering rule they used for the final version of their dataset, nor did they release the dataset itself.
2. The dataset is multi-turn with some of the questions dependent on the model’s response in the previous turn. Regenerating the response in the first turn might make the user’s prompt in the subsequent turns obsolete, which may introduce confusion to the student model.

We attempt to compare against ShareGPT with the following approach. We use the filtered version available at: https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/tree/main and randomly select 1000 examples and only take the first (user, assistant) interaction. From a preliminary experiment, SFT on just the first turn performs just as well as SFT on all turns. We then regenerate the assistant response with GPT-4-Turbo.

Our pipeline still significantly outperforms regenerating responses to prompts from ShareGPT.

Can the authors discuss weakness 3?

The reviewer brings up a valid concern. However, we would first like to point out that the numbers reported in Table 4 were from Instruct-SkillMix-D. When we include the results from Instruct-SkillMix (our main pipeline), even the WR is significantly higher for our pipeline.

Nonetheless, to address the concern about length-controlled being an unfair metric for models trained on Alpaca 1K Longest, we regenerate responses to randomly selected 1000 examples from the Alpaca 52K dataset.

We refer the reviewer to the following section in the meta-comment: Gains from pipeline vs. strong teacher model, Part 1: Rewriting responses to ShareGPT and UltraChat

Although Alpaca 1K Random performs better on the length-controlled metric than Alpaca 1K Longest, it falls significantly under our pipeline. This shows the strong performance of the models is not entirely due to a strong teacher model, but can be attributed to our novel pipeline.

We hope that most of your concerns have been addressed. If so, as the discussion period nears the end, we would appreciate it if you could reconsider your assessment. We’d be happy to engage in further discussions. We would be very happy to engage further in discussions to clarify any remaining concerns that you might havez

Response to Questions

Have you investigated whether different model architectures or sizes hit the 4K example ceiling at different points?

Please see our response to the first weakness.

Could you explain the choice of k=2 for skill combinations? Have you explored other values?

As shown in Table 3, we explore k=1 and k=2. We did not explore larger values of k since we did not observe significant enough gains from k=1 to k=2.

How would the method perform with different teacher models (e.g., Claude, PaLM)?

Please see the following portion in the meta-comment: Gains from pipeline vs. strong teacher model, Part 2: Generalizability of pipeline with different teacher model.

Would it be possible that combining synthetic data with human annotations potentially break through the 4K example ceiling?

We would like to emphasize that the ceiling achieved using just 4K examples is already quite high (i.e., it has not been achieved by other SFT-only efforts. The performance surpasses SFT on ShareGPT (where responses are written by humans). Please see the performance of Vicuna-13B v1.5 in Table 7 (Appendix B). While it may be possible that human annotations could help, one of the main contributions of this paper was to minimize the need for human annotation / intervention so as to reduce the cost required for generating the dataset.

Could you elaborate on potential approaches for quality control in the data generation process?

One of the main contributions of this paper was to minimize the need for human annotation / intervention so as to reduce the cost required for generating the dataset. As hinted above, we suspect that most human interventions may reduce performance.

Please see Table 13 (Appendix G) and the following table (which reports the numbers for Instruct-SkillMix instead of Instruct-SkillMix-D). When we train on 4 different subsets of the same size from our dataset, the resulting models have similar performance across the benchmarks.

However, Table 6 (also included below) shows the negative effect on the trained models when we intentionally introduce 20% of data that has uneven quality (shorter in length or poor quality). The difference between the two settings helps us deduce that the data generated from our pipeline has a uniform quality.

Could you provide analysis of model performance on longer-form tasks and multi-turn conversations?

While we appreciate the reviewer bringing up this point, this is not in scope of the project, and all generated data was a single (instruction, response) pair. However, our pipeline is adaptable to other settings, and it is easy to inject human inductive bias if needed. By changing the prompt to specifically ask for skills and data for multi-turn conversations, we can generate data that are relevant to multi-turn conversations. See an example of a generated list of skills / and an example data point.

EXAMPLE SKILLS

itinerary_customization, problem_solving

EXAMPLE DATA

###Human: I need to plan a trip to New York for a conference but I also want to squeeze in some sightseeing. Can you help me with that?\n### AI: Absolutely! When will you be going, and how many days will you be staying in New York?\n\n### Human: I'm attending the conference from April 10th to 12th, but I can stay until the 15th.\n### AI: Great! Would you prefer to do your sightseeing before or after the conference days? [rest omitted]

The paper provides limited investigation of how different teacher models might affect results. Lack of this raises questions about the method's generalizability.

We refer the reviewer to the following sections in the meta-comment:

1. Gains from pipeline vs. strong teacher model, Part 1: Rewriting responses to ShareGPT and UltraChat
2. Gains from pipeline vs. strong teacher model, Part 2: Generalizability of pipeline with different teacher model

The relationship between skills and model performance remains inadequately explored, with no clear metrics for assessing skill quality or coverage.

The question refers to “Quality and coverage,” which in prior approaches to instruction tuning (e.g., UltraChat) was decided by humans. While there is no doubt that the set of skills matters, we present evidence that it may be simplistic to use humans to judge this. We reported on two methods to come up with a list of skills and find that performance differs a lot, and especially with respect to brittleness to presence of “bad” data. (Instruct-SkillMix-D vs Instruct-SkillMix in Table 7, Appendix B). This is mystifying. The experiment suggested it is best to use skills derived directly using the metacognition of the teacher model — rather than from making the model extract skills from pre-existing seed datasets.

But interestingly, the performance of skills depends on what other model is used for generation. The best performance is achieved when the skills are sourced from the same model as the one generating the data (see table below). In other words, there may be no optimum set of skills, and the set may change with the teacher and even the student. Our paper opts to highlight such phenomena, which is an important contribution for design of instruction-tuning datasets.

We will add the above discussion and thank the reviewer for bringing this up.

Response to Weaknesses

The paper's most significant limitation is the performance plateau at 4K examples, with no clear explanation or analysis of learning curves as dataset size increases. This is compounded by limited investigation of whether different architectures or model sizes might hit different ceilings.

All methods plateau at some point, and no prior method had hit the performance level of our method despite using far more data (including efforts with humans in the loop such as Dolly and ShareGPT). Ultimately, an SFT cannot improve on the teacher, and in our experiments stronger models like Llama3 8B plateau pretty much when they achieve 50% win-rate against the teacher on held-out queries. The high performance on evals (e.g., on AlpacaEval 2.0) points to good OOD generalization. A weaker model or a weaker pipeline generalizes less well and even on training may not match the teacher’s performance on training data.

For detailed learning curves on different dataset sizes (and different architectures), we refer the reviewer to the following parts of our initial submission: Table 7 (Appendix B) and Figures 3 and 4 (Appendix E.2). As shown in Figure 4, the trend appears to depend significantly on the model, with different architectures and sizes reaching different performance ceilings. Since our original submission, we have also trained Gemma2 2B on Instruct-SkillMix datasets of varying sizes (see table below). Taking these results into account along with Table 7, we still see that different architectures and sizes hit different plateaus. This suggests that the observed plateau is influenced by both dataset size and the specific model architecture and size.

The strength of our paper is our simple and powerful pipeline. We hope it will cause a rethink of current approaches and motivate other efforts to improve the method; e.g. (a) better ways to extract skills (b) curating Q&A according to types of questions (e.g., essay questions, fixed-format questions as in IF-EVAL, or on a ranking of skills based upon how they contribute to downstream performance.

The evaluation methodology relies heavily on AlpacaEval 2.0 and lacks assessment of long-form generation and multi-turn conversations. The use of both teacher and grader models from the same model family (GPT-4) raises concerns about potential systematic biases.

We thank the reviewer for bringing up a good point. However, we would like to highlight that we chose AlpacaEval 2.0 as an evaluation for instruction following (not necessarily long-form generation / multiturn conversations). Please also see the response to the last question (i.e. one could easily use the Instruct-SkillMix pipeline and specify answers of different forms and word-length).

Regarding systematic biases due to teacher and grader models this was addressed in the paragraph Effect of choice of grader on page 8. Using Claude 3 Opus as a grader conserves the relative ranking between models, but favors our models even more than does GPT-4.

We also refer reviewer to the meta-comment (Gains from pipeline vs. strong teacher model, Part 2: Generalizability of pipeline with different teacher model) where we use Claude-Sonnet-3.5 for the Instruct-SkillMix Pipeline, GPT-4 as the grader, and still see performance gains compared to baselines.

Also, the methodology lacks a principled approach for determining the optimal number of skills or combinations, and provides no systematic quality metrics for the generated data.

Yes, the optimal set of skills as well as number of skills is left for future work. One issue is that we also lack good evaluations, and so our focus was on showing that our new ideas lead to strong OOD generalization, i.e. performance on current evaluations. The paper explores the performance of SFT on combinations of k skills for k=1 and k=2. Here, k=2 is not significantly better than k=1, but this is possibly due to limitations of the evaluations. In a somewhat different setting (i.e. with different task and different list of skills), recent work by Zhao et al. (https://arxiv.org/abs/2409.19808) shows that fine-tuning on combinations of k skills yields notable improvements when applied to both out-of-distribution combinations of k skills, and as well as more than k skills. Potentially those ideas can yield better evaluations than AlpacaEval 2.0.

We hope that most of your concerns have been addressed. If so, as the discussion period nears the end, we would appreciate it if you could reconsider your assessment. We’d be happy to engage in further discussions. We would be very happy to engage further in discussions to clarify any remaining concerns that you might have