Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Thanks for your thorough evaluation and insightful feedback on our submission.

While the paper addresses generalization, there is a risk that the generated synthetic data might overfit to the taxonomy's structure, potentially missing out on more nuanced, real-world instructions

GLAN is customizable. One can always introduce rare fields/disciplines to the taxonomy to capture nuanced, real-world instruction data as needed. Sometimes, the real-world instructions might be too difficult to “summarize” (into a certain discipline/field) manually. One workaround is to prompt GPT-4 with some of your instructions to generate the field or discipline names. The most frequently occurring field/discipline names can then be incorporated into the taxonomy.

Are there any measures in place to ensure the generated synthetic data's diversity and prevent redundancy?

Yes. As described in line 161 to 172 (Section 2.4), we incorporate randomly sampled class sessions and key concepts to our instruction generation prompt to ensure the diversity of our generated instructions (also see the full prompt in Table 6 of Appendix A.4). For each discipline, we have approximately 4 million unique combinations (of randomly sampled class sessions and key concepts) and in total (126 disciplines) we have over 500 million such unique combinations, which guarantees that the 10 million instructions we generate exhibit significant diversity.

what is the whole taxonomy of the human knowledge and capabilities?

As described in line 186 (section 3.1), we uploaded the taxonomy of human knowledge and capabilities used in this work as the supplementary material. Below is an example of the leaf node (i.e., the “History” discipline), where the parent node is the “Humanities” field and the grandparent node is the “academic” category.

{"topic": "History", "meta_topic": "academic, Humanities"}

And can each task (e.g., gsm8k, arc) be categorised into any sub category?

It depends. Specifically, gsm8k aligns well with the mathematics discipline in our taxonomy. In contrast, tasks like ARC and MMLU span multiple disciplines, corresponding to various nodes within our taxonomy. It is challenging to determine which specific discipline would most benefit the reasoning-related BBH task. However, after training on the 10 million generated instruction data, we observed a notable improvement in the reasoning capabilities of LLMs.

Are the effects of each category orthogonal to each other? i.e., ablating data from a child category does not affect tasks in another child category. It would be beneficial if the authors could provide some preliminary results.

According to our ablation experiments on the Mathematics discipline, the answer is yes. We use 15K data generated from the Mathematics discipline (from our taxonomy) and 60K data generated from all the other disciplines (also from our taxonomy). We run experiments in three settings (i.e., only with the 15K math data, only with the 60K data in other disciplines and combining the 15K and the 60K data) and results are as follows.

The results from the GSM8K benchmark indicate that the mathematical capabilities are predominantly derived from the data generated within the Mathematics discipline. When combined with the 60K data from other disciplines, the mathematical performance remains mostly unchanged. Similarly, the coding and reasoning capabilities appear to be derived from the data of other disciplines (see results on HumanEval and BBH), and the same trends are observed when this data is combined with the math data. Thank you for bringing this interesting question to us. We will include the above explanations and results into our revised manuscript.

All the results and discussions mentioned above will be included in our updated manuscript.

Thank you again for your constructive reviews.

Thank you for taking the time to review our submission and providing constructive feedback.

The novelty is limited as similar top-down designs have been utilized in many previous works.

Regarding "similar top-down designs in previous works", could you please share specific references or examples? This would help us further clarify the distinctions between GLAN and those approaches.

Besides, the key contributions of GLAN include:

• Scalability and Independence from Pre-Existing Data: As described in Section 2 (see lines 82-95), GLAN enables LLMs such as GPT-4/GPT-4o and GPT-3.5/GPT-4o-mini to generate a vast amount of instruction-response data from scratch, exceeding 500 million data points (i.e., 500 million unique combinations of knowledge points), without relying on pre-existing data samples. This approach contrasts with previous methods like Evolve-Instruct and Self-Instruct, which depend on pre-existing data samples (see lines 29-37). For instance, WizardCoder [1] utilizes 20K provided samples, resulting in a final data size of 80K after applying the Evolve-Instruct method.

• Broad Coverage Across Disciplines: GLAN spans a wide range of domains, currently covering 126 disciplines. This extensive coverage ensures that the generated data is diverse and comprehensive. For a detailed overview of the 126 disciplines, please refer to Table 12 in the appendix or the supplementary materials.

• Easy Customizability: As detailed in lines 74-77, GLAN offers easy customization options, allowing users to add new fields, disciplines, or subjects into the taxonomy without disrupting previously generated content. This flexibility ensures that GLAN can be adapted to various needs and updated as new knowledge areas emerge.

Besides, the main experimental results in Table 1 appear mediocre compared to other methods.

Without utilizing task-specific training data of these tasks, we achieved the highest or second-highest results as shown in Table 1 (see the “GLAN” row). Specifically, our approach yielded the best results for MBPP, BBH, ARC-E, ARC-C, and MMLU, and secured the second-best results for HumanEval, GSM8K, and MATH (also see line 249 to 258). These results demonstrate that after instruction tuning, GLAN excels on multiple dimensions from mathematical reasoning, coding, reasoning, and academic exams with a systematic data generation approach.

Besides these general capabilities as demonstrated on general benchmarks, GLAN also excels on instruction following as shown in Table 4 and Table 5.

Thanks again for your review!

[1]. Z. Luo et al. “WizardCoder: Empowering Code Large Language Models with Evol-Instruct”. 2023

We appreciate your detailed review and the valuable suggestions provided for improving our work.

No use of actual human curriculum

We did not use human curriculum explicitly to build our taxonomy because:

1. Initially, we intend to automate the whole generation process and found GPT-4 is good at listing the taxonomy structure of human curriculum.

2. We have a verification process for the taxonomy (see Section 2.1 and Section 3.1). Actually, all of our annotators searched for existing human curriculum taxonomies for verification. That said, we actually leverage human curriculum implicitly and our taxonomy overlaps significantly with existing human curriculum.

3. There are numerous different versions of the human curriculum on the web and determining which version to use is challenging. We instead generate the taxonomy using GPT-4 (we assume GPT-4 has read all these versions and can produce a version with above average quality) and then verify each node inside it.

We also did not use human created syllabi. Because the syllabi developed within human curricula are numerous, and collecting them all to ensure high-quality and comprehensiveness is impractical for us.

We will add these discussions to our updated manuscript.

Computation cost not compared: The paper does not provide a comparison of computational costs with similar methods, such as WizardLM. For instance, GLAN training required approximately 8 days using 32 A100 GPUs to generate 10 million instructions, but no direct comparisons are made to illustrate the efficiency or cost-effectiveness relative to other approaches.

For the computational cost, API cost and GLAN’s differences from other approaches, please refer to the Author Rebuttal addressed to all reviewers.

The method is limited by the performance of GPT-3.5/4: The quality of the generated taxonomy and syllabus heavily depends on the capabilities of the underlying LLMs used in the process, namely GPT-3.5 and GPT-4. In general, GLAN does not inform how we can improve capabilities of models beyond GPT4. But also, does not consider the cost of generating 10 million instructions.

• Model Dependency: GLAN is not limited to GPTs. It can be applied to any strong close-source (e.g., Claude, Gemini) or open-source (e.g., Llama-3.1 70B/405B, Nemotron-4, Mistral Large 2, etc.) LLMs.. We chose GPT-3.5/4 for our experiments to best demonstrate the effectiveness of our method at the time of writing.

• Improving Model Capabilities Beyond GPT-4: This is still an open research problem, focusing on how a language model can self-improve with its own generations. Solely leveraging GLAN may not directly enhance capabilities beyond those of GPT-4. But we believe the diverse instructions GLAN can produce across a wide range of domains and tasks can at least address the question of “where to improve in model self-improving?”

• Cost of Generating Instructions: Please refer to our Author Rebuttal at the top.

High variability in results (Table 2): There is significant variability in GLAN's performance across different categories, with particularly weaker results in humanities and social sciences compared to STEM fields. The authors should address this, also discuss the document proportion of each taxonomy, and potentially see if there is a correlation between the data size and performance.

• High variability in results: We carefully examined the disciplines in our taxonomy (esp. for those related to MMLU subjects). We found there are 19 disciplines related to MMLU and only 6 of them are STEM disciplines. In our final experiment, we generated almost the same number of examples for each of these 19 disciplines. Consequently, we have more non-STEM data than STEM data. Therefore, as mentioned in line 264 to 269, the strong STEM results may be due to CoT reasoning.

• Ablation on duplicated subjects: We did not do ablation on duplicated subjects for the following reasons. 1) The duplication is not very severe in our view. We have in total 15,751 subjects and 7,030 of them are unique. Since we repeat the subject generation for each discipline for 10 times (described in line 192), the duplication of subjects here is reasonable. 2) We manually inspected the most frequent subjects and they looked reasonable for us.

Thanks again for your review! All the discussions and analysis above will be added to our updated manuscript.

Initially, we intend to automate the whole generation process and found GPT-4 is good at listing the taxonomy structure of human curriculum.

"found GPT-4 is good at listing the taxonomy" : this is based on what metric? was there a scientific study to test this?

our taxonomy overlaps significantly with existing human curriculum.

Is there a scientific analysis of this claim?

computation cost comparison.

I was looking for a training compute cost comparison since all baselines may have been trained for different durations, which can be a big confounder in the "perceived quality" of the data.

Overall, found the answer to "high variability" quite unconvincing, and a reluctance to scientifically examine the phenomenon that is underneath the variability (data density etc.).

Thank you for your careful review and the thoughtful comments on our submission.

However, I am wondering if there are newer topics, for example (within the medical area, we have the new topic called "Covid-19".) Since GLAN is very dependent on LLMs, the main area of concern would be ensuring that the LLMs that GLAN depends on remains updated.

Actually we find in our generated dataset there are 38,749 instructions and 28 syllabus containing the keyword “covid-19”. We believe the GPT-4 we used in experiments is aware of the Covid-19 topic. However, there is no syllabus or discipline exactly focusing on “Covid-19”. A quick fix is to instruct newer GPT-4 (i.e., gpt-4o) using prompts described in Section 2.2 and 2.3 (also see Table 6) to generate a syllabus or subject list on “covid-19”. The generated syllabus looks good to us and we show part of it in the following due to space limitation.

### College Level Syllabus on COVID-19
#### **Introduction**
This course provides an in-depth study of COVID-19 …
#### **Class Details**
**Class 1: Introduction to Coronaviruses**
- **Description**: Overview of coronaviruses, …
- **Knowledge Points**:
  1. Structure and classification of coronaviruses.
  2. History of coronavirus outbreaks.
  3. Transmission mechanisms.
  4. Symptoms and disease progression.
  5. Comparison with other respiratory viruses.
- **Learning Outcomes & Activities**:
  - Understand the basic virology of coronaviruses.
  - Activity: Research and present a comparison between SARS, MERS, and COVID-19.

---
**Class 2: Virology of SARS-CoV-2**
…
**Class 3: Epidemiology of COVID-19**
…
**Class 4: Clinical Presentation and Diagnosis**
…

Thanks to the “easy customization” feature of GLAN (line 74 to 77), we only need to generate instructions for the newly added “covid-19” topic without re-generating the entire dataset.

While the paper claims scalability, there is limited discussion on the computational resources required for generating the synthetic data at scale. Practical constraints related to computational costs and time could be a potential weakness. It was mentioned in the checklist that it is very computationally expensive to repeat experiments.

For the computational cost and API cost, please refer to the Author Rebuttal at the top.

Besides, we believe there are two types of scalability: (1) the ability to generate a large number of diverse data points and (2) the ability to generate each data point at a low cost. In Checklist 7 (Experiment Statistical Significance), we noted, “We did not include error bars in the experiments due to the high computational demands.” This statement does not contradict GLAN's scalability. This is because generating an additional 10 million data points to compute error bars is indeed computationally expensive. On the contrary, the scalability of GLAN is precisely why it is challenging to repeat the experiments. Our objective is to enhance the general capabilities of large language models (LLMs). Achieving this goal appears to necessitate a large dataset, which in turn results in high computational/API costs.

Line 269: Why do you say that the reason why errors coming from humanities and social science questions is due to CoT? Could it be due to the lack of knowledge, because it was not trained on that knowledge?

The lower performance on humanities and social science questions is unlikely to stem from a lack of knowledge. The scope of knowledge for the generated questions is predominantly governed by generated syllabuses. Therefore, it is implausible that GPT-4 would create syllabuses for humanities and social sciences with significantly narrower knowledge coverage compared to those in STEM disciplines.

We believe CoT contributes more to the lower performance on humanities and social science questions. In our earlier experiments using the MMLU benchmark, we evaluated both w/ CoT and w/o CoT settings.While the overall performance was superior w/ CoT (thus leading to its adoption), a nuanced observation emerged: CoT was more effective for STEM questions, whereas the absence of CoT proved advantageous for humanities and social science questions. It may be because “CoT may help the multi-step reasoning in STEM multi-choice questions, while humanities and social sciences questions involve more memorization and single-step reasoning, where CoT may introduce additional errors.”

Figure 2: Any reason why for HumanEval and BBH datasets, the scores dropped even though the GLAN data size increase?

The score drop happens within the 50k to 1M range. The main reason is probably due to the relatively small average size of data points per discipline. We have 126 disciplines in total and on average we have:

• 2K examples per discipline for a total of 200K examples,
• 4K examples per discipline for a total of 500K examples,
• 8K examples per discipline for a total of 1M examples.

We do observe a significant leap in performance from 1M to 10M examples on HumanEval and BBH, when data points per domain is significant enough.

We will add computational/API cost related discussions or from the checklist/appendix to the main paper.

Thanks again for your review! All discussions and results above will be added to our revised manuscript.