Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Generate instructions from an open-source frontier model and train on it to test the self-training capabilities of GLAN.

The scaling trends look really promising! But 1. how do GLAN instructions compare with other methods given the same quantity? can plot as a comparison; 2. will the performance ever saturate? worth more experiments and analysis.

We sincerely appreciate your constructive suggestions and agree that the proposed experiments would provide valuable insights into our approach. However, conducting these experiments requires significant computational resources and time to obtain reliable results. While we have already begun exploring this direction, the two-week rebuttal period is insufficient to produce meaningful findings. We intend to incorporate these results into future work to provide a more comprehensive analysis.

Does GLAN generate longer instructions than other methods? I think it might be a suitable method to scale the length of instructions, which might be an interesting direction.

Thanks for your interesting question. To answer this question, we compute the average prompt and response length of GLAN-10M, Alpaca-52k (self-instruct) and WizardLM_evol_instruct_70k (evol-instruct). The results are as follows and as you expected:

This may be due to GLAN's ability to generate complex instructions in areas such as math, physics problem-solving, essay writing and more.

Thank you once again for your valuable feedback! We will incorporate all the results and explanations mentioned above into the revised manuscript.

Lack of Evaluation on Bias and Ethical Concerns: The paper acknowledges the potential risks of amplifying biases in generated data but does not present specific strategies for bias detection or mitigation, which is critical for real-world deployment.

We sincerely appreciate your thoughtful feedback regarding bias and ethical concerns. These are undoubtedly critical considerations for the deployment of LLMs in real-world applications. While we acknowledge the importance of addressing these challenges, this work primarily focuses on synthetic data generation. As such, a comprehensive exploration of bias detection and mitigation strategies is beyond the immediate scope of this research.

Nevertheless, to detect and mitigate bias and ethical issues in our method/model (or other models trained on synthetic instructional data), one can adopt methods for bias detection and mitigation in recent work [1][2][3].

[1] Fan, et al. (2024). BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs. https://arxiv.org/abs/2407.10241

[2] Narayan et al. (2024). Bias Neutralization Framework: Measuring Fairness in Large Language Models with Bias Intelligence Quotient (BiQ). https://arxiv.org/abs/2404.18276

[3] Gallegos et al., (2024). Bias and Fairness in Large Language Models: A Survey. https://aclanthology.org/2024.cl-3.8/ Computational Linguistics, Volume 50, Issue 3 - September 2024

Table 1: how many instructions are there to train each model?

As stated in [1], WizardLM v1.0 is trained on 250K instructions. The exact number of instructions used for WizardLM v1.2 is not disclosed, but we assume that at least the same quantity was used. Similarly, [2] discusses WizardMath v1.0 but does not specify the exact number of instructions used for training. It also remains unclear whether additional instructions were used for training WizardMath v1.2.

We would also like to highlight that the 10M instructions we generated include tasks beyond the scope of current benchmarks, such as instructions related to fishing and textiles. To the best of our knowledge, none of the benchmarks used in this paper cover these domains. Therefore, solely evaluating GLAN on existing benchmarks may not provide a comprehensive assessment of its capabilities. This limitation on evaluations similarly applies to frontier LLMs.

[1]. C. Xu et al. “WizardLM: Empowering Large Language Models to Follow Complex Instructions”. 2023

[2] Luo et al. “WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct”

Method Feasibility: The method is somewhat novel, but its feasibility in the real world is questionable. While the comparative methods could generate instructions for self-training, I suspect that GLAN is bounded by the knowledge of the generator, e.g., GPT-4 in this paper. It significantly impedes the deployment of GLAN as a method to instruction-tune frontier models, if not to distill from GPT-4 with yet another trick.

Thank you very much for your insightful comments, which are truly inspiring for future research in this direction. However, we would like to clarify several aspects.

On “Comparative methods could generate instructions for self-training” and “to instruction-tune frontier models”

First, we must clarify that "generating instructions for self-training" is not the same as "instruction-tuning frontier models." The only comparative method that has demonstrated the ability for self-training is self-instruct, which expands 175 seed instruction-response pairs using GPT-3 base (davinci) to generate additional instructions. These instructions are then used to fine-tune GPT-3 base. The resulting fine-tuned GPT-3 model (GPT-3 Self-Instruct) significantly outperforms GPT-3 base and is only slightly behind GPT-3 Instruct (text-davinci-001) on SuperNI evaluations. However, improving a frontier LLM (which is already fine-tuned through supervised training and further refined using RLHF or PPO) is considerably more challenging than improving a base model.

On “feasibility in the real world is questionable”

In many real-world applications, such as mobile or desktop software, deploying frontier LLMs locally is often impractical due to resource constraints. Small language models (SLMs), such as those with 3B or 7B parameters, offer a more cost-effective solution and are well-suited for local deployment. Our research demonstrates that GLAN can substantially enhance the performance and capabilities of these SLMs, making them a viable alternative for resource-limited scenarios.

On instruction-tune frontier models or improving frontier models

To systematically improve frontier LLMs, a broad and diverse set of prompts is essential for identifying areas or domains that require improvement. GLAN can generate instructions across a wide range of domains and tasks, providing the fuels needed for self-improvement in frontier LLMs. Besides the initial input prompts, we believe a good reward model is also needed to guide the improvement of response quality.

Baselines: With the last point being said, at least a few comparative methods to generate instructions, like self-instruct or evol-instruct, should be compared with GLAN. After all, the paper claims the instruction generation method as its core contribution. However, no baseline method is even compared.

Thank you for highlighting this! We have actually compared GLAN with both self-instruct and evol-instruct in Table 1. Specifically, “WizardLM v1.2” [1] is based on evol-instruct, while “Mistral CodeAlpaca” [2], which focuses on the coding domain, is based on self-instruct. GLAN outperforms both across all benchmarks. We will include these clarifications about WizardLM v1.2 and Mistral CodeAlpaca in the revised version of the paper.

[1] Xu et al. (2024). WizardLM: Empowering Large Language Models to Follow Complex Instructions. https://arxiv.org/pdf/2304.12244

[2] https://github.com/sahil280114/codealpaca

We sincerely thank Reviewer h7cZ for your review and are grateful for the time you spent on our submission. We are glad for the acknowledgment that our approach is effective for ensuring diversity of instruction tuning and valuable for the LLM community. Below we would like to give detailed responses to each of your comments.

W1: novelty and research contribution of GLAN

We would like to highlight the research contribution of our paper to address potential concerns regarding novelty.

• As far as we know, GLAN is the first method that does not rely on existing seed topics or instructions to synthesize data from scratch. This approach brings scalability and flexibility to instruction tuning data, especially by filling in the gaps for underrepresented and niche domains.
• In terms of taxonomy construction, our method first induces LLMs to recursively decompose high-level concepts into smaller concepts for instruction generation by autonomously searching within the knowledge space of large LLMs. This approach offers unique advantages, including better coverage and high automation.
• At the performance aspect, GLAN surpasses the progress seen across many open instruction model recipes released recently. Unlike previous instruction tuning methods that are limited to benefiting specific tasks, GLAN demonstrates consistent improvements across a wide range of tasks and domains.
• At the data and resource aspect, as recognized, the curated 10M high quality instruction-response pairs are valuable for future work.

Compared to the mentioned works (Camel and MathScale), we would like to highlight the unique strengths of GLAN:

• Generalizability: Unlike Camel, which focuses on role-playing tasks, and MathScale, which is tailored to the mathematics domain, GLAN aims to enhance capabilities across a broad range of tasks. GLAN covers a wide range of domains, with mathematics being just one of them.
• Scalability: GLAN can generate over 500 million unique combinations of knowledge points for instructional data generation, making it highly scalable.
• Customizability: New fields, domains, or disciplines can be easily added to GLAN's taxonomy, providing flexibility and adaptability.

We will add an in-depth discussion on comparison with the mentioned works in the revised version.

W2: The scalability of the proposed method is questionable.
We hope to clarify that our method's scalability is not limited by human verification and API costs, the reasons are as follows:

A. Human Verification

• First, as mentioned in lines 153 and 89, human verification is at the discipline level rather than at the concept level. The cost of human verification is low due to the limited number of disciplines in the taxonomy (126 disciplines in total).
• Second, human verification is only required in the initial step when constructing the taxonomy. Subsequent steps in the GLAN process are fully automated, significantly minimizing the cost of human verification. Since we have already produced the entire taxonomy, subsequent scaling up no longer requires any additional human verification.

B. LLM API Cost
We would like to clarify that the whole GLAN methodology for data synthesizing is not limited by API costs.

It is a model-agnostic methodology, allowing users to select any open-source or closed-source models based on their requirements. For example, users who deploy free open-source models such as Llama-3 72B will incur a data generation cost of $0.

In our main implementation, we adopted GPT4 as the large model for distillation only to obtain a higher quality of instruction tuning data for future work.

W3: the presentation of the proposed knowledge taxonomy
Thanks for your great suggestion. We'll add a figure in the revised version to illustrate the definition, examples, and final quantity of the knowledge hierarchy for better understanding.

W4: "The method is only evaluated on the Mistral model."
As shown by previous works $1$ , the effectiveness of general-purpose instruction-tuning data is consistent across different models. Our experiments follow a similar setting with similar works, such as Orca $2$ , Phi $3$ , and WizardLM $4$ . When conducting our experiments, we utilized the Mistral-series models, which were among the best open-source models available. We believe that the evaluation results of the Mistral models are transferable to more recent open-source models.

It is worth mentioning that GLAN surpasses the progress seen across various open instruction model recipes released recently. As shown in Table 1, the Mistral 7B model tuned with GLAN has already surpassed all other models of similar size in other open instruction model recipes. The GLAN-7B model even outperforms the 13B WizardLM v1.2 on most benchmarks.

Thank you for your review and the valuable feedback on our submission.

W1: Concern about novelty and related works

The novelty of our paper may appear limited compared to existing works such as [1], [2], and [3]. However, we aim to clarify that our method, GLAN, distinguishes itself by recursively decomposing high-level concepts into smaller, more granular elements. This recursive breakdown, unlike previous approaches, enhances the diversity and richness of synthesized instructions. By breaking down topics into their constituent subtopics, GLAN facilitates efficient knowledge distillation from Large Language Models (LLMs) to Small Language Models (SLMs). We consider prior works as special cases of GLAN, with a tree height of 1.

Additionally, while [4] shares similarities with our approach in using syllabus to prompt LLMs, our method stands out by generating syllabus autonomously, broadening coverage and diversity of the synthesized instructions.

W2: Concern about API cost

Regarding the high-cost API concern you raised, we would like to clarify that although our current implementation involves high-cost APIs like GPT4, the method is adaptable to open-source LLMs like Llama 3 [5]. We intend to include experiments using data generated by Llama 3 in the final paper to demonstrate versatility.

W3: Lack of multi-turn conversation data

Your observation regarding the absence of multi-turn conversations in our dataset is valid. We recognize the importance of multi-turn dialogue for practical applications and plan to address this in future work by incorporating multi-turn conversation data.

Q1: Necessity of human evaluation

Regarding the necessity of human evaluation, our decision to employ human assessment was primarily to refine the taxonomy for efficiency. While human involvement can pose scalability challenges, algorithmic approaches like automated pruning based on content overlapping of the leaf node’s syllabus could streamline this process, ensuring scalability without compromising data quality.

We appreciate your feedback and will incorporate these clarifications and expansions into the final paper.

Reference:

[1] Phi-2: The surprising power of small language models

[2] Enhancing Chat Language Models by Scaling High-quality Instructional Conversations

[3] Baize: An open-source chat model with parameter-efficient tuning on self-chat data*

[4] Instruction Tuning with Human Curriculum

[5] https://ai.meta.com/blog/meta-llama-3/

Thank you for your questions and insightful suggestions regarding our work!

Q1. Clarification on Human Effort in Taxonomy Creation

Human intervention is only required at the initial stage of building the taxonomy.

Specifically, we begin by prompting GPT-4 multiple times with the query "list all fields of human knowledge and capabilities", resulting in the collection of hundreds of disciplines. To manage the presence of duplicated or similar disciplines, a majority voting process is conducted among 3 human annotators to determine which disciplines should be retained. Once this initial curation is completed, the subsequent generation steps within GLAN are fully automated to minimize the need for human involvement.

Q2. Exploring Open-Source Models for Data Generation

You are correct—GLAN is designed to be model-agnostic, meaning it can operate with both closed-source models (e.g., GPT-4/GPT-3.5) and strong open-source models (e.g., LLaMA-3 72B). In our experiments, we chose to use closed-source models to best demonstrate GLAN’s potential at that time. However, we anticipate that GLAN can achieve comparable results when applied to high-performing open-source models, making it more applicable.