WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions

Dear Reviewer, we thank you for your valuable comments and the time you spent reviewing our work!

Please find below a detailed discussion of the points you have raised:

Weaknesses:

Weakness 1: The paper lacks the number of baselines for comparison.

As shown in the table in the section of New Baselines of General Response, we add more baselines such as Baize, CAMEL, Tulu. The extensive results indicate that our WizardLM also significantly surpasses these open source models, which demonstrates that the convincing quality of instruction-following dataset crafted by Evol-Instruct. We don't report the Orca due to it's not an open-source model. We add discussion about Orca in the relate work of updated main submission.

Weakness 2: The process of generating training instances largely relies on a single LLM (ChatGPT).

To verify that our designed prompt and method can be transferred to other LLMs, we chose LlaMA-2-70B-Chat as a substitute for ChatGPT. Following the same prompts and design choices outlined in the paper, we re-executed the entire evolution process starting from Alpaca data and trained the WizardLM-13b(LlaMA-2-70B-Chat) model with randomly selected 70k data. Please refer to below table for details, although LlaMA-2-70B-Chat's ability is weaker than ChatGPT, WizardLM-13b(LlaMA-2-70B-Chat) still significantly outperforms Alpaca-13b trained solely on Alpaca data and even outperforms Vicuna-13B trained with much more ChatGPT tokens.

Weakness 3: In Figure 5, the paper calculates the difficulty of the instruction by simply using ChatGPT. The paper should calculate the difficulty by using different evaluators or other metrics.

In order to explore the ability of ChatGPT to perform difficulty analysis, we sample 600 instructions (each dataset has 100 instructions) and use the more powerful GPT4 model and 5 well-educated human annotators together for difficulty assessment. The assessment results are in the following Table. The results show that ChatGPT, GPT4, and manual annotation show a high degree of consistency in the trend of difficulty changes.

Questions:

Question 1: For human evaluation, how many labelers labeled the same instances? Could you share the exact inter-labeler agreement between human labelers?

As mentioned in the Section 4.4 Human Evaluation, we recruit 10 well-educated annotators to judge the whole test set. All of the Kappa scores are greater than 0.6 (WizardLM vs. Alpaca is 0.67, WizardLM vs. Vicuna is 0.64, WizardLM vs. ChatGPT is 0.63), which indicates the good agreement among the annotators.

Question 2: Does using the full 250K instructions lead to better performance on the automatic evaluation setting?

Yes, as shown in the following table, we also report the performance of model trained on 250K instructions, it achieves better performance on all the automatic benchmarks than the model trained on 70K instructions.

We hope these resolve all your concerns about this paper!

We would like to thank the reviewer for engaging so thoroughly with both our paper and the rebuttal. We hope the following addresses all remaining concerns:

Weakness 1: The bold should be on Tulu for MMLU, not WizardLM for Table 1. Please update the paper. (53.19 vs 52.92)

Thanks so much for the knidly reminder. We have updated the Table 1 in the latest revision.

Weakness 3: How did you define the concept of "difficulty" for the experiments? Since difficulty is inherently subjective (depends on the evaluator), is there any standard to make the boundaries clearer (ex) Provide examples corresponding to the difficulty to the labelers)?

Yes. Given an instruction, we let the labelers give an overall score on a scale of 1 to 10, where a higher score indicates a higher difficulty.

In order to clarify and unify the labelers' understanding of difficulty, we have prepared some topics for each with three related instructions on 2, 5, 9 difficulty levels respectively.

Example-1: (General topic)

Example-2: (Code topic)

Example-3: (Math topic)

Question 1: So does it mean that "each" of the 10 annotators evaluated the "whole" set?

Yes, each annotator has evaluated the whole set.

We want to thank you for you review. Please let us know if any of your points were not addressed properly, or if you have any additional questions.

Question 4: Would incorporating more human feedback during the instruction evolution process help in refining the quality of instructions? If so, how?

Sure, human feedback is important in the evolution process, one of our on-going research project is "human-in-the-loop evolution", which indicates that the human feedback would improve the instruction quality and accuracy, we will public related research in the future.

detailed response: Sure, human feedback is important in the evolution process. What we refer to as "human-in-the-loop evolution" aligns very well with your idea of "incorporating more human feedback during the instruction evolution process helps in refining the quality of instructions". This is a direction we are currently exploring. Similar to OpenAI's RLHF method, we evolve an instruction multiple times, hire people to score and rank the evolved instructions. Then, we train a reward model for instruction quality based on this human-annotated data. When executing Evol-Instruct, we select high-scoring instructions based on this reward model and set the weight of the gradient during fine-tuning according to the score. Cases with high instruction scores have a larger gradient update weight, while cases with low scores have a smaller gradient update weight.

Question 5: How scalable is the Evol-Instruct process for even larger models or datasets in the future?

Besides the WizardLM-13b (52K self-instruct data as the seed), we add the experiments of three new trained wizard models: lager models WizardLM 65b and WizardLM 70b which is trained from Llama-1 65 and Llama-2 70B, and WizardLM-13b (70K ShareGPT data as the seed). The results in the following table indicate that our Evol-Instruct shows scalable performance for larger models or datasets.

Question 6: What’s your opinion of learning with Evol-Instruct and in-context learning with demonstrations? The correlations and differences, and the pros and cons.

In-context learning can allow Large Language Model (LLM) to learn a new task with only a few examples given. For tasks like Evol-Instruct that make instructions more complex, which are not common in the training data for LLM, enabling in-context learning is expected to help LLM perform better in evolution. In fact, in our In-Depth Evolving of Complicating Input, we use in-context learning to enhance the evolution effect. For details, please refer to Appendix D. In-context learning is a process where the LLM learns a new task based solely on a few given examples, without doing any Back Propagation (BP). Evol-Instruct is aimed at constructing complex instructions to enhance the performance of large pre-trained language model supervised fine-tuning. Therefore, for learning a new task, the advantage of in-context learning is that it requires very few resources, while Evol-Instruct requires more resources but can achieve better results.

We hope these resolve all your concerns about this paper!

Dear Reviewer, we thank you for your valuable comments and the time you spent reviewing our work!

Please find below a detailed discussion of the points you have raised:

Weaknesses:

Weakness 1: Reliance on LLMs.

We agree with your concern about the use of LLMs, thus one of our current research directions is introducing human feedback to the evolution process.

detailed response: We agree with your concern about the use of LLMs. We consider the elimination of potential biases introduced in evolution as an important direction for our research. A feasible method is to use human feedback as a signal to alleviate the inherent bias of the LLM during the evolution process. Although LLM has the issue of bias, the advantage of using it instead of humans for instruction evolution is that the cost is low enough and the speed is faster.

Weakness 2: Uncertainty in Evolution Direction:

Yes, this version of Evol-Instruct neglects the detailed control over the direction and topics of evolution, how to make it controllable and in line with specific goals is a topic worth research. For this paper, we mainly want to demonstrate that evolving instructions can improve the supervised fine-tuning performance of pre-trained LLM.

Weakness 3: Potential Overfitting:

Yes, executing the Evol Instruct multiple times may lead to getting stuck in a specific topic and not being able to cover a wider range of topics. This is also why we propose the branch of In-Breadth Evolving in addition to In-Depth Evolving, which is an attempt to solve this problem. From Chapter 4.5 of the main submission and Appendix F, it can be seen that the diversity of topics in our evolved instruction data is actually better than the original data. How to further improve the problem of overfitting is a topic worth continuous exploration.

Weakness 4: Paper Organization:

Thanks for your kindly advise, we have reorganized and updated our paper about section "3 APPROACH".

Questions:

Question 1: How do you ensure that the evolved instructions maintain their intended purpose and don't deviate from the original goal?

• (1) To ensure the accuracy of the evolution results, we did a lot of case study and refine the prompts, all the authors have tried almost 300 cases, and the quality meet ours requirements.
• (2) We also propose a "Elimination Evolving" method, which uses ChatGPT to measure whether the evolved instructions "have same constraints and requirments" and have "same depth and breadth of the inquiry" compared to the original one, if they are equal, we will eliminate it.

detailed response:

• (1) To ensure the accuracy of the evolution results, we did a lot of case study and refine the prompts, all the authors have tried almost 300 cases, and the quality meet ours requirements. For example, we find that the Complicating Input of In-Depth Evolving is difficult for ChatGPT to accurately perform evolution in a zero-shot setting. We adopt a few-shot approach, by providing a few appropriate examples, to ensure that this evolutionary method can work properly with ChatGPT.
• (2) We also propose a "Elimination Evolving" method, which uses ChatGPT to measure whether the evolved instructions "have same constraints and requirments" and have "same depth and breadth of the inquiry" compared to the original one, if they are equal, we will eliminate it.

Question 2: Are there any measures to prevent potential biases introduced during the instruction evolution process?

We fully agree with you that the evolution may introduce some potential biases. We propose two new methods:

• (1) Use ChatGPT to measure if there is any toxic or bias in the new instruction and answer
• (2) Use RLHF to improve alignment.

detailed response: We fully agree with you that the evolution may introduce some potential biases. We propose two new methods: (1) Use ChatGPT to measure if there is any toxic or bias in the new instruction and answer (2) Sample some evolved instructions and hire human annotators mark out the possible types of bias in them. Based on this data, train a classifier to filter out those biased evolved instructions during the evolution process.

Question 3: How does WizardLM handle instructions that are not part of the evolved set?

To better handle the instruction types as much as possible, we propose the In-Breadth Evolving, which aims to enhance topic coverage, skill coverage, and overall diversity.

detailed response: To better handle the instruction types as much as possible, we propose the In-Breadth Evolving, which aims to evolve a completely brand-new instruction that is different from the current one to be evolved. In-Breadth Evolving can enhance topic coverage, skill coverage, and overall diversity. Section 4.5 of the paper and Appendix F demonstrate that the diversity of topics in our evolved instruction data is actually better than the original data.

Dear Authors,

Thank you very much for the response to my proposed issues.

I have reviewed them and most of them can address my concerns. However, some explanation are too simple and lack details. For example,

• Could you clarify "human-in-the-loop evolution"? How do you make it?
• The sufficient experimental analysis on the updated results on three new trained wizard models. The analysis in Section 4.5 is a little superficial. For example, Why does WizardLM-13b (ShareGPT Seed) perform worse on MMLU, GSM8k? Why does WizardLM-70b perform worse on ARC and HellaSwag? These observations are contrary to your conclusions in Section 4.5: t (i) the ShareGPT is a better seed for evol-instruct, (ii) larger evolved data size can improve model capacity, and (iii) our proposed Evol-Instruct method is not dependent on ChatGPT, other strong open source model such as Llama-2 is also a good substitute for ChatGPT.

I advise the authors pay more attention to the expression and detail analysis in this paper, and explain my questions more detailedly. I would decrease my scores if I don't receive a revised manuscript (changes had better be highlighted) and circumstantial response with more detailed evidences and analysis.

Best Wishes,

Reviewer tNq7

Dear Reviewer tNq7,

Thank you for your constructive comments and valuable suggestions. We have carefully checked our response to your reviews, and have added some details to Weakness 1 and Question 1~4, with the new complete detailed responses highlighted in italics. In order to answer your questions about our three new wizards, we have added some experiments. We also have updated the corresponding content into the latest manuscript. We hope the following responses can address your concerns.

Why does WizardLM-13b (ShareGPT Seed) perform worse on GSM8k?

To investigate the reason, we add a new experiment: we random sample 2000 instructions from ShareGPT and Alpaca data respectively, then use ChatGPT to judge whether an instruction is "math" related, we find that the ShareGPT only contains 4.3% math data, and Alpaca data contains 11.8% math data, thus we think that less math training data ratio results in worse preformance on GSM8k of WizardLM-13b (ShareGPT Seed).

We have also updated above analysis on "Section 4.5 ABLATION STUDY - Training with different data (seed, size), evol model, and base model size" and the ChatGPT judgement prompt on Appendix G in the latest manuscript.

Why does WizardLM-13b (ShareGPT Seed) perform worse on MMLU? Why does WizardLM-70b perform worse on ARC and HellaSwag?

As shown in the following two tables, we report the model checkpoints performance on different epochs (2.5, 2,75, 3).

In this paper, we train our model with 3 epochs and only reported the performance of the final checkpoint to align with previous works.

For 13B models, we can see that the best performance always appears on WizardLM-13b (ShareGPT Seed) for each benchmark except GSM8k. And for 65b/70b models, we also see that the WizardLM-70b is the best one on all the benchmarks.

Therefore, we think this is mainly caused by the fluctuations on some benchmarks in model training.

We have also updated above analysis and following tables on Appendix L in the latest manuscript.

We hope you find these revisions address your concerns. Please let us know if any of your points were not addressed properly, or if you have any additional questions.

Best regards,

Paper 6631 Authors.

Weakness 4. The WizardEval dataset is too small and the paper contains little detail about its construction, so we do not know if the dataset can serve as a sound and fair evaluation benchmark.

To better introduce our WizardEval, we has updated and added some details to the Appendix on latest version of paper:

• (1) Figure 6 in Appendix G illustrates the distribution of the instances and skills in our WizardEval set. It contains 29 distinct skills that represent the main requirements of humanity, such as Coding Generation & Debugging, Math, Reasoning, Complex Formats, Writing, Extensive Disciplines, and so on. It consists of 218 instances, each of which is an instruction for a specific skill.

• (2) We compared our test set with Vicuna’s test set, which is a benchmark dataset for evaluating instruction following models. We found that Vicuna’s test set only 80 instances and 9 skills and is much smaller and less diverse than ours.

• (3) Figure 4 in main submission shows how the difficulty and complexity of the test data vary across different instances. Our test data has a more uniform distribution, meaning that it contains instructions with different levels of difficulty and complexity. On the other hand, Vicuna and Alpaca have a skewed distribution, meaning that they mostly contain instructions with low difficulty and complexity. This indicates that these two corpus are not able to handle the evaluation on more complex and demanding scenarios.

Weakness 5: I do not understand the argument in the section Analysis of In-breadth Evolving.

In-breadth Evolving aims to enhance topic coverage, skill coverage, and overall dataset diversity. To examine (qualitative analysis) the breadth (diversity) of different dataset, we firstly use BERT to encode each instruction and get its embedding with 768 dimensions, then use a dimension reduction algorithm named t-SNE to reduce embedding dimension to 2, finally we apply a clustering algorithm k-means to partition the instructions of each dataset into 20 clusters for an intuitive visualization. As shown in the Figure 6 in the paper, the data points of our dataset are more dispersed than ShareGPT and Alpaca (Self-Instruct), which indicates the better topic diversity in our instructions.

Minor Issues and Writting:

Thanks for your kindly suggestions for our grammatical and citations issues. We has updated a new version of paper to improve these aspects, please refer to the new version and thank you again.

Questions:

Question 1: How do we know the answers to the newly created prompts are correct?

To improve the alignment between the new prompts and our evolution target, we propose "Elimination Evolving" as shown in the paper, it use two strategies to judge the quality of instruct: ChatGPT and manual rules.

Question 2: How do we know the difficulty and diversity scores by ChatGPT are correct? Please do not say because other papers did similar things. The other papers could be wrong, too.

• (1) In this paper, we only use ChatGPT to measure the "difficulty and complexity" (which are synonyms). And as mentioned in the response to the Weakness 5, we describe how we measure the diversity score with BERT, t-SNE, and k-means.

• (2) To investigate the correctness of the difficulty score by ChatGPT, we add a new experiment to measure agreement of difficulty judge between ChatGPT and humans: We randomly select two instructions from the six datasets - Alpaca, ShareGPT, C1 to C4 - with equal probability each time, forming a pair. In total, we have selected 300 instruction pairs. Then, we ask ChatGPT and 5 well-educated human annotators to judge which one is more difficulty in one instruction pair, the Kappa score between humans is 0.68, and the Kappa between ChatGPT and human (majority voting) is 0.66, which indicates the good agreement among the ChatGPT and human annotators.

We hope these resolve all your concerns about this paper!

Dear Reviewer, we thank you for your valuable comments and the time you spent reviewing our work!

Please find below a detailed discussion of the points you have raised:

Major Issues:

Weakness 1: The authors do not compare against diverse instruction tuning datasets such as Natural Instructions, Supernatural Instructions, and the training data of FLAN-T5.

For a more comprehensive comparison and study, we extract 70k data from the Supernatural Instructions dataset and fine-tune llama-13b with the same training hyperparameters as ours, resulting in the model LlaMA(SNI). The experimental results, as shown in above Table, indicate that LlaMA(SNI) performs well on classic QA benchmarks, such as MMLU, surpassing Alpaca, Vicuna, and WizardLM. However, its performance is less impressive on math, code and open-domain test-beds like MT-Bench.

Weakness 2: The experiments focus on Llama and ignore other LLMs such as T5, Falcon, Mistral (which has a model not instruction tuned), and so on.

To verify whether our proposed Evol-Instruct can be applied to pre-train models outside of Llama, we chose to train Mistral-7B on Alpaca data and the evolved Alpaca data respectively. The results of the benchmarks are shown in the following table.

Weakness 3.1: The evaluation of difficulty and diversity by ChatGPT is unconvincing.

We just use the ChatGPT to post analyse the "difficulty" distribution of the generated instructions, but we do not use this analysis results to guide the data generation or model training. In order to explore the ability of ChatGPT to perform difficulty analysis, we sample 600 instructions (each dataset has 100 instructions) and use the more powerful GPT4 model and 5 well-educated human annotators together for difficulty assessment. The assessment results are in the following Table. The results show that ChatGPT, GPT4, and manual annotation show a high degree of consistency in the trend of difficulty changes.

To investigate the correctness of the difficulty score by ChatGPT, we add a new experiment to measure agreement of difficulty judge between ChatGPT and humans: We randomly select two instructions from the six datasets - Alpaca, ShareGPT, C1 to C4 - with equal probability each time, forming a pair. In total, we have selected 300 instruction pairs. Then, we ask ChatGPT and 5 well-educated human annotators to judge which one is more difficulty in one instruction pair, the Kappa score between humans is 0.68, and the Kappa between ChatGPT and human (majority voting) is 0.66, which indicates the good agreement among the ChatGPT and human annotators.

Weakness 3.2: It is also dubious to report only the average score of difficulty and diversity, as these are clearly different concepts.

In the Figure 5, we only shows the average score of "difficulty and complexity" (which are synonyms), and this score do not contains the degree of "diversity".
Furthermore for "diversity", as shown in the "Section 4.5 - Analysis of In-breadth Evolving" and Figure 6 in Appendix F , we use t-SNE and k-means over BERT embedding of instructions to indicate the topic diversity between self-intruct, ShareGPT and evol-instruct dataset.

Dear Reviewer, we thank you for your valuable comments and the time you spent reviewing our work!

Please find below a detailed discussion of the points you have raised:

Weakness 1.1: The technical contribution of the proposed method is not very significant because compared to self-instruct, it is only adding the command for LLM to generate more complex instruction.

Motivation & Novelty: The Self-Instruct[1] is to address the problem of "human-written instruction data that is often limited in quantity", aiming to "provide an almost annotation-free method" based on LLM to automatically generate more instruction data. Our Evol-Instruct is not aimed at increasing the quantity of instructions but rather explores whether increasing the complexity of individual instruction contributes to improving the supervised fine-tuning performance of LLM. Our novelty lies in: (a) To best of our knowledge, we are the first to explicitly propose that increasing the complexity of instructions is beneficial for improving the instruction-following capabilities of pre-trained language models. (b) We propose a brand-new automated method to enhance the complexity and difficulty of existing instruction data without the need for human annotators.

Both automatic and human evaluations consistently indicate that the model trained using Evol-Instruct (WizardLM) surpasses the baseline model (Alpaca) trained using Self-Instruct with a large margin in a broad spectrum of benchmarks.

reference:
[1] Self-Instruct: Aligning Language Models with Self-Generated Instructions

Weakness 1.2: The success of the proposed method largely depend on the abilities of powerful LLMs such as ChatGPT.

In the experiment, the baselines we compared, Alpaca-13b and Vicuna-13b, their training data also came from ChatGPT. Our experiment shows that under the condition of using the same powerful LLM, the model trained by Evol-Instruct will perform better on a wide range of benchmarks. Further, when we use a weaker model LlaMA-2-70B-Chat to replace ChatGPT to execute Evol-Instruct, the trained model is still stronger than the models (i.e., Alpaca-13b and Vicuna-13b) trained by ChatGPT outputs. Please see the details in our response to your Weakness 4.

Weakness 2: The authors compare WizardLM with Vicuna by using the same amount of generated instructions. However, the cost or token consumption used for collecting the datasets (especially compared with Alpaca) should also be controlled or at least mentioned.

Our motivation is to investigate whether, under the condition of keeping the number of instructions constant, simply increasing the complexity of each instruction can be beneficial to the SFT performance of LLMs. More token consumption is a reasonable expectation for this approach. We have counted the number of tokens required for generating 70k ShareGPT data, Alpaca data, and our 70k WizardLM data. Detailed statistics are as follows: ShareGPT(70k) 118.3M, Alpaca 3.02M, WizardLM(70k) 18.84M. To verify when the token consumption is exactly the same, our method can still outperforms the self-instruct method for generating Alpaca. We randomly selected 11,300 samples from our 70k data, totaling 3.02M tokens, equivalent to Alpaca. We then trained the Wizard-13b(3M) model. The results, as shown in the following table, indicate that Wizard-13b(3M) still significantly surpasses Alpaca-13b in all benchmarks. It's worth noting that the model we trained with only 3M Tokens can surprisingly perform comparably to the Vicuna-13b trained with 118M Tokens.

Weakness 3.1: Comparisons with other methods for instruction data generation methods

As shown in the table in the section of New Baselines of General Response, we add more baselines such as Baize, CAMEL, Tulu. The extensive results indicate that our WizardLM also significantly surpasses these open source models, which demonstrates that the convincing quality of instruction-following dataset crafted by Evol-Instruct.

Weakness 3.2: More detailed ablation study is required to make the results more convincing.

We agree with you, seed instructions is very important for the model performance. So we choose the training data of Alpaca-13b as the seed instructions for our execution of Evol-Instruct. The main submission's Table 1 experimental results show that, based on the same seed instructions, WizardLM-13b after executing Evol-Instruct significantly outperforms Alpaca-13b on a series of benchmarks with a large margin. Although ShareGPT has different seed instructions from Alpaca data, its quality is widely recognized as far superior to Alpaca data. We chose to compare with Vicuna-13b, which uses ShareGPT as training data, in order to demonstrate that even a relatively low-quality instruction-following dataset, after applying our proposed Evol-Instruct, can surpass higher-quality data. To further verify the effectiveness of our method, we use ShareGPT as a new seed, and apply our evol-instruct method on it to produce a new 70k dataset, then train a model named WizardLM-13b (ShareGPT Seed). As shown in the following table, the average performance across nine benchmarks of this model is even higher than the previous version of WizardLM (the seed is Alpaca data). The consistent results of Alpaca data and ShareGPT as evolutionary seeds indicate that Evol-Instruct can significantly improve the performance of fine-tuning pre-trained language models using specific instruction data.

Weakness 4: Also, it would be very helpful to test the proposed data generation methods using other LLMs (e.g., LLaMA) instead of ChatGPT to better understand the proposed data synthesis pipeline.

To verify that our proposed Evol-Instruct method is not dependent on ChatGPT, we chose LlaMA-2-70B-Chat as a substitute for ChatGPT. Following the methods outlined in the paper, we re-executed the entire evolution process starting from Alpaca data and trained the WizardLM(LlaMA-2-70B-Chat) model with randomly selected 70k data. Please refer to below table for details, WizardLM(LlaMA-2-70B-Chat) still significantly outperforms Alpaca-13b trained solely on Alpaca data and even outperforms Vicuna-13B trained with much more ChatGPT tokens.

We hope these resolve all your concerns about this paper!