WizardCoder: Empowering Code Large Language Models with Evol-Instruct

Thank you for your thorough review! We are grateful for your professional feedback. We are particularly encouraged that you think our method is innovative and our paper is good!

Weakness: In section 4.3, Table 1, Result of pass@1(%) on HumanEval and MBPP, the result of pass@1(%) on MBPP is not as good as the result on HumanEval. HumanEval only tests a narrow distribution and can be overfitted. The benchmark needs to be upgraded.

We acknowledge the need for a more comprehensive set of benchmarks in the coding domain. Our evaluation extends beyond HumanEval, encompassing a diverse set of benchmarks, including MBPP, DS-1000, and MultiPl-E. Notably, when compared to SOTA open-source software (OSS) models, our models showcase substantial improvements across various benchmarks. Specifically, our models achieve approximately a 20-point improvement on HumanEval, a 5.2-point improvement on MBPP, a 7.4-point improvement on DS-1000 (insertion), and an average 9.3-point improvement on MultiPL-E. These significant enhancements underscore the effectiveness of our work. Additionally, it's worth noting that Codellama-instruct only achieves a 2-point improvement compared to Codellama on MBPP.

Question 1: In section 3.2, TRAINING WizardCoder," We iteratively employ the Code Evol-Instruct technique on this dataset consisting of around 20k samples to produce evolved data." What is the number of iterations and the criteria for completion?

We conducted a total of 4 iterations. Figure 3a reveals that the optimal performance is achieved after 3 rounds of evolutions on the dev set. Based on this observation, we made the decision to discontinue further iterations.

Question 2: In section 3, when iterating the Code Eval-Instruct on the sample dataset, what model is used to generate evolved data? Does the difference in the quality of pre-trained model outputs lead to different results?

Please refer to the section "2. Common Response to using different evolution execution models" in the Overall Response. We have provided the results of using GPT-4 as the evolution execution model.

We have mentioned in Section 4.2 that we use GPT3.5 (gpt3.5-turbo) to generate evolved data. We conduct additional experiments, replacing GPT3.5 (gpt3.5-turbo) with GPT-4 to generate evolved rounds. HumanEval Pass@1 scores increased from 73.2 to 73.8 for 34B and from 59.8 to 62.2 for 15B. Using a superior LLM does contribute to better evolved data; however, the improvement is not strongly correlated with the coding performance of the evolution executing model (88.4 for GPT4 and 73.2 for GPT3.5).

In summary, we found your comments insightful and hope that our responses have addressed any questions. If you have additional questions, please feel free to reach out. Once again, we sincerely appreciate your thoughtful review!

Thank you for your thorough review! We are grateful for your dedication throughout the review period. Your positive comments, "proposed instruction-tuning method is valuable" and "the idea is interesting", are sincerely appreciated.

Weakness 1: It is not clear how the authors came up with the list of heuristics for data evolution. This unclarity makes such approaches less applicable to wide range of tasks.

Question 3: Can you clarify how did you come up with the heuristics? Did you eliminate any of them based on the final outcome? Was the process a trial-error approach or there were some insights behind making any of the decisions?

We created these five heuristic methods to increase the complexity of code instructions, aligning with the characteristics of the code domain and previous research on code models' limitations. The first three heuristics naturally escalate coding task difficulty by incorporating additional requirements, replacing common requirements, or introducing extra reasoning steps. This aligns with observed trends on platforms like LeetCode, where tasks with more requirements or intricate reasoning are considered more challenging. The fourth heuristic introduces erroneous code as an adversarial sample, inspired by prior research in attacking pre-trained code models [1][2], serving to assess the robustness of the code LLM. The fifth heuristic, emphasizing time and space complexity, aligns with the significance of optimization in the coding domain. Previous studies [3] indicate that tasks requiring optimized program performance pose heightened difficulty, making this direction a valuable avenue for complexity improvement.

[1] Yang, Zhou et al. “Natural Attack for Pre-trained Models of Code.” 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE).

[2] Jha, Akshita et al. “CodeAttack: Code-based Adversarial Attacks for Pre-Trained Programming Language Models.” AAAI 2022.

[3] Madaan, A. et al. "Learning Performance-Improving Code Edits." (arxiv, 2023)

Weakness 3: While the idea is interesting, but it seems very incremental compared to prior work and the contributions are limited.

Please refer to "1. Common Response to novelty and contributions" in the Overall Response regarding the novelty and contributions of our work.

Question 1: Table 4, the Pass@1 score decreases as the number of iterations increases. Do you have any insights as why this is happening?

In the upper part of Table 4, the Pass@1 scores exhibit fluctuations rather than a monotonic decrease; it's important to highlight that only the fourth round notably deteriorates. This decline may be attributed to two key factors. First, continual evolution could lead to overly complicated instructions, rendering them unreasonable. Second, beyond a certain round, instructions may become excessively complex, posing a significant challenge to the evolution executing model (i.e., gpt3.5-turbo) in generating suitable responses.

In the lower part of Table 4, the decreasing scores are normal. As instructions become more complex, the length of samples increases, leading to a reduction in the number of samples in each round to control for the same token count.

It's crucial to recognize that individual metrics in Table 4 for a specific round with complex data might not fully represent the overall impact on the final model performance. Figure 3 provides insights, indicating that merging different complex-level data can result in performance gains.

Question 2: How the performance of your model increases as you use less capable models for generating evolving data? In addition, do you think we can use the same model to generate the data for next iteration?

Please refer to "2. Common Response to using different evolution execution models" in Overall Response. We have provided the results of using less capable models as evolution execution models.

We explored using the open-source models (OSS) CodeLlama-Instruct-34B for generating evolved instructions. However, it demonstrated relatively low coding performance in response generation. To address this, we fine-tuned it using the code-alpaca dataset and utilized this model for response generation. The OSS-based Code Evol-Instruct tuned models also exhibited performance improvements compared to the code-alpaca fine-tuned model, increasing from 65.9 to 70.1 for 34B and from 45.7 to 55.5 for 15B. Despite CodeLlama exhibiting weaker performance than GPT3.5 (gpt3.5-turbo), the performance gap significantly diminishes when utilizing our Code-Evol Instruct to generate evolved rounds (73.2 vs. 70.1).

Generating data for the next iteration using the same model is challenging because the OSS code model is trained to generate code responses and not to generate evolved instructions.

Overall, we hope that our responses clarify your questions. Please let us know if you have further questions, and thank you again for your review!

Question 9: Section 5 paragraph 1, "amalgamate the training set", what about using a simpler word like "merge the training set"?

Thank you. We updated it in Section 5 paragraph 1.

Question 10: Section 5 paragraph 2. The Reviewer assumes more rounds equals more complexity. But it's better to make it explicit and provide some examples to show what does complexity actually mean in this context.

Yes. More rounds equals more complexity. We have provided some evol examples in Appendix D.

Question 11: Section 5 "Complexity and Similarity". This paragraph didn't really talk about complexity, unless more rounds = more complexity?

Yes.

Question 12: Table 4. It's better to show the confidence interval on this table, if possible, since 400 examples is not that large a test set.

The primary objective of Table 4 is to underscore the pivotal role of instruction complexity in enhancing coding performance. The table effectively demonstrates that models trained with any round of round 1-4 data (with evolved complexity) outperform round 0 data (without evolved complexity) with a substantial gap. Given the clear difference in performance between any evolved round and non-evolved data, the presentation of confidence intervals is unnecessary to prove that evolved data is better.

Question 13: Section 6: conclusion. ".. the pivotal role of instruction complexity in enhancing coding performance". Reviewer doesn't think this claim holds well given the analysis in the paper. For example, in Table 4, later rounds (assuming more complexity) lead to a lower performance (or statistically insignificant changes).

We appreciate your constructive comments. What we intended to convey is that making instruction more complex benefit code model fine-tuning. To better substantiate our claim and eliminate possible misunderstandings, we would like to further explain Table 4 and the corresponding experimental results. Table 4 demonstrates that models trained with any round of round 1-4 data (incorporating evolved complexity) exhibit significantly better performance than round 0 data (devoid of evolved complexity). Though rounds 1-4 are not a monotonous process with increasing complexity, they are all better than the code-alpaca data with low complexity. This indicates that complexity helps to achieve good performance. Figure 3 further supports this by showcasing that the combination of rounds 0-3 achieves the best performance, indicating the beneficial impact of introducing instruction complexity. We sincerely hope that the clarification could address your concern.

Question 1: Where does the new coding solution come after expanding the instruction? Did we use the base model itself, or GPT-3 to generate them?

We use the same model (gpt3.5-turbo) to generate the code solutions.

Question 2: Did we do any deduplication of the expanded instructions?

The seed data, employing the self-instruct technique, is inherently deduplicated. Given that the expanded instructions derive from distinct coding tasks, there is no need for deduplication in the subsequent evolved process.

Question 3: Did we verify the quality of new instructions by executing the code generated?

We have tried to execute all the generated Python codes. All of them can run without any compile errors.

Question 4: Figure 1 lower plot. It's better to use pattern instead of color to separate two data classes to make it more friendly for color-blindness.

Thank you for your suggestion. We updated Figure 1 lower plot with different patterns.

Question 5: Section 1 paragraph 2. Reviewer thinks "Code" need not to have capitalized first letter.

Thank you for your suggestion. We updated Section 1 paragraph 2.

Question 6: Section 2 Related Work. We should also mention CodeLlama's self-instruct work and compare the differences. https://arxiv.org/abs/2308.12950

We have mentioned the CodeLlama-Series models in the related work. The CodeLlama-Instruct model is also one of the baseline models in our experiments. We include extra discussion of this model in the related works section.

Question 7: Page 4, Code Evolution Heuristic Methods table. Please explain the 4th instruct's purpose and examples of expansion. "Provide a piece of erroneous code as a reference to increase misdirection". It's not clear to the Reviewer what this is doing and why it would be effective.

For a specific instruction, this method includes an error code in it. For example, the original instruction is: Write a Python function to tell me what the date is today.

The evolved instruction is: Write a Python function to tell me what the date is today, using the datetime module. The function should return a string in the format of “Today is YYYY-MM-DD”. For example, if today is November 17, 2023, the function should return “Today is 2023-11-17”. As a reference, here is a piece of erroneous code that does not work as intended:

import datetime
def get_date():
    today = datetime.date.today()
    return "Today is " + today

This erroneous code serves as an adversarial sample designed to challenge the code LLM, akin to previous works focused on attacking pre-trained code models. [1][2]

[1] Yang, Z., Shi, J., He, J., & Lo, D. (2022). Natural Attack for Pre-trained Models of Code. 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), 1482-1493.

[2] Jha, A., & Reddy, C.K. (2022). CodeAttack: Code-based Adversarial Attacks for Pre-Trained Programming Language Models. AAAI Conference on Artificial Intelligence.

Question 8: Page 4, Code Evolution Heuristic Methods table. Last prompt about increasing time/space complexity: how can we verify the response actually achieved it?

This heuristic, which underscores time and space complexity, aligns with the importance of optimization in the coding domain. Previous studies [3] suggest that tasks demanding optimized program performance present heightened difficulty, rendering this direction a valuable avenue for complexity improvement. Ideally, all responses should achieve 100% compliance with the specified complexity requirements. Given our use of GPT3.5(gpt3.5-turbo) to generate responses, we can express a reasonable level of confidence that the responses are well-aligned with the stipulated requirements.

[3] Madaan, A., Shypula, A., Alon, U., Hashemi, M., Ranganathan, P., Yang, Y., Neubig, G., & Yazdanbakhsh, A. (2023). Learning Performance-Improving Code Edits. ArXiv, abs/2302.07867.

Thank you for your insightful reviews! Your professional feedback provides valuable guidance for writing a more comprehensive and competitive paper. We are particularly encouraged that you found our WizardCoder's performance impressive!

Weakness 1: Missing some key details. Where does the new coding solution come after expanding the instruction? Did we use the base model itself, or GPT-3 to generate them? Did we do any deduplication of the expanded instructions? Did we verify the quality of new instructions by executing the code generated?

The coding solution is generated by the same model that produces the evolved instructions, specifically gpt3.5-turbo. The seed data, employing the self-instruct technique, is inherently deduplicated. Given that the expanded instructions derive from distinct coding tasks, there is no need for deduplication in the subsequent evolved process. We have tried to execute all the generated Python codes. All of them can run without any compile errors.

Weakness 2: Weak analysis. Why does round 4 have a negative impact on the quality? The paper mentioned "some evolution process will fail" but provided no detail. Other analysis related questions will be in the "questions" section.

Our empirical results reveal a potential saturation point from Code Evol-Instruct after a specific number of rounds, namely 4. This decline may be attributed to two key factors. First, continual evolution could lead to overly complicated instructions, rendering them unreasonable. Second, beyond a certain round, instructions may become excessively complex, posing a huge challenge to the evolution executing model (i.e., gpt3.5-turbo) to generate suitable responses.

Weakness 3: Writing quality can be improved. The paper uses more sentences to repeatedly claim WizardCoder's performance (which is indeed impressive) but vague on implementation details and analysis.

We present the implementation details in Section 4.2, with the evolution prompts discussed in Section 3.1 and the training process outlined in Section 3.2. The prompt formats are provided in Appendix A, baseline details in Appendix B, and data filtering details in Appendix C.

Furthermore, our analysis in Section 5 encompasses various aspects, such as the impact of different numbers of evolution rounds, the relationships between complexity and quantity, and the correlations between complexity and similarity.

Weakness 4: Inconsistency in the data. Figure 2 reported GPT-4 has a HumanEval pass@1 of 88.4, while Table 1 showed 67.0.

In Footnote 5, Page 6, we have discussed why the scores in Figure 2 and Table 1 are different. In Figure 2, the scores are reported by the Eval-Plus leaderboard with OpenAI's APIs. In Table 1, the scores are reported by OpenAI's report and CodeLlama's paper. In order to make the conclusion more rigorous, all the claims regarding GPT3.5(ChatGPT) and GPT4 in our paper rely on results obtained through OpenAI's APIs.

Thank you for taking the time to review our rebuttals, and we appreciate your ongoing consideration of our submission.

We have conducted additional experiments to replace the evolution execution models with the same family of our models. The results are presented in the "Overall Response: 2. Common Response to using different evolution execution models". For your convenience, we also include some results here:

As you suggested, we explored using the same family of open-sourced models (OSS), specifically the CodeLlama-Instruct-34B, for executing the entire process of Code Evol-Instruct. The OSS-base Code Evol-Instruct tuned models also exhibited performance improvements compared to the code-alpaca fine-tuned model, increasing from 65.9 to 70.1. This demonstrates that our proposed method is still effective without GPT-3.5.

To eliminate any possible misunderstandings, we are willing to explain our experimental setup for you in detail. Our experimental setup is reasonable lies in that our baseline code-alpaca is also generated from the GPT3.5 model but using a different method (i.e., self-instruct). As shown in the upper part of Table 4 of Section 5 in the main submission, when ensuring a similar amount of training data, the model (rounds 1-4) fine-tuned from Code Evol-Instruct created data is superior to the model (rounds 0) fine-tuned with code-alpaca data.

Overall, our Code Evol-Instruct emerges as the pivotal factor in driving performance enhancement, rather than some implicit data leakage from GPT-3.5. We sincerely hope that the clarification could address your concern.

Question 4: Section 3.1: We optimised the evolutionary instructions by eliminating deepening and complicating inputs, as well as In-Breadth Evolution. What does this mean? Not clear how 1 and 2 are different, can you describe precisely with an example?

Thank you for your reviews. We have updated the section 3.1 to make it more clear.

Example: Original Instruction: Write a MongoDB query to select all documents in a collection where the field 'category' is 'clothes' and the 'brand' field is not equal to 'Nike'.

The first method "Add new constraints and requirements..." asks the model to include more requirements in the instruction: Write a MongoDB query to select all documents in a collection where the field 'category' is 'clothes' and the 'brand' field is not equal to 'Nike', and the 'price' field is greater than or equal to 100 and less than or equal to 500.

The second method "Replace a commonly used requirement..." asks the model to replace the requirements in the instruction: Write a MongoDB query to select all documents in a collection where the field 'material' is 'cashmere' and the 'designer' field is not equal to 'Chanel'.

Question 5: Can you provide details of the compute infrastructure involved and hyperparameters in fine-tuning? (number of GPU hours, batch-size, sequence length) and other details

The details of hyper-parameters are included in Section 4.2. We train our models with 24 V100 GPUs, 15 hours for 15B, and 32 hours for 34B.

Question 6: The abstract (and some places in the paper) comment on model sizes of Claude and Bard as 'largest closed-source LLMs', this is not verifiable, so I suggest the authors re-word this without commenting on their model size and avoid other such colloquial writing present in the paper to improve its soundness.

Thank you for your suggestion. We re-word this without commenting on their model size.

Typo Intro Para 3: several key adaptions --> adaptations

Thank you. The typo has been fixed.

Question 1: Figure 3: Why are results from evol round 0 + 1 + 2 + 3 + 4 worse than 0 + 1 + 2 + 3? Do returns from Evol-Instruct start to diminish or turn negative after a certain number of rounds? This is not discussed or explored in this paper.

Our empirical results reveal a potential saturation point from Code Evol-Instruct after a specific number of rounds, namely 4. This decline may be attributed to two key factors. First, continual evolution could lead to overly complicated instructions, rendering them unreasonable. Second, beyond a certain round, instructions may become excessively complex, posing a huge challenge to the evolution executing model (i.e., gpt3.5-turbo) to generate suitable responses.

Question 2: Analysis - Complexity and Quantity: This section is not clear to me. What are the results shown in Table 4? How does Table 4 make one conclude that the gains are not due to increase in samples or tokens? Are the 4 rows in Table 4 pass@1 metric for model checkpoints when sequentially trained on each round?

In Table 4, all models are fine-tuned separately from scratch, not sequentially. We fine-tune the models with only the specific round data, which contains a similar number of samples (upper part) or tokens (lower part). The results show that with a similar number of samples or tokens, the models trained with evolved data achieve better pass@1. We have updated this section to make it clear.

Question 3: How dependent is the performance of Evol-Instruct on the evolving model? Will the performance substantially improve if we use GPT-4 to generate evolved rounds? Will it worsen if the evolving model is changed to other open sourced alternatives?

Please refer to the section "2. Common Response to using different evolution execution models" in the Overall Response. We have provided the results of using GPT-4 or open-sourced models as evolution execution models.

In additional experiments, we replaced GPT3.5 (gpt3.5-turbo) with GPT-4 for generating evolved rounds. HumanEval Pass@1 scores increased from 73.2 to 73.8 for 34B and from 59.8 to 62.2 for 15B.

Furthermore, we explored using the open-source models (OSS) CodeLlama-Instruct-34B for generating evolved instructions. However, it demonstrated relatively low coding performance in response generation. To address this, we fine-tuned it using the code-alpaca dataset and utilized this model for response generation. The OSS-based Code Evol-Instruct tuned models also exhibited performance improvements compared to the code-alpaca fine-tuned model, increasing from 65.9 to 70.1 for 34B and from 45.7 to 55.5 for 15B.

While GPT-4 demonstrates superior coding performance with a score of 88.4 compared to GPT3.5 (gpt3.5-turbo) with a score of 73.2, leveraging a more powerful model for generating evolved rounds does not result in a proportional performance gain (73.8 vs. 73.2). Conversely, despite CodeLlama exhibiting weaker performance than GPT3.5 (gpt3.5-turbo), the performance gap significantly diminishes when utilizing our Code-Evol Instruct to generate evolved rounds (73.2 vs. 70.1). This underscores the observation that employing a stronger model for evolved rounds can somewhat enhance performance, but the extent of improvement is not strongly correlated to the evolution executing model's coding performance. Our Code Evol-Instruct emerges as the pivotal factor in driving performance enhancement.

Thank you for your insightful reviews! Your professional feedback provides valuable guidance for writing a more comprehensive and competitive paper. We are particularly encouraged that you found our results on code benchmarks impressive!

Weakness 1: While results on code benchmarks are impressive, novelty of the scientific methodology itself is quite limited as it is an adaptation of Evol-Instruct for Code.

Please refer to the common response 1 regarding the novelty and contributions of our work.

Weakness 2: Missing human assessment - It is not clear how useful the final fine-tuned model is outside the benchmarks that focus exclusively on functional correctness. Model hasn't been tested on developer productivity tasks like completion, code refinement.

Tasks related to developer productivity can serve as useful assessments of the practical applicability of code models in real-world industrial scenarios. However, even the most advanced LLM, GPT-4, faces challenges when applied to real-life industrial coding tasks. Thus, the most popular evaluation method for code LLMs research is the execution-based function-level code snippet generation, where the generated code snippet/patch undergoes unit tests to ensure functional correctness. We follow this research line to explore methods to improve the performance of function-level code snippet generation. Our evaluation strategy aligns with the previous and contemporary SOTA works in this research line, such as Starcoder and Codellama, as well as works accepted by ICLR23 and KDD23, including CodeGen and CodeGeeX. These works similarly exclude human assessment of code.

The chosen evaluation tasks, namely HumanEval, MBPP, DS-1000, and MultiPL-E, collectively cover a spectrum of diverse code completion and insertion scenarios. Specifically, HumanEval addresses function-level code completion, MBPP focuses on task-level code generation, DS-1000 entails statement-level code completion and insertion tasks within the data science domain, and MultiPL-E focuses on diverse programming languages code completion. This comprehensive evaluation framework ensures a thorough examination of the model's utility across various dimensions.

Weakness 3: Not clear if data leakage has been prevented. Does the evolved data or seed data overlap with HumanEval or other benchmarks' test set?

To prevent data leakage, we implemented a filtering process. Similar to the method in Section 5 (Complexity and Similarity), we use all test samples as queries to retrieve the top 5 samples from training data, leveraging the SOTA embeddings model, gte-large. Subsequently, GPT-4 is employed to assess the similarity between the retrieved samples and the test sample. If a match is found, the sample is excluded. Details of the prompt and a more thorough discussion are presented in Appendix C.

Weakness 4: The paper assumes the reader to be familiar with Evol Instruct, and does not provide sufficient context to follow the method.

Thank you for your suggestion. We have included the general Evol-Instruct discussion in the related work section.