AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data

W1: Concerns of the presentation.

• Thank you for your valuable advice on improving our presentation! Our motivation stems from the following two considerations: 1) Existing Code LLM pre-train methods typically use multi-source data, while Code LLM fine-tuning methods focus on developing high-quality single-source datasets; 2) The current open-source Code LLM community already has various high-quality single-source fine-tuning data. To better utilize existing data resources and contribute to the Code LLM community, we are pioneering the integration of multi-source data for Code LLM fine-tuning.
• Additionally, we have provided more examples in the global response PDF that demonstrate how our method harmonizes inherent conflicts in multi-source data. We will also refine our presentation in the latest version of the manuscript.

W2&Q1: Concerns about the selection of data customized by AlchemistPrompt.

• Sorry for the confusion. We calculate the Conditional Perplexity Discrepancy (CPD, refer to lines 248-252 of the manuscript) and selectively chose data with higher CPD values for AlchemistPrompts harmonization. Conditional perplexity discrepancy is an indicator of how data affects the complexity of model-generated responses under given conditions (i.e., instructions), and its calculation formula is $CPD = Perplexity(instruction + response) − Perplexity(response)$. The level of CPD reflects the impact of the conditional instruction on the complexity of the generated response. Specifically, a high CPD indicates that the perplexity of the generated response significantly increases under the presence of a conditional instruction, which usually reflects a poor alignment between the instruction and the response, the instruction may be unclear or not specific enough, or insufficient contextual information, thereby increasing the difficulty of model response generation. By analyzing high CPD values, we can identify cases where instructions and responses are poorly aligned and more effectively optimize data quality. As deeply analyzed in Figure 8 of the manuscript, AlchemistPrompts can effectively harmonize the discrepancy between instructions and responses.

Q2: Why should we expect AlchemistCoder models to be better generalists in NL tasks?

• Indeed, similar findings are also present in existing work in some other fields. Based on our insights, we attribute this phenomenon to the following three points:
  • Enhanced reasoning abilities: Coding abilities themselves reflect reasoning capabilities. Additionally, multi-source integration significantly increases the diversity of fine-tuning data, which often includes detailed analyses, reasoning, and explanatory annotations, all of which contribute to improved model reasoning abilities.
  • Better instruction-following abilities: AlchemistPrompts supplements instructions with details about programming languages, algorithm concepts, and code characteristics corresponding to the responses, representing an optimization of instruction/response alignment specific to coding capabilities. Thus, AlchemistPrompts can refine the alignment within instruction-response pairs and enhance the instruction-following abilities of fine-tuned models.
  • Improved context understanding: As shown in Figure A4~A11 of the manuscript, both our AlchemistPrompts and code comprehension tasks can provide training for the context-understanding capabilities of models.
• In summary, our fine-tuning data includes a substantial amount of natural language descriptions in addition to code snippets, leading to the improvements from fine-tuning that are not limited to coding abilities. The enhancements in the three abilities mentioned above also benefit tasks MMLU for multitasking language understanding, BBH for comprehensive reasoning, and GSM8K for mathematical reasoning.

W1: Concerns of AlchemistPrompts.

• Thanks for your insightful concerns! To fairly compare with other models, we provide a new version of Table 1 in the global response PDF, which includes details of training corpora sources for fine-tuned Code LLMs. Compared to methods that heavily rely on GPT-4 to generate entire new datasets, we strive to minimize our dependence on GPT-4 and only generate AlchemistPrompts for 5% of the data. More importantly, existing Code LLM methods usually use strong models (e.g., GPT-4) to directly generate code for fine-tuning, whereas we do not. The AlchemistPrompts we generate are concise textual descriptions that do not include code (refer to Figures A4 and A5 in the manuscript), fundamentally differing from other methods in both their intended goals and practical effects. To sum up, we do not rely directly on the code capabilities of strong models for supervision and our method achieves more effective optimization of fine-tuning data with greater diversity, higher quality, and lower cost for empowering AlchemistCoder to obtain promising and comprehensive code capabilities.
• Additionally, we highlight the importance of combining AlchemistPrompts with multi-source data to achieve exceptional performance and we present more examples in Figure R1 of the global response PDF that demonstrate how our method harmonizes inherent conflicts in multi-source data. This is fundamentally different from fine-tuning on a single-source high-quality dataset. The synergy between AlchemistPrompts and multi-source data sets AlchemistCoder apart from other models and demonstrates a new path for enhancing code LLMs. Our contributions also lie in introducing new insights into improving prompts and designing instruction fine-tuning tasks to develop better open-source models: 1) The design philosophy and utility of AlchemistPrompts (please refer to our response to Reviewer q6UB W1); 2) The data construction process itself reflects higher-level capabilities and can guide model training.
• Indeed, GPT-4 holds advantages in code and general capabilities and this urges us to continue advancing towards our goal: exploring techniques to optimize large models and bridging the gap between open-source and closed-source models. We will open source our fine-tuning data to contribute to the Code LLM community and explore the expansion of the proposed method to other LLM tasks.

Q1: For different portions of AlchemistPrompts other than 5%, does the general increasing trend of accuracy on multi-source data in Table 4 maintained?

• Yes, although 5% of AlchemistPrompts significantly contribute to performance improvement, other proportions (1–20%) of AlchemistPrompts also maintain a similar trend of enhancement. It is important to note that code data from different sources may vary greatly in language style and content, including question types, code style, presence of comments, test cases, etc. Therefore, mixing multi-source data has a double-edged sword effect: it provides necessary diversity but may also introduce significant domain gaps. To effectively bridge this gap while maintaining diversity, adding concise corpus generated by the same Alchemist model (i.e., AlchemistPrompts with similar language styles) to a small amount of data can effectively address this issue. Additionally, AlchemistPrompts are beneficial for refining the alignment within instruction-response pairs to enhance the instruction-following abilities of fine-tuned models.

W1: Missing ablations.

• Thanks for your meticulous suggestions! We provide a new version of Table 1 in the global response PDF, which includes details of the training corpus used for fine-tuned Code LLMs.
• Here, we reorganize the following table based on Table 4 to more clearly demonstrate the impact of multi-source data, harmonization, and code comprehension:

  Method	HumanEval (Pass@1)	MBPP (Pass@1)
  Baseline (CodeLlama-Python-7B)	37.8	57.6
  + Multi-source data (w/o data decontamination)	54.6	57.9
  + Multi-source data (w data decontamination)	59.8	58.2
  + Harmonizations (AlchemistPrompts)	72.0	63.4
  + Code comprehension (Instruction Evolution)	71.3	65.8
  + Code comprehension (Data Filtering)	73.8	67.7
  + Code comprehension (Code Review)	74.4	68.5

• Due to inherent conflicts, the improvement brought by directly mixing multi-source data is limited (refer to the second/third row of the table above and DirectMix-L-7B in Figure 1). We believe that the key efficacy of our method lies in optimizing the data to possess the following characteristics conducive to effective model training:
  • Balance between diversity and source gap: Multi-source mixing may bring necessary diversity but harmful domain gaps. Adding concise corpus generated from the same Alchemist model (i.e., AlchemistPrompts with similar language styles) can effectively bridge this gap.
  • Better alignment within instruction-response pairs: AlchemistPrompts supplements instructions with details about programming languages, algorithm concepts, and code characteristics corresponding to the responses, representing an optimization of instruction/response alignment specific to coding capabilities. As shown on the right side of Figure 8, the instructions customized by AlchemistPrompts are closer to the responses in the feature space. Meanwhile, as shown on the left side of Figure 8, Conditional Perplexity Discrepancy (CPD) quantifies the difficulty change in generating responses before and after adding specific inputs (e.g., instructions) to the model (the smaller the value, the easier it becomes). The generally smaller CPD values after adding AlchemistPrompts reflect the facilitating effect of AlchemistPrompts on improving model performance.
  • Reflecting code comprehension: Instruction evolution task data, which communicates the relationship between data before and after evolution, addressing a gap in existing work using instruction evolution data. Data filtering and code evaluation task data, provide the model with a deep understanding of both high-quality and low-quality code.
• In addition, we have explored which data features are harmful to model training (refer to the fourth row of the table above and lines 162-171 of the manuscript):
  • Responses that are overly concise and devoid of code. These answers usually provide straightforward replies to the instructions, overlooking both the code solution and explanatory annotations. Moreover, these instances often contain overly simplistic questions in the instructions.
  • Code solutions that are either non-compilable or do not pass test cases (pertaining to particular samples).

W2: Python-based evals.

• Indeed, similar to existing work, we focus on Python due to its popularity and the wealth of benchmark resources available. We have provided results on multilingual code generation (refer to Table 2 in the manuscript, including C++, Go, Java, and JavaScript) and results on general benchmarks (refer to Table 5 in the manuscript, including MMLU for multitask language understanding, BBH for comprehensive reasoning, and GSM8K for mathematical ability). These extensive experimental results demonstrate that our AlchemistCoder series models deliver comprehensive code capabilities and reasoning abilities.
• Additionally, each single source data itself contains multiple programming languages, not just one language per source, making it challenging to separately ablate the corpus for each language. Under the efficacy of multi-source integration, our fine-tuning data includes nearly 60% of various programming languages (refer to Figure 5 in the manuscript, including C, Java, HTML, SQL, etc.), aiming to contribute more universally powerful open-source models to the Code LLM community.

Q1: Clarification of the ablation study.

• Sorry if that statement was confusing. Code data from different sources may vary significantly in language style and content, including question types, code style, presence of comments, test cases, etc. Therefore, multi-source data mixing is a double-edged sword: it provides necessary diversity but can also bring large domain gaps. Adding concise corpus generated from the same Alchemist model (i.e., AlchemistPrompts with similar language styles) to a small amout of data can effectively bridge this gap while maintaining diversity. Besides, the inclusion of 5% AlchemistPrompts also represents a balance between cost and performance.

Q2: Is code comprehension data augmentation a novel contribution?

• Yes, we pioneer novel code comprehension tasks for fine-tuning Code LLMs. Meanwhile, we also aim to provide new insights into the construction of fine-tuning data: the data construction process itself reflects higher-level capabilities and can guide model training.

L1: The Reliance on GPT-4.

• Our method does indeed rely on Alchemist models (i.e., GPT-4), and we have discussed this in section A of limitations. Compared to methods that heavily rely on GPT-4 to generate entire new datasets, we strive to minimize our dependence on GPT-4 and only generate concise AlchemistPrompts for 5% of the data.
• More importantly, this urges us to continue advancing toward our goal: exploring techniques to optimize large models and bridging the gap between open-source and closed-source models. And we will open source our fine-tuning data to contribute to the Code LLM community.

W1&W2: The novelty of the proposed method.

• Actually, our AlchemistPrompts are entirely different from existing instruction evolution techniques in several aspects:
  • Different designed goals: Instruction evolution techniques are designed to "expand into a richer and more complex set of instructions," whereas the goal of AlchemistPrompts is to harmonize multi-source data and instruction-response pairs. In short, the former aims at "expansion", while the latter focuses on "harmonization".
  • Different operational methods: Instruction evolution techniques start from an initial set of instructions only and typically involve multiple rounds of iterative generation for all data. In contrast, AlchemistPrompts use only a small amount of data, require both instructions and responses, and only need to generate once. Additionally, instruction evolution techniques require generating new responses, whereas our AlchemistPrompts do not.
  • Different practical effects: In terms of diversity, instruction evolution techniques significantly expand the data, while AlchemistPrompts suppress excessive diversity in multi-source data by incorporating corpora with similar language styles. Regarding controllability, instruction evolution techniques generate data with reference to a broad evolutionary direction, resulting in relatively high randomness, while AlchemistPrompts focus on fine-grained alignment within individual instruction-response pairs, thereby providing stronger controllability. For data alteration, AlchemistPrompts only insert concise corpus into the instructions, whereas instruction evolution techniques make substantial changes to both instructions and responses.
• Additionally, our AlchemistPrompts explore a novel application of hindsight relabeling in the domain of Code LLM fine-tuning, which differs fundamentally from previous methods. Specifically:

  Method	Designed Purpose	Relabeled Object	Generation Mode	Relabeling Period	Experience Source
  Previous Methods [a,b,c,d]	Alignment for Preferences	Conditional Goal	Handcrafting/Scripting	Postprocessing	Human
  AlchemistPrompts (ours)	Harmonization for Multi-source Data	Data Instruction	LLM Generation	Preprocessing	LLM

• More importantly, our core contributions lie in pioneering the application of multi-source data in the field of Code LLM fine-tuning and we are the first to unveil inherent conflicts in multi-source code corpora. To achieve this, we propose logically progressive components of our method: AlchemistPrompts and code comprehension tasks. These key ideas introduce new insights into improving prompts and designing instruction fine-tuning tasks, enabling our fine-tuning data to be more diverse, higher quality, and lower cost for empowering AlchemistCoder with promising and comprehensive code capabilities.

Reference

[a] Andrychowicz M, et al. Hindsight experience replay. NeurIPS 2017.

[b] Li A, et al. Generalized hindsight for reinforcement learning. NeurIPS 2020.

[c] Packer C, et al. Hindsight task relabelling: Experience replay for sparse reward meta-rl. NeurIPS 2021.

[d] Korbak T, et al. Pretraining language models with human preferences. ICML 2023.

W3: Additional analysis on the effectiveness of the proposed method.

• Thanks for your considerate suggestions! For additional analysis on the effectiveness of our method, please refer to our response to Reviewer fYHy W1.

Q1: More insights into the relative contributions of data diversity and the harmonization process.

• Code data from different sources may vary significantly in language style and content, including question types, code style, presence of comments, test cases, etc. Therefore, multi-source data mixing is a double-edged sword: it provides necessary diversity but can also bring large domain gaps. Adding concise corpus generated from the same Alchemist model (i.e., AlchemistPrompts with similar language styles) to a small amount of data can effectively bridge this gap while maintaining diversity.
• In terms of balancing diversity and harmonization, for single-source data, the relatively lacking diversity is more important; for multi-source data, diversity is naturally introduced during mixing, so harmonizing conflicts becomes more crucial.

Q2&L1: Scalability & Data Complexity

• Thanks for your insightful advice! For scalability, due to limited computational resources, we focus on smaller models that are more popular and practical in the Code LLM field. For larger models, we will discuss this in our limitations and will open source our fine-tuning data to contribute to the Code LLM community, welcoming developers to apply it to larger models.
• In terms of data complexity, we assume that an increase in data complexity implies the inclusion of more diverse data sources. We deconstruct this into a higher degree of (possibly excessive) diversity and more varied quality. In such a scenario, the harmonization effect introduced by our method becomes increasingly critical, which may imply a greater need for a higher proportion of AlchemistPrompts to achieve optimal performance. As for extreme cases, this essentially means fine-tuning towards generalist models rather than specialized ones (e.g., Code LLMs), where the bottleneck tends to be the balance of various capabilities, i.e., like a seesaw.

L2: Data Bias

• Thank you for emphasizing the aspect of data bias. Our AlchemistPrompts enhance the instruction-following abilities of models, which can potentially mitigate biases. For example, if the model has a bias towards responding with Python code when the programming language is not specified, the inclusion of programming language declarations in AlchemistPrompts helps to alleviate this bias. We have not delved into this yet and will discuss it in our limitations.

Dear Reviewer q6UB,

We wish to express our gratitude for your extensive review and supportive feedback. Your feedback of minor questions about data complexity impacts is invaluable to us, and we are fully committed to thoughtfully incorporating your insights to enhance our paper. As the discussion phase is nearing its end, we are warmly concerned whether our rebuttal addresses your concerns.

It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Best regards,

Authors of Paper #10234