Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining

Thank you for your detailed review! We provide our responses as follows.

Q1: Regarding the openness of this work, will the authors consider making the data pipeline and models publicly available? This will greatly help future studies on Code LM pretraining.

Great point. We will try the best to open source the pipeline, model, and the annotator to facilitate future research in code LLM pretraining. However, due to proprietary reasons, this process would be long and we hope the reviewers can understand. We have also revised our paper to cover more technical details and insights.

Q2: Regarding the evaluation, can the authors provide some explanations as to why BigCodeBench results are omitted in most of the tables?

The primary reason is that BigCodeBench is a very comprehensive benchmark covering 1,140 task instances, each of which can involve multiple function calls from external libraries, and this makes evaluation on every checkpoint costly with our resource constraint. We are happy to incorporate more evaluation on BigCodeBench in the final revision.

Q3: Can the authors share more insights on how current setups documented in the paper can be generalized to larger Code LMs (e.g., 10B+)?

Good question. While we are unable to verify the exact effectiveness of 10B+ code models due to the cost of pretraining them, we believe the data insights from our work can generalize to larger code models as well. Specifically, for larger code LMs, we can make the annotator cover more scenarios since they are able to hold more knowledge internally. For bigger models, it can also be beneficial to train for a longer token horizon.

Thank you for your valuable review and suggestions! We provide our response as follows.

Q1: ...why is Python set as the primary language data, and how can it be adjusted for other languages?

Thank you for the great point. The primary reason we set Python as a focus during enhanced pretraining is to align with prior work, such as StarCoder and CodeLlama, which emphasize specific Python variants. While those approaches dedicate a final fine-tuning stage exclusively to Python, our method balances this by adjusting the Python mix ratio to 50%, while preserving the relative proportions of other languages. This ensures that Python is adequately emphasized without overshadowing other languages. It’s important to note that in the earlier phases of pretraining, we adhered to a natural distribution of languages without prioritizing Python.

Q2: ...performs significantly better than the first one.", could you further explain the reason behind this conclusion?

The primary reason for improved performance when grouping by language before repository is that grouping by repositories can lead to training instances containing mixed file types, such as configuration files and programming files. During training, we align the compute, meaning that the "grouping by repositories" approach processes fewer tokens specifically from programming files. Moreover, because files are randomly ordered, code files from different languages are often unrelated. Consequently, each training example may end up including two completely unrelated files, which can negatively impact learning.

A promising hybrid approach could involve grouping files by language within each repository. This method ensures that training examples can include multiple programming language files while maintaining the cohesion of files in the same language within each group.

We have put these explanations in the revised paper.

Q3: ...what is the difference between alignment and fine-tuning of pre-trained models?...

We would like to clarify that the term "alignment" in this context specifically refers to "distribution matching" and is distinct from the use of the term "alignment" in the context of training paradigms. We have revised the wording in our manuscript accordingly for better clarity.

Thank you for your review and suggestions! We address your questions as follows.

Q1: ...Similar methodologies have been employed, such as using GPT-4 as a data annotator. This raises questions about the uniqueness of the authors' contributions.

We agree that there are many approaches relying on GPT-4 for generating instruction tuning data. However, using GPT-4 for annotating the huge pretraining corpora is cost-prohibitive and does not scale effectively. Instead, our method leverages a BERT-based annotator, which is significantly more efficient for both filtering and annotation purposes.

The key innovation lies in combining data filtering and data synthesis—a novel approach that we first proposed. Additionally, we provide actionable insights, such as optimal re-warmup strategies during continued pretraining, which align with Reviewer 4fJs’s feedback.

Q2: ...The evaluation would benefit from the inclusion of additional baselines, such as OpenAI's GPT models...

Thanks for your suggestion. We did not include GPT models at the first place primarily because SnowCoder focuses on small code models that are open-source. Given the close nature of OpenAI models, it is difficult to draw conclusions from comparisons with them. The primary goal of SnowCoder is to provide insights and offer practical guidelines for researchers working on pretraining, and to benefit future pretraining work. Here are the additional evaluation results for OpenAI's GPT series and Anthropic's Claude series:

We are happy to include them in the final revision if the reviewer believes they are important.

Q3: Lower than Phi-1.5-1.3B on HumanEval+ MBPP+...

Great question! It’s important to note that HumanEval+ comprises just 164 self-contained Python problems, making it a relatively narrow benchmark for evaluating general-purpose code generation. In contrast, BigCodeBench is far more comprehensive and practical. It challenges LLMs to tackle 1,140 fine-grained tasks across 139 libraries and 7 domains, requiring the ability to invoke multiple function calls as tools.

Moreover, Phi-1.5-1.3B uses GPT-3.5 and GPT-4 to generate over 30 billion textbook-like tokens —this is a significantly more expensive approach than SnowCoder. At current pricing for GPT4-o models at $2.5 per 1M tokens, this will take $75,000 to generate one time; usually when experimenting, one can easily expect to pay 2x that or more. This is a significant expense. This dataset is heavily skewed toward HumanEval-like problems, which may explain its advantage on such benchmarks. In comparison, Arctic-SnowCoder-1.3B’s specialized pretraining on high-quality code tokens prioritizes real-world applicability over narrowly optimized generalization to benchmarks like HumanEval+.

Thank you for your valuable review and suggestions! We put our responses to your questions as follows.

Q1: ...how the quality annotator specifically improves the model’s learning? What are the key features or patterns it identifies that contribute most to the performance boost?

Good questions. Please kindly refer to Table 4, where we studied different data combinations used to train the quality annotator. In Figure 2, we showed that the downstream performance on a given benchmark directly correlates with the ROC-AUC score of the quality annotator on that benchmark. A higher ROC-AUC score indicates that the annotator can rank the corresponding downstream distribution higher. And this is how we conclude that the key to high-quality data is its alignment with the distribution of downstream applications.

Q2: How does the synthetic data generated by Llama-3.1-70B differ in quality or characteristics from the high-quality tokens...

Great point. We leveraged Llama-3.1-70B to rephrase a lower-quality code file into a higher-quality variant, where one of the key instructions is to let the model make sure the code file can improve a developer's ability in terms of language comprehension, reasoning, algorithms, and mathematics. Due to the text length limit in rebuttal, we have put several concrete synthetic data examples in Appendix A in our revised paper. From the examples, we can observe that a notable pattern of synthetic data is that it contains more documentation, natural language explanations, and comprehensive functionality. These features make the data more informative and boost the effectiveness of model learning.

Q3: Your results suggest that grouping by language before repository improves performance. Could you elaborate on why this approach works better?...

The primary reason for improved performance when grouping by language before repository is that grouping by repositories can lead to training instances containing mixed file types, such as configuration files and programming files. During training, we align the compute, meaning that the "grouping by repositories" approach processes fewer tokens specifically from programming files. Moreover, because files are randomly ordered, code files from different languages are often unrelated. Consequently, each training example may end up including two completely unrelated files, which can negatively impact learning.

A promising hybrid approach could involve grouping files by language within each repository. This method ensures that training examples can include multiple programming language files while maintaining the cohesion of files in the same language within each group.

We have put these explanations in our revised paper.

Q4: ...how do you envision scaling this approach for larger models or different domains...?

We believe that further scaling up the training tokens can lead to significant performance improvements, as evidenced by the trends observed during the continued pretraining and enhanced pretraining phases. However, our experiments indicate the importance of aligning the training distribution with downstream distributions. Misalignment in data distribution poses a critical challenge during the pretraining of SnowCoder, as it often results in diminishing returns. Specifically, while the validation loss continues to decrease, downstream performance plateaus or fails to improve.

Q5: ..How do you ensure these benchmarks reflect real-world programming challenges...

Great point. We choose HumanEval+ due to its wide adoption and its robustness compared to HumanEval. While HumanEval+ primarily focuses on self-contained coding tasks with just 164 problems, BigCodeBench is a comprehensive coding benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. Also, the selected benchmarks are well suited for evaluating pretraining models, while most other real-world programming challenges will require instruction tuning, which is not the primary focus of this paper. We believe the combination of the two benchmarks can cover most practical coding scenarios.