To Code or Not To Code? Exploring Impact of Code in Pre-training

We would like to thank R YwMY for their very positive review of our work, highlighting the novelty and comprehensiveness of the study. We greatly appreciate their claim that our work is “first-of-its-kind, comprehensive study on how code data in pre-training affects LLMs” and noting that our work “provides actionable guidance for future LLM development”. We address each individual concern below:

> Limited Details on High-Quality Code Data: A significant portion of the paper’s conclusions, particularly in Section 3.4, emphasize the positive impact of incorporating high-quality code data on LLM performance across different task categories. However, the description of the high-quality code data is limited to that it is proprietary and synthetically generated. The lack of transparency about the dataset's characteristics, such as the source, data format, or complexity raises questions about what specifically constitutes "high quality" and how these qualities directly influence performance. More details on criteria for data quality would strengthen the paper and enable the research community to apply similar quality standards in future work.

We thank R YwMY for suggesting furnishing more details about what specifically constitutes “high quality” code data. Our primary focus in the given work is to systematically study the impact of code data on downstream tasks across (1) the impact of code across different stages of pre-training, (2) the specific properties of code that contribute to improvement, and (3) how these effects scale. Our synthetic code data was created using problem statements which were used to create Python solutions that are formally verified. The main criteria to create good synthetic code data is to use a performant teacher model to generate the code data and couple with formal verification tests to ensure generation is high quality and correct. This is similar to recent work that has shown the importance of teachers [1] and formal verification [2].

[1] Odumakinde, Ayomide, et al. "Multilingual arbitrage: Optimizing data pools to accelerate multilingual progress." arXiv preprint arXiv:2408.14960 (2024).
[2] Xin, Huajian et al. “DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search.” ArXiv abs/2408.08152 (2024).

> Potential Bias in Generative Ability Evaluation: The evaluation of generative ability relies on the LLM-as-a-Judge approach, specifically through win-rate scores on the Dolly-200-English dataset, which is designed to reflect open-ended, non-code tasks. The authors attribute improvements in generative quality to the inclusion of code data in pre-training, arguing that code data enriches the model’s ability to generate preferred outputs. However, using LLMs to evaluate other LLM outputs has known limitations[1], including potential biases toward verbosity, structured formatting, and positional advantage. Moreover, the inclusion of markup-style code may enhance readability and formatting, making responses more appealing to the LLM evaluator. To mitigate ambiguity, qualitative examples of output differences would be helpful in illustrating how the inclusion of code data contributes to generative improvements beyond format and readability alone.

We thank R YwMY for their thoughtful comments about the potential biases in LLM-as-a-judge evaluations. We acknowledge the well-documented limitations of using LLMs to evaluate other LLM outputs, including potential biases toward verbosity, formatting, and positional advantages. In fact, these known limitations were precisely why we included LLM-as-a-judge evaluation as one of multiple evaluation axes in our study. By incorporating this method alongside other rigorous benchmarks, we aimed to provide a holistic and nuanced picture of model performance. We take R YwMY’s suggestion to include qualitative examples of output differences in illustrating how there are improvements beyond format and readability.

> Could the authors clarify whether the Balanced LM (50% text and 50% code) incorporates markup-style data in its training mix? The paper specifies that the Code-initialized model includes 80% code data and 20% markup-style data, but it remains unclear whether this type of data is also included in the Balanced LM.

We thank R YwMY for their question. The Balanced LM’s code data includes markup style data in the same proportion as the code-initialized model, i.e. the 50% code data is made up of 80% code and 20% markup style data (therefore, 40% code and 10% markup data in full mixture). We will add this clarification to the manuscript.

> Minor Issues

We thank R YwMY for these suggestions, and acknowledge these issues. These will be updated in the final manuscript.

> The pre-training models used in this study are relatively small (470M and 2.8B parameters), which limits the generalizability of the findings. Larger models may be necessary to substantiate conclusions regarding the effects of code data on NLP pre-training.

Thanks for your comments. However, our choice of 470M and 2.8B parameter models was deliberate given the comprehensive nature of our study. To conduct thorough ablations and ensure reliable conclusions, we pre-trained 64 models in total where each pre-training run for 200B tokens takes 4,736 TPU-chip hours for 470M and 13,824 TPU-chip-hours for 2.8B parameter models and each cooldown run for 40B tokens takes 1,024 TPU-chip hours for 470M models - a significant computational undertaking even at these scales. Pre-training this many variants at larger scales would be prohibitively expensive.

While we agree that exploring larger architectures would provide further confirmation, as we experimented with two model scales (470M and 2.8B) and found the same trends hold for both scales, we believe that our findings transfer to larger models. Our findings offer valuable and actionable insights about the role of code in pre-training data mixtures.

> Have the authors considered evaluating the pre-trained models on additional natural language tasks to better understand the generalization effects of code data?

We thank R ujyE for their suggestion. We evaluated our models on 14 natural language datasets where these datasets cover a diverse set of task types for reasoning and world knowledge such as question answering, natural language inference, sentence completion, and coreference resolution. Additionally, we included an open-ended generation task to validate our conclusions beyond academic benchmarks. Finally, we included 2 code datasets as given in Table 1 (Appendix). This is a very comprehensive list in terms of tasks types, datasets, domain to enable consistent evaluation signal at our chosen model scales, allowing us to draw reliable conclusions about the effects of code data.

Additionally, thanks to R PKei suggestions for exploring model performance on mathematical tasks, we take the additional time during this rebuttal period to evaluate the cooldown 470M models on GSM8K. The results show that including code in any stage of the pre-training improves performance compared to the model where no code has been seen pre-training, continual pre-training, and cooldown. The most performant model in this comparison has seen the code data in all stages including cooldown where it leads a significant improvement (from 2.9 to 4.12, +42% relative gain).

Finally, it is important to note that, we carefully picked these benchmarks by filtering out additional benchmarks where any experimented model performance was below random. Therefore, we made sure that all the experimental models’ performance achieves non-trivial performance in selected benchmarks.

We would like to thank R ujyE for the time and effort for their feedback and for highlighting the insights provided and the multiple experimental configurations of our work. We address each individual concern below:

> The findings are somewhat unsurprising, as incorporating diverse data sources (like code) in pre-training is known to improve model performance—a principle demonstrated by previous studies, including PaLM (Chowdhery et al., 2022), Gopher (Rae et al., 2022), and Bloom (Workshop et al., 2023). This paper extends these findings by analyzing code quality and proportion but does not significantly depart from established conclusions, such as the importance of data quality in pre-training (https://arxiv.org/abs/2207.05579).

While previous works [1][2][3] have noted correlations between code pre-training and improved performance, none have conducted systematic ablations to understand the mechanisms and extent of this relationship. For example, in PaLM, they only noted the use of code data from Github together with its percentage in their data mixture without any systematic analysis and ablation. This is similar to Gopher and Bloom papers. Our work is a first-of-its-kind which makes distinct contributions by systematically investigating how and when code data improves model performance.

Our study is the first to comprehensively examine (1) the impact of code across different stages of pre-training such as pre-training from scratch, continued pre-training and cooldown, (2) the specific properties of code that contribute to improvement such as use of code adjacent data, or high-quality synthetic data, and (3) how these effects scale. Rather than simply confirming that code helps, we provide detailed insights into when and how it helps, including comprehensively quantifying its impact on specific capabilities like natural language reasoning and world knowledge. This systematic study across multiple different model development axes and provides insights for model development at the pre-training stage.

[1] Chowdhery, Aakanksha, et al. "Palm: Scaling language modeling with pathways." Journal of Machine Learning Research 24.240 (2023): 1-113.
[2] Rae, Jack W., et al. "Scaling language models: Methods, analysis & insights from training gopher." arXiv preprint arXiv:2112.11446 (2021).
[3] Muennighoff, Niklas, et al. "Scaling data-constrained language models." Advances in Neural Information Processing Systems 36 (2023): 50358-50376.

> Figure 4 suggests that including code data may actually reduce performance on some natural language tasks, which appears contradictory to the paper’s main claim. This inconsistency weakens the overall reliability of the conclusions; [and] Question 1: Given the observed performance drop on certain NL tasks, how does the study reconcile this finding with the broader claim of code data’s benefits for NLP tasks?

We would like to clarify to R ujyE that Figure 4 actually shows the impact of varying percentage of code data in pre-training: as the proportion of code is increased in pre-training while there is indeed a reduction in performance on world knowledge tasks, we consistently observe improvements in natural language reasoning tasks till about >50% of code proportions. This distinction is important and helps illuminate the different ways code data influences model capabilities.

The apparent reduction in world knowledge tasks performance which contain benchmarks TriviaQA and Natural Questions Open. These benchmarks primarily test knowledge retrieval rather than reasoning capabilities. This is not surprising, as code data would not be expected to enhance a model's ability to recall specific facts about world knowledge. As the proportion of code is increased there are no data sources to acquire the required knowledge in the pre-training mix, resulting in degradation of performance. However, considering the trade-off including reasoning tasks, code generation tasks and world knowledge, using the right amount of the code data leads to better average performance overall.

We would also like to point out that when continual-pretraining from a model that has seen code data (Section 3.1, balanced -> text model), we find that there is a relative improvement of 4.1% over a text-only base model even for the world knowledge task. Therefore, use of code data with the right amount and combined with multiple stage of pre-training strategy benefits for NLP tasks.

As the discussion period has only a few days remaining, we wanted to ask R ujyE if there are any follow-up points we can clarify. We have responded to all concerns raised, and also included new results on a mathematical reasoning benchmark (GSM8K) where we show that code data plays a beneficial role as it is used in different stages of pre-training. We are also happy to engage in further discussion. If there are no other points of clarification, we would kindly ask that Reviewer ujyE to consider increasing their score to reflect.

Thank you for your rebuttal. While your response has explained the two questions I raised, I remain unconvinced based on the current answers:

• Given the computational resources available for this study (e.g., 13,824 TPU-chips), I think experimenting with larger models (e.g., 7B) is not prohibited. Even conducting a subset of the main experiments on larger models would provide more robust insights.

• The authors argue that the paper has been evaluated on 14 datasets, but why are the most conventional text classification benchmarks excluded, which are naturally the first choice to evaluate LLM capabilities?

We would like to thank R SkF7 for the time and effort for their positive feedback and for highlighting the importance of the area studied, and the analysis of code quality. We also express gratitude towards them for emphasizing positively that our approach, “addresses a crucial aspect of model performance with implications for both research and practice” and that our approach “provides valuable insights into enhancing model effectiveness”. We address the individual concerns below:

> Limited Technical Contribution: The study primarily consists of empirical analyses without introducing new methodologies or frameworks. While the findings are good, the technical contribution could be perceived as limited due to the absence of novel approaches or theoretical advancements.

Thanks for your comments. While we do not introduce new methodological frameworks, we want to respectfully clarify that our research fills a critical knowledge gap through systematic empirical investigation of the impact of code in LLM pre-training. Despite the widespread practice of including code in pre-training data mixtures for LLMs, there has been limited rigorous analysis of its impact on non-code tasks before our work. While theoretical advancements are valuable, equally important are rigorous empirical studies that challenge assumptions and provide concrete evidence for improving model and data design. Our work falls into this latter category, offering actionable insights about the critical role of code data during pre-training and its effects on downstream non-code tasks. Considering the wide-spread use of LLMs and the cost of pre-training, we believe that our work results in significant and impactful research.

> Limited Scale: The analysis is restricted to models with 470M and 2.8B parameters, which may not capture the performance dynamics across a broader range of model sizes and architectures.

We deliberately chose the model scales of 470M and 2.8B parameters, given the computational demands of our experimental design. We pre-train 64 models in total where each pre-training run for 200B tokens takes 4,736 TPU-chip hours for 470M and 13,824 TPU-chip-hours for 2.8B parameter models and each cooldown run for 40B tokens takes 1,024 TPU-chip hours for 470M models. This is an enormous endeavour given the scale and computational resources required.

While we agree that exploring larger architectures would provide further confirmation, as we experimented with two model scales (470M and 2.8B) and found the trends hold for both scales, we believe that our findings transfer to larger models.

We thank R SkF7 for acknowledging the rebuttal and their comments.

We would like to thank R PKei for their positive feedback and for highlighting the “extensive ablation study” that shows that “adding code data yields substantial benefits across tasks beyond coding”. We also express gratitude towards them for emphasizing positively that our approach, “addresses a crucial, relatively unexplored question”, “offers a fresh perspective on potential benefits of code data” and that “the methodology is rigorous”, “provides significant implications for pre-training dataset design”. We address the individual concerns below:

> For the scaling experiments, some key information could be explored further. For instance, does the trend observed in Figure 4 hold consistently across models of different scales? Additionally, how can the trends identified be extended or generalized to larger models?

Thanks for your comment. For Figure 4, we pre-trained 6 models by varying code percentage before continual pre-training. Given the resource intensive nature of pre-training (requires training 6 larger models for this experiment), we have only conducted this ablation using 470M parameters models (pre-trained for 200B tokens).

However, we believe that a similar trend would hold for the larger models. In the context of continued pre-training experiments (Section 3.2), we pre-train 2.8B parameters models where we see a similar trend with 470M models where the code initialized model (i.e. code -> text) outperforms the balanced initialized model (balanced -> text) in reasoning tasks, and falls slightly behind in world knowledge tasks.

> I am also curious about the model's performance on mathematical tasks, as these, like coding, are highly logical and reasoning-intensive. Including a benchmark evaluation on mathematical tasks could provide valuable insight into whether similar gains extend to other logic-focused domains.

R PKei suggestions around exploring model performance on mathematical or logic-focused tasks are insightful. We take the additional time during this rebuttal period to evaluate the cooldown 470M models on GSM8K. We only evaluate the models that underwent cooldown training as for this model size, only after cooldown, the models achieve nontrivial performance on GSM8K.

As can be seen from the results, including code in any stage of the pre-training improves performance compared to the model where no code has been seen pre-training, continual pre-training and cooldown. The most performant model in this comparison has seen the code data in all stages including cooldown where it leads a significant improvement (from 2.9 to 4.12 - +42% relative gain).

> In terms of writing, certain critical details could be clarified. For instance, it may not be immediately clear in the text which model size (e.g., 470M or 2.8B) is being evaluated in figures such as Figures 4 and 5. Adding such context in the captions or main text would enhance clarity for readers.

We thank R PKei for pointing out improvement required in clarity for certain details. We will update the main text, captions and related text by adding the information of the model size (470M parameters) for Figures 4 and 5 to improve readability and provide clear details.

> In the sentence "Figure 8 shows the results of xxx" in Section 3.2, it seems that "Figure 8" should actually be "Figure 3." May be a typo?

Thanks for finding this typo. Yes, it should be Figure 3, we have corrected this in Section 3.2.

As the discussion period has only a few days remaining, we wanted to ask R PKei if there are any follow-up points we can clarify. As suggested by PKei, we have evaluated our cooldown experimental variants on a mathematical reasoning benchmark (GSM8K), and show that code data plays a beneficial role as it is used in different stages of pre-training. We have responded to all concerns raised, and are also happy to engage in further discussion. If there are no other points of clarification, we would kindly ask that Reviewer PKei consider increasing their score to reflect.