Textbooks Are All You Need

We thank the reviewer for their time and effort in reviewing our paper and highlighting our models impressive performance. Below is our response for addressing some of the concerns raised in the review

Q. The authors make a concerted effort to prove the "synthetic exercise" dataset curated in their work does not contain contaminated data from the test sets - however, I could not find a similar investigation for the "synthetic textbook" dataset.

We were able to do a thorough decontamination analysis for HumanEval on our CodeExercises dataset as both datasets were of the same type (short python functions) and are self-contained code snippets – this meant that we can devise robust versions of similarities beyond string matching. We note that many decontamination procedures in the literature (including The Stack dataset) resort to string-matching or min-hash matching against a test set which we show is not very robust (e.g., our analysis picked up similarities that N-gram analysis did not).

On the other hand, our synthetic textbook is a fundamentally different NLP-heavy dataset. The code snippets in the text are often very short and used mostly for illustrative purposes: e.g, exercises or usage specific functions or structures like print and if-then, class definition etc. These are not directly comparable to HumanEval questions which have specific function level algorithmic style. Thus, we did not find this dataset suitable for in-depth ast-style analysis.

Q. Another significant weakness is that finetuning results for other models (e.g. StarCoder) are missing - these results can answer the question, "Are textbooks all you need or just the exercises?"

We emphasize that the benefit of “textbook” approach not only lies in the final performance but also in overall training compute. Compared to our model, all other baselines are pretrained with orders of magnitude larger compute. StarCode is 10x larger – and even finetuning on it is expensive computationally. It is possible that CodeExercise finetuning will boost the performance of StarCoder and other models too, but that does not diminish the importance of “textbook” quality data for pretraining. We also highlight from our paper that even without any finetuning, our phi-1-base model trained on CodeTextbook dataset achieves 29% HumanEval performance with a mere 1.3B parameter model. The previous smallest model that achieves close to 30% performance on HumanEval was Replit-Finetuned at 2.7B parameters, which was trained with 100 times more training tokens than us Replit (2023).

We thank the reviewer for their time in reviewing our work and highlighting the strengths of our work on methodology and empirical findings. Please find the answers to your clarifications below.

Q. The authors are motivated to include high-quality data (i.e., textbook). However, they also use the data generated by gpt-3.5, which may have a lot noises. How to justify the motivation because it seems contradictory to have both high-quality and low-quality data.

While GPT-3.5 is far from perfect and has noises, the comparison point here are random code/text snippets from web which even after filtering, which typically have much higher levels of non-educational content. We find that with the right prompting, especially for simple coding problems the quality of GPT-3.5 generated data has far more educational value compared to a typical random text from Github repos (which form the large part of web data) or StackOverflow. The harder issue to tackle with using Synthetic data is that the generated text might not have sufficient coverage of rarer and more complex use-cases. This is why we use the mix of web-filtered data with synthetic data targeted to address specific shortcomings we observe in the webdata. We also note that our prompts were tailored to the generating model (GPT3.5)’s known strengths and weaknesses. Taking the example of synthetic textbooks, these were designed to cover the entirety of beginner programming topics, but at the same time, the topics selected were fine-grained enough that GPT-3.5 can reliably generate high accuracy text. We ensured diversity through variations in format and target audience. Of course, if we use a more powerful GPT-4 model for our generations the quality of models will very likely improve further, but generating data at our scale using GPT-4 is significantly more expensive and slower (due to access limits).

Q. In the caption of Figure 1, it is not light / dark green. maybe light / dark blue? Thank you for pointing this out, this was a typo (we forgot to edit text after changing the figure) and we will fix it.

Q. In section 2, the authors are motivated by: We conjecture that language models would benefit from a training set that has the same qualities as a good “textbook”: it should be clear, self-contained, instructive, and balanced. It may be too intuitive without any justification. What about other high-quality data in addition to textbook? For example, legal documents, government document?

We certainly considered official sources of coding textbook data, there are two issues we encountered: (a) much of such high-quality material including real textbooks are copyrighted material, (b) the size of the datasets we managed to obtain while looking for such sources was very small (<<100MB) that it did not end up making much of a difference.

Q. In Section 2.1, the authors rely on the GPT-4 as an annotators to filter the data. Why can we rely on it? I am aware of some studies that using GPT-4 as an annotator. However, the ability of GPT-4 on this specific task is unknown. Justifications are needed. For example, you may want to show the comparison between the data that GPT-4 thinks high-quality v.s. low-quality.

Indeed this is an imprecise measure. We rely on it only after many iterations of manual quality checks and prompt refinements – we had small batches of data scored with GPT-4 and human labelers and look at correlations and iterate on the prompts based on feedback. We provide a typical example of high and low educational value samples from The Stack in Page 4.

We thank the reviewer for their time in reviewing our work and highlighting the impressiveness of our model. Please find the answers to your questions below.

Q. Can you please briefly introduce the training data and method used by CodeGen-Mono, Replit and StarCoder? Will be helpful in understanding how Phi-1 is different and perhaps will further highlight the important of data quality.

Thank you for raising the point. For Codegen-Mono we don’t know the exact dataset that was used in the training, but their paper states that they use combination of Python source code data from Github and NLP data from The Pile. Both these datasets are large corpuses of web data without additional filtering. StartCoder and Replit were trained on the deduplicated subset of The Stack (which in turn has github source code cleaned of PII and permissive licenses among other things). Both these models also use multiple code languages in pretraining. These models were trained on a total of 1T and 500B tokens, respectively.

In comparison Phi-1, (a) we start with only the python subset of The Stack and stackoverflow, (b) but uses filtering to reduce the size to 20-25% of the original datasets (6B unique tokens), and (c) strategically add targetted synthetic data in pretraining and finetuning. We train on 7B unique tokens for slightly less than 8 epochs. Of the choices we made, the first choice of using python-only was not strategical and made for restricting the scope so we can do more controlled experiments. The other two choices were the new contributions in this work.

Q. As mentioned in Sec 3.2, fine-tuning on coding exercises makes unrelated tasks easier to distill from pretraining. If I'm understanding it correctly, during pretraining, languages other than Python are used. Is it possible to further support this claim by evaluating on non-Python coding tasks?

Unfortunately, our pretraining data had only python code snippets, but this would be a good experiment to expand on our model for multilingual code. This could certainly help distil information from other languages, although one consideration experimentally is that if finetuning is fully in python the model might be overfit to the python syntax. To avoid this, the right approach might be to mix in python CodeExercises with small amounts of filtered dataset from other languages – this way we can see the benefits of training with python-CodeExercises on other languages without being confounded by the syntax overfitting.

Q. It was mentioned that Phi-1 has emergent abilities that were not observed in Phi-1-base. In Sec 3, examples were used to support this claim. Is it possible to evaluate this more systematically and qualitatively?

The main surprising emergent ability we found was that the model got better at API usage of popular apis even if the finetuning dataset did not have those API’s. Unfortunately, there are no standard benchmarks we could find for evaluating API usage, so we show more qualitative examples of the correct usage.

Q. Would be helpful to include GPT-4 performance on the 50 new unconventional coding problems.

Thank you for suggesting it. We will add this number. Below is the updated Table.

We thank the reviewer for highlighting the strengths of our work on demonstrating the benefits of quality and the performance of the resulting model.

Q. However, there are significant gaps regarding the creation of the training data. … assessing the paper as a standalone piece makes it challenging to be certain that the authors' methodology doesn't introduce any data leaks potentially favoring their results.

We acknowledge this as a limitation of our work for academic publication. We have added as much detail in the paper as we thought were appropriate to balance between proprietary interests and openness to research community. We additionally release our model artifact for further research, exploration, and independent evaluation. While this is not ideal or 100% transparent, we nevertheless believe that our paper captures important aspects of LLM training and is important for the research community. As noted in the paper it is also not unprecedented in our research community where highly influential papers have not disclosed dataset details (e.g. Minerva which also attribute their main contribution to be data).

Q. The approach taken to assess leakage against the finetuning data is somewhat nebulous. In their n-gram overlap analysis, they offer no rationale for their choice of 13-grams to gauge overlap, quickly concluding that “n-grams are not refined enough to find similar code snippets”. Subsequently, they switch to embeddings, commenting, “We observe that the embedding distance effectively captures code pairs,” without substantiating this claim. and What verification, if any, was performed to rule out that the synthetic pre-training data generated is not highly similar to the existing samples in HumanEval?

Many standard datasets in literature perform decontamination by using exact substring matches or N-grams matches to remove test set from training set: exact substring (extreme version of N-gram with N=len(human-eval-question)) was used in The Stack dataset which is the defacto standard dataset for code models; one of the largest NLP datasets DOLMA (https://blog.allenai.org/dolma-3-trillion-tokens-open-llm-corpus-9a0ff4b8da64) uses 13-gram like ours. In our 13-gram detection we did not detect any true-positive contamination in over 100k code snippets of CodeExercises (we had 4 false positives and all else were negative). Given the widespread use of N-grams for decontamination, we do not consider our conclusion unfounded.

That said since we were ourselves not satisfied with this common procedure which is why we go beyond this standard practice and use other techniques for determining code-similarity, we show how code-snippets that are similar in principle are not detected by N-gram but are captured by our metrics.

Q. Although they explore leakage in the finetuning data, the same scrutiny isn't applied to their synthetic data.

We were able to do a thorough decontamination analysis for HumanEval on our CodeExercises dataset as both datasets were of the same type (short python functions) and are self-contained code snippets – this meant that we can devise robust versions of similarities beyond string matching. We note that many decontamination procedures in the literature (including The Stack dataset) resort to string-matching or min-hash matching against a test set which we show is not very robust (e.g., our analysis picked up similarities that N-gram analysis did not).

On the other hand, our synthetic textbook is a fundamentally different NLP-heavy dataset. The code snippets in the text are often very short and used mostly for illustrative purposes: e.g, exercises or usage specific functions or structures like print and if-then, class definition etc. These are not directly comparable to HumanEval questions which have specific function level algorithmic style. Thus, we did not find this dataset suitable for in-depth ast-style analysis.

We thank the reviewer for their time and effort in reviewing our paper and highlighting the strengths of computational benefits and decontamination analysis. Now we address the main concerns raised in the review.

Q. The paper has low novelty, and the importance of data quality in LLMs is well known. & Q. What are the unique contributions compared to prior work on tailored training data and prompted training?

On this point, we respectfully disagree with the reviewer about the lack of novelty in our paper. While the philosophy of good quality training data being good for learning is not new, the predominant trend in LLMs has been on scaling-up data and model sizes, as evidenced by numerous papers on scaling laws. To our knowledge, our paper is among the first to provide strong and compelling evidence on the effectiveness of high-quality data to achieve state-of-the-art performance in LLMs. The novelty in our work also lies in specific approaches we took to generate and use "textbook quality" data. Our paper contributes uniquely in several ways:

• Targeted synthetic data: We show the effective usage of targeted synthetic data to address specific gaps in existing data sources: e.g. our synthetic textbooks provide useful signals for the model by enhancing signal from natural language text that explains simple algorithms in detail.
• Filtering methodology: We used a combination of GPT-4 (frontier LLM), a small LM classifier, and human-in-the-loop for textbook-quality data filtering. Such methodologies incorporating frontier LLMs strategically in the loop are in the early stages of experimentation and our paper contributes to it. We believe identifying best practices in this area is significant for future models.
• Overall, our model achieves top-tier performance on algorithmic code generation (next only to latest GPT models) despite being 10x-100x smaller in size compared to the other top models. This provides strong evidence that selecting high quality data in creative ways is key to unlocking efficient learning.

Q. The paper is only evaluated mainly on short Python functions. How would it perform on more complex, real-world coding tasks.

We intentionally limited our model's scope (at the beginning of the project) to Python programming on algorithmic tasks. This choice was guided by our hypothesis that high-quality data is more critical to learn reasoning and planning-based skills, which are arguably among the hardest to learn from data scaling alone. Nevertheless, we discuss qualitative examples in the paper show that despite our narrow focus, our model was able to perform well on broader tasks of using popular APIs and understanding general natural language formats. We also discuss the limitations of the models in the appendix. Note that we use qualitative evaluations in these sections as there are no standard benchmarks covering broad use-cases for code generation.

Q. How was the quality of the synthetic textbook data evaluated? What measures were used to ensure it is high quality and diverse?

For our synthetic datasets, we primarily relied on manual evaluation to assess quality and diversity. We used the performance of the resulting (small) model when trained on representative examples of generations as quantitative validation for our manual quality checks. We note that the evaluation of quality and diversity is subjective and depends on the specific task at hand; as such, there are no good metrics to gauge utility for LLMs. We considered these difficulties and the limitations of manual evaluation when crafting prompts for the generation process. The prompts were tailored to the generating-model's (GPT3.5) known strengths and weaknesses. Taking the example of synthetic textbooks, these were designed to cover the entirety of beginner programming topics, but at the same time, the topics selected were fine-grained enough that GPT-3.5 can reliably generate high accuracy text. We ensured diversity through variations in format and target audience.

Q. How do the results of synthetic data, and using textbooks generalize to other domains beyond code? Does this approach advance the scientific understanding of training language models for code generation?

Our paper focuses on code generation as a proof of concept to demonstrate the benefits of using textbook-quality data in training LLMs. This serves as a compelling standalone use-case and a representative example for other tasks that require reasoning-based skills. We are aware of similar follow up attempts in more general language understanding and reasoning tasks.

We thank the reviewer for their time reviewing the paper and highlighting our strengths in efficiency, advantages to the research community, and clarity. Below we address the questions raised in the review.

Q. Key details on the data generation process are not shared for "proprietory reasons", yet this is central to the paper, its proposition and its results.

We acknowledge this limitation of our work for academic publication. We have added as much detail in the paper as we thought were appropriate to balance between proprietary interests and openness to research community. We additionally release our model artifact for further research, exploration, and independent evaluation. While this is not ideal, we nevertheless believe that our paper captures important aspects of LLM training and is important for the research community. As noted in the paper it is also not unprecedented in our research community where highly influential papers have not disclosed dataset details (e.g. Minerva which also attribute their main contribution to be data, and GPT-3 are notable examples).

Q. The lack of a broader impacts section weakens this paper.

We have discussed some of the practical implications of the work in the Conclusions section. We will additionally discuss the environmental benefits of training small models and the opportunities for mechanistic understanding experiments. We will also clarify (based on Reviewer QdXo’s comments) that some of the hidden cost include the amortized cost of the building GPT3.5, GPT4 in the first place that can be used in turn for targeted generation. If there are other issues/benefits that the reviewer would like us to address, we are happy to add them to the paper.

Q. Would you argue that the structure and compositionality of programming code is critical for achieving such high levels of performance from small models and datasets....

The composability and structure of code generation was certainly helpful for us in effectively selecting high quality data through filtering and generation. For example, it is intuitively easier to define specifications for filtering existing web-datasets for good quality algorithmic code compared to more imprecise and subjective skills such as common-sense reasoning. We can build our textbooks with sufficient coverage based on what we consider beginner level syllabus for programming and ensure coverage.

There is nothing special about python itself, so we don’t have a reason to believe why the same methodology will not extend to other languages and structured tasks. Even for unstructured and subjective tasks like common-sense reasoning, while it might be harder to filter and generate for what would constitute a “textbook” quality data, it is not impossible and can be beneficial to think of creative ways to increase the signal for reasoning skills in the training corpus.

Q. Do you think that the code exercises factor into improving performance by encouraging the model to self-critic whatever it generates, given that it seems to append exercises to generations?

This is a possible hypothesis we have not tested yet but can be studied as part of mechanistic understanding of how models learn to code using our model as a base model.

Q. Where do you think this approach is most likely to fail? With what kinds of tasks/data?

We discuss the limitations of our model in the appendix and in the conclusions. One common pattern of limitations we see is that the model as-is does poorly on tasks that are easier with some form of memorization. For example, in terms of api usages, since our model is not trained on all of python code available on the web, it has little to no knowledge of less known libraries. We believe this can be remedied in the future by appropriate continual training where we can add additional knowledge incrementally. Additionally, as noted earlier, there are some skills (e.g., common sense reasoning) where filtering or generating "textbook" quality data is more subjective and will require creative new ideas.

Q. Does phi-1 have ICL abilities? And are there other emergent capabilities that can be for seen and easily tested?

Since we don’t have concrete benchmarks for all types of coding abilities, our explorations in this space are more qualitative – we share some of the findings on emergent abilities in Section 3 and in Appendix.