Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models

We thank the reviewer for the helpful feedback!

Long-context modeling. To clarify, we did not say that our work focuses on long-context modeling. In line 79, we describe our setup as in line with “long token horizon training”, which is different from long-context modeling. Long token horizon training (see [1]) involves a relatively large token budget (compared to the number of high-quality data available for training) and thus necessitates making multiple passes over the training set. We simulate this setup by fixing the total number of tokens curated with different baselines and training on the resulting datasets for 2-4 epochs.

Include per-task results. We have included an analysis of per-task performance difference at the 3B model scale in Appendix D.2 Figure 6. At this scale, mixing raw and synthetic tokens significantly improves over using only the former on MMLU, ARC Easy, ARC Challenge, SQUAD, COQA, BoolQ and some BigBench tasks (Repeat copy logic + language identification).

Are the performance improvement consistent? We have generated more synthetic tokens with REWIRE and obtained additional results at the 7B model scale. To summarize, the same observations hold: (i) compared to training on only the highest-quality raw data, mixing it with our high-quality rewritten texts leads to 12% improvement on MMLU and 2.5% improvement on average across 22 tasks of DCLM’s CORE, (ii) training on the raw & synthetic mix is also more effective than having access to 2X more web data. Refer to the Global Response for more details.

Figure 1 is unclear. We will try to improve the visualization in the next iteration of the paper.

[1] Su, Dan, et al. "Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset." arXiv preprint arXiv:2412.02595 (2024).

Thank you for the response. I would like keep my scores the same.

Thank you for your time and for the review!

Comparison to prior work. To clarify, our contribution lies in providing a general-purpose method to make better use of low-quality web documents that would normally be discarded by SOTA filters. This is especially pertinent in the context of internet data soon being relatively much less available compared to compute resources. Given this motivation, we view the papers you suggested as related but not “highly similar” to our work:

We have discussed "Synthetic Continued Pre-training" [1] in Related Work. [1] synthesizes new knowledge from books, which is more in line with the technical data synthesis direction (e.g. Phi [2]). In contrast, our work doesn’t particularly focus on any type of content. Besides, in [1] the data generation was done at a much smaller scale (for continual training setting) - the authors generated a total of 455M tokens, which is just 6% of the pre-training set size in our smallest experiment (and >30x less than what we generated). We will clarify these differences further in the Related Work discussion.

For ProX [3], we were not aware of this work but will add the citation. ProX’s data refinement approach was applied to existing high-quality documents (e.g. DCLM-baseline). We do not find the refinement operations specified in Table 1 of [3], such as removing noisy lines, string normalization, etc., to be applicable or helpful for improving lower-quality documents (i.e. the emphasis of our work). Furthermore, the refinement procedure happens through a set of predetermined, rule-based function calls, while in our pipeline, the enhancement can be more fluid based on the LLM’s reasoning process.

Given prior work that prompts LLMs to rewrite data, this limits the novelty of the work. While the high-level idea of rewriting is not new, to the best of our knowledge, our work is the first to (i) generate synthetic data at a large scale for pre-training, allowing us to mix synthetic tokens with web-scraped tokens with 1:1 ratio, (ii) maintain the quality and diversity of the synthetic texts at such a large scale, such that adding in as much synthetic data as web data during pre-training does not harm but rather improves performance of the base model. The latter is non-trivial.

Analysis of number of documents with knowledge gaps and rewriting it fills in the knowledge gap. We have provided analysis in the same vein in Figure 3 of our paper. If we consider Wikipedia-style rephrasing as paraphrasing and not modifying the knowledge gap, we find that our pipeline goes beyond rephrasing and changes the knowledge representation in the majority of the raw documents (see the green curve versus the yellow curve).

Lacking analysis on structural diversity. We posit that the structure of the generated texts still largely follows the structure of the original documents, e.g. if the web-scraped document contains a poem, then the LLM would deduce that the task is to write a poem and output that correspondingly. We do not have a compute-friendly way to measure the structural diversity of the synthetic texts (besides asking an LLM to annotate all documents), or to assess how structural diversity affects model generalization (besides trying different ways to sample documents using the LLM annotations and pre-training on different data variations from scratch). These questions could be the scope of another paper altogether, see [4] as an example.

Cost of data generation/ Does the compute cost justify the performance gain? We have stated the compute costs used for data generation in Appendix B. In the Global Response posted above, we also highlight our new results at 7B scale with larger performance gains: mixing carefully filtered raw texts and rewritten texts leads to 12% improvement on MMLU and 2.5% improvement on average across 22 tasks, compared to training on only the former. This shows that the performance improvement from our method increases with scale. We emphasize that this makes the method especially useful in the current LLM scaling landscape, where AI hardware is becoming more efficient and compute is getting cheaper, while the stock of naturally existing internet data is slowly being exhausted. Consequently, it is worth spending more compute on creating new datasets for pre-training.

Modifications to Figure 1. Thanks for the suggestions, we will consider these changes in the next iteration of the paper.

[1] Yang, Zitong, et al. "Synthetic continued pretraining." arXiv preprint arXiv:2409.07431 (2024).

[2] Li, Yuanzhi, et al. "Textbooks are all you need ii: phi-1.5 technical report." arXiv preprint arXiv:2309.05463 (2023).

[3] Zhou, Fan, et al. "Programming every example: Lifting pre-training data quality like experts at scale." arXiv preprint arXiv:2409.17115 (2024).

[4] Wettig, Alexander, et al. "Organize the Web: Constructing Domains Enhances Pre-Training Data Curation." arXiv preprint arXiv:2502.10341 (2025).

We thank the reviewer for their constructive feedback!

Statistics on the increase in useful training documents after filtering. For filtering after rewriting, we only retain the top 10% of the highest scoring documents. The exact number of pretraining synthetic tokens that come from rewriting can be found under the “Dataset size” column in Table 1. We have provided further analysis of the increase in useful training documents in Section 4.1 of the paper. To summarize, about 81.7% of the new synthetic tokens used for pretraining come from recycling low-quality web texts that are not selected by DCLM-baseline.

The role of meta-cognitive abilities. We appreciate that the reviewer recognizes that our rewriting prompt leverages LLM’s meta-cognitive abilities, which uses chain-of-thought reasoning to analyze the original documents in terms of facts, goals, etc. We experimented with several versions of the rewriting prompts at smaller data and model scales, and found that the prompts that leverage meta-cognitive abilities lead to better pre-training data (measured by downstream performance). Besides, we have checked the effectiveness of the prompt via manual inspection (see Appendix for document examples), as well as quantitative analysis. Specifically, through the semantic similarity analysis (Figure 3), we demonstrate that our pipeline goes beyond simple fixes (e.g. paraphrasing or stylistic changes) to improving the semantics of the original texts as a whole. For the rebuttal period, we have also run additional factuality evaluations, and found that mixing in the rewritten data improves truthfulness and knowledge capabilities of the resulting model. Refer to the Global Response for more details.

Missing citation. This work came out in May after our COLM submission, we will add the citation in the next version of the paper.

Thank you for the reply. I would still like to keep the same score.

Thank you for the valuable suggestions!

Why train a new fastText classifier? The DCLM fastText classifier was trained on the natural text distribution (scraped from Common Crawl) and thus its discriminative/ filtering capability is hampered when deployed on our synthetic text distribution (i.e. texts that are considered out-of-distribution). As a result, we have to retrain the fastText classifier. We follow the DCLM classifier’s training recipe by using the same datasets (OpenHermes 2.5 + ELI5) for positive examples, but replacing the negative examples to be i.i.d. samples from our synthetic data pool before any filtering. This allows us to stay as close to the DCLM setup as possible, isolating the data distribution (raw/ synthetic/ mixed) as the main factor influencing downstream performance.

Choice of training dataset for classifier. We assume that by "one being synthetic and the other being social media", you are referring to the positive examples used to train the fastText classifier. We agree that in general, improvements in filter design can improve the quality of the synthetic data further. Our choice of positive examples simply follows the DCLM paper as we want to keep changes in the filtering pipeline to a minimum. We note that the DCLM paper did experiment with various choices of positive examples (see Table 5 of the paper), including Wikipedia, and found that using OH-2.5 + ELI5 led to the best filtered datasets. Therefore, we adopt the same design choice.

Analysis of hallucination. It is true that there are cases where the rewriting pipeline elaborates on the existing content and adds in details (which themselves could be new facts). We find that defining a knowledge base against which hallucination is measured is not tractable in our setup. Furthermore, a lot of existing hallucination detection benchmarks use LLMs as part of the pipeline, models that are themselves susceptible to hallucination. As a proxy to measure the impact of including rewritten content on factuality, here we compare the performance of training with and without synthetic texts, on TruthfulQA and on the World Knowledge subset of DCLM Extended tasks (Jeopardy, MMLU, BigBench QA Wikidata, BigBench Misconceptions, ARC Easy, ARC Challenge, TriviaQA). We find that adding in rewritten data doesn’t harm, but rather improves truthfulness and knowledge capabilities of the resulting model.