Organize the Web: Constructing Domains Enhances Pre-Training Data Curation

Thank you for your review and for engaging closely with our work! We are glad you found that our methodology is overall sound and that our claims are supported by clear evidence. However, you raised several points with respect to evaluation and novelty, which we hope to address below.

Comparing to greater number of clusters in SemDeDup and Quad

The central premise of our paper revolves around the question of how to partition a monolithic dataset into meaningful subsets for data mixing. Therefore, we consider different ways of structuring the data into an equal number of partitions and apply a fixed domain mixture optimization algorithm (for which we use RegMix). We believe this is a fair experimental setting given our paper’s goals. However, the reviewer correctly points out interesting future extensions, such as studying the effect of domain granularity.

The two mentioned papers, SemDeDup [1] and Quad [2], are fantastic related works, but neither of them perform k-means clustering with the goal of learning a data mixture. Instead, SemDeDup [1] performs clustering to compute “prototypicality” scores and select data points that are least similar to the cluster center — therefore performing sample-level filtering — and the DCLM paper finds that it performs worse than the RefinedWeb baseline (Table 4). Quad [2] employs fine-grained clustering to obtain robust and efficient influence estimates across groups and then employs binary thresholding to select groups of similar data points from a corpus. Their proposed algorithm is designed for fine-grained group selection instead of data mixture optimization and requires careful tuning of the alpha and gamma hyperparameters. We agree with the reviewer that it would be interesting to add Quad to the experiments, but we would not think of it as a baseline for domain mixing, but rather compare it to FineWeb-Edu and DCLM-fasttext and see whether Quad would also benefit from reweighting domains. We hope to add these experiments in the final version of the paper.

DCLM Core is more informative and comprehensive than OLMES

DCLM Core may contain a more comprehensive range of tasks, but in our experiments, we noticed substantial variance with the DCLM Core tasks at the 1B parameter scale (up to 1 point) and many tasks were close to random guessing performance. In fact, the DCLM paper acknowledges that other evaluation frameworks such as LightEval may be more suitable for evaluating small models. We also note that data mixing is a more subtle intervention on the training data than quality filtering, requiring more stable evaluations to reveal small differences between models. Ultimately, we found OLMES to be more reliable and informative. We also note that DCLM Core does not include MMLU, although MMLU is undoubtedly one of the most popular benchmarks for LLMs. We agree with the reviewer that it would be an interesting avenue for future work to investigate better and more diverse proxy data for domain optimization, but this was ultimately not the focus of our work. Instead, we chose MMLU and HellaSwag as they are widely known in the research community and used in almost all previous data selection works.

It would be better to make domain construction more automated with less human effort

Since pre-training data curation is so crucial in shaping the behavior of a model that may potentially be used by millions of people, we argue that there is some value in retaining human oversight in the data curation process! Nevertheless, we expect future methods to automate the process of taxonomizing and structuring the web with much finer granularity.

Relation to Bai et al.

Thank you for bringing this interesting paper to our attention! We agree that there are similarities with our work and we will definitely highlight them in the next version of our paper. Most notably, Bai et al. [3] also contribute a topic classifier of 13 topics derived by summarizing k-means clusters, which we will compare to our 24 topics. From our analysis in Section 2.4, we would expect k-means clusters to reflect topic information, whereas it would be harder to obtain format domains this way. Besides contributing a high-quality set of format domains, we believe our work makes several independent contributions: (1) We show that topic and formats are approximately orthogonal and their distributions can be combined, (2) We propose a new, simple and effective way of combining data mixing with quality-based data selection, (3) We study the implicit domain biases of two popular quality filters.

Questions: Unfortunately, we have reached the rebuttal space limit and cannot respond to the interesting questions here

[1] Abbas et al. "SemDeDup: Data-efficient learning at web-scale through semantic deduplication"

[2] Zhang et al. "Harnessing Diversity for Important Data Selection in Pretraining Large Language Models"

[3] Bai et al. "Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining"

Thank you for your kind review! We are glad you found the claims in the paper clear and supported by convincing experiments!

Assumption of selecting topics and formats independently

We don’t mean to make an assumption in the theoretical sense, but we simply propose to form a mixture distribution of p(topic, format) = p(topic) p(format) given individual mixture distributions p(topic) and p(format). This is motivated by Section 2.3, where we find that topics and formats often have close to zero pointwise mutual information, i.e., they co-occur almost as though they were independently distributed.

Thank you for your review of our paper! We are glad that you highlighted our extensive experiments and analyses, as well as the scalability and practical value of our proposed approach. We would like to briefly address two of your points.

Reliance on RegMix

In the paper, we clearly acknowledge that we use RegMix as an existing technique, since our focus is not on contributing another method for optimizing domain ratios given a set of data sources. Instead, we believe that the greater novelty of our work is to shift attention to defining the subsets for effective and large-scale data mixing. Nevertheless, we still extend RegMix to a novel setting, where we directly optimize the task loss of two downstream benchmarks (the bits-per-byte of the correct answers), instead of the perplexity on a corpus such as C4. In Appendix C, we also discuss several new modifications to RegMix that facilitate this, including fitting multiple per-task regression models and employing an iterative search method.

Scalability to improve multiple evaluation metrics is unclear

We will make predictions of the topic and format mixtures for additional target tasks (the existing RegMix checkpoint only need to be evaluated on new tasks) and include these in the appendix in the final version of the paper. In our current draft, we focus on only two target tasks, MMLU and HellaSwag, since (a) these tasks are well known in the community and are commonly used for evaluating data selection due to their relevance and low variance (b) we can more easily associate characteristics of the combined mixture with one of the two tasks (c) it was not clear to us how many tasks should be averaged (e.g., should we assign a higher weight to MMLU than one of multiple commonsense tasks?) (d) we reserve more tasks to be used for evaluating generalization to “unseen” tasks. In fact, we find that mixtures for MMLU and HellaSwag generalize well across many popular benchmark tasks. However, we agree with the reviewer that finding the “right” target tasks will remain a question for future research.

Thank you for your detailed review and for regarding our paper as a solid contribution to the field with strong empirical results. We are especially grateful for the detailed suggestions with respect to the typos and clarity in Section 5, and the many interesting and relevant citations. We will improve the paper with your feedback and discuss these references in the related works section. We will also use the extra page of the final version to add a clear limitations section before the impact statement!

Domain mixing results for only two downstream tasks

We focus on MMLU and HellaSwag, since these tasks exhibit interesting qualitative differences, are widely known in the community and are used by most prior works as stable and relevant metrics for data selection. However, we will include topic and format predictions for additional tasks in the appendix of the next version. Training additional models on many target task mixtures is unfortunately beyond our current computational resources and we believe that is equally important to hold out a set of tasks for measuring generalization to “unseen” tasks.

Questions:

1. Applicable to more general LLM corpora?

While we specifically designed our taxonomy with the goal of structuring many terabytes of web data, our domain classifiers can be applied to any text domain (we will also release a classifier that does not require an URL) and we would expect our models to produce reasonable results. Adapting the taxonomy is more complicated and computationally expensive — one would need to annotate additional data and retrain the domain classifiers — but our open-sourced scripts should make these operations run smoothly!

2. GPU time for training domain classifier?

Great question! Fine-tuning the domain classifier is relatively cheap. The first stage of training takes less than 8 H100 hours and the second stage less than 1 H100 hour. By far the most extensive stage is annotating 100k web pages with Llama-3.1-405B, which takes approximately 600 H100 hours.

3. Why gte-base-en-v1.5?

gte-base-en-v1.5 is the best small-scale embedding model on MTEB at the time of writing with support for longer documents (up to 8192 tokens). Unlike bag-of-word representations (like tf-idf), the neural encoder can produce embeddings which also reflect semantic and conceptual aspects of the source document.

4. Avoiding contamination

The task examples for fitting RegMix and evaluating our final models are from completely disjoint sets, and we never use final test examples for fitting domains. Since our domain mixtures are very low-dimensional, it is unlikely to leak substantial information from the test sets, but it is still best practice to use a different split for mixture optimization.

5. Takeaways from Figure 4

DCLM-fasttext and FineWeb-Edu are currently used to create some of the most widely used pre-training datasets and Figure 4 reflects the composition of these datasets. By comparing to the RegMix predictions, it also visualizes how the pre-training distributions change in the final four rows of Table 1, giving us some indication of which types of content are either “overlooked” or “overvalued” by the quality classifier. For example, we observe that FinWeb-edu has a strong inherent topic bias for Science, Health, and History, but overly filters out more diverse topics. Meanwhile, DCLM-fasttext uniquely has strong preferences for the formats Comment Sections and Creative Writing. We will expand on this discussion in the next version of the paper.

6. Perplexity scores in Table 2

These are computed on the same held-out data of the baseline corpus! We will add this for clarification.