FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language

We thank reviewer 5fxD for the very positive assessment of our work and for highlighting the deduplication and rehydration parts of the paper.

How did you choose which documents to keep after deduplication? Since Zyda2 [1] reports different quality levels for documents within the same cluster, did you investigate the impact of your selection?

We thank the reviewer for their thoughtful question regarding deduplication. While in this work we simply chose a random document to keep for each cluster, we plan to further explore data diversity within duplicate clusters in future work. We believe that a simple change to our rehydration pipeline could provide some gains: instead of simply repeating a document N times, we could instead sample N distinct (and potentially while maximizing their diversity) documents from each cluster.

The Zyda2 findings are quite interesting but we would like to point out that in our understanding, the specific result regarding the drastically different classifier scores for extremely similar documents (with only a handful of words changed) suggest a limitation of the classifiers themselves, and not that the same document without “More You may also like…” produces a significantly stronger model.

Besides training monolingual datasets, a multilingual dataset will likely also be used for training multilingual models. Therefore, I believe it would strengthen the paper to compare a multilingual model on FineWeb2 with those trained on existing alternatives.

While we agree in principle, a multilingual model introduces additional confounders such as the proportion of data from each language (mixing), as well as cross-language transferability (we would not be sure that observed gains in a language would be exclusively due to the improved quality of that language’s data). Additionally, we would still be limited by scalability issues in terms of evaluation tasks: besides translation tasks which are generally available, we’d still need to curate benchmarks for other tasks for each additional language.

We would like to thank reviewer faKG for their detailed review and for raising interesting points regarding the tail end of minority languages. We address their concerns below.

Appendix

We agree with the reviewer’s concerns regarding the use of the Appendix and will update our draft to better integrate some of the results the reviewer mentions.

Tail end of minority languages

We believe the reviewer raises a very relevant point regarding languages that have no (or extremely low quality) content. We share the reviewer’s concerns regarding the quality of languages with very few documents in the final dataset, which is what motivated us to develop the “Precision filtering lower resource languages” section, inspired by work related to the reference the reviewer mentions (Caswell et al. (2020); Bapna et al. (2022)). The version of the table with language composition present in the draft is from before we applied this final step, hence why those tail end languages are shown with such short document counts. We will update the paper with the final data and likely replace the table with plots showing the composition, in line with the reviewer’s criticism of the excessive number of appendix pages, half of which are taken up by this final table.

Tokenizer and evaluation

for the ablation experiments with the 9 canary languages: why use the Gemma tokenizer instead of just training language-specific ones (with a smaller vocab)? I don't think it makes much sense to use a 250k vocab to train monolingual models (would also save some computation for the ablations)

Not training our own tokenizer (usually on our own data) ensures that we do not unfairly benefit our own dataset. Since most languages do not have state-of-the-art small vocabulary tokenizers available, we opted instead to choose from among the well-known multilingual vocabularies with a large vocabulary size. Furthermore, though we agree with the reviewer regarding the additional cost, using the same tokenizer for all languages allows us to simplify our codebase and avoid accidental mistakes.

line 320: average of ranks - what exactly are the ranks?

We will clarify this part. We compute the aggregate score of each method for each language (as described in the evaluation section) and then sort the methods by this score (rank 1 means a method achieved the highest aggregate score in a given language, etc) and average the ranks across languages.

Typography

We thank the reviewer for pointing out typographical errors and will correct them.

We would like to thank reviewer RcCQ for their positive assessment of our work, and specifically for recognizing the importance and potential impact of our released resources and findings. We appreciate the relevant questions raised by the reviewer regarding evaluation and address their concerns below.

Appendix

We agree with the reviewer’s concerns regarding the use of the Appendix and will update our draft to better integrate some of the results the reviewer mentions.

Evaluation tasks

Relatedly, the proposed selection of evaluation tasks is vaguely defined without giving enough background on why these choices were necessary.

Is the reviewer referring to the final list of evaluation tasks (presented in Appendix A.5.3) or to the criteria used to select tasks (mentioned in Section 3.3 and formally defined in Appendix A.5)?

In the first paragraph of Section 3.3 we justify the need to select tasks ourselves, since for most languages no “standard” evaluation suites are available, and many tasks have subpar quality (originating from machine translation).

We would be happy to update Section 3.3 if the reviewer is willing to clarify which part of the evaluation setup they believe requires further background. Additionally, we can prioritize moving information from Appendix A.5 into the main text to help clarify.

Non-random performance early on: L159-162: Does this argument conflate task difficulty with task usefulness? The lack of signal from ablations at a specific model scale could be attributed to limited model capacity, rather than indicating that the task itself is not useful.

The reviewer raises a great point that we are happy to clarify. We consider a task to be useful if it allows us to make informed comparisons between different datasets/models at the scale we run our experiments. This means that a task that provides non-random performance at a large scale but not at our small scale (due to its difficulty) is not useful for our experiments (even if it could be at a larger scale). We explicitly mention this as a limitation in the Conclusion: “[...] we studied “high-signal” properties of each task at the very early stages of model training, and so it is possible that the properties could change significantly as training progresses, making some tasks more viable.”

While the authors present results on "unseen" languages, given that the tasks are selected based on the model learning signal itself, there is a risk of circular reasoning.

This is a reasonable concern that we were not able to avoid due to the previously mentioned lack of established benchmarks for most languages. Out of a concern that our task selection could unfairly benefit our own dataset, we selected tasks using a similar approach to the canary languages: we relied on other datasets (excluding our own) to quantify task-signal.

Deduplication

L260-262: could it be because the deduplication was done globally per language? Was there any analysis on cross-language "information" overlap?

While we did not perform the mentioned analysis, we investigated a related question: Is the different impact per-language related to the degree of duplication in each language? We looked at the percentage of data removed during deduplication for each language, but the languages sorted by this value (see table below) do not align with the order of performance impact: French and Chinese, two languages that both show little benefit from deduplication, are ranked on opposite ends of the table. As both the language with the most (removed) repeated content and the one with the least showed very similar performance effects after deduplication, it seems that the degree of duplication is not the leading factor in the disparate results we observe across languages. We also considered other hypotheses, but were similarly unable to draw any definitive conclusions.

believe it would be useful to discuss this in the main body or even as a footnote next to the key criteria as opposed to the conclusion because they ground the usefulness of the metrics used for selection for use cases (e.g. model sizes).

Indeed we should have made it clearer when we first introduce the task selection process that it targets the specific scale at which we run our ablations, we will update our draft to highlight this aspect.

We thank reviewer A835 for their thorough review. The reviewer highlights the important contributions provided by our pipeline and high quality dataset. We address the points raised by the reviewer below.

Appendix

The reviewer’s main concern has to do with the way the Appendix was used in our paper. While the way we structured the paper stemmed from an attempt to fit most of the content that we considered relevant in the 9 page limit, we fully agree with the reviewer that the current structure does not make for the best reading experience.

We are committed to address this issue and will update our draft to improve the general flow and content imbalance between the main body of the paper and the Appendix, and in particular to make important results clearer in the main body of the paper.

Under-explored sections

The reviewer mentions that certain sections of the paper are under-explored. We believe this concern is related to the previous one, and will strive to make the different sections clearer. Specifically, for deduplication (Section 4.3, last paragraph, impact on data quality) we will mention that even the languages that seem to not benefit from deduplication see large performance uplifts during rehydration (that relies on deduplication); for filtering recipes (Section 4.4, statistics on each language on different corpora, different metrics etc.), we will include illustrative examples with some metrics across datasets; stopwords (Section 4.4.1, why at least two words from the stopwords list?) this follows from the original Gopher approach (we will clarify).

Misc questions

The paper talks about “early-signal” benchmarks and “high-signal” benchmarks. Are they the same? Is there a difference?

We consider there to be a difference: high-signal benchmarks allow us to properly compare different datasets/models with a certain degree of confidence regarding the meaningfulness of score differences. Early-signal benchmarks provide this “high-signal” behaviour in the early stages of pre-training, which is the setting we use for our experiments. That said, we do use these two terms somewhat interchangeably in the paper and will change this to be clearer in our draft.

Section 4.1: the mentioned “blocklist” was not questioned at all but simply accepted as fit for purpose?

This blocklist is employed in previous literature, specifically in FineWeb (Penedo et al., 2024) and in RefinedWeb (Penedo et al., 2023). In the latter, a thorough analysis of this blocklist is provided. We will update our draft to point readers to RefinedWeb’s analysis.

Several times phrases like “the concurrent work HPLTv2.0” are mentioned. Why exactly is HPLTv2.0 singled out and labelled as “concurrent work”?

This is a relevant work that was published at a late stage of the development of our own paper, which is why it was not included in, for example, our baseline datasets list that was used for task selection. We agree with the reviewer that always referring to it as “concurrent” can be excessive, and will update our draft accordingly.

The choice for GlotLID (paragraph 3 on page 5) needs a more comprehensive explanation.

We will attempt to more clearly justify this decision in the main body of the paper. Our reasoning (as we detail in Appendix A.6.1) was primarily related to the larger number of labels supported (which allows us to have data in many more languages) as well as to some unique design decisions of GlotLID compared to other LID systems: it explicitly includes scripts, using script detection to narrow down possible labels and allowing us to tailor filtering to each script (in languages with multiple ones); UND labels for unsupported scripts (so that they are not assigned to another label); and specific “noise” labels such as text decoded with the wrong encoding, binary content, or misrendered PDFs, preventing it from being classified as a natural text language.

Section 4.4.3: I would’ve expected a discussion of mixed-languages documents here (or somewhere else in this paper).

We believe mixed-language documents can be useful for multilingual training (they essentially provide “free” parallel data) and that language classifier confidence scores may also be affected by this type of data, but indeed we have not specifically discussed them in our work. Could the reviewer please clarify which specific aspects of mixed-language documents they would like to see discussed in the paper?

Typography changes

We deeply thank the reviewer for taking the time to compile different typography issues found in our work, which we will update with most of the proposed changes.