Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale

Thanks for recognizing the practicality of our method and its strong empirical results across model scales and datasets. For your questions and suggestions:

$\textrm{\color{blue}Question 1}$

The comparison in Table7 in the appendix would be important to show in Table2, and the "kept ratio" would also be important in Table 2.

Thank you for your careful reading and for recognizing that ProX achieves a higher kept ratio than FineWeb-Edu. We will update the presentation accordingly in our revised version. Regarding the kept ratio of rule-based filtering methods on RedPajama-V2, we summarize the results as follows. Notice that aggressively ensembling these filtering methods does not show very obvious improvement (+0.2%) but only keeps 50% of the data.

On FineWeb, we observe the following:

Despite keeping more than 3× the data, ProX achieves comparable downstream performance to FineWeb-Edu, highlighting its effectiveness in retaining high-quality content.

$\textrm{\color{blue}Question 2}$

Here all baselines only perform document filtering, so showing separate rows for either document filtering, chunk rewriting, or both would be important to understand the source of the gain.

In Table 3, we use only prox's doc refining performance to compare with other baseline data selection methods (please see our experiment setting from line 294 to line 296) This is mainly to show that using a very small LM on quality refinement task is indeed very simple and effective compared to other data selection baselines.

$\textrm{\color{blue}Question 3}$

The chunk-level rewriting could be compared to other approaches of rewriting text. A good baseline would training the same ProX base model to generate the cleaned document (with headers, footers, URLs, navigation removed) directly.

Thank you for the insightful suggestion. While rewriting is a valid approach to improve pre-training data quality, it incurs high inference costs due to full-document generation and relies heavily on large models. Recent work like WRAP and Nemotron-CC uses models over 7B (e.g., Mistral NeMo 12B). In contrast, ProX improves data quality with much lower compute by using a small 0.3B model and concise outputs. We believe efficient rewriting would require a strong base model and leave this as future work.

$\textrm{\color{blue}Suggestion}$

Include deduplication as a standard part of the data curation pipeline and its relation to ProX.

Thank you for the suggestion! We have discussed in the related work section that deduplication plays an important role in data preprocessing—it reduces redundancy, improves training stability, and enhances efficiency. ProX is orthogonal to deduplication, as it focuses on the quality of individual samples rather than redundancy. In practice, ProX can be applied after deduplication to further improve data quality while saving compute. We will include this discussion in the revised related work section.

$\textrm{\color{blue}Question 4}$

Can you explain what seed data the fasttext baseline is trained on?

To ensure a fair comparison, we re-trained the fastText classifier on the exact same training data used for our ProX doc-level refining models. All documents labeled with drop_doc() are treated as negative samples, while those labeled with keep_doc() are treated as high-quality samples. We trained the language model on FastText filtered data from scratch using all the same configuration as the other experiments in Table 2.

However, why not add FineWeb-Edu to it (which should also achieve a 1.2% avg performance gain if I'm not mistaken)?

I also believe it is important to mention the relationship to the FineWeb-Edu prompt in the main text.

I would not be surprised that a fine-tuned 0.3B model could regenerate the source document with headers / footers / URLs removed.

Thank you for recognizing our extensive experiments, method soundness and novelty! We are happy to answer all your questions.

$\textrm{\color{blue}Question 1}$

What is the main reason why documents get lengthier after cleaning? I would expect the opposite (maybe shorter documents are getting deleted?)

Yes, as you correctly pointed out, the primary reason for the longer average document length in ProX-refined data is that general text corpora—even after extensive rule-based filtering—still contain many short and noisy documents. These low-quality documents are often discarded during the document-level refining stage, leading to a curated dataset with naturally longer documents.

At the chunk-level refining stage, we observe a 5–10% reduction in total tokens depending on the corpus, and document lengths correspondingly decrease. However, they still remain consistently longer than those in the original corpus, indicating that ProX effectively removes noise while retaining substantial and meaningful content.

$\textrm{\color{blue}Question 2}$

Maybe you're plotting the frequency and not density?

Thank you for your careful reading! After checking it, we do not perform normalization on the Y-axis, thus showing the frequency instead of density. We will correct this typo in the updated version.

Thank you for recognizing our work! We are truly delighted to see your appreciation of ProX's novelty, efficiency, and effectiveness in improving data quality. Regarding your question:

$\textrm{\color{blue}Question 1}$

Can the exact rules for C4, Gopher, and FineWeb be made available in the appendix?

Thank you for the suggestion. For these rule-based filtering methods, we primarily refer to the implementations provided in the FineWeb codebase. We are happy to summarize them below and will include the full details in the appendix in our revised version.

• C4: We re-implement the C4 filtering rules, including:
  1. Filtering lines with overly long words or too few words
  2. Removing citations, placeholder text (e.g., lorem ipsum), JavaScript, bracket content, and web policy mentions
  3. Discarding documents with too few sentences
• Gopher: We adopt Gopher’s rules, including:
  1. Minimum and maximum word count per document
  2. Symbol-to-word ratio
  3. Average word length per document
  4. Heuristics such as bullet point filtering
• FineWeb: We use FineWeb’s official implementation, which includes:
  1. Punctuation ratio checks
  2. Duplicate line removal
  3. Filtering based on newline character frequency
  4. Additional combined filters integrating the above rules

We hope this clarifies our reference to these rule-based methods, and we’ll ensure their presentation in the appendix for reproduction.

$\textrm{\color{blue}Question 2}$

Could experiments/ analyses be conducted to isolate ProX's performance improvement due to per-sample application versus its particular rules.

Regarding the performance of ProX in isolation, we would like to clarify two points:

1. ProX inherently and functionally covers other rule-based filtering methods, as it leverages a language model to learn and apply data selection criteria, effectively capturing patterns encoded by handcrafted rules.
2. During our preliminary study, we explored training the refining model using data filtered by FineWeb rules as negative samples. Compared to using the original raw data(a subset of Red-Pajama-V2), ProX achieved approximately +1.0% improvement in downstream performance, indicating its ability to identify lower-quality documents.

Furthermore, we observed that more aggressive filtering tends to yield better performance. The use of quantifiable scores such as Edu Score and Format Score offers us the flexibility to adjust filtering thresholds and control the trade-off between data quality and quantity.

$\textrm{\color{blue}Question 3}$

Could the paper provide an example of how a practitioner/ domain expert would add more complex functions to ProX and which parts of the procedure they would need to adjust?

Regarding the addition of specific rules or filtering strategies, we believe it is important to distinguish between document-level and chunk-level refining:

• At the document level, if a domain expert aims to remove biased content (e.g., safety issues or toxic data), they can simply annotate relevant keywords and mark the corresponding documents with a drop_doc label. This allows for efficient and targeted removal based on expert knowledge.
• At the chunk level, domain experts may need to:
  1. Identify specific types of noise with clear patterns — such as frequently repeated dates in forum pages, pervasive advertisement slogans, or unnatural line breaks that disrupt the reading flow.
  2. Abstract the noise removal strategy into a reusable function — for example, leveraging regular expression (regex) syntax for flexible and customizable pattern-based filtering.
  3. Use model or human annotation to create seed data, which can then be used to train a refining model tailored to the noise patterns of interest.

We hope this explanation clarifies our approach.

So I am wondering if we could look at what happens if the underlying rules of ProX are the same as the other rules. Here's an example of how I would imagine doing this; take a Gopher rule: Gopher rule: "Minimum and maximum word count per document" --> this traditionally requires some hardcoded number (i.e., if # words < 10 then discard) To do a ProX version of this rule, you could create the seed data by prompting the LLama-70b model to flag documents if they appear too long or too short, but the key thing is that you don't enforce a hard rule, like 10 words.