Strengths:

Practical considerations and scenarios

It is commendable the authors for thoroughly exploring a diverse range of distinct and practical data constraints and scenarios. Unlike many dataset papers that make a particular assumption of constraint and conduct experiments solely around that, which can sometimes limit practical applicability, this paper investigates scenarios such as:

• Reproducing Muennighoff et al.'s work on diminishing return of repated data
• Simulating scenarios where "the dataset we are trying to manipulate is closer in size to the desired token count"
• "fix the token budget and the number of unique tokens used for training"

This broad practical scope significantly enhances the utility and relevance of the findings for real-world LLM training. These results will be valuable to different training goals, such as small models, large models, inference-optimal models, etc.

Practical findings and training methods

The paper finds that adjusting weight decay will help reduce the diminishing return of repetition, and also provides a practical data sampling method. The weight decay method remind me of [1], where the authors theoretically study the interplay of weight decay and data set size, by considering the model weights as a EMA of the weight updates. This finding here highlights that dataset study should not be considered simply as a "preprocessing step" before training.

The study offers actionable strategies for optimizing LLM training with constrained resources, whether due to limited unique data volume or specific compute budgets.

Weakness

Writing

The paper presents a list of distinct experimental settings and comparisons. However, the overall organization and presentation could be improved to make it easier for readers to find and consistently follow the flow of these numerous experiments, sometimes lacking clear initial roadmaps or explicit transitions between different scenarios.

[1] https://arxiv.org/abs/2502.15938

Strengths:

• The paper spans multiple model sizes, a wide range of tasks, and detailed ablations (perplexity, core metrics, MMLU), which makes the findings very robust.

• Beyond just “here’s a trick,” the authors provide hyperparameter settings, pseudo-code for count manipulation, and appendices with implementation details—bonus points for reproducibility.

• While prior work has touched on data filtering and sampling, the explicit pairing of multi-epoch repeats, plus the doc-level repeat allocation, limited-resource budget regimes.

Weaknesses:

• Key insights (e.g., that boosting weight decay is required to make repeats helpful) are buried. Figure 1’s default curves actually mislead unless you already know to jump to Table 1. There’s no high-level roadmap or a guiding “flow” diagram, so it reads like disjointed blog posts stitched together.

• The “why” behind count manipulation isn’t motivated with simple analogies or visuals. Readers unfamiliar with corpus building jargon may struggle to grasp why document-level repeats beat global strategies.

• Juggling four different model scales × three evaluation metrics × multiple sampling schemes makes it hard to see the forest for the trees. A more focused presentation (or a distilled summary table) would help.

Strengths

The paper is well-written and explores the interaction between data quality and repetition in a series of generally well-designed experiments. The results provide useful insights for practitioners when limited curated data becomes a constraint.

The first group of experiments in Section 3 mainly explore how the effects of repetition varies on 3 datasets (C4, a replication of RefinedWeb, and DCLM-baseline) across different scales:

1. The first experiment (Figures 2 and 3) explores how repetition affects performance for each of the 3 datasets at the 1B scale. Each run starts with training the model to the Chinchilla optimal duration with unique tokens from the datasets. Then 3 different separate runs are done with different size pools of the data; for each run the perplexity on the C4 validation set is reported as well as the averaged performance on a set of downstream tasks. For DCLM, the same experiment was performed at the 3B scale.

2. The second experiment (Figure 4) is similar except it fixes the total training budget across runs and then subsets the data to achieve the desired number of repetitions. This is done for different model sizes and durations on the DCLM-baseline data; each model size/duration also includes the baseline performance on one epoch of RW v2.

3. The third experiment (Table 1) explores using weight decay to mitigate the performance degradation from repeating tokens. Here the experiments are fixed at model size 12.6B parameters, duration 252B tokens. The number of repeats and magnitude of the weight decay is then varied. Figure 1 then emphasizes a scenario where weight decay can be used to meaningfully improve performance at the high repeat scale.

Section 4 then turns to how to "create better datasets relative to a token budget by explicitly manipulating the counts of individual document," focusing on repeats at the document rather than dataset level. Two main techniques are explored: subsampling methods and count manipulation. Count manipulation fills the token budget with documents in descending order according to some quality metric and utilizes different rules for how many documents to keep at each level of quality. The authors emphasize that count manipulation becomes fundamentally different from filtering in that high quality documents can be upsampled so that they appear more times in the new dataset than they do in the original. The performance of these strategies is explored in Tables 2-4.

Weaknesses

See the questions section for some concerns regarding Section 4 where I think the motivation behind some of the design choices could be made clearer.

Overall, some of the figure legends and captions could be improved to make the experiments easier to parse for the reader. A few specific examples (primarily to aid in improving the paper in revisions):

The caption for Figure 4 says "Legend denotes model size and multiplier to scale tokens seen at Chinchilla optimal." I presume this means that this is setting the total duration for each run in terms of a multiplier on the Chinchilla optimal duration, i.e. the duration for each run is the multiplier times 20x the number of parameters. However, because you are also changing the number of repeats this notation is not immediately clear. My suggestion would be to label the 6 colors with a more verbose description of the model size/duration settings and then have 2 more labels that explain that the solid and dashed lines correspond to the 2 datasets.

On the left hand side of Figure 1, since there are only 4 compute settings it would be useful to include the model size/training duration in the figure or caption. I assume from the $x$-axis label that the duration is the Chinchilla optimal for each model size, so just the model size could be sufficient. On the right hand side, it would help to clarify that 12.6B 252B refers to the model size and duration (later you have things like DCLM 230B where the number refers to the dataset size and not the duration). Also in Figures 1 and 2 you use the abbreviation RW v2 which isn't explained until page 4.

On the right hand side of Figure 1, it looks like "DCLM 12.6B 252B" is the WD 1x column from Table 1 and "DCLM 12.6B 252B WD" is the max performance for each row, which is mostly WD 2x. Some explanation of this would be helpful to make it clear why the curves match at 1 epoch. Also I presume that the largest compute scale in the left panel is the 12.6B parameter, 252B duration scale explored in the right panel. Using some marker to indicate that the 1 epoch and 10 epoch points have matching values on the left panel could be helpful given that the current shared colors are confusing (the 1 epoch point is coming from the purple line on the left).

Strengths

• A Counter-intuitive and Foundational Insight on Dataset Repetition: The paper's most impactful contribution is the robust demonstration that repeating a smaller, aggressively filtered dataset for multiple epochs can outperform training for a single epoch on a significantly larger, but lower-quality, superset. This finding provides a clear, evidence-based strategy for navigating the increasingly critical issue of data scarcity and directly challenges the prevailing "single-epoch" training orthodoxy in the LLM community.

• A Novel, Granular Approach to Data Composition: The work moves beyond coarse, dataset-level repetition to explore the more nuanced concept of document-level repetition. By introducing "count manipulation," the authors reframe dataset creation as a principled optimization problem, where the frequency of each document is explicitly controlled based on quality metrics. This represents a more powerful and flexible paradigm than simple filtering or uniform sampling.

• A Practical Roadmap for Data-Constrained Scaling: Together, these insights offer a practical roadmap for training high-performance models when the volume of unique, high-quality data is a limiting factor. The paper forces a critical re-evaluation of long-held assumptions and suggests that the community may have prematurely dismissed the utility of multi-epoch training, a standard practice in other deep learning domains.

Weaknesses

• Uncertain Extrapolation to Frontier-Scale Training Regimes: The primary weakness of the study lies in the scale of its experiments. While substantial for academic research, the models are trained on token budgets in the hundreds of billions. This is approximately an order of magnitude smaller than the over 30 trillion token datasets now commonly used to train state-of-the-art frontier models.

• The Question of Shifting Scaling Laws: The core uncertainty is whether the observed phenomena will extrapolate linearly to these much larger compute scales. It is plausible that scaling laws may shift, and the optimal trade-off between data quality, quantity, and repetition could be qualitatively different in a multi-trillion token regime. Therefore, it remains an important open question whether the benefits of heavily repeating a 1T token high-quality dataset would still hold against a single pass through a 15T token moderately-filtered corpus.