Thank you for your valuable feedback! We address your concerns below:

Q1: Randomness of the experiment results

A1: Thank you for raising this point. We acknowledge that a limitation of our study is the lack of multiple runs with different random seeds for most experiments, primarily due to the high cost of pre-training (our main 160B run required 1,536 H100 GPU hours). That said, we would like to emphasize the following:

(1) Single-run experiments are standard in LM pre-training studies, given the resource constraints. Our setup follows established practice, consistent with prior work (Xie et al., 2023; Li et al., 2024; Wettig et al., 2024). It is also generally accepted that pre-training exhibits less variance than fine-tuning.

(2) As you noted, Table 13 shows low variance of our pre-training experiments (3 different seeds and different subsets of data), particularly when averaging across the full evaluation suite. We believe that the result we present is significant and is not due to randomness.

(3) Figures 3–4 demonstrate consistent gains from MeCo across a range of model sizes and datasets, further supporting that the improvements are not artifacts of randomness but reflect meaningful trends.

Q2: Lack of zero-shot results.

A2: In this paper, we followed OLMES’s setting, which adopts curated 5-shot examples to reduce evaluation variance. But we agree that adding zero-shot results offers a better picture of model performance. We add a zero-shot evaluation for our main 1.6B, 160B, DCLM experiment as following:

As we can see, the MeCo model still achieves significantly better performance on zero-shot.

Q3: Lack of theoretical contributions

A3: We acknowledge that this paper did not provide a theoretical justification for MeCo’s effectiveness or a rigorous analysis of how MeCo changes the training dynamics. However, the main contribution of this paper lies in proposing the method and uncovering this interesting phenomenon (metadata conditioning accelerates pre-training), which is both novel and possesses significant empirical impact. Theoretical understanding of such a method is challenging due to the nature of pre-training, and is also beyond the scope of this empirical study.

That said, we included empirical ablations and hypotheses to shed light on the possible inner-workings of MeCo. In Sec 5.2 and Table 5, we showed that using hashed URLs can achieve similar performance as natural URLs, suggesting that the semantic meaning of the URLs is not necessary for better pre-trained models, and MeCo mostly provides signals to group similar documents together. We agree with reviewer TsUA that with the grouping information, the models can either learn to upweight certain “higher-quality” domains (such as Wikipedia) or learn an implicit curriculum that helps accelerate training—but it is still unclear to us how exactly MeCo changes the training, and it warrants further investigation.

We also highlight a recent preprint, Zhu et al., On the Power of Context-Enhanced Learning in LLMs. Though their setting is synthetic and differs from ours, their theoretical analysis shows that context-enhanced learning—such as providing metadata at the beginning of the sequence without actually optimizing cross entropy loss on these tokens—can improve sample efficiency. We found the result insightful and encourage the reviewers to check it out as well.

Thank you for your valuable feedback! We address your concerns here:

W1: Evaluation only includes commonsense reasoning and only reflects a few sources such as Wikipedia.

A1: We evaluate our models by using OLMES (Gu et al., 2024), the industry-standard evaluation behind AI2’s OLMo models (Groeneveld et al., 2024). Similar tasks are also used in Llama and Open LLM Leaderboard. While our evaluation does not include some of the latest popular LLM benchmarks on math (GSM8K, MATH), coding (HumanEval), graduate-level QA (GPQA), or instruction following (AlpacaEval, IFEval), we emphasize that at our experiment scale (1B model, 160B tokens), models do not achieve any meaningful performance on math, coding, and graduate-level QA—making these benchmarks uninformative. Additionally, since our study focuses on pre-training instead of post-training, the models are not expected to perform well on instruction-following tasks. Other studies that similarly investigate pre-training (e.g., Wettig et al., 2024; Yu et al., 2024) adopt comparable evaluation setups.

We also stress that OLMES covers more than just “commonsense”. For example, MMLU benchmarks model knowledge on a diverse set of subjects, including math, medicine, and economics; OpenbookQA and TruthfulQA test models’ factual correctness. This suite of tasks has been used by numerous pre-training studies and industry LLMs, and are considered to cover a diverse range of capabilities and to picture a holistic picture of the model performance. We respectfully disagree with the claim that the evaluation reflects only a narrow set of sources such as Wikipedia—for example, websites like personal blog-posts often benefit models’ commonsense performance.

W2: MeCo only discussed the case where all metadata is available. Most curated raw texts do not contain metadata.

A2: We respectfully disagree with the reviewer on “most curated raw texts do not contain metadata”. Most pre-training sources, such as CommonCrawl, C4, FineWeb, RefinedWeb, and DCLM, provide at least the URL information. For companies that perform their own data crawling, retaining metadata like URLs is standard practice.

W3: Explanation of the pre-training efficiency

A3: Thanks for raising this point! We included empirical ablations and hypotheses to shed light on the possible inner-workings of MeCo. In Sec 5.2 and Table 5, we showed that using hashed URLs can achieve similar performance as natural URLs, suggesting that the semantic meaning of the URLs is not necessary for better pre-trained models, and MeCo mostly provides signals to group similar documents together. We agree with reviewer TsUA that with the grouping information, the models can either learn to upweight certain “higher-quality” domains (such as Wikipedia) or learn an implicit curriculum that helps accelerate training—but it is still unclear to us how exactly MeCo changes the training, and it warrants further investigation.

We also highlight a recent preprint, Zhu et al., On the Power of Context-Enhanced Learning in LLMs. Though their setting is synthetic and differs from ours, their theoretical analysis shows that context-enhanced learning—such as providing metadata at the beginning of the sequence without actually optimizing cross entropy loss on these tokens—can improve sample efficiency. We found the result insightful and encourage the reviewers to check it out as well.

Thanks for your positive review! We are glad that you found our method interesting and our experiment design sound. To answer your questions:

Q1: Chiang and Lee, 2024 show that providing different source information often does not affect RAG results. How to interpret this?

A1: Thanks for providing the reference and we’ll include a discussion of this work. We believe the influence of metadata depends on both the metadata type and the model’s training. As noted by Chiang and Lee, publication time significantly impacts RAG performance, while source does not—indicating higher sensitivity to temporal information. However, under MeCo training, models also become sensitive to source metadata as it’s explicitly conditioned on such information—this is evidenced by our conditional inference result.

Q2: Why does MeCo work? Can you release intermediate checkpoints?

A2: Thanks for raising this point! We included empirical ablations and hypotheses to shed light on the possible inner-workings of MeCo. In Sec 5.2 and Table 5, we showed that using hashed URLs can achieve similar performance as natural URLs, suggesting that the semantic meaning of the URLs is not necessary for better pre-trained models, and MeCo mostly provides signals to group similar documents together. We agree with the reviewer that with the grouping information, the models can either learn to upweight certain “higher-quality” domains (such as Wikipedia) or learn an implicit curriculum that helps accelerate training—but it is still unclear to us how exactly MeCo changes the training, and it warrants further investigation.

We will also make sure to release all intermediate checkpoints to facilitate future research.

Thank you for your suggestions and questions! We appreciate that you recognize the paper’s contributions and strengths. To address your concerns:

W1: Lack of theoretical explanation—how does MeCo change the training dynamics?

A1: Thank you for raising this point! We acknowledge that this paper did not provide a theoretical justification for MeCo’s effectiveness or a rigorous analysis of how MeCo changes the training dynamics. However, the main contribution of this paper lies in proposing the method and uncovering this interesting phenomenon (metadata conditioning accelerates pre-training), which is both novel and possesses significant empirical impact. Theoretical understanding of such a method is challenging due to the nature of pre-training, and is also beyond the scope of this empirical study.

That said, we included empirical ablations and hypotheses to shed light on the possible inner-workings of MeCo. In Sec 5.2 and Table 5, we showed that using hashed URLs can achieve similar performance as natural URLs, suggesting that the semantic meaning of the URLs is not necessary for better pre-trained models, and MeCo mostly provides signals to group similar documents together. We agree with reviewer TsUA that with the grouping information, the models can either learn to upweight certain “higher-quality” domains (such as Wikipedia) or learn an implicit curriculum that helps accelerate training—but it is still unclear to us how exactly MeCo changes the training, and it warrants further investigation.

We also highlight a recent preprint, Zhu et al., On the Power of Context-Enhanced Learning in LLMs. Though their setting is synthetic and differs from ours, their theoretical analysis shows that context-enhanced learning—such as providing metadata at the beginning of the sequence without actually optimizing cross entropy loss on these tokens—can improve sample efficiency. We found the result insightful and encourage the reviewers to check it out as well.

W2: Arbitrary duration choice for cooldown

A2: We provide an ablation study on the duration of cooldown in Table 14 (Appendix B.3), which demonstrates that 10% cooldown leads to competitive performance.

W3: Though the authors conducted evaluation on some standard benchmarks, there lacks (1) non-English tasks and (2) evaluation on robustness across more diverse tasks.

A3: Thanks for raising this point! We evaluate our models by using OLMES (Gu et al., 2024), the industry-standard evaluation behind AI2’s OLMo models (Groeneveld et al., 2024). Similar tasks are also used in Llama and Open LLM Leaderboard. While our evaluation does not include some of the latest popular LLM benchmarks on math (GSM8K, MATH), coding (HumanEval), graduate-level QA (GPQA), or instruction following (AlpacaEval, IFEval), we emphasize that at our experiment scale (1B model, 160B tokens), models do not achieve any meaningful performance on math, coding, and graduate-level QA—making these benchmarks uninformative. Additionally, since our study focuses on pre-training instead of post-training, the models are not expected to perform well on instruction-following tasks. Other studies that similarly investigate pre-training (e.g., Wettig et al., 2024; Yu et al., 2024) adopt comparable evaluation setups.

That said, we acknowledge that the evaluation suite we use focuses on English tasks, which is a common limitation in language model pre-training studies.