2 OLMo 2 Furious (COLM’s Version)

Thank you for the helpful review and for appreciating our fully open, competitive language models and highly detailed paper. We would like to respond to the two points raised:

Limited novelty, lacking new ideas or methods

We respectfully disagree with the reviewer on this point.

First, we believe novelty in ideas or methods are one—but not the only—means to achieve impactful research for the community. The goal of our work is to support the broader language model research community to build, understand, and advance the science of language models. The result of our work is that our findings and data/code/model/etc artifacts are fully open while also maintaining a compute-efficient training recipe that produces models that are best among fully open models while highly comparable to leading weights-only models. If we view novelty is a means to impact, we would like to point to comments by Reviewers:

• Maj8 (“Fully open-sourced code and data significantly benefit the AI community while serving as an accessible resource for academia”) and
• 9NGC (“A very well written paper providing LLM training recipes which are going to be extremely useful for the community...I believe CoLM is a great venue for papers like this.”).

That being said, we would still like to point to where our main contributions bring novelty. First, we introduce some novel ideas for training stability, including repeated n-gram filters for training spikes and a more stable model initialization, and support them with ablations (Appendix E). On the topic of mid-training, at the time of writing, the area was underexplored in literature, especially not to the extent as we have done with experiments on learning rate schedules amid two-stage training, combined with data curriculum experimentation, as well as model souping results. We believe our exploration in mid-training goes far beyond what can be considered mere standard application.

Finally, if we look at some of the most significant reports from language model developers that have shaped our field significantly, often those behind leading open-weights models, they often comprise of a combination of (i) novel ideas/methods, (ii) novel compositions/applications of existing ideas/methods, and (iii) experimentation of existing ideas/methods that may have worked in one setting but have yet to see broader validation. We believe the modern full development cycle of language models requires combining all of these and we have crafted our paper to best reflect how these parts compose together.

Concern with usage of term “Pareto-Optimal”

The reviewer has expressed concern with our usage of the term “Pareto-optimal”; we believe this is a fair critique and would like to explain our thinking here.

First, we recognize we should more clearly define what we mean by Pareto-optimal, since it can be defined relative to many different criteria. The reviewer is right that relative to the criterion of Performance vs Number of Parameters, we are not Pareto-optimal. In our work, we are specifically defining our usage relative to Performance vs Training Compute which is proportional to Number of Parameters x Number of Tokens. That is, while the reviewer considers the criterion the model size (which is the key property at inference time), we are considering the training cost (which is the key property at development time). We believe our usage is still important as the focus of our work is about the development recipe—namely, training stability and mid-training strategies.

Second, we recognize the reviewer’s concern with our claim. In our work, we show that our training strategy produces a 32B model only 1-2 points difference from Qwen 2.5 32B while using 1/3rd the training tokens. Yet, the reviewer has rightfully pointed out that this does not mean that Qwen 2.5 32B, had it been trained on 1/3rd the training tokens, would not be better than our AnonModel 32B. While the counterfactual comparison would be valuable, the reality is that this data point does not exist nor is it possible to replicate as Qwen do not release their data, so we will never know for sure.

Nevertheless, we believe our fully open training recipe has demonstrated high flops efficiency. In our camera ready revision, we will replace the term “Pareto-optimal” with “Compute efficient” in places like the title and abstract where it could be confusing to readers without clarification. We believe this will help the COLM community members better discover and identify with our work (“compute efficient LMs” is under call for papers). And we will more clearly articulate in our abstract and introduction that our goal is cost-efficient model development and making these practices and artifacts accessible to others. We thank the reviewer for helping us make this improvement!

Thank you for the shout-out, we really appreciate it! We are excited to share the work and appreciative that the reviewer recognizes this will be a significant contribution and extremely useful for the community through our transparency and detail of our language model development. Even considering our work to be ”an example of how reports on new model should look like” is praise that encourages us to do more! We agree that CoLM is a great fit for this paper. Finally, we thank the reviewer for their critique on our bold face usage in Tables 2 and 3; you are right and we will change these to reflect standard bold usage (for best results) and use midrule for AnonModel delineation in the camera ready. If there are more questions or further clarifications, we would be happy to address them!

We thank the reviewer for the very positive feedback! We are happy to hear that they assess the work is great and that they cannot find any reason to reject it! We appreciate them highlighting our two-stage training strategy, our transparent and full technical details, the accessibility of our released artifacts for the research community, and the competitive performance of our models. Space-permitting, we will do our best to follow your suggestion to move Table 14 to the main text, thank you!

Given your strongly positive assessment with no negatives, we are wondering about your provided score of 7. If you have other questions or want further clarifications, we would be happy to address them if that would help give the reviewer confidence towards giving our work a higher score?

We thank the reviewer for their detailed comments and for highlighting our contributions towards transparency and competitiveness of open language modeling research! Our response is as follows:

Comparison with Qwen 3

The paper submission deadline was March 28th and Qwen 3 was released April 28th, so we could not have included these numbers in the paper. But we agree that it would be interesting to include these! We try our best to include up-to-date results (we even included Gemma 3 which was released mid-March), and we will include comparisons with Qwen 3 and additional models for which we have enough time to do the comparisons, in the camera ready!

Paper's scope is too broad, needs more details & ablations

We respectfully disagree with the reviewer on this point. We believe modern language model development involves many interconnected components (e.g. data, training, modeling, engineering, etc.) that cannot be meaningfully separated for scientific study without losing critical insights about their interactions:

• For example, our Sec 5 on mid-training combines both training (learning rate decay) and data (curriculum), which at the time of writing was highly under-documented/explored in the literature. The value of this investigation was a combination of both data and model training.
• If we look at some of the most significant reports from language model developers that have shaped our field significantly, they often also cover a broad scope of topics necessary to capture the full development of the model. Even the Qwen 3 paper, which the reviewer has asked us to compare to, does this.

We have done our best to maintain this important coverage of the full developmental cycle while also maintaining full openness in data, code, and more beyond what is typically seen in these language model development reports. Many papers in our field go into great detail on a specific slice of language model development, but far fewer works capture the breadth of what is needed to complete full language model pretraining; we believe our end-to-end account is complementary to the many component-specific papers that exist today.

Of course, we recognize the core concern the reviewer has, which is that covering these topics comes at a tradeoff between breadth and depth. This is difficult to navigate given COLM’s 9 page constraint, but we’d like to point the reviewer to the extensive ablations we’ve included in the Appendix E that hopefully satisfies the reviewer’s interest in depth of details:

• We have included experimental evidence of the impact of all our modeling stability improvements—n-gram filter (Appendix E1), new initialization (Appendix E2), reordered norm and QK-norm (Appendix E3.1), decreasing $\epsilon$ in AdamW (Appendix E4.1), removing weight decay (Appendix E4.2)---each change with an accompanying figure showing its impact.
• We have also included ablations of our mid-training interventions—learning rate decay rate/duration + two-stage training (Appendix E5), impact of data curriculum (Sec 5.2 Table 4), impact of mid-training (Appendix Table 14).

Limited technical novelty, better suited for a workshop

We respectfully disagree with the reviewer on this point. We believe that our paper goes above and beyond a typical workshop paper. We point to our response above about the value of works documenting the full development cycle, and that we have done so with extensive ablation results either in the main paper or Appendix E. On the topic of novelty, much of our experimentation on mid-training is underexplored in literature, and some of our ideas in stability are novel, including repeated n-gram filters for training spikes and a more stable model initialization. We believe in modern language model development, it’s important to have a balance between individual novel ideas, novel composition of existing ideas, and rigorous validation of existing ideas. We have done all three in this work in such a way so that our findings and data/code/model/etc artifacts are fully open while also maintaining a compute-efficient training recipe that produces models that are best among fully open models while highly comparable to leading weights-only models. We believe this work will have broad impact and is of high interest to the wider COLM community.

Why switch to GQA?

We thank the reviewer for bringing attention to this change from our 7B, 13B models (which used MHA) to our 32B model (using GQA). We made this change because it would provide faster inference (the KV cache for a 32B model would be enormous and it would be hard to deploy in practice). We will certainly add an explanation of this in the camera ready version!

Improve presentation of Table 2

Thank you for your suggestions on how to improve the structure of Table 2! We will make sure to improve the clarity of the paper and table for the camera ready version.