Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Thank you for the detailed review and for highlighting our novel, open and well-performing post-training recipe!

• Regarding the sequence of preference alignment and RLVR: It is first important to note that Llama 4 was released after the submission of this paper (March 28th vs. April 5th). At the time of this submission, our initial investigation showed a slight improvement of running RLVR after the DPO step. Generally, we didn't see degradation on tasks after using RLVR. However, we agree with the reviewer that the future work could show a detailed experimentation on the order of these steps.
• Regarding Reasoning Models: the main purpose of this work is to establish post-training recipes for generalist language models (such as gpt4o or deepseek v3). Since the submission of this work, several lines of work study the role of RLVR to build reasoning models. We agree that future work should include developing post-training recipes for general-purpose reasoning models.

We thank the reviewer for highlighting that our paper bridges the gap between open and closed post-training methods, by releasing open recipes, code, data and an evaluation benchmark. As the reviewer pointed out, this paper is distilled from a much longer technical report. We will make sure to improve readability and to clean up the Appendix (thank you for pointing out the appendix - paper reference mismatches!), for the camera-ready version.

Answers to the questions:

On prompt selection and the number of prompts: The final number of prompts and their composition were determined according to the data mixing results, for both SFT and DPO.

• For SFT, we ran careful data ablations to balance performance across our evaluations, and we found that adding “too much” data from one domain, for example math, degraded non-math evaluations, while not providing further benefit on math-specific evaluations. We therefore downsampled the prompts to maintain performance with fewer samples.
• Similar concerns were valid for DPO. We additionally used smaller prompt subsets for DPO because of cost considerations and also because earlier experiments showed that scale matters less compared to the diversity of the responses (and some realities of needing to ship the model at some point even if we have more opportunities!).

On dataset licensing: Licensing is a very important topic – thank you for further feedback on the area that is too often ignored! When sourcing prompts, we took consideration of the licenses of the original datasets, and we even manually tracked the provenance of the subsets of datasets and removed those with issues (for example ShareGPT). Regarding some of the specific datasets mentioned in the review:

• For PersonaHub (CC-BY-NC-SA) we generated (prompt, response) pairs ourselves, conditioned on the personas and we discarded all persona description. Our data does not contain the personas that are covered by that license. For future models we’re exploring regenerating the persona’s to get out of this licensing grey area.
• For the data derived from Llama: we apply our recipe to llama models and therefore decided that using llama derived data would be fine for our purposes. We comply with the llama3.1 license and named our models and data with the prefix “Llama-3.1”.
• For data composed of other frontier model outputs: we follow standard practice of the community that has used and released many datasets with outputs from these models. In doing so, we try to remind users in the dataset cards that the data is subject to the terms and conditions of those providers, but historically there has been no meaningful enforcement of them.
• For the No Robots dataset: We are removing that from future versions and will be releasing models without it ASAP.

We thank the reviewer for their thoughtful comments and we are happy to see that the reviewer appreciates the comprehensiveness and diversity of our experiments and results.

• On the novelty of RLVR: Our approach to RLVR is inspired by prior work using RL for improving reasoning skills (e.g., Kazemnejad et al., 2024), and we further extend it to cover more domains, are the first to use asynchronous PPO for RLVR training, carefully ablate key aspects, and evaluate RLVR’s effect on a broad range of evaluations, not just limited to mathematics. More specifically,
  • We show that RLVR itself can extend beyond comparing to ground-truth answers with our IFEval results, where there are no ground truth labels involved (as we instead provide reward based on how well the model’s output adheres to the given constraints).
  • While prior approaches did indeed use iterative SFT, we instead use an efficient asynchronous PPO setup (section 6.2), and we cover more domains and evaluations than prior work.
  • We ablate core aspects of the setup, including value model initialization, RM scores, and the effect of differing policy initializations.

We note that while there is indeed a current boom of ‘reasoning’ models, our work is prior to much of this work and also has a unique focus: rather than explicitly trying to target reasoning skills and reasoning skills alone, we instead focus on how to improve general non-thinking performance whilst maintaining (or improving) in domains beyond math. As such, we believe that our approach is novel (as said by reviewer yFmp) and contains many useful findings for the field.

• On DPO: Our experiments (shown in tables 6 and 7) show that the pipeline that includes the DPO step significantly improves a model’s general chat capabilities over the SFT step (+20 points on AlpacaEval 2, for example, for the 8B model). Additionally, we applied RLVR to a subset of domains that were verifiable, i.e. precise IF and math, and the DPO step was therefore needed to further strengthen the rest of the skills. Our experimentation showed that using RLVR after DPO is superior to using DPO after RLVR. For future work we could explore getting rid of the DPO step and instead only use RL (RLHF + RLVR) and we could experiment with the order of the pipeline too! The phrasing specifically of “Many current post-training scheme proceed from SFT to GRPO or directly start with GRPO.” is for reasoning models such as DeepSeek R1 or OpenAI’s o3. The training recipe introduced in this work and resulting models are designed for generalist models such as DeepSeek V3 V3 or GPT 4o. We plan to explore that style of training in future work.