SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

We sincerely thank the reviewer for the insightful comments and suggestions. We appreciate that the reviewer finds our work simple and has a strong potential to be adopted in practice. We are also encouraged that the reviewer finds our paper “clearly written”.

It could've been nicer if it was discussed more prominently in earlier literature review sections. Currently it is only discussed in Section 5.1.

We thank the reviewer for relating our work to the broader RL literature. We will discuss our relation with existing work in RL that is not about language model alignment earlier in our updated manuscript, and extend our discussion in the appendix.

Methods like Direct Nash Optimization (Rosset et al 2024 https://arxiv.org/abs/2404.03715) do mix samples from online policy and offline policy together, hence SimpleMix is not the only paper which mixes online and offline policy samples. DNO could've been an interesting baseline, at least a non-iterative version.

We thank the reviewer for bringing up the interesting discussion on DNO. Although the authors of DNO defined it as “a batched on-policy algorithm” (page 9 in [1]), DNO selects the best and worst response from N on-policy generations and one off-policy generation from gpt-4-turbo to perform DPO, therefore it can be seen as an hybrid method. As suggested by the reviewer, we conduct an experiment comparing SimpleMix with one iteration of DNO on Tulu-3.1-8B-SFT, where we sample N = 4 generations and mix it with the off-policy “chosen” generation from Ultrafeedback, and select the best and worst response according to our oracle reward model. We report the results below:

Alpaca Eval 2.0

Compared to the setting in [1], we changed the number of on-policy generations (5 -> 4), the off-policy model (gpt-4-turbo -> the chosen response in ultrafeedback), and the oracle reward (gpt-4-turbo -> Skywork/Skywork-Reward-Gemma-2-27B) to make the results comparable to our work. We will make sure to add discussions on DNO in our latest manuscript.

[1] Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences (Rosset et al., 2024)

We thank the reviewer for the thoughtful review and we would like to extend our gratitude for the reviewer finding our method straightforward and our experiment results clear. We hope to answer the questions and resolve the concerns of the reviewers below:

The idea itself isn’t very novel—just mixing two data sources is fairly standard.

• On its novelty: Our contribution is not limited to mixing of the two data sources, our work also 1) shows that the two data sources are complimentary (Section 3) - which provides a possible explanation for the performance improvement; 2) conduct detailed ablations (Section 5) on data mixtures and other data curation strategies. To the best of our knowledge, our work is the first to carefully study the interplay between on- and off-policy data in preference optimization.

• On simplicity: While we acknowledge that the method is simple, we believe this is our strength. SimpleMix does not introduce additional hyper-parameters. To put it in reviewer 9f8w’s words, “SIMPLEMIX method is a straightforward but effective way to improve language model alignment without additional computational overhead.”

The reported performance improvements are modest, raising doubts about real-world significance.

• Improvements: While the reported performance improvements may appear modest, they are averaged across eight benchmarks. As reviewer fktC denotes “The improvement is modest, but the consistency makes a convincing case.”
• Real-world significance Tulu 3 [1] is a concurrent effort that is similar to our setting and demonstrates a 3-point average improvement across benchmarks. This is a meaningful gain, especially considering that mixing the two data sources incurs minimal cost. As Reviewer fktC denotes, our method “has a strong potential to be adopted in practice.”

Can you explain why Tulu specifically benefits more from SIMPLEMIX? Is there something specific about Tulu’s training or data that makes mixing data sources especially useful here? In Figure 8, SIMPLEMIX shows very little improvement or sometimes no improvement over the baseline (pure on-policy or off-policy data) when applied to the LLaMA instruct-tuned model across all four task categories (math, coding, creative writing, recommendation). This again relates the Q1 above: it seems that mixing is not useful when the model is already aligned?

We aggregated the two questions here: we conjecture that Tulu-8B-SFT have not gone through a preference optimization process while Llama-3.1-8B-Instruct have already gone through extensive SFT and DPO training, thus making it easier to improve Tulu’s performance compared to improving Llama-3.1-8B-Instruct’s performance.

References: [1] Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024)

We sincerely thank the reviewer for the thoughtful comments and suggestions. We are thankful that the reviewer finds our method “straightforward but effective” and our results “demonstrate clear trends”. We hope to resolve the concerns below:

Benchmarks like Alpaca Eval 2.0 and Ifeval involve LLM-based evaluation, which can be biased or manipulated. Including more human evaluation results (e.g., actual user studies) would add credibility to the findings.

It is possible there may be some misunderstanding of the scope of our evaluation. In our work, we have chosen to report scores of nine benchmarks, only one of them is LLM-based evaluation (Alpaca Eval 2.0). The other 8 benchmarks include but are not limited to world knowledge acquisition (MMLU), commonsense reasoning (Hellaswag), precise instruction following (IFEval), open-domain question answering (OpenQA). We believe that this is a practical and comprehensive setting since the same benchmarks are also adopted by FineWeb [6] in pre-training data curation. Detailed descriptions of our evaluation benchmarks can be found at Appendix D. We acknowledge that adding real user studies would improve the credibility of our work. Since the cost of high quality human annotators is prohibitively large, we opted to use Alpaca Eval 2.0 due to it having a 0.95 correlation with real human annotators while only having 0.8% of the cost according to [7].

The paper lacks a theoretical foundation to explain why SIMPLEMIX works better than other hybrid approaches.

There exists a mismatch between theory of existing works and practice in LM alignment, and we conjecture that might be the reason why SIMPLEMIX works better:

• HyPO [1] proposes to perform an off-policy DPO while using on-policy data to minimize the KL divergence between the current policy ($\pi_\theta$) and the reference policy ($\pi_\text{SFT}$). The motivating assumption for adding the additional KL regularization is that the “DPO implicit reward” is unbounded and can lead to infinite KL. In practice, the literature has witnessed the opposite trend [2, 3, 4] where KL regularization might not be necessary for LM alignment (because of the reference model’s unreliability, leading to unreliable KL values). Our work combines on- and off-policy data without explicit regularization of the KL compared to HyPO, removing the strong KL regularization in HyPO might be a reason that SimpleMix works better.

• DPO-Mix-P [5] samples from an interpolation between $\pi_\theta$ (the current policy) and $\pi_\text{SFT}$ (reference policy). The theoretical foundation of DPO-Mix-P is designed for faster convergence in terms of DPO loss (achieving a lower DPO loss for less iterations). In practice, the recent works have shown that lower DPO loss doesn’t always go hand-in-hand with better alignment (“Alignment Gap” [6]). Therefore, achieving a lower DPO loss, or equivalently, a higher ranking accuracy, might not contribute to a better aligned model.

While the study identifies task-dependent benefits of on- and off-policy data, it does not explore task-specific weighting or adaptation. A potential extension could involve dynamically adjusting the data mixture ratio based on task type.

In section 3, we have shown the complementariness of on- and off-policy data on carefully selected subtopics in Alpaca Eval 2.0. However, in reality, user prompts rarely fall into a single category as many queries require both understanding the user’s personal preference, but also following verifiable constraints. For example, the query from UltraFeedback “explain cache keys to me like im 5” requires both objective knowledge about cache keys and adjustments of tone and style. Therefore, we decided to be safe and only combine two sources of data to achieve a balanced performance across all types of tasks. We agree with the reviewer that algorithmic adjustment of the sampling method (on- or off-policy) based on the prompt category would be a promising future direction.

References:

[1] The Importance of Online Data: Understanding Preference Fine-tuning via Coverage (Song et al., NeurIPS 2024)

[2] Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (Xu et al., ICML 2024)

[3] SimPO: Simple Preference Optimization with a Reference-Free Reward (Meng et al. NeurIPS 2024)

[4] DAPO: An Open-Source LLM Reinforcement Learning System at Scale (Yu et al., 2025)

[5] The Crucial Role of Samplers in Online Direct Preference Optimization (Shi et al., ICLR 2025)

[6] Preference Learning Algorithms Do Not Learn Preference Rankings (Chen et al., NeurIPS 2024)

[7] FineWeb: decanting the web for the finest text data at scale (Penedo et al., 2024)

[8] MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (Ni et al., NeurIPS 2024)