Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

We thank the reviewer for their valuable comments and respond to the weaknesses and questions below.

While the algorithm claims faster convergence, given the small T=3~5 in practice, it's unclear whether the theoretical analysis provides meaningful guidance for real applications (since these are just bounds).

We would like to clarify that our theoretical bounds for both OMD and optimistic OMD share the same constant, and the improvement remains notable even for small values of $T$. For instance, at $T = 5$, the convergence rate improves from $1/\sqrt{5}$ to $1/5$, yielding a 2× gain. While this improvement becomes more pronounced as $T$ increases, it is common in the RLHF and LLM alignment literature [Xie et al., 2024; Wu et al., 2024; Zhang et al., 2024] to develop theoretical analyses for relatively large $T$, even though practical experiments often use small values such as 3 or 5.

The paper may need to supplement comparisons under equal computational budgets, as the algorithm's T=3~ 5 might correspond to baseline methods' T=6~8, suggesting that despite theoretically faster convergence, it may not be superior under limited computational resources.

We would like to clarify our online training setup, which follows from prior LLM alignment literature. In each iteration $t$, we construct the preference dataset $D_t$ by sampling a new set of prompts and generating responses using the current policy. Crucially, the prompt set changes across iterations. To ensure a fair comparison, we run all methods for the same number of iterations, ensuring that they are trained on the same set of prompts.

Regarding computational cost, all baselines are trained for two epochs per iteration. In the Mistral experiments, ONPO uses the same total number of training epochs, one for $\pi'\_{t+1}$ and one for $\pi_{t+1}$, resulting in the same computational cost. In the LLaMA experiments, we use two epochs for $\pi'\_{t+1}$ and one for $\pi_{t+1}$, adding just one extra epoch per iteration. However, the majority of computation time per iteration (approximately 3–4 hours) is spent on generating the preference dataset, and the additional epoch only adds about 1.5 hours, which is relatively minor. Moreover, we experimented with training the baselines for three epochs per iteration but did not observe performance improvement.

The proposed method requires maintaining and training two models simultaneously, introducing additional computational overhead.

We would like to clarify that $\pi'\_{t+1}$ and $\pi_{t+1}$ are trained sequentially rather than simultaneously. After constructing the preference dataset $D_t$, we first minimize $g_t$ to obtain $\pi'\_{t+1}$, and then minimize $g_{t+1}$ to obtain $\pi_{t+1}$. The log ratio of $\pi'\_{t+1}$ in the objective $g_{t+1}$ is fixed and precomputed. As a result, there is no need to maintain both models in memory at the same time, and the training does not introduce additional memory overhead from simultaneously handling two models.

Results where the baseline outperforms ONPO are underlined." This method of presenting results in Table 1 can easily lead to misunderstandings.

Thank you for pointing this out. We agree that the current presentation could be misleading and will revise it in the next version.

Is it possible to implement the algorithm at the minibatch level? This could potentially lead to larger T, even if it introduces increased noise.

We assume the reviewer is referring to using only a minibatch of prompts in each iteration to train the model. While this approach could reduce the computational cost per iteration and allow for a larger $T$, a small set of prompts may not provide sufficient training signal to accurately obtain $\pi_{t+1}$, which serves as the target policy in mirror descent. Our current online training setup follows prior RLHF literature, which typically uses a large batch of prompts per iteration. That said, exploring a minibatch-level implementation is an interesting direction for future work.

We thank the reviewer for their valuable comments and respond to the weaknesses and questions below.

Lack of Statistical Significance Reporting

We acknowledge the importance of statistical significance as raised by the reviewer. However, in contrast to traditional machine learning, LLM experiments are extremely resource-intensive, making statistical significance reporting relatively rare in the LLM community. This is not only the case in academic work, including closely related papers such as Iterative DPO [Dong et al., 2024], SPPO [Wu et al., 2024], and INPO [Zhang et al., 2024], but also in large-scale technical reports such as the Qwen3 Technical Report [Yang et al., 2025]. Here we report standard deviations on the AlpacaEval 2.0 benchmark, one of the most widely used and representative alignment benchmarks:

As shown, the standard deviations are relatively small and ONPO consistently outperforms all baselines by a statistically meaningful margin.

Missing Empirical Comparison with Closely Related Methods

We address the reviewer’s concern by discussing each method individually:

Nash-MD. The Nash-MD algorithm cannot be directly implemented in practice for two main reasons: (1) computing the geometric mixture policy is computationally intractable, and (2) estimating the expected win rate between two policies is also challenging in practice. As a result, practical implementations of Nash-MD require heuristic approximations. At the time of our project, we did not find the official implementation, and prior related works did not include comparisons with Nash-MD.

During the rebuttal period, we made our best effort to implement a variant of Nash-MD using the TRL trainer. Since our setting provides only binary preference signals rather than scalar win probabilities, we approximated the win rate by assigning a value of 1 to the preferred response and 0 to the other. The results on LLaMA3-8B-SFT are shown below.

As shown in the table, ONPO consistently outperforms Nash-MD across all three evaluation benchmarks.

DNO. As stated in lines 224–232, the original DNO algorithm is not directly implementable in practice. Rosset et al. [2024] proposed a practical variant, DNO-Prct, which replaces the intractable update with the DPO objective. As a result, DNO-Prct can be interpreted as an iterative version of DPO and our experiments already include a comparison with Iterative DPO.

Online IPO. There are two possible ways to implement online IPO. One is to apply the same loss transformation technique used in IPO and INPO; in this case, online IPO reduces to INPO without the KL stability term, which has been shown to underperform compared to INPO, see Table 3 in Zhang et al. [2024]. The other option is to use policy gradient methods to directly optimize the objective. However, as discussed in lines 215–223, policy gradient methods often suffer from high variance and can lead to unstable learning. Furthermore, we could not find any official policy gradient implementation of online IPO. We believe this is also why prior works have not included it in empirical comparisons.

Taken together, we would like to emphasize that we have already compared ONPO with state-of-the-art preference alignment methods, including SPPO and INPO, demonstrating the effectiveness of our approach.

We thank the reviewer for their valuable comments and respond to the weaknesses and questions below.

While performance improvements against other models have been established, they have been done so for models of size 8B/7B. the question as to whether the same performance improvement would be had if the model sizes are scaled up to larger sizes remains open.

Due to limited computational resources, our experiments are conducted on 7B/8B base models. This limitation is common in the LLM alignment literature, where experiments on larger-scale models are rarely presented. We acknowledge that it remains an open question whether the same performance gains hold at larger model scales, and we plan to explore this direction in future work as resources allow.

Their is an assumption of a preference oracle here. Is that a realistic assumption? It appears not to be the case but I am open to the authors explaining why it might be.

We would like to clarify that the preference oracle is not an assumption but a general formulation of the underlying human preferences we aim to align with. Specifically, given a prompt $x$ and two responses $y_1$ and $y_2$, the oracle gives the probability $\mathbb{P}(y_1 \succ y_2 \mid x)$ that humans would prefer $y_1$ over $y_2$. This offers a mathematical abstraction of the process that humans generate preference labels.

Commonly used models such as the Bradley–Terry (BT) model is a restrictive special case of the general preference formulation, where $\mathbb{P}(y_1 \succ y_2 \mid x)$ is assumed to follow a specific parametric form, e.g., the sigmoid of the reward difference $R(x, y_1) - R(x, y_2)$. In contrast, we only assume access to a binary preference signal sampled from $\mathrm{Ber}(\mathbb{P}(y_1 \succ y_2 \mid x))$. As long as we are comfortable with modeling the human feedback as a probability distribution, this is the minimal assumption. This binary signal plays a similar role to labels in supervised learning, it is essential for model training, and all LLM alignment methods rely on such preference data for supervision.

The self play here is done for a single turn. While I am not very familiar with the practical aspects of RLHF this seems significant.

As discussed in Appendix B, our method can be extended to the multi-turn setting. However, implementing the multi-turn setting would necessitate using deep reinforcement learning techniques and policy gradient methods, which are known to be computationally expensive and unstable during training. Our primary goal in this work is to develop a computationally lightweight method and focus on the single-turn setting, leaving the implementation of the multi-turn extension as future work.

How much more computational resources would be needed for a multi turn setup?

We do not have precise estimates for the additional computational resources required for a multi-turn setup. However, based on our experience from related projects involving policy gradient methods for post-training large language models, achieving satisfactory performance typically requires 10 or more epochs. In contrast, our current method uses only 2–3 epochs per iteration.

We thank the reviewer for their valuable comments and respond to the weaknesses and questions below.

Only compares against DPO and 2 minor methods. Other methods in this area (SimPO, ORPO, R-DPO, RL-based methods etc) are not explored.

We would like to clarify that the two baselines we compare against, SPPO and INPO, are state-of-the-art approaches for online preference alignment.

The methods mentioned by the reviewer are designed for the offline alignment setting, where the algorithm optimizes a learning objective over a fixed, pre-collected dataset. In contrast, our work focuses on the online setting, where no response pairs are pre-collected and the model must iteratively generate responses and improve itself based on collected preferences. As stated on line 263, prior work has shown that online or iterative alignment methods can outperform their offline counterparts [Guo et al., 2024; Dong et al., 2024]. For this reason, we compare ONPO with other online methods to ensure a fair and meaningful evaluation.

Among the offline methods, SimPO has demonstrated strong performance. To further strengthen our comparison, we implemented an iterative variant of SimPO, where the current model generates responses and optimizes the SimPO objective at each iteration. As shown below, ONPO significantly outperforms it across all three evaluation benchmarks:

No discussion of experimentation setup. Hardware, training time, codebase, etc. Makes it hard to assess practical trade-offs between methods.

Details of our experimental setup are provided in Section 5.1 and Appendix D. As noted on line 564, all experiments were conducted on 8×A100 GPUs with 40GB memory each.

We apologize for not reporting the actual training time in the main text. All methods follow the same online training setup and differ only in their learning objectives. In each iteration, approximately 3–4 hours are spent generating the preference dataset. Each baseline is then trained for two epochs, which takes around 3 hours. In the Mistral experiments, ONPO uses the same total number of training epochs per iteration, one for $\pi'\_{t+1}$ and one for $\pi_{t+1}$, resulting in equivalent computational cost. In the LLaMA experiments, we use two epochs for $\pi'\_{t+1}$ and one for $\pi_{t+1}$, introducing only one additional epoch (~1.5 hours). We also tested training the baselines with three epochs per iteration and did not observe performance improvements.

Our codebase is adapted from Dong et al. [2024], and all baselines are implemented following their official code.

Method assumes access to a general preference oracle (possibly learned), but does not consider the effect of noisy/imperfect oracle feedback. Since human preferences can be inconsistent, this is a critical.

We want to first emphasize that our preference oracle is a general formulation of the underlying human preferences we aim to align with. It is not “learned” as the reviewer suggested, since the oracle is simply a mathematical abstraction of the process that human generates preference labels. Specifically, given a prompt $x$ and two responses $y_1$ and $y_2$, the oracle gives the probability $\mathbb{P}(y_1 \succ y_2 \mid x)$ that humans would prefer $y_1$ over $y_2$.

It is unclear to us what the reviewer means by “inconsistent”. The reviewer might be thinking of properties of commonly used models such as the Bradley-Terry (BT) model, which assumes that $\mathbb{P}(y_1 \succ y_2 \mid x)$ follows a sigmoid function of some underlying reward/utility function $R$. In this case, a real human feedback may indeed be “inconsistent” with the BT model in that the feedback may not be consistent with any reward function. In contrast, we only assume access to a binary preference signal sampled from $\mathrm{Ber}(\mathbb{P}(y_1 \succ y_2 \mid x))$. As long as we are comfortable with modeling the human feedback as a probability distribution, this is really the minimal assumption. This binary signal plays a similar role to labels in supervised learning, it is essential for model training, and all LLM alignment methods rely on such preference data for supervision.

We acknowledge that modeling and mitigating the effect of noise in preference signals is an important direction for future work. In our case, stochastic noise will simply change the function $\mathbb{P}(y_1 \succ y_2 \mid x)$, and our algorithm will still align with the preference function modified by noise. How to align with the “original” preference is analogous to handling noisy or corrupted labels in supervised learning. However, we consider this direction to be orthogonal to our primary focus, which is to develop a more effective alignment method with provably faster convergence. We will include a discussion of this point in the revised version.

Even without full implementation, demonstrating ONPO on multi-turn with toy examples would help validate its broader application and scope.

We appreciate the suggestion. However, the core challenge of the LLM alignment problem lies in enabling large models to learn complex human preferences, which are difficult to meaningfully represent in toy examples. As discussed in Appendix B, extending our method to the multi-turn setting would require incorporating deep reinforcement learning and policy gradient techniques, which are computationally expensive and often unstable. Given our focus on developing a computationally lightweight method, we chose to concentrate on the single-turn setting in this work and leave the implementation and evaluation of the multi-turn extension to future work.

Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024.

Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024.