We are appreciative of the reviewer's helpful comments.

Comment: For the right $\lambda$, QPO can closely match EXPO ... results of Figure 4 support this ... so how beneficial is EXPO in real-world cases with tuned $\lambda$.

Response: Actually, there are no values of the QPO hyperparameter that can simultaneously match EXPO in both the good and bad input prompt regimes as depicted. For example, consider IPO: for the good cases $\log \lambda > 0$ is optimal, while for the bad cases $\log \lambda \approx -1$ is optimal. We provide another illustration of this point at this link; only change is adjusting $\pi^*$. And overall, we regard reduced sensitivity to $\lambda$ as a positive even in real-world scenarios, as hyperparameter tuning can be expensive.

Comment: Ambiguity of Figure 8 ... it looks like all methods converge to the optimal policy, not reference.

Response: Figures 7 and 8 are meant to be viewed in tandem (and we can add explicit text to better convey their relationship). From Figure 7, when $\lambda$ is large we observe training convergence to $\pi_{ref}$ values (0.2 and 0.4). Meanwhile in Figure 8 we show fully converged solutions for a range of $\lambda$ values varying from small to large. For large $\lambda$ (right side of each plot) we observe that all models produce $\pi_{ref}$ as expected. Conversely, for small $\lambda$ (left side of each plot) we see that only EXPO maintains the optimal policy. Again, we can easily clarify this in a revised appendix.

Comment: What exactly about the QPO objectives is problematic and exactly how EXPO addresses the issue ... higher-level intuitive explanation would be helpful.

Response: Good suggestion. We agree that accessible explanations are valuable for conveying some of the subtle messages associated with our work. In this regard, the paragraph beginning on Line 172 (righthand column) may be helpful. There we provide a more intuitive perspective for the specific QPO cases of DPO and IPO; similar ideas hold more broadly, but are decidedly more involved to present, hence the technical aspects of our submission. There is also another natural entry-point for understanding where QPO objectives begin to fall short. For simplicity, consider only the special case of DPO, which is advertised as producing the minimal solution to equation (4) instantiated with an optimal reward. Next observe how (4) behaves as $\lambda \rightarrow 0$. Basically, once the KL term is ignored, the remaining loss will be trivially optimized by a degenerate policy assigning all probability mass to a single response; see equation (43) for the derivation. In contrast, the EXPO loss is explicitly designed to reflect the full optimal BT policy in this same limit (not just a mode). Appendix E.2 provides further details regarding why this distinction is important.

Comment: ... it would be beneficial to explicitly spell how IPO and DPO are expressed in terms of the QPO objective.

Response: Great suggestion, and this is easy to include in a revised appendix.

Comment: Given that the similarity in performance between the QPO and EXPO methods varies with lambda, it would be beneficial to show how the different lambda values impact performance on the real world tasks.

Response: As we have clarified in a previous response above, QPO and EXPO performance is quite distinct as $\lambda$ is varied across our synthetic experiments. And while we do agree that further ablations with real-world data would be interesting, such testing incurs significant time and computational cost that we unfortunately cannot accommodate. That being said, in Appendix B.3 we mention $\lambda$ stability for EXPO under a limited testing range.

Comment: It is not clear to me how Figure 3 addresses the interpolation goal ...

Response: Indeed Figure 3 only shows the interpolation extreme on the optimal policy side (i.e., when $\lambda$ is small). However, Figures 7 and 8 in Appendix B.2 complete the full picture, as we ran out of space in the main text. For reference, there is also a pointer on Line 346 (righthand column).

Comment: Good to add the distance between the reference and optimal policies to the "Good Cases" plot in Figure 4 ...

Response: By design, the distance between $\pi_{ref}$ and $\pi^*$ is zero in Figure 4, top plot. Please also see Line 355 (righthand side) in the text.

Comment: How well tuned is lambda for Anthropic-HH and IMDB experiments ... relationship with Figure 4.

Response: Please see our responses above regarding Figure 4, and the distinct performances of QPO vs EXPO once both good and bad cases are considered collectively. In terms of tuning $\lambda$ on real-world data, we only explored a few distinct values because of computational cost; see also Section B.3.

We appreciate the constructive comments, and address the main points as follows (grouping where appropriate).

Comment: First/Main limitation: I am not convinced by the first limitation of existing approaches. The considered setting seems a bit synthetic to me as it only considers two types of prompts: good and bad, where $\pi_{ref} = \pi^*$ for good prompts. However, it is very likely that $\pi_{ref}$ is not optimal for all the prompts in the datasets. A more realistic setting is that the authors can prove the proposed methods can improve over over all the prompts without requiring $\pi_{ref} = \pi^*$ for good prompts.

Response: We remark that it is commonplace for theoretical results to involve simplifying assumptions, otherwise many analytical steps become infeasible. That being said, in the present circumstance the point is not necessarily to reflect the most realistic scenario possible. Instead, the goal is to show that even in an idealized case DPO-like methods perform below expectations, the implication being that such subpar performance is unlikely to disappear even under broader conditions. We also reiterate that our assumption (for Theorem 3.1) is not that $\pi_{ref} = \pi^*$ for all prompts, but rather, only for so-called ideal good cases.

Comment: Second limitation: The synthetic evaluations support the limitation claim and the advantage of the proposed method over the existing method. However, more evidences are needed to demonstrate using the real-world dataset. For example, to address the second limitation. the authors can show the diversity of the proposed methods over existing methods while ensuring good generation quality.

Response: As demonstrated in Figure 5 of the submission, our proposed method achieves high-quality generation on multiple real-world datasets. However, to quickly explore diversity per the reviewer's suggestion with limited time during the rebuttal period, we evaluate using the basic setup, sampling method and metrics from Section 5.3 of this paper (although some experimental details, like max_token are not reported in the paper; for these we adopt our settings). Results are shown in the new table below, where generally EXPO displays higher diversity relative to DPO. We also normalize the entropy by token length to avoid bias towards longer responses; other metrics are already implicitly normalized w.r.t. length.

We thank the reviewer for pointing out that our paper is clear, based on convincing evidence, and supported by interesting and relevant theory. The reviewer also provided many constructive comments; with limited space to reply, we prioritize the main critiques as follows:

Comment: Methodology is sound, but warrants testing on larger scale models, e.g., in 8B parameter range as opposed to 2.8B range.

Response: We agree with the reviewer that the community is trending towards experimentation with larger models, particularly on less theoretically driven papers. But even so, many recent papers still lean heavily on 2.8B-sized models or smaller for analyzing preference methods and drawing comparative conclusions between approaches. As representative examples, the DeepSeek GRPO paper [14, Figure 5] uses a 1.3B model to compare online vs offline preference learning; likewise 1.4B-2.8B models are used for a similar purpose in [9, Table 1] and [13, Figure 3]. We also note that other more theoretical works such as the IPO paper (Azar et al.,2024) contain no real-world experiments at all and yet remain highly influential to the field. In any event, we are not disputing the value of larger models; instead we are merely advocating that high quality contributions are still possible without them, especially when granting that compute resources and timing constraints are often limiting factors (as is the case for us within the rebuttal period).

Even so, during the rebuttal window we tried fine-tuning a Llama-3.1-8B model using Lora to reduce the computational burden, and indeed EXPO is better than DPO in this limited setting. However, complete testing with further baselines or a full parameterization is unfortunately not feasible at this time.

Comment: Recent evidence suggests that online preference learning generally performs better ... so relevance of offline approaches may be waning.

Response: This is a very reasonable point to consider, and addressing it helps to further advance the relevance of our paper. In this regard, the suggested references from the reviewer are extremely helpful. Our overall response is three-fold:

1. The inference (from references [5,6,12,13,14] and others) that online preference learning is generally superior is to date mostly derived based on testing with QPO approaches, and usually DPO alone (or in the case of [6], an even earlier rejection-sampling approach). Hence the open possibility remains that offline approaches that mitigate DPO deficiencies (like our EXPO) might reset the scales relative to online alternatives. And crucially, many of the specific arguments provided for why DPO is outperformed by online approaches like PPO do not apply to our EXPO. For example, in [5] it is argued that a key DPO limitation is that it does not exploit unlabeled prompt-only data, which can then lead to undesirable biases. However, as we point on on Lines 322-324, EXPO can naturally incorporate such unlabeled prompt-only data.

2. Many references that argue in favor of online preference learning nonetheless suggest that DPO is still valuable when reformulated as an online or related on-policy iterative approach; see for example [6,9,10,11,13]. Some (e.g., [11]) even explicitly advocate for generalizing beyond DPO to online versions of IPO and the like. Hence our analysis of QPO models in general, and EXPO in particular, remains quite relevant in a wide variety of revised online settings.

3. Even if we concede that offline preference learning is sub-optimal, in many scenarios it is still considerably simpler, possibly even with convergence guarantees. As such, for prototyping or resource-constrained environments offline learning may enjoy some durable advantages.

Comment: Relation to literature, missing references ... [10] should be cited and discussed, but it is not. [11] should also be cited ...

Response: References [10] and [11] are empirically-driven studies that advocate strongly for iterative versions of DPO, e.g., exploiting DPO implicit rewards over multiple on-policy rounds. See also our comments above that include reference to [10,11]. Overall, these works are quite complementary to our own, and we can definitely cite them in our revision. Likewise for other references the reviewer has kindly provided.

Comment: Questions 1-5 ...

Response:

1. Memory consumption is the same as DPO.
2. Using unlabeled samples for (16) may help some, but further study is needed.
3. SIC and WIC differ strongly as $\lambda$ becomes small. For SIC the minimal loss equates to an optimal preference distribution, while for WIC we only obtain the mode of an optimal distribution. Appendix E.2 contains further details.
4. We have not tested this (and it could depend on the implementation), but intuition suggests that EXPO may still provide a benefit in some cases.
5. Multi-turn extensions of EXPO represent an interesting direction for future work.

Thanks for checking many of the technical details of our paper, including proof and empirical materials, while acknowledging the novelty of our approach in addressing underappreciated limitations in prior preference optimization methods. We respond to points of critique as follows.

Comment: Limited experiment results on real world tasks. Would like to see its performance on Llama-3-8B on AlpacaEval 2.

Response: During the short rebuttal window, we tried fine-tuning a Llama-3.1-8B model using Lora to reduce the computational burden, and indeed EXPO is better than DPO in this limited setting. However, complete testing with further baselines or a full parameterization is unfortunately not feasible at this time.

Comment: The same algorithm name ExPO has already been used in "Weak-to-Strong Extrapolation Expedites Alignment".

Response: Thanks for the notice, we could easily change the name to avoid any confusion.

Comment: Can EXPO handle non-BT preference models?

Response: Good question. In principle yes, but some theoretical results would need to be reconsidered.