Design Considerations in Offline Preference-based RL

We thank you for your encouraging feedback on our work. We address your key questions below.

1. Relaxing Assumption 3.1: You are correct in noting that having an exponential dependence on the size of log-ratios is not ideal. But this limitation is shared by a line of prior works [1], [2], [3]. While it would be desirable to weaken this dependence, we also note that the degradation of learning quality as the log probability ratios grow large during learning is quite clear in our experiments. As we see in Figures 1 and 2, the drop of log probabilities under the learned policy coincides with degradation in the performance of DPO. Since the reference policy has well-behaved log probabilities, the probability ratios look nearly identical to the learned policy’s log probabilities here. So we also think that the bounds capture a phenomenon actually realized in the experiments qualitatively, even though we might be able to improve the precise functional forms from being milder than exponential in future work.

2. Generating model underlying squared loss: We thank you for the insightful question about the generating model underlying the squared loss. We refer you to Appendix A, which provides this discussion for general link functions, as well as gives the specific preference generation model for squared loss in lines 591-592: $P(\omega=1|x, y, y’) = \frac{1}{2} + \frac{R^\star(x, y) - R^\star(x, y’)}{2}$.

We thank you again for your thoughtful feedback, and will add these discussions to the final version of the paper.

We thank you for your thoughtful feedback on our work. You are right in noting that the main contributions of our work are theoretical, and not in proposing new methods. We address your main concerns with our theoretical results below.

Squared loss and realizability:. You are correct that it is not the case that the squared loss is a clear win across all the theoretical criteria, and makes a less natural realizability assumption. We note that we can use standard arguments to extend our analysis to the case where the realizability assumption holds uniformly, up to an error $\epsilon$. We did not include this for clarity of exposition in the submission, but can easily add this to the final version. More generally, our view is that realizability is an unverifiable assumption in practice, but is perhaps more likely to hold for rich model classes such as LLMs. At the same time, the curvature of the loss is something we can control better. So an approach that primarily favors the squared loss as a default objective based on the curvature properties is reasonable practically, since it will yield good results, unless realizability is violated, in which case a practitioner can switch to other losses, such as logistic. This tradeoff between the intuitive modeling niceness versus optimization and learning tractability is often encountered in the literature on generalized linear models as well, and we will expand on this discussion in the final submission.

Choice of loss function affects benchmark policy: You are also correct that the choice of the base policy affects the benchmark policy. As we note on line 161 (right), the most natural benchmark, which would be independent of these choices, cannot be recovered by the class of offline methods including DPO and variants which are studied here. That said, the base policy is a part of the loss function. It is quite standard even in the literature on simpler classification and regression setups that when the loss changes, the convergence point changes accordingly, and we can only analyze convergence to the optimal solution according to the chosen loss. Indeed, as we notice in Remark 3.8, the choice of base policy has no explicit role in the convergence bound, but only plays a more subtle role in influencing the realizability assumption and the benchmark policy. We think it is still useful to uncover the different points of convergence that different choices used in practice correspond to, so that we can prefer methods which correspond to desirable points of convergence.

Gap between the performance measures in theory and experiments: We agree that it would have been great if we could just have a similar KL bound or an easy surrogate in our experiments as well. Unfortunately, without access to the unknown benchmark policy, this is an impossible task. Consequently, we measure the most natural quantity that we could think of empirically, and that is also consistent with the broader literature. While not perfect, we believe this is still insightful. If the reviewer has additional recommendations for performance measures that are measurable, and that would narrow the theory-practice gap, we would be happy to include them in the final submission.

Notation: We apologize for the inconsistency regarding the notation for loss, we will fix it. The notation $\Pi_{\mathcal{R}}$ is a typo from an earlier version. It should just be $\Pi$.

Thanks again for your insightful comments. We hope that our responses address some of your concerns, and we would be happy to discuss further if you have additional feedback.

We thank you for your encouraging and thoughtful comments.

Severity of restrictions due to Assumption 3.5: You are correct in noting that Assumption 3.5 places some restrictions on the loss function, but we believe that these restrictions are relatively mild and standard as we discuss below:

1. Please note that the curvature condition is not pointwise (i.e. per x, y, y’ triple), but only in expectation, and not uniform, but only local around the optimal policy $\pi^\star$. So the required assumption is local curvature of the expected loss around the optimal solution, which is a very common assumption in optimization and convergence theory.
2. The curvature assumption, while not explicitly stated, is responsible for the e^B term in almost all prior works on analysis within the Bradley-Terry-Luce model with the logistic loss, such as [1], [2] and [3]. Please also see Remark 4.9 in [1] for an explicit mention of this aspect. In fact, even in simpler settings such as analysis of generalized linear models, a similar curvature assumption can be seen in [4], Assumption 1.

While we agree that weakening such conditions further would be highly desirable, addressing this goes well beyond the study of preference-based learning methods, and would be a significant undertaking to be carried out in a separate work.

Differences to analysis in IPO paper: We apologize for missing the discussion of differences from the analysis in the IPO paper in the submission. The authors in IPO indeed note the issue of the curvature in the link function, and use this to motivate the IPO algorithm. However, there is no explicit quantitative analysis of the role of the link function beyond a simple example, since the focus is more on obtaining a practical algorithm for the identity link function case. We instead derive precise convergence guarantees, and also highlight the role of offline data coverage which is not captured in the IPO paper.

We thank you again for the insightful comments, and we will update the final submission with the discussions above, as well as address your other suggestions.

References:

• [1] Sharp Analysis for KL-Regularized Contextual Bandits and RLHF, arXiv 2411.04625.
• [2] Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons, ICML 2023.
• [3] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint, ICML 2024.
• [4] Parametric Bandits: The Generalized Linear Case, NeurIPS 2010.

Thank you for the encouraging feedback. We will add the suggested references [1, 2] to our discussion.