Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective

We thank the reviewer for the positive feedback and insightful suggestions! We address your comments as follows.

1. Methods And Evaluation Criteria & Experimental Designs Or Analyses

1.1 Comparison with other online algorithms

Regarding online RLHF methods, we first point out that other existing methods (XPO, IPO [1] etc) cannot handle transfer settings, hence they are not directly comparable. Besides, the core technique in empirical TPO is the "win rate-based source policy selection via UCB", which allow us to adapt to the best source RMs and switch back to normal online learning if no benefits in transfer, without prior knowledge on task quality. Therefore, we view other online methods (e.g. XPO) not as competing baselines, but rather as complementary approaches that can be enhanced with our transfer learning techniques.

To make it clear, in our next revision, we will replace “DPO” (line 11, Alg. 3) with $Alg_{PO}$, which serves as a placeholder for any Policy Optimization oracle (e.g. DPO, XPO, IPO etc). This change aligns with the usage of generic placeholder $Alg_{OL}$ in TPO (Alg. 1).

To support this point, similar to Table 1 in paper, below we report win rates comparisons when we instantiate $Alg_{PO}$ by optimizing XPO and IPO loss, respectively, keeping other settings the same as Sec. 6 (except using learning rate 1e-5 in IPO). These results demonstrate that our transfer learning techniques can be effectively combined with different policy learners, leading to consistent performance improvements. We believe this highlights the modularity and generality of our framework, and we thank the reviewer again for prompting this valuable point.

• Empirical TPO (Alg. 3) by replacing DPO (line 11) with XPO ||Without Transfer|Purely Exploit ROUGE|Purely Exploit T5-Large| |:-: |:-:|:-:|:-:| |Iter 1|52.3$\pm$1.1|53.4$\pm$0.8|50.2$\pm$0.3| |Iter 2|51.6$\pm$1.3|54.7$\pm$1.6|49.1$\pm$1.3| |Iter 3|52.2$\pm$1.6|53.8$\pm$2.9|49.2$\pm$1.1|
• Empirical TPO (Alg. 3) by replacing DPO (line 11) with IPO ||Without Transfer|Purely Exploit ROUGE|Purely Exploit T5-Large| |:-: |:-:|:-:|:-:| |Iter 1|52.3$\pm$1.0|50.4$\pm$1.6|49.9$\pm$0.4| |Iter 2|55.2$\pm$1.4|52.3$\pm$0.3|50.1$\pm$0.3| |Iter 3|55.3$\pm$1.1|51.8$\pm$0.5|50.3$\pm$0.5|

1.2 Additional benchmarks

We consider summarization task for the following reasons:

1. Summarization task is important and it is also widely used in RLHF [2, 3].
2. Summarization task is quite suitable for our reward model transfer setup. There are various choices for additional reward models with different qualities, such as similarity scores (ROUGE, BERTScore) with human expert summaries, advanced LLMs, etc.

We believe it is an interesting direction to evaluate our algorithms with other LLMs and benchmarks (e.g. those in [4] as suggested). Given that our experiments already effectively demonstrate the advantage of our proposed approach and due to limited computational resources, we leave further evaluation to the future works.

2. Questions For Authors

2.1 Why not estimate policy value directly

The main reason is that under the Bradley-Terry assumption, the preference distribution $\mathbb{P}_{r^*}(\cdot|s,a,\tilde{a}) = \sigma(r^*(s,a)-r^*(s,\tilde{a}))$ is invariant under the transformation that $r^*(s,a)\rightarrow r^*(s,a)+b(s)$, where $b$ is an arbitrary state-dependent function. As a result, we can at best identify the true reward $r^*(s,a)$ up to a state-dependent shift, which can largely bias the policy evaluation and make it unreliable.

In contrast, introducing $J_\beta(\pi_{\text{ref}})$ as the baseline can cancel out the bias term. This enables consistent estimation for the value difference $J_\beta(\pi^*_{r^w})-J_\beta(\pi_{\text{ref}})$.

2.2 Clarification on the second term in Eq. (8)

Firstly, notice that the second term of Eq. (8) takes the minimum over $\sum_w1/\Delta(w)$ and $\sqrt{Wt}$. When $\Delta(w)$ is very small, $\sqrt{Wt}$ avoids the upper bound being extremely large.

Secondly, as in most regret bounds in online learning, our result characterizes the worst-case behavior. Analogous to multi-armed bandit (MAB) settings, for fixed $T$, in the worst case we may have $\Delta(w)=\sqrt{T/W}$, resulting in regret that matches our bound.

Moreover, there is a simple refinement can be done: additionally taking the minimum over the current bound and $\sum_w\Delta(w)N(w,T)=O(\sum_w\Delta(w) T)$, where $N(w,T)$ denotes the number of times we transfer from model $r^w$ in $T$ iterations. This may align more closely with the reviewer’s intuition. In fact, this basic regret bound was actually the starting point of deriving Thm. 4.3.

[1] A General Theoretical Paradigm to Understand Learning from Human Preferences

[2] Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning.

[3] BOND: Aligning LLMs with Best-of-N Distillation

[4] Reinforcement learning from human feedback with active queries

We thank the reviewer for the positive feedback and constructive suggestions! We address your specific comments in the following.

1. Claims And Evidence

1.1 About Theorem 3.2

As correctly pointed out, the sub-optimality gap is $\tilde{O}(T^{-1/2} + Cov(\Pi) T^{-1})$, and that’s why we claim the suboptimality/regret does not depend on complexity measure after finite time. As long as $T \geq \Omega(Cov(\Pi)^2)$, we have $\tilde{O}(T^{-1/2} + Cov(\Pi) T^{-1}) = \tilde{O}(T^{-1/2})$, and $T$ becomes the only dominating term. We will make this point clear in our revision.

1.2 A few rigorousness Issue

Thanks for pointing them out! We will revise our statements to improve their rigor as recommended.

2. Theoretical Claims

Firstly, notice that the second term of Eq.(8) takes the minimum over $\sum_w 1/\Delta(w)$ and $\sqrt{Wt}$. When $\Delta(w)$ is very small, $\sqrt{Wt}$ avoids the upper bound being extremely large.

Secondly, as in most regret bounds in online learning, our result characterizes the worst-case behavior. Analogous to multi-armed bandit (MAB) settings, for fixed $T$, in the worst case we may have $\Delta(w) = \sqrt{T/W}$, resulting in regret that matches our bound.

Moreover, there is a simple refinement can be done: additionally taking the minimum over the current bound and $\sum_w \Delta(w) N(w,T) = O(\sum_w \Delta(w) T)$, where $N(w,T)$ denotes the number of times we transfer from model $\pi^*_{r^w}$ in $T$ iterations. This may align more closely with the reviewer’s intuition. In fact, this basic regret bound was the starting point of deriving Theorem 4.3.

3. Experimental Designs Or Analyses

Thanks for raising this valid point. Despite difference in design, both TPO and its empirical version are grounded in the same core theoretical insight—transfer from policy with better coverage for $\pi^{\*}\_{r^{\*}}$ (i.e. low $Cov^{\pi^{\*}\_{r^{\*}}|.}$). The value estimation is preferable from a theoretical standpoint, but it is often computationally expensive in practice. To address this, our empirical TPO uses win rate as a proxy, because it is more scalable and characterizes the lower bound for $Cov^{\pi^{\*}\_{r^{\*}}|.}$ (Lem. 5.1). We view this as a reasonable trade-off between theoretical rigor and empirical scalability.

4. Other Comments Or Suggestions

Could we leverage...while achieving an improved regret bound?

This is a very interesting question. We conjecture it is possible to design more elegant algorithms with better regret bound, and we believe it is a highly valuable direction. We hope our work can serve as an initial step and inspire future developments, and we leave this for future work.

We thank the reviewer for the feedback. It seems there may be some misunderstandings regarding our setting and several our key claims. To clarify, we start with a general remark, followed by detailed point-to-point responses. We hope our replies improve the clarity of our submission and help the reviewer evaluate our paper.

General Remark for Clarification

As stated in Sec. 1&2 and Fig. 1, we target at improving sample efficiency in online RLHF by transferring from imperfect source RMs, i.e. learning $\pi^\*_{r^\*}$ from online human feedbacks associated with $r^\*$, while leveraging auxiliary RMs $r^1, \dots, r^W$.

• Our theoretical method TPO (Alg 1&2) is not offline but instead an online reward transfer algorithm, and we just use an offline subroutine (RPO) to compute a transfer candidate (Lines 6–7 in Alg. 2). Besides, we did not claim Thm. 3.2 is a contribution for offline literature. Instead, we claim it improves the existing convergence rate of online RLHF methods, which motivates "self-transfer learning". Our core theoretical contribution lies in analyzing the benefits of transfer learning in online RLHF. As stated in Thm. 4.3 and Sec. 4.2, TPO improves the online RLHF results given good source RMs and self-transfer learning.
• Empirical TPO (Alg. 3) is closely aligned with the theoretical insights behind TPO—transfer from policy with better coverage for $\pi^{\*}\_{r{\*}}$ (i.e., low $Cov^{\pi^{\*}\_{r^{\*}}|.}$). TPO follows it by selecting policy with its value gap (upper bound of $Cov^{\pi^*_{r^*}|.}$ by Lem. 3.1), while empirical TPO utilizes win rates (lower bound of $Cov^{\pi^*_{r^*}|\cdot}$ by Lem. 5.1, but more scalable).

1. Claims and Evidence

Theorem 3.2...violates the authors' claim.

In Thm 3.2, we only omit $\log|\Pi|$—the log covering number. We apologize that our wording is not precise enough, and will revise to "no dependence…up to log-covering number factors". However, such log terms are standard even in supervised learning, and removing any policy coverage terms in previous online RLHF bounds is a significant improvement.

We also clarify that Thm. 3.2 is about convergence rate, not regret bound, and it is only part of the contributions.

The method...not offline…not online…

We apologize for the confusion. We study reward transfer in online setting, and the term "offline" appears in paper just because we use an offline method (RPO) to compute a policy. We will reword "offline" to "distilled" to avoid confusion.

the contribution...to…RPO and XPO is not clear

We reply to it together in point 2 below.

2. Methods And Evaluation Criteria & Experimental Designs Or Analyses

Given that we study the online setting, offline methods (e.g. RPO) operate under different assumptions and objectives and are naturally not comparable.

Regarding online RLHF methods, note that other existing methods (XPO, IPO, etc) cannot handle transfer settings, hence they are not directly comparable. Besides, the core technique in empirical TPO is the "win rate-based source policy selection via UCB", which adapts to the best source RMs and switches back to normal online learning if no benefits in transfer, without prior knowledge on task quality. Therefore, we view other online methods (e.g. XPO, IPO) not as competing baselines, but rather as complementary approaches that can be enhanced with our transfer learning techniques. For limited space, we refer to "1.1 Comparison with other online algorithms" in our response to Reviewer yCvD for additional discussion and experimental support.

Our main goal in experiments is to show the effectiveness of empirical TPO, and our experiment results clearly demonstrate these. Besides, win rate is a standard metric in evaluating RLHF methods. We note that accuracy is not typically used as a standard metric for summarization tasks.

3. Theoretical Claims

​​Removal of...in the offline learning literature is not a contribution…

There is a misinterpretation of our contribution. We study the online setting and our main contribution is to improve sample efficiency in online RLHF by transfer learning.

...bounded policy ratio is very restrictive…

Such an assumption is standard in online RLHF literature [1,2]. Besides, as stated in Footnote 1 (Page 3), it is not essentially an assumption, but an additional preprocessing step given a realizable policy class, because $r^* \in [0, R]$ implies $||\log \pi^*_{r^*}/\pi_{ref}||_\infty \leq R/\beta$.

4. Other Strengths And Weaknesses

...empirical algorithm, differs significantly from the main algorithm…the source selection algorithm is totally changed

We respectfully disagree with it. Both our TPO and empirical TPO share the same insight (see second bullet in general remark).

[1] Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf

[2] Self-Exploring Language Models: Active Preference Elicitation for Online Alignment

We thank the reviewer for the positive feedback and constructive suggestions! We address your specific comments in the following.

1. Claims And Evidence

1.1 About Claim in Thm. 3.2

Thank you for the suggestion. We will follow it and clarify our claim in our next revision.

...why the second phase is also independent of $\beta$?

As predicted by the offline learning theory, the offline policy converges to $\pi^*_{r^*}$ at a rate of $\tilde{O}(Cov^{\pi^*_{r^*}|\pi_{mix}^T} T^{-1/2})$, where $\pi_{mix}^T := \frac{1}{T}\sum_{t=1}^T \pi^t$ is the uniform mixture of the policies collecting data. Technically, in the second phase, $\pi_{mix}^T$ is very close to $\pi^*_{r^*}$, and as implied by Lem. 3.1, $Cov^{\pi^*_{r^*}|\pi_{mix}^T}$ is at constant level, which results in the $\tilde{O}(T^{-1/2})$ rate. From this view, $\beta$ only matters in how fast $Cov^{\pi^*_{r^*}|\pi_{mix}^T}$ reduces to constant as $T$ grows.

1.2 About structure of Section 3 and 4

Thank you for pointing it out, and we will try our best to make it more understandable. The core insight behind our discussion in Sec. 3 and 4 (and also empirical TPO in Sec. 5) is that "select and transfer from the policy with the best coverage of $\pi^{\*}_{r^{\*}}$". It may help the reader to grasp the main algorithm if we highlight this statement earlier (e.g. Sec. 3.2).

2. Methods And Evaluation Criteria & Experimental Designs Or Analyses

It is also not clear to me whether Table 1 is a 'good' empirical result…

Win rate is a common metric in evaluating the LLM performance. A "(50+$p$)%" win rate against no-transfer baseline roughly means that, per 100 prompts, the LLM fine-tuned by empirical TPO is preferable on $50+p$ out of 100, which is $2p$ more than no-transfer baseline ($50 - p$ out of 100).

Besides, the performance gap may be further enlarged by increasing training batch size and training epochs, incorporating better source reward models, adjusting test-time decoding temperature, etc. Notably, the quality of the inital policy also affects the magtitude of the performance gap---the more near-optimal the initial policy is, the smaller the maximal possible performance gap can be, although empirical TPO can still outperform the no-transfer baseline.

Therefore, our primary goal in experiments is to demonstrate the potential of our transfer learning techniques in improving sample efficiency, especially, reward transfer remains an underexplored area in online RLHF.

...is the point that only three iterations were needed?

Here we follow the related literature [1,2], where running for three iterations is a common choice in algorithm evaluation (due to limited computational resources).

Would the trend in improvement continue with additional iterations?

As reported in Fig. 2 in Appx. I, in the 3rd iteration, empirical TPO already switches back to online learning without transfer, as the online policy can already outperform the best source policy in win rates. If we continue training for more iterations, the comparison with the no-transfer baseline effectively reduces to running the same online algorithm but with a better initial policy. Overall, we can expect the advantage of empirical TPO to persist, although the performance gap may decrease as both of them are converging to the optimal policy.

3. Relation To Broader Scientific Literature

To our knowledge, most existing transfer learning literature, especially those theoretical ones, focuses on pure-reward maximization setup. We are the first to identify special structure (Lem. 3.1) induced by KL regularization, and propose novel transfer learning principles specialized in this setting.

4. Other Strengths And Weaknesses

Thank you for your suggestions! We will take them into consideration in our next revision.

[1] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint

[2] Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF