Policy-labeled Preference Learning: Is Preference Enough for RLHF?

We sincerely thank the reviewers for their time and thorough evaluation of our paper. We have organized our responses to your comments below. If any of our responses fail to address the intent of your questions or if you have remaining concerns, please let us know.

1. Final practical implementation

We agree that it would be beneficial for readers to have a clearer view of the final implementation in the main text, and we will revise to address this.

2. Regret-based preference model attribution

Thank you for clearly outlining the relationships with the literature. We will revise and add a reference to [Knox et al. 2022] to clearly attribute the origin of the regret-based preference model as referenced in CPL.

3. Including the meaning of theoretical results

We appreciate your insight that providing intuitive explanations for our theoretical results can further enhance the reader's understanding. We have endeavored to include interpretive statements following each theoretical result. Could you please kindly indicate if there are specific parts of the theoretical results that you believe would benefit from additional clarification?

4. Theoretical justification

• (Lemma 3.2): We have confirmed that [Rafailov et al. 2024a] derived the same theoretical results as [Rafailov et al. 2024b], and we will cite this in the paragraph explaining Lemma 3.2.

• (Theorem 3.4): While [Shaikh et al. 2024] presented their results in a contextual bandit setting, our Theorem 3.4 extends this to MDPs. This extension is non-trivial due to the presence of stochastic transitions. Though [Zeng et al 2024] achieved results more similar to ours than [Shaikh et al. 2024], their proof applies only to deterministic transitions. Thus, Theorem 3.4 makes a distinct theoretical contribution by addressing environmental stochasticity.

5. Theorem 3.6

First, Corollary 1 of [Hejna et al. 2024], which motivated Theorem 3.6, showed that maximizing the optimal advantage is equivalent to learning the same optimal policy as maximizing the reward. In this context, Theorem 3.6 was written to make it intuitively easier to understand what kind of policy is learned from a reinforcement learning perspective when maximizing negative regret. The reason for considering only the preferred segment here is to interpret this intuitive meaning more clearly. For less preferred segments, the PPL objective can be expressed as:

$\arg \min\_{\pi\_{\psi}}\Big(\mathbb{E}\_{\zeta^- \sim \mathcal{D}}[-Reg^{\pi^-}\_{\pi\_{\psi}}(s^- ,a^-)-\alpha \log \pi\_{\psi}(a^- |s^-)]\Big) \equiv \arg\max\_{\pi\_{\psi}}\Big( \mathbb{E}\_{\zeta^- \sim \mathcal{D}}[\bar{D}\_{KL}(\pi^- || \pi\_{\psi} ; s^-, a^-)]\Big)$

However, since the left-hand side represents cost minimization rather than reward maximization in traditional reinforcement learning, it is difficult to interpret intuitively. To avoid this ambiguity, we only dealt with the preferred segment perspective.

6. Dataset Generation and Evaluation Procedure

• Dataset Generation: During SAC training, we saved checkpoints of models showing 20%, 50%, and 100% success rate. We loaded policy model to generate 20K segments, and for dense cases, randomly sampled 2.5K segments. For labeling procedure, we used $V^{\pi^*}(s_k)-V^{\pi^*}(s_0) + \sum_{t=0}^{k-1} r(s_t,a_t)$ from CPL, and calculated $V^{\pi^*}$ using 64 MCMC samples with a 100% success rate critic $Q^{\pi^*}$. For fair comparison, all algorithms were trained with the same labels, and when policy labels were needed in PPL, logits were calculated using policy information stored during rollout.

• Evaluation Procedure: We conducted evaluations following the same method as CPL: (1) run 25 evaluation episodes, (2) calculate averages across 8 neighboring checkpoints, (3) average across 4 seeds, and (4) report the maximum values in the table.

For clarity of experimental design, we will add the provided response to the appendix.

7.Retrain the SAC oracle for accurate advantage estimation

In the CPL dataset creation process, data segments are added to the replay buffer of the oracle SAC and then re-trained, which helps lower the TD error and improve the accuracy of advantage estimation. While we do not adopt the same procedure, we used only a single SAC oracle and trained all algorithms with the same labels. Therefore, it is difficult to conclude that the inaccuracy of advantage estimates caused performance degradation only for CPL.

One likely reason for CPL's struggle with noisy labels appears to be similar to the likelihood mismatch issue discussed earlier. Since CPL does not consider policy labels, it assumes all training data segments originate from an optimal policy. We suspect that when noisy labels occur, suboptimal data segments might be incorrectly treated as if they were generated by the optimal policy, potentially making the model more sensitive to noise and degrading its learning performance.

We sincerely thank the reviewers for their time and thorough evaluation of our paper. We have organized our responses to your comments below. If any of our responses fail to address the intent of your questions or if you have remaining concerns, please let us know.

1. Paper position disconnection with introduction

To emphasize that the PPL algorithm learns policies without explicitly learning a reward function, we mentioned in the Introduction that we are adopting the Direct Preference Optimization framework. Although the DPO paper, which introduced this framework, has been studied in the LLM domain, we noted in line 25 (right column) that its deterministic transitions make it an unsuitable setting for demonstrating the advantages of PPL.

2. Label usage in experiments

In the experiments in Section 4.2, PPL assumed that policy labels are accessible. However, since extracting reliable policy labels is nontrivial, in Section 4.3 we designed and experimented with PPL-deterministic, which utilizes pseudo labels.

3. Analysis under noisy label

Thank you for your comment regarding noisy pseudo-labels. We are currently conducting experiments in which we add Gaussian noise, $\mathcal{N}(0, \sigma^2)$to the ground-truth labels. In the button-press environment, we observed that performance degrades as the noise level $\sigma$ increases. Specifically, at $\sigma$=0.1, the performance was lower than that of PPL but higher than that of PPL-deterministic. However, at $\sigma$=0.3, the performance dropped below that of PPL-deterministic. This indicates that if the pseudo-label is not sufficiently close to the policy label, its performance will be inferior to the proposed PPL-deterministic, thereby confirming that the pseudo-label used in PPL-deterministic is a reasonable alternative.

4. Likelihood mismatch and role of coverage

As described in Figure 3 and in the section on likelihood misalignment, the likelihood mismatch issue does not primarily arise from dataset coverage. Instead, it occurs when the algorithm assumes that trajectories are always generated by the optimal policy. As shown in Figure 3, let us consider the black trajectory is generated by the optimal policy, while the red trajectory is produced by a suboptimal policy. Let's consider the case where all four trajectories—which ensure complete data coverage—are included in the dataset. Without the policy label, the estimated MDP appears as shown in the right panel of Figure 3, and it is evident that it still misinterprets the ground-truth MDP depicted in the left panel.

5. Misleading title

Thank you for your detailed feedback. However, we believe that the scenario considered in our work—where data is generated by heterogeneous/diverse policies—is more justified and encompasses a broader range of realistic situations than merely assuming a homogeneous dataset. Previous studies using offline datasets have not taken into account the policy that generated the dataset. In contrast, our paper highlights this point and proposes an algorithm that leverages policy information, thereby emphasizing that in RLHF settings, both preference and policy information should be considered. We consider the current title appropriate, but if you feel that this context is not sufficiently conveyed, we would greatly appreciate any further suggestions.

6. Comparing with DPO and DPPO

Since the [Rafailov et al 2024] presents an algorithm designed for bandit settings, it could not be directly applied to sequential tasks such as robotic manipulation, and therefore was not compared separately. In the case of the DPPO, since CPL assumes a fixed standard deviation Gaussian for the policy, its loss simplifies to an L2 loss between actions—identical to the implementation used in DPPO [An et al. 2023]. We confirmed this similarity based on the official CPL GitHub repository, so no separate comparison was performed.

7. Other offline likelihood (policy mismatch) literature

To the best of our knowledge, we have not found any offline RL algorithm that considers likelihood mismatch or policy mismatch.

References

• [Rafailov et al 2024] : Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization:Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.
• [An et al 2023] : An, Gaon, et al. "Direct preference-based policy optimization without reward modeling." Advances in Neural Information Processing Systems 36 (2023): 70247-70266.

We sincerely thank the reviewers for their time and thorough evaluation of our paper. We have organized our responses to your comments below. If any of our responses fail to address the intent of your questions or if you have remaining concerns, please let us know.

1. Generalization to other domains

Since PPL is particularly beneficial for sequential and stochastic scenarios, we determined that a robotic manipulation task is more appropriate than a language domain with deterministic transitions, and thus conducted our experiments accordingly. If extended to the language domain, exploring the multi-turn LLM scenario you suggested would be an excellent direction for future work.

2.LLM literature reference

Recent research on RLHF has been particularly active in the LLM domain, and our PPL algorithm which directly learns policies without explicitly learning a reward function shares similarities with the Direct Preference Optimization framework. We briefly mention this connection in the main text. However, since our experiments focus on sequential RL tasks such as robotic manipulation, we decided not to deeply discuss related work specific to LLM tasks to avoid reader confusion.

3. Response to [Other Comments and Suggestions]

In Equation (2), we noticed that the derivation of Equation (3) is referenced as being in Appendix B.1, but that reference appears about one paragraph after the equations are presented. To reduce any potential confusion for readers, we will adjust the sentence placement. Additionally, the explanation for the subscript * is provided later than its initial usage. Since removing the subscript * does not change the original meaning of Equation (2), we will remove it from that equation. We will also correct the typo in Lemma 3.3 as you pointed out.

We sincerely thank you for your time and effort in reviewing our paper. We have organized our responses to your comments below. If any of our reconstructed responses miss the intent of your questions or if there are remaining concerns, please let us know so we can address them.

1. Difference between PPL and cited literature

The regret-based preference model used in the CPL algorithm, which defines regret in terms of the optimal advantage function, was first introduced in [Knox et al. 2022]. Similarly, [Knox et al. 2024] interprets preferences using the same optimal advantage-based regret model, arguing that the Bradley-Terry model yields an approximation of the optimal advantage function rather than a true reward function. The regret preference models used in CPL, [Knox et al. 2022] and [Knox et al. 2024] are both essentially based on optimal advantage, which is a different design from the regret model proposed in PPL.

2. Response to [Other Comments and Suggestions]

Thank you for pointing out the typo, and I will incorporate it into the revised version for better clarity. Regarding Figure 4, it was designed as a single plot to enable an at-a-glance comparison of the distributions across the three datasets. However, we recognize that overlapping colors make it difficult to distinguish the overlapping regions. Given the space constraints in the main text, we will either separate the plots or incorporate an envelope to improve boundary delineation.

References

• [Knox et al 2022] : Knox, W. Bradley, et al. "Models of human preference for learning reward functions." arXiv preprint arXiv:2206.02231 (2022).
• [Knox et al 2024] : Knox, W. Bradley, et al. "Learning optimal advantage from preferences and mistaking it for reward." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 9. 2024.