DPO Meets PPO: Reinforced Token Optimization for RLHF

Thank you for your review and support. Below are our response to your questions.

Q1: Assumption 3.1 is not a rigorous assumption. The parameters $A$ and $\xi$ have not been defined. What are their possible ranges of values? The current statement is not an assumption without conditions $A$ and $\xi$. In addition, it is helpful to add more discussions on the intuition of this assumption and why it makes sense in the considered problem.

A1: The parameter $A = |\mathcal{A}|$ is the action set size (Line 220), and $\xi$ is a constant introduced in Assumption, to clarifiy this, we revise Assumption 3.1 as

Assumption 3.1. There exists a response $y = y_{1:H}$ satisfying $\pi^*(y|x) \ge A^{-\xi}$ for some $0 \le \xi \le H$.

We have also stated in the paper that "By the pigeon-hole principle, there must be a response $y$ such that $\pi^*(y | x) \ge A^{−H}$, implying that $\xi ≤ H$ naturally. In practice, $\xi$ is usually much smaller than $H$ because the language model tends to choose the optimal response rather than making a random guess."

More specifically, there are at most $A^H$ possible responses/trajectories $y = y_{1:H}$ (each step has $A$ actions (tokens) to choose from and the sequence length is at most $H$). Therefore, at least one response is chosen with a probability of at least $A^{-H}$. Moreover, given a prompt, the LLM will only generate several likely tokens with high probability, which makes the final generation probability of the most likely response significantly larger than the random guess probability $A^{-H}$. We denote this as $A^{-\xi}$ with $\xi \ll H$.

Q2: There is a gap between the Algorithm 1 (Theoretical Version) and Algorithm 2 (Practical Version). The theorems were proved for Algorithm 1, whole all experiments were done based on Algorithm 2. If Algorithm 1 is not implementable in practice, why to introduce it? Can you prove the suboptimal gap bound for the Algorithm 2 directly?

A2: We introduce the practical version of the theorem primarily to demonstrate that token-level MDPs with KL constraints can be learned sample-efficiently. We believe this constitutes a new and important theoretical advancement that deserves emphasis. We also acknowledge that proving the suboptimality gap bound for Algorithm 2 directly is challenging; in fact, even for standard PPO, its theoretical analysis often requires simplifications, such as replacing the GAE estimation with an optimistic estimation. Thank you for your question, and we will highlight our motivation for introducing the theoretical version and discuss its current limitations.

Q3: A few highly related key references were only mentioned in Section A of the Appendix. For example, Zeng et al. (2024) also considered token-level DPO. Can authors add more discussions on the difference and the technical novelty beyond this paper?

A3: Sure. We will include more discussion in the revision. Meng et al. (2024) propose SimPO by modifying the DPO objective, replacing the reference model with response length, and adding a margin threshold. Zeng et al. (2024) also consider a token-level reward and leverage this insight to develop token-level DPO, which performs better than the original DPO. These are beyond the scope of our work. In contrast, we utilize the implicit token-level reward provided by the original DPO as the dense token-level reward for RL training.

Q4: The authors used "DDPO" and "DPPO" in plots and discussions. Are they all different?

A4: Thanks for pointing out the typo and we will correct them. Both "DDPO" and "DPPO" indicate the baseline that uses RTO reward delayed to the last token.

Thank you for your review and support. Below are our response to your questions.

Q1: I have a question on the results. SimPO method outperforms PPO by a large margin (Table 1). Do you have any intuition for this?

A1: Our observation is that the RL-free methods are good at fitting the style over the deep RL methods, while the human/GPT-4 preference are easily hacked by the preference bias (see Figure 3 of [1] and related discussions). Compared to the DPO, SimPO further removes the KL constraint so it is more easily to achieve a higher in-domain score (AlpacaEval). However, we also notice that SimPO will also often hurt the OOD reasoning performance (see Table 9 of their paper for example).

[1] From Lists to Emojis: How Format Bias Affects Model Alignment

Q2: Literature seem to suggest that RL-free offline methods are outperformed by online methods, such as PPO, but the results in this paper seem to suggest differently.

A2: Indeed, almost all closed-source state-of-the-art LLMs, including ChatGPT, GPT-4, and the recent DeepSeek R1, are trained using RL-based methods like PPO and GRPO. However, the open-source community has struggled to replicate these RL training techniques effectively. Our work aims to narrow this gap by exploring both dense and sparse reward approaches.

Thank you for reviewing our paper. Below are our responses.

Question Regarding Theory: Theoretical Proofs: Could you provide a complete proof for the sample efficiency claim in Appendix B?

Response: We have provided complete and rigorous proofs for both Proposition 3.2 and Theorem 4.2 in Appendix B. Should you have any specific questions or require further clarification, we are happy to address them. Regarding our extension mentioned in Remark B.3, we have acknowledged that this is a standard result in RL literature (see e.g., Theorem 2 in [1] for a detailed proof). Given that this standard extension is not central to the main claims of our paper, we opted to provide a concrete reference rather than including a detailed proof. We appreciate your understanding.

[1] Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization. https://arxiv.org/pdf/2007.06558

Question Regarding Experiments:(1) DPO’s Role: How does DPO-derived token-wise reward correlate with human preferences? Is there empirical validation beyond benchmark scores? (2) Baseline Details: Were PPO and DPO baselines trained with identical hyperparameters (e.g., KL penalty, reward scaling)? (3) Data efficiency: The claim that RTO achieves PPO-level performance with 1/8 of the data is compelling but requires validation across multiple seeds.

Response: Thank you for your question.

(1): We have added the following results to illustrate that DPO-derived token-wise rewards effectively capture human preferences. For the pairwise data, we measure consistency between human choices and DPO reward choices (reward implies an order). We observed a 79.23% consistency in the training data and 72.72% in the test data. Additionally, we want to emphasize that the performance of DPO-derived token-wise rewards used in reinforcement learning training, evaluated by widely recognized benchmarks, is a key standard for assessing its quality and is also the central focus of our work. To further illustrate the generalizability of DPO-derived reward, we've extended our methods to another RL algorithm REINFORCE++ [1].

The following table presents the Alpaca Eval 2 benchmark scores, further demonstrating the ability of DPO rewards to effectively capture human preferences and the broader applicability of our RTO method.

[1] REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models

(2): We have included the detailed hyperparameter choices in Appendix E. The DPO and PPO hyperparameters differ, and both sets were derived from well-tuned hyperparameters provided by the OpenRLHF project. For your convenience, we present important hyperparameters here: The KL penalty coefficient ($\beta$) is $0.01$ for PPO and $0.1$ for DPO. Other PPO hyperparameters are clip coefficient $\epsilon=0.2$ and GAE $\lambda=0.95$. We train a reward model for PPO, and another tiny reward model for RTO ($r_\text{MLE}$) following the standard practice. The reward scaling of PPO is $1$ for reward model and $\beta=0.01$ for KL reward penalty. For RTO, we use $\beta_3=1$ for reward model, $\beta_2=0.01$ for KL reward penalty, and $\beta_1=0.05$ for DPO reward.

(3): Thanks for the suggestions. Our implementation is based on OpenRLHF and our (also the community's) experience (across hundrends of LLM RL training in either preference learning or math/coding task) is that with the well-tuned recipe, the LLM RL is very stable and not greatly affected by the choice of random seed. As a result, we believe our findings are sufficient to demonstrate the sample efficiency of our claim.

Thank you for reviewing our paper. We summarize your questions and address them as follows.

Q1: MDP fomulation and Previous Theoretical Work

A1: We agree with the reviewer that the existing PPO is implemented in an MDP with zero reward for all but the last token, but we would like to claim that this is essentially a bandit problem. This point is explicitly stated in Section 3.5 of the seminal InstructGPT paper [1]. In contrast, a token-level MDP with dense rewards is a real multi-step decision-making problem (see Proposition 3.2 for a theoretical gap).

We will emphasize the importance of KL constraint in claim (1), to different from existing theoretical work on dueling RL. Moreover, we would like to emphasize that these works are purely theoretical works and does not lead to corresponding paractical algorithm leveraging dense rewards.

Finally, the token-level MDP terminology is mainly used to emphasize the use of dense rewards, in contrast to the bandit formulation with sparse rewards, which aligns with the reviewer's understanding. This token-level MDP was rigorously introduced first in our paper, and it is well recognized by the community [2]. Therefore, we prefer to maintain the terminology of token-level MDPs with KL constraints while highlighting the differences from existing bandit formulations and theoretical dueling RL work.

[1] Training language models to follow instructions with human feedback

[2] Value-incentivized preference optimization: A unified approach to online and offline rlhf

Q2: MLE in (4.1) at the trajectory level

A2: The MLE objective in (4.1) looks like at the trajectory level because the preference signal is usually given in the trajectory level and the cumulative token-level reward is the trajectory-level reward. However, even the trajectory-based preference reward can naturally induce a cumulative token-level reward. This is more evident for problems with step-wise structure like mathematical reasoning [3].

[3] Entropy-Regularized Process Reward Model

Q3: Theorem 4.2 depends on $\lambda$ and the role of $\lambda$

A3: The parameter $\lambda$ is just a regularization parameter for theory (ensure $\Sigma_{\mathcal{D}}$ has inverse), and our Theorem 4.2 holds for any $\lambda>0$. You can simply regard $\lambda = 1$.

Q4: Verify Assumption 4.1 by construction

A4: If we choose $\phi(s, a) = 1_{(s, a)}$ as the one-hot vector and $\theta_{(s, a)}=r(s, a)$, then the assumption is satisfied with $d = |\mathcal{S}| \times |\mathcal{A}|$. However, the reward function may exhibits some low-dimensional respresentation $\phi \in \mathbb{R}^d$ with $d < |\mathcal{S}| \times |\mathcal{A}|$, and thus we assume Assumption 4.1 with potentially small $d$ and do not make the explicit construction. This is also the main motivation of linear bandit and linear MDP.

Q5: Experiments: (1) limited base model and baselines. (2) application to other RL algorithms; (3) hyperparameter $\beta_3$?

A5: (1) Regarding baseline selection, we focused on the most relevant (DPO, PPO), strong (R-DPO, SimPO), and concurrent (SimPO, TDPO) methods. Given the limited performance of numerous other alignment algorithms reported in SimPO, we opted not to include them in our comparisons. Due to time limit, we are still trying to add experiments on other base models. We appreciate your understanding. (2) We extend our experiments to REINFORCE++ (RF++) [5], an alternative RL algorithm. We selected RF++ since

• Unlike reasoning tasks, chat alignment typically does not involve sampling multiple responses per prompt like RLOO and GRPO.
• RF++ exhibits significant differences from PPO, notably the absence of a critic model.

The table below shows the power of RTO when applied to RF++.

[4] REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models

(3) Since only relative ratio matters and to minimize tuning, we fix $\beta_2$ the same value PPO uses, choose $\beta_3 = 1$, and tune $\beta_1$ (see discussion below (4.7)). In our internal experiments we found our algorithm are quite robust to the choice of $\beta_1$. Thus we believe that conversely, RTO is not sensitive to the choice of $\beta_3$.

Q6: Reward shaping

A6: Our algorithm can be interpreted as a form of potential-based reward shaping, with $F(s, s' = (s, a)) = \Phi(s') - \Phi(s) = \log \frac{\pi(s')}{\pi_{\mathrm{ref}}(s')}-\log \frac{\pi(s)}{\pi_{\mathrm{ref}}(s)}=\log\frac{\pi(a|s)}{\pi_{\mathrm{ref}}(a|s)}$, where $\Phi$ is the potential function and $F(s, s' = (s, a))$ denotes the token-wise reward function added to each token. Thank you for the insightful question and we will emphasize this in the revision.

Regarding your minor suggestion, we would emphasize the issue of the reward modeling type directly, rather than mentioning the open-source implementation.