From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

We would like to thank the reviewer for the detailed feedback! We already provided an answer about the evaluations in Section 5.1. in our response to reviewer UQbN and to your second question in our response to reviewer hRDL, which we will not repeat due to response length limitations.

“Section 5.2 attempts to make a connection between guided decoding and DPO by showing that DPO improves…”

A number of recent works ([1]-[3] for reference), have explored using search-based methods with a value function during inference time, which has shown to improve the quality of outputs. Moreover [1] has also shown such benefits when using a DPO base model as well. The goal of section 5.2 is to theoretically show the equivalence between such search methods and DPO likelihood-based search and provide empirical proof. As such we do not aim to explain the performance improvement of beam search, but to make the point that we do not need an explicitly trained separate value function. Instead we can utilize the implicit value function learned by the DPO model, by leveraging the connection between implicit value functions and LLM likelihoods in DPO, which is the main theoretical contribution of our work as outlined in Section 4.

“I do not clearly see the equivalence between Eq (2) and the formulation of RLHF in the previous studies. Are they exactly the same?”

Prior classical RLHF works usually formulate the RLHF problem using bandit terms, however in practice the PPO algorithm (and similar policy-gradient based methods) optimize the token-level objective of Eq. 2.

Thank you for pointing out a number of typos, we will fix these in our final version!

[1] Controlled Decoding from Language Models, Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, Ahmad Beirami, 2024

[2] Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding, Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, Asli Celikyilmaz, 2024

[3] Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training, Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, Jun Wang, 2024

We would like to thank the reviewer for the detailed review and kind words!

“While the original derivation of DPO relies on the fact that Q∗(x, y) = r(x, y)...”

Indeed, the original DPO work uses a bandit setting which can be represented as a trivial single-step MDP, which does not require any sequential considerations. In that setting the value function Q(x,y) is trivially r(x,y), but as we argue in later sections, this does not seem to represent the real sequential nature of language problems.

“Question 2: Could you expand on this difference? It seems to be and important aspect to judge the novelty”

Garg et al. (2022) is an imitation learning algorithm, while we focus on the problem of improving language models from (offline) user feedback. Hejna & Sadigh (2024) trains a parameterized value function using the feedback dataset and then separately attempts to extract a policy from that value function. All experiments are based on continuous control and the approach has been hard to scale to LLMs. In our work we have a unifying formulation through max-entropy RL, where we represent the LLM itself as both a policy and an implicit value function and do not use separately parameterized models. Moreover, the proofs in those works state something similar, but critically different: that there is a bijection between optimal rewards and value functions under a fixed policy. Both our setting and results are different. We show that there is a bijection between the reward function and optimal Q-function (not Q-function of a fixed policy) and that this holds in the Token-MDP with discount factor one (while prior works require discount factors less than one in order to invoke contraction properties of the Bellman operator). This requires a different proof technique than used in prior work.

“I fail to see why this is not self-evident. Scaling the logits by f(x) does not affect the probability distribution. Could you please clarify this?”

This comment links the theoretical reward-smoothing formulation of Ng et. al. (1999) as represented in Section 4.2 to the practical LLM implementation. We agree that from the logit's perspective, it is clear that a constant shift does not affect the final model outputs, but represents the reward shaping potential $\Phi(s)$. However, from a token-level RL perspective this was originally non-obvious until Ng's. work. Here we extend this reward shaping formulation to the feedback setting, which we state in Theorem 1.

We would like to thank the reviewer for the detailed feedback and kind words!

“Section 5.1 reports a case study..”

We follow the same training procedure as original DPO work, including SFT pre-training and DPO fine-tuning including the official codebase and default hyperparameters, besides varying the beta coefficient, which we report in our results.

“Again connected with the previous point, the evaluation feels very anecdotical…”

We appreciate your point! Note that we have included additional examples in the Appendix. Furthermore, additional evaluations on TL;DR have exhibited similar trends -- we can include more of these examples in the final paper.

We believe larger scale studies are indeed warranted, but face challenges:

1. Quantifying credit assignment in reinforcement learning in general is challenging, but becomes increasingly so in free-form text generation, where overall quality might not be attributable to specific tokens but larger scale features of the writing.
2. Datasets for such analysis are quite limited.

One potential domain where credit assignment might be easier to evaluate is multi-step reasoning tasks. Recent work [1], trains a DPO implicit reward model as a verifier for sequence-level reasoning. It is possible to compare step-level results from such a verifier to data with explicit step-level annotations [2]. We believe this is a valuable study, but in of itself can be quite expansive work to correctly train such models (which has warranted several publications in its own right) and is beyond the scope of our current work!

We would appreciate it if the reviewer has any particular suggestions on this front!

“Did they try completing this step for other open-source models...” We only considered the 2.8B Pythia model due to compute constraints, but we believe our results will transfer to stronger models as well.

“Does a model trained only with SFT behave in a similar fashion…”

Our credit assignment experiments all evaluate the DPO implicit reward, as outlined in Section 4.2. As such in of itself the SFT model does not induce a directly comparable reward function.

[1] V-STaR: Training Verifiers for Self-Taught Reasoners, Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, Rishabh Agarwal, 2024

[2] Let's Verify Step by Step, Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe, 2024