On a Connection Between Imitation Learning and RLHF

Dear Reviewer hdJf,

Thank you very much for carefully reading our response and increasing your score! We are very happy that our response has addressed your comments. We genuinely appreciate your support and will include the new results in the main paper.

Regarding your observation, this was indeed a typo. We mistakenly wrote SLiC as KTO. Nevertheless, we are actively working on comparing our method with KTO on benchmarks.

We have updated the KTO results in the main Table 2 of the updated submission PDF. The results show that our DIL still achieves better performance than KTO on most widely used benchmarks, particularly on AlpacaEval 2.0 and MATH, demonstrating that DIL can more effectively preserve reasoning abilities compared to KTO while aligning with human preferences.

Sincerely,

The Authors

I have adjusted my score given the reviews and the replies. One last detail is the KTO results are missing despite being mentioned as a baseline in the paper, so it would be better if added.

Q5. In 6.3 you discuss learning dynamics DPO, SimPO, and DIL however Figure 3 does not have DPO, is the discussion from some other paper?

A5. Thank you for your questions. Yes, the issue of decreasing likelihood of chosen responses in DPO has been widely noticed by many recent works [2,3,4]. Given the superior performance of SimPO over DPO, we only included SimPO in Figure 3 as the motivating example.

Q6. Do you have additional results on MT-Bench or Arena Hard?

A6. Thank you for your questions. To answer your question, we have included additional comparisons on Arena-Hard given it is an updated version of MT-Bench in Table 2 (Comparison results on Mistral-7B-Base are also shown in the table below.) We report the win rate (WR) against the baseline model following SimPO. We find our DIL still achieves superior performance on Arena-Hard compared to baselines

We wholeheartedly appreciate your suggestions and comments. Nevertheless, we believe there are indeed misunderstandings and not fatal to the major contributions of our manuscript. We have made extensive efforts to address your comments and believe that they adequately address all your concerns. We believe that the reviewer's insightful comments can be easily and effectively addressed in the final version. Could you please consider increasing your score to reflect the efforts of your review and our rebuttal.

[1] MiniLLM: Knowledge Distillation of Large Language Models. ICLR 2024.

[2] Iterative Reasoning Preference Optimization. NeurIPS 2024

[3] Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment. NeurIPS 2024.

[4] Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. In Arxiv.

Dear reviewer hdJf, we appreciate your perception of our contributions on connecting RLHF with imitation learning. We believe that there are some important misunderstandings. Please see our clarifications below:

Q1. Equation 13 is the reverse KL between the behavior and optimal policy however knowledge distillation is forward KL between teacher (optimal) and student(behavior). KD (forward KL) is distribution fitting or mean seeking whereas reverse KL is mode seeking which makes the policy focus on high-reward regions rather than fitting the entire distribution with forward KL as in SFT. Overall, the KD claim by the paper is incorrect....

A1. We apologize for any confusion and believe there may have been some misunderstandings. The knowledge distillation in Equation (13) that we specifically mentioned here is indeed a "reverse" version, focusing on minimizing the reverse KL divergence, as also mentioned in recent work MiniLLM [1].

Yes, Equation (13) is a known result from DPO, but our key contribution is not Equation (13). Please see our response to Q3 for detailed justification why our DIL differs from DPO.

Q2. In standard RLHF, the reward model is a separate LLM with an additional MLP to predict the scalar reward. So by training a reward model, one does not imitate the expert or optimal policy.

A2. Thank you for your comments. Although the choice of the loss function and optimization procedure differs, the central aim of our work is to emphasize that the optimal policies of RLHF and DPO are theoretically the same as the imitation learning process. In essence, all these methods aim to discover identical optimal policies, i.e., the chosen response distribution. Specifically,

• Equation (12) shows that imitation learning loss between the distribution chosen response and energy-based policy is exactly the same as the reward loss based on BT assumption.

• Equation (13) (also shown in DPO) shows that imitation learning between optimizing policy response and energy-based policy is exactly the same as the RL loss.

• Thus, RLHF with two steps (reward learning and policy learning) can be viewed as conducting imitation learning between optimizing policy and the distribution of chosen response.

Q3. I dont think DIL is novel because it is the backtracking of the derivation of the DPO objective. After all, the 16th equation is the same as the 14th equation of the DPO without the partition. DIL is redefining the reward function of DPO, excluding the density ratio estimation part. All in all, I believe this part(excluding density ratio) is already present in DPO.

A3. Thank you for your comments. We believe there are indeed some misunderstandings. Our work indeed significantly differs from DPO in several important ways:

• Theoretical Insight: We are the first to show that the objective of current alignment methods, such as DPO and RLHF, can theoretically be viewed as fitting the chosen response distribution by minimizing the reverse KL divergence.
• General Framework: We provide a general framework based on Bregman divergence to directly optimize the reverse KL divergence between the policy and the chosen response distribution.
• Empirical Results: We demonstrate that our framework effectively alleviates the decrease in the likelihood of chosen responses and achieves better performance on reasoning-intensive tasks, addressing important limitations in DPO.

Q4. You mention that DIL does not depend on Bradley-Terry but you introduce new reward training with different objectives such as LSIF, UKL, and BCE which are essentially replacements for BT, so doesn't the DIL still rely on some preference modeling assumption?

A4. Thank you for your comments. As shown in our paper, DIL and DPO indeed share the same assumption, as they both solve the same imitation learning objective (i.e., minimizing the reverse KL divergence between the policy and the chosen response distribution).

However, as demonstrated in our paper, DPO relies on the BT assumption (pairwise loss) to cancel out the normalization term. Due to the self-normalized property in Equation (22), our DIL generalizes to a broad class of objectives that do not rely on pairwise comparisons, unlike DPO/SimPO. Since DPO/SimPO only learn to preserve the relative ordering between the likelihoods of the chosen and rejected responses, they reduce the likelihood of the chosen response, resulting in poor performance in reasoning tasks.

Thank you for the rebuttal!

Thank you for your comments. Please refer to our responses Q6 to Reviewer JHUL, where we provide additional results demonstrating that our improvements are significant compared to SFT, DPO, and SimPO.

Thank you for providing these! These do increase my confidence in the signifcance of the results. Nevertheless, could the authors also provide standard errors or CIs?

Also, looking further at Table 2, is it possible SFT should be bolded instead of DIL for Mistral MMLU-PRO? (27.58 > 27.44)

We apologize for any confusion. What we intend to convey is: "With the estimated density ratio reward, the surrogate imitation learning objective in Equation (17) can be solved using any RL algorithm. However, this two-step process is complex and often unstable." ... We have updated the main paper to clearly articulate this.

Thank you, after having another look, this clarification does help with the reading of the paper!

Yes, we have included the base model (SFT) performance in Table 2 of the updated submission.

Great, thanks for adding that into the table - it gives a good starting point to compare to for all methods.

The reason we chose "vs. SFT" and "vs. Chosen" is to strictly align all settings with those in the original DPO paper.

I agree it's generally a good idea to be consistent with prior work. However, that shouldn't restrict the authors from adding additional results here. It makes sense that in the DPO paper, the authors didn't include "vs. DPO", because they'd be comparing against themselves which is maybe not very meaningful. However, in this paper, it makes a lot of sense to do a head-to-head comparison with DPO and maybe SimPO in Table 3, instead of having to infer the strength of one method over the other indirectly through the performance against some other baseline (SFT and Chosen in this case).

Yes, it is negative log-likelihood. We apologize for the confusion and have updated the figure accordingly.

Actually, since it's negative, is it just log-likelihood? The version I'm seeing still seems to just say "likelihood". Though thank you for making the other changes in the figure!

Sorry for your confusion. We also utilize a change-of-variables approach (critic function to policy using Eqs. (23) and (26)).

I'm still a bit confused here. Are the authors plugging the result from Eq. 26 ($f^*(x, y) / \beta = \log \frac{\pi_{\mathrm{chosen}}(y | x)}{\pi_{\mathrm{ref}}(y | x) c(x)}$) into Eq. 25? If so, then you don't even need Eq. 23, since Eq. 26 gives essentially the same result? If the authors could clarify their exact steps in a bit more detail, that would be helpful.

Further questions

1. Why is DPO not included in Fig. 1 & 3? From the reply to some of the other reviewers I understand it's because SimPO is the motivating example, but I still think adding DPO could help put the results in those figures in better perspective, especially since DPO is still widely used.

2. In 4.1, why is it justified to take $\pi_{\mathrm{ref}}(y | x) = 0.5 \mathbb{I}(Y = y_l) + 0.5 \mathbb{I}(Y = y_w)$ for sampling from the reference distribution?

Finally, thanks for taking the time to address all the other minor clarifications/typos!

Dear reviewer MhY4, we appreciate your efforts and detailed comments very much! However, we believe that there are some misunderstandings. Therefore, we would like to provide a point-by-point response to your comments.

Q1. While the empirical results seem to consistently outperform prior methods, I’m a bit worried about the statistical significance since the margins seem rather small sometimes (e.g. for Table 2, the improvements are almost always smaller than 1 percentage point).

A1. Thank you for your comments. Please refer to our responses Q6 to Reviewer JHUL, where we provide additional results demonstrating that our improvements are significant compared to SFT, DPO, and SimPO.

Q2. The exposition of the math/theory in the paper could have been a bit clearer (section 4). It took me some time to understand what actually is the final objective that DIL optimizes, and how it came to be. This is because, for example, at the end of section 4.3 the authors state “With the estimated density ratio reward, the surrogate imitation learning objective in Equation (17) can then be solved with any RL algorithms.”, which initially made it seem like DIL would have to resort to RL optimization anyways…

A2. We apologize for any confusion. What we intend to convey is: "With the estimated density ratio reward, the surrogate imitation learning objective in Equation (17) can be solved using any RL algorithm. However, this two-step process is complex and often unstable." To address this, in section 4.4, we introduce a simpler approach that directly optimizes the imitation learning objective. This method bypasses the need for RL training and density ratio estimation by leveraging a change-of-variables approach. We have updated the main paper to clearly articulate this.

Response and Clarification to Questions:

Thank you for your detailed questions! Below are our responses and clarifications. We will update the submission to include these clarifications.

1. Mistral-7B-Base is the same as Zephyr-7B-SFT. We have corrected the typo.
2. This was a typo. We mistakenly wrote SLiC as KTO. We have corrected it.
3. Yes, we have included the base model (SFT) performance in Table 2 of the updated submission.
4. The "average" refers to the average win rates computed by GPT-4 when comparing the SFT-generated responses with the chosen responses in the dataset.
5. The reason we chose "vs. SFT" and "vs. Chosen" is to strictly align all settings with those in the original DPO paper.
6. As shown in the original DPO objective under the BT assumption, it maximizes the expected relative difference between the implicit 7. rewards of the chosen and rejected responses. Thus, while these methods preserve the relative ordering between the likelihoods of the chosen and rejected responses, they may reduce the absolute likelihood of the chosen response.
7. The reviewer is correct. $\pi_{\text{chosen}}$ refers to the probability of generating the chosen response in the preference dataset.
8. The tabular setting refers to cases where the state and action spaces are small, and both state functions and action-state functions are represented as tables.
9. Yes, it is negative log-likelihood. We apologize for the confusion and have updated the figure accordingly.
10. The term “infinite models” was a mistake; it should be “infinite computation.” We apologize for the typo.
11. It should indeed be $\pi_{\text{chosen}}$. We apologize for the confusion.
12. There should be a log operation here. We apologize for the typo.
13. The characteristic refers to the self-normalized property. DPO relies on the BT assumption (pairwise loss) to cancel out the normalization term. Due to this self-normalized property, our imitation learning approach generalizes to a broad class of objectives that do not rely on pairwise comparisons.
14. Sorry for your confusion. We also utilize a change-of-variables approach (critic function $f$ to policy $\pi$ using Eqs. (23) and (26)).
15. Yes, the results are compared to some base model performance. Please see the updated Table 2, where we provide the performance of the base models (SFT).
16. The improvements are over SimPO. We apologize for the confusion and have clarified this in the updated submission.

Dear reviewer sbFs, we appreciate the reviewer's perception of our contributions to both empirical and theoretical analysis, and we thank the reviewer for their insightful questions. Please find our detailed responses below:

Q1. The amount of data needed for satisfactory alignment with DIL compared to other methods is not clear.

A1. Thank you for your comments. We apologize for any confusion. The term "efficient" does not refer to data efficiency; instead, it highlights that, compared to PPO in RLHF, our DIL approach is more efficient in terms of computation, speed, and engineering effort. DIL eliminates the need for an additional stage of training a reward model and, during policy training, does not require decoding online responses (which is typically slow) or training an additional value model.

Q2. All the models in the experiments are smaller (<10B parameters) so it’s not clear how effective DIL would be for larger models.

A2. Thank you for your suggestions. Following your suggestion, we conducted additional experiments on Mixtral-8x22B-Instruct-v0.1. From the following results, we find that the DIL still achieves better performance compared to baselines. These results have also been included in Table 8 in the updated submission:

Q3. Since DIL doesn’t suppress the likelihood of dispreferred responses as much as SimPO, how does this affect alignment from a safety perspective? Is the model more prone to generate harmful responses?

A3. Thank you for your questions. Based on the results on Anthropic Helpful and Harmless, we observe that our DIL achieves significantly better performance than SimPO, demonstrating that DIL is less prone to generating harmful responses. We hypothesize that there may be two potential reasons for this:

• (i) Although DIL does not suppress the likelihood of dispreferred responses, the margin likelihood between preferred and dispreferred responses in DIL is at the same level as SimPO, demonstrating that DIL still has the capability to distinguish between preferred and dispreferred responses.
• (ii) Additionally, not all rejected responses in preference datasets are low-quality. It may be the case that both chosen and rejected responses are high-quality, but the chosen response is slightly better. In this case, it may not be ideal for the model to excessively reduce the likelihood of rejected responses.

Q5. "Here, we use the set of rejected responses $y_l ∼ \pi_{ref}(y | x)$ to approximate the expectations under $\pi_{ref}(y | x)$". This is simply a wrong assumption. I do not know why the authors have chosen to make the assumption, but it feels like a contrived way to come to Equation 24 and form a connection to a DPO like objective.

A5. Thank you for your insightful comments! We believe there may be some misunderstandings. The assumption is indeed reasonable. It is acceptable to use the set of rejected responses to approximate the expectations since both chosen and rejected responses are indeed sampled from $\pi_{ref}(y | x)$, as also demonstrated in [1,2].

Furthermore, we can use both chosen and rejected responses to approximate this expectation. We acknowledge that this approximation may have bias, and there are indeed many ways to approximate the expectations [1]. However, we have shown that with this approximation, DIL achieves better performance, as demonstrated in the following table. Additionally, we demonstrate that DPO with CPC as a density estimation method can be viewed as an imitation learning objective using this approximation, which we believe provides valuable insights. We have updated the paper to clarify this.

From the table, we can observe that utilizing rejected responses to approximate the expectation achieves better performance, which is reasonable as the goal is to decrease the likelihood of the rejected response rather than the chosen one.

[1] Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models. In Arxiv

[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In NeurIPS 2023

Q6. The results are on relevant benchmarks, but the improvement over DPO seems minor in most cases.

A6. Thank you for your excellent suggestions regarding potential additional experiments to justify the improvements over DPO. We want to emphasize that the improvements over DPO are indeed significant, especially on reasoning-heavy tasks. Recently, many researchers have observed that DPO generally decreases downstream task performance, particularly on reasoning-heavy tasks like Math and Coding. To verify this, we have included additional comparisons with DPO on more reasoning-heavy tasks, such as Coding (HumanEval, LeetCode, MBPP) and Math (GSM8K, MATH, TheoremQA).

From the table, we observe that DIL achieves significant improvements over DPO. Moreover, it demonstrates that DIL more effectively preserves reasoning abilities, such as the mathematical and abstract reasoning skills of the base SFT model, and even significantly outperforms SFT in many cases. Consequently, DIL imposes a lower alignment tax, given its strong performance on the preference alignment benchmark, AlpacaEval 2.0.

We gratefully appreciate your time in reviewing our paper and your comments. We have made extensive efforts to address your comments and believe that they adequately address all your comments. The reviewer's comments are mainly about some clarifications and are not fatal to the contributions of our manuscript; we believe that the reviewer's insightful comments can be easily and effectively addressed in the final version. We would be grateful if the reviewer could increase the score.

Dear reviewer JHUL, we appreciate your efforts and detailed comments very much! However, we believe that there are some misunderstandings. Therefore, we would like to provide a point-by-point response to your comments.

Q1. My first main gripe with the paper is that the idea that RLHF in its entirety is performing imitation learning seems to stand on shaky foundations. A lot of leaps from one objective to another are required to arrive at this conclusion and a lot of the nuances in the differences between different objectives get lost along the way …

A1. Thank you very much for your insightful comments!

We agree that the title "RLHF secretly performs imitation learning" may overstate the case, particularly given the nuances of the transition between imitation learning and reinforcement learning within the RLHF pipeline. We have revised the title in the updated submission to more accurately reflect this perspective: “DIL: Direct Imitation Learning for Preference Alignment and Connections to RLHF.”

Although the choice of the loss function and optimization procedure differs, the central aim of our work is to emphasize that the optimal policies of RLHF and DPO are theoretically the same as the imitation learning process, they aim to discover identical optimal policies, i.e., the chosen response distribution. We would like to kindly remind the reviewer of our contributions compared to works based on RL as inference [1].

• Theoretical Insight: We are the first to show that the objective of current alignment methods, such as DPO and RLHF, can theoretically be viewed as fitting the chosen response distribution by minimizing the reverse KL divergence.
• General Framework: We provide a general framework based on Bregman divergence to directly optimize the reverse KL divergence between the policy and the chosen response distribution.
• Empirical Results: We demonstrate that our framework effectively alleviates the decrease in the likelihood of chosen responses and achieves better performance on reasoning-intensive tasks, addressing important limitations in DPO.

[1] Reinforcement learning and control as probabilistic inference: Tutorial and review In ArXiv.

Q2. The paper seems to be derived backwards as some of the connections made feel slightly contrived upon reading. E.g. the jump from the original objective to knowledge distillation mentioned above. The steps taken to arrive at a DPO like objective from density ratio estimation etc.

A2. Thank you for your comments. The knowledge distillation steps we provided are intended to demonstrate that the standard KL-regularized RLHF problem and DPO both aim to discover the same optimal policies as those obtained by conducting imitation learning on the chosen response using reverse KL divergence.

Based on these insights, we propose a more general framework based on Bregman divergence to directly optimize the reverse KL divergence between the policy and the chosen response distribution. We have updated the paper to clarify the derivation step further.

Q3. The knowledge distillation connection seems tenuous (and already known), it seems more straightforward to think of the entire process as imitating a better policy as in MARWIL [2] or as chaining a specific variant of an improvement operator and distillation as already derived in detail for many different variants in [3].

A3. Thank you for your comments and mentioning these related works. We totally agree with the reviewer that knowledge distillation is well-known. However, The knowledge distillation steps we provided are intended to show that the standard KL-regularized RLHF problem and DPO both aim to discover the same optimal policies as conducting imitation learning on the chosen response with reverse KL divergence, which are inherently different from these works as also shown in our response to Q1. We will discuss more past works on knowledge distillation in the updated submission.

Q4. E.g. " In this section, we connect RLHF to the imitation learning framework. We show that RLHF is a special case of imitation learning problem by defining the following specialized energy-based mode" in front of Eq 9, which very clearly is already derived in the DPO paper and literature on RL as probabilistic inference. It is fine to re-state such derivations but then please say: we built on a well known connection between RL and energy based models/probabilistic inference.

A4. Thank you for your suggestions. We totally agree with the reviewer that the energy-based model in Eq. 9 is well-known; however, our contribution lies in establishing the connection between RLHF/DPO and imitation learning over the chosen response in the preference data (see our detailed response to Q1). We have updated the paper and rephrased this sentence as: “We build our analysis on a well-known connection between RL and energy-based models.” We apologize for your confusion.

Dear ICLR Reviewer JHUL,

We greatly appreciate your time and the insightful comments provided during the review of our paper.

We have made extensive efforts to address all your questions, suggestions, and misunderstandings in our response and believe that they adequately address your concerns. The reviewer's comments primarily focused on clarifying certain claims and experimental details. We have addressed these points in our response by providing detailed explanations, including clarifications on specific claims and additional results. We believe that the reviewer's insightful comments can be effectively and easily addressed in the final version.

As you and other reviewers have noted, our work offers valuable insights, builds a very interesting connection between imitation learning and RLHF, and presents a practical effective framework. Since your comments do not impact the major contributions of our manuscript, we would be grateful if you could consider increasing the score.

We are extremely grateful for your time.

Best regards,

The Authors

Dear ICLR Reviewer JHUL,

We greatly appreciate your time and the insightful comments. We have made extensive efforts to address all your questions, suggestions, and misunderstandings in our response and believe that we have addressed your concerns.

In your original review, you mentioned, “If the authors can provide some more rationale and clean up the derivations, the score could be improved.”

Thus, we sincerely want to confirm if there are any additional clarifications you would really like us to address. We would be grateful if you could consider increasing the score.

Best regards,

The Authors

Dear ICLR Reviewer JHUL,

We appreciate your recognition of our contribution and agree that our current version addresses your concerns and could be considered for publication as is.

Our work provides an intriguing explanation and insight for DPO/RLHF from the perspective of imitation learning and presents a new alignment objective inspired by this insight. We will shorten the entire derivation following your suggestion.

Thank you once again for your thoughtful review and valuable feedback!

Sincerely,

The Authors

Thanks to the authors for adjusting the paper and replying to my concerns. Most of the minor concerns have been addressed and the additional results and ablations (e.g. re. the choice of sampling dataset for estimating Eq 24) are appreciated.

I think the paper is improved and a bit clearer, especially putting emphasis more on connections between RLHF and IL seems a good choice in the title.

I still maintain that some of the derivations are lengthy and the fact that the ablations reveal that the choice of the sampling dataset for Eq. 24 makes a huge difference (and is a choice that is clearly made to make the algorithm resemble DPO as much as possible) makes me feel that the entire derivation could have been much shortened and the paper just been presented as a better alternative to DPO.

Nonetheless the paper does provide potentially useful insights to the community and in its revised version could be considered for publication as is (albeit a larger rewrite would probably improve it further by quite a bit). I have thus adjusted my score upwards.