Large Language Model Alignment via Inverse Reinforcement Learning from Demonstrations

We sincerely thank the reviewer for their time and thoughtful feedback. We value the opportunity to clarify the novelty, contributions, and focus of our work, as well as address any concerns raised. Below, we provide detailed responses to the key points.

1. Responses Regarding Novelty

We respectfully disagree with the assessment that our work lacks novelty. Importantly, our paper predates several of the works mentioned by the reviewer.

(1). Timeliness:

To the best of our knowledge, before the submission deadline of ICLR, there were only three papers on this topic, they were made publicly available later than our work. We are happy to provide additional evidence by compliant with ICLR policies.

• Our paper is 6 months earlier than, has been acknowledged, and cited in Wulfemier et al. (Sep. 2024), as their prior related work.
• Our paper is 3 months earlier than DITTO, which is currently a concurrent submission to ICLR 2025.
• Our paper is 1 month earlier than Tajwar et al. And the paper focused on sub-optimal, on-policy data in RLHF.
• Our paper is 1 month earlier than Rafailov et al. The problem we characterized with MDP in our paper is more general than theirs — AfD as IRL is different from the conventional RLHF setup.

We would like to emphasize that the ideas introduced in our paper were developed and documented well before the appearance of these works. We believe this timing further supports the originality and relevance of our contributions.

(2). Discussions on Ziebart et al. and SPIN in Our Paper

In our submitted manuscript, we have appropriately cited and acknowledged relevant prior works, including those by Ziebart, which lays the foundation for our approach.

On SPIN, we had a section discussing SPIN as an important related work. In our initial submission, we had a page starting from line 1003 to line 1058 discussing the link and difference between our work and SPIN. As per suggested and acknowledged by reviewer iqxv that this section itself has a unique contribution to the community, we have moved this section to the main text in our revision (lines 316 - 418, highlighted by blue text). Please refer to our updated manuscript for the revision!

(3). Distinct Focus and Contribution:

While some relevant ideas have been later on discussed and empirically studied in related works, our approach differs significantly. Those related works can serve as additional support for our key insights — as the reviewer pointed out — some of their experiment results shared similar discoveries with ours.

Moreover, the focus and techniques used in those papers are different. The focus of DITTO is primarily on individualized and few-shot alignment using demonstrations, whereas our work takes a broader approach, addressing general alignment from demonstration setups. This distinction marks an important difference in the scope and application of the methods.

Additionally, our work introduces a two-stage explicit reward modeling method, which is distinct from the direct alignment methods discussed in SPIN, DITTO, and Wulfmeier et al. We believe that exploring both direct alignment and reward modeling approaches, as we do in our paper, provides a meaningful contribution to the field. The existence of multiple approaches in this area should not be viewed as competing but as complementary solutions that advance the field in different ways.

(4). From IRL Foundation to LLM Alignment with IRL

The reviewer correctly points out that Ziebart laid foundational work, and we have cited and built upon it. However, we would like to emphasize that our work extends this foundational knowledge by proposing novel methods that are specifically tailored for learning from demonstrations in LLMs. This is not simply a reiteration of past work, but rather an extension and adaptation of existing ideas to a new and impactful context.

(5). Summary and Request for Re-evaluation

We understand the reviewer’s concerns given their knowledge of the recent advancements in the field (but unintentionally missing the factual temporal order!). We hope that the points above help clarify the novelty and contribution of our work. We would also appreciate the reviewer’s recognition of the soundness of our method based on existing empirical studies and agree that it is beneficial for papers in the same domain to support one another in advancing the state of the art.

We would kindly request reviewer wScp to consider re-evaluating our work, taking into account the above clarifications, and consider the novelty and significance of our contributions in the broader context of the field.

I would like to thank the reviewer for their detailed responses to my questions. I have read the paper again and provide both a) responses to the author’s rebuttal and b) a new review of the work and an updated score.

Response to authors

On Novelty

Timeliness: I apologize, my first response may have placed an over-importance on the relationship between the timing of prior work and this current paper. That being said, I am unable to reproduce the timelines given by the authors.

• I have placed r to Q* at Apr 18, 2024: making the authors work a month later, not a month earlier.
• I have placed DITTO at June 2nd, making the authors work only a handful of days earlier, not 3 months after
• I place Tajwar et al. at Apr 22, making the authors work a month later, not a month earlier.

However, I don’t consider the exact timelines here as a crucial component of my assessment of the work. I simply meant to show that several ideas present in this paper were published in work prior work, though not in the exact same form.

There are even other earlier works that do Inverse-RL in the token level MDP, like: Sequence Match: Imitation learning for autoregressive sequence modeling by Cundy and Ermon in 2023 uses Inverse RL in the token-level MDP. ON-POLICY DISTILLATION OF LANGUAGE MODELS: LEARNING FROM SELF-GENERATED MISTAKES by Agarwal et al. which was published at ICLR 2024.

Discussion of SPIN Thanks for bringing this to my attention. I did not read this initially as it was in the appendix. I understand that DPO makes assumptions about the data, and that the authors approach uses a different model than SPIN to generate negatives.

Distinct Focus and Contribution I understand that DITTO is focused on few-shot alignment, and the SPIN / Wulfmeier et al is focus on SFT or alignment. This response left me with some questions, as it is just stated that this work is designed to address “general alignment from demonstrations” which I assume is just learning from demonstrations. This is a big area, and after re-reading the paper I believe some of the author's contributions are new, and some are not. I’ll discuss this more later.

From IRL Foundation to LLM Alignment I agree that the authors have presented new ideas in their work. I just think a large part of the work as it is currently is focused on re-iterating that we can consider each step of generation an MDP, that forward KL = BC and that reverse KL = IRL which can be optimized adversarially. I am interested in the authors new experiments on how to optimize these objectives, and ablations over the way in which the discriminator is learned. I wish the paper focused on these as the contribution, which I believe are novel and useful, rather than the general framework.

Clarifications of Key Contributions

1. Framework: This unified framework for distribution matching has existed for a long time outside of LLMs, and works in LLMs have already used the unified framework, even though it was not explicitly written up.
2. Sec 3.2: I believe this is interesting and is new contribution. I wish the authors spent more time discussing this, and made this more of a core contribution of the work. However, section 3.2 begins only on page 6. I think the work would be much improved if it focused primarily on how to effectively learn a reward function from demonstrations for inverse RL, and ablating and explaining these choices.

Regarding timeline, I respectfully disagree on the author's points: 4. “The first to introduce the idea of LLM alignment from demonstrations” I think this was done in SeqMatch (f-divergence minimization) and perhaps SPIN which were both definitively published before this work 5. “Inverse-RL Perspective” I think this was done by seqmatch and other works which took the distribution matching perspective over occupancy measures, but did not call it InverseRL. 6. Previous works have shown IRL is effective, but I don’t believe this diminishes from the authors contributions which show that IRL can be effective! I think instead the authors should re-focus their work around their contributions.

On Experiments

• I see that the reviewers get better performance using demonstrations than pairwise preferences. I have further questions and weaknesses about this I detail in my new review of the work.
• “We theoretically and empirically highlighted super-demonstrator performance”:

1. Could the authors kindly point me to where the theoretical evidence of super-demonstrator performance is, and where the proof is that SPIN cannot achieve it.
2. Could the authors clarify what they are empirically measuring super demonstrator performance with respect to? For example, doing better than the preferences isn’t evidence of this because the demonstratiosn weren’t generated by the same policy!

Focus on Reward Model Experiments These are interesting, and I appreciate the authors bringing these up again. I have discussed them in my new review of the work.

New Review

The authors present AfD, a general framework for alignment using demonstrations. That unifies SFT and inverseRL by looking at different directions of the KL divergence.

Strengths

• Several works have demonstrated that the quality of the reward model is extremely important to performance. I think the authors' experiments address an interesting and important question.
• I think the experiments in Section 4.2 are very interesting – demonstrating how the choice of which data to use for a reward model impacts down-stream performance.

Weaknesses

• The authors work primarily focuses on their introduced framework for distribution matching in the Token level-MDP. However, 1) this general framework/knowledge has been known for a long time in RL/imitaiton literature, so I am uncertain if this can be claimed as a contribution. 2) even within the landscape of LLMs, several prior works have used the distribution matching framework as justification for their methods, citing literature in imitation learning. This casts doubt as to whether or not the author's proposed unified framework is adding anything. At present, I do not believe the author's unified framework constitutes a novel contribution, as prior works have already used the same ideas, but did not feel it was necessary to present such a framework. I would be happy to see a survey or overview paper on this topic, but I am unsure that ICLR is the best venue for it. I again believe the work would be stronger if the authors focus on the reverse-KL case, and how to best learn a reward model for it. For citations, I refer the author to my response to their rebuttal.

• The authors claim to be doing Inverse-RL with their method in Section 3.2 and Section 4.2, but they learn the reward model / discriminator from a static dataset. Since the inverseRL objective requires the policy to be optimized under its own distribution, its unclear how a static discriminator will actually optimize for the objective the authors claim. This is a bit strange as the authors for most of the work claim that they are introducing a framework for IRL, but then don’t actually seem to do inverse RL in the end.

• The authors perform their experimental evaluation on only a single task. Most contemporary papers have a more extensive evaluation.

• The experimental evaluation raises a number of questions, which I bring up in the “questions” section of my review. I think these questions need to be discussed to gain confidence in the experimental section.

Questions My questions are largely concerning the experimental evaluation.

• The authors use GPT4 to generate demonstration data. However, in Section 4.1 the authors compare finetuning on data generated by GPT4 (SFT AfD, DPO AfD) to finetuning on data from the original dataset (SFT Preferred, DPO Preference). The only conclusion I can draw from this experiment at present is that the GPT-4 data is better, preferred by the gold RM, or easier to fit than the original data. Why is comparing between the original data, and data distilled from GPT-4 a valid comparison on this task? Especially because GPT-4 has already likely been trained to be “helpful and harmless”.

• Again, for Section 4.3 the authors use GPT-4 as a critic. However, the BoN IRL-RM and SFT models were trained on data generated by GPT-4, which GPT-4 likely prefers. Is there a reason for why this is a valid comparison? Would GPT-4 not be biased to its own responses, and thus prefer SFT/BoN IRL-RM to the BT-RM.

• Could the authors show experiments that work on another dataset?

• Could the authors clarify what the theoretical and empirical evidence of extrapolation beyond the demonstrator is as this was repeatedly brought up in rebuttal? The proof of this in Brown et al 2019 made specific requirements on the sub-optimality of the demonstrator, ie there were requirements on the demonstrator’s average performance in relation to the true optimal policy.

• Could the authors explain the relationship between the objective that they actually solve for in the experiments (Init-SFT) versus the theoretical section (Reverse-KL occupancy matching). I understand that occupancy matching can be done with a discriminator/RM, but the distribution on which the discriminator is trained is important.

Concluding Thoughts I still think that a paper which focused on the best way to learn a discriminator from demonstrations would be impactful and insightful. However, at present the work spends a lot of time / focus on other parts, and in my opinion as a result does not give the “learning a discriminator” part proper treatment. Moreover, I still have questions regarding the experimental evaluation which underpins this, as the authors base many of their conclusions off comparisons between models trained on GPT-4 demonstrations and models trained on the original HH data. This seems like it might be problematic, particularly when using GPT-4 as a critic.

We sincerely thank reviewer wScp for their consideration of re-evaluating our paper!

Although the author response window is narrowing down and our response may be limited by the approaching deadline, we deeply appreciate such an opportunity to further clarify the contribution and a few other aspects of our work. In the following, we would respond to the reviewer’s comments grouped by topics.

1. Motivations of introducing the framework before the technical method.

We thank our reviewer for their affirmative comment on our method section (Sec. 3.2)! We agree the explicit reward modeling part, together with the technique we introduced to overcome the reward hacking challenge, are the major technical contributions of our work. However, we would like to respectfully argue the current presentation design of the paper is motivated by the following considerations:

(1). For a self-contained paper.

Providing the prerequisite knowledge of Inverse RL and the MDP formulation is necessary for the follow-up discussion of e.g., distribution matching in our paper.

We understand those prerequisites might be straightforward to our knowledgable reviewers, yet for our general potential readers who are not familiar with MDPs and RL, our work is self-contained such that any reader familiar with the basic concept of LLMs to be able to understand our paper.

(2). Connecting research areas and inspiring future innovations.

Bridging the research area of Inverse RL and Imitation learning with the general task of alignment — including both alignment from preference feedback and demonstrative feedback is useful to the community. While the idea is not totally new by itself and has been discussed in the pre-LLM era (e.g., as we have discussed in the related work section, and pointed out by the reviewer), we believe it is beneficial to highlight its importance in the LLM alignment setups. It is worth letting more people understand the connection between RLHF and AfD through the lens of Inverse RL: RLHF and AfD are two different instantiations of Inverse RL for LLM alignment, the difference is in how to generate the reward signal.

We would highlight such a unified framework can inspire potential future innovations beyond RLHF or AfD — the essential idea is not to use a specific data type for alignment, but more on leveraging different types of datasets to build the reward model. The current focus of the literature mainly focuses on RLHF — using preference-based feedback — except for the few recent exceptions (concurrent work) pointed out by our reviewer.

(3). Disclosing the motivation of the proposed method.

Describing the token generation process using the MDP language is essential because it can be used to formally define the forward and inverse process. Within such a framework, we are able to formally discuss the alignment problem using the RL and Inverse RL perspective, introduce the distribution matching methods that have been applied in the literature, and adapt those methods according to the property of the AfD task (Sec. 3.2)

2. Why explicit reward modeling can be superior to implicit methods (e.g., SPIN, SFT)

(1). On the sub-optimality of demonstration data

We fully agree and are glad to see the reviewer resonates with our high-level insight in reward extrapolation (cf. Brown et al. 2019). The reviewer is very correct in pointing out if the demonstrations are optimal, then there would be no further space for improvement.

In the context of LLM alignment, even the responses generated by GPT4 are far from optimal. On one hand, this is because the concept of helpfulness or harmless themselves may not be transitive. On the other hand, GPT4 will reject to give answer to many queries in the harmless dataset due to its filtering mechanism. Therefore, the key insights of Brown et al. 2019 that extrapolating over reward models can improve over sub-optimal demonstration can be applied in our context.

In our previous response and also in our paper, we highlighted the core difference between the direct alignment methods without a reward model, such as SPIN, which is upper bounded by the demonstration quality, because they always consider the demonstration data to be positive, and the current generation as negative.

This can be proven by contradiction: assuming in SPIN the LLM to be aligned outperforms the expert demonstration, then, in the next generation, the algorithm will treat those (better) generations as negative samples and optimize the policy to decrease the probability of generating such responses — this contradicts with the objective of increasing the probability of generating higher-quality responses.

(2). Policy optimization objective with our explicit reward models

Define

$r_{c}(y|x)=\log \bar{\pi}_{SFT}(y|x) - \log \pi_0 (y|x)$

The policy optimization objective with our explicit reward model becomes

$\arg\max_{n} r(y_n|x) = \arg\max_{n}(\log\bar{\pi}_{SFT}(y_n|x) -\log \pi_0(y_n|x) )$

In such an objective, we can directly calculate the probabilities of generating any $y_n$ from those parameterized models, the parameters in the $\pi_{SFT}$ model are frozen in calculating the probabilities, and the N samples are generated by this $\pi_{SFT}$ model;

To understand the theoretical interpretation of such an objective, we note

$\max_\pi KL(\pi||\pi_0)-KL(\pi||\pi_{SFT})=\max \mathbb{E}_{y\sim\pi}[r_c(y|x)]$

Therefore, optimizing the generation using BoN with regard to $r_c$ (i.e., the order statistics) can be interpreted as simultaneously maximizing the KL divergence between the order statistics and $\pi_0$ and minimizing the KL divergence between the order statistics and $\pi_{SFT}$. This exactly leads to the extrapolation behavior as we desired!

3. On evaluations.

(1). The pursuance of a comprehensive comparison in our work.

Firstly, we would like to highlight the efforts made in our work to make the results reliable — we have used both golden-reward evaluation and GPT4-as-a-judge to provide a comprehensive evaluation of the proposed methods. Our evaluation strictly follows the best practices in the literature.

(2). Motivation for using the HH Dataset (Harmless and Helpful tasks)

Please kindly let us reiterate that the most important motivation for using the Harmless and Helpful datasets is that the open-sourced golden reward models exist on those tasks, and those tasks are well-studied in the literature.

On other datasets such as UltraFeedback, policy evaluation can only be based on GPT4-as-a-judge, which is much more expensive than using open-sourced golden reward models and may suffer from the challenge of why is GPT4 able to do the evaluation. Should the reviewer agree with us among many literature that GPT4-based evaluation is reliable, we are happy to run experiments on UltraFeedback (which can be costly).

(3). Can GPT4 evaluations be trusted?

First of all, we would like to highlight the fact that our evaluation is based on 2 methods, the effectiveness of the proposed method is verified by both evaluation metrics.

“the BoN IRL-RM and SFT models were trained on data generated by GPT4, which GPT4 likely prefers.” We would respectfully disagree with the reviewer on this ungrounded claim — the base models we used are not GPT4, and their generated contents are different from GPT4. More importantly, we would like to highlight that the objective and takeaway of Section 4.3 is about the relative improvement achieved from SFT to Inverse RL, rather than the absolute performance. Therefore, the takeaway of Section 4.3 is isolated from whether or not GPT4 would prefer the contents generated by AfD on some specific demonstrations — as existing literature using GPT4 evaluation did.

Once again, we sincerely thank the reviewer for taking the time to review our paper. Despite the limited time remaining, we are still eager to address any further concerns or questions you may have.

I would like to thank the authors for their response. I found the clarificaitons helpful, but think we are still not aligned on a few points.

Motivations of the Framework I agree with the authors that some background is necessary for the paper. However, my issues with this part of the paper remains as follows:

1. The authors posit in their response and the paper text that this framework is a new contribution. I think we disagree on this point.
2. The length of discussion on the framework leaves less time to discuss the later contributions in text.
3. The presented framework is not well connected with the method the authors propose later in the paper. For example, in their response the authors detail the objective that the experiments optimize for, which is not exactly the reverse-KL. Why is this objective not in the main text of the paper? I think there should be a discussion in the main text.

On Explicit Reward Model I agree with the point that explicit reward modeling can be better. I also thank the authors for detailing the proof by contradiction, and providing the objective. I believe these should be more central components of the paper, as at present they are not reflected within the text.

However, I think a few points are missing from the paper:

• The authors should be explicit that they are only providing a proof of contradiction that SPIN cannot achieve super demonstrator performance. I had the impression that there was a theoretical proof that the authors method could achieve super-demonstrator performance. While empirically this seems to be the case, the distinction between theoretical and empirical results should be clear.
• the steps to go from reverse KL to the authors objective should be in the paper (I think I see how it is done)

Evaluations

1. The distinction I am making is that while other works use GPT-4 as a critic, they do not necessarily use GPT-4 to generate the data. A more grounded approach in my mind would be to take a demonstration / SFT dataset, and label preferences between them. Then the demonstrations and preferences come from the same distribution, and one is not biased towards the evaluator. There have been several works (including DITTO) which note the bias of GPT-4 as a critic when judging its own outputs.

2. Other datasets: the authors might consider showing their results on other common datasets used for SFT, or even datasets in preference based learning, like TLDR or even IMDB sentiment. Again, the key distinction I am making is that the authors experiments use GPT-4 to generate the demonstrations, then use GPT-4 as a critic. I am arguing that using demonstrations that are not generated by GPT-4 would be more reliable when evaluating with GPT-4.

3. I might be mis-understanding the draft. However, I was basing this discussion off of this statement: “Demonstrations were generated using the OpenAI GPT-4 API, with detailed prompting strategies…” where the prompting strategy inputs the dataset and asks GPT-4 to complete it. Then, the AfD models use these GPT-4 demonstrations for reward learning. In this sense, the AfD models are trained on data generated by the critic, while the baselines are not. Please correct me if I am wrong.

Overall Recommendation I agree with many of the points the authors bring up in their responses, and find the responses very useful. My recommendation remains that the authors re-vamp the writing of the paper to focus less on the “framework” such that their method can be adequately explained within the main text of the paper. Several details and contributions discussed here would make the main body of the paper much stronger, but are omitted because of space.

It is disappointing to observe a reviewer making claims that contradict the actual content of the paper, especially while having a closely related submission of their own. Intentional misrepresentation undermines the integrity of the review process. The community deserves better.

5. Policy optimization

We thank the reviewer’s interest in the results with parameterized policy optimization methods (e.g., PPO). In our manuscript, those results were deferred to Appendix E.7.

In Figure 6, we conducted policy optimization using PPO and compared the results with those in BoN. We find that PPO outperforms BoN with a small number of N (=50) and can achieve on-par performance with BoN with N = 500.

6. Golden reward and win rate

Yes, if we use the golden reward model rather than the learned ones to pick the best response out of N responses and evaluate it, it will outperform all other competitors and achieve a win rate of 1.0

7. The ablation study results in Figure 4

In Figure 4, we experimented with different reward modeling choices to highlight the challenges analyzed in Section 3.2 — the heterogeneous problem (also known as reward hacking) in reward modeling.

When the curves go down, it means using those reward modeling choices can not lead to an effective reward model — those reward models hacked the discriminative tasks and they are not good choices for reward modeling.

To be more explicit, this is because those reward models focus on the incorrect aspect of responses in identifying whether they are positive samples or negative samples. Therefore, optimizing toward increasing those reward values can not lead to an increase in the golden reward values.

8. Discussion on SPIN / Direct methods

Thank you for the suggestion, and we are glad to hear that our insights resonate with you. We have moved the discussion on SPIN to follow Section 3.2, as we agree that this adjustment enhances the clarity of our ideas and better distinguishes our work from related studies.

Once again, we would like to thank the reviewer for their insightful comments and thorough reading to improve our paper. Should there be any leftover concerns, please don’t hesitate to let us know and we will do our utmost to address them!

We thank the reviewer for their encouraging comments on our presentation and for their insightful comments. We would respond to each of the concerns and questions in turn:

1. MaxEnt RL / KL Constraints

Thank you for highlighting this point! In Section 2.2, we intentionally omitted the notion of KL constraints to maintain conciseness and simplify the notation. KL constraints or MaxEnt regularization can indeed be interpreted as an additional (omitted) objective beyond alignment in this context.

In this work, we avoided incorporating KL terms in both entropy constraints and alignment objectives to minimize potential confusion for readers. However, integrating our method and framework into a MaxEnt/KL-regularized setting is a promising future direction, given the demonstrated successes of such approaches.

2. Related Work

We thank the reviewer for sharing the related work discussing the intransitivity of RLHF! We have added the discussion in our revision (please see Page 1, line 45 in our updated manuscript).

3. SFT = BC = action matching is well known, but BC = action matching = trajectory distribution matching is new

From an RL perspective, we agree with the reviewer that equating SFT to BC is not surprising.

However, in RL literature, BC is typically understood as "action (marginal) distribution matching," which is known to suffer from the compounding error problem. Interestingly, we highlight that in the context of LLM alignment, BC goes beyond action distribution matching to also include trajectory distribution matching. This arises from the deterministic concatenation of sentences and tokens during generation (i.e., state transitions). This unique aspect allows us to unify SFT and IRL-based AfD methods within a single distribution matching framework.

We believe this novel equivalence between marginal distribution matching and trajectory distribution matching is an important contribution to the community.

Furthermore, this equivalence, along with the use of forward KL in establishing it, naturally motivates our exploration of using reverse KL for distribution matching within the same framework. This enables us to explain the differing alignment behaviors and properties induced by these distinct distribution-matching objectives.

We thank the reviewer iqxv for their further feedback!

On BC and Forward KL.

We thank our reviewer for raising their further question on the relationship between BC and the trajectory forward KL. To better support the discussion, we would like to formally highlight the differences below:

In BC, the objective is forward distribution matching of actions, i.e.,

$J_{BC}(\pi)=-\mathbb{E}_{(s,a)\sim \beta } \left[\log(\pi(a|s)) \right]$

In general trajectory distributional matching, the objective is

$J_{FKL-\tau} = KL(d^\beta(\tau|s_0)||d^\pi(\tau|s_0))= \mathbb{E}_{(\tau|s_0)\sim\beta} \left[\log d^\pi(\tau|s_0) \right]$

where we use $\tau |s_0$ to denote the trajectories starting from state $s_0$: $\tau |s_0 = \\{s_0, a_0, s_1, a_1,...\\}$. In general, such an objective is intractable and it is different from $J_{BC}(\pi)$ since the calculation of the trajectory density requires access to the transition probability of $p(s'|s,a), \forall s, a$.

In the MDP of LLM token generation (in alignment), we know $p(s'|s,a) = 1, \mathrm{for}~ s'=\mathrm{concate}(s,a), \mathrm{and}~ 0 ~\mathrm{otherwise}$. Therefore, with this specific context can we show the equivalence between trajectory distribution matching using forward KL and BC. (cf. Equation (6) in our manuscript).

In the IRL literature, it has been argued that the forward distribution matching on occupancy measure (joint distribution of state-action pairs) is different from the distribution matching of actions (BC) [1, 2]. Different from the IRL literature, we study trajectory matching instead since responses as trajectories are more meaningful than tokens in the context of LLM alignment. Another derivation that might be interesting to our reviewer is that --- if we follow the RL literature to consider the occupancy measure matching (i.e., matching the distribution of incomplete sentences), we will have the following objective with forward KL:

We thank the reviewer for their time and thoughtful review of our paper, as well as their encouraging recognition of our innovation in providing an IRL framework for LLM alignment. Below, we would like to address their concerns and questions in turn:

Q1: Evaluation with golden reward models

We thank the reviewer for highlighting the need for further clarification regarding our motivation for using golden reward models in evaluation.

1. Open-Source and Reproducibility On one hand, evaluating with golden reward models is significantly more cost-effective than relying on commercial APIs like GPT-4. Moreover, leveraging open-source reward models enhances reproducibility and facilitates fair comparisons in future research. This approach is a well-established practice in the reward modeling literature [1–6].

2. Evaluating with GPT-as-a-Judge That said, we agree with the reviewer that incorporating GPT-as-a-judge provides valuable complementary insights into evaluating the performance of various methods. To this end, we included GPT-based evaluations in Section 4.3 (Table 2). Notably, the results obtained from GPT-based evaluation are broadly consistent with those derived from golden reward models.

3. Using both evaluation methods as cross-validation Recent studies have raised concerns about the reliability of LLM-based evaluations [7–10]. To address this, we employ multiple evaluation metrics, combining golden reward models with GPT-as-a-judge. This dual approach enables a more comprehensive and reliable assessment of our proposed method.

We hope this dual evaluation strategy underscores the rigor and thoroughness of our assessment and addresses the reviewer’s concerns.

Q2: A More Direct Comparison to SPIN.

We thank the reviewer for suggesting that a comparison with SPIN in the main text would be more informative than including it solely in the appendix. In the original manuscript, discussions and comparisons with SPIN were deferred to Appendix A.

In the revised version, we have moved the discussion and the illustrative example to the main text, with updates highlighted in blue.

1. Achieving Super-Demonstrator Performance: A Fundamental Limitation of SPIN

A key distinction between our method and SPIN lies in the flexibility and potential of our approach to achieve super-demonstrator performance in LLM alignment, as empirically validated in our experiments.

SPIN operates under the explicit assumption that the current policy (initially the SFT policy) is always weaker than the demonstrations. Consequently, at convergence, the aligned LLM's performance is upper-bounded by the performance of the demonstration dataset, as the demonstrations are consistently treated as positive examples in implicit reward modeling.

In contrast, our method, rooted in Inverse RL, explicitly learns a reward model and extrapolates over it. This reward modeling mechanism allows for leveraging task scores to enhance performance beyond the demonstrators. On the other hand, naively adopting SPIN's setup—using the SFT checkpoint's generations as negative examples and the demonstrations as positive examples—can adversely impact heterogeneity and lead to suboptimal performance. As demonstrated in our experiments, the IRL approach consistently achieves super-demonstrator performance across both tasks.

2. Empirical Comparisons with SPIN

To substantiate our analytical insights, we provide empirical comparisons against SPIN in Table 4 of Appendix A.5, which we now summarize in the table below:

The table reinforces our key insight: while SPIN focuses on imitating expert demonstration behavior, our IRL-based approach surpasses the demonstration performance, achieving significantly higher scores than the demonstrators.

By moving this discussion and comparison into the main text, we aim to provide a clearer understanding of the advantages of our method over SPIN.

Once again, we would thank reviewer ZCzk for their time and effort devoted to reviewing and improving our paper.

In our author response posted 11 days ago, we have addressed each of our reviewers' concerns in a detailed, point-by-point manner. We hope the responses have addressed the outstanding questions and the reviewer would consider raising their score if those questions have been appropriately answered.

As the author-reviewer discussion phase nears its conclusion, we would like to kindly ask if there are any remaining concerns or additional feedback that we can address. In the limited time remaining, we are committed to making every effort to resolve any outstanding issues!

We thank the reviewer for their time and thoughtful review of our paper. As the author-reviewer discussion window is narrowing to its end, we would like to summarize the reviewer’s main concerns, and how our previous responses addressed those concerns.

1. Motivations of golden reward model evaluation

In their original review, our reviewer ZCzk pointed out that it would be helpful to include further clarification regarding the motivation for using golden reward models for evaluation.

In our response, we highlighted the motivations of Open-Source and Reproducibility; and Comprehensive evaluation methods using both the golden reward model and GPT-as-a-critic.

2. A more direct comparison to SPIN.

In their original review, reviewer ZCzk recommended including a more direct comparison to SPIN.

In our response, we

• 1. Discussed and compared our method with SPIN in Appendix A.
• 2. Moved the discussion and the illustrative example to the main text in our revised manuscript
• 3. Highlighted the key difference between our method and SPIN is the potential of Achieving Super-Demonstrator Performance
• 4. Empirically, we provided the following empirical results in Table 4 of Appendix A.5.

3. Difference between RLHF and AfD.

In their original review, reviewer ZCzk asked questions centering on the difference between RLHF and AfD.

In our response, we highlight that AfD and RLHF differ in the dataset they work with — AfD works with expert demonstration dataset, while RLHF works with pairwise preference dataset. While RLHF can suffer from the difficulty of noisy labels, high annotation costs, and privacy concerns, AfD does not suffer from those challenges and can effectively build reward models and align LLMs.

Once again, we thank our reviewer for their insightful comments in improving our paper. In the limited author-reviewer discussion period, we are still eager to address any further concerns from our reviewer.

We thank the reviewer for their time and thoughtful review of our paper, as well as their encouraging recognition of our contribution to streamlining LLM alignment tasks through a distributional matching perspective. Below, we address the concerns and questions raised in turn:

1. Details for building reward models

In our implementation, we use Equation (11) to train a discriminator, where the logits of the discriminator correspond to the reward values as defined in Equation (12). Specifically:

$D_\phi(y|x) = \sigma(`logits`(y|x)) :=\sigma( r(y|x))$

here, $D_\phi(y|x)$ is a binary classifier trained to distinguish samples generated by the SFT model from those produced by the initial model.

To aid readers in understanding our implementation, we provide Algorithm 1 in Appendix E.1.

Please let us know if there are further specific questions about any aspect of the reward modeling implementation.

2. Objective of Section 4.3

We appreciate the reviewer for highlighting the need for clarification regarding the objective of Section 4.3.

The primary goal of Section 4.3 is to evaluate our proposed method using GPT-4 as a judge—a widely adopted and complementary evaluation metric. In our work, we employ a dual evaluation approach, utilizing both golden reward models and GPT-4 as evaluative benchmarks. The rationale for this dual approach is as follows:

• Golden Reward Models: Evaluating with golden reward models is significantly more cost-effective than relying on commercial APIs like GPT-4. Moreover, using open-source reward models enhances the reproducibility of our work and enables fair comparisons in future research. This practice aligns with established standards in the reward modeling literature [1–6].
• GPT-4 as a Judge: Incorporating GPT-4-based evaluation provides additional insights into the performance of various methods, complementing golden reward evaluations. As shown in Section 4.3 (Table 2), our results indicate that GPT-4-based evaluations are broadly consistent with those derived from golden reward models.

To address this point, we have revised the description in Section 4.3 to improve clarity and ensure our evaluation approach is well-articulated.

3. Why Our Work Falls into the IRL Method Category.

We thank the reviewer for raising the question about method categorization. In the RL literature, both Imitation Learning (IL) and Inverse RL (IRL) aim to learn a policy from a demonstration dataset, typically involving trajectory distribution matching [e.g., 7,8]. The key distinction is that IRL explicitly learns a reward model, while IL does not.

• In IL, the objective is to match the behavior of the learned policy with the demonstrator.
• In IRL, an intermediate step involves building a reward model from the demonstration dataset, which is then used to optimize a policy to match—or even surpass—the demonstrator's performance.

Due to space constraints, we deferred the detailed discussion of these differences to Table 3 and Appendix A in the manuscript.

In our work, we explicitly build a reward model using the demonstration dataset, hence the method falls to the category of IRL rather than IL. As has been demonstrated in our experiments, such an explicit reward modeling step is essential for achieving super-demonstrator performance, rather than merely matching their performance. (cf. Figure 4.)

We thank the reviewer for their time and thoughtful review of our paper. As the author-reviewer discussion window is narrowing to its end, we would like to summarize the reviewer’s main concerns, and how our previous responses addressed those concerns.

1. Building reward models

In the original review, reviewer 5fua asked about the details of reward modeling.

In our response, the following explanations have been provided:

• 1. Detailed the implementation using Equation (11);
• 2. Explained why the logits of classifiers could be used as reward signals;
• 3. Provided Algorithm 1 in Appendix E.1;

2. Section 4.3

In the original review, reviewer 5fua raised their concern on the objective of section 4.3 (experiment section where we use GPT4-as-critic for reward model evaluation)

In our response, we highlighted the objective of leveraging a dual evaluation approach in our work is to enhance the reliability our conclusions.

In addition, as per suggested by Reviewer wScp, we additionally experimented using the GPT3.5 demonstrations (using golden reward models for evaluation) and UltraFeedback (using GPT4o for evaluation). All of our experimental designs aim at enhancing the reliability of the conclusions in our work.

3. Link to IRL literature

In the original review, reviewer 5fua raised the question of Why this paper falls into the IRL method category.

In our response, we highlighted the usage of an explicit reward modeling step in our method making our method an IRL algorithm. To further enhance the clarity, we discussed the key distinctions between Inverse RL and Imitation Learning. We would refer to Table 3, Figure 4, and Appendix A for more details in our updated manuscript!

4. On the forward and reverse KL

In the original review, reviewer 5fua requested an additional discussion on the forward and reverse KL.

In our response, we discussed the key insight of using the forward and reverse KL in our distributional matching objective from the following 4 perspectives:

• 1. Leveraging SFT-format data beyond SFT.
• 2. Mode-seeking behavior with learned reward models.
• 3. Enabling inference-time optimization with reward models.
• 4. Achieving super-demonstrator performance.

5. Workflow in AfD and RLHF

In the original review, reviewer 5fua raised a question on comparing AfD and RLHF.

In our response, we answered this question first by highlighting that AfD is an alternative to RLHF rather than an intermediate step. We then provided a comparison table that summarizes the key difference between the workflows of AfD and RLHF: Both methods have Reward Modeling and Policy Optimization stages, AfD works on demonstration data, and RLHF works on pair-wise preference data.

6. Citation format

We thank reviewer 5fua for pointing out the citation commands. In our updated manuscript, we have updated the reference format.

Once again, we thank our reviewer for their insightful comments in improving our paper. In the limited author-reviewer discussion period, we are still eager to address any further concerns from our reviewer.