Direct Language Model Alignment from Online AI Feedback

We reply to each of the review’s concerns below.

My first concern is regarding the novelty of the paper. It seems that the language model annotator is essentially a preference model. Therefore, OAIF can be seen as a method of online direct alignment algorithm with access to a preference model. The author mentioned several previous work with on-policy generation and online feedback but in need of a reward model. How is OAIF different from different from these method if we simply plug in the language model annotator as the reward model in their methods?

Please note our work is novel in the following aspects:

• We firstly propose AI feedback to address the offline and distributional shift issue of DAP methods
• We firstly demonstrate the adaptability of alignment via prompting AI feedback model (Section 4.6)
• We firstly show the feasibility of weak-to-strong alignment via OAIF (Section 4.7)

At line 118 the author pointed out that RM might suffer from distribution shift because the training data of RM might not share the same distribution with $\pi_\theta$. However, it seems to me that using language model as preference annotator cannot bypass this problem since the language models' pretraining corpus or the finetuning corpus relating to preference labeling has a similar distribution with $\pi_\theta$.

This is a great question! We assume the large-scale pretraining/finetuning and model scaling equips LLM with strong generalization, which could generalize over responses from different policy models when used as an AI annotator. This has been evidenced by the Reward Bench (https://huggingface.co/spaces/allenai/reward-bench, see Gemini 1.5 Pro performance based on zero-shot prompting) results where LLM achieves comparable and even better performance than dedicated reward models by just prompting.

How is OAIF's performance compared to other online methods like RSO and IterativeDPO? I think that these methods might also be included as baselines since reward model can also be taken by AI annotators.

Please notice that OAIF is orthogonal to RSO as RSO develops the sampling strategy rather than preference labeling, and we consider iterative DPO as a concurrent work.

We reply to each of the reviewer's concerns individually below.

The improvement by extending online is under expectation as it introduces more datasets and training budgets.

We may disagree with this argument as it is unclear how much improvement would be considered as “expected”. We showed empirical results, from both human and LLM, that OAIF performs better than several strong baselines, which we believe are solid evidence to support the superiority of OAIF.

The contribution is limited. The only difference compared to the prior method is substituting the reward model of prior methods (Iterative DPO) to LLMs, though I agree the explicitly static reward model may introduce the model distributional shift problem.

We consider the distributional shift problem, an overlooked problem in the literature, as one major bottleneck when applying DAP methods, and we consider iterative DPO as a concurrent work.

Besides, we’d like to remind the reviewer that the proposal of online learning with AI feedback is just one of our contributions. The high controllability of OAIF via prompting LLM (as shown by the length control experiments in Section 4.6) and the feasibility of weak-to-strong alignment in Section 4.7, we believe, are solid contributions to the community.

Some drawings or comparisons are not fair enough. (a). Table 1 explicitly avoids the limitation of this method by leveraging the feedback from LLM, though it is another variant of the "reward model". (b). Figure 3, the training step is not an approximate x-axis as the online DPO variant has been heavily fine-tuned offline.

As stated in your comments, “the explicitly static reward model may introduce the model distributional shift problem”, which highlights the significance of leveraging feedback via large off-the-shelf LLMs, which are not specialized on generations from a single policy.

We don’t understand the question (b). We perform OAIF based on SFTed models rather than offline DAP models. Could the reviewer clarify their point?

There are no theoretical foundations, or new plausible explanations, aside from more datasets and the online budget, for the further improvement of the online variant DPO.

First, we’d like to highlight that our work strictly follows the theoretical foundations of the corresponding DAP methods, since online learning itself doesn’t break the foundation.

Secondly, we consider our work as an empirical study. We believe our empirical evidence is solid enough to support all our claims.

Thanks for identifying our contribution to addressing the off-policy sampling issue via the proposed online AI feedback (OAIF) method. We have addressed each concern as described below.

The rationale for why on-policy learning brings performance gains is not well clarified. The cited reference [1] does not provide strong support for this claim. There is no experimental evidence that on-policy sampling encourages exploration.

Our empirical analysis in Figure 8 (Appendix) demonstrates a distributional shift for off-policy responses, where off-policy responses get significantly lower generation probability than on-policy responses. Intuitively, using off-policy responses enforces the model to adapt to these examples apart from learning the preferences, which increases the learning difficulty.

Most experiments are conducted with the closed-source LLM Palm; evaluating state-of-the-art open-sourced LLMs would enhance generalizability.

Please note PaLM is available to use and can be fine-tuned on Vertex. We are also aware of a reproduction of this work by HuggingFace which supports our concolusions and will be announced soon.

It is unclear how much of the performance gains are due to on-policy sampling versus online feedback.

We included a comparison between on-policy sampling (with a fixed reward model) and OAIF in line 265 and 266 of Section 4.4, which shows that OAIF obtains significantly better performance.

The reasons why utilizing online on-policy data can avoid overfitting and improve performance should be further analyzed and discussed.

Thanks for this suggestion! We consider that the online feedback to dynamically sampled on-policy responses offers timely signals for the learning which helps avoid overfitting to some extent. Please also note that OAIF allows high and straightforward controllability on the alignment by modifying the prompt as demonstrated by the length control experiments (Section 4.6), and enables weak-to-strong generalization (Section 4.7).

Is it correct to categorize RSO and iterative DPO as on-policy generation in Table 1?

Yes, both RSO and iterative DPO use on-policy sampling, i.e. the responses are sampled from the current policy model during training.

What new opportunities and challenges arise when applying DAP to online settings? Did you encounter common issues of DAP methods, such as overfitting, in the online setting? What are the differences in these issues between online and offline settings?

These are great questions! We believe online learning is the right way to unlock the potential of DAP algorithms, but how to select responses and collect preference data from the current policy model and the feedback model during online learning can be challenging.

Online DAP, like OAIF, still suffers from overfitting, while the behavior and bias are different. For example, offline DPO often shows clear overfitting patterns with decreased win rates against the SFT baseline; OAIF, in contrast, often produces much longer outputs, which we consider as a type of overfitting. However, such overfitting is adjustable by modifying the AI feedback prompt.

Where are the experimental results to support the superiority of using LLMs over RMs to provide online feedback in Line 267?

We performed automatic evaluation by Gemini Pro and showed that OAIF outperforms LLM+RM with a win ratio of >70%.

Thanks for your response. I would keep my previous evaluation.

We respond to the questions you have as follows.

The idea is straightforward but lacks theoretical proof. The proposed method combines DPO and AI feedback, unlike the constitutional AI paper, which integrates PPO with AI feedback. However, this point is minor. Given the abundance of concurrent work [1-7], the authors should further develop the theoretical analysis of their approach to strengthen their method.

While we appreciate the recognition of our work’s simplicity which we deem valuable for both academia and industry, we need to highlight that our work identifies the offline issue of DAP methods (not just DPO) and demonstrates the significance of online learning via OAIF with solid experimental evidence. We consider this as a convincing contribution to the community rather than a “minor point”; these concurrent works listed further supports the significance of the problem investigated in our work.

We are surprised by the argument that our work lacks theoretical proof, since we didn’t develop new DAP algorithms but make their learning online and on-policy. Take DPO as an example. The equivalence between the optimal policy and optimal reward function is built on online and on-policy learning; but traditional DPO often adopts preference data collected and fixed ahead of the training and even from other policy models, leading to offline and even off-policy learning. Our work ensures the optimality of these DAP algorithms.

Different methods should use an equal amount of training data. In the second epoch of onlineDPO, although the prompts remain the same as in the first epoch, the responses and rank information differ due to online generation.

The ability of leveraging newly generated responses and new rank information is exactly the strength of OAIF, which is infeasible for offline DAP methods.

We believe this should be considered as an advantage of OAIF rather than a weakness.

Recent results on Reward Bench indicate that small reward models are more effective than LLM critiques. The iterative DPO methods are similar to OAIF DPO. A performance comparison between OAIF and various iterative DPO methods using cheaper reward models, as both address the off-policy issue, is essential and should be included.

We also noticed recent results on Reward Bench, particularly that generative LLMs perform on-par with and even better than dedicated reward models. This is exciting evidence supporting the reliability of AI feedback thus the potential of OAIF.

A comparison between reward model and OAIF has been provided in line 265 and 266 of Section 4.4. Our experiment showed that the reward model (trained at once) significantly underperformed OAIF from a generative model.

We also highlight that iterative DPO requires retraining reward models frequently, which not only complicates the training process but also incurs significant cost, making the “cheaper” argument debatable. In contrast, OAIF supports quick reward prototyping as shown in our length controllability experiments (Section 4.6), where using the reward model is non-trivial.