RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Thank you for your thoughtful comments and time spent reviewing our paper. We address each point below:

Public Availability of PaLM 2 Models

Thank you for raising the concern regarding the availability of PaLM 2 models. PaLM 2 models are publicly available through Google’s PaLM API (https://developers.generativeai.google/guide/palm_api_overview) and Vertex AI (https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models), which enhances the reproducibility of our results. We will explicitly mention this in the revised paper to ensure clarity for readers.

Experimenting on More Tasks

We conduct experiments on two more tasks to further strengthen our findings - specifically on question/instruction answering and harmless dialogue generation. For question/instruction answering, we find that RLAIF and RLHF both outperform the SFT model with 54% win rates (statistically significantly different from 50%). For harmless dialogue generation, we find that the RLAIF and RLHF outperform the supervised baseline, with SFT, RLHF, and RLAIF responding with harmless replies 64%, 76%, and 88% of the time, respectively. Across all 4 tasks, we find that RLAIF achieves comparable results to RLHF.

We acknowledge that there are many more tasks that could be investigated, and we believe that the positive results on these four tasks provide a promising indication that RLAIF can achieve performance comparable to RLHF.

Additional contributions to the paper

To enhance the contribution of this work to the research community, we have conducted several additional experiments that expand the scope of our investigation. Specifically:

• We have compared RLHF and RLAIF on a new task: harmless dialogue generation using Anthropic's Helpful/Harmless preference dataset
• We have also investigated the performance of RLAIF with an AI labeler of the same size as the policy
• We have experimented with bypassing the reward model by directly using feedback from a LLM
• We have conducted experiments combining RLHF and RLAIF

These additional experiments will be incorporated into the revised paper, providing a more comprehensive and impactful contribution to the field.

Thank you for your time and for reviewing our paper carefully. We greatly appreciate your thoughtful and detailed suggestions. We address your comments below:

Relationship Between AI Labeler Alignment, RM Accuracy, and Downstream RL Policy

We acknowledge the importance of thoroughly understanding the relationship between RM accuracy and downstream RL results. To address this concern, in the revised paper, we will include the RM accuracies for both human feedback RMs and AI feedback RMs in Section 4.1. Additionally, we will report the RM accuracies for the two RMs trained on different AI labels in Section 4.6, titled "End-to-End Sensitivity to AI Labeler Alignment.”

While we recognize the importance of understanding the relationship between RM accuracy and downstream RL results, we believe this is a broad research question deserving a comprehensive study of its own. Additionally, the nature of RL fine-tuning and human evaluation demands substantial resources, making an in-depth exploration of this relationship outside the scope of our work. Nevertheless, we recognize the significance and potential of this research direction.

We would also like to emphasize that all RMs are trained following a standardized procedure. This consistency in training procedure of the RM does not favor any specific approach, allowing for a fair evaluation of the different methods employed.

Human Evaluation Statistics

To answer your questions about human evaluation, we constructed a set of 2k rating instances across all of our RL experiments, which includes evaluations for additional experiments that we are adding to the revised paper (see last section of this reply). Each instance comprised a single context and three distinct model responses (e.g., RLAIF, RLHF, SFT), resulting in a total of 6k unique input-output pairs subjected to human evaluation. Additionally, each instance was assessed by three independent raters, which we will use to calculate inter-annotator agreement. These details will be incorporated into Section 3.4 "Human Evaluation" of the revised paper to enhance the transparency of our evaluation methodology.

Length-Controlled Results

You raise an excellent point about calculating the length-controlled results for RLHF vs. RLAIF directly. We will calculate this quantity and incorporate the result in Appendix F.

Other Questions

Are the SFT baselines using only the preferred responses in the datasets?

We do not conduct SFT on the preferred responses. For summarization, we solely train on the supervised data from the TL;DR dataset (note that this is distinct from the TL;DR preference dataset). For other tasks, we do not conduct SFT at all. We will make this clear in the revised paper.

Is there a reason why the reward models are also initialized from the SFT models (described in Section 3.3)? Their output spaces are different. Is it a random try or based on some statistics?

We initialize the RM from the SFT model because we have found that initializing the RM from the SFT model improves RM performance. The hypothesis behind this is that the RM must be able to adapt to the scoring task, and the SFT model adds more domain-specific training. This is also an observation shared by other researchers - see https://arxiv.org/pdf/2009.01325.pdf, https://notesonai.com/RLHF+-+Reinforcement+Learning+with+Human+Feedback, and https://huyenchip.com/2023/05/02/rlhf.html.

What is the baseline used for REINFORCE (mentioned as “we use REINFORCE with a baseline” in Section 3.3)? Is it the output of the value model in the authors’ setup?

As you mentioned, we use the value function as the baseline in REINFORCE. See Appendix C for details.

Lastly, we will implement your recommendations on “Presentation suggestion” and fix the point regarding section 4.2. Thank you for pointing these out.

Additional contributions to the paper

To enhance the contribution of this work to the research community, we have conducted several additional experiments that expand the scope of our investigation. Specifically:

• We have compared RLHF and RLAIF on a new task: harmless dialogue generation using Anthropic's Helpful/Harmless preference dataset.
• We have also investigated the performance of RLAIF with an AI labeler of the same size as the policy
• We have experimented with bypassing the reward model by directly using feedback from a LLM
• We have conducted experiments combining RLHF and RLAIF

These additional experiments will be incorporated into the revised paper, providing a more comprehensive and impactful contribution to the field.

Thank you for your time and for reviewing our paper carefully. We greatly appreciate your suggestions and address them to the best of our ability below:

Contribution Value of this Work

We have conducted an additional experiment in which the AI labeler matches the size of the policy and RM. Here, RLAIF surpasses the initial SFT policy, achieving a win rate of 68%. We hope this addresses your initial concern, and we have incorporated this into Sections 3.2 and 4 of the revised paper.

We also believe that our experiments with the larger AI labeler remain valuable to the research community. We agree that it is not entirely unexpected that a weaker model improves based on feedback from a larger model, given that RLHF improves a model with feedback from humans, which can be viewed as a super-capable model. However, the significance of our work lies in quantifying the performance gap between AI and human feedback. Our findings, which indicate that directly replacing human feedback with LLM feedback can achieve comparable results to RLHF, are novel and have not been explored in previous work. We believe this is a valuable contribution to the research community.

As you have suggested, we will also make it clearer that we have two scenarios - one where the AI labeler is larger than the policy, and another where it is the same size.

Qualitative Analysis

In response to your concerns regarding the qualitative analysis, we have implemented three changes.

First, we have initiated human evaluation to assess whether the observed trends exist at scale. If we find evidence of the trends, we will provide hypotheses as to why these trends occur. The results of this evaluation will be incorporated into the revised Section 5.

Second, to reflect the explorative nature of this analysis, we have renamed Section 5 from "Qualitative Analysis" to "Qualitative Observations." This emphasizes that the findings represent observations rather than definitive patterns.

Impact of AI Labeler Alignment on Downstream RL Policy

To address your concern regarding the lack of direct comparison between the two model outputs in Section 4.6 “End-to-End Sensitivity to AI Labeler Alignment”, we have added a head-to-head evaluation of the win rates of the two RLAIF policies. We find that the policy trained with higher preference label alignment (78% alignment) achieves a win rate of 59% over the policy trained with lower alignment (76% alignment). We will incorporate these results into the revised paper.

We also acknowledge the inherent subjectivity and noise associated with human preference ratings. To mitigate this issue, we employed three raters to evaluate each rating instance. Additionally, the prompt provided to the AI preference labeler closely mirrors the instructions given to human evaluators, leading us to expect a strong correlation between AI Labeler Alignment and the final policy's performance.

Despite these measures, we recognize the possibility that higher AI Labeler Alignment may not necessarily translate to improved downstream performance. Due to the resource-intensive nature of conducting these experiments and human evaluation, we do not intend to conduct extensive studies to fully explore this relationship, which is deserving of an entire study on its own. However, we will clearly acknowledge that we cannot definitively conclude that higher AI Labeler Alignment consistently leads to better downstream performance.

Framing of Paper

To make it explicit that we did not introduce RLAIF, we have amended the Abstract to directly credit Bai et al.. In the second sentence of the abstract, we write, “RL from AI Feedback (RLAIF) is an alternative solution introduced by Bai et al. that generates preferences using an off-the-shelf LLM in lieu of human annotators.”

Given that our work is the first to study RLAIF in such depth, we believe that a fair compromise is to leave “RLAIF” in the title while updating the abstract to make it abundantly clear that RLAIF was first introduced elsewhere.

Additional Contributions to the Paper

To enhance the contribution of this work to the research community, we have conducted several additional experiments that expand the scope of our investigation. Specifically:

• We have compared RLHF and RLAIF on a new task: harmless dialogue generation using Anthropic's Helpful/Harmless preference dataset
• We have also investigated the performance of RLAIF with an AI labeler of the same size as the policy
• We have experimented with bypassing the reward model by directly using feedback from a LLM
• We have conducted experiments combining RLHF and RLAIF

These additional experiments will be incorporated into the revised paper, providing a more comprehensive and impactful contribution to the field.

Final Word

If we have adequately addressed your concerns, we would greatly appreciate a re-evaluation of your initial rating. Otherwise, please let us know your additional concerns.

We sincerely appreciate your time and insightful feedback on our paper. We value your thoughtful suggestions and address each point in detail below:

The Effect of In-Context Learning on AI Labeler Alignment

We have also conducted experiments on harmless dialogue generation (see last part of reply), where we found that adding exemplars increases AI Labeler Alignment. Our observations suggest that the optimal prompting techniques are task-specific.

We do not believe that low-quality exemplars are to blame for ICL not working. For all tasks, we carefully handpicked high quality exemplars representative of the preference task. We also measured alignment for "Base 1-shot" on summarization using 10 random exemplars, yielding a maximum/minimum alignment of 76.1%/74.4%, none surpassing the 76.1% alignment of "Base 0-shot". We will add these details to the paper.

Interestingly, existing literature suggests that the role of exemplars and chain-of-thought prompting is still not fully understood (e.g., [1] and [2] show that corrupting exemplars and chain-of-thought surprisingly yield performance gains). We agree that deeper understanding of exemplars is an important research area.

[1] Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? - https://aclanthology.org/2022.emnlp-main.759.pdf

[2] Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters - https://arxiv.org/pdf/2212.10001.pdf

Whether Differences in AI Labeler Alignment are Meaningful

We studied the sensitivity of RL-trained policies to the AI labeler accuracy, revealing that a small disparity can lead to a significant difference in win rates. Specifically, a policy trained with preference labels achieving 78% AI Labeler Alignment outperforms one trained with preference labels achieving 76% alignment, securing a win rate of 59% (statistically significantly above 50%). We will report these results in Section 4.6 "End-to-End Sensitivity to AI Labeler Alignment”.

While we cannot definitively conclude that higher AI Labeler Alignment consistently translates to superior RL policies from this one experiment, this result suggests that such a correlation at least exists in certain cases. Due to the expensive nature of conducting these experiments and human evaluation, we agree that further investigations are necessary and defer more comprehensive study to future work.

AI Labeler Alignment vs. RM Accuracy

We acknowledge the importance of thoroughly understanding the relationship between AI Labeler Alignment and RM accuracy. To address this concern, in the revised paper, we will include the RM accuracies for the two RMs trained on different AI labels in Section 4.6 "End-to-End Sensitivity to AI Labeler Alignment.”

We also would like to note that training and evaluation for all RMs are done in the exact same way.

Is Human Eval Aligned with Preference Datasets?

We acknowledge the potential for discrepancies between the human raters who curated the preference datasets and the individuals we engaged for evaluation. Additionally, differences in training or instruction may introduce further variations. To minimize the impact of such mismatches, we meticulously replicated the settings provided in the original papers. Also, as you pointed out, it is possible that the AI labeler may provide more accurate assessments than the humans who initially rated the dataset in some cases.

Incorporating an agreement analysis between our human evaluators and the preferences in the preference datasets is an excellent suggestion. If we can afford additional human evaluation, we will integrate this analysis into the camera-ready version to further substantiate the value of our human annotators' feedback.

Regarding the possibility that annotators pick the longest response, our post-hoc analysis suggests that human evaluators continue to favor RLAIF and RLHF model outputs even after controlling for response length. This post-hoc analysis is similar to the approach employed by Stiennon et al. in their work "Learning to Summarize from Human Feedback" (https://arxiv.org/pdf/2009.01325.pdf).

Additional contributions to the paper

To enhance the contribution of this work to the research community, we have conducted several additional experiments that expand the scope of our investigation. Specifically:

• We have compared RLHF and RLAIF on a new task: harmless dialogue generation using Anthropic's Helpful/Harmless preference dataset
• We have also investigated the performance of RLAIF with an AI labeler of the same size as the policy
• We have experimented with bypassing the reward model by directly using feedback from a LLM
• We have conducted experiments combining RLHF and RLAIF

These additional experiments will be incorporated into the revised paper, providing a more comprehensive and impactful contribution to the field.