Your Weak LLM is Secretly a Strong Teacher for Alignment

W5: Difference with self-rewarding language models

Our work aims to answer a fundamentally different research question from Yuan et al. [1] and bears several distinctions in problem formulation, methodology, and key findings.

Difference in feedback source and mechanism. Yuan et al. start from the response of a relatively strong model, specifically Llama 2-70B, and utilize the same size model itself as both the generator and evaluator of responses. This model assesses its own outputs through an internal LLM-as-a-judge mechanism, effectively serving as a self-rewarding system. In contrast, our work employs a much weaker LLM to provide external rewards from the model as feedback to train a more powerful student model. Our central goal is to explore whether a model with weak capability can provide effective feedback for alignment, which is different from Yuan et al. In the setting of Yuan et al., the size of the student and teacher are consistent, while in our setting the student is weaker than the teacher.

Difference in model training. The self-rewarding model in Yuan et al.'s work undergoes iterative training, refining its evaluation capabilities alongside its response generation. Our method, however, focuses on training the weak LLM specifically for feedback provision, without iterative self-improvement, and then assesses its effectiveness in guiding the student model's alignment.

Difference in key finding. Yuan et al. demonstrate that a strong model can effectively self-align through internal feedback mechanisms. In contrast, our research demonstrates that feedback from a weak LLM can match or even surpass human feedback in certain alignment tasks. This finding suggests that smaller, less resource-intensive models can be effectively employed in alignment processes, offering a cost-efficient alternative to larger models.

[1] Yuan, Weizhe, et al. "Self-Rewarding Language Models." ICML 2024.

W6: Significance w.r.t. RLAIF

We appreciate your perspective. Our study offers several significant contributions that go beyond the scope of RLAIF.

First, while RLAIF relies on pure AI feedback by prompting off-the-shelf models (e.g., GPT-4) to generate preference, our framework operates in a fundamentally different way. As discussed in response to W1, our framework is based on a hybrid approach, where the weak LLM is trained on a task-specific alignment dataset and produces preferences through a reward mechanism rather than simple prompting. Importantly, we find that substituting GPT-4 with a weak LLM in a prompt-based setup, as done in RLAIF, is not effective. Instead, our method is formulated as a semi-supervised alignment framework (Section 3.1), setting it apart from RLAIF’s problem formulation and design.

Second, Section 4.5 of the RLAIF study suggests that alignment improves with larger AI labelers, which may (mis)lead the community to believe that smaller models are inadequate for providing effective feedback. Our findings challenge this notion by demonstrating that weak LLMs can serve as competent supervisors for stronger student models, offering reward signals that enable the student models to perform on par with, or even surpass, those trained with human or GPT-4 feedback. To the best of our knowledge, this is the first time such a finding has been revealed. We believe such a finding is both novel and significant because it points the research community in an unexpected but promising direction. This opens new avenues for future research aimed at refining and expanding the use of weak signals in AI alignment strategies. This contributes to the broader discourse on developing efficient and effective methods for aligning AI systems with desired outcomes.

Lastly, our paper offers more than evaluation---we conduct in-depth understanding on the effectiveness of weak LLM feedback for alignment in Section 5, shedding light on instances where it surpasses human feedback. Such a qualitative understanding is missing in RLAIF, which is our unique contribution. We believe this nuanced understanding is valuable to the research community, underscoring the potential of leveraging weak AI signals for scalable alignment solutions.

We thank the reviewer for the valuable and constructive feedback on our work. We are glad you recognize the importance of the problem and the comprehensiveness of our experiments. We address your comments and questions in detail below.

W1: why frame weak LLM feedback as the "middle ground"?

Thank you for raising this question. We are happy to clarify further. We position weak LLM feedback as a "middle ground" because our framework is inherently hybrid---the weak LLM is trained in a supervised manner utilizing some labeled human preferences, which then generates AI feedback on unlabeled dataset. This positioning highlights a unique balance between scalability and cost efficiency. Human feedback faces scalability challenges, whereas feedback from large LLMs is highly scalable but computationally expensive. Weak LLM feedback provides a practical, scalable alternative that mitigates the trade-offs of both human and high-capacity LLM feedback, which represent two extremes on the feedback spectrum.

W2: 'Weak' or 'small' LLM

That's an insightful point. We fully agree that "weak" and "small" imply different concepts and should be used with precision. In our study, the term "weak LLM" is intended to emphasize the limited capability of the preference model rather than size alone. Although our weak LLMs are indeed smaller in scale, they do not perform at the level of what might be considered a "moderate" reward model. We offer three pieces of evidence to clarify why our "weak LLM" is indeed weak:

• First, our weak LLMs are not trained on extensive, high-quality human preference data. We use only a subset of the inherently noisy HH-RLHF dataset, which is far less comprehensive than large-scale datasets like UltraFeedback typically used to train high-performing reward models.
• Second, as shown in Table 1, the weak LLM (OPT-125M) achieves only 60.6% accuracy (or consistency with human feedback label) on the test set of HH-RLHF, indicating its performance constraints.
• Finally, our evaluation of the weak LLM’s preferences on RewardBench (details in response to W4) further confirms its constrained capabilities.

For these reasons, we believe "weak LLM" accurately captures the intended model characteristics in our study. This also makes our finding novel and interesting, that using feedback from weak LLM can not only match but potentially exceed human feedback in alignment tasks.

W3: Presentation of Section 2

Thank you for the suggestion on the presentation. Section 2 introduces essential mathematical notation and conventions specific to DPO/PPO, which we included to ensure clarity and consistency throughout the paper. While these details may enhance accessibility for a broader audience outside the alignment community, we agree that moving this content to the appendix would allow us to maintain a self-contained reference without diverting attention from our main analysis and results. We will incorporate this change.

W4: Evaluate the weak LLM on the RewardBench

As suggested, we evaluate the weak LLM (OPT-125M) through the RewardBench, and report the results in the table below.

We thank you for the positive and constructive feedback! We are deeply encouraged that you find our work clear, convincing, and likely to have a significant impact on future LLM alignment practices. Below we address your comments and questions in detail.

W1/Q3: Comparison using the DPO instead of using the reward model

You raise an excellent question. Our choice to utilize DPO fine-tuning for weak LLM is grounded in two key considerations.

First, by utilizing DPO, we maintain a consistent and unified framework throughout the entire training process, aligning both the reward model and the target student model under the same optimization paradigm. This consistency facilitates more straightforward integration in practice and is a common basis for the teacher-student framework (where both teacher and student models are optimized for a similar objective). Additionally, by fine-tuning the LLM with DPO, we effectively embed the reward model within the weak LLM itself. As you correctly noted, DPO inherently acts as an implicit reward model, since it's fundamentally formulated as a Bradley-Terry loss.

That said, we recognize that when the weak LLM’s role is limited to producing reward signals, it is feasible to replace it with a traditional reward model. To explore this, we conducted additional experiments comparing DPO-trained weak LLMs with traditional reward models. The results indicate that both approaches yield comparable performance, with both surpassing the human labeler.

Q1: Clarification on introduction

Thank you for the great suggestion. We agree that clarifying the role of the “weaker supervised LLM” will help readers better understand its middle-ground position. We will modify the introduction following your suggestion.

Q2: The preferred answer in the Figure 1

Thank you for the careful read and for catching this issue! We understand how this could be confusing for readers, given the human annotation (following the HH-RLHF label) is potentially incorrect. We will replace it with another example where the preference is clear.

Q4: Why call the "gold" reward model

The terminology of "gold reward" mostly follows the naming convention in literature. For example, previous work [1] referred to a strong reward model's reward as a "gold" reward. The model is considered "gold" due to its comprehensive training on extensive, high-quality preference datasets, enabling it to serve as a proxy for assessing response quality. Considering that the reward for answer quality in the real world is difficult to simulate, we choose to use the SOTA reward model in RewardBench to simulate the gold reward for evaluation.

[1] Scaling laws for reward model overoptimization ICML 2023

Q5: GPT-4 evaluation with the prompt to directly identify the better response out of the two

That's an interesting suggestion. Following your suggestion, we perform the GPT-4 evaluation using the prompt from Prometheus 2 [1], directly asking GPT-4 to identify the better response out of the two. We report the win-rate in the table below on different models. Similar to our main experiments, the weak LLM feedback is based on OPT-125M. The win rate for a model trained with weak LLM feedback approaches 50% for all cases, indicating that its performance competitively matches that of using human feedback.

[1] Kim, Seungone, et al. "Prometheus 2: An open-source language model specialized in evaluating other language models." arXiv preprint arXiv:2405.01535 (2024).

Q6: Default ratio used in experiments

The default ratio used in experiments is 1/2. We have added this to our manuscript - thanks for catching this issue!

Q7: The wrong x axis label of Figure 7

Thanks again for your careful read! We have fixed the x axis label in Figure 7 as $\beta$.

W2/Q1: Swap the positions of the two responses and conduct the experiment.

We completely agree with your concern! In fact, our implementation and evaluation already incorporate the suggested approach: we randomly swap the positions of the two responses with a probability of 0.5 before inputting them into GPT-4. This means that outputs from the model trained with weak LLM feedback and the one trained with human feedback are equally likely to appear in either position. We will make sure to clarify this in our manuscript - thanks again for catching this issue.

Q2/W3: Evaluation on more complex task

As suggested, we additionally evaluate the effect of our framework by testing it on the task of generating code from natural language, using Spider [1], an open-source dataset for SQL code generation. We collected model error outputs paired with ground truth code, which are used as code preference dataset. Then we use the method proposed in Section 3 to collect code preference annotated with weak LLM (OPT-125M). This dataset was compared against feedback generated by GPT-4 using the RLAIF framework.

The performance of the OPT-1.3B student model under different feedback sources is summarized in table below. The results show that feedback from the trained weak LLM significantly enhances the student model's code generation ability compared to directly prompting GPT-4 for code preferences. This demonstrates the ability of weak LLM feedback to generalize effectively to more complex tasks.

[1] Yu, Tao, et al. "Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task." arXiv preprint arXiv:1809.08887 (2018).

Q3: Why is it that training on data labeled by weak LLMs on unlabeled data can outperform the training set labeled by humans?

Thank you for this insightful question. We provide an in-depth analysis of weak LLM feedback quality in Section 4. Specifically, in L388-L414, we examine the quality differences between weak LLM and human feedback. In cases where weak LLM feedback diverges from human feedback, we find that in 44.3% of these instances, the response chosen by the weak LLM has a higher gold reward score. This result suggests that weak LLM preferences are superior to human feedback in nearly half of these cases. As you pointed out, the comparable performance between weak LLM-labeled and human-labeled data can be attributed to the fact that human feedback is inherently imperfect and often contains noise. We include additional qualitative examples in Appendix B to further demonstrate the noise in human feedback.

We thank the reviewer for the thorough and constructive feedback. We are encouraged that you recognize the significance of our work. Below we address your comments and questions in detail.

W1: Difference w.r.t. Burns et al. (2024)

We discuss the differences w.r.t. Burns et al. (2024) in Section 5 (L461-L476). To recap, there are several major distinctions to note.

• [Problem formulation is different] First, as you noted, while Burns et al. focused on simpler tasks like reward modeling and binary classification with scalar or categorical outputs, we tackle the more complex task of language generation and alignment, which involves a significantly more sophisticated output space. This shift in focus introduces a novel exploration of weak LLM feedback for alignment---a direction Burns et al. did not address.
• [Evaluation framework is different] Additionally, while Burns et al. evaluated their models with accuracy metrics suitable for classification tasks, we use a different evaluation framework tailored to the generative nature of the alignment task. This involves using gold reward models and GPT-4 evaluations and performing an in-depth qualitative study to understand the human vs. LLM feedback (Section 4). All of our experimental results and analysis sections are novel.
• [Conclusion is different/opposite] Lastly, we draw an entirely opposite conclusion from Burns et al. In particular, they reported models trained with weak LLM feedback are noticeably worse than those trained with human feedback, especially for reward modeling, our findings challenge these conclusions. Considering that human preferences are noisy and unreliable, accuracy on the preference test set for alignment is a questionable metric. Our works show contrary conclusions when directly measuring the generation performance of the aligned model, and demonstrate for the first time that using feedback from weak LLM can not only match but potentially exceed human feedback in alignment tasks. Contrary to the pessimistic view presented by Burns et al., our results advocate for the promising potential of leveraging weak AI signals for alignment. This opens new avenues for future research aimed at refining and expanding the use of weak signals in AI alignment strategies.

W2: Potential bias in evaluation using gold RM and GPT-4

This is an excellent question, and we address it in three parts.

First, it's important to acknowledge that using model-based approaches for evaluating open-ended generations remains a significant research challenge. Although GPT-4 and gold reward models are not flawless (as shown in our study), they are widely used in the literature due to their scalability and strong correlation with human preferences.

To further enhance the reliability of evaluation, we incorporate multiple top-performing reward models from RewardBench [1]. This consists of several independently trained reward models, helping to mitigate biases and errors from any single model. We present a table where each row corresponds to a different reward model highly ranked in RewardBench. This approach demonstrates that models trained with weak LLM feedback either match or surpass those trained with human feedback across various evaluation metrics, providing a more balanced and comprehensive assessment.

Lastly, we complemented our model-based evaluations with human evaluations to assess the quality of the responses generated by models trained with weak LLM feedback versus those trained with human feedback. The average win-rate across 100 pairs of responses is 52%, closely aligning with our main findings that weak LLM feedback is at least as effective as human feedback.

[1] Lambert, Nathan, et al. "RewardBench: Evaluating Reward Models for Language Modeling, March 2024." URL http://arxiv.org/abs/2403.13787.

W1.4: GPT-4 with more advanced prompts to provide alignment feedback.

This is a great idea! As suggested, we use the more advanced prompt from Prometheus 2 [1] to provide alignment feedback. We report the comparison in the table below, where we measure the performance of the target model (OPT-1.3B) under different sources of feedback. The adoption of a more advanced prompt does improve the average gold reward from 4.52 (RLAIF) to 4.76, though weak LLM feedback remains competitive.

[1] Kim, Seungone, et al. "Prometheus 2: An open-source language model specialized in evaluating other language models." arXiv preprint arXiv:2405.01535 (2024).

Q1/W2: Why the weaker LLM to provide alignment feedback is referred to as a task-specific model? What about the generalization aspect?

The weaker LLM is referred to as a task-specific model because preference datasets are designed with distinct criteria tailored to specific objectives. For example, HH-RLHF defines preferred responses as those that are both helpful and harmless, aligning the model with these particular goals. Conversely, datasets like TL;DR prioritize concise and accurate summarization, reflecting different definitions of "preference." As each dataset encodes its own standards, models trained on these datasets inherently align with the specific tasks they target. This task-specific alignment explains why a model trained on HH-RLHF should not be expected to distinguish preferences for TL;DR, as the two tasks are guided by fundamentally different criteria.

Furthermore, we clarify that our framework is not aimed at building a general-purpose weak LLM. Instead, it is designed for scenarios where a small amount of human-labeled data is available, offering a significantly more cost-effective alternative to full human supervision. For any given task, a weak LLM trained on this subset of human preference data can provide feedback to achieve alignment performance comparable to full human supervision, yielding substantial savings in annotation costs. While cross-task evaluation is an interesting avenue for general-purpose reward models, it is outside the scope of our approach.

Lastly, if the objective is to create a general-purpose labeler capable of generalizing across diverse domains and tasks, a model like GPT-4 would be more suitable. However, there is a tradeoff: general-purpose models such as GPT-4 excel in cross-task generalization but are computationally expensive and can be suboptimal for task-specific alignment. In contrast, our framework gives practitioners an additional choice. When working with task-specific datasets and limited human feedback, our method provides an efficient, scalable alternative, allowing practitioners to decide whether to leverage a general-purpose model like GPT-4 or rely on our framework tailored for resource-constrained settings.

We thank the reviewer for the thorough and insightful feedback. We appreciate that you recognize the importance of the research problem, our clear presentation, and our positioning with respect to prior studies. We address your additional comments below.

W1.1: Differences w.r.t. RLAIF, comparison with RLAIF employing weaker LLMs as the reward model.

We fully agree with you on this point. We already dedicated such a discussion and a systematic comparison in our manuscript L267-L297 and Figure 2(b). For your convenience, we elaborate on the differences below and provide additional experiments.

First, while RLAIF relies on pure AI feedback by prompting off-the-shelf models (e.g., GPT-4) to generate preference, our framework operates in a fundamentally different way. In particular, our framework is based on a hybrid approach, where the weak LLM is trained on a human-labeled alignment dataset and produces AI feedback through inherent reward mechanisms rather than simple prompting. Our method is formulated as a semi-supervised alignment framework (Section 3.1), setting it apart from RLAIF’s problem formulation and design. Importantly, we find that substituting GPT-4 with a weak LLM (OPT-125M) in a prompt-based setup, as done in RLAIF, is not effective.

Moreover, Section 4.5 of the RLAIF study suggests that alignment improves with larger AI labelers, which may (mis)lead the community to believe that smaller models are inadequate for providing effective feedback. Our findings challenge this notion by demonstrating that weak LLMs can serve as competent supervisors for stronger student models, offering reward signals that enable the student models to perform on par with, or even surpass, those trained with human or GPT-4 feedback. To the best of our knowledge, this is the first time such a finding has been revealed to the research community. We believe such a finding is both novel and significant because it points the research community in an unexpected but promising direction. This opens new avenues for future research aimed at refining and expanding the use of weak signals in AI alignment strategies. This contributes to the broader discourse on developing efficient and effective methods for aligning AI systems with desired outcomes.

W1.2 Higher-quality preference dataset

As suggested, we conduct additional experiments by employing the revised version of HH-RLHF [1] for evaluation. We select the high-quality samples that are in the top 70 percentile in terms of the mean preference difference. We report the gold reward of the target model (OPT-1.3B) trained with the weak LLM feedback (OPT-125M) and human feedback. Our conclusion holds consistently on such higher-quality human preference dataset, where using a weak LLM to provide a preference label for alignment can match or even exceed the performance of using full human feedback.

[1] Wang, Binghai, et al. "Secrets of rlhf in large language models part ii: Reward modeling." arXiv preprint arXiv:2401.06080 (2024).

W1.3: LLMs with larger sizes

We agree on the importance of validating our findings across different model sizes. We already provide the results with larger-size target models in Figure 2(a) and Figure 3(a), including OPT-1.3B, OPT-2.7B, OPT-6.7B, OPT-13B, Mistral-7B, Llama-2-7B, and Gemma-7B. Additionally, we provide results in Figure 2(b) using larger-size models as the supervisor, including OPT-1.3B, Llama-3-8B, and GPT-4. Due to computing resource constraints, the largest model we were able to train is of size 13B.

During rebuttal, we conducted more experiments by taking the Mistral-7B as the target model, and comparing its performance trained with different sources of feedback: OPT-125M, OPT-1.3B, OPT-2.7B, and human feedback. Our conclusion holds consistently that using a weak LLM, with a size as small as 125M, to provide preference feedback for alignment can match or even exceed the performance of using pure human feedback.

Thank you very much for the detailed responses and the extensive additional experiments, which have addressed some of my concerns. However, I still have two major concerns:

1. In your response to Q1/W2, you explained that, compared to GPT-4, you approach trains a weaker LLM on task-specific data to provide better signal for aligning with particular objectives. Thus, your weaker LLMs yield better performance on specific domains while GPT-4 would be more suitable in general-purpose settings. I think this explanation reasonable, however, this scope and limitation were not clearly stated in your paper, which risks misleading readers. Furthermore, if your method is only effective in specific scenarios, its contribution and significance may appear less compelling compared to general-purpose approaches like RLAIF.

2. While I agree that using a weaker LLM has potential efficiency advantages, I remain skeptical of its claimed superiority in effectiveness. The supplementary experiments provided additional evidence:

• GPT-4, with advanced prompts, shows promising results that are comparable to those of the weaker LLM.
• When evaluated on higher-quality human-labeled data, the observed improvement is much less pronounced.
• Not all evaluation metrics consistently support the superior performance of the weaker LLM.
• The effectiveness of weaker LLM also increases as the scale gets larger, from OPT(125M) to OPT-1.3B and Llama-3-8B.

Moreover, the paper does not include comparisons where a large-scale LLM is trained in your framework as the supervisor (W1.3). Based on these points, I find the claimed effectiveness advantage of the weaker LLM to be unconvincing.

W3: More diverse and more reliable evaluation metrics should be considered.

This is an excellent point, and we address it in three parts.

First, it's important to acknowledge that using model-based approaches for evaluating open-ended generations remains a significant research challenge. Although GPT-4 and gold reward models are not flawless (as shown in our study), they are widely used in the literature due to their scalability and strong correlation with human preferences.

To further enhance the reliability of evaluation, we incorporate multiple top-performing reward models from RewardBench [1]. This consists of several independently trained reward models, helping to mitigate biases and errors from any single model. We present a table where each row corresponds to a different reward model highly ranked in RewardBench. This approach demonstrates that models trained with weak LLM feedback either match or surpass those trained with human feedback across various evaluation metrics, providing a more balanced and comprehensive assessment.

Lastly, we complemented our model-based evaluations with human evaluations to assess the quality of the responses generated by models trained with weak LLM feedback versus those trained with human feedback. The average win-rate across 100 pairs of responses is 52%, closely aligning with our main findings that weak LLM feedback is at least as effective as human feedback.

[1] Lambert, Nathan, et al. "Rewardbench: Evaluating reward models for language modeling." .

W4: Actionable insights from in-depth analysis.

Our in-depth analysis in Section 4 provides several important insights that advance our understanding of alignment and inform future research directions.

First, we delve into the quality of weak LLM versus human feedback, offering direct explanations for why weak LLM feedback performs as effectively as human feedback for alignment, which underpins our main findings. Notably, our analysis reveals that in nearly half the cases where weak LLM feedback diverges from human feedback, the weak LLM's choices surpass human annotations in quality, highlighting its potential to complement and even enhance human feedback.

The analysis also sheds light on the importance of addressing noise in the human-annotated data. By identifying weaknesses in existing datasets, we advocate for improved data curation and preprocessing strategies, which can in turn enhance the effectiveness of training weak LLMs to provide feedback. This aligns with our demonstration in response to W1.2.

Moreover, with more qualitative analysis, we reveal weak LLM feedback and human feedback diverge significantly in cases with subtle quality distinctions. Advanced models like GPT-4 show low preference consistency in these scenarios, reflecting the inherent challenge of distinguishing between closely matched responses. This points to the need for improved mechanisms to handle ambiguity in feedback collection, offering actionable directions for future research to tackle these nuanced cases.

Due to space constraints, we include additional qualitative analyses in Appendix C, such as (1) understanding systematic vs. random noise in weak LLM feedback, (2) correlations between generation quality in models trained with weak LLM versus human feedback, and (3) the impact of the KL coefficient on alignment performance when leveraging weak LLM feedback. We believe these insights provide valuable contributions to the community.

We will revise the manuscript to ensure these actionable insights are more prominently discussed and effectively linked to their practical implications for advancing alignment research.

Q2: The meaning of “pre-trained strong model"

The pre-trained strong model in line 371 refers to the model without any fine-tuning or alignment.

Q3: Why GPT-4 underperforms compared to smaller models.

That's an insightful question! While directly prompting GPT-4 to generate preferences demonstrates significant potential, it is still prone to biases, which can negatively impact preference quality. Additionally, GPT-4 is not explicitly optimized to distinguish preferences for specific tasks, unlike our weak LLM, which is trained on task-specific data to align with particular objectives. Our experimental results show that task-specific LLMs provide more reliable feedback, leading to improved alignment performance compared to relying solely on general-purpose models like GPT-4. Our qualitative analysis in Section 5 (Lines 415-448) further highlights the unreliability of GPT-4’s feedback in certain contexts.

We thank the reviewer for taking the time to read our response and actively engaging in this dialogue. We are happy to clarify the concerns further.

Our paper is not about establishing performance superiority but rather about demonstrating the promise of weak LLMs for alignment relative to human feedback. The central focus of our study is the contrast between weak LLM vs. human feedback, as reflected in our primary claim:

"The alignment performance using weak LLM feedback closely matches or even surpasses that of using human feedback."

We carefully chose wording such as "closely match" to avoid implying outright superiority. Below, we address your concerns in more detail.

---

General-purpose vs. task-specific

We respectfully disagree that our paper misleads readers, as the task-specific scope has been explicitly mentioned in multiple sections of the paper: L96 (introduction), L295 (experiments), and L508 (related work). That said, we acknowledge the value of further emphasizing this scope and its limitations and will revise the manuscript to ensure they are more clearly articulated.

We’d also like to clarify that our goal is not to claim performance superiority over RLAIF but to highlight the trade-offs between performance and computational efficiency. RLAIF is powerful but requires (1) heavy prompt engineering and (2) expensive compute. Our framework provides practitioners with an additional choice, and can co-exist with RLAIF in complementary way: for task-specific datasets with limited human feedback, our method offers a cost-effective, scalable alternative to GPT-4.

The task-specific nature of our method does not diminish its value; instead, it underscores its practical utility in addressing real-world constraints. We believe our work remains compelling for practitioners handling large-scale data who cannot justify the expense of resource-intensive models or may favor light models for sustainability concerns.

---

While I agree that using a weaker LLM has potential efficiency advantages, I remain skeptical of its claimed superiority in effectiveness.

We would like to reiterate that our paper is not about claiming superiority. Our central claim "The alignment performance using weak LLM feedback closely matches or even surpasses that of using human feedback." is supported by both qualitative and quantitative evidence, including additional experiments:

• Feedback from GPT-4 with more advanced prompt or high-quality human labels can indeed enhance the student model's performance, which is an expected conclusion. However, it is important to note that both comes with cost. Collecting GPT-4 feedback with higher-quality prompts often demands greater computational resources and relies heavily on prompt engineering. Similarly, obtaining high-quality human labels is more labor-intensive. In practical applications, balancing efficiency and effectiveness could be a critical consideration. Notably, our findings reveal that with weak LLM feedback achieves significantly higher efficiency while delivering comparable performance to the student model learned from the feedback from GPT-4 with more advanced prompt or high-quality human labels.
• Using higher-quality human feedback data, using weak LLM feedback achieves gold reward 5.23, which closely matches that of using human feedback 5.19. This is consistent with our central claim.
• Our human evaluation indicates a win-rate of 52%, closely aligning with our main findings that weak LLM feedback is at least as effective as human feedback. The same trend can be observed using gold reward model based evaluations. Across all five models, weak LLM feedback consistently demonstrates close or slightly better performance than human feedback.
• Section 3.3 demonstrates that alignment performance is nearly comparable across different supervisor sizes, suggesting that supervisor size has a less significant impact on feedback effectiveness. We additionally provide experiments using Llama-2-13B (due to computational constraints, we cannot train larger than 13B) as the supervisor for comparison. Additional experiments further confirm this: using Llama-2-13B achieves nearly identical performance to OPT-125M (7.90 vs. 7.93).

We thank the reviewer for the insightful feedback and positive evaluation of our work. We appreciate the acknowledgment of our approach's potential to reduce both computational costs and reliance on extensive human feedback. We address your comments and questions in detail below.

W1/Q1: Consistency of weak LLM feedback

As suggested, we perform additional investigation by evaluating the consistency of weak LLM feedback across multiple runs. Specifically, we trained 5 instances of the weak LLM (OPT-125M) with different seeds, resulting in a set of varied weak LLMs. Across different samples, the variance in feedback preference was measured at 0.04, which suggests a relatively low variance and a consistent alignment in feedback for the weak LLM trained with different seeds. This result demonstrates that the weak LLM can reliably provide consistent preferences across different runs.

W2/Q2: How do the authors ensure that the gold reward model and GPT-4 are reliable in evaluation? Are other potential evaluation metrics considered?

This is an excellent question, and we address it in three parts.

First, it's important to acknowledge that using model-based approaches for evaluating open-ended generations remains a significant research challenge. Although GPT-4 and gold reward models are not flawless, they are widely used in the literature due to their scalability and strong correlation with human preferences [1].

To further enhance the reliability of evaluation, we incorporate multiple top-performing reward models from RewardBench [2]. This consists of several independently trained reward models, helping to mitigate biases and errors from any single model. We present a table where each row corresponds to a different reward model highly ranked in RewardBench. This approach demonstrates that models trained with weak LLM feedback either match or surpass those trained with human feedback across various evaluation metrics, providing a more balanced and comprehensive assessment.

Lastly, we complemented our model-based evaluations with human evaluations to assess the quality of the responses generated by models trained with weak LLM feedback versus those trained with human feedback. The average win-rate across 100 pairs of responses is 52%, closely aligning with our main findings that weak LLM feedback is at least as effective as human feedback.

[1] Liu, Yang et al. "G-eval: Nlg evaluation using GPT-4 with better human alignment." arXiv preprint arXiv:2303.16634 (2023).

[2] Lambert, Nathan, et al. "RewardBench: Evaluating Reward Models for Language Modeling, March 2024." URL http://arxiv.org/abs/2403.13787.

Q3: The paper listed many LLM alignment methods in the related work. Did the authors compare their method with these alignment techniques?

We included an extensive list of alignment approaches in our related work section to provide readers with a comprehensive overview of this actively developing field. While most alignment methods discussed in the related work focus primarily on training strategies, our approach distinctly emphasizes the feedback sources, which is orthogonal to alignment techniques. Our experiments are designed to highlight the role of feedback sources rather than the alignment approaches themselves. Specifically, we benchmarked the performance of our method against both human feedback-based methods and other AI-driven feedback methods (such as RLAIF) discussed in the related work.

With that being said, exploring the use of weak LLM feedback in conjunction with more advanced and sophisticated alignment approaches presents a compelling avenue for future work. We believe integrating our approach with these more complex alignment strategies could yield even more promising performance, enhancing both efficiency and effectiveness in model alignment.