Thank you for your insightful review. We appreciate your recognition of the paper's clarity and the novelty of our approach in aligning RL policies using LLMs. We address your questions below.

Concern 1: In reality, this may not be the case, and additionally, specifying an exhaustive list of principles may not be practical.

We thank the reviewer for the insightful comment. We have added a new paragraph to discuss the limitation of the Principle Design Challenges, as copied below:

Principle Design Challenges: Crafting robust and encompassing principles for SALMON is intricate, mainly due to the unpredictability of the myriad scenarios a model might encounter during the RL stage. Balancing potentially conflicting principles introduces complexities that can yield unexpected results. We advocate for the participation of a diverse group, including ethicists and other stakeholders, to refine these guiding principles. It is crucial to recognize that distinct contexts and applications will necessitate unique strategies. We present our approach not as a universal solution but as a starting platform, aiming to foster expansive community discourse. In practice, many companies and organizations already have an exhaustive list of principles for their employees, which is usually named as Business Conduct Guidelines (BCG).

On the other hand, we would like to point out that a recent work (published after the ICLR deadline) [1] shows that some general “good-for-humanity” principles can provide high-level guidance to choose outputs that are best for humanity, and achieve similar performance to specific principles. We leave the exploration of SALMON with less amount of general principles as future direction.

Question 1: It is not entirely clear how the principles and exemplars affect the policy learning.

Please see “Direct Evaluation of the Principle-Following Reward Model” under the General Response for how principles affect the preference of the reward model.

As an approximation of PPO, we also provide the best-of-n samples of how principles affect the results of ranked policy outputs in Section. D of the appendix.

Question 2: What happens in scenarios where two or more principles are in conflict with each other?

As the discriminative preference labels are decided by the principle with the most pronounced difference (Appendix D). The scenarios of conflict principles are determined by the most significant differences between two responses.

Question 3: would it be possible to impose a hierarchy of principles during training?

We thank the reviewer for the insightful suggestion. We envision that a hierarchy of principles (e.g., safety over helpfulness) is possible and would explore this direction as our future work.

Question 4: is the set of principles general enough to be applicable to any scenario?

Please refer to our response to Concern 1.

[1]: Kundu, Sandipan, et al. "Specific versus General Principles for Constitutional AI." arXiv preprint arXiv:2310.13798 (2023).

Thank you for your insightful review and the positive feedback on our paper. We are glad that you found our paper well-written and organized. It's particularly encouraging to hear your appreciation for the practical intuitiveness of SALMON and its efficiency in reducing human annotation costs compared to the prevalent RLHF method. We address your questions below.

Concern 1: can the authors measure a quantitative result of the principle-following reward model?

Please see “Direct Evaluation of the Principle-Following Reward Model” under the General Response.

Concern 2: Even though Llama-2-70B with SALMON can provide better alignment score (7.4 MT-Bench score) than Llama-2-70B with RLHF (PPO) (6.9), there is still large gap to GPT-4 (9.0) and ChatGPT (7.9).

We would like to point out that it is unfair to directly compare SALMON with state-of-the-art proprietary models. SALMON uses an open-source LLaMA-2-70B base language model, while GPT-4 uses a much larger and stronger base language model, as evidenced in the LLM capability evaluations (e.g., MMLU, BIG-BENCH). Therefore, we use LLaMA-2-Chat-70b (6.9) as the primary baseline for comparison, and show that SALMON (RLAIF) can outperform the prevailing RLHF method on general alignment.

Concern 3: However, enhanced RLHF methods such as DPO (Direct Policy Optimization) and P3O (Pair-wise Policy Optimization) have been proposed and shown that they can achieve better reward scores than PPO-based RLHF. It would be better to compare SALMON with recent RLHF methods.

P3O [1] is released to the public in Oct. 2023, before the ICLR deadline. It is impossible for us to compare SALMON against it.

For DPO [2], please see “Compare SALMON with recent RLHF alternatives: DPO, ResT, and SliC-HF” under the General Response.

Concern 4: if SALMON is better to maintain the token distribution of the reference LLM

We perform the comparison of the aggregated KL-divergence of SALMON with and without RL-time inference intervention, but do not observe significant difference.

We would like to point out that the reward-hacking distribution shift in the RLHF training usually does not come with a significant increase in KL-divergence, especially when these RLHF methods have a KL-penalty loss term in the reward (Eq. 2).

Question 1: what is the main advancement of SALMON compared to Constitutional AI?

Please see our General Response.

Question 2: if better base LLMs than Llama-2-70B are used, can SALMON further reduce the gap to GPT-4 and ChatGPT?

As a general and scalable alignment paradigm, SALMON is not specialized on LLaMA-2 family LLMs. We believe better base LLMs + SALMON could further reduce the gap to GPT-4 and ChatGPT, as evidenced by the recent works [6,7] showing that RLAIF has a very steep scaling curve (e.g., Figure 4 in [1] and Figure 6 in [2]).

Question 3: If some enhanced RLHF methods such as DPO and P3O are used instead of a PPO-based method, is SALMON sill provide better performance than those methods?

Since PPO (RLHF) is the only method that has proven effective in large-scale real-world settings [3,4,5], and our method outperforms LLaMA-2-Chat-70b (best PPO-trained non-distilled model) under fair comparison, we believe SALMON would also achieve strong performance with other non-RL (offline RL) PPO alternatives. For example, we found sampling responses from the SFT model, using a principle-following reward model to label the preference, and performing DPO-based optimization can achieve strong performance.

Please see “Compare SALMON with recent RLHF alternatives: DPO, ResT, and SliC-HF” under the General Response for more information.

---

[1] Wu, Tianhao, et al. "Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment." arXiv preprint arXiv:2310.00212 (2023).

[2] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." arXiv preprint arXiv:2305.18290 (2023).

[3] OpenAI. “Introducing ChatGPT” (2022).

[4] OpenAI. “GPT-4 Technical Report” (2023).

[5] Anthropic. “Introducing Claude” (2023).

[6] Bai, Yuntao, et al. "Constitutional ai: Harmlessness from ai feedback." arXiv preprint arXiv:2212.08073 (2022).

[7] Kundu, Sandipan, et al. "Specific versus General Principles for Constitutional AI." arXiv preprint arXiv:2310.13798 (2023).

Thank you for your constructive feedback on our work involving SALMON and the development of Dromedary-2-70b. We are glad that you appreciate the clarity and relevance of our paper, particularly in addressing the timely problem of preference data collection for reward models. Your recognition of our model's impressive performance and the novel approach to AI feedback is encouraging. We address your questions below.

Concern 1: Some lack of novelty compared to Constitutional AI

Please see “Novelty Compared to Constitutional AI” under the General Response.

Concern 2: I don’t see any reason that SALMON rewards cannot be hacked to give degenerative responses or undesirable responses, if trained long enough using RL.

In LLM-based AI alignment, reward hacking, or reward over-optimization [1,2,3], means that an RL-optimized model might exploit certain vulnerabilities in the fixed reward model, thereby artificially boosting its score without genuine performance improvement.

We would like to clarify that reward hacking in LLM alignment can happen with perfect natural language, but often comes with noticeable problematic behavioral traits. For example, in this paper, we identify three types of reward-hacking behaviors in our Figure 3 without degenerated responses. Here, unnatural or degenerated responses would not occur due to either too low reward scores or too-high KL penalty.

Perhaps the degenerated responses would occur if the policy models were trained really long. But in practice, the PPO steps of RLHF/RLAIF on large-enough LLMs are typically within 1000 steps [4], and the degenerated responses do not occur in this short RL period.

Concern 3: The SFT-Model acting as a judge can already be used as a reward model.

The SFT-Model cannot act as a judge for the following reasons:

Note that at inference time, as opposed to training, all the reward models need to predict a scalar for a single output, without requiring access to its paired output. LLMs are typically better at doing pairwise comparison, but worse at directly assigning a score to a single answer [4,5,6]. Therefore, the SFT-Model in RLAIF is usually only used for labeling AI-generated preferences with pairwise comparison. Even for pairwise comparison, the performance of the SFT-Model still falls behind the specially trained reward models. Please see “Direct Evaluation of the Principle-Following Reward Model” under the General Response for more information. Aggregating many principles with the SFT-Model, as the reviewer pointed out, can be very inefficient at implementation.

Concern 4: How well does SALMON do without the Preference Model Pre-training?

We found the SFT-Model generated preferences can be comparable to pre-trained preference models, especially on the helpfulness dimension. Please see “Direct Evaluation of the Principle-Following Reward Model” under the General Response for more information.So SALMON is still possible to achieve strong self-alignment performance without PMP.

However, we would also like to point out that Preference Model Pre-training (PMP) is a common practice in RLHF [8,9] to improve the sample efficiency and the asymptotic performance.

Concern 5: Prior works deem the tendency of RLHF to produce lengthier responses as a negative.

While some recent work [7] found that “When optimizing for helpfulness, RLHF has been consistently observed to drive models to produce longer outputs”, we have not found any work that claims the lengthier responses as a negative. Instead, [7] argues that longer outputs often prove to be more informative and useful. We are open to further discuss this aspect and would appreciate any additional references the reviewer could provide to enrich our understanding.

Question 1: What happens if the answers are indistinguishable based on the principle?

Our system does have the “no preference” option when the logprobs of judging either Output (a) and Output (b) as the better option are equal.

Question 2: What is the size of the final preference dataset that is generated by the Self-aligned SFT model?

The preferences are collected for each principle. We collected a total of 98k = 10 (#principles in Table 6) x 9.8k (#prompts OASST1) principle-specified preferences, and used those to construct around 40k multi-principle preferences.

Question 3: The highest log probability can be misleading by itself.

We calculate the log probabilities of the SFT-Model for preferring Output (a) or Output (b), not the log probabilities of the answer themselves. The AI-generated preference process is illustrated in Figure 2 (Collecting Principle-Driven Synthetic Preferences).

Thank you for your insightful review of our paper. We're glad to hear that you appreciate the simplicity and general applicability of our SALMON technique, as well as its potential in domain-specific applications. Your recognition of how our method can generate synthetic preference data with minimal human supervision and its integration with Dromedary-2 to achieve top-notch performance across benchmarks is encouraging. We address your questions below.

Concern 1: It would be good to show that the methods can also be leveraged to improve the performance of smaller models such as 7B or 33B

Recent works [1,2] show that RLAIF has a very steep scaling curve (e.g., Figure 4 in [1] and Figure 6 in [2]), that is, the model’s discriminative performance is not very far from random guess when the model size is around 22B. On the other hand, the SFT initialization of SALMON, i.e., Principle-Driven Self-Alignment [3], also requires a strong enough base language model (LLaMA-65b/70b). Therefore, we have only experimented with the strongest open-source LLM with 70B parameters.

However, we are optimistic and enthusiastic about compressing the alignment behavior from these large self-aligned models into smaller models, for example, using the Dromedary-2-70b model as the oracle generator/evaluator model to generate distillation data for smaller models. [4] That would be an important direction for deploying these models.

Concern 2: I believe this method could potentially be adapted to some other tasks such as code generation.

We appreciate the reviewer’s insight. We do believe our SALMON (RLAIF) method can be adapted to more difficult tasks where it’s very hard or costly for humans to annotate preferences. We are actively pursuing this direction as follow-up work.

[1] Bai, Yuntao, et al. "Constitutional ai: Harmlessness from ai feedback." arXiv preprint arXiv:2212.08073 (2022).

[2] Kundu, Sandipan, et al. "Specific versus General Principles for Constitutional AI." arXiv preprint arXiv:2310.13798 (2023).

[3] Sun, Zhiqing, et al. "Principle-driven self-alignment of language models from scratch with minimal human supervision." arXiv preprint arXiv:2305.03047 (2023).

[4] Tunstall, Lewis, et al. "Zephyr: Direct Distillation of LM Alignment." arXiv preprint arXiv:2310.16944 (2023).