Rule Based Rewards for Language Model Safety

We are grateful to the reviewer for the constructive feedback and for pointing out areas where the clarity of our submission can be improved.

Terminology and Conceptual Clarity: Thank you for pointing out that the large amount of terms can lead to confusion. As mentioned in the general response, we will be sure to include additional tables (such as Table 2 in the rebuttal pdf) in the main text and appendix to clarify, as well as improve the clarity of writing in the main text.

Feature Extraction and Content Categories: We apologize for any confusion regarding feature extraction. In our methodology, content categories for prompts are pre-labeled using an automated labeling method and are not determined via feature extraction. This auto-label is necessary for the RBRs as it directly defines the desired response type for a given PPO prompt. Therefore, the propositions can directly look at just the completion as the desired response type is known. This information is also necessary for human data collection - while the annotators label the content category, and use their content category labels to decide which behaviors are ideal and non-ideal - we still need the auto labeled content categories to curate the set of prompts sent to the annotators to ensure there is a good balance of prompts as a random sample of prompts will mostly contain “safe” comply prompts and only contain a very small percentage of unsafe prompts. We will make sure to clarify these details in a future version.

Ranking of Completions: We also apologize for any confusion regarding ranking of completions and will be sure to clarify in a future version. To synthetically generate our weight fitting set, we start with the train split of our safety prompts and criteria for a set of desired completions for each prompt. For example, for a prompt that should be hard refused, we can decide we want the following set of 4 completions: one perfect hard refusal (ideal), two bad completions with randomly sampled bad refusal traits, such as judgement and/or illogical continuation, and one completion that contains the requested disallowed content. Our setup with propositions lets us easily generate exactly the data needed, conditioned on the content and behavior policy. We can use the natural language descriptions we already have to prompt for diverse completions. The goal is to have synthetic completions representing an ideal completion, a few sub-optimal completions, and an unacceptable completion for every prompt. For each desired completion, we iteratively synthetically sample a candidate completion from a prompted Helpful-Only model, and use our RBRs, the moderation model and other quality LLM filters to confirm it contains the desired traits (ex. we did indeed generate a judgy bad refusal) and resample if necessary. While these completions are not guaranteed to be perfectly what we want (ex. A bad refusal labeled as judgy may not actually be judgy), we find they are generally pretty accurate through a quick spot check of 20 completions per response type, response label combo. This way, we can use the fact that a completion we generated by prompting for ideal should be ranked higher than a completion we generated by prompting to be bad. We release some example weight fitting data that was generated using this procedure in data/weight_fitting_data and provide example code for using it in weight_fitting_example.ipynb.

Alleviating Distributional Shift Issues in Reward Modeling: We agree that rule-based rewards may provide more robustness to many potential issues, such as distributional shifts! For example, we have found that RBRs are be more robust to issues with reward over-optimization which we plan to talk about in a future version (see Section (5) Human Data Reward Over-Optimization in the rebuttal pdf).

Connection to model spec: We agree that it seems that OpenAI’s model spec could be an interesting and more complex situation with expanded scope to apply RBRs.

Prompt Design for Feature Extraction: To tune our prompts, we first label a “gold” set by hand that contains (prompt, completion) pairs and whether a each proposition is true or not. We use this to tune classification-prompts for auto-LLM proposition grading by hand until high accuracy is achieved on the set. We are happy to say that in our supplemental code release, we were able to provide the exact classification-prompts used in our experiments (config/proposition_prompts.yaml) as well as this gold set (data/rbr_gold_data for data and analyze_RBR_gold_data.ipynb for basic stats).

We thank the reviewer for their insightful comments and suggestions.

Generality of the Approach: Our integration of rule-based systems with reinforcement learning is designed to be model-agnostic, and we believe it should generalize to other LLMs. In a revised version, we will include an additional plot (Figure 1 in the rebuttal PDF) that illustrates the RBR works on a variety of RMs with different tendencies. Including additional experiments to show it also can correct an RM trained with old, outdated data.

Chain-of-Thought (CoT) Prompting and Alignment Issues: In our current work, we actually do not employ CoT to design the auto-LLM classification prompts for propositions. We did test using CoT for this, however we found it did not improve performance when using a ‘Large’ sized grader, most likely because our propositions were already broken down into simple binary tasks which were easy enough to classify. Additionally, for all propositions, we prompt tune and evaluate using a human labeled gold set (released with our code in the global rebuttal: see data/rbr_gold_data for the dataset and analyze_RBR_gold_data.ipynb for basic stats of the dataset) to further mitigate auto-LLM errors. We predict that CoT will be an important strategy when we move to other domains that have more complexity so we thank the reviewer for raising this important direction of future work to keep in mind.

If the comment is about how RBRs can be used to improve the properties of CoTs, this is an interesting question. Our approach is agnostic to the prompts and response, and could be applied to any property that a grader LLM can evaluate effectively. It would be interesting future work to explore applying such an approach to questions like faithfulness of CoTs.

For evaluations, we test both with and without CoT and report average scores, we are open to providing more granular evaluation results separating CoT and non-CoT results if it would reduce concerns.

Comprehensive Rules in RBRs: To cover a comprehensive range of safety concerns without becoming overly complex, we adopt a modular approach to rule design. Each rule addresses a specific aspect of safety policy, and the framework allows for the incremental addition or modification of rules as needed. This modularity ensures that the system remains manageable and adaptable to evolving safety requirements.

Handling Conflicting Rules: Our framework ideally works with rules that are mutually consistent. Currently, any conflicting rules are handled at the behavior policy design level. If we accidentally design conflicting rules, we will have to resolve at the policy level and rerun the entire process - however because our approach relies on synthetic data, it is much faster than having to recollect human data. We will add discussion about this point in a future version.

We thank the reviewer for the thorough evaluation and constructive feedback.

Relation to Prior Work and Novel Contributions: Our work builds upon and extends concepts from prior studies such as Sparrow, particularly in using rule-based approaches for enhancing safety in LLMs. However, our approach introduces several novel contributions:

• Sparrow focused on using human feedback - we instead focus on utilizing AI feedback. Sparrow was trained on 14.5k conversations generated and labeled by humans, whereas our approach required only human labeling for ~500 synthetically generated conversations. We hope that this cost, effort, and time reduction is impactful for the field.
• To integrate their Rule RM with their preference RM, they utilize a fixed combination method. In contrast, our approach involves fitting a model to ensure that the final reward effectively and correctly ranks completions.
• Lastly, we skip the step of distilling rules into RM data, a process that we found could be finicky ( see section (5) Human Data Reward Over-Optimization in the rebuttal PDF) and focus on incorporating the rules as directly as possible into PPO training. We will further elaborate on the differences between Sparrow and other related work in a future version of the paper.

AutoLLM grading accuracy: We provide the accuracy per proposition of the LLM grader in Table 7 in the appendix of the original submission. In a future version, we will add a compressed version of this table averaged across all propositions to the main text and make more references to the full table in the main text. Additionally, for more reproducibility, in our code release (see global rebuttal), we release the gold dataset used to score the proposition accuracies in data/rbr_gold_data (see analyze_RBR_gold_data.ipynb for basic stats of the dataset).

Grader LLM prompts: We are happy to be able to provide the exact classification-prompts for the propositions we used in our code and data release in config/proposition_prompts.yaml. We used a “large” sized helpful-only SFT engine as a grader engine.

Synthetic Data Quality: The labels of the synthetic data are most likely noisier than the human labeled gold data set as they are synthetic and have not gone through human verification. We spot-checked 20 examples from each response-type (ex. hard-refuse) and response (ex. Perfect refusal, bad refusal, etc) and found the response labels were mostly accurate. We will add this analysis to the appendix in a future version. We additionally release example synthetic data generated using the procedure detailed with our code release (data/weight_fitting_data, see weight_fitting_example.ipynb for description and usage) to allow external parties to study its accuracy as well.

Other models for combining the RBR and RM: We combine the scores given by our “large” sized helpful-only RM with the RBR scores by fitting a linear model of features. Note that “large” is the largest model size considered in these experiments. During the research process, we did plan on moving to non-linear models if necessary, however we found the performance of a linear model was already quite good based on the analysis given in Figure 3 of the original submission. In the future if we extend this work to even more complex domains, we do anticipate possibly needing a more complex model (such as possibly fitting a small neural network).

Higher over-refusal Rate: We believe the high over-refusal rate of human data could be due to a few possible reasons: We find annotators tend to be a bit more over-cautious in terms of assigning label categories (ex. For borderline refusal prompts, they would lean towards marking the prompt as hard_refuse). Additionally for human data, there is the question of balancing the RM data, (we include some discussion of this in the rebuttal PDF section (5) Human Data Reward Over-Optimization), we choose a 1-1 balance of hard-refusal and comply by default for human RM data, but it is possible that other ratios would result in different performance. One advantage of the RBR is that it removes the need to tune RM data balance, which we found to be a difficult parameter to tune.

Confusing Terms: As mentioned in the global response, we will be sure to include your helpful feedback and include clarifying tables as well as clarifying the text in a future version.