Preference Data Annotation with Guided Density Ratios

We thank the reviewer for their insight and constructive feedback. Below are responses to the reviewer’s specific questions.

W1:

We address the novelty concern of prompting in three aspects:

1. Beyond generative tasks: While system prompts and ICL examples are commonly applied to generative tasks, our work is the first to study their effects on density ratio reward models. Previous works (Lambert et al., 2024; Lin et al., 2024;) have used density ratio rewards, but none have employed instructions or prompts to guide or customize the reward function. To our knowledge, prompting has not been widely used to evaluate the likelihood of samples in general.

2. Counter-intuitive to the DPO theory (see Common Response): DPO implicit reward (density ratio between DPO-ed and pre-DPO models) is considered to directly approximate the ground truth reward distribution on training data. Applying prompts or instructions to an optimized reward model is counter-intuitive, and is thus not adopted by other works. We propose a different perspective to density ratio through Strong-over-Weak Hypothesis, which allows for prompting to improve controllability.

3. Rules and principles as prompts in preference annotation: The ability to craft rules and principles to supervise AI models is indeed critical for safe and controllable AI development. Prompting is a cost-effective way to encode “human preferences” that can vary across user groups and evolve over time. Our training-free approach offers significant advantages over static classifiers, which must be retrained with updated data. A concurrent work from OpenAI(Mu et al., 2024) also addresses the issue by proposing rule-based prompting for generative safety classifiers.

W2:

We appreciate the comment but may not have fully understood reviewer question. Could the reviewer clarify what is meant by the "updated model," the “source” model, and the “target” model?

If the question pertains to the choice of models in the density ratio formulation, Response-to-All-Reviewers (Part 1) extensively discusses our updates and findings on the topic.

W3 (Experiment Design):

We revise the experiment design section (Section 4.2) to clarify the setting. Below are answers to specific questions:

Q1:
Generative Reward uses the Nous-Hermes-2-Mistral-7B-DPO model to generate preferred or dispreferred judgments. The same system instruction, ICL examples, and routing mechanism were used for the generative baseline. But it still reports poor performance (equivalent to random guessing). Generative reward typically needs very strong models to be reliable.

Q2:
We initially excluded results using chat-hard prompt template to avoid confusing readers, since this setup is not eventually adopted in our final method. We include the result for this setup in the updated Table-1. Chat and ChatHard are essentially the same category of samples, so we merge them into general Chat category. The templates for ChatHard lowered performance on the "simple" Chat domain, raising concern over its generalization. Thus, we eventually did not adopt it for our final method.

Q3:
Results for Nous-Hermes-2-Mistral-7B-DPO are included both as density ratio rewards in the DPO vs. SFT sections of Table 1 and as Generative Reward. We have updated Table 1 for clarity.

Q4:
We updated the table to better explain our model choice. Fixing the strong model as Nous-Hermes-2-Mistral-7B-DPO, we experiment with using Base or SFT as the weak model. Results show that using SFT as denominator yields better performance, so follow-up experiments mainly use the DPO vs. SFT model pair.
We also test SFT vs. Base (neither trained on preferences), demonstrating that the density ratio between them can function as a reward signal. Surprisingly, this approach successfully aligns the Llama-3-8B model after applying prompt guidance, as shown in line 439 of Table 2.

Q5 & W4:
We add detailed ablation studies on prompts and ICLs in Appendix C. Takeaways and findings from these ablations are available in Response-to-All-Reviewers (Parts 2, 3).

Q1:

The router in our pipeline is a classifier, which can be evaluated using standard classification performance metrics. It is implemented using zero-shot prompting (Figure 14) on Mixtral-8x7B-Instruct-0.1, achieving 83% accuracy in 3-way classification (Chat, Safety, Reasoning). Despite its imperfections, the router has limited negative impact on performance, as shown in Table 1: the score with a perfect router is 83.4, compared to 82.6 with the zero-shot router.

References:

Rule-Based Rewards for Language Model Safety, Mu et al., 2024

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization, Lin et al., 2024

RewardBench: Evaluating Reward Models for Language Modeling, Lambert et al., 2024

Thank you for your constructive feedback. Can we make two more points that hope to clarify your novelty concern and our scientific contribution?

1. We have made significant updates to the paper, including the introduction of the Strong-over-Weak Hypothesis, which highlights the relationship between model choices and reward generalization. This finding could be particularly relevant to the community grappling with inconsistent or suboptimal outcomes when using DPO implicit rewards—a special case of density ratio rewards. Our analysis suggests that these challenges may be mitigated by the insights presented in our work. Result is illustrated in Figure-1, hoping you may find compelling and worthy of consideration.

2. I’d like to highlight again the novelty and impact of our proposed method: it is the first to employ prompts to customize density ratios as reward signals, leading to results that exceed conventional expectations. It points to the direction of customizing density ratios as untrained, off-the-shelf reward signal, and we are building a library to make its usage simple and accessible.

We thank the reviewer for their interest in our work and their constructive feedback. To address common questions and clarify the draft revisions, we have included a Response-to-All-Reviewers (Parts 1, 2, 3). Below are responses to the reviewer’s specific questions.

W1:

To address the concern that “prompts and in-context examples…could be labor-intensive,” we discuss the exact cost of the safety prompt presented in the paper. The safety prompt required less than one day of work by one author. This time included drafting and iterating the instructions on a held-out set and preparing a set of in-context examples for each domain to ensure diversity. Note that we did not expend extra effort on optimizing these examples to achieve the results presented in the paper. Further discussion and ablation studies on prompt and ICL design are provided in Response-to-All-Reviewers (Parts 2, 3) and Appendix C.

The reviewer raises an excellent point regarding the limited cross-domain generalization of prompting (“prompts and in-context examples…may not generalize well across all tasks”), and we agree that domain mismatch could lead to sub-optimal reward performance. To address this, we propose a domain router in the annotation pipeline (end of Section 3.2). This router ensures that each data point is routed to the most relevant domain prompt for reward annotation. Our experiment (Table 2) compares the alignment results of the annotation pipeline with and without the domain router, showing that the domain-router approach (e.g., AlpacaEval 40.6) outperforms using a single fixed domain prompt (e.g., math/coding 30.0, safety 36.0).

W2:

We appreciate the reviewer’s suggestion to experiment with even larger models using our method. We have initiated experiments with meta-llama/Meta-Llama-3-70B-Instruct, but the significant computational overhead associated with sampling/annotation and training at this scale presents a challenge. We will report the results as soon as they are available and include them in the revised draft.

Notably, our current experiment already demonstrates that two relatively weak Mistral-7B models, without further training, can align the Llama-8B model to achieve GPT-4 level performance on AlpacaEval2.0 and ArenaHard. It will be fascinating to see if a similar super-alignment phenomenon can be achieved with the Llama-70B model.

Q1:

We believe that our density ratio-based method has applications beyond language models and holds promise for vision-language models (VLMs). Technically, as long as a model can input a sequence of tokens (e.g., text or image patches) and output their likelihood, our method can be applied.

Moreover, based on the Strong-over-Weak hypothesis outlined in Response-to-All-Reviewers (Part 1), we hypothesize that in the VLM setting, if we can identify two models where their difference is driven by human preference for generated images and the gap is sufficient, the difference in their density estimations can provide a quality/preference signal. We look forward to seeing follow-up work exploring this direction.

While the authors claim that the prompt design took "less than one day,", it does not address the inherent difficulty of iteratively refining prompts and in-context examples when handling diverse, ambiguous, or overlapping domains. This issue may limit the practical utility of their approach. This concern on designing these prompts share similarity with the other two reviewers.

Thank you for your constructive feedback.

We fully understand and appreciate your feedback on the prompting process, which may limit the use-case of the method.

But we want to highlight that the findings of the result do not depend on the prompting process or techniques. Density ratio reward improves from 77.2 to 80.1 on RewardBench by using a simple instruction “You are a helpful AI assistant.” Our more complex iteration of prompting only aims to show the it can specialize the density ratio towards different domains. Most noteworthy, by time of publishing, prompted density ratio scores the highest on the Safety domain of RewardBench, which is quite surprising as it out-performs specially trained safety classifiers. We believe those results sheds light on the strength of customization that clearly describing the "criterions of preference" is key to achieve improve performance for density ratio rewards. To make our method more useful, we are also building a library to use existing instructions to guide density ratio reward for it to be more accessible to the community.

Besides this, can we make two more points that hope to highlight our updated paper and contributions?

We have made significant updates to the paper, especially the introduction of the Strong-over-Weak Hypothesis, which highlights the relationship between model choices and reward generalization. This finding could be particularly relevant to the community grappling with inconsistent or suboptimal outcomes when using DPO implicit rewards—a special case of density ratio rewards. Our analysis suggests that these challenges may be mitigated by the insights presented in our work. Result is illustrated in Figure-1, hoping you may find compelling and worthy of consideration.

I’d like to highlight again the novelty of our proposed method: it is the first to employ prompts to customize density ratios as reward signals, leading to results that exceed conventional expectations. It points to the direction of customizing density ratios as untrained, off-the-shelf reward signal.

We thank the reviewer for their interest in our work and for their constructive feedback. Based on your suggestions, we update the drafe and add experiments. The primary update involves highlighting our first contribution: proposing the Strong-over-Weak density ratio reward framework and including experiments to support our thesis. Additionally, we conduct a study on prompt design, details in Response-to-All-Reviewers (Part 2,3) , to address concerns about the labor cost of designing prompts.

Q1:

Our experiments cover various domains, including Chat, Safety, and Coding/Math, as shown in Table 1. We include details about our prompt design process in Section C and add ablation studies on in-context learning (ICL) examples for various domains in Section C.1. Our work is not solely focused on the safety domain but is intended to present a general framework for customizing density ratio rewards.

Q1 & Q3:

The ability to craft rules and principles to supervise AI models is indeed critical for safe and controllable AI development (Mu et al., 2024) and offers significantly greater value than relying solely on manual annotation efforts.

To address the reviewer’s question, the safety prompt presented in Figure-3 requires less than one day of work by one author. This time included drafting and testing the instructions on a held-out set and preparing the set of in-context examples to ensure diversity. It’s worth noting that we did not expend additional effort optimizing these examples to obtain the results shown in the paper. See Response-to-All-Reviewers (Part 2) for detail.

In contrast, manual annotation entail much higher costs:

• Collecting data and training a reward model: Top-performing reward models (e.g., RM-Mistral-7B with a score of 80.4 and ArmoRM-Llama-3-8B with a score of 90.4 in Table 1) require approximately 570k diverse preference samples, costing around $570 ($1 per sample), in addition to the cost of training the reward model.
• Directly collecting human feedback data: Our experiments annotated 1,920,000 samples of preference data (Best-of-32 sampling) using our method. Annotating this manually would require a team of professional annotators working for months, costing approximately $1.92M ($1 per sample).

Q2:

We agree with the reviewer that human preferences can be complex. As outlined in Contribution #1 in the introduction, our method's first design choice is selecting appropriate strong and weak model pairs capable of capturing the complexity of human preferences. Prompts serve an auxiliary role, refining and guiding the density ratio signal.

For instance, a vanilla density ratio (without prompting) achieves 77.2 on RewardBench after careful selection of model pairs, as shown in Table 1.

We also observe cross-domain generalization:

• Shown in ablation Section C, the simple prompt "You are a helpful assistant" improves performance on the Reasoning domain, despite not explicitly targeting it.
• Similarly, the Safety prompt + ICL in Table 1 enhances performance in both the Safety and Reasoning domains. The system instruction acts as a prior, enabling LLMs to generalize across tasks and domains.

Q4 & Q5:

The system prompt and ICL examples establish principles and guardrails that the model must follow. In cases of conflict between preference instructions and user instructions, preference instructions (defined in the system prompt) take precedence. For example, if a user requests unsafe information conflicting with safety principles, responses that disregard safety principles are penalized, while safe responses are rewarded.

The rules are human-defined principles and are intended to be as such. As AI creators, we should have the ability to supervise principles AI systems shall follow, adjusting them for various deployment scenarios.

We write Response-to-All-Reviewers (Part 3) and Section C.1 to address your question on ablating prompt design. For instance, we do observe diminishing returns after incorporating more than three rules in safety prompt, and the fifth rule (as shown in Figure 6) led to performance declines, so it was not adopted.

Q6:

We appreciate the constructive feedback. We agree that studying the relationship between strong and weak models is crucial to clarifying and strengthening Contribution #1. In response, we have expanded this topic as the Strong-over-Weak hypothesis, providing additional experiments as empirical evidence. Please see Response-to-All-Reviewers (Part 1) for more details.

References:

• Rule-Based Rewards for Language Model Safety
  Tong Mu et al., 2024, Link

• RewardBench: Evaluating Reward Models for Language Modeling
  Lambert et al., 2024, Link