R.I.P.: Better Models by Survival of the Fittest Prompts

We thank the reviewer for acknowledging our contributions.

1. effectiveness of RIP if reward models are of smaller size, and from a different model family

Thank you for this insightful feedback. To address the reviewer’s question, we select a lightweight non-Llama-based reward model “Ray2333/GRM-gemma2-2B-rewardmodel-ft” (https://huggingface.co/Ray2333/GRM-gemma2-2B-rewardmodel-ft), which is a gemma2-2B based reward model, to annotate and then DPO finetune a Llama3.1-8B-Instruct model.

Below are the results on RIP filtering using reward scores by this Gemma2-based RM. By curating less than 5k out of 20k prompts, we can improve Llama3.1-8B-Instruct DPO models from 41.1 to 49.9 on AlpacaEval LC-winrate, showing similar improvement using ArmoRM filtering (LC winrate improved from 48.4% to 57.8%).

Ray2333/GRM-gemma2-2B-rewardmodel-ft (ranked 36th on Reward Bench) is ranked below ArmoRM on RewardBench, and the performance gap in the two reward model quality also affects the performances of finetuning Llama3.1-8B-Instruct with reward model annotations(i.e. better-quality reward model leads to better winrate of finetuned models). However, in both cases RIP filtering demonstrates its effectiveness.

Moreover, In the paper, we show the effectiveness of RIP filtering using various reward signals (ArmoRM, LLM-as-a-Judge, human), one of which is to use human annotated rewards from HelpSteer2 dataset as filtering criteria. We show in Table 4 that RIP filtering using human rewards also improve LLama3.1 winrates across all 3 benchmarks. In addition, Table 20 (performances on valid set when filtering using a single criteria), highlights that curating prompts of smaller human reward gaps boost performance.

We really appreciate the reviewer’s feedbacks, and hope that these experiment results can not only address the reviewer’s question, but further strengthen the effectiveness of RIP filterings under various reward model quality.

2. In table 9, I found several entries listing a lower bound of the reward gap, i.e., GAP > 0.042 in WildChat datasets. Does that contradict with the "smaller GAPs work better" hypothesis?

We really appreciate the reviewer for pointing out the typo. It should be GAP < 0.042 instead of >, since we are filtering out prompts with larger reward gaps. We will correct these typos in our updated version.

1. One caveat is that the claims are very much scoped to alignment performance on llm-judge benchmarks, which is a very particular domain.

Thank you for bringing this to our attention. We acknowledge that in our draft, we demonstrated the effectiveness of RIP on alignment performances in general instruction-following tasks. To further validate its capabilities, we conducted additional testing on reasoning domains. Here are the results:

We used our method to filter science reasoning 15k data [1] with inf_orm reward model (https://huggingface.co/infly/INF-ORM-Llama3.1-70B). As you can see that with our RIP method, we successfully filtered out over 90% of data, however, the model performance remains the same on reasoning benchmarks. [1] Yuan et al. NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions

2. It would still be useful to validate results with human annotators instead of entirely relying on model-as-judge results.

We agreed that mixed human evaluation is the preferred approach. However, due to cost constraints, we limited our human evaluation to 100 examples. The results showed our RIP successfully filtered out 88% of the noisy data identified by human evaluators. We will provide additional details in the appendix.

3. Does model performance and/or behaviour when dealing with such ambiguous prompts change?

This is a thought-provoking question! We evaluated our method on three widely used benchmarks: Alpaca Eval, Arena-Hard, and Wildbench. Notably, Wildbench utilizes data sourced from real users in chatbot-like scenarios, which provides a more realistic and representative testbed for our approach.

4. The authors use 3 elements for RIP (response reward, response length, reward gap), but do not evaluate and ablate each component

We performed an ablation study on each component in Tables 19 and 20. Specifically, we trained Llama3.1-8B on data filtered by each component individually and evaluated the trained model on our validation set (reporting Armo reward on the validation set). This is our standard approach for checkpoint selection. The numbers in Tables 19 and 20 represent ArmoRM scores on the valid set, which are close to each other due to the use of Armo scores distribution. Although the differences may seem small (e.g., 0.01), they correspond to significant performance differences. We only tested performance on the validation set initially, as running model evaluations on three benchmarks (Alpaca Eval, Arena Hard, and Wildbench) is time-consuming, given their reliance on the GPT4 API. However, we recognize that this may make it challenging for readers to interpret the numbers in Table 19. To address this, we further tested the best checkpoints within Table 19 on Alpaca Eval (see our comment to Reviewer D5vn).

5. The authors do not explore models beyond llama models

To show the effectiveness of our RIP filtering beyond Llama models, we:

(1). Finetune Gemma2-9B-it model with SimPO using the dataset (princeton-nlp/llama3-ultrafeedback-armorm) which are Gemma2 generations on ultrafeedback annotated by ArmoRM. Applying RIP on Gemma2-9B finetuning further improves Gemma2 performance on AlpacaEval from 69.48 to 73.81 by filtering out 50% train data.

Train Size Comparison

(2). Finetune Llama3 Model using a Gemma-2-2b based reward model (see our comment to Reviewer eddG).

6. You might imagine that filtering on length is less effective for a length-normalized method?

To further validate our approach, we tested our method on SimPO, a well-known length-normalized variant of the DPO algorithm.

Filter Metrics Comparison

These findings from our SimPO experiments are consistent with our previous DPO experiments, which demonstrated that Rejected Armo is the most effective metric. The addition of rejected length also proved to be highly effective, while gap filtering provided some benefits, albeit to a lesser extent than the other two metrics.

We appreciate your feedback and thank you for the opportunity to strengthen our paper. We hope our comments have addressed your concerns and questions, and we look forward to your further consideration of our work.

We thank the reviewer for highlighting our strength on the novelty of RIP filtering and its significant improvements as compared to traditional prompt-based filtering methods.

1. Could the authors provide a more detailed explanation regarding why the rejected response length is chosen as a filtering metric? Is this measure specifically intended to support Hypothesis 2, which suggests that low-quality prompts produce a broader variance in responses?

We thank the reviewer for offering the opportunity for us to clarify our hypothesis. We cited several studies in our paper Line 124 (e.g. [1]) showing correlation between response length, response quality and final performance. Given Hypothesis 1 “Low-quality prompts are likely to produce low-quality responses”, we thus select the length of the lowest-scored responses (a.k.a the rejected response) as one of the filtering metrics, in addition to rejected response score to measure quality of the rejected response. While rejected response length might also be correlated with response variance, we consider a more straightforward metric, the reward gap between chosen and rejected responses, to capture variances in our hypothesis 2.

2. "comprehensive case studies or explicit examples that directly compare samples filtered out by this criterion with those that are selected."

We thank the reviewer for pointing out the importance of detailed illustrations to justify our hypothesis. We have included such analysis in the paper appendix due to page limits.

In Table 26, we summarized 4 clusters of prompts in Figure 5 t-SNE plot that are being filtered out due to large response variance. For each cluster, we manually go over the 50 ~200 instructions, and summarize patterns of those rejected instructions in the “Rejected Reason” column. We further include “side-by-side” comparison between sample instructions filtered out by this criterion with those being selected in “Rejected Instructions” and “Accepted Instructions”, to further illustrate the hypothesis 2.

In Table 25 we extract 4 clusters of prompts from Figure 4 t-SNE plot that contain thousands of prompts filtered out due to low quality rejected responses. For each cluster, we count the percentage of prompts filtered out due to shorter length (below the rejected response cutoff). We also summarize their patterns in the “Description” column and list sample rejected instruction in “Rejected Instruction” column. Those 4 clusters we investigated are predominantly prompts filtered out by RIP (barely any survived prompts). To further address the reviewer’s question on side-by-side comparison, we included 2 other clusters that contain both selected and filtered prompts (See plot in https://anonymous.4open.science/r/projects-37B8/tsne.jpg)

In addition to visualizing the examples, we also conduct GPT4 analysis into quality of the filtered out prompts in Section 6.2 by each criterion, to justify our hypothesis. We also add human eval on 100 examples, due to length limit, we add one example here

We will add more of those analysis in the appendix with side-by-side comparisons to illustrate our RIP filtering criteria. We hope these further analysis, in addition to our Table 25~26 will help address the reviewer’s clarification question.

[1] Zhao, H., etc, N. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning.

1. Lack of ablations and clarity surrounding the effectiveness of each of the 3 criteria individually.

Due to the paper's length constraints, we have included an ablation study in the appendix (Table 19 and 20, line 990). In this study, we conducted data filtering experiments using each criterion individually, including filtering based on chosen reward, rejected reward, average reward (between chosen and rejected), chosen length, rejected length, and gap. We reported the results on our validation set.

As shown in the tables, when applying individual filtering criteria, we found that rejected reward is the most effective criterion, followed by rejected response length. The gap criterion also provided some improvement, although it was not as effective as the other two criteria. To further validate these findings, we tested the individual criteria on the Alpaca evaluation.

2. Lack of novelty: beta-DPO also filters based on gap.

We acknowledge that beta-DPO also employs gap-based filtering, which may raise concerns about the novelty of our approach. However, as evident from Table 19 and Table 20, our primary focus lies in the effectiveness of Rejected Reward and Rejected Length as filtering criteria in addition to reward gap, which outperform the gap-only-based criterion.

3. beta-DPO filtering as baseline.

Thank you for the suggestion. We acknowledge that beta-DPO also employs gap-based filtering, which we cite in our paper. However, there are three key differences between their approach and ours:

(a). Online vs. Offline Filtering: Beta-DPO's filtering is online, meaning they filter out data in every batch, whereas our approach filters data offline. This offline filtering enables more flexible and efficient generation pipelines, particularly for weak-to-strong generation scenarios. For instance, finetuning Llama3.3-70B-Instruct on prompts RIP filtered by a smaller Llama3.1-8B model outperformed (Alpaca LC-winrate improved from 54.3 to 64.5, Arena-Hard from 70.5 to 76.7).

(b). Gap Size Thresholds: Unlike beta-DPO, which removes both small and large gaps, our method removes bigger gaps only.

(c). Probabilistic vs. Deterministic Filtering: Beta-DPO's filtering is probabilistic, resulting in incomplete data removal, whereas our approach uses deterministic filtering to ensure thorough removal of unwanted data.

Given these differences, and as previously mentioned, our method prioritizes Rejected Reward and Rejected Length criteria over gap-based filtering, which have demonstrated superior effectiveness in our experiments. Consequently, when submitting our draft, we did not include a direct comparison with beta-DPO's filtering results. However, we appreciate your suggestion and have since conducted additional experiments to evaluate beta-DPO filtering on our experimental setting:

Thank you for diligently reviewing our work. We hope that we have thoroughly addressed all of your questions and concerns. Furthermore, we conducted additional experiments to strengthen our paper, including performing extra reasoning tasks, expanding our model suite beyond Llama by finetuning a Gemma-based model and two other reward models (GRM-Gemma2-2B-RewardModel-FT and INF-ORM-Llama3.1-70B), and applying RIP with SimPO.