AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset

We thank the reviewer for your thoughtful and constructive review! We are glad that you recognize our work for tackling an "important problem" and for thoroughly investigating various strategies through a "large number of ablation studies". We will address your comments and questions in detail below:

Reasons To Reject #1

It would be beneficial to include explicit hypotheses for each experiment and additional supporting experiments to validate these hypotheses.

We appreciate the reviewer's insightful suggestion regarding the inclusion of explicit hypotheses and additional supporting experiments to enhance the robustness and credibility of our work.

We agree that a clear articulation of hypotheses is beneficial. In our current manuscript, our "Takeaways" (Section 4.2, 4.3, 4.4) function as empirically derived principles, which serve as our explicit hypotheses for each dataset component. Each of these "Takeaways" is investigated through a series of controlled experiments within Sections 4.2, 4.3, and 4.4, designed to validate its impact.

Furthermore, to validate the synergistic effect of combining these principles, Section 4.5 ("Combined Impact") presents an overarching experiment demonstrating cumulative gains when integrating the optimal choices.

Finally, our Cross-Verification (Section 4.6 and Appendix C.3) specifically aims to enhance robustness by validating our findings across different annotator models and instruction sources, thereby providing additional supporting evidence for the generalizability of our hypotheses.

We would like to clarify if our understanding of "hypotheses" as these explicitly stated "Takeaways" is accurate. Please let us know if there is any misunderstanding on our part.

Reasons To Reject #2

Expanding the scope to include more diverse models would significantly strengthen the findings.

Thank you for this important question regarding the generalizability of our findings across different base models. This is indeed a common question raised by all reviewers, so please refer to our Global Response for Using Another Base Model.

As the discussion period is nearing its end, we would like to offer our continued availability to address any remaining questions or points of clarification you may have. Should there be any aspect we haven't fully addressed, or any misunderstandings on our part, please do not hesitate to let us know. We are committed to ensuring all your concerns are resolved.

Thank you again for your valuable time and insightful feedback throughout this process.

Dear Reviewer E4jA, as the discussion period is drawing to a close, would you mind taking a look at the author response to see if it adequately addresses your concerns?

Best, AC

We sincerely thank you for your incredibly positive and constructive feedback! We are thrilled that you found our manuscript to contain "a wealth of insights for preference dataset curation" and recognized the AIR framework as a "useful way of disentangling different aspects of preference learning dataset curation". We will address your comments and questions in detail below:

Reasons To Accept #3

Confirmatory experiments with another base model would be nice.

Thank you for this important suggestion about the generalizability of our findings across different base models. This is indeed a common question raised by all reviewers, so please refer to our Global Response for Using Another Base Model.

Questions To Authors #1

It would be useful to provide some recommendations that generalize beyond the wholesale set of recommendations identified in the work, since there may be partial constraints on dataset collection.

We fully agree with the reviewer's insightful recommendation to provide guidance that generalizes beyond our 'wholesale set' of optimal recommendations, especially for scenarios with partial constraints on dataset collection. While our work identifies the most performant strategy when unconstrained, the fundamental strength of the AIR framework lies in its component-wise analysis. This systematic dissection allows practitioners to:

• Identify and work around constraints: Even if certain components are fixed due to collection limitations (e.g., being restricted to pairwise annotations), the framework helps understand the expected performance impact of that constraint.
• Optimize unconstrained elements: By understanding the isolated impact of each component, AIR enables targeted optimization of the remaining flexible elements based on our empirically validated principles.
• Discover new performant mixtures: The framework inherently provides a methodology for systematically exploring and quantifying the trade-offs of different strategy combinations, thereby allowing for the discovery of other effective mixtures tailored to specific partial constraints.

We will emphasize this practical utility more explicitly in the revised manuscript. This indeed opens up a crucial avenue for future work to formally explore and delineate these performant mixtures under various real-world constraints.

Questions To Authors #2

In some cases, the impacts of combining different interacting strategies is not super clear, e.g., in Figure 4, how do relative score margins, score thresholds, and on/off-policy mixing strategies interact? Are these plots made while holding other strategies constant?

We appreciate your providing this opportunity to clarify our experimental setup. We agree that the current description in the paper could be more explicit regarding the interactions and controlled variables.

To confirm, all our experiments are conducted under a controlled variable methodology. For instance, when investigating the impact of absolute score threshold in response pair construction (as shown in Figure 4, middle plot), we define three settings: High, Mid, and Low, based on the score of the chosen response (line 292). For each chosen response, we then carefully select its corresponding rejected response. During this selection, we control the relative score margin to be between Δ=2 or 3 for all pairs. Furthermore, for this specific analysis, all chosen and rejected responses are initially off-policy, meaning they are not generated by the model currently being trained.

We acknowledge that this level of detail was not sufficiently clear in the original manuscript. In the revised version, we will enhance the clarity and detail of each experimental setup description to ensure that the interactions between strategies and the controlled variables are explicitly and comprehensively presented.

Thank you for addressing my concerns and questions with a quick turnaround of experimental results. I also agree that the scope of the paper lies in LLM-as-a-Judge paradigm and thus, comparisons to human annotations are not needed in this work. I have also read the global response and convinced that this work's findings is generalizable to other model architectures. Overall I am satisfied with all the discussion and additional results, would love to see some of these incorporated in the final draft.

Questions To Authors #2

In line 204, "Explanations/coarse guidelines offer negligible gains (≤ 0.28) over simpler baselines" why is this considered negligible gains? Did you implement a single-explained to better understand this effect?

We thank the reviewer for pointing out this detail and for the opportunity to clarify our findings. Our statement that "Explanations/coarse guidelines offer negligible gains (≤0.28) over simpler baselines" (line 204) refers to the modest impact when comparing Pair-Guided (41.30) and Pair-Explained (41.75) directly against Pair-Basic (41.47). Specifically, Pair-Guided (which introduces guidelines) showed a slight decrease of -0.17, while Pair-Explained (which requires an explanation first) showed a slight increase of +0.28. This is distinct from the difference between Pair-Explained and Pair-Guided.

To further clarify this effect, we implemented and tested both Single-Explained and Single-Guided annotation strategies. The results, as reported in Table 1 within the "Global Response for Using Another Base Model," show that Single-Guided (31.4354) and Single-Explained (31.2454) actually resulted in a decrease compared to Single-Basic (32.333).

Questions To Authors #3

The difference in performance across various annotation designs are not well-investigated to understand whether this is noise or not.

We thank the reviewer for prompting a deeper look into our conclusion regarding intricate annotation workflows introducing noise. To address your concerns directly, we have conducted further analyses for the revised manuscript:

• We have now included error bars in some experimental results to quantify performance variability and consistency.
• We also added experiments for Single-Guided and Single-Explained strategies to Table 1. These, combined with existing results, suggest that the observed patterns are less about random noise and more about the systematic behavior of LLM annotators.
• Our existing cross-verification (Section 4.6) demonstrated consistent findings across different datasets and annotator models, which further supports the robustness of our conclusions.
• We have also conducted a new experiment with an alternative base model, further reinforcing the generalizability of our principles. (As shown in Global Response for Using Another Base Model)

Thank you for pushing us to strengthen this investigation.

Questions To Authors #4

Did you also study which RM (classifier or generative) had higher agreement with human annotations?

We appreciate this pertinent question. Our analysis in Section 4.2.1 focuses on which type of LLM-based RM yields better results when used to construct preference datasets for DPO training. However, this comparison was done on downstream LLM alignment performance, not framed as an "agreement with human annotations" study. We acknowledge that a direct investigation into the inter-annotator agreement between different LLM-based RMs and human judgments would provide valuable insights into the quality and biases of LLM annotations. This is an important area that we would consider for future work to further validate the reliability of LLM-as-a-judge paradigms.

Questions To Authors #5

How would these findings change or not change with the base model or a different architecture?

Thank you for this constructive suggestion regarding the generalizability of our findings across different base models. This is indeed a common question raised by all reviewers, so please refer to our Global Response for Using Another Base Model.

Questions To Authors #6

These papers can be cited to better support this work.

We thank the reviewer for these valuable citation suggestions. We agree they will strengthen our paper. This paper, Kim, Joongwon, et al. (2024), is already cited in line 104, relevant to our discussion on instruction selection. We will integrate the remaining three papers into our revised manuscript.

Reasons To Reject #2

This work did not adequately study the impact of LLM annotations vs Human annotations and caveats with LLM annotations (for e.g. LLMs tend to favour their own generations) and how these impact preference dataset design.

We appreciate the reviewer's insightful comment. We actually considered the potential for self-favoring bias and conducted a preliminary study. To investigate this, we sampled 1,000 instructions. For each instruction, responses were generated by both Meta-Llama-3.1-8B-Instruct and Qwen2-7B-Instruct. These responses were then judged independently by Meta-Llama-3.1-70B-Instruct and Qwen2-72B-Instruct. The win rates we observed were as follows:

• Win rate (Llama as Annotator): {'Llama': 59.1%, 'Qwen': 40.9%}
• Win rate (Qwen as Annotator): {'Llama': 55.1%, 'Qwen': 44.9%}

While these results indeed suggest the presence of a self-bias effect, its impact was not overwhelmingly large. To further mitigate this, we intentionally diversified our response generation pool beyond just the Llama and Qwen families. As shown in Table 2, out of the 17 models we used for response generation, only 4 are Llama-based and 3 are Qwen-based. The remaining models, including the Gemma-2 series, Phi-3 series, Mistral series, MiniCPM3-4B, and DeepSeek-V2-Lite-Chat, belong to different architectural families. This broad diversity means any self-bias effect on our overall results is limited.

Regarding human annotation, our work operates within the LLM-as-a-Judge paradigm due to the high cost and scalability challenges of human annotation for large datasets (e.g., ~30k preference pairs per DPO run). The paper's core contribution is optimizing these LLM-annotated dataset components. While direct comparison with human annotations and a comprehensive analysis of LLM judge biases are important, they were beyond the scope of this systematic analysis.

Questions To Authors #1

Report the error bars.

We appreciate the reviewer's excellent suggestion to include error bars. Firstly, it's important to distinguish between evaluation types. For objective benchmarks such as Eurus and LiveBench, where model responses are generated using greedy decoding, the evaluation results are deterministic and remain consistent across multiple runs. Therefore, variability (and thus error bars) is not applicable to these benchmarks.

However, for subjective LLM-as-a-Judge benchmarks (MT-Bench, ArenaHard, AlpacaEval 2.0, WildBench (V2)), which utilize GPT-4o or GPT-4 series models for evaluation, results can exhibit some variability due to the nature of LLM judgment. As each single evaluation can incur significant costs (around $10), constrained by evaluation resources, we selected ArenaHard as a representative LLM-as-a-Judge benchmark to report error bars for the results presented in Figure 2 and Table 1.

Here are the updated results for ArenaHard (totally 3 runs including the original result, using a 95% confidence level), reporting the average result and the confidence intervals:

Figure 2 (Left)

Figure 2 (Right)

Table 1

These results demonstrate that the evaluations performed by GPT-4-0314 LLM-as-a-Judge are relatively stable, with confidence intervals indicating consistent performance across runs, even given the inherent variability of LLM-based judgments. The evaluations were conducted with a temperature of 0.6 and top_p of 0.95.

We sincerely thank the reviewer for your thorough and constructive feedback! We are very pleased that you recognize our work for its "systematic study and nuanced understanding" of preference dataset components. We will address your comments and questions in detail below:

Reasons To Reject # 1:

The study does not study the interactions between these components that may affect LLM performance.

We appreciate the reviewer's insightful comment regarding the limited study of interactions between dataset components. While our initial work focused on isolating individual impacts and demonstrated the cumulative effect of our findings in Section 4.5, we agree that a deeper understanding of component interactions is very important.

We acknowledge that exhaustive pairwise interaction studies were not included in the original manuscript due to the substantial experimental cost (each setting requiring a DPO run and extensive evaluation, often with expensive APIs like GPT-4/GPT-4o). However, in direct response to your valuable feedback, we have conducted additional preliminary interaction experiments during the rebuttal period, specifically exploring the interplay between Instruction Selection and Response Pair Construction. Due to current experimental and evaluation API costs, these experiments were evaluated on ArenaHard, a benchmark widely recognized in recent work (e.g., Qwen 3).

Exp 1: Instruction Variance Vs. Relative Score Margin

We divided instructions into low, mid, and high variance groups, consistent with Section 4.3.1. For each group, we investigated the optimal response pair score margin (based on margin distribution to balance pair count across groups for DPO training). We controlled other variables and evaluated results on ArenaHard.

Analysis: We observe that while the optimal relative score margin can differ slightly for instructions with fixed variance, the superior performance of low-variance instructions remains consistently significant across all margin settings.

Exp 2: Instruction Variance Vs. Absolute Score Threshold

Using the same instruction variance groups, we explored the optimal response absolute score threshold (low/mid/high settings consistent with Section 4.4.2, balancing pair count for DPO training). We controlled other variables and evaluated results on ArenaHard.

Analysis: These results show that for instructions of any variance, selecting high-score responses consistently yields the best performance. Conversely, the benefit of low-variance instructions is consistently evident across all absolute score thresholds.

These two experiments indeed demonstrate relevant interactions between instruction selection and response pair construction. However, they also reinforce our core conclusions: for instance, the consistent benefit of low-variance instructions, and that higher absolute scores are generally preferred. The specific optimal margin might vary, adding a layer of nuance. We will integrate these findings into the revised version of the paper, emphasizing these interactions and highlighting that a more exhaustive study of all component interactions remains a valuable direction for future work.

Figure 3 (Left):

Figure 3 (Right):

Figure 4 (Left):

Figure 4 (Middle):

Figure 4 (Right):

As can be seen, the experimental results from Sections 4.2, 4.3, and 4.4 in the main text show significant consistency for OLMo-2-1124-7B-SFT, with the average results being consistent. This indicates that our core findings generalize well across different base model architectures.

We will integrate these new base model experimental results into the revised version of our paper to further highlight the credibility of our findings.