Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

Dear Reviewer dZTX,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[Q1] How do you choose the initial seed data? Are the data randomly chosen as seeds? Are there differences in human preferences across data? How do you handle such differences?

The initial seed data is selected through random sampling from the UltraFeedback dataset, which could include diverse human preferences. We do not explicitly address these preference variations, but we observe that SPA remains effective across different random seed selections and seed sizes, as demonstrated in Tables 3 and 4. This robustness stems from SPA’s use of direct preference judgment, which leverages the LLM’s intrinsic knowledge about human preferences derived from the overall seed data, rather than relying on specific data instances like in-context learning approaches.

On the other hand, we’d like to mention that there is a field of research that aims to better capture diverse human preferences in LLM’s responses [1,2]. Since SPA can easily incorporate the modifications in the reward modeling or training objective, we believe that the proposed SPA can be jointly applicable to these existing works.

---

[Q2] In section 4.1, can you elaborate on the sampling process used to generate the responses y1 and y2? In particular, does the model use specific diversity techniques to ensure diversity of reactions, or does it rely purely on randomness in the generation process?

The responses $y_1$ and $y_2$ are sampled purely based on randomness without applying specific diversity techniques, as described in lines 195-196 of the original draft. However, our framework is flexible and can incorporate various diversity sampling techniques if desired. Given the demonstrated effectiveness of diversity techniques in iterative preference learning (e.g., sampling N>2 responses, then choosing best/worst as the preferred/dispreferred responses [3,4]), we believe that incorporating such methods could further enhance the performance of our framework by enhancing the diversity of the responses.

---

[Q3] How does it ensure the reliability of the expanded data?

First, the proposed direct preference judgment to label the preference between responses (in Section 4.1) yields more reliable data expansion. This is because it leverages intrinsic reward signals directly derived from the target LLM, which are continuously refined through iterative updates. In contrast, prior methods that expand data rely on preference labels generated via implicit prompting or fixed external reward models, limiting their reliability and adaptability.

Additionally, the proposed self-refinement mechanism in Section 4.2 further strengthens the reliability, by addressing labeling noise within the iterative preference learning framework. This self-refinement step uses a logit interpolation to approximate outputs from a more strongly aligned LLM, enabling effective noise detection. By reducing labeling noise in the expanded preference data, this technique enhances the overall reliability and accuracy of the generated data.

We highlight that the enhanced reliability of the expanded data with these components is demonstrated through our experimental results. For example, our SPA approach significantly outperforms prior methods, with improvements such as a 3.52% increase in length-controlled (LC) win rate on AlpacaEval 2.0. Moreover, SPA achieves superior alignment performance using only 3.3% of the seed preference data, compared to DPO method that uses the full 100% seed data.

---

[1] Zhou et al., Beyond One-preference-for-all: Multi-objective Direct Preference Optimization., arXiv:2310
[2] Pitis et al., Improving Context-aware Preference Modeling for Language Models., arXiv:2407
[3] Wu et al., Self-play Preference Optimization for Language Model Alignment., arXiv:2405
[4] Rosset et al., Direct Nash Optimization: Teaching Language Models to Self-improve with General Preferences., arXiv:2404

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

Dear Reviewer dZTX,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much.

Best regards,
Authors

Dear Reviewer NqES,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1] 'De-coupled noise preference detection' is not stated clearly enough in section 4.2. Based on my understanding, zθ~ is used to substitute for zθ in the 'Self-refinement' part, which is also supported in Algorithm 1. If I am correct, it would be easier for readers to understand if the final usage of the approximated logits and labels are stated in the main text.

You’re correct; $z_{\widetilde{\theta}}$ (Eq. 11) is used to substitute for $z_{\theta}$ in the self-refinement step (Eq. 10). Following your suggestion, we have explicitly stated this in Section 4.2 (lines 256-257 in the revised draft) for clarity.

---

[W2, Q3] Lacks some explanatory discussion on why this method can work on such a small subsets and even perform better than DPO with the full dataset. Does this method elicit the latent human preference knowledge from pre-trained model, or the 3.3% subset is already enough to define the human preference in UltraFeedback dataset?

Thank you for the insightful question. As noted in your first conjecture, SPA is indeed able to elicit latent preference knowledge from the pre-trained LLM, allowing it to perform effectively even with a limited number (e.g., 3.3%) of the labeled preference data. Specifically, this effectiveness is achieved through two main techniques in SPA:

• Direct Preference Judgment (Section 4.1) enables efficient and reliable data expansion, by leveraging intrinsic reward signals directly derived from the target LLM, which are continuously refined through iterative updates.
• Self-Refinement Mechanism (Section 4.2) further enhances the reliability of expanded data, by addressing labeling noise through the iterative preference learning. By reducing labeling noise in the expanded preference data, it enhances the overall reliability and accuracy of the generated data.

We also highlight that our experiments support this claim well:

• SPA without seed data (Figure 4 and Table 8): it is assumed that there is no seed preference data (i.e., 0%), and the instruction-tuned LLM is directly used to generate preference data without initial DPO step (1st line in Algorithm 1). Here, the proposed SPA continuously improves the alignment performance, which supports the effectiveness from the elicitation
• SPA with varying number of seed data (Table 3): SPA is applied with varying number of seed preference data (0.8% to 10%), and SPA is consistently effective and the improvement grows with increased seed data. It opposes the second part of the conjecture as it implies that the 3.3% subset does not fully capture the UltraFeedback dataset’s human preference knowledge. Remarkably, SPA yields better alignment performance than DPO with full data, if the given seed data is sufficient (e.g., >= 1.7%) to provide the effective guidance to elicit LLM’s intrinsic knowledge about human preference.

Overall, these results indicate that the effectiveness of SPA is from eliciting LLM’s intrinsic knowledge about human preference rather than learning the given seed preference data well.

[Q1] What is the model used for the 'LLM-as-judge' method in Table 2 ? Have you tried using πi−1 in this baseline?

For the ‘LLM-as-Judge’ method in Table 2, we used a fixed model fine-tuned specifically for evaluating preferences between responses. Specifically, the training dataset for this model was constructed by converting seed preference data into pairwise comparison prompts, following approaches similar to [1,2]. Then, the common SFT model (used for initialization in other baselines like DPO and SPA) was further fine-tuned on this dataset via supervised learning. More details, such as the training process and the pairwise comparison prompts, are included in Appendix B.2 of the original draft.

Regarding the use of $\pi_{i-1}$ in this baseline, we conducted new experiments; at the 2nd iteration, the evaluation model was initialized with the resulting model from the 1st iteration and fine-tuned with the same preference evaluation dataset. The evaluation results, as denoted by LLM-as-Judge (Iter. 2, prev. init), on AlpacaEval 2.0 are presented below along with other methods at the 2nd iteration. While this approach yielded somewhat improved alignment compared to the fixed model, SPA still significantly outperformed this baseline. This underscores that SPA’s superior performance arises from its novel preference evaluation techniques rather than the specific evaluation model used. We have added these results and the corresponding discussion to Appendix D of the revised draft.

\begin{array}{c|ccc} \hline \text{AlpacaEval 2.0} & \text{LC Win Rate (\%)} &\text{Win Rate (\%)} \newline \hline \text{LLM-as-judge (Iter. 1)} & 8.88 & 8.01 \newline \hline \text{LLM-as-judge (Iter. 2, orig)} & 9.49 & 8.46 \newline
\text{LLM-as-judge (Iter. 2, prev. init)} & 9.74 & 10.09 \newline \hline \text{SPA (Iter. 2, ours)} & 15.46 & 19.91 \newline \hline \end{array}

---

[Q2] There are lines out of the border in references and appendix.

Thank you for the careful reading and pointing out the editorial problems. We have corrected these in the revised draft (p.13 and p.15).

---

[1] Bai et al., Constitutional AI: Harmlessness from AI Feedback., Anthropic 2022
[2] Lee et al., Aligning Large Language Models by On-Policy Self-Judgment., ACL 2024

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

Dear Reviewer NqES,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much.

Best regards,
Authors

Dear Reviewer NqES,

We are glad to hear that we have addressed most of your concerns. Also, thank you for raising the score!

Please don't hesitate if you have any further questions.

Best regards,
Authors

Dear Reviewer BPdQ,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1] Can the authors provide a little more description about the "length control" aspect of AlpacaEval 2.0 in the main paper? This setting is used in nearly all results, but is not explained clearly where first introduced (Section 5.2)

Thank you for the constructive feedback. The length-controlled (LC) win rate in AlpacaEval 2.0 [1] is a newly introduced evaluation metric designed to reduce bias toward longer responses when using LLMs as judges [2,3]. To achieve this, a regression model is trained to separate the contributions of response length and quality, based on data from leaderboard submissions. The LC win rate then estimates win rates by neutralizing the effect of response length, focusing purely on quality. As demonstrated in [1], this metric correlates more closely with human evaluation [4] than the standard win rate. We have added these details about this “length control” aspect of AlpacaEval 2.0 in the revised draft (lines 304-305).

---

[W2] What is "gold label" (Table 1, 5)? Is this the Ultrafeedback preference data? Please make this explicit in the writeup

The term “gold label” refers to the “ground-truth preference label” provided by the UltraFeedback dataset. In our experiments, we use the UltraFeedback dataset in two distinct ways. First, a portion serves as the seed preference data, $\mathcal{D} = \{(x, y_l, y_w)\}$, directly using its ground-truth labels. Second, another portion of the dataset is used solely as a prompt set, $X_i = \{x\}$, by discarding the original labels $y_l$ and $y_w$ to allow to use it for new data generation at each iteration; thus, this portion does not utilize “gold labels.” Therefore, in Tables 1 and 5, we denote “gold label” to clarify the amount of ground-truth preference data utilized, ensuring a fair comparison between methods that rely on different quantities of labeled data. We explicitly mention this in the revised draft (lines 294-295).

---

[Q1] Typos.

Thank you for the careful reading and pointing out the typo! We have corrected this in the revised draft.

---

[1] Dubois et al., Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators., arXiv:24.04
[2] Wang et al., How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources., NeurIPS 2023 Datasets and Benchmarks Track
[3] Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena., NeurIPS 2023 Datasets and Benchmarks Track
[4] Chiang et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference., arXiv:24.03

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors