Diverse Preference Learning for Capabilities and Alignment

We thank the reviewer for their thorough technical feedback. We address each point below:

Theoretical Analysis and Derivations:

• We have corrected the copying errors in Equations 8, 25, and 26 in the appendix. These were transcription mistakes and do not affect the theoretical results. We thank the reviewer for catching these.
• Equations 22 and 23 are correct as written, since entropy has a negative sign: H(\pi) \= -\sum_y \pi(y) \log \pi(y)
• We've revised notation to use proportional symbols consistently
• We've renamed propositions and corollaries to reflect their section locations for clarity

Implementation Details: We have expanded Appendix B with comprehensive experimental details (including for all training runs) to aid reproducibility. We will also release our code upon acceptance. We include a few key pieces of information here:

• Reward model: Trained using Mistral-7B-Instruct with LoRA and reward head on HH-RLHF using Bradley-Terry loss. We note that since the Bradley-Terry loss is shift-invariant, only relative differences between reward values are meaningful.
• Diversity metrics: We updated the Section 4.1. Diversity Metrics section to connect our concrete diversity metrics with the conceptual types of diversity they measure: “our diversity metrics roughly measure 1) general semantic diversity (embedding distance metric), 2) logical diversity or diversity of viewpoints (logical disagreement metric), and 3) diversity in response content and ideas (content diversity metric)”. We also updated the appendix with the mathematical formula for embedding cosine distance and details on the embedding model used
• Sample generation: We use 16 samples per prompt to obtain robust per-input diversity measurements and low-variance quality estimates
• Hyperparameters: Hyperparameters and training data specifications for all training runs are now included

Choice of Models: We used Mistral-7B-Instruct for diversity experiments as it performed well in chat settings. For mathematical reasoning and calibration, we found better results using the Zephyr finetuning approach $1$ (which starts from Mistral-7B base), as it is a stronger finetuning recipe than using HH-RLHF. For consistency, we will update all experiments in the main paper to use the Mistral-base/Zephyr setup in the camera-ready version.

Additional Baselines: Thank you for highlighting relevant work on diverse preference learning. We have already implemented and will include full results comparing against f-DPO in the camera-ready version. However, Inverse RLignment addresses a fundamentally different setting (learning from demonstrations) than our work, which focuses specifically on understanding and addressing diversity loss in pairwise preference learning algorithms such as RLHF and DPO. Our theoretical analysis of KL regularization and proposed solution are specific to the preference learning setting, making direct empirical comparisons less meaningful. We have added a discussion of Inverse RLignment to our related work section to highlight these complementary approaches.

Mathematical Task Performance: While DPL shows mixed results on easy/medium difficulty problems, it demonstrates consistent improvements on hard problems in terms of accuracy and sample efficiency:

• GSM8K:
  • +10% vs vanilla DPO, +4% vs temperature-scaled DPO at 128 samples
  • DPL needs 84 samples to match DPO@128 (34% savings)
  • DPL needs 106 samples to match temp-scaled DPO@128 (17% savings)
• MATH:
  • +4% vs vanilla DPO, +1% vs temperature-scaled DPO at 128 samples
  • DPL needs 88 samples to match DPO@128 (31% savings)
  • DPL needs 113 samples to match temp-scaled DPO@128 (12% savings)

Regarding the meaning of DPO $t=1.2$ in Figure 3 – throughout the paper, we used the notation $t$ to refer to token-level temperature and $\\alpha / \\beta$ to refer to DPL’s global temperature. This is defined mathematically in Section 3.2. However, for the sake of clarity, we updated the Figure 3 caption to explain that t refers to token-level temperature.

Empirical $\\beta$ Range: We've strengthened our claim about typical $\\beta$ values with citations to foundational RLHF and DPO works which use $\\beta \\in$ 0.01, 0.1 $$: 1) OpenAI's InstructGPT, 2) Cohere's REINFORCE paper, 3) Original DPO paper, 4) Allen AI's Zephyr technical report

$1$ Zephyr: Direct Distillation of LM Alignment

We sincerely thank the reviewer for their positive and constructive feedback. We are glad they found our analysis of the mode collapse problem well-posed and convincing. We are also glad they appreciated our principled solution and found our experiments comprehensive.

We address their comments and questions below:

Missing Baselines: We thank the reviewer for the suggestion that including more thorough baselines would provide additional context for evaluating our method's performance. We have already implemented and will include full results comparing against top-k, top-p, and min-p sampling in the camera-ready version. We have updated Figure 2 with complete min-p experiments and will add additional results as they finish.

Preliminary results show DPL outperforms min-p sampling on the embedding and logical disagreement metrics, especially at high temperatures. However, on the content diversity metric, min-p performs slightly better than DPL. This is related to the non-monotonicity of this metric at higher temperatures. This is where DPL shines, and thus DPL looks worse on this metric. We discuss this in the next section of this review.

Regarding REINFORCE, we note that recent work replacing PPO with REINFORCE for LLM alignment (e.g., Wei et al. 2024) still incorporates KL regularization in the RL objective. However, if the reviewer had a specific REINFORCE variant in mind that excludes KL terms, we would be very interested in potentially including it as an additional baseline.

Points Below the Pareto Frontier (Figure 2): The reviewer raises a good point about the apparent brittleness suggested by points far below the Pareto frontier. These points reflect a phenomenon where high temperatures can actually decrease measured diversity on certain prompt-based metrics. This causes them to appear below the Pareto curve instead of to the right of it. Specifically:

• Content, surface form, and perspective diversity metrics (Appendix B) show non-monotonic relationships with high temperatures.
• Embedding-based and logical diversity metrics increase monotonically
• We clipped extreme outliers that would have appeared far to the left of the vanilla DPO point (these points were always token-level scaled or token-level scaled with min_p, never DPL). This hides some of the non-monotonic behavior for these methods.

We updated the results section of Section 4.1 with this discussion. We also labeled points exhibiting this kind of non-monotonic behavior with their parameter values in Figure 2 and Appendix B.

Figure 3 Readability: We thank both reviewers for noting the difficulty in reading Figure 3. We have updated it with:

• An improved color scheme (switching from green/blue to orange/blue)
• Greater contrast between different temperature gradients
• Added best-of-N accuracy numbers in tabular form in Appendix C

We thank the reviewer for their comments again and will continue to update the paper as we obtain full baseline results with top-p and top-k sampling.

We thank the reviewer for their constructive feedback, particularly their suggestion to include additional baselines, which has helped strengthen our empirical validation.

We're pleased to share that we have now included comprehensive baseline comparisons in Figure 2, including:

1. Min-p sampling
2. Top-p (nucleus) sampling
3. Top-k sampling
4. f-DPO, another preference learning algorithm intended to preserve output diversity (these results are in Figure 5 of the Appendix due to space constraints)

These results show DPL maintains competitive or superior performance against these established methods, particularly on embedding distance and logical disagreement metrics.

We have also enhanced the paper with:

• Improved Figure 3 readability with a new column showing accuracy gains relative to DPO
• An extensive theoretical and empirical analysis in Appendix C examining the relationship between temperature, diversity, and problem-solving performance

Given that we've addressed the reviewer’s concerns about baseline comparisons and Figure 2, would the reviewer consider raising their score? We thank the reviewer again for their feedback and would be happy to address any remaining questions or concerns.

We sincerely thank the reviewer for their thoughtful and detailed feedback. We appreciate their acknowledgment of our method's principled approach and comprehensive results. Below, we address the reviewer’s main points and questions.

Relationship Between Diversity and Problem-Solving Performance: The reviewer raises important questions about the relationship between temperature and problem-solving performance. We have significantly expanded our analysis to better illustrate these relationships:

1. Quantitative Improvements: While the visual presentation in Figure 3 may have obscured the gains, DPL shows consistent, meaningful improvements over baselines on hard splits:

• GSM8K:
  • +10% vs vanilla DPO, +4% vs temperature-scaled DPO at 128 samples
  • DPL needs 84 samples to match DPO@128 (34% savings)
  • DPL needs 106 samples to match temp-scaled DPO@128 (17% savings)
• MATH:
  • +4% vs vanilla DPO, +1% vs temperature-scaled DPO at 128 samples
  • DPL needs 88 samples to match DPO@128 (31% savings)
  • DPL needs 113 samples to match temp-scaled DPO@128 (12% savings)

2. Temperature-Performance Relationship: We have added an extensive analysis and discussion in Appendix C showing that:
   • Each problem has a unique optimal temperature, which is related to the problem’s difficulty (i.e. single-sample success probability). In particular
     1. Easy problems benefit from lower temperatures as they don't require extensive exploration
     2. Medium problems show mixed results as they operate in a transition regime
     3. Hard problems benefit from higher temperatures up to an optimal point, after which DPL degrades more gracefully than token-level scaling
   • Best-of-N sampling benefits from higher variance sampling strategies as N increases. We tie this to the concavity of the best-of-N success rate function.
     Our added analysis provides a theoretical explanation for the observed behavior and empirical evidence supporting this theory.
3. Best-of-N as Inference-Time Scaling: We note that best-of-N is a widely-used inference-time scaling approach. While we agree that exploring other scaling strategies (e.g., beam search) would be interesting, recent work shows they can underperform best-of-N in high-sample settings (e.g., https://arxiv.org/abs/2408.03314 Figure 3 left). Best-of-N sampling also has the advantage of relying only on additional samples, so improved performance of DPL does not correlate with other aspects of the inference-time scaling method. Overall, we believe best-of-N is a sufficiently robust, popular, and representative inference-time scaling approach to evaluate our method on.

Figure 3 Readability: We thank the reviewer for noting the difficulty in reading Figure 3. We have:

• Changed to a more distinct color scheme (orange/blue)
• Increased contrast between temperature gradients
• Added results in tabular form to Appendix C

Cause of Points Below the Pareto Frontier (Figure 2): These points reflect a phenomenon where high temperatures can actually decrease measured diversity on certain prompt-based metrics. This causes them to appear below the Pareto curve instead of to the right of it. Specifically:

• Content, surface form, and perspective diversity metrics (Appendix B) show non-monotonic relationships at high temperatures
• Embedding-based and logical diversity metrics increase monotonically
• We clipped extreme outliers that would have appeared far to the left of the vanilla DPO point (these points were always token-level scaled or token-level scaled with min_p, never DPL). This hides some of the non-monotonic behavior for these methods.

We updated Section 4.1 results with this discussion. We also labeled points exhibiting this kind of non-monotonic behavior with their parameter values in Figure 2 and Appendix B.

Additional Core Contributions: While the reviewer focused on problem-solving results, our paper makes several other key contributions:

1. We identify KL regularization as a central cause of diversity loss in aligned LLMs
2. We provide a theoretical analysis using social choice theory to show how KL regularization amplifies majority preferences.
3. We propose DPL as a principled solution that provably restores distributional representation.
4. We empirically validate our method across multiple settings beyond just problem-solving.

We thank the reviewer again for their thoughtful comments and note that this feedback was very helpful in improving the paper.