Sample-Efficient Alignment for LLMs

Thank you for your insightful questions and valuable suggestions to add more experiments. We hope to address your concerns below.

Q1: Why is SELM excluded?

Thank you for raising this point. From both the SELM paper and our personal communication with the authors, we understand that SELM is theoretically equivalent to XPO. Their release dates are also nearly simultaneous (May 29, 2024 for SELM vs. May 31, 2024 for XPO), and we consider them concurrent works based on the same core idea. This was our primary reason for not including SELM separately in our experiments, as noted in line 794.

Additionally, we observed that SELM's official implementation uses an offline preference dataset for training, which diverges from both their proposed algorithm and the online alignment setting our work focuses on. We mention this discrepancy in the text for transparency, although it was not the reason for omitting SELM as a baseline.

We also wish to clarify a potential misconception: training with an offline dataset, as in SELM’s implementation, does not qualify as offline Best-Arm Identification (BAI). Both E&E (Exploration & Exploitation) and BAI assume an online interaction loop, where responses are generated and feedback is collected in real time. The distinction lies in evaluation:

• E&E focuses on the quality of every online decision,
• BAI evaluates the final learned model after online training.

We will revise the text to make these clarifications more explicit.

Q2.1 (Weakness 1): Novel aspects of SEA

The main novelty of this paper lies in clarifying E&E and BAI scenarios in the context of LLM alignment, proposing a unified Thompon sampling-based algorithm for both E&E and BAI, and providing novel solutions to the technical challenges when conducting TS in high dimensional space like LLM. These developments have practical implications that can more efficiently scale LLM alignment from human preferences. We also present extensive empirical evidence to validate the method's effectiveness with an extendable codebase that can benefit future research.

We will revise the manuscript to make these contributions more explicit and better emphasize the technical and practical significance of SEA.

Q2.2 (Weakness 2): How does SEA mechanistically differ from prior methods?

Although APL and XPO aim to improve sample efficiency through exploration, both are tightly coupled to DPO as their loss function. Specifically:

• APL selects pairs with the largest implicit reward margin (from DPO) to guide exploration, relying entirely on the DPO reward proxy.
• XPO adds an explicit optimism term to the DPO loss, effectively injecting a global exploration bias but still constrained within the DPO framework.

In contrast, SEA introduces a loss-agnostic, algorithmic framework based on contextual dueling bandits and Thompson sampling (TS). Instead of relying on DPO-specific signals, SEA maintains a posterior over reward models and samples duels in a Bayesian manner to explore uncertainty regions more systematically.

Key mechanistic differences include:

• Exploration principle: SEA uses posterior sampling (TS), while APL relies on handcrafted scoring functions within DPO and XPO encourages global optimism (also within DPO).
• Policy optimizer independence: SEA is compatible with any downstream optimizer (e.g., DPO, IPO, SLiC), while APL/XPO are tied to DPO.

We believe this principled, modular approach enables broader applicability and better scalability in online alignment. We will revise the paper to highlight this distinction more explicitly.

Q3: Difference between the 'policy-guided search and mixed preference learning techniques', and 'sampling-based optimization in bandit literature and contrastive training and model-based RL concepts'

Thanks for this insightful question. "Policy-guided search" is a computationally efficient way of getting the "best arm" in the large language response space, thus it's related to high-dimensional approximate sampling in the literature [1,2]. In a simple bandit, once the reward model is sampled, finding the best arm for the sampled reward model is computationally easy. This becomes computationally difficult in the LLM setting as the number of “arms” grows exponentially with sequence length motivating the use of sampling-based optimization. Our novelty lies not in this part, but in the “mixed preference learning” part where we propose a method for efficiently maintaining a sampler to sample good sequences for a mix of reward functions instead of sampling for a single reward function in sampling-based optimization. In "mixed preference learning", we mix in both the real preference data together with preferences labelled using all the different reward models that we maintain, when learning the LLM policy. The learned policy can hence be used to sample sequences that are good for all the different reward models rather than for a single reward model. The optimization is usually done with contrastive training (e.g., DPO). This is also related to model-based RL where the world model's simulated data can be used to learn the policy; here our world model contains the set of the different reward models that are still being explored.

Limitations: 0.4B RM and Pythia may be outdated and the robustness of SEA under noisy feedback

Thank you for this valuable suggestion. To further validate the generality of SEA, we conducted additional experiments with the following new setting:

• Base LLM: meta-llama/Llama-3.2-1B-Instruct
• Prompt Dataset: HuggingFaceH4/ultrafeedback_binarized (with 60k/2k train/test splits)
• Reward Oracle: Skywork-Reward-V2-Llama-3.1-8B
• Reward Model Backbone: Skywork/Skywork-Reward-V2-Llama-3.2-1B
• Preference noise: preference is flipped with 10% probability if enabled
• Evaluation temperature: 0.6

This setup uses a different model family (Llama 3) and a more realistic dataset (UltraFeedback). We also leverage a stronger reward oracle (Skywork-V2-8B) and a larger reward model backbone (Skywork-V2-1B).

The table reports win rates against the initial model during training. SEA-DPO consistently learns faster and converges to a higher win rate than Online-DPO, reinforcing the effectiveness and generality of our approach across model families and datasets.

In addition, we tested both SEA and online DPO under noisy preference feedback (with 10% chance the preference is flipped). As shown in the table, SEA is robust to feedback noise and shows superior learning efficiency and final performance over the baseline.

---

Thank you again for your detailed feedback and suggestions, and we hope our responses and new experiments were able to address any remaining concerns. Please do let us know if you have any further questions as well as what would be expected for score improvement.

---

[1] Mazumdar, Eric, et al. "On approximate Thompson sampling with Langevin algorithms." international conference on machine learning. PMLR, 2020.

[2] Osband, Ian, et al. "Approximate thompson sampling via epistemic neural networks." Uncertainty in Artificial Intelligence. PMLR, 2023.

Thank you for recognizing the scientific value of our work and its originality, as well as your constructive feedback. We hope to address your concerns below.

Q1. Include more validation points with at least one dataset and another model family Thank you for this valuable suggestion. To further validate the generality of SEA, we conducted additional experiments with the following new setting:

• Base LLM: meta-llama/Llama-3.2-1B-Instruct
• Prompt Dataset: HuggingFaceH4/ultrafeedback_binarized (with 60k/2k train/test splits)
• Reward Oracle: Skywork-Reward-V2-Llama-3.1-8B
• Reward Model Backbone: Skywork/Skywork-Reward-V2-Llama-3.2-1B
• Evaluation temperature: 0.6

This setup uses a different model family (Llama 3) and a more realistic dataset (UltraFeedback). We also leverage a stronger reward oracle (Skywork-V2-8B) and a larger reward model backbone (Skywork-V2-1B).

The table reports win rates against the initial model during training. SEA-DPO consistently learns faster and converges to a higher win rate than Online-DPO, reinforcing the effectiveness and generality of our approach across model families and datasets.

In addition, following reviwer amjx's suggestion, we tested both SEA and online DPO under noisy preference feedback (with 10% chance the preference is flipped). As shown in the table, SEA is robust to feedback noise and shows superior learning efficiency and final performance compared to the baseline.

Q2. Add more technical description for Algorithm 2, put both Algorithm 2 and Figure 5 in the main body

Thank you for the helpful suggestion to improve the clarity and accessibility of our method. Due to space constraints in the original submission, we placed Algorithm 2 and Figure 5 in the appendix. In the camera-ready version, we will leverage the additional page allowance to move both into the main body, as recommended.

We will also expand the technical description of Algorithm 2 to clarify its key steps. In particular, the additional computational overhead of SEA compared to vanilla Online DPO arises from two components:

1. Over-sampling candidate responses for each query to construct duels. This step is embarrassingly parallelizable, and the number of candidates can be tuned to trade off compute cost with labeling/sample efficiency.
2. Inference and training of Epistemic Reward Models, which quantify uncertainty. These models are lightweight compared to LLM generation, requiring only a single forward pass per sample, and can be further optimized using techniques like LoRA.

Overall, while SEA introduces some additional computation, it remains highly efficient and practical in online learning settings, and we will clarify this in the revised algorithm description.

Q3. Include an estimate of the energy consumption and the carbon emission

Thank you for raising this important point. All experiments were conducted using NVIDIA A100 GPUs. Across all runs (including multiple seeds and ablations), we estimate a total GPU usage of approximately 17,000 GPU hours, which corresponds to an estimated 1,275 kg CO2 emissions, based on the Machine Learning Emissions Calculator [1].

We were mindful of our compute usage and followed standard practices to limit unnecessary runs while ensuring statistical significance. We will include this estimate and the reference to the emissions calculator in the revised version.

---

Thank you again for your detailed feedback and suggestions, and we hope our responses and new experiments were able to address any remaining concerns. Please do let us know if you have any further questions as well as what would be expected for score improvement.

---

[1] Lacoste, Alexandre, et al. "Quantifying the carbon emissions of machine learning." arXiv preprint arXiv:1910.09700 (2019).

Thank you for your insightful comments and suggestions to include related work. We will include this relevant paper appropriately in our revision.

We also hope to mention that we have added new experiments with another model family (Llama) and a more realistic dataset (UltraFeedback), and even with noisy preference feedback to address other reviewers' concerns. Please feel free to take a look at the results, which we believe further strengthen the practical applicability of the proposed method.

---

Thank you again for your insightful feedback, and we hope our responses and new experiments were able to address any remaining concerns. Please do let us know if you have any further questions as well as what would be expected for score improvement.

Thank you very much for your support!

Best regards,

The authors