SAIL: Self-improving Efficient Online Alignment of Large Language Models

Thank you for your detailed comments. We are currently running the requested experiments and will post our complete responses with results soon. We appreciate your patience.

Why did the author just include MMLU as the downstream task metric? They should incorporate more tasks (e.g., ARC-Challenge) like the similar self-improvement work SPIN (ICML24) to better illustrate their contribution.

Response: Thank you for your suggestion. Let us first explain why we did not apply many different downstream task evaluation datasets to our experiments. One reason is that we have incorporated 5 other metrics including the widely used MT-Bench scores and AlpacaEval 2.0 length-controlled win-rates. Another reason is that the UltraFeedback fine-tuning dataset we used is primarily designed to consider 4 different aspects, namely instruction-following, truthfulness, honesty, and helpfulness, and therefore it may not be very useful to improve the model's capability on reasoning datasets like MMLU and ARC-Challenge.

Nevertheless, we agree that adding more evaluation datasets would strengthen our experimental analysis. Following the reviewer's suggestion, we added the ARC-Challenge dataset as part of the evaluation and reconducted the experiments on Llama-3 (8B) and ARC-Challenge. We see similar observations as on MMLU, the SAIL methods bring larger improvements than the DPO baseline.

The results show that our improvements are larger than the DPO baseline, although the baseline improvement is small.

In the alignment area, it's better to conduct experiments in the Arena-Hard benchmark since it's a common metric to evaluate the alignment ability.

Response: Thank you for your suggestion. We agree that the Arena-Hard benchmark is recently becoming a widely used benchmark. We use the Arena-Hard-Auto repository and adapt their newly introduced Style Control (SC) method, which follows an update of Chatbot Arena. Following the reviewer's suggestion, we added the Arena-Hard benchmark as part of the evaluation, and reconducted the experiments on Llama3 (8B). The observation is similar as on MT-Bench, where we clearly see the SAIL methods can lead to significantly larger improvements than the DPO baseline. We plan to add Arena-Hard evaluations to other experiments in the manuscript soon.

The method does not improve much in the AlpacaEval 2.0 Score. The author should give a detailed explanation. And why not use metrics like length-controlled win rate?

Response: Thank you for your careful observation and question. We would like to clarify that we are already using the length-controlled (LC) AlpacaEval 2.0 win-rate metric in our evaluations. We will make this clearer in the table header of Table 3.

Regarding the fact that the AlpacaEval 2.0 scores on LLama-3 (8B) do not improve compared to the baselines, we believe this is because our base model, the instruction-finetuned LLama-3 (8B), is already trained to perform exceptionally well in terms of helpfulness, which is the focus of the AlpacaEval benchmark. Additionally, the preference dataset we used, UltraFeedback, may not provide significant further enhancement in the helpfulness aspect. This is supported by the slight decrease observed in the AlpacaEval score for the standard DPO baseline as well (see Table 3, results on LLama-3). Therefore, we think these AlpacaEval 2.0 results on LLama-3 (8B) may not indicate that SAIL is ineffective; it may be simply caused by an ill-suited combination of base model, finetuning dataset, and evaluation benchmark.

We also further conducted experiments on the Zephyr (7B) model as the backbone, whose AlpacaEval 2.0 win-rate is lower. We still train on the UltraFeedback preference dataset and the other experiment setups are unchanged. In this experiment, we see a larger improvement of the SAIL method compared to the standard DPO baseline (Zephyr-7B-Beta).

Authors should compare more advanced preference optimization algorithms like ORPO and SimPO. And current results are not impressive for the alignment community.

Response: Thank you for raising this insightful point. We see ORPO and SimPO are two recent work which propose a different objective than the standard RLHF, and achieve remarkable improvements in terms of alignment performance and efficiency.

Our work focus more on bringing standard RLHF to a bilevel optimization framework and propose an effective and efficient approximate algorithm on top of it. We can see some new preference optimization methods including ORPO and SimPO have one fundamental difference from our approach: they do not explicitly incorporate the KL regularization term. The absence of the KL regularization term allows these methods to optimize more aggressively for the reward function by deviating significantly from the reference model. In contrast, our approach is specifically grounded in the standard RLHF, where the KL regularization term ensures that the model remains aligned with the reference distribution while optimizing for the reward function. This distinction makes direct comparisons with ORPO or SimPO less meaningful theoretically, as those methods omit the KL regularization and adopt a fundamentally different optimization objective design.

However, we think our work, although developed adhering to the standard RLHF setup, can be compatible and combined with some recent advanced preference optimization algorithms, despite their differences in optimization setups and objectives. This is because we can reformulate their alignment problem as bilevel optimization, and go through the derivation as done in the paper. Taking SimPO as an example, we can treat their reward model definition (Equation (4) in the SimPO paper) as the solution of the upper level optimization (replacing Equation (4) in our manuscript), and adopt their modified Bradley-Terry objective with reward margin (Equation (5) in the SimPO paper) to replace the standard one (Equation (10) in our manuscript). By applying these changes and rederiving the extra gradient terms, we can formulate an adaptation of our method to the SimPO objective. We will implement this combined algorithm, which adapt our methodology to the SimPO objective, and compare with the SimPO as a baseline.

Recently many different alignment objectives and algorithms have emerged; it is an interesting question to discuss the compatibility and combination of our method with each objective. We will add more relevant discussions to the appendices, but due to the fact that the compatibility problem with each design is a non-trivial question, this process may incur considerably more work, and we hope the reviewer understands that this effort cannot be fully reflected by the rebuttal period. But we will continue to expand the discussion as the wide compatibility to other designs also strengthens our contribution to the community. We thank the reviewer for raising this insightful point.

Limited Exploration of Alternative Utility Functions: The method relies on the Bradley-Terry preference model, which may not be optimal for all RLHF applications. Future work could benefit from exploring alternative utility models that account for more nuanced preference data. SAIL currently relies on the Bradley-Terry preference model. Have you considered experimenting with other preference models, and do you anticipate any impact on alignment performance if different utility functions are used?

Response: Thank you for this insightful comment. Indeed, our current method relies on the Bradley-Terry (BT) preference model, and exploring alternative preference models is an exciting direction for future work. Since this is one of the initial works establishing a rigorous foundation for iterative RLHF, we focused on fundamental methods to clearly convey the core idea of Bilevel RLHF.

Our work reveals a crucial insight: the BT preference model plays a critical role in ensuring strong concavity of the lower-level problem within our bilevel optimization framework. This mathematical property enables us to derive a closed-form solution, which is key to simplifying the bilevel problem into single-level optimization using the DPO trick. However, this approach may not readily extend to more complex or non-convex preference models, as they could introduce additional optimization challenges.

We agree that extending the framework to accommodate alternative utility functions, particularly those capable of capturing more nuanced or domain-specific preferences, is a valuable research direction. Exploring these extensions could uncover interesting trade-offs between expressiveness, computational feasibility, and alignment performance, and we plan to address this in future work.

Scalability Concerns for Larger Models: Although the paper demonstrates SAIL’s effectiveness on LLMs with up to 8B parameters, additional scaling experiments would strengthen the paper's claims about computational efficiency for significantly larger models. The paper demonstrates SAIL's efficiency with models up to 8B parameters. Could you share any considerations or expected challenges for scaling SAIL to significantly larger models, such as those with over 100B parameters?

Response: Thank you for this insightful question regarding the scalability of SAIL to larger models exceeding 100B parameters. We would like to share our considerations and expected challenges:

1. Primary Overhead Sources: For the main SAIL methods—SAIL-PP and SAIL-PR—the major overhead compared to standard DPO comes from response generation and reward evaluation. The additional gradient terms computed (as per Equations (9) and (13)) are low-dimensional relative to the model parameters or inputs. This results in minimal time and memory overhead, even for models with over 100B parameters.
2. Challenges Similar to Online RLHF Training: Scaling SAIL to larger models involves challenges common to most online RLHF training methods. To achieve computational efficiency and enable training on machines with limited resources, we recommend using Parameter-Efficient Fine-Tuning (PEFT) techniques not only for training but also during generation, as we have implemented in our code.
3. Technical Considerations: There may be additional overhead when switching between training and generation modes, as well as interfacing with the reward model. Utilizing an optimized training framework that minimizes these overheads is crucial. Our current implementation adapts TRL's DPOTrainer, but it is not fully optimized or tested for models larger than 100B parameters. Further optimization is needed to handle the increased scale effectively.

We believe that with these considerations and optimizations, SAIL can be effectively scaled to significantly larger models while maintaining computational efficiency.

Regarding the three variants of the SAIL method, Table 3 shows that in the Eval-Reward and MT-bench columns, the SAIL method performs worse than the baseline DPO. Please clarify whether these experimental results undermine the assertion that the SAIL method is superior to the baseline DPO.

Response: Thank you for your thorough analysis of our experimental results. In Table 3, we observe that among our variants, only SAIL-DP demonstrates marginally lower performance than the baseline DPO in Eval-Reward and MT-Bench metrics. However, this observation does not affect our broader conclusions regarding the effectiveness of our two primary SAIL implementations: SAIL-PR and SAIL-PP.

Let us clarify the key points:

• SAIL-DP employs a distinct methodology, utilizing responses from the offline dataset with self-generated preference labels. This contrasts with SAIL-PR and SAIL-PP, which generate responses online. Additionally, SAIL-DP operates with a reduced number of preference labels compared to standard DPO.
• While SAIL-DP shows slightly decreased performance in Eval-Reward and MT-Bench metrics, it achieves notable improvements in Reward Margin. This is particularly significant given its reduced preference label requirements and minimal computational overhead.

These findings support our overall conclusion regarding SAIL methods' superiority over baseline DPO. We will enhance the manuscript to better articulate the distinct characteristics and trade-offs of each SAIL variant.

There is a large amount of blank space below Section 6.1. Is there any missing content in this part of the paper?

Response: Thank you for pointing this out. The blank space below Section 6.1 is not due to missing content; it is a LaTeX formatting problem. We will address this in the updated manuscript.

As a practitioner, at least the presentation/writing wasn't clear enough to agree that SAIL provides a unified framework for those who might want to consider using online RLHF in future works. I would personally suggest adding a section explains about how one could use SAIL instead of iterative DPO methods, as well as a huge emphasis on how the provided code could be used.

Response: Thank you for this valuable suggestion. We will enhance the manuscript by adding a paragraph that addresses the limitations of current online iterative RLHF methods. In the final draft, we will expand upon the following points to better articulate the significance of SAIL:

• We will emphasize that iterative methods fail to account for interdependencies during the reward learning phase, specifically the dependency of policy-generated trajectories that result in distribution shift.
• To address these dependencies in a principled manner, we demonstrate the necessity of reformulating the alignment problem as a bilevel optimization problem, as expressed in equation (3).
• While bilevel optimization presents significant computational challenges due to its requirement for complex second-order information, making it computationally intensive.
• To overcome this, we leverage RLHF's special structure and the closed-form solution of the KL-regularized problem to transform it into a single-level problem without compromising generality, leading to our proposed SAIL approach.
• Finally, we develop a self-improvement mechanism that replaces the human-in-the-loop component by utilizing the implicit reward function as defined in equation (11).

There is a huge emphasis on trying to improve reward models (on RewardBench) to mitigated reward model overoptimization & train better LMs. I am curious if given a fixed budget/time limit, whether one should try to employ online RLHF methods or try to enhance reward models in general.

Response: Thank you for raising this insightful point. Indeed, there has been significant emphasis on improving reward models (through initiatives like RewardBench and new VLM benchmarks, particularly from AllenAI etc.) which has successfully addressed certain issues such as length bias. While we acknowledge the value of this approach in addressing specific challenges, we believe the underlying issue is more fundamental and encompasses response quality more broadly. The effectiveness of reward models is intrinsically dependent on training with optimal or high-quality response pairs. However, this presents a significant challenge, as it necessitates training on an extensive corpus of responses to ensure comprehensive coverage.

Our proposed bilevel optimization framework addresses this challenge by providing an efficient mechanism for concurrent training of the reward model and policy. This approach enables dynamic collection of task-relevant response pairs, resulting in more targeted and effective training.

I would suggest adding an explanation of what is the limitation of online RLHF methods that the paper could not address. For example, it is still unclear on what is the best practice to "whether to discard instances from a preference dataset that have a subtle difference on the preference strength" or "would it be beneficial to employ more models when gathering responses when consisting a preference dataset".

Response: Thank you for this valuable suggestion regarding the limitations of online RLHF methods. We will include a comprehensive discussion of these limitations in the revised manuscript. Our theoretical insights and experimental analysis reveal an important finding: preference datasets containing diverse responses yield more informative gradients, which are essential for effective model updates. Conversely, responses with only subtle differences in preference strength generate minimal gradients, resulting in negligible improvements.

Our work leaves several promising directions unexplored. One particularly intriguing possibility is the development of a curriculum-based approach that initially leverages diverse responses and progressively incorporates responses with closer preference values. Such an approach could optimize the learning process by capitalizing on response diversity in early stages while refining alignment as the model converges. This aligns with the natural progression we observe in model training, where response similarity tends to increase as the model approaches convergence, particularly in scenarios with low uncertainty in optimal response generation. This area represents a promising avenue for future research.

We have now posted detailed responses to questions that do not heavily depend on experimental validation. We are diligently working on the remaining experimental evaluations and will provide comprehensive results, along with any necessary response updates, in the coming days. We sincerely appreciate your thoughtful feedback and understanding as we work to thoroughly address all comments and strengthen our paper.