Self-Improving Robust Preference Optimization

We thank the reviewer for their valuable comments and insightful questions. We have addressed the concerns and queries raised by the reviewer in this response.

Questions:

What is $R^+_∗$ in line 162? It is the set of strictly positive real numbers.

Relation to Continual Learning: We thank the reviewer for highlighting the potential connection between our work and continual learning (CL), particularly in relation to the specific work cited. While there may be conceptual overlaps, we believe that our study is better framed within the context of continual self-improvement rather than traditional continual learning. The primary distinction lies in the approach we take: the variant of SRPO presented in this paper trains the self-improvement policy in a single, static training phase rather than iteratively adapting to new tasks or data distributions over time, as is typical in CL.

Continual learning often focuses on managing trade-offs such as overcoming catastrophic forgetting while adapting to new tasks or data. In contrast, our work centers on developing a robust self-improvement policy that enhances model outputs iteratively but within a fixed training framework. That said, extending SRPO to the continual learning setting, where the self-improvement policy evolves as new data or tasks are introduced, could be a promising direction for future research. Such an extension could explore dynamic adaptation of the self-improvement policy, potentially integrating ideas from continual learning to balance adaptation and retention in long-term iterative improvement processes.

We thank the reviewer for their valuable comments and insightful questions. We have addressed the concerns and queries raised by the reviewer in this response, as well as in the revised manuscript submitted alongside it. Additionally, we are in the process of preparing another revised manuscript, which will include new experimental results specifically aimed at addressing the reviewer’s concerns regarding the experiments.

Weaknesses

Similarity in Performance with baselines: It is accurate that, in the case of zero revisions, our method performs comparably to IPO and still outperforms DPO in the case of TLDR. (Though our new results on the challenging Arena Hard and our prior results on XSUM shows that SRPO outperforms both IPO and DPO w/o revision.) However, the true strength of SRPO becomes evident after multiple revisions, where it consistently achieves superior performance compared to IPO when subjected to the same number of revision cycles. This demonstrates that SRPO utilizes the same amount of compute more effectively, leveraging its iterative refinement process to deliver better outcomes than its competitors. By optimizing each revision step, SRPO ensures that the improvements compound over iterations, highlighting its efficiency in achieving high-quality results over multiple refinement steps.

Theoretical Justification for Combination Loss: We will formalize this theoretical result in Theorem 1, which will be thoroughly detailed in Sections 4.1 (Main result), 4.2 (Proof) of the revised submission (to be included with this rebuttal). This result serves to justify the use of the combination loss in our framework. Specifically, Section 4.2.3 will be dedicated to presenting a rigorous proof of the combination loss, providing a clear and comprehensive explanation of its mathematical foundations.

Questions

IPO/DPO multi-step prompt: Yes, for consistency and fairness in evaluation, we provide the models trained by IPO and DPO with the same in-context prompt used for SRPO. This ensures that all models are tasked with improving their outputs under identical conditions, allowing for a direct and meaningful comparison of their ability to refine and enhance completions. By using the same prompt structure, we aim to isolate the differences in performance attributable to the underlying training and optimization methods, rather than discrepancies in the prompts or contextual inputs.

Robustness of PPO: PPO and DPO optimize the same underlying objective, and as a result, they are both subject to similar limitations in terms of robustness. Specifically, their performance can be sensitive to the behavior policy $\mu$, which governs the data distribution used for training. This dependency can lead to a lack of robustness, as noted in prior work (e.g., Section 3.3 of Munos et al., 2023). While PPO is often used in online settings where the behavior policy can be dynamically adjusted, this sensitivity still poses challenges when transferring to offline or fixed-policy scenarios.

In this submission, we focus primarily on offline methods, as our goal is to address the challenges associated with static datasets. Consequently, we have chosen to compare SRPO against DPO, which is directly relevant to our context. However, extending the evaluation to include PPO under comparable conditions is an interesting direction for future work.

Over-optimization in SRPO: Yes, over-optimization is observed in SRPO as well. Similar to other direct alignment methods, SRPO experiences a decline in performance, measured by win-rate, after approximately 3-4 epochs of training. This phenomenon occurs because prolonged training can cause the model to overfit to the specific patterns in the training data or over-optimize for the alignment loss, potentially at the expense of generalization. In essence, the model becomes overly tailored to the preference data it has seen, which may not fully represent the diversity or complexity of real-world tasks.

To mitigate this issue, careful strategies such as early stopping, regularization techniques, or validation-based checkpoints could be employed to prevent over-optimization during training. Additionally, exploring ways to improve the diversity and representativeness of the training data, or leveraging ensemble methods, may help SRPO maintain robust performance over extended training epochs. Further analysis and experimentation are warranted to understand the nuances of over-optimization in SRPO and develop strategies to address it effectively.

Munos et al 2023: Nash Learning from Human Feedback

In response to the reviewers' concerns regarding the similarity in performance between SRPO and other baseline methods, when using same amount of compute, we would like to direct attention to our latest results. Specifically, on the challenging Arena Hard benchmark, SRPO demonstrates consistent and significant improvements over a range of prominent preference optimization methods, even without additional revisions (i.e., no significant extra compute needed at inference time to achieve better performance). A snippet of these results is presented in the following table (Full report of results is included in Appendix D of revised manuscript):

Win-rates against Llama 3.1 8B Instruct (Judged by GPT 4)

This evidence provides a compelling demonstration of SRPO's superiority over prior approaches, especially in, Arena hard arguably one of the most challenging benchmarks in generative AI.

We thank the reviewer for their valuable comments. We have addressed the concerns and queries raised by the reviewer in this response, as well as in the revised manuscript submitted alongside it. Additionally, we are in the process of preparing another revised manuscript, which will include new experimental results specifically aimed at addressing the reviewer’s concerns regarding the experiments.

Weaknesses:

The paper lacks multiple empirical comparisons: We are currently evaluating SRPO on the ultra-feedback dataset, which has been designed to provide deeper insights into the model's ability to align with complex human preferences. The ultra-feedback dataset allows for a more rigorous testing of SRPO's performance across diverse scenarios, helping to assess its scalability and effectiveness in handling nuanced feedback. We will present the results of these evaluations, including detailed analysis and comparisons, in a revised draft that will be submitted before the rebuttal deadline.

Comparison with SimPO and RPO: We are in the process of launching experiments for SimPO and RPO to thoroughly evaluate their performance in comparison to SRPO. The results of these experiments, along with a detailed comparison between SimPO, RPO, and SRPO, will be included in the revised version of the manuscript, which will be submitted prior to the rebuttal deadline. Additionally, we will expand the discussion in the related work section to provide a comprehensive comparison of SimPO and RPO, highlighting their methodologies, strengths, and limitations relative to SRPO. This will ensure that the context and significance of our contributions are clearly articulated.

Significance of improvement: We acknowledge that SRPO involves a trade-off between achieving higher-quality completions and the additional computational overhead required to compute these improved completions. However, we believe this trade-off is justified when the additional compute results in significantly better performance. Utilizing extra computation enables us to achieve high-quality outputs even with smaller models, offering several practical advantages. For instance, this approach facilitates the deployment of high-performance language models on edge devices or resource-constrained environments without demanding excessive memory or computational resources. By optimizing the balance between computational cost and model performance, SRPO provides a pathway for creating efficient, scalable, and portable language solutions that meet diverse application needs.

In response to the reviewer request for more experiments, we conducted additional experiments to evaluate the self-improvement capabilities of SRPO by post-training on a Llama 3.1 8B base model using Ultra Feedback dataset (Cui et al 2024), and comparing its performance against IPO and DPO. Below is a snippet of the results (the full results is available in the Appendix D of the revised manuscript) using the Arena-Hard prompts.

Win-rates against Llama 3.1 8B Instruct (Judged by GPT 4)

The new findings further reinforce our previous observations: SRPO significantly outperforms both DPO and IPO after one revision. Additionally, SRPO demonstrates competitive performance with IPO (within margins of error) in the zero-revision scenario, while also outperforming DPO at this stage.

Cui et al 2024 UltraFeedback: Boosting Language Models with High-quality Feedback

-------------------------------------------------------------------------------Important Result Update------------------------------------------------------------

We conducted a more comprehensive evaluation of our results by calculating the AI win rates across all checkpoints sweep trained with SRPO, IPO, and DPO (due to AI API quota limit we could not do this exhaustive hyper sweep before).

The analysis revealed that, if it is tuned carefully, SRPO consistently outperforms all baselines by a significant margin across all revisions. In contrast, similar exhaustive evaluations and hyper-parameter tuning for IPO and DPO did not show any notable improvements in comparison to previously reported results. A snippet of updated win-rates on Arena Hard are presented below (complete report of our result will be included in the revised manuscript).

Win-rates against Llama 3.1 8B Instruct (Judged by GPT 4--- UPDATED---)

In response to the reviewers' request for a comparison between SRPO, SIMPO, and RPO, we provide a snippet of win-rate results in the table below trained on Ultra-feedback dataset and evaluated on the challenging Arena-Hard prompts. (Full report of results is included in Appendix D of revised manuscript.)

Win-rates against Llama 3.1 8B Instruct (Judged by GPT 4)

Our analysis indicates that SRPO outperforms both SIMPO and RPO by a significant margin across all metrics. Notably, this superior performance is achieved despite SRPO not utilizing the additional cross-entropy regularization term present in RPO, which is known to enhance performance of RPO in comparison with vanila DPO.

We thank the reviewer for their valuable comments and insightful questions. We have addressed the concerns and queries raised by the reviewer in this response, as well as in the revised manuscript submitted alongside it. Additionally, we are in the process of preparing another revised manuscript, which will include new experimental results specifically aimed at addressing the reviewer’s concerns regarding the experiments.

Weaknesses:

1. Our derivations in Section 4 establish a theoretical foundation for the convergence of SRPO to the minimax solution of the objective defined in Equation 1. In the revised submission (to be included with this rebuttal), we will formalize this result as Theorem 1, detailed in Sections 4.1 (Main result) , 4.2 (Proof). This theorem demonstrates that the complex optimization objective of SRPO can be reformulated as a standard supervised learning problem, which is scalable and solvable using an offline human preference dataset.

2. In the revised submission (to be included with this rebuttal), we have refined the motivation in Section 2 to more effectively highlight the importance of learning self-improvement policies and provide a detailed explanation of why this is critical for training GPT-style models. Specifically, we address the issue of exposure bias in language models, a challenge that arises because these models generate tokens sequentially based on their prior outputs. This sequential generation can lead to error accumulation when an incorrect token is produced, propagating mistakes throughout the output. We emphasize that self-refinement models offer a promising solution to this problem by enabling models to correct their own outputs. However, training such models through supervised fine-tuning (SFT) requires constructing datasets with both initial completions and their corrected versions, a process that is resource-intensive and susceptible to overfitting to specific patterns or neglecting the refinement process altogether. To address these limitations, we introduce SRPO as an alternative approach based on direct preference optimization. SRPO teaches models to align less-preferred outputs with more-preferred ones, focusing on the mechanics of improvement rather than simply predicting the most-preferred completion. This approach provides a more scalable and natural framework for learning self-improvement policies.

3. We are in the process of preparing these evaluations and will include head-to-head comparisons in a revised submission prior to the rebuttal deadline.

Questions:

1. In Section 2 of the revised manuscript (to be submitted with this rebuttal), we will provide a more in-depth explanation of the significance of self-improvement policies. Specifically, we will emphasize how self-improvement can effectively address the issue of exposure bias in language models, a critical challenge where error accumulation arises due to sequential token generation. Additionally, we will highlight the necessity of advancing existing methods for training self-improvement policies to enhance their effectiveness and scalability.

2. We will include detailed information on computing the win rate against humans in Appendix C.3 of the revised manuscript (to be submitted with this rebuttal). Regarding the accuracy of AI evaluators compared to humans, we refer the reviewer to prior work, such as Arawjo et al. (2023), which establishes the reliability and accuracy of AI-based evaluators.

Arawjo et al. (2023): Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences.