ROPO: Robust Preference Optimization for Large Language Models

This paper examines the unavoidable presence of noise in preference learning and its significant impact on the performance of Large LLMs. Previous research has only slightly reduced the negative effects of noise, which persists during the training phase. Additionally, efforts to filter out noisy samples often lead to increased computational costs. To address these challenges, the paper introduces the ROPO framework, which combines noise tolerance and the filtering of noisy samples. It also incorporates the technique of rejection sampling to further enhance performance. Specifically, the authors derive a loss function through mathematical derivation designed to suppress the gradients of samples with high uncertainty. This approach prevents the model from overfitting to noisy samples while simultaneously identifying them. The effectiveness of the ROPO framework is demonstrated across three datasets in both practical and artificially noisy scenarios.

The paper studies preference alignment under the condition when there are poorly-annotated preference pairs. The authors propose a robust preference optimization (ROPO) framework with two key considerations, (1) a noise-robust loss function that suppresses the gradients of samples that the policy model is uncertain about; (2) A robustness-guided rejection sampling technique designed to balance the filtering of noisy samples with the preservation of important information from queries that might otherwise be discarded.

In the experiments, the authors demonstrate that the policy model aligned with ROPO shows the least drop in performance (win rate against a reference model as judged by GPT-4) with an increasing proportion of injected noise in the training data. The injected noise includes both artificial noise, such as flipping the preference labels of training pairs, and practical noise, where responses from a larger model are blindly assumed to be preferred over those from a smaller model.

This paper introduces the RObust Preference Optimization (ROPO) framework, a method designed to improve preference alignment in large language models (LLMs) by addressing the challenges posed by noisy preference data. ROPO employs a noise-tolerant loss function and an iterative process that integrates noise filtering during training. Additionally, ROPO includes a robustness-guided rejection sampling technique to retain valuable data while filtering noise. Experiments show that ROPO outperforms existing methods under various noisy conditions, offering a scalable and effective approach to aligning LLMs with human preferences without the need for external models.

The paper presents the RObust Preference Optimization (ROPO) framework, a novel approach to improving preference alignment in LLMs under noisy conditions. By integrating a noise-tolerant loss function and robustness-guided rejection sampling, ROPO aims to mitigate the impact of noisy preference data while preserving valuable information. Experimental results across various datasets demonstrate that ROPO outperforms existing methods in both real and artificially noisy scenarios, achieving better alignment with human preferences without increasing computational overhead.

Strengths:

• The paper addresses the important challenge of annotator noise in preference alignment, which is a critical issue for the development of reliable policy models.
• It includes a theoretical analysis that outlines DPO's limitations in distinguishing between noisy and clean samples and introduces a regularization loss as a mitigation strategy.
• Experimental results show improvements over DPO in noisy scenarios.

Weaknesses:

• Novelty: The contribution of the proposed approach is limited, as the primary contribution is the addition of a regularization term, which is not significantly different from the original DPO approach. Furthermore, the use of loss values to identify model-uncertain samples in the robustness-guided rejection sampling procedure closely resembles concepts explored in prior work.
• Complexity: The method introduces multiple hyperparameters, such as the trade-off parameter alpha, the sample filtering ratio, and beta from DPO, which require extensive experimental tuning. This added complexity could reduce the practical value of the approach.
• Evaluation: The evaluation is constrained by the use of limited test datasets, primarily AlpacaEval and Reddit. This lack of coverage across multiple instruction-following benchmarks such as Wildbench, Arena-Hard, and MT-Bench limits the generalizability of the findings. Additionally, the paper does not sufficiently define or characterize noisy data, making it unclear how noise is identified and mitigated.
• Clarity: While Figure 1 outlines the framework of ROPO, the absence of an overall description makes it challenging to understand how ROPO's components—e.g., noisy sample filtering, rejection sampling, and noise tolerance training—are integrated into an iterative workflow. This could make the work difficult to reproduce.

The evaluation and clarity issues were partially addressed during the discussion period, with the addition of new experiments and a more formal presentation of ROPO in the appendix. However, the addition of significant new experiments (e.g., key results such as human evaluation) would ideally require a fresh round of reviewing. Furthermore, reviewers remain concerned about the novelty and complexity weaknesses mentioned above, even after considering the authors' latest comments and updated paper. In light of these concerns, I recommend rejection.

Reject