Perplexity-aware Correction for Robust Alignment with Noisy Preferences

The paper proposes Perplexity-Aware Correction (PerpCorrect) to improve the alignment of large language models (LLMs) by detecting and correcting noisy preferences (NPs). PerpCorrect identifies NPs using the differences in perplexity (PPLDiff) between chosen and rejected responses. The method involves aligning a surrogate LLM with clean validation data, refining it with reliably clean and noisy data, and correcting NPs based on PPLDiff. Experiments demonstrate that PerpCorrect enhances alignment performance, is practical with a small amount of validation data, and is compatible with various alignment techniques.

This paper presents a novel method called PerpCorrect for robust alignment in large language models, particularly in the presence of noisy preferences in training data. PerpCorrect addresses noisy preferences by evaluating the perplexity difference (PPLDiff) between selected and rejected responses. By leveraging PPLDiff to identify and rectify these noisy preferences, PerpCorrect generates a denoised training dataset. Experimental results demonstrate that PerpCorrect achieves state-of-the-art alignment performance with only a small amount of validation data.

The paper introduces Perplexity-aware Correction (PerpCorrect), a method for robust alignment of large language models (LLMs) with noisy preferences (NPs). PerpCorrect detects and corrects NPs by analyzing the perplexity difference (PPLDiff) between chosen and rejected responses. The approach involves aligning a surrogate LLM with clean validation data, iteratively refining with reliable training data, and ultimately producing a denoised training dataset. Experiments demonstrate that PerpCorrect significantly improves alignment performance and is compatible with various alignment techniques.

The paper proposed a novel method to mitigate the preference noise in alignment. The authors first provide insights into how the PPLDiff can recognize the noisy preferences and then use the PPLDiff to select and correct noise preferences. Extensive experiments demonstrate that the method can significantly improve the robustness under high noise ratios.

We thank all reviewers for their insightful comments and suggestions. Please find our replies below.

[R1] We discuss the complexity and additional computation time of method as follows.

The additional computation time is primarily due to PerpCorrect. Both the theoretical and practical times are reported in the table below (using XXX to represent the theoretical time required for robust alignment) and are followed by a detailed analysis.

In theory, during the PerpCorrect process, we need to calculate PPLDiff and train the surrogate model in each epoch. The computation time introduced by PerpCorrect (Stage II) is approximately $\frac{T}{3}$ that of the robust alignment (Stage III and baseline methods).

• The calculation of PPLDiff in each epoch requires only $\frac{1}{3}$ of the time needed for robust alignment (Stage III). The primary computational load in robust alignment arises from the complexity of forwarding and back-propagation, while the complexities of gradient updates and parameter updates are relatively low. Additionally, back-propagation takes twice as long as forwarding. In addition, the calculation of PPLDiff only requires forwarding.
• For surrogate model training, PerpCorrect utilized data points that represented $t\times \alpha$ of the total dataset during epoch $t$. Since both $\alpha$ and $t$ are small, the time required for surrogate model training can be approximately ignored.

In practice, Our entire robust alignment pipeline (~24h) takes only twice as long as the baseline (~12h). We set $T=5$ and $\alpha=2\%$, and used the AdamW optimizer. The practical efficiency of the PerpCorrect is due to the use of fp32 precision by the AdamW optimizer, which increases the GPU's calculation time during the robust alignment process.

[R2] We provide a discussion about the need for clean datasets and potential solutions as follows.

We only need 50 clean data points, which constitutes less than 0.5% of the entire dataset. In practice, such small-scale annotation is feasible. Reviewing and labeling data are essential steps to enhance data quality. For instance, Scale AI utilizes a large workforce to review and ensure the high quality of their data [1].

To address the challenge of obtaining sufficient clean datasets, one potential approach is to employ the LLM-as-a-judge method, which allows models to self-annotate and thereby reduce reliance on manually curated clean data. This method has been discussed in recent works [2] [3] [4], and could serve as a promising solution to this limitation.

[R3] We repeated the experiments three times with the same settings for Tables 3, 4, and 6, and the average reward accuracy and standard deviation are reported in the PDF file.

[1] Scale AI, https://scale.com/docs

[2] Self-Rewarding Language Models, ICML 2024

[3] Constitutional AI: Harmlessness from AI Feedback, Arxiv 2024

[4] Trustllm: Trustworthiness in large language models, ICML 2024

All four reviewers provided positive recommendations for this work and recognized that the problem of noisy preference is significant, the proposed method is novel, the experiments are comprehensive. The reviewers also pointed out that it can be better if the authors can discuss the limitation of requiring high-quality clean validation dataset, and report the standard deviation in the final version. As a result, I recommend this paper for acceptance.