Understanding Generalization of Preference Optimization Under Noisy Feedback

The authors studied the impact of noise in feedback on the alignment of large language models (LLMs) through preference optimization. In particular, they considered a fairly general alignment loss which includes DPO, IPO, and SLiC. Moreover, they assumed that the "feature backbone" $g(x)$ of the model (the term is not clearly defined in the paper but it seems that it refers to the last layer of the model) is fixed (which I think it means that weights of the model are frozen) and only the "unembbeding matrix" $W$ is learnable. They modeled the output of the model by softmax$(Wg(x))$ and analyzed the dynamic of gradient flow over $W$ under some assumptions on the distribution of the input (the von Mises-Fisher (vMF) distribution). The authors defined a notion of risk for preference learning (Definition 3.1) and they provided upper bounds on the expected population risk (Theorem 3.2) through the gradient flow analysis (under some assumption on the significance of noise (which is modeled by a Bernoulli RV with parameter $\epsilon$) in the feedback and the run time $t$). Based on their analysis, they observed that the expected risk is bounded by $1/(1-c\epsilon)^2$ when $\epsilon$ is sufficiently below $1/2$ and then decreases linearly with $\epsilon$ when $\epsilon$ gets close to $1/2$. They verified their analysis on some synthetic data and also a real setting.

This paper studies the problem of preference optimization about the impact of noisy feedback. The key theoretical observation is that a formula of the risk upper bound that increases with extent of noise. When the noise rate approaches 1/2, the expected risk transitions grows at a linear rate. The theory also reveals that stronger concentration, more samples, and contrasting directions for positive and negative samples yields tighter bounds and slower degradation in accuracy as the noise rate increases. Empirical evaluations are done to support the theoretical findings.

The paper studies preference optimization with noisy preference feedback. Theoretically, the paper analyzes a fairly broad family of algorithms under softmax-linear parametrization of the language model policy. Experimentally, the paper mostly focuses on DPO loss both in controlled settings (but also considers some other losses) and on standard RLHF benchmarks. The main technical tool is a notion of reward margin, essentially the implicit reward difference between prefered and dis-preferred responses and the main concept insight is that stronger "concentration" (less variance of the data), more samples, and increasing the "angle" between positive and negative directions can improve performance.

Comments:

1. I have a basic issue with the premise of this paper. The original derivation of DPO (and these related losses) already incorporates noise in the preference data. The data is assumed to be generated by the Bradley-Terry model, in which preference pairs appear via P(y_1>y_2) \propto \sigma(r(y_1) - r(y_2)). This is already noisy! And the BT model can nearly model the "massart" noise model considered here, all "true positives" have reward r_1 and all true negatives have reward r_2 such that \sigma(r_1 - r_2) = 1-eps. So why not just reframe the paper this way?

2. The setup for the analysis is highly specialized and I feel this is rather limiting. Why is the vMF distribution natural or reasonable? Why not use a more general noise model, like the original BT? Why this specific policy parametrization? (I am ok with using softmax linear, but I think this should be at the token level over the responses, which I don't think is what is happening in the paper)

3. I am not particularly convinced by the controlled experiments. The settings nearly perfectly matches the one in the theoretical analysis. Thus the only conclusion one can make is, roughly, the theorem is correct. But I do not know if the controlled experiments have much bearing on practice.

4. The real-world experiment is more interesting, but I am not sure if we should view this as validation of the theory, because we do not see the 1/(1-x)^2 type behavior very clearly. Instead we see linear behavior, which could be presumably explained by many other models. As just a concrete example, in noisy classification (PAC learning with classification noise) we expect that the error rate of ERM scales linearly with the noise level, i.e., we expect err(ERM) \leq OPT + poly(1/n) and OPT will incur error rate equal to the noise level. So this is an equally valid, and much more general/simple/convincing, explanation for the experimental results? I get the stated reason for why we do not see the 1/(1-x)^2 behavior, but the point remains that no other competing hypothesese have not be convincingly ruled out.

5. I looked at the prior work Im-Li-2024a and I noticed that nontrivial portions of the text seem to be copied verbatim. I am not sure if this is problematic or not...

Overall: It's quite possible I am missing something and I'm open to having my mind changed about this. But, I feel the paper should just work in the standard BT model, should consider a more general setup for the analysis, and needs to have a more convincing experimental section. As it stands, I feel the paper is below the bar for ICLR.

This paper explores the generalization of preference optimization in large language models under noisy human feedback, providing theoretical guarantees and empirical validation in both synthetic and real-world settings.

This paper addresses the impact of noisy feedback on preference optimization for aligning large language models (LLMs) with human preferences. While most existing work assumes noise-free feedback, this paper provides generalization guarantees under noisy conditions. The authors establish guarantees for noisy preference learning across various optimization losses (e.g., DPO, IPO, SLiC) and present a general model describing how generalization decays with the noise rate. Empirical results on contemporary LLMs validate the practical relevance of these findings. The reviewers raised many critical questions, but the authors did not provide any responses. As a result, I recommend rejection.

Reject