A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO

This submission studies the interplay between privacy and robustness in both RLHF and DPO, two main alignment methods for language models. It shows that, when considering a linear reward model for RLHF and a log-linear policy class for DPO, the problem of offline alignment reduces to parameter estimation for logistic regression under private and corrupted labels. Using this reduction, one can derive suboptimality upperbound for the policy learned for both RLHF and DPO, based on parameter estimation error for logistic regression under private and corrupted labels.

Summary after rebuttal: I asked a question from the authors after their rebuttal about their considered corruption model in the LTC case. However, the authors did not reply to this question. Yet, I am going to keep my score for the other results in the paper.

The authors provide a theoretical framework to analyze the suboptimality gap of the learned policy in offline alignment, in the presence of privatized and corrupted labels. Specifically, they reduce two main paradigms, the Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), to the parameter estimation problem of logistic regression under some linear model assumptions. The suboptimality gap is then upper bounded by the estimation error of the logistic regression parameters. They demonstrate that, with a “shifting and scaling” loss function, the estimation error can be upper-bounded when labels are privatized (i.e., random response mechanism) and corrupted (i.e., an adversary modifies partial labels arbitrarily), quantifying the robustness of the alignment training to noisy labels.

The paper considers alignment problems such as RLHF or DPO, in a robust private setting. In this setting, we are given a preference dataset where each example contains an input text $s$, two actions $a_0, a_1$, and a label $y \in \{0, 1\}$ denoting which action is preferred in the example. Commonly, we assume the label $i$ is sampled with probability proportional to $exp(r(s, a_i))$, where $r$ is a reward function on the text $s$ and action $a$ In RLHF we train an intermediate reward model on the dataset which is then used to optimize the policy, in DPO the policy is directly optimized on the dataset. In the robust, private setting there are two sources of error on the labels (1) corruption, where some fraction $\alpha$ of the labels are corrupted by an adversary after inspecting the dataset, and (2) local DP, where we apply randomized response to change the label of each example (with probability $1 / (1 + e^\epsilon)$ for some $\epsilon$) to provide privacy to user. One can consider the LTC or CTL settings, where local DP is applied and then corruption occurs, or vice-versa. The authors give an analysis for both RLHF and DPO, in the LTC, CTL, and "CLC" setting (where corruption occurs both before and after local DP).

For RLHF the analysis assumes that the reward function is the dot product of some ground (bounded norm) ground truth $\theta^*$ and some known function $\phi(s, a)$, which was done by most of the prior work. They consider two algorithms: One that chooses a policy maximizing the expectation over the policy of this dot product for a current parameter estimate of $\theta^*$, and one that choosing a policy that maximizes the minimum this expectation can increase relative to a reference policy even if the parameter estimate is perturbed by a bounded amount. Under the linear model assumption, the authors show the labels correspond to a logistic regression model and bound the suboptimality of their algorithms. For DPO, they assume the optimal policy falls a log-linear model and prove similarly that the labels in DPO follow a logistic regression model and bound the suboptimality of a policy that is a function of an estimate of the ground truth in the logistic regression model.

Using these results, the authors design algorithms for getting the parameter estimate of $\theta^*$, by shifting and scaling the received labels based on the privacy of randomized response. They give bounds for this parameter estimate's error, with and without corruption, and with and without a uniform coverage assumption (the covariance of the features has lower bounded eigenvalues). Combining the parameter estimation bounds with the suboptimality bounds for the algorithm using the parameter estimate, they derive suboptimality bounds for RLHF and DPO in the LTC and CTL settings, with or without uniform converage. The bounds consist of (1) a bias term depending on the corruption parameter $\alpha$, which is larger for LTC than for CTL, demonstrating that LTC is fundamentally harder) and (2) terms decaying to 0 as $1/\sqrt{n}$ in the dataset size $n$, matching the noiseless rate.

update after rebuttal

I remain in support of accepting the paper

The paper develops a unified theoretical framework analyzing the impact of label corruption and privacy on two primary offline alignment methods: RLHF and DPO. The authors focus on the interplay between LDP and adversarial label corruption, formalizing three noise models: CTL (corruption then LDP), LTC (LDP then corruption), and CLC (corruption before and after LDP). A key conceptual contribution is a reduction from the offline alignment problem to logistic regression under these noise models. Major findings include:

• LTC is provably more challenging than CTL, even under linear models.
• The analysis yields new state-of-the-art theoretical guarantees for offline alignment under privacy-only or corruption-only regimes.
• Novel estimation bounds are derived for logistic regression under the joint noise setting, leading to suboptimality bounds for RLHF and DPO algorithms.
• The authors also provide practical algorithmic implications, including a new estimator and empirical verification of the separation between LTC and CTL on GPT2.

This paper presents a unified and rigorous theory to understand the impact of label corruption and local differential privacy on offline alignment methods including RLHF and DPO. All reviewers recommend acceptance due to the paper’s novel reduction to logistic regression under joint noise, insightful separation between CTL and LTC settings, and new bounds that advance the literature. One limitation is that the current result heavily relies on the linear-model assumption. Overall, the theoretical contribution still makes it a strong submission that deserves acceptance.