Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment

Claims And Evidence (1): The study explores only a narrow set of tasks, primarily math and reasoning, with minimal analysis. Key questions remain: How does the algorithm perform on a broader range of tasks?

A: Thanks for your question. We are conducting experiments on the new tasks.

Claims And Evidence (2): How much corruption exists in the preference data, and to what extent can the proposed approach mitigate it?

A: Thanks for your questions. It is hard to count the number of corrupted labels in the large datasets, but we are sure there are corruptions in the AI-generated labels, so we use the performance gap (by LC win-rates and accuracies) between our algorithm and the 3 algorithms that do not directly tackle corruption (i.e., pessimistic/optimistic DPO algorithm, length-regularized DPO algorithm and vanilla DPO algorithm) to roughly reflect the extent of corruption mitigation.

However, we are working on the new experiments by corrupting x% of the labels in the preference data with various x.

Claims And Evidence (3): To what degree is reward overoptimization a concern, and why is tuning the beta parameter in vanilla DPO insufficient to address it?

A: Thanks for your questions. Note that the robust DPO algorithm (corruption-only), length-regularized DPO algorithm (verbosity-only) and vanilla DPO algorithm in our experiments do not directly deal with over-optimization. Tables 1 and 3 show that the LC-win rates of these 3 algorithms are 0.57%-1.68% lower than those of our DPO-COV algorithms. Table 2 shows that these 3 algorithms are 0.38%-3.16% less accurate than our DPO-COV algorithm. These performance gaps show the effect of overoptimization.

Tuning beta alone can still cause overoptimization, because the KL-regularization could only control the scale of the gradient per training example, while adding the pessimistic term can further modify the gradient direction [1].

[1] Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., ... & Wang, Z. Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Questions For Authors: How much more effective is the proposed pessimistic MLE compared to the regular KL-penalized objective in reducing overoptimization? What baseline policy is used in the experiments?

A: Thanks for your questions. The pessimistic MLE is proposed not by us but by (Liu et al., 2024c; Cen et al., 2024; Ji et al., 2024; Yang et al., 2024), which have empirically demonstrated that the pessimistic MLE significantly outperforms the regular KL-penalized objective (i.e., vanilla DPO) in reducing overoptimization. For example, in Table 1 of (Liu et al., 2024c), the win rate of pessimistic MLE VS vanilla DPO is 56%. As to the baseline policy $\pi_{\rm base}$, Line 307 left side said "Algorithm 1 takes $\pi_{\rm base}(\cdot|x)$ as the distribution of the preferable responses $a_i^w$ given $x_i=x$, which is well covered by $\mathcal{D}$."

Questions For Authors (1): Please discuss and compare your results with [1] for robustness.

[1] Chowdhury, Sayak Ray, Anush Kini, and Nagarajan Natarajan. "Provably robust DPO: Aligning language models with noisy feedback." ArXiv:2403.00409 (2024).

A: Thanks for bringing this important work to our attention. [1] proposes a robust DPO method that solves data corruption well by incorporating a certain fixed label flipping probability $\epsilon\in(0,1)$ into the likelihood function, but it does not explicitly solve over-optimization and verbosity, while our work solves all these three issues simultaneoulsly. As to theoretical results, both works achieve generalization error rates of $\mathcal{O}(1/\sqrt{n})$ with $n$ possibly corrupted samples, but [1] uses expected true reward as the generalization error measure, while our generalization error measure (Eq. (20)) strikes a trade-off among expected true reward, distance to the reference policy (to solve over-optimization) and length penalty (to solve verbosity). The experimental results are not directly comparable as they involve different tasks and datasets. We will cite [1] in our revised work.

Questions For Authors (2): What is the motivation to tackle the corruption, overoptimization, and verbosity simultaneously in the real world?

A: Good question. All these three issues exist and can yield undesirable consequences in the real world as will be elaborated below. That motivates us to tackle these issues simultaneously.

The "Corruption" paragraph in the introduction reveals the reason and dangers of corruption in the real world: "However, preference labels given by human may be corrupted due to inexperience, inattention, personal bias, unclear context, and even malicious falsification (Bukharin et al., 2024). For instance, when fine-tuning LLM for automated content moderation on social media, malicious annotators may mislabel harmful contents like misinformation and hate speech as preferable, which misleads the LLM to generate such harmful contents." Hence, we need to tackle corruption.

The "Overoptimization" paragraph in the introduction reveals the reason and undesirable consequence of overoptimization: "RLHF and DPO may overoptimize the reward model, yielding LLM responses of high estimated reward but low actual quality (Gao et al., 2023; Casper et al., 2023)." This contradicts the goal of RLHF and DPO to make LLM responses helpful, honest, and harmless. Hence, we need to tackle overoptimization.

The "Verbosity" paragraph in the introduction said "LLM aligned by vanilla RLHF and DPO is likely to prefer verbose but possibly low-quality responses". In the real world, verbose responses can waste users' time and thus lose users. This consequence will be added to the revision. Hence, we need to tackle verbosity.

Questions For Authors (3): What is the reason for designing $y_i\xi_i$ instead of $\xi_i$ in loss function in Lines 158-160?

A: Good question. As shown in the footnote in the page 3, using $y_i\xi_i$ ensures that $\mathbb{P}(y_i|a_i^{(1)},a_i^{(-1)})=\sigma[r^*(x_i,a_i^{w})-r^*(x_i,a_i^{\ell})+y_i\xi_i^*], y_i\in\{-1,1\}$ is a valid probability measure satisfying $\sum_{y_i\in\{-1,1\}}\mathbb{P}(y_i|a_i^{(1)},a_i^{(-1)})=1$. To elaborate, since $y_i=1$ means $a_i^{w}=a_i^{(1)}$ and $a_i^{\ell}=a_i^{(-1)}$, we have $\mathbb{P}(y_i=1|a_i^{(1)},a_i^{(-1)})=\sigma[r^*(x_i,a_i^{(1)})-r^*(x_i,a_i^{(-1)})+\xi_i^*]$. Similarly, since $y_i=-1$ means $a_i^{w}=a_i^{(-1)}$ and $a_i^{\ell}=a_i^{(1)}$, we have $\mathbb{P}(y_i=1|a_i^{(1)},a_i^{(-1)})=\sigma[r^*(x_i,a_i^{(-1)})-r^*(x_i,a_i^{(1)})-\xi_i^*]=1-\sigma[r^*(x_i,a_i^{(1)})-r^*(x_i,a_i^{(-1)})+\xi_i^*]$, so $\sum_{y_i\in\{-1,1\}}\mathbb{P}(y_i|a_i^{(1)},a_i^{(-1)})=1$.

Questions For Authors (4): The assumption 3 is all-policy coverage. Can your method improve to single-policy coverage?

A: Assumption 3 only involve two fixed policies, the baseline policy $\pi_{\rm base}$ and the true optimal policy $\pi_{r^*}$. Why does the reviewer think it is all-policy coverage? Actually our Assumption 3 is the same as Assumption 2 of [1], which is claimed by [1] as single-policy coverage.

[1] Ji, X., Kulkarni, S., Wang, M., and Xie, T. (2024). Self-play with adversarial critic: Provable and scalable offline alignment for language models. ArXiv:2406.04274.

Claims And Evidence: The only claim I find a bit problematic is the verbosity reducing one. The main issue is that verbosity reduction is addressed through a universal length penalty which would not take into account prompt-specific length requirements. The results don't include numbers on average response lengths, so it is unclear how well the length regularizer works. Another concern is that reducing length might affect reasoning performance because of shortened reasoning chains, but the evals on reasoning and math benchmarks don't sustain this.

A: Prompt-specific length is a good future direction, which may be tackled by using length penalty coefficient $\omega(x)$ that depends on the prompt $x$. The larger length required for $x$, the smaller $\omega(x)$ should be.

We are working on experiments to include the numbers on average response lengths. Thanks for your suggestion.

For reasoning, we could use small length penalty coefficient to guarantee intact necessary reasoning steps, while trimming unnecessary steps and tokens. In the future, we may apply different levels of length penalty to different reasoning steps, to ensure a concise description of each step, rather than to reduce the number of steps.

Generalizability: It is hard to judge the performance of DPO-COV with just win rates against GPT-4 on AlpacaEval. It would be nice to see a few more benchmarks (ArenaHard and MT-Bench) or different base models (llama3-8b).

A: Thanks for your suggestion. We are working on the new experiments.

Effect of Corruption levels in data: It would be helpful to see how different levels of corruption in the preference data affects alignment. The authors speak about malicious actors deliberately corrupting preference data as a motivation, so it would be helpful to see how robust this method is to an issue like that.

A: Thanks for your suggestion. We are working on the new experiments by corrupting x% of the labels in the preference data with various x.

Overoptimizations: An analysis into the patterns of reward hacking that were observed in the vanilla DPO run vs. DPO-COV run would also help illustrate how all these techniques work together to make a better model.

A: Thanks for your suggestion. We are working on the new experiments.

Experimental Designs Or Analyses (1): To show the LC-win rate, Argilla-DPO-Mix-7K is used, while performance is compared using reasoning tasks such as GSM8K and ARC. Why are different tasks required?

A: Argilla-DPO-Mix-7K is the preference dataset we use for training, while GSM8K and ARC are benchmarks used for evaluation.

Experimental Designs Or Analyses (2): In addition, in the experiments section, readers may expect that DPO-COV is empirically robust against corrupted preferences and reward overoptimization. However, why does the performance on reasoning tasks reveal such desirable properties of DPO-COV?

A: Math and reasoning are two important categories of tasks in LLM evaluation [1], the superior performance of DPO-COV over these benchmarks compared to vanilla DPO exactly show that our method is robust against corrupted preferences and reward overoptimization.

[1]. Gao, L. et. al. (2024a). A framework for few-shot language model evaluation. https://zenodo.org/records/12608602.

Experimental Designs Or Analyses (3): Among the proposed methods, only the performance of offline DPO-COV is reported.

A: The performance of online methods is reported in Appendix A due to limited space of the main text.

Other Comments Or Suggestions:

The equations have poor readability. For example, in Equation (12), (18), (23), and (24), what is the role of $\{$ and $\}$? In addition, the line breaks in the equations are very hard to follow.

A: Thanks for your suggestions. We will simplify these equations respectively as follows. $\min _ {r\in\mathcal{R},\xi\in\mathbb{R}^N} [\max _ {\pi\in\Pi} \mathcal{L} _ {N,\lambda}(r,\xi)+\eta V _ {\beta,\omega}(\pi,r)],$

$\min _ {\pi\in\Pi _ {\mathcal{R}}} [\mathcal{L} _ {N,\lambda}(r^{\pi},\xi^{\pi})+\eta V _ {\beta,\omega}(\pi _ {r^{\pi}},r^{\pi})],$

$\pi_{t+1}\in{\arg\min} _ {\pi\in\Pi}\min _ {r\in\mathcal{R},\xi^{(t)}\in\mathbb{R}^t}[\mathcal{L} _ {t,\lambda}(r,\xi^{(t)})-\eta V _ {\beta,\omega}(\pi,r)],$

$\pi _ {t+1} \in {\arg\min} _ {\pi\in\Pi _ {\mathcal{R}}} [\mathcal{L} _ {t,\lambda}(r^{\pi},\xi^{\pi,(t)})-\eta V _ {\beta,\omega}(\pi_{r^{\pi}},r^{\pi})].$

The { } in the original equations unnecessarily repeat the expressions of the functions $\mathcal{L} _ {N,\lambda}$ (similarly $\mathcal{L} _ {t,\lambda}$) and $V _ {\beta,\omega}$ above that have already been defined in Eqs. (8) and (10) respectively, so we removed such repetitions.

Question about Analysis: I do not see any assumptions about noise; therefore, I may assume very large noise. In this case, preferences become random, making it impossible to guarantee the performance of any RLHF algorithm, including DPO-COV, since the given data contains no information about preferences. However, Theorem 1 guarantees the performance gap, which suggests that DPO-COV can obtain a reasonable solution. Could you clarify this confusion?

A: The convergence rate in Theorem 1 is $\mathcal{O}(\sqrt{||\xi^*|| _ 1/N})$ where $||\xi^*|| _ 1=\sum _ {i=1}^N|\xi_i^*|$ is the norm of the true noise. Right after Theorem 1, we implicitly assume the upper bound on the true noise by saying "Hence, as long as $||\xi^*|| _ 1\le\mathcal{O}[\log(N)]$ (much weaker than Assumption 4.2 of (Bukharin et al., 2024) that there exist constants $c_0,c _ {\infty}>0$ such that $\xi^*$ has at most $c_0$ nonzero entries and they range in $[-c_{\infty},c_{\infty}]$), the generalization error rate (22) has the order of $\mathcal{O}[\log(N)/\sqrt{N}]$."

Question about objective (20): The authors suggest that objective (20) is a desirable objective. However, I am not sure that (20) is truly a desirable goal, as there are many ways to address the verbosity issue that are not aligned with objective (20) (For example, SimPO or GRPO, etc.)

A: I agree that there are many methods to address the verbosity issue, and also there can be many generalization measures (including our Eq. (20)) that accounts for verbosity. All these methods and measures align in the final goal to strike a trade-off between expected reward and length. Therefore, as long as the length penalty coefficient controlling such trade-off is proper for a specific task, Eq. (20) is a desirable goal.