Risk-aware Direct Preference Optimization under Nested Risk Measure

We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.

Response to Other Comments Or Suggestions: The running …… "Theorem 3.6 Restated"?

We apologize for the confusion caused by our oversight. We will correct these errors in the next version.

Response to Question 1: Your reported …… objective function?

It should be clarified that our proposed $Ra-DPO$ method maintains both:

1. A natural and simple loss function (the sum of DPO loss and negative sequence risk ratio), which can be seen in Figure 1;
2. Comparable training efficiency, with no substantial increase in training time requirements.

Note: Firstly, in the paper, the lemmas and theorems presented serve exclusively to demonstrate the theoretical validity of our approach when accounting for risk considerations. Specifically, we introduce nested risk measures (a nonlinear function) in token-level policy optimization and then prove that maximizing the objective function will result in policy improvements. This method is more likely to effectively balance alignment performance and model drift, thereby preventing model failure in certain aspects and ultimately achieving an improvement in reward accuracy.

Additionally, as shown in Figures 6, we can observe that our method achieves higher reward accuracy when implemented with Pythia-1.4B as the base model on the Anthropic HH dataset. And, we provide several additional experimental results, including a numerical example, several evaluation results of LLMs, the results using nested ERM (entropic risk measure) [1-3] and the results with different seeds, to demonstrate the effectiveness of our algorithm. For details, please refer to the link 🔗Additional_Experiment_Results.

Response to Question 2: Would your …… those settings?

Thank you for the valuable question. We would like to provide an extensive discussion about risk-awareness and corresponding experiments.

Risk-Awareness: In this paper, we introduce nested risk measures to enhance risk-awareness in LLM alignment, which induces a conservative policy during the process of aligning LLMs, which enables the model to remain closely aligned with a reference LLM, thereby preventing significant deviations and maintaining its superior decision-making and reasoning abilities. This is highly valuable in real-world applications, where aligning general-purpose LLMs with human values and intentions (higher reward) without compromising its decision-making and reasoning abilities (lower KL divergence).

Experiments: Additionally, from the perspective of output verification (e.g., harmful content moderation, toxicity detection), we recommend a hybrid approach that combines Safe RLHF [4] with risk-sensitive measures. This approach independently models both cost and reward functions while accounting for cost distributions. However, it may require more computational resources due to the need to train additional models. Nonetheless, our method can serve as a foundational groundwork for such potential approaches. Moreover, we plan to conduct further research in this direction in the future.

It is noteworthy that our additional experiments also demonstrate examples of using LLMs (DeepSeek and GPO-4o) to evaluate the performance of various algorithms. The experimental results can be found at the link 🔗Additional_Experiment_Results.

References:

[1] Föllmer, Hans, and Alexander Schied. Convex measures of risk and trading constraints. Finance and stochastics, 2002, 6: 429-447.

[2] Hau, Jia Lin, Marek Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted MDPs. In AISTATS, 2023.

[3] Fei, Yingjie, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret. In NeurIPS, 2020.

[4] Dai, Josef, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In ICLR, 2024.

We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.

Response to Claims And Evidence:

Risk awareness: In this paper, risk awareness refers to the sensitivity to risks arising from deviations from the reference model. It induces a risk-averse policy to enable the model to remain closely aligned with a reference LLM and maintain its superior decision-making and reasoning abilities.

Experimental Benchmarks: From the perspective of output verification, we recommend a hybrid approach that combines Safe RLHF [1] with risk-sensitive measures, which independently models both cost and reward functions while accounting for cost distributions. However, it may require more computational resources due to the need to train additional models. Nonetheless, our method can serve as a foundational groundwork for such potential approaches.

Research significance: Notably, a critical consideration in alignment research involves balancing performance and model drift. Current approaches (e.g., DPO [2] and TDPO [3]) establish that optimal alignment should simultaneously maintain minimal deviation from the reference model (lower KL divergence) while aligning with human values (higher reward). F-DPO [4] studies this trade-off under varying divergence constraints.

Additional Experiment Results: We provide additional experimental results to further demonstrate the effectiveness of our approach at the link 🔗Additional_Experiment_Results.

Response to Methods And Evaluation Criteria:

The fine-tuning of LLMs typically involves two key stages: supervised fine-tuning (SFT) and preference alignment. During the alignment phase, the post-SFT model typically serves as the reference model (the most reliable model available), which generally demonstrates robust reasoning and decision-making capacities. Importantly, significant deviation from the reference model generally leads to capability degradation, which inherently constitutes substantial risk. Thus, the primary challenge in LLM alignment lies in balancing alignment performance and model drift (maintaining reasoning and decision-making abilities).

Response to Theoretical Claims:

Theoretical Breakthroughs. We introduce the nested risk measures to enhance the model's risk sensitivity. Our theoretical advancements primarily include:

1. Incorporating nested risk measures into token-level policy optimization and providing a closed-form solution. Importantly, our method maintains a natural and simple loss function shown in Figure 1.
2. Establishing the connection between risk-aware value functions and optimal policies. The key technical contributions lie in:
   • A risk-aware advantage function design under nested risk measures;
   • Proof of Bellman-type model equivalence with the Regret Preference Model under nested risk measures.

State Augmentation. We argue that state augmentation is essential, particularly for token-level generation in LLM alignment. The inherent contextual dependencies of text tasks and the non-Markovian characteristics of risk sensitivity naturally necessitate this approach. Compared to DPO-style methods, it introduces negligible additional complexity and computational overhead.

Response to Experimental Designs Or Analyses:

To ensure clarity, we provide additional clarification as follows: In Figures 2 and 3, each algorithm is represented by two curves: the darker-colored one shows the raw curve, while the brighter-colored one displays the smoothed version. All curves share one identical random seed, primarily because (1) LLM training requires substantial computational resources, and (2) we followed conventional practice by initializing the trained model with parameters from the reference model (the post-SFT model). Also, we add experimental results under multiple random seeds to the "The results with different seeds" folder in link 🔗Additional_Experiment_Results.

Response to Questions For Authors:

We apologize for the confusion caused by our oversight. The operator "o" denotes the concatenation of the state and action at time step $t+1$. In the set of equations (5), the second input to $V$ should be $y^{<t+1}$. We appreciate this correction to the writing error. $\pi(\cdot | [x, y^{<t}])$ represents the policy when taking an action given the state $[x, y^{<t}]$. In the next version, we will correct these errors.

References:

[1] Safe RLHF: Safe reinforcement learning from human feedback. ICLR, 2024.

[2] Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.

[3] Token-level direct preference optimization. ICML, 2024.

[4] Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. ICLR, 2024.

We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.

Response to Methods And Evaluation Criteria: Yes, the …… different datasets.

In Appendix Figures 6 and 7, we present experimental results conducted on the Anthropic HH dataset using Pythia-1.4B and Pythia-2.8B as base models. We implemented $TDPO_2$, and $Ra-DPO_2$ with different risk control parameters.

In addition, we provide several additional experimental results, including a numerical example, several evaluation results of LLMs, the results using nested ERM (entropic risk measure) [1-3] and the results with different seeds, to demonstrate the effectiveness of our algorithm. For details, please refer to the link 🔗Additional_Experiment_Results.

Response to Essential References Not Discussed: The essential …… future work.

We sincerely appreciate the reviewers' recognition of our work. In Appendix A, we examine key factors that introduce risks in the alignment of LLMs, where we highlight the crucial factor that "there exist conflicts and contradictions among human preferences (or choices)", while implying that the 'transitive' relationship between preference annotations may not always hold. Of course, we acknowledge that we inadvertently omitted citation to the important reference https://arxiv.org/abs/2409.19325. In the new version, we will include the relevant citations.

Response to Question 1: In Line …… Line 293.

In Line 378, Section 4.3, the sampling temperature coefficient is a parameter of AlpacaEval [4], which is a tool to evaluate instruction-following language models based on human annotations. In this paper, we adhered to the default settings in the official AlpacaEval implementation. It should be explicitly noted that this parameter bears no direct relationship to the loss function of our proposed Ra-DPO algorithm.

Response to Question 2: Have you …… automatic metrics?

Thank you very much for your valuable suggestions. Using human evaluators to assess alignment quality beyond automatic metrics is indeed very convincing. However, as is commonly known, this approach incurs substantial labor costs. For this reason, numerous research efforts are now exploring methodologies that utilize LLM-based evaluators as substitutes for human evaluators. Many studies [4-7] have shown that LLM-based auto-evaluators have become a key component of the LLM development process due to their cost-effectiveness and scalability compared to human-based evaluation. In this paper, we use AlpacaEval, a fast and affordable benchmark for chat LLMs that uses LLMs to estimate response quality.

References:

[1] Föllmer, Hans, and Alexander Schied. Convex measures of risk and trading constraints. Finance and stochastics, 2002, 6: 429-447.

[2] Hau, Jia Lin, Marek Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted MDPs. In AISTATS, 2023.

[3] Fei, Yingjie, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret. In NeurIPS, 2020.

[4] Dubois, Yann, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.

[5] Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2023.

[6] Li, Tianle, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024.

[7] Lin, Bill Yuchen, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770, 2024.

We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.

Response to Weaknesses 1:

We apologize for the confusion caused by failing to give a clear explanation and would like to re-clarify our motivation.

Before restating our motivation, we first clarify the following facts:

1. The reference model, typically a post-supervised fine-tuned model, demonstrates robust decision-making and reasoning capabilities. Current approaches, including DPO [1], KTO [2], and TDPO [3], establish that optimal alignment should simultaneously maintain minimal deviation from the reference model (lower KL divergence) while aligning with human values (higher reward).
2. Experiments in TDPO [3] have demonstrated the advantages of examining divergence against a reference LLM on a more granular, token-by-token basis.
3. The TDPO [3] focuses only on the expected reward (a risk-neutral criterion), thereby neglecting the characteristics of the reward distribution beyond the mean.

Based on the aforementioned facts and corresponding experimental results, a critical conclusion emerges: significant deviation from the reference model typically indicates a heightened risk of degradation in decision-making and reasoning capabilities.

Motivated by the above facts and conclusion, we introduce nested risk measures to enhance risk-awareness in LLM alignment. Here, “risk” specifically denotes potential hazards arising from deviations relative to the reference model. This is highly valuable in real-world applications, where aligning general-purpose LLMs with human values and intentions (higher reward) without compromising its decision-making and reasoning abilities (lower KL divergence). We hypothesize that risk-aware models employing nested risk measures (e.g., CVaR and ERM) will systematically reduce the probability of policy options with potential catastrophic consequences (failure, harmful, or deceptive) during policy optimization.

Response to Weaknesses 2:

We provide several additional experimental results, including a numerical example, several evaluation results of LLMs, the results using nested ERM (entropic risk measure) [4-6] and the results with different seeds, to demonstrate the effectiveness of our algorithm. For details, please refer to the link 🔗Additional_Experiment_Results.

Response to Weaknesses 3:

We provide discussion about why risk-aware preference optimization can lead to higher reward accuracy as follows:

As shown in Figures 6 and 7, we can observe that risk-aware preference optimization (Ra-DPO) achieves higher reward accuracy when implemented with Pythia-1.4B as the base model on the Anthropic HH dataset. This effect disappears when using Pythia-2.8B, which we attribute to the greater potential for reward accuracy improvement in smaller models. This is evident from several failed experiments we conducted: smaller models (Pythia-14m, Pythia-70m, and Pythia-160m) are more prone to model drift after thousands of iterations, resulting in empty outputs or invalid responses (extremely brief answers that fail to address the question). The proposed risk-sensitive method, incorporating risk-awareness into the token-level objective function, addresses this through risk-averse policy optimization. This method is more likely to effectively balance alignment performance and model drift, thereby preventing model failure in certain aspects and achieving an improvement in reward accuracy.

Response to Weaknesses 4:

Thank you for the valuable suggestions. We add the experimental results using nested ERM to demonstrate the effectiveness of our algorithm, which is conducted on Anthropic HH dataset with Pythia-1.4B serving as the base model. The experimental results can be found at the link 🔗Additional_Experiment_Results. Experimental results show that our $Ra−DPO_2$ algorithm (with Nested-ERM) also achieves consistently lower KL divergence and higher reward accuracy compared to baseline methods.

References:

[1] Rafailov, Rafael, Archit Sharma, Eric Mitchell, et al. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023.

[2] Ethayarajh, Kawin, Winnie Xu, et al. Model alignment as prospect theoretic optimization. In ICML, 2024.

[3] Zeng, Yongcheng, Guoqing Liu, et al. Token-level direct preference optimization. In ICML, 2024.

[4] Föllmer, Hans, and Alexander Schied. Convex measures of risk and trading constraints. Finance and stochastics, 2002, 6: 429-447.

[5] Hau, Jia Lin, Marek Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted MDPs. In AISTATS, 2023.

[6] Fei, Yingjie, Zhuoran Yang, et al. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret. In NeurIPS, 2020.