We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.

W1: I fundamentally …… in memory).

Response: Thank you for the valuable suggestions. Below, we re-clarify the rationale behind defining "risk" as divergence from the reference policy and provide an explanation of the SimPO [1] compared to the DPO.

Regarding the define of "risk":

Indeed, as the reviewer pointed out, the core point of preference optimization is to make the model less harmful, more helpful, and truthful. However, drawing upon existing literature and our own insights, we would like to emphasize:

• A trade-off between alignment objectives and model fidelity is still necessary, although the original(reference) model may not be "safe" or "correct". For example, in LLMs safety alignment tasks, a simple objective is to enable the model to reject unsafe responses while preserving original reasoning capabilities [2-3]. A response that is safe but logically incoherent or semantically uninformative is of little practical value. Therefore, many studies [4-5] typically formulate such tasks as constrained reward maximization problems.
• KL divergence has typically been used to penalize excessive deviations from a reference (critic) model [6-7]. In fact, numerous studies have reported that KL constraint offers many beneficial effects, such as balancing exploration and exploitation, ensuring stability and robustness, preventing catastrophic forgetting, and preserving the model’s fundamental capabilities [8-9].

Regarding SimPO:

SimPO [1] is a reference-free preference optimization algorithm that not only achieves superior performance, but also significantly reduces memory consumption. However, there are also studies indicating that SimPO has the following drawbacks:

• As pointed out in [10], the lack of a reference model in SimPO reduces training robustness and necessitates stricter conditions to prevent catastrophic forgetting. The experimental results in the paper (Table 1) show that SimPO does not consistently outperform DPO. Specifically, SimPO demonstrates a significant advantage over DPO when training the Qwen2.5-7B-Base model, while DPO slightly outperforms SimPO when training the Qwen2.5-7B-Instruct model.
• As noted in [11], different from DPO, SimPO reintroduces complexity through dual parameters $(\beta, \gamma)$, which introduce additional complexity on hyperparameter tuning.
• As pointed out in Subsection 3.8.2 of [12], the reward function in SimPO emphasizes length-normalized next token prediction, which raises concerns about a potential deviation from the original alignment objective.

From our perspective, DPO and SimPO serve as representative examples of reference-based and reference-free preference optimization methods, respectively. A comprehensive comparison in terms of performance, stability and robustness, hyperparameter tuning complexity, and computational efficiency reveals that each approach has its own trade-offs. Consequently, they are suited to different applications and both continue to represent valuable research directions.

W2: Regarding the evidence of our paper's motivation.

Response: We have already provided some supporting evidence in our response to W1. Below, we present additional direct evidence:

• As pointed out in [7], in stylistic text continuation tasks, optimizing reward without constraints leads to incoherent continuations, whereas introducing a KL constraint encourages the policy to remain close to a language model trained on BookCorpus.
• As noted in [13], TDPO outperforms DPO while maintaining a lower KL divergence. Furthermore, we implemented our Ra-DPO and reproduced both TDPO and DPO baselines. As shown in the sampling results from AlpacaEval (see Supplementary Material, directory: "LLMs_eval_results / Q&A_evaluation_examples"), DPO (with higher KL) tends to generate longer responses in many question-answering tasks, some of which contain several repetitive statements and lack logical coherence. In contrast, such issues are rarely observed with TDPO and Ra-DPO.
• Several of our unsuccessful experiments suggest that model instability or collapse is frequently associated with increased KL divergence. For instance, when setting $\alpha=0.1$, both TDPO2 and Ra-DPO show significantly elevated KL divergence, leading to poor performance on the AlpacaEval.

W3: Regarding the definition of risk.

Response: In the new version, we will appropriately incorporate the explanation of risks into the main text.

W4-5 & Q1-3: Regarding the hyperparameters chosen and search.

Response: Hyperparameters chosen: As mentioned in Section 4, we set the hyperparameters following the original KTO [14] implementation. Specifically, in the official KTO codebase, the default value of $\beta=0.1$ can be found in the file “HALOs/config/loss/dpo.yaml”, and the default value of $lr=5.0e^{-6}$ is specified in the file “HALOs/config/config.yaml”. This configuration is also widely adopted in other literature, including in implementations of DPO (Hugging Face version) [5] and TDPO [13].

Hyperparameter search: We have conducted extensive hyperparameter searches for various algorithms, including DPO, TDPO2, and Ra-DPO2, across different settings of $\beta$ and learning rate (lr). However, due to the NeurIPS 2025 policy prohibiting the inclusion of external links or PDF in the rebuttal, we summarize the key results in the tables below.

Table 1. Performance of various algorithms under different $\beta$ settings. (Anthropic HH, Pythia-1.4B, H100, $lr=5.0e^{-6}$)

Table 2. Performance of various algorithms under different learning rates. (Anthropic HH, Pythia-1.4B, A100, $\beta= 0.1$)

Note:

1. Sharpe ratio [15, 16], a new evaluation metric, is introduced. A higher Sharpe ratio indicates that a model attains a given performance gain with lower behavioral deviation.
2. The data in the tables represent the average over the final stable phase of training. It is worth noting that DPO's KL divergence and Sharpe ratio are often unstable.
3. The experiments in Table 1 were conducted on two H100 GPUs, each with 80GB of GPU memory, while the experiments in Table 2 were conducted on two A100 GPUs, each with 40GB of GPU memory.

Q4: Regarding the "risk evaluation" of model.

Response: Thank you for the valuable suggestions. Measuring the divergence between reference model and policy model is a common practice [3, 5, 9, 13]. Introducing an external "risk evaluation" model could be a feasible direction for exploration; however, several foreseeable challenges may arise. For instance, during the initial training phase, the distribution of the trained model may differ significantly from that of reference model, which could destabilize or even collapse the training process. Selecting an appropriate reference model would thus be a critical challenge. In our future work, we will conduct further experiments in this direction.

It is note that in our response to Reviewer mp5k, we provide detailed results on policy entropy, Sharpe ratio, and KL divergence across different models (including Pythia-1.4B and DeepSeek-R1-Distill-Qwen-1.5B), which may offer additional insights into this issue.

Reference:

1. Simpo: Simple preference optimization with a reference-free reward. NeurIPS, 2024.
2. Safe RLHF: Safe Reinforcement Learning from Human Feedback. ICLR, 2024.
3. Stepwise alignment for constrained language model policy optimization. NeurIPS, 2024.
4. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
5. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.
6. Trust region policy optimization. ICML, 2015.
7. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
8. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017, 114(13): 3521-3526.
9. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. ICLR, 2024.
10. Pre-dpo: Improving data utilization in direct preference optimization using a guiding reference model. arXiv preprint arXiv:2504.15843, 2025.
11. Repo: Relu-based preference optimization. arXiv preprint arXiv:2503.07426, 2025.
12. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216, 2024.
13. Token-level direct preference optimization. ICML, 2024.
14. Model alignment as prospect theoretic optimization. ICML, 2024.
15. The sharpe ratio. Streetwise–the Best of the Journal of Portfolio Management, 1998, 3(3): 169-85.
16. Adjusting for risk: An improved Sharpe ratio. International review of economics & finance, 2000, 9(3): 209-222.

We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.

W1: Even though …… (Token-level Direct Preference Optimization, ICML 2024).

Response: As the reviewer points out, our method is an extension of TDPO. However, it is important to emphasize that we introduce nested risk measures to effectively enhance the risk sensitivity of the policy optimization process. Our key theoretical contributions include:

• Incorporating nested risk measures into token-level policy optimization and providing a closed-form solution. Importantly, our method maintains a natural and simple loss function shown in Figure 1 (the sum of DPO loss and negative sequence risk ratio).
• Establishing the connection between risk-aware state-action value functions and optimal policies. The key technical contributions lie in:
• A risk-aware advantage function design under nested risk measures;
• Proof of Bellman-type model equivalence with the Regret Preference Model under nested risk measures.

Main experiments results include:

• In the paper, we evaluate the effectiveness of the proposed method across various text generation tasks and analyze its sensitivity to the risk control parameter. The results show that our method effectively suppresses the risk of undesirable behavior while improving overall performance.
• To further demonstrate the algorithm's properties, we provide additional analyses on policy entropy, Sharpe ratio [1–2], hyperparameter sensitivity, and other key aspects for each baseline. These can be found in Table 4 and in responses to the other reviewers. The results show that models trained with Ra-DPO2 achieve high reward accuracy, preserve policy entropy (indicating sustained exploration), attain a high Sharpe ratio (reflecting superior risk-adjusted performance), and maintain low KL divergence (ensuring proximity to the reference policy), highlighting the robustness and broad applicability of our approach.

W2 & Q1: Regarding the computational complexity during training time.

Response: Thank you for the valuable suggestions. Before analyzing computational complexity, it is worth noting that we reformulate the recursive Bellman equation into a classical Bellman equation by constructing an augmented-state MDP despite introducing nested risk measures. As a result, in terms of computational complexity, the main difference between our algorithm and TDPO lies in the use of risk measures such as CVaR or ERM, as opposed to the expectation. Below, we present their time and space complexity.

Table 1. Complexity of different risk measures.

For a more intuitive presentation, we present performance and runtime of various algorithms on A100 and H100 GPUs under different models and datasets in the following tables.

Table 2. Runtime of various algorithms. (IMDB, A100)

Table 3. Runtime of various algorithms. (Anthropic HH, A100)

Table 4. Performance and runtime of various algorithms. (Anthropic HH, Pythia-1.4B, H100, $\beta=0.1, lr=5.0e^{-6}$)

Experimental analysis:

• Several variants of Ra-DPO consistently achieve high reward accuracy and Sharpe ratios, while maintaining low KL divergence. Notably, these improvements are achieved without a significant increase in training runtime.

Note:

1. In Table 1, B denotes the batch size, T denotes the sequence length, and N denotes the number of samples per distribution.
2. The Sharpe ratio is a widely adopted metric for assessing risk-adjusted performance. In the context of language model alignment, we adopt it to assess the risk-adjusted reward. A higher Sharpe ratio indicates that a model attains a given performance gain with lower behavioral deviation—thus reflecting more stable and reliable training dynamics.
3. The runtime reported in Tables 2 and 3 was measured on two A100 GPUs, each with 40 GB of memory, while the runtime in Table 4 was measured on two H100 GPUs, each with 80 GB of memory.
4. Ra-DPO2 (CvaR-v2) is another version of Ra-DPO2 (CvaR), which effectively alleviates computational pressure by using top-k instead of quantile when calculating CVaR. Notably, our experiments show that the impact of this substitution on metrics such as reward accuracy and KL divergence is negligible.

W3: Even though Ra-DPO can…… and 53.5, respectively.

Response: We acknowledge that the win rate of our method on AlpacaEval is not significantly higher than that of the baselines. However, it is important to note that the core idea of our approach lies in incorporating risk-awareness by employing a class of nested risk measures, which achieve a better balance between alignment performance and model drift under risky conditions. The token-level sequential risk ratio constraint makes the model more conservative in exploration when facing high-risk situations, leading to generated responses that are shorter and more moderate in length. As a result, our method achieves better performance in terms of length-controlled win rate as show in Table 1.

Moreover, there are several experimental results are worth highlighting:

• Figure 13 in Appendix C.4 presents results on the Anthropic HH dataset using Pythia-1.4B as the base model. For all algorithms, we report the average performance across three random seeds, with the shaded area indicating one standard deviation around the mean. The figure shows that, under both the CVaR-based and the ERM-based risk measure with µ = 5, the proposed algorithm achieves reward accuracy comparable to that of the baseline methods, but with greater stability, while maintaining consistently low sequential KL divergence.
• In our responses to the other reviewers, we provided results on policy entropy, hyperparameter search, and other aspects for each algorithm. The results show that Ra-DPO (ERM) exhibits the most desirable risk-aware behavior among the evaluated methods, as evidenced by its high and stable Sharpe Ratio, as well as well-preserved policy entropy.
• In the Supplementary Material under the directory " Q&A evaluation examples using LLMs\3-Evaluation results of LLMs", we present a comparative evaluation of our algorithm and baseline methods using DeepSeek and GPT-4o, based on a selected set of questions. The evaluation dimensions include:
  • Riskiness: Whether the answer has potential risks or problems.
  • Effectiveness: Whether the answer can effectively solve the problem.
  • Relevance: Whether the answer closely revolves around the core of the question.
  • Redundancy: Whether the answer has unnecessary repetitions or redundant information.

The evaluation results show that our algorithm (Ra-DPO) performs well in terms of riskiness and relevance, but there are differences in effectiveness and redundancy.

The original conclusion is as follows:

• Some algorithms (such as Ra-DPO1 (CVaR, $\alpha=0.5, \mu=0.97$) and Ra-DPO2 (CVaR, $\alpha=0.5, \mu=0.97$) can provide high-quality practical information, while the effectiveness of other algorithms (such as Ra-DPO1 (CVaR, $\alpha=0.5, \mu=0.99$) ) and Ra-DPO2 (CVaR, $\alpha=0.5, \mu=0.99$) ) is relatively low and their redundancy is relatively high.
• Future research should focus on optimizing algorithms to improve their performance in terms of effectiveness and redundancy, thereby enhancing their overall performance.

Q2. Is there any trade-off in incorporating risk-awareness into DPO or TDPO?

Response: There exists a certain trade-off when incorporating risk awareness into TDPO. As shown in the responses to W2 and Q1, Ra-TDPO2 achieves higher performance and maintaining lower KL divergence, while requiring slightly more running time.

Q3. In Figure 3, why does Ra-DPO with ERM work worse than Ra-DPO with CVaR?

Response: This may be related to the inherent characteristics of the CVaR and ERM risk measures themselves. CVaR directly optimizes performance on the worst-case samples, whereas ERM applies exponential weighting uniformly across all samples, potentially diluting attention to extreme poor-performing cases. Additionally, ERM is more sensitive to hyperparameters, as its performance varies significantly with the choice of parameter. As shown in Figure 3, when the risk coefficient of Ra-DPO2 with ERM is set to $\mu=5$, it achieves performance comparable to that of Ra-DPO2 with CVaR.

Limitations: The authors provide …… can be in Appendix.

Response: In the new version, we will move the discussion on limitations from the appendix to the main text.

Reference:

1. Sharpe W F. The sharpe ratio. Streetwise–the Best of the Journal of Portfolio Management, 1998, 3(3): 169-85.
2. Dowd K. Adjusting for risk: An improved Sharpe ratio. International review of economics & finance, 2000, 9(3): 209-222.

We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.

Q1 & Q3. Regarding the model's entropy, practical metrics and model baselines.

Response: To more directly demonstrate the characteristics of our algorithm, we have added experiments with multiple metrics under different model baselines.

Model baselines: We conducted experiments with Pythia-1.4B and DeepSeek-R1-Distill-Qwen-1.5B [1] on the Anthropic HH dataset.

Practical metrics: We added policy entropy and Sharpe ratio [2-3] to evaluation our method. The Sharpe ratio is a widely adopted metric for assessing risk-adjusted performance. In the context of language model alignment, we adapt it to assess the risk-adjusted reward. A higher Sharpe Ratio indicates that a model attains a given performance gain with lower behavioral deviation—thus reflecting more stable and reliable training dynamics.

However, due to the NeurIPS 2025 policy prohibiting the inclusion of external links or PDF in the rebuttal, we summarize the key results in the table below.

Table 1. Performance and runtime of various algorithms. (Anthropic HH, Pythia-1.4B, H100, $\beta=0.1, lr=5.0e^{-6}$）

Experimental results in Table 1:

• The model trained with DPO exhibits low reward accuracy, high KL divergence, and a negative Sharpe ratio, indicating poor alignment performance and unstable policy optimization.
• Models trained using different Ra-DPO variants maintain a stable policy entropy of approximately 2.93 across the entire training process. In contrast, models trained with baseline methods show a consistent downward trend in entropy, persisting until the end of training.
• Several variants of Ra-DPO consistently achieve high reward accuracy and Sharpe ratios, while maintaining low KL divergence. Notably, these improvements are achieved without a significant increase in training runtime.

Table 2. Performance and runtime of various algorithms. (Anthropic HH, DeepSeek-R1-Distill-Qwen-1.5B, H100, $\beta=0.1, lr=5.0e^{-6}$)

Experimental results in Table 2:

• Experiments conducted using the DeepSeek-R1-Distill-Qwen-1.5B model yield patterns similar to those reported in Table 1, validating the consistency of our findings.
• A closer look at Table 2 reveals a key distinction: while DPO and TDPO2 exhibit marginally negative Sharpe ratios, indicating limited risk-adjusted performance, Ra-DPO continues to achieve a substantially higher Sharpe ratio. This underscores the stability and generalizability of Ra-DPO under varying model backbones.

Experimental analysis:

• A comprehensive analysis of Tables 1 and 2 reveals that Ra-DPO2 delivers consistently strong performance across two distinct model backbones: Pythia-1.4B and DeepSeek-R1-Distill-Qwen-1.5B. Specifically, models trained with Ra-DPO2 attain high reward accuracy, preserve policy entropy (indicating sustained exploration), achieve a high Sharpe ratio (reflecting superior risk-adjusted performance), and maintain low KL divergence (ensuring proximity to the reference policy). These results highlight the robustness and broad applicability of our approach.

Note:

1. The values of reward accuracy, KL divergence, and Sharpe ratio reported in the Table 1 and 2 represent the average over the final stable phase of training, after convergence. The policy entropy of DPO and TDPO2 does not converge, but instead continues to decrease gradually throughout training.
2. The experiments in Tables 1 and 2 were conducted on two H100 GPUs, each with 80GB of GPU memory.
3. In our responses to the other reviewers, we provide additional analyses on complexity analysis, Sharpe ratio, hyperparameter sensitivity, and other key aspects for each baseline.
4. Ra-DPO2 (CvaR-v2) is another version of Ra-DPO2 (CvaR), which effectively alleviates computational pressure by using top-k instead of quantile when calculating CVaR. Notably, our experiments show that the impact of this substitution on metrics such as reward accuracy and KL divergence is negligible.

Q2. How does the …… generation lengths.

Response: We appreciate the reviewer’s insightful comment on the complex interaction among answer length, reward shaping, gradient variance, and overall E2E performance. Indeed, disentangling these factors is non-trivial and remains an open challenge in RL-based language generation. Due to time constraints, a comprehensive analysis is currently infeasible. However, as shown in Tables 1 and 2 and our supplementary material, several key points can be observed:

• There exists a certain trade-off when incorporating risk awareness into TDPO. As shown in Tables 1 and 2, Ra-TDPO2 attain high reward accuracy, preserve policy entropy, achieve a high Sharpe ratio, and maintain low KL divergence, while requiring slightly more running time.
• While our method does not explicitly control for output length, we observe in the Supplementary Material that models trained with Ra-DPO2 tend to generate more concise and focused responses without sacrificing reward, suggesting a potential regularization effect.

We will include a discussion on this point in the final version.

Regarding Supplementary Material:

• In the directory: "LLMs_eval_results / Q&A_evaluation_examples" in Supplementary Material, the sampling results from AlpacaEval show DPO (with higher KL) tends to generate longer responses in many question-answering tasks, some of which contain several repetitive statements and lack logical coherence. In contrast, such issues are rarely observed with TDPO and Ra-DPO.

• In the directory " Q&A evaluation examples using LLMs\3-Evaluation results of LLMs" in the Supplementary Material, we present a comparative evaluation of our algorithm and baseline methods using DeepSeek and GPT-4o, based on a selected set of questions. The evaluation dimensions include:

  • Riskiness: Whether the answer has potential risks or problems.
  • Effectiveness: Whether the answer can effectively solve the problem.
  • Relevance: Whether the answer closely revolves around the core of the question.
  • Redundancy: Whether the answer has unnecessary repetitions or redundant information.

The evaluation results show that our algorithm (Ra-DPO) performs well in terms of riskiness and relevance, but there are differences in effectiveness and redundancy.

Reference:

1. Guo D, Yang D, Zhang H, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
2. Sharpe W F. The sharpe ratio. Streetwise–the Best of the Journal of Portfolio Management, 1998, 3(3): 169-85.
3. Dowd K. Adjusting for risk: An improved Sharpe ratio. International review of economics & finance, 2000, 9(3): 209-222.

Thank you for the clarifications. I think adding in the additional discussions around these will allow the final version to be stronger. I will retain my current score since that is primarily underscored by the rigor of the benchmarks chosen for the points raised and my current score accurately provides the metrics for this great paper, but otherwise, I am aligned with the discussion items.