SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety

Thank you for your constructive feedback and comments.

[Definition of Harmless Ratio]

The definition of the harmless ratio is the ratio of harmless responses to all responses. In the model-based evaluation, the beaver-7b-unified-cost model is trained to generate a negative cost for safe answers and a positive cost for unsafe answers. In contrast, during GPT-4 evaluation, we let GPT-4 generate a harmlessness score between 0 and 10, where a safe answer should score higher than 5, and an unsafe answer should score lower than 5. This process is also based on the prompts in Safe RLHF, and we presented the details of our prompts in Appendix C.2.2. of our paper.

[Inconsistency in Helpfulness Evaluation]

Please refer to our general response.

[Further Analysis]

In this work, we focus on deriving a DAA for safety alignment. For future work, we plan to extend our algorithm to handle multiple objectives, such as various binary and continuous safety measures. As part of this future work, we will also address the reviewer's suggestion to analyze the trade-off between model helpfulness and harmlessness across harm categories, such as animal abuse, child abuse, and others.

Thank you for your constructive feedback and comments.

[Different Dataset vs. Safe RLHF]

First, it is challenging to determine which dataset is used in the Safe RLHF paper, as Safe RLHF involves three stages (Beaver-v1, -v2, -v3), each utilizing different datasets. Moreover, the Safe RLHF dataset is not yet fully available and is still being updated.

When we began this research, only a 33K dataset was available in Huggingface. However, the dataset has been updated with a larger version. For the following two reasons, we conducted our experiments using SafeRLHF-30K: After the update, some data entries became inconsistent. For example, certain samples (x,y1,y2) were labeled such that y1 is safer than y2, yet y1 is also marked as unsafe while y2 is marked as safe. We aimed to use a fixed dataset to ensure consistency and avoid issues arising from dataset updates. We emphasize that this decision was made to maintain experimental clarity and consistency, with full acknowledgment of the Safe RLHF authors' valuable contributions.

As the reviewer provided, [2] Huang, etc. One-Shot Safety Alignment for Large Language Models via Optimal Dualization. also used the same dataset.

[Inconsistency in Helpfulness Evaluation]

Please refer to our general response.

[Unnormalized Scores]

We additionally evaluated the performance of SafeDPO and Safe RLHF using three random seeds, reporting the average and standard error. Here, unnormalized rewards refer to the original rewards obtained by the reward model beaver-7b-unified-reward. In the revised paper, we have also reported unnormalized scores in the Appendix B.6.4.

Thank you for your constructive feedback and comments.

[Efficiency of SafeDPO]

(Related to Q1, Weakness 1)

Please refer to our general response.

[Stability of Algorithms]

(Related to Weakness 2)

We additionally evaluated the performance of SafeDPO and Safe RLHF using three random seeds, reporting the average and standard error. Here, unnormalized rewards refer to the original rewards obtained by the reward model beaver-7b-unified-reward. In addition, we have incorporated this result into Appendix B.6.4 of the revised version.

[Comparison with MODPO]

(Related to Q2)

First, define reward functions similar to the SafeDPO setting. Let $r\_1$ be the reward function for the helpful preference, $r\_2$ be the reward function for safety, and $\mathbf{w}=(1,1)$. Then, we can obtain the following MODPO objective for $k=1$:

$

-\mathbb{E}_{\mathcal{D}_1}\bigg[\log\sigma\bigg(\beta\log\frac{\pi_\theta(y_w\vert x)}{\pi_\text{ref}(y_w\vert x)}-\beta\log\frac{\pi_\theta(y_l\vert x)}{\pi_\text{ref}(y_l\vert x)}-m(x,y_w,y_l)\bigg)\bigg],

$

where

$

m(x,y_w,y_l)=r_2(x,y_w)-r_2(x,y_l)\in\lbrace-\infty,0,\infty\rbrace.

$

Here, the differences between the SafeDPO objective and this modified MODPO objective are as follows:

1. Additional terms: $-(\tilde h\_l - \tilde h\_w)\Delta$ for SafeDPO and $-m(x,y\_w,y\_l)$ for MODPO.
2. Reordering preference: In SafeDPO, if $y\_l$ is safe and $y\_w$ is unsafe, the preference is reordered as $\tilde y\_w=y\_l$ and $\tilde y\_l=y\_w$. In contrast, MODPO does not consider reordering preferences.

From 1, consider the following case: If we set

$

r_2(x,y)=\begin{cases}0 & \text{if $y$ is safe} \\ -\infty & \text{otherwise}\end{cases},

$

then following challenging arises: when $m=\infty$ (i.e., $y\_w$ is safe and $y\_l$ is unsafe) or $m=-\infty$ (i.e., $y\_w$ is unsafe and $y\_l$ is safe), it becomes impossible to minimize the objective since the gradient signal becomes zero or $\pm\infty$. To address this limitation, we can instead set

$

r_2(x,y)=\begin{cases}0 & \text{if $y$ is safe} \\ -\Delta & \text{otherwise}\end{cases},

$

then $m(x,y\_w,y\_l)=(-h\_w + h\_l)\Delta$, which avoids the problem of infinite values and aligns more closely with the SafeDPO objective. However, another issue still arises due to the difference (2): even if $y\_l$ is safe and $y\_w$ is unsafe, the objective attempts to increase the probability of $y\_w$, which is unsafe, and decrease the probability of $y\_l$, which is safe. Therefore, although a similar objective to SafeDPO can be derived from the MODPO objective, a simple modification of MODPO is insufficient to properly address the safety alignment objective.

Consequently, MODPO is not well-suited for handling binary variables, whereas SafeDPO cannot address continuous safety variables. As the reviewer pointed out, both MODPO and SafeDPO can be utilized to enhance the safety of LLMs by leveraging continuous and binary safety variables, respectively. For future work, we aim to extend our algorithm to address multiple objectives, incorporating both binary and continuous safety measures. We believe that combining MODPO and SafeDPO represents one of the most promising directions to achieve this goal.

[Related work]

In the revised version, we have moved the related work section to precede the main section, shifting it from Section 4 to Section 3.

Thank you for taking the time to provide such thoughtful feedback. Your positive remarks and suggestions are really helpful in improving the quality of our work.

Thank you for your constructive feedback and comments.

[Contribution]

(Related to Weakness 1)

First, we apologize for the potential impact on readability caused by the use of many notations. We acknowledge that the use of multiple notations can make the paper harder to follow. We are actively considering ways to improve readability while preserving the formal derivations of our formulations, and we will address this in a future revised version.

(1), (2), please refer to our general response.

(3) The core idea of SafeDPO is reordering: (safe and helpful response) > (safe but unhelpful response) >> (unsafe response). However, Eq. (16) (SafeDPO without $\Delta$) is insufficient to reflect the preference for (safe response) >> (unsafe response). Thus, we introduce $\Delta$ to address this issue. Although $\Delta$ is not needed for infinitely many samples, with finite samples, a larger $\Delta$ strongly encourages preferring safe responses (even if they are unhelpful) over unsafe responses (although they are helpful). Therefore, $\Delta$ balances the trade-off between helpfulness and safety when finite samples are used - this explains why safety increases while helpfulness decreases as $\Delta$ increases. In addition, with a very large $\Delta$, SafeDPO suffers from degeneration because SafeDPO with large $\Delta$ becomes similar to the unlikelihood method introduced in the DPO paper, as explained in the Appendix B.5.

[Theoretical Analysis]

(Related to Weakness 2, 3)

First of all, $r(x,y)$ and $c(x,y)$ are unknown reward and cost functions, which are not accessible in practice. As noted in the reviewer’s comment, $r(x,y)$ in SafeDPO serves the same role as $r(x,y)$ in DPO. Since these functions are not accessible, they must be replaced by accessible terms, such as helpfulness preferences and safety indicators.

However, such replacements can introduce bias, that is why we aim to prove that our replacements are theoretically unbiased. To address this, we introduce three propositions.

Theoretically, if $r$ and $c$ were accessible, sampling preferences from $r\_c$ would suffice to solve the safety alignment objective implicitly. However, we cannot access these functions, we replace this sampling with helpfulness preference and safety indicators provided in the dataset. Proposition 3.2 demonstrates that the probabilities derived from sampling helpfulness preferences, safety indicators, and reordering preferences are equivalent to the probabilities obtained by sampling preferences from $r\_c$. In other words, informally, our replacement is theoretically equivalent to sampling directly from $r\_c$.

[Efficiency of SafeDPO]

(Related to Weakness 4)

Please refer to our general response.

[Hard to understand]

(Related to Weakness 5)

For now, we have added Figure 2 to help clarify the flow of the main section, and we will further improve readability in a future revised version.

Thank you for taking the time to provide such thoughtful feedback. Your feedback is not only helpful but also motivates us to refine our study further.

Thank you for your constructive feedback and comments.

[Contribution of SafeDPO]

(Related to Weakness 1)

Please refer to our general response.

[Efficiency of SafeDPO]

(Related to Q2, Weakness 2)

Please refer to our general response.

[Theoretical Analysis]

First, we apologize for any unclear parts. We are currently revising the main section to improve clarity, but this process is time-consuming. During this rebuttal, we are primarily focusing on clarifying our explanation of the derivations.

(Related to Q1)

Theoretically, if $r$ and $c$ were accessible, sampling preferences from $r\_c$ would suffice to solve the Eq. (14). However, we cannot access these functions, we replace this sampling with helpfulness preference and safety indicators provided in the dataset. Proposition 3.2 demonstrates that the probabilities derived from sampling helpfulness preferences, safety indicators, and reordering preferences are equivalent to the probabilities obtained by sampling preferences from $r\_c$. In other words, informally, our replacement is theoretically equivalent to sampling directly from $r\_c$.

(Related to Q3, Weakness 3)

For theoretical derivations, we introduce two assumptions, Assumptions A.1. and A.2., as described in Appendix A. Based on this question, we have updated this section in the revised version.

Assumption A.1: This assumption assumes that the reference model can generate at least one safe response with probability greater than $\epsilon$. Intuitively, this is essential because aligning a LM to generate safe answers requires the existence of at least one safe response. Furthermore, this is not a strong assumption since, in principle, we can always provide a safe but uninformative response, such as “we cannot answer this question”. While such a response may lack utility, it makes this assumption satisfied.

Assumption A.2: This assumption simplifies the derivation by restricting the range of rewards. Without this assumption, the theory can still be derived by shifting and rescaling the reward function to map (x,y) pairs into the fixed range with high probability. However, adopting this assumption allows for a more concise and clear formulation of the derivation.

[About $\Delta$]

(Related to Q4)

The core idea of SafeDPO is reordering: (safe and helpful response) > (safe but unhelpful response) >> (unsafe response). However, Eq. (16) (SafeDPO without $\Delta$) is insufficient to reflect the preference for (safe response) >> (unsafe response). Thus, we introduce $\Delta$ to address this issue. Although $\Delta$ is not needed for infinitely many samples, with finite samples, a larger $\Delta$ strongly encourages preferring safe responses (even if they are unhelpful) over unsafe responses (although they are helpful). Therefore, $\Delta$ trades-off between helpfulness and safety when finite samples are used.

[Human Evaluation]

(Relate to Ethics Concern)

Participation is entirely voluntary, with individuals 18 or older asked to evaluate the safety and helpfulness of language model responses. While the task involves potentially sensitive or harmful content (e.g., curse words, violence, or adult material), participants are informed of the risks upfront and can opt out or skip questions at any time. Privacy is strictly maintained, and participants can withdraw without any penalty. We also inform participants that the purpose of this study is to enhance the safety of language models, and any concerns can be directed to the provided contact.

References

[1] Jiang, Albert Q., et al. "Mixtral of experts." arXiv preprint arXiv:2401.04088 (2024).

[2] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

[3] An, Soyoung, et al. "EXAONE 3.0 7.8 B Instruction Tuned Language Model." arXiv e-prints (2024): arXiv-2408.

[4] Lai, Xin, et al. "Step-dpo: Step-wise preference optimization for long-chain reasoning of llms." arXiv preprint arXiv:2406.18629 (2024).

[5] Pang, Richard Yuanzhe, et al. "Iterative reasoning preference optimization." arXiv preprint arXiv:2404.19733 (2024).

[6] Liu, Zixuan, Xiaolin Sun, and Zizhan Zheng. "Enhancing llm safety via constrained direct preference optimization." arXiv preprint arXiv:2403.02475 (2024).

[7] Huang, Xinmeng, et al. "One-Shot Safety Alignment for Large Language Models via Optimal Dualization." arXiv preprint arXiv:2405.19544 (2024).

[8] Dai, Josef, et al. "Safe rlhf: Safe reinforcement learning from human feedback." arXiv preprint arXiv:2310.12773 (2023).

After carefully reviewing the meta-review, there may have been a significant misunderstanding that requires scientific clarification.

1. The draft has never claimed that Equation (8) is equivalent to Equation (9). Unfortunately, this false assumption appears to have been the primary basis for the rejection.

• The authors seem to believe that these two equations are not equivalent.

2. In fact, the objective is explicitly derived from Equation (8).

• As substantiated by Assumptions A.1 and A.2, as well as Propositions 4.1, 4.2, and 4.3.

Given the positive evaluations from all 5 reviewers who carefully assessed both the theoretical and empirical contributions, it is concerned that the meta-review may have disproportionately relied on a misinterpretation of our theoretical framework. While fully respecting the meta-reviewer’s right to adopt a critical stance, when overriding multiple reviewers’ positive assessments, it is essential that any critiques are grounded in clear and scientifically verifiable evidence.

While fully appreciating ICLR's commitment to maintaining a thorough review process, it is necessary to express significant concerns regarding the meta-review of our manuscript.

The meta-review identifies our work as flawed based on an incorrect assumption—one that our submission never claimed. If there are indeed issues with the logical flow of our arguments, we would be grateful for the opportunity to address them in detail. Ensuring that the review accurately reflects the content and contributions of the submission is essential for supporting the high standards that ICLR represents. This principle should apply to the meta-review as well.