Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization

Thank you for the feedback! It seems that your critique and questions focus on the significance of our theoretical analysis and the nature of the experiments on the SORRY-Bench dataset (Section 6). We address these points below.

Significance of Our Theoretical Analysis

We agree that the theory is rather simple in terms of the complexity of calculations. That said, we believe that the utility of a theory need not depend on its complexity, rather on the insights and actionable guidelines that it provides.

As demonstrated by the experiments of Section 3, 5, and 6, our theory identifies salient mechanisms that drive likelihood displacement — a widely observed phenomenon, which was not well understood — for contemporary models and standard preference datasets. Specifically, it characterizes how the model’s embedding geometry drives likelihood displacement, shedding light on why it can occur even in simple settings. This stands in stark contrast to prior work, which attributed likelihood displacement to various complexities in the preference learning pipeline (as discussed in Section 3, and elaborated upon in Appendix B). Furthermore, as shown in Section 5, the CHES score brought forth by our analysis accurately identifies which samples lead to likelihood displacement, whereas alternative similarity measures fail to do so (e.g., the edit distance between preferences, which was suggested by [1] as a potential source of likelihood displacement). We therefore view the relative simplicity of the analysis as a strength rather than a weakness.

SORRY-Bench Experiments (Section 6)

We would like to first emphasize that the main focus of our work is on understanding why likelihood displacement occurs and its undesirable implications, as opposed to proposing a new algorithm for improving the alignment of language models. As specified above, through a combination of theory and experiments, our results substantially advance the existing understanding of likelihood displacement and give a quantitative tool for identifying which samples are expected to cause it in a given dataset. In particular, it is not true that we include only experiments considering the refusal rate of models, as suggested by the review.

The purpose of the experiments in Section 6 is to demonstrate: (i) in a clean, yet realistic setting, undesirable outcomes due to likelihood displacement; and (ii) the potential of the CHES score for mitigating such undesirable outcomes through data filtering. By no means do we claim that one should only consider the refusal rate of a model during alignment. Rather, the experiments demonstrate that, even when only attempting to train a model to refuse unsafe prompts, likelihood displacement can cause the refusal rates to drop (instead of increasing). As discussed in Section 6.2, this is particularly striking since there is no incentive for the model to not refuse a prompt.

Note that, if one would additionally include general instruction following data, then that would only further incentivize a drop in refusal rates. To verify, we ran experiments analogous to those of Section 6 while including samples from UltraFeedback in the training set (in addition to the SORRY-Bench preference data). As expected, this led to a more severe decrease in refusal rates — e.g., the refusal rate of Llama-3-8B-Instruct dropped from 74.4% to 16.7% (in contrast to a drop to 33.4% when training only preference data from SORRY-Bench).

Further Exploring the Utility of the CHES Score in Additional Settings

While Sections 5 and 6 demonstrate the potential of the CHES score for mitigating undesirable effects of likelihood displacement, as stated in the conclusion section, we agree that further investigation is necessary to assess its utility more broadly (e.g., for improving performance in general instruction following settings, as measured by metrics other than the refusal rate). However, this is a non-trivial endeavor, which we believe falls outside the scope of the current work, and is more a promising direction for future research.

We note that in general instruction following settings, it is difficult to categorize whether likelihood displacement is benign or catastrophic, as opposed to the safety alignment setting of Section 6, where refusal responses are benign and non-refusals are catastrophic. Thus, a major challenge is to detect additional settings where likelihood displacement is catastrophic and requires mitigation. For example, it is possible that likelihood displacement will lead to degradation in performance only for some types of prompts and evaluation metrics.

Conclusion

Thank you for taking the time to read our response. We hope that it addresses your concerns, and would greatly appreciate it if you would consider raising your score. Please let us know if you have any further questions; we will gladly expand.

Thank you for the feedback and for highlighting the clarity of our work and the value of our theoretical and empirical contributions! We treat your comments and questions below.

1. The unconstrained feature model is not realistic and an oversimplification. But the reviewer does appreciate the attempt towards analyzing the phenomenon of likelihood displacement from a rigorous theoretical perspective.
2. The theoretical results are restricted in analyzing the local change of log probability of preferred labels, while a comprehend analysis of the entire dynamic of the changes in the log probabilities is unknown, even for the simplified single data and single output token case. Such analysis can give better characterization of whether the log preferred response probability goes up or down after direct preference learning.

We agree that the unconstrained features model simplifies the training dynamics, abstracting away the role of the architecture. However, we believe that the utility of a theory need not depend on its complexity, rather on the insights and actionable guidelines that it provides. As demonstrated by the experiments of Section 3, 5, and 6, our theory identifies salient mechanisms that drive likelihood displacement. In particular, the CHES score brought forth by it accurately identifies samples leading to likelihood displacement (Section 5), whereas alternative similarity measures fail to do so. This is a clear indication that our analysis, despite the adoption of the unconstrained features model and its local nature, captures the causes for likelihood displacement in practical settings with standard preference datasets and contemporary models. We therefore view the relative simplicity of the analysis as a strength rather than a weakness.

With that said, as stated in the conclusion section, extending our theory by studying the evolution of log probabilities throughout training and how the architecture choice influences likelihood displacement (i.e. going beyond the unconstrained features model) can be fruitful directions for future work.

Answers to Questions

1. In Section 4.2.1., Line 297, why the orthogonal component of $\mathbf{W_{y^{+}} - W_{y^{-}}}$ w.r.t. $\mathbf{W_{y^{+}}}$ is $\mathbf{W_{y^{-}}}$? It seems that from Theorem 1 if the likelihood displacement happens (in this specific setting of single token response), the two unembeddings should be well aligned, i.e., $\langle \mathbf{W_{y^{+}}} , \mathbf{W_{y^{-}}} \rangle$ is relatively large, and in this case the projection to the orthogonal space of $\mathbf{W_{y^{+}}}$ is not likely to be introduced by $\mathbf{W_{y^{-}}}$. Can the authors explain more about the intuition here?

This is a great question! We will gladly explain. Denoting by $\Pi_\perp$ the projection operator onto the subspace orthogonal to $\mathbf{W_{y^{+}}}$, it holds that $\Pi_\perp (\mathbf{W_{y^{+}}} - \mathbf{W_{y^{-}}} ) = - \Pi_\perp ( \mathbf{W_{y^{-}}} )$. This is what we meant by saying that $\mathbf{W_{y^{-}}}$ introduces the orthogonal component in line 296. We clarified this point in the updated manuscript; thank you for the question!

Regarding when likelihood displacement occurs in the single token case, as shown in Theorem 1 and discussed in the succeeding text, there are two factors that govern when the preferred response decreases. Indeed, one of them is $\langle \mathbf{W_{y^{+}}} , \mathbf{W_{y^{-}}}\rangle$. Note that it is possible for this inner product to be relatively large and for $\mathbf{W_{y^{-}}}$ to also have a non-negligible component orthogonal to $\mathbf{W_{y^{+}}}$. As reported in Table 13, this is the case for the models and tokens considered in Section 3.

Thank you for the feedback, and for highlighting the strength of our analysis and the potential of the CHES score as a preference data curation tool! We treat your comments and questions below.

1. The scope of the experimental scenarios is a concern. While the experiments include several models and datasets, the scenarios are somewhat limited. They primarily focus on refusal responses, which are single-token. To verify the robustness of the findings, other scenarios, including multi-token responses and diverse types of responses, are encouraged to test out.

We would like to point out that actually, the majority of our experiments consider settings with multi-token responses, including standard preference datasets that contain diverse response types. In particular, in Section 5 we demonstrate over the UltraFeedback and AlpacaFarm datasets that the CHES score can accurately identify samples causing likelihood displacement, whereas alternative similarity measures fail to do so. Moreover, in the experiments of Section 6, the (refusal and non-refusal) responses are not just single tokens, but rather generations either from the language model being trained (e.g. Llama-3-8B-Instruct) or from a diverse set of models.

Only the experiments of Section 3 consider single-token responses. Their purpose is to demonstrate that, even in such simple settings, likelihood displacement can occur and be catastrophic, in contrast to prior work attributing likelihood displacement to various complexities in the preference learning pipeline.

2. From the experiment result in Section 6, it looks like using CHES for data filtering improves the performance a lot on top of DPO. It would be great if the authors can further explore the effect of CHES filtering on more realistic tasks like finetuning on ultrafeedback dataset with models tested in the paper, compared with other baselines.

As discussed in the conclusion section, we agree that further investigation is necessary to assess the utility of the CHES score more broadly (e.g., for improving performance in general instruction following settings via data filtering or for selecting distinct preferences from a pool of candidate responses). However, this is a non-trivial endeavor. Since the main focus of our work is on understanding why likelihood displacement occurs and its undesirable implications, we believe that further exploring uses of the CHES score falls outside the scope of the current work, and is more a promising direction for future research.

We note that, in general instruction following settings, it is difficult to categorize whether likelihood displacement is benign or catastrophic, as opposed to the safety alignment setting of Section 6, where refusal responses are benign and non-refusals are catastrophic. Thus, a major challenge is to detect additional settings where likelihood displacement is catastrophic and requires mitigation. For example, it is possible that likelihood displacement will lead to degradation in performance only for some types of prompts and evaluation metrics.

Answers to Questions

1. How does likelihood displacement evolve over longer training periods? Does it diminish or intensify as the training goes?

Empirically, we observe that when the preferred response probability starts decreasing (in some cases it can increase in the initial steps), it usually keeps decreasing for the remainder of training. This falls in line with experiments of prior work (see, e.g., Figure 5 in [1], Figures 16 and 17 in [2], and Figure 3 in [3]).

2. Is there any recommended threshold for CHES filtering?

For data filtering methods (e.g., [4,5]), the threshold that one should use usually varies depending on the setting and is selected heuristically, e.g., by considering multiple thresholds and selecting based on performance over a validation set. Similarly, we carried out experiments while filtering out different percentages of data. As noted in Section 6, keeping up to 15% of the original samples led to analogous results (i.e. an improvement in refusal rates). Beyond that, as when training on the full dataset, likelihood displacement caused refusal rates to drop. Going forward, exploring guidelines for automatically selecting such a threshold for the CHES score, and data filtering methods in general, is a valuable direction for future work.

Conclusion

Thank you for taking the time to read our response. If it addresses your concerns, we would greatly appreciate it if you would consider raising your score. Please let us know if you have any further questions; we will gladly expand.

Thank you for the support and for highlighting the potential impact of our work! If any questions arise during the discussion period, please let us know and we will gladly respond.