Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

We appreciate the effort and time of the reviewer (tR7t). We would like to address the reviewer's valuable feedback as follows:

While the authors provided implementation details, it is unclear why they used a 7B model with closed weights when a wide range of open-source models of this size is available.

We acknowledge that using open-source models would enhance reproducibility, but due to the constraints of this study, we focused on a controlled setup. In Section 4.1, we offer full details on the models used (7B and 35B parameter models) and the datasets (ULTRAFEEDBACK and BINARIZEDPREF), including exact versions and sizes. While the 7B model is closed-source due to the use of proprietary models or data, we have described these in as much detail as possible. Furthermore, we posit that our findings are likely generalisable to other LLMs, as most LLMs (e.g., Llama, Gemini) are based on standard transformer architectures (Vaswani et al., 2017). For example, the Llama model family has very standard features such as RoPE embeddings (Su et al., 2024).

We also refer the reviewer to the response to reviewer dJek. In summary, the 35B model has openly available weights, and we considered it of scientific importance to compare the 35B model to a smaller model from the same family, rather than one where we have limited or no visibility into the full data selection and training process. There is considerable value in the ability to compare our observations across model sizes. There is also the (unfortunate but very real) more practical limitation of computing resources. The Llama3 models are available in 8B and 70.6B, and would not have allowed us to carry out the full suite of experiments planned. While we value and are aligned with the reviewer’s point of view, we consider an experimental setup with a closed 7B and an open-weight 35B model a fair compromise for the sake of sharing innovative research, and we hope that the reviewer can be convinced to reconsider accepting our research contributions in this spirit.

This aligns with the observations of [1], as probability mass moves towards out-of-distribution (OOD) examples (and to avoid overoptimization, we should prevent leakage of mass to OOD sequences). Therefore, the findings do not seem to present new insights.

We respectfully disagree with this interpretation. The reviewer suggests that [1] posits probability mass moves toward out-of-distribution (OOD) examples and that preventing this leakage is necessary to avoid overoptimisation. In contrast, our work presents an opposing view: we demonstrate that a slight decrease in the likelihood of chosen sequences—effectively redistributing probability mass—can improve output diversity and ultimately enhance model performance. This perspective challenges the idea that avoiding OOD leakage is the sole factor in alignment. Instead, our findings highlight the critical role of balancing likelihood adjustments across chosen and OOD sequences, providing fresh insights into over-optimization mechanisms.

When demonstrating that increased probabilities for chosen sequences can also lead to overoptimization, it is important to explore the probabilities associated with OOD examples.

We argue that our findings serve to indirectly address the reviewer's concern. Since the total probability mass must sum to one, increasing the probabilities for chosen sequences necessarily reduces the probability of OOD examples. Thus, this exploration is implicitly addressed through the principle of probability mass conservation. Our findings emphasize that excessive likelihood increases for chosen sequences can still lead to over-optimisation, underscoring the importance of carefully managing likelihood shifts during alignment training. Nonetheless, we consider this a point of interest and would appreciate the opportunity to discuss this further with the reviewer.

We appreciate your thoughtful comments and the opportunity to clarify and expand on the points raised. Below, we address your concerns in detail.

As I mentioned in my review, I am concerned about Figure 2. Do I understand correctly that this phenomenon can be observed around the 600th step, where the win-rate is at its peak while the probabilities of chosen sequences are slightly reduced?

Thank you so much for your question. Yes, that is correct.

Also, from what I see in this figure, it does not seem that the entire observed results could be considered "an opposing view". Rather, we see that during optimization, it could be beneficial to slightly reduce the probabilities of chosen sequences. However, they further decrease over the course of training. If so, do you have any explanations for this behavior? As a researcher in the alignment field, I am more interested in providing insights into the behavior of methods rather than results merely indicating that some behavior could occur.

Thank you so much for your question.

Why does the reduced likelihood improve model performance? A reduced completion likelihood gives the model a better chance to select alternative tokens. As shown in Figure 2, the probabilities of the chosen sequences align with the model’s Cross-Input Diversity, which enhances its ability to generalise to unseen scenarios. This is reflected in an increased win probability, indicating improved performance.

Why does overly reducing the likelihood harm model performance? At a certain point in training, reducing the completion likelihood too much can negatively impact the model's performance. For example, as explained in Section 4.4, during the early stages of training, Per-Input Diversity (measured by entropy over the top-k tokens, where k = 10) increases. This indicates a broader distribution of selected tokens and a more uniform output distribution for the next token prediction. Considering that the better completion likelihood is decreasing across the training, the increase of entropy at the beginning phase indicates that those tokens from better completion have a higher probability at the initial policy model over other tokens in the top k. The decrease better completion likelihood gives the model a better chance to select other tokens, which increases diversity and enhances generalisation, as reflected in the win probability. However, this trend reverses after a certain point. As Per-Input Diversity (entropy) starts decreasing, the model begins to over-prioritise certain tokens. This suggests that those tokens in the better completion now have an overly low likelihood, lower than other tokens in the top k. Despite this, Cross-Input Diversity keeps increasing, which indicates that the model is still generating diverse outputs, but now it includes tokens that are less relevant or nonsensical, i.e., tokens that humans do not prefer.

When discussing different tasks, it is essential to consider that they are not equal. The Ultrafeedback dataset consists of rejected sequences that are slightly worse than the chosen ones. (...) I do not understand what constraints prevented the authors from using widely accessible models for evaluation.

We appreciate your concern regarding the use of proprietary datasets and their potential impact on reproducibility. We want to clarify that significant portions of our work are fully reproducible:

• Our study includes the open-sourced preference learning dataset UltraFeedback, which has been validated for its quality and utility, as evidenced by its successful use in training the open-source model Zephyr.
• The 35B model weights are openly available. Our methodology is detailed enough to enable replication with similar open models (e.g., models like Aya-Expanse-8B that share our architectural foundations)

The decision to use the BINARIZEDPREF dataset, while proprietary, was motivated by its unique characteristics that complement the publicly available dataset, allowing us to explore a broader range of alignment datasets. Additionally, we provided extensive details on this dataset in the rebuttal revision, including its construction and the quality differences between chosen and rejected sequences. We believe this transparency mitigates some of the limitations associated with proprietary datasets.

It is also worth noting that the use of proprietary resources is common in the research community. For example, numerous published papers from Google rely on proprietary datasets and models, such as PaLM. We believe this aligns with standard guidelines for evaluating work based on its scientific contributions rather than the openness of every resource, particularly when sufficient details are provided to ensure reproducibility and understanding.

If anything is unclear or requires further elaboration, please don't hesitate to ask, and we would be more than happy to discuss and refine our approach.

We appreciate the effort and time of the reviewer (PsR8). We would like to address the reviewer's valuable feedback as follows:

The two indicators are so trivial, that people already know this before. Thus these two findings might not be counted as contributions.

The reviewer raises an important point which we have clarified in the main text of the paper. We respectfully disagree with the observation that the indicators introduced are trivial. While intuitively plausible, these indicators have not been systematically formalised or studied in the context of DAAs prior to our work. Our contributions lie in:

• Formalizing these indicators: Connecting diversity reductions and performance trends explicitly to over-optimization in DAAs.
• Quantitative empirical validation: Demonstrating these trends across diverse datasets, models, and methods. If the reviewer is aware of prior studies explicitly formalising these indicators, we kindly request references for the appropriate acknowledgement. Even in the event that there exists previous work that we are unaware of that has studied these effects, we consider both the breadth and depth of experiments across which we validate these findings to be considerable strengths and a contribution in their own right.

The experiments are restricted. Some DAAs with alternate objective (CPO, SimPO, ...) (such as length normalization) are not tested, making the claim less solid.

We respectfully note that our experiments include three popular DAAs: DPO, IPO, and SILC, which were selected based on prior work [1]. We will explore additional objectives like CPO and future work. We also refer the reviewer to the response to reviewer dJek where, in summary, we highlight the high compute resources to run such experiments and which do not directly strengthen the core findings of our work. We agree with the reviewer that extending our exploration to additional DAAs would be of significant interest, which further validates the contributions made in this work, but doing so is computationally infeasible and beyond the scope of this work. We thus leave such exploration to future work.

[1] Scaling laws for reward model overoptimization in direct alignment algorithms.

In Figure 3, why the DPO/IPO 7B ultrafeedback model cannot gain much improvement compared with initial checkpoint?

Our goal is not to train the model to optimize performance but rather to understand the relationship between likelihood and model performance. The limited improvement observed in the DPO/IPO 7B ultrafeedback model reflects our focus on studying the dynamics of over-optimisation, rather than fine-tuning for maximum performance gains.

We thank the reviewer once again for their valuable and constructive review. We hope that our revised upload has served to address the main points for improvement identified and look forward to continuing to strengthen the clarity of the work through further discussion.

We appreciate the effort and time taken by the reviewer (dJek) to provide such a detailed and constructive review. We respond to the main weaknesses and questions below and incorporate much of this in a revised version of the paper, which we believe considerably strengthens our contribution.

Over-optimization itself is already well-studied in [1]. Some results, like Figure 1, seem to be consequences of the high KL divergence. I think NLL for y_w and KL are highly correlated metrics. It would be useful to see a figure with KL on the x-axis and NLL(y_w) on the y-axis.

Thank you for the suggestion. While prior work [1] has examined over-optimisation through the lens of KL divergence, our analysis uncovers a more intricate relationship. Specifically:

• Rafailov et al. focuses on KL divergence as a primary metric for assessing over-optimization. In contrast, our work introduces a multi-dimensional approach, analysing patterns in likelihood, entropy dynamics, and probability mass distribution.
• KL divergence vs. completion likelihood: KL divergence does not directly correlate with the completion likelihood. Both increases and decreases in completion likelihood can result in higher KL divergence from the reference model. KL divergence is more about how far the model should move, while our likelihood analysis is more about which direction the model should move.

We will include an additional figure plotting KL versus NLL(y_w) in future work, which we believe will further clarify this relationship and strengthen our conclusions.

While Figure 1 supports the claim that there is no correlation between NLL(y_w) and win rate in general, Figure 2 shows that, for the given method and hyperparameters, the best step can be determined by observing NLL(y_w). More precisely, I would suggest tracking the difference between NLL from the previous and current checkpoints. If this value becomes larger than a certain threshold, one can stop training and thereby obtain the strongest checkpoint. Therefore, from the reported I would say that tracking NLL(y_w) could be sufficient to detect over-optimization across all methods (instead of proposed methodology in Section 4.4, which depends on method).

We respectfully disagree with the suggestion that tracking NLL(y_w) differences could serve as a universal stopping criterion. Our experiments show that:

• Optimal NLL ranges differ significantly across DAAs, making a universal threshold impractical (evident in Figures 2-4).
• Using NLL thresholds would require method-specific calibration, defeating the purpose of having a general detection mechanism.
• Consider the comparison between win rates in the first row of Figure 2 and NLL(y_w) in the second row of Figure 2, across all three DAAs, we can consistently see that win rates peak earlier than tracking NLL(y_w) would suggest, even accounting for effects like smoothing/moving averages or deltas to previous checkpoints. NLL(y_w) also fluctuates considerably, and would require a separate representative held-out validation set, and method-specific thresholds as mentioned above. It can and should be used as a signal, but relying on it alone is suboptimal.

In contrast, our proposed method provides an algorithm-agnostic framework by leveraging additional signals such as diversity and provides signals that correlate more reliably with likelihood over-optimisation across different DAAs, which is also particularly beneficial in the common situation of selecting between different DAAs for best performance.

ORPO loss

We acknowledge the reviewer’s observation regarding the distinction between ORPO’s loss formulation and DPO. In general, KL loss (where $\pi_{\text{ref}}$ is used) can be interpreted as a form of regularisation. In ORPO, KL regularisation is replaced with SFT (using a cross-entropy loss) over preferred completions. This forms the basis for our claim that ORPO can be viewed as a combination of DPO and SFT over preferred completions. This approach is conceptually similar to SLiC-HF [3], which also uses cross-entropy regularisation instead of KL.

We hope this clarification helps to resolve any confusion, and we welcome further discussion on this point.

The bend in the NLL shown in Figures 2-4 is a promising indicator.

We sincerely appreciate your suggestion that the bend in NLL shown in Figures 2-4 is a promising indicator. We have incorporated your insightful observation into our discussion, as it aligns closely with our core findings and adds further depth to the narrative of our paper. Additionally, we emphasise that identifying and leveraging such signals, including NLL, represents a novel aspect of our work that has not been explored in prior studies.

Model choice

We thank the reviewers for their feedback. Below, we address the points raised to clarify our methodology and approach:

Reproducibility of Our Work. A significant portion of our study is reproducible and utilizes open-source resources. Specifically:

• We employed the UltraFeedback dataset, an open-sourced preference learning dataset that has been validated in prior work for its quality and practical utility. This dataset successfully supported the training of Zephyr, a fully open-source model, underscoring its efficacy.
• The 35B model weights used in our study are openly available, ensuring researchers can directly reproduce our results. Furthermore, our methodology is sufficiently detailed to enable replication with alternative open models such as Aya-Expanse-8B, which shares architectural foundations with our work.

It is also worth noting that the use of proprietary resources is common practice in the research community. For example, numerous published papers from Google rely on proprietary datasets and models, such as PaLM, to advance research in large-scale language modelling. We believe this aligns with standard guidelines for evaluating work based on its scientific contributions rather than the openness of every resource, particularly when sufficient details are provided to ensure reproducibility and understanding.

References:

[3] SLiC-HF: Sequence Likelihood Calibration with Human Feedback.

If anything is unclear or requires further elaboration, please don't hesitate to ask, and we would be more than happy to discuss and refine our approach.

We thank the reviewer for engaging in valuable and insightful discussions, particularly over the weekend.

• NLL/KL Relationship. The NLL/KL relationship described by Ravailov's work assumes that completions are sampled according to $\pi_{\text{ref}}$, which is not always the case in our setting. In practice, sampling is often governed by the model’s learned distribution, which may differ from $\pi_{\text{ref}}$.
• KL Divergence Dynamics. It is important to note that the overall KL divergence can only increase during training, as the model’s distribution $\pi_{\text{model}}$ gradually deviates from the reference distribution $\pi_{\text{ref}}$. This is a consequence of learning, where the model adapts to the data and moves away from $\pi_{\text{ref}}$, making direct control of KL divergence challenging.
• Challenges in Offline Settings. As detailed in [1] (Section 4), controlling KL divergence in offline scenarios is inherently difficult. This is because offline loss correlates poorly with actual KL divergence, except in cases where the model distribution remains extremely close to $\pi_{\text{ref}}$. Empirical evidence (e.g., observed order-of-magnitude differences in KL for minimal changes in offline loss) suggests that even small variations in offline loss can lead to significant and unpredictable changes in KL divergence. Given our reliance on fixed datasets without access to online feedback, adjusting KL divergence between $\pi_{\text{model}}$ and $\pi_{\text{ref}}$ in a controlled manner is infeasible in our setup.

While we would like to highlight once again that the relationship between KL and NLL is not central to addressing the research questions our work investigates, and that irrespective of the relationship, observations around scaling effects (particularly with larger models than previously studied) serve to motivate our work and insights. Nonetheless, we agree with the reviewer that the KL vs NLL scatter plot is interesting and will add value to our work. This is a valid and important point; we should have addressed this better in our original paper and included it. This was an oversight on our part.

Due to time and access constraints, we may not be able to generate this plot before the end of the discussion period (although trust us, we are trying), however, we will commit to including the plot in the camera-ready version of the paper.

Once again, we thank the reviewer once again for your engagement and highly valuable contributions.

[1] Generalized Preference Optimization: A Unified Approach to Offline Alignment.

Thank you for the response. Sorry for the pause in discussion. I would like to clarify my thoughts about the NLL vs. KL question.

I found that in [1], the SFT model was used as a starting point for the DAA algorithms; therefore, from this perspective, $KL(\pi_\theta \| \pi_{ref})$ and NLL(y_w) are expected to have similar values, because the SFT was learned with the NLL(y_w) objective.

I noticed that you use a slightly different setup (correct me if I'm wrong) with instruction versions of the base models, which were probably trained and maybe aligned on different datasets. In that case, when $\pi_{ref}=\pi_{inst} \neq \pi_{sft}$, your claims may be right.

From this perspective, I would like to emphasize two points:

• Is NLL correlated with KL when $\pi_{ref}$? Yes.

• Is NLL correlated with KL in your case? Probably no.

However, I still think that in [1], it was shown that extreme KL values (in the case where $\pi_{ref} = \pi_{sft}$) lead to suboptimal performance; therefore, you might have the same conclusion that extreme values of NLL(y_w) yield suboptimal performance.

In your case, if you were to train the base model with SFT loss as a first stage, in the second alignment stage with DAA, you would probably see the aforementioned correlation.

It is important to note that the overall KL divergence can only increase during training.

The likelihood of y_w also tends to decrease (Figure 7 in [1], Figure 2 in [2]) during training, which is a known problem of DPO-like DAAs without an additional NLL(y_w) loss term.

While we would like to highlight once again that the relationship between KL and NLL is not central to addressing the research questions...

Agreed. However, these results are presented in Figure 1, which is typically used to present the main contribution of the work and therefore draws a lot of the reader's attention.

Additionally, a significant part of the introduction starts with the idea that higher LL(y_w) does not always produce better results, which is presented as a new idea (lines 70-73), but from my perspective, this is not the case.

The second contribution mentioned in the introduction, discussed in lines 80-82 regarding overoptimization signals, is also questionable to me because it is method-dependent.

Due to time and access constraints, we may not be able to generate this plot before the end of the discussion period.

I understand the authors' position as a person, but as a reviewer, I have to rely on the results of experiments that the authors could have prepared during the discussion period.

Overall, the experiments provided by the authors could be interesting for the alignment community, but the main questions for me are the interpretation and analysis of the obtained results.

[2] Noise Contrastive Alignment of Language Models with Explicit Rewards

If there are any points I may have overlooked or if you would like to discuss any aspect further, please do not hesitate to response.

We appreciate the effort and time taken by the reviewer (VwQC) to provide such a detailed and constructive review. We respond to the main weaknesses and questions below and incorporate much of this in a revised version of the paper, which we believe considerably strengthens our contribution.

The paper only studies single epoch training when DPO typically needs 2-3 epochs for best performance. This important limitation is not well justified or analyzed.

We chose a single-epoch training setup to focus on understanding the likelihood over-optimisation in DAAs. This approach aligns with prior work [1] demonstrating that over-optimization can manifest in as little as one epoch. While multi-epoch training might yield stronger performance, our findings are specifically targeted at uncovering fundamental dynamics rather than maximising benchmark scores. In future work, we plan to extend our analysis to multi-epoch settings to evaluate the consistency of our conclusions.

[1] Scaling laws for reward model overoptimisation in direct alignment algorithms.

The BINARIZEDPREF dataset lacks crucial details about its construction and preference collection. If preferences come from GPT-4 rather than humans, the generalization of findings is questionable. Could you provide more information about BINARIZEDPREF construction, including how preferences were collected and validated? The source of preferences is particularly important.

We acknowledge the need for clarity regarding dataset construction. We have provided further information on the construction of BINARIZEDPREF in response to the question below and will add this detail to a revised version of the paper.

The BINARIZEDPREF collection process used a robust multi-source approach combining professional annotators, multiple independent annotation pipelines, and various validation methods. The foundation comes from professional annotation services (~70% of data), with rigorous quality control through multi-annotator consensus, adversarial validation sets, and specialized verification datasets for issues like hallucination and repetition. We've ensured broad domain coverage, incorporating specialised modules for code generation, RAG interactions, STEM, and medical domains while maintaining strong multilingual capabilities across French, Spanish, Korean, Japanese, German, and Italian - including dedicated datasets for handling code-mixing and language transition cases. Quality control is implemented through multiple layers: consensus-based annotation (1-3 annotators depending on complexity), dedicated adversarial validation sets, and specific datasets targeting quality aspects like anti-repetition, length control, and format adherence. The data is predominantly recent (2024), with carefully weighted components and explicit test sets for key capabilities. We use strategic copy multipliers (up to 5x) for crucial capabilities, and the entire dataset is organised into functional groups (multilingual, code, RAG) to ensure balanced training across all target capabilities.

To further validate the generalisation of our findings, we also tested on open-source datasets in our paper, such as UltraFeedback, demonstrating consistent trends and supporting our conclusions.

The primary evaluation uses GPT-3.5-turbo as a baseline which feels dated. Testing against stronger models like LLaMA-3 would strengthen the findings.

We have already included evaluations against modern LLMs, including GPT-3.5-Turbo, GPT-4o, Claude-3-Sonnet, LLaMA-3-8B, and LLaMA-3-70B-Chat. These results are detailed in Figure 7 of the Appendix. While GPT-3.5-Turbo serves as a baseline, the inclusion of stronger models like LLaMA-3 further validates our findings.

While the single-epoch training question is particularly intriguing, we respond to the motivation above and leave exploring multi-epoch training as an extension for future work. We thank the reviewer once again for their detailed and valuable feedback and hope that our revisions have served to clarify the contributions of our work, which we believe will be extremely valuable for informing future research in this direction.