SimPO: Simple Preference Optimization with a Reference-Free Reward

We thank the reviewer for acknowledging the clarity and comprehensive evaluation of our paper. We address your raised points as follows.

KL regularization and safety: We would like to present our most recent results based on the gemma-2-9b-it model. We measured the original gemma-2-9b-it model, its DPO variant, and the SimPO variant using Sorry-Bench (Xie et al., 2024 [1]), a benchmark for testing the refusal behavior of language models when faced with unsafe queries. Our findings indicate that further tuning the Gemma model on the UltraFeedback dataset with either DPO or SimPO increases the attack success rate, probably due to the lack of safety-related data in UltraFeedback. However, we also find that SimPO is safer than DPO. Therefore, we believe that SimPO does not raise more safety concerns than DPO.

We further direct the reviewer to the PDF attached to our general response, where we demonstrate that even without explicit KL regularization, SimPO achieves a comparable KL divergence from the initial SFT model as KL-regularized objectives like DPO.

[1] SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors

Why is length-controlled WR important: The length-controlled win rate (LC-WR), introduced by Dubois et al. (2024) [2], effectively addresses the length bias issue prevalent in model-based evaluations such as AlpacaEval. This metric has demonstrated a better correlation with human judgments compared to raw win rates [2]. Hence, it has been adopted as the preferred metric for ranking models on the AlpacaEval leaderboard. Given its significance and widespread acceptance in the field, we have incorporated the length-controlled WR as one of our primary evaluation metrics in this study.

[2] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

Is SimPO more fragile and sensitive to data quality? Our extensive experiments across various settings do not indicate that SimPO is more susceptible to low-quality data than other methods. In fact, the UltraFeedback dataset used in our study inherently includes a variety of data quality levels, as the responses are generated by randomly-sampled models from a diverse pool, including relatively weak models (e.g., Alpaca-7B). Despite this variability in data quality, SimPO consistently demonstrates significant empirical advantages over other baselines.

We hypothesize that SimPO's robustness to data quality issues stems from two key factors:

1. The use of a small learning rate, which is the common practice in preference optimization algorithms and naturally prevents significant divergence from the initial model.
2. The inherent robustness of some large language models (e.g., gemma models), which allows them to maintain performance even when exposed to some degree of low-quality input.

These factors combined may provide sufficient protection against catastrophic forgetting, even without the need for explicit KL regularization.

Dear Reviewer Wuuc,

Thank you once again for your thoughtful review and valuable feedback! We appreciate your support!

Sincerely,
Authors

We thank the reviewer for acknowledging the novelty, simplicity, and significant empirical results of SimPO!

Fine-grained case studies: Thank you for the suggestion! We refer the reviewer to Figures 8 and 9 in our original submission, as well as Figure 2 in the PDF attached to our general response. These examples intuitively demonstrate why SimPO outperforms DPO:

• SimPO can produce better-structured answers than DPO. (Figures 8 and 9 in our paper)
• SimPO can generate more concise and clear responses than DPO. (Figure 2 in the rebuttal PDF)

We acknowledge that qualitative analysis of model outputs is becoming increasingly challenging, given the breadth and depth of the questions in these benchmarks. We will provide more qualitative cases in our next revisions.

Online methods: Thank you for the suggestions! We are actively working on an online version of SimPO. Specifically, we alternate between generating online preference optimization data and training the model with the generated data. Here are some initial results we have for a stronger version of the model — gemma-2-9B-it:

We find that the online version of SimPO continues to improve performance with each additional iteration.

Tasks other than chat, e.g., reasoning tasks like coding and maths: In our initial manuscript (also shown in the table below), we did find that after further training on the Llama-3-Instruct-8B model, the chat ability of the models enhances at the sacrifice of the degradation on GSM8k (math) and MMLU (knowledge). However, after more exploration, we find that such a performance degradation is largely attributable to the lack of robustness of the base model (i.e., Llama-3) rather than the SimPO objective. We’d like to present the following new results from the gemma-2-9b-it model. The last two rows in the table below demonstrate that SimPO largely retains general knowledge (MMLU) and even slightly improves the original model’s math ability (GSM). Additionally, it significantly enhances coding ability, as demonstrated by the Arena-Hard benchmark, which primarily consists of real-world coding questions.

We'd like to thank the reviewer for the thoughtful feedback. We address your questions as follows.

Without KL to the SFT, one can violate the constraint and do better with the reward for that specific or similar task, but what about the performance in other tasks that the pre-trained/SFT model was good at? While reward hacking is theoretically possible without explicit regularization, several practical factors mitigate this risk: (1) the learning rate, (2) the preference dataset, and (3) the SFT model’s robustness to forgetting.

1. The learning rate of preference optimization algorithms is typically very small (e.g., 5e-7), which naturally prevents significant divergence from the SFT model.
2. The preference datasets are often constructed to cover a broad range of tasks and domains, helping retain the model's existing knowledge and task versatility.
3. Large language models generally have a substantial capacity to learn from new data without catastrophic forgetting of previously learned tasks. This robustness can mitigate the need for explicit regularization, similar to how instruction tuning (SFT) retains the model’s pretrained knowledge without KL regularization.

The above factors combined might be sufficient to ensure that the model learns human preferences while retains its generalization capabilities. Empirically, with appropriate hyperparameters, SimPO can result in a similar KL divergence to DPO (see our response to your last question) and comparable academic benchmark performance (see Table 9 in the Appendix). Additionally, we demonstrate new results in the table below that fine-tuning gemma2-9b-it with SimPO yields significantly improved instruction following ability without degradation on academic benchmarks including math (GSM) and general knowledge (MMLU).

DPO has a closed-form unique solution due to the strong convexity of the KL regularized RLHF objective resulting in the particular objective. Without KL, is the objective still strongly convex? If not, what are the insights that this objective will have some good convergence properties? The derivation of the DPO objective from RLHF relies on the specific condition of achieving the optimal policy $\pi^*_{\theta}$. However, this assumption is rarely met in practice due to the inherent challenges in optimizing deep neural networks. Therefore, there's no guarantee that DPO faithfully implements the original RLHF objective. Our work departs from this assumption: we directly address the potential misalignment between the DPO objective's reward metric and the decoding objective. This focus allows us to tackle the practical limitations of DPO without relying on idealized assumptions.

Furthermore, comparing preference optimization (PO) objectives in terms of convexity or convergence is inherently difficult. All these objectives, including RLHF (PPO), DPO, and SimPO, are generally non-convex with respect to the model parameters $\theta$. This is because the policy model $\pi_{\theta}$ is parametrized by multi-layer, non-linear Transformer architectures, making it a non-convex function with respect to $\theta$. Consequently, any PO objective that depends on $\pi_{\theta}$ will also be inherently non-convex with respect to the model parameters.

The motivation for the objective is heuristically driven and unclear from where it comes from: Our SimPO objective is systematically derived by aligning the reward in training with the generation likelihood in decoding:

1. We use the sequence likelihood objective, optimized by decoding algorithms, as the reward metric for winning/losing responses during preference optimization.
2. We then apply the Bradley-Terry objective over these reward metrics to formulate the SimPO objective.
3. The target reward margin hyperparameter in SimPO is analogous to the margin loss in SVMs and represents the home advantage in Bradley-Terry models.

Can you provide the KL regularization of your model to the SFT and show how much it deviates from the SFT in comparison to baselines? In the PDF attached to our general response, we illustrate the KL divergence of SimPO and DPO. Figure 1 shows that (a) increasing $\beta$ in DPO can reduce the KL divergence, but (b) it can also overly constrain the model from effective learning of the preference dataset, resulting in lower generation quality. Therefore, there is a trade-off between learning preference data and staying close to the initial model. With appropriate hyperparameters, SimPO can achieve similar KL divergence to DPO.

We’d like to thank the reviewer for acknowledging the novelty and simplicity of our proposed approach. We address your raised points as follows.

The target reward margin $\gamma$ requires extra tuning: We acknowledge that the newly introduced target reward margin requires additional tuning. However, we’d like to emphasize that within a reasonable range of $\gamma$ values, SimPO consistently outperforms the DPO baseline, as shown in the table below (also Figure 3a in the manuscript). Furthermore, we find that using $\gamma$=1.5 generally yields sufficiently good results across all settings. While slight tuning can further improve performance, it is not mandatory.

Lack of mathematical understanding: While it will be challenging to derive a rigorous theoretical understanding yet, we offer two possible explanations:

1. SimPO's reward, the average log-likelihood of a sequence, closely aligns with the objective used during decoding. Although it's challenging to prove rigorously, random sampling with a temperature of 1 is likely to match SimPO's reward metric better than either (1) DPO or (2) SimPO without length normalization.
   • DPO involves a reference model in its formulation, which is not used during decoding.
   • Analogous to beam search, which ranks candidate sequences of varying lengths using average log-likelihood, length normalization is crucial when comparing sequences of different lengths (i.e., winning and losing responses) for reward calculation in SimPO. The reward without length normalization is biased, as evidenced by the tendency to assign higher likelihoods to longer sequences (Figure 2). This can result in excessively long outputs, such as degeneration into repetitive tokens.
2. Another perspective for understanding the method is that the SimPO reward decouples the reward from the sequence length. According to [1], sequence length positively correlates with model-based judgment. SimPO’s disentanglement allows us to train the policy without biasing it to generate longer sequences merely to achieve a higher reward. Instead, it encourages the model to learn and focus on the differences beyond sequence lengths.

We hope this helps clarify the effectiveness of our approach!

[1] A Long Way to Go: Investigating Length Correlations in RLHF

DPO with an offset (Amini et al., 2024): We conducted additional experiments to determine if adding an offset (Amini et al., 2024) would further enhance DPO. After tuning $\gamma$, we found that it does not lead to any further improvement. Specifically, we used $\gamma$=0.1 for mistral-base and $\gamma$=0.0 for mistral-instruct. Increasing gamma beyond these values resulted in worse results.

The objective of DPO inherently includes an instance-wise target reward margin $\gamma_{\text{ref}}$, as shown below:

where $\gamma_{\text{ref}} = \beta \log \pi_{\text{ref}}(y_w \mid x) - \beta \log \pi_{\text{ref}}(y_l \mid x)$. This may explain why adding an extra margin to DPO will not be as effective as it is with SimPO. We will add these results and discussions to our revision!

Discuss about KL divergence: In the PDF attached to our general response, we illustrate the KL divergence of SimPO and DPO across different $\beta$, measured on the winning responses from a held-out set during training. In summary:

• We find that SimPO, even without KL regularization, yields a reasonably small KL divergence from the policy model to the SFT model, which is comparable to DPO under its optimal $\beta$.
• Within the range of $\beta$ explored, increasing $\beta$ reduces the KL divergence for both DPO and SimPO, but it can also overly constrain the model from effective learning of the preference dataset, resulting in lower generation quality. This suggests that KL divergence is not monotonically related to reward scores or model quality (Gao et al., 2023 present a similar phenomenon). We hope this helps clarify the reviewer’s concern about KL divergence!

Thanks for the insightful suggestion to test the discrepancy between the reward metric and decoding methods! We follow the reviewer's suggestion to test beam search with different beam sizes (e.g., 1, 5, 10) using the mistral-base-SimPO model. We find that increasing beam sizes does increase the generation quality measured by win rate, as shown in the following table. It's even better than the number reported in our paper! We think that this helps validating that aligning rewards and decoding metrics more explicitly is beneficial. However, a clear downside is that the runtime significantly increases as beam size increases, and we think this might be why people stick with greedy decoding/sampling instead of using beam search for LLM generation.

Thank you for the thoughtful comments; this is an excellent question! To clarify how length normalization (LN) impacts preference learning, we need to consider three factors:

• Length bias of total log-likelihood: As you mentioned, longer sequences typically accumulate a lower (more negative) summed log-likelihood, inherently disadvantaging longer sequences.
• Length difference in preference pairs: We found that, in many instances, the winning response ($y_w$) is longer than the losing response ($y_l$).
• Objective of preference learning: Preference learning aims to assign higher rewards to winning responses ($y_w$) compared to losing ones ($y_l$).

Without LN, the reward metric $r_{\text{SimPO w/o LN}}(x,y) = \log \pi_\theta(y \mid x)$ is based on the total log-likelihood. Due to the length bias, longer winning responses ($y_w$) generally have a lower total log-likelihood compared to shorter losing responses ($y_l$). To correctly rank these $y_w$ higher, the model must overcome this bias by assigning disproportionately high probabilities to each token in $y_w$ to ensure the reward on $y_w$ exceeds that on $y_l$. This compensatory effect can cause the model to become miscalibrated, as illustrated in Figure 2(c), where it begins to excessively favor longer sequences. As a result, during decoding, such an ill-calibrated model tends to generate longer sequences. Essentially, LN acts as a calibration mechanism to prevent this overcompensation in the training stage.

To further illustrate this for a mathematical understanding, let's analyze the loss of SimPO with and without LN. For simplicity, we omit the expectation symbol in the following formulas:

• With LN: $L_{\text{SimPO}} (\pi_\theta) = - \log \sigma \left( \frac{\beta}{|y_w|} \log \pi_\theta(y_w|x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l|x) - \gamma \right)$
• Without LN: $L_{\text{SimPO w/o LN}} (\pi_\theta) = - \log \sigma \left( \log \pi_\theta(y_w|x) - \log \pi_\theta(y_l|x) - \gamma \right)$

It can be seen that without LN, the reward difference $\Delta r = \log \pi_\theta(y_w|x) - \log \pi_\theta(y_l|x)$ is based on total log-likelihoods, and is generally more negative when $y_w$ is longer than $y_l$ due to the length bias. Consequently, $\sigma(\Delta r - \gamma)$ approaches 0, and thus the final loss for such instances, $- \log \sigma(\Delta r - \gamma)$, will be very large. This heavily biases the model towards learning such preference pairs. As demonstrated in Figure 2(a), this can result in the failure of learning the opposite cases where $y_w$ is shorter than $y_l$ (the loss for these cases will be very small by applying a similar reasoning).

We hope this addresses the reviewer's question, and we’ll include these clarifications in the revision. If you find this explanation satisfactory, we kindly ask you to consider raising the score, thank you so much for your time! We are also happy to help further clarify any questions you have!