We are very grateful for your recognition and encouragement of our work. We are deeply touched. Regarding your two questions:

Typo in L46

There is indeed a typo on line 46. It should be Figure 1. We made an error when referencing the figure, which caused confusion in understanding. We sincerely apologize.

About Figure suggestion

Regarding 2a, you provided a very sincere suggestion. We will correct the figure according to your recommendation in the subsequent version. We apologize that the figure is too small, causing reading inconvenience.

About the failure of joint optimization

We also feel it is very regrettable. There are many possible reasons that lead to the failure of joint optimization. We think the problem mainly lies in data quality. We will dive into this question further and try to solve this in future work.

Thank you again for your support and recognition of our work!

Thank you for your kind suggestions.

About learning rate selection

Regarding the first point in the weaknesses, we also tried other learning rates. For the Llama model, with a learning rate of 1e-6, the DPO results were 18.83 and 17.26, which are also better than the commonly used 2e-5 learning rate. Our SFT learning rate adjustment is a trick based on theoretical properties, and the results with other learning rates on Llama hopefully demonstrate the universality of small learning rates for performance improvement.

About model selection

We followed SimPO's experimental settings to facilitate using their results as baseline comparisons. SimPO uses exactly the same settings as ours, and such model configurations are very common in preference learning, as can be seen in SimPO, DPO, ORPO, and GEM. So, maybe this setting is enough?

About SFT and DPO relationship

Our work mainly focuses on the connection between SFT and DPO. As we mentioned in the experimental section L268, a good SFT starting point may not necessarily score high on benchmarks, so SFT performance has variability, but we achieved significant improvements on algorithms like DPO and SimPO.

About representation-level analysis

In Section 4.4, our analysis of logits changes, and in Section 4.5, our visualization of logits value changes during the SFT process, are both demonstrations of what the model is doing at the representation level.

About Key parameters

Our work does not involve the key parameters you mentioned, such as clipping bounds in pre-clip and the threshold in advantage filtering. Your question has left us very confused.

About Typo in Line 45

The figure reference on line 45 is incorrect, and we sincerely apologize.

About why increments emerge after post-alignment training?

SFT mainly provides a better initial point for DPO in the post-training process. As we limited the update magnitude during the SFT stage via kl or a small learning rate to ensure the exploratory nature of the SFT checkpoint, it may not perform well on specific downstream tasks. However, it maintains output diversity and acts as a better starting point for alignment algorithms, which contributes to better results after performing alignment algorithms.

About why KL is suitable

The distance relationship between the SFT model and the base model is also relative. When LLM pre-training converges, the cross-entropy loss is around 1. However, for the original maximum entropy, taking Qwen as an example, the entire vocabulary size is 128,256, so the maximum entropy at uniform distribution is log(1/128,256) = 5.11. It is significantly larger than the cross-entropy loss. When the maximum entropy is achieved, it completely loses the characteristics of language distribution. Therefore, although the SFT stage changes output probabilities, as long as the output is natural language, the cross-entropy loss constraint is reasonable. We constrain the relationship with the base model to maintain the rich natural language modeling in the base model.

We apologize for the interruption. We would like to know if our reply has addressed your questions or if you need any further clarification from us.

Thank you for taking the time to read our work. Your judgment about the importance of KL is very accurate. There may be some aspects we haven't expressed clearly, so we would like to restate our method here:

• The post-training process typically includes multiple learning methods such as SFT and RL, and their theoretical connections are underexplored. To deeply understand what the SFT process is actually doing, we return to the maximum entropy imitation learning optimization objective and make corrections that align with language modeling, adding a cross-entropy term to the entropy term. The goal is to keep the model from deviating from the language distribution while maintaining explorability.
• We follow the derivation approach of non-adversarial imitation learning and obtain a series of imitation learning objectives. When the f-divergence is chosen as total variance, it exactly equals the commonly used SFT objective function. We also obtained some new insights about the SFT process, which we list as follows
• The above derivation process proves that SFT is actually also learning implicit reward. According to Equation 2, SFT is equivalent to searching for the most suitable reward in the optimal reward-policy space based on the given expert demonstrations. At the same time, DPO also proves that in reference learning, we can learn implicit rewards to directly fit preference signals. So implicit reward can naturally serve as a unified perspective to connect with DPO.
• We found that when taking total variance, the original KL derivative term -log π_ref is a zero-order term in stochastic gradient descent and does not contribute to the gradient. In RL setting, the role of KL is mainly to constrain the magnitude of single-step updates. We constrain the step size of single-step optimization in the same way by reducing the learning rate.
• Since the derivation process has the same mathematical structure as DPO, we simultaneously extend the connection between LLM logits and Q-function from the DPO process to the SFT context. We found that the logits of LLM during the SFT process can also serve as a Q function. They represent the expected future return starting from the current state and can serve as a judgment of the quality of the current state-action pair.

About the cross-entropy term in KL will be small

You proposed a thoughtful question! In fact, this value is relative. When LLM pre-training converges, the cross-entropy loss typically ranges from 1.x to 2.x(https://arxiv.org/pdf/2407.21783). For the original maximum entropy, taking Qwen(https://arxiv.org/abs/2309.16609) as an example, the entire vocabulary size is 128,256, so the maximum entropy at uniform distribution is log(1/128,256) = 5.11, which is significantly larger than the cross-entropy loss. At this point, it completely loses the characteristics of language distribution. Therefore, although the SFT stage changes output probabilities, as long as the output is natural language, the cross-entropy loss constraint is reasonable.

What is $V^*$

For any given reward, there must be a corresponding optimal policy. The expected future return that the current policy will obtain in a certain state is defined as $V^*$.

The KL deficiency problem in SFT

Your judgment about KL deficiency in SFT is absolutely correct! This is also one of the core contributions of our work. We found that the KL term, which originally plays a huge role in the imitation learning objective, becomes ineffective due to the corresponding term under total variance, i.e., -log π_ref in Equation 5, being a zero-order term. Therefore, we made some corrections to common SFT tricks, including small learning rate correction and other imitation loss corrections. We found that these effects all show significant improvements.

Finally, thank you again for your insights, and we look forward to further discussion with you!

We are very grateful that you are willing to spend time carefully reading our work and for your high recognition of this research.

About Different Reward Structure

Our work mainly provides theoretical relevance for understanding the SFT process from the perspective of implicit reward. Thus, alignment algorithms that use implicit reward can easily establish connections with our theoretical framework. For different reward structures like complex multi-turn feedback, there are already some works based on implicit reward, such as https://arxiv.org/pdf/2406.14868, that can establish connections with our work via implicit reward. At the same time, because in our imitation learning process, we target the finest-grained reward function, i.e., token-level reward, trajectory reward, or step-level reward can all be obtained through additive aggregation from token-level reward, allowing for very natural theoretical extensions.

About Logits-to-Q

Your interest in the logits-to-Q extension is very insightful. As we mentioned in L187, the logits-to-Q properties in the DPO process have already been pointed out in previous work. Our work derived a consistent mathematical structure in the SFT process and extended the theory. There are works(https://arxiv.org/abs/2502.01456, https://arxiv.org/abs/2404.18922) that have already used the logits-to-Q properties in the DPO process to guide the learning process and achieve good results. Practically, we found that in the experiments conducted in Section 4.4, the logits of LLMs show greater differences when they have consistent judgments(Avg std. 5.81) about quality compared to when they have opposite judgments (Avg std. 2.76). Since the statistical data shows that the quality judgments of the two models are positively correlated in the experiments, we suggest treating the conclusion of non-validity as random fluctuations caused by tokens having insignificant contributions to the final reward. We hope this can make the logits-to-Q question more convincing.

Thank you again for taking the time! We are very grateful for your recognition of our work!

Thank you for the response. I appreciate the argument that token-level rewards can extend naturally to multi-turn or fine-grained feedback settings, as well as empirical hints you provided on the logits-to-Q property. It would be reassuring if the narrowness of the task domain in the experiments were addressed as well.

Thank you for your patient reading and your very insightful questions.

About logits-to-q

Your question about logits-to-q is very perceptive. There are three points we would like to discuss with you:

1. As we mentioned in L187, the relationship between logits and the q function in the DPO process has also been discussed in previous works(https://arxiv.org/abs/2404.18922, https://arxiv.org/pdf/2404.12358). We derived a consistent mathematical structure based on the SFT optimization objective and extended the logits-to-q conclusion from the DPO process to the SFT process. Assumption 2 is similar to that made in previous work.
2. Beyond theoretical proof, we provide indirect experimental evidence in Section 4.4. Since Q-V functions represent expected returns and their numerical values indicate the quality of state-action pairs, we showed that different models share some common judgments in their quality assessments. This indirectly proves the correctness of logits-to-q.
3. After thoroughly investigating the validity of logits-to-q, we find that the magnitude of logits changes is more pronounced (Avg std. 5.81) when two models share similar judgments, compared to when they have opposite assessments of state quality (Avg std. 2.76). This also supports the validity of Assumption 2. We hope these explanations can make the logits-to-q conclusion and corresponding assumption 2 more convincing.

A better SFT starting point for DPO may not show better performance on metrics

Thank you for your interest in the phenomenon we proposed. To the best of our knowledge, the commonly used methods for selecting starting points for DPO are greedy. The conclusion we proposed in L268 may serve as a contribution to correcting misconceptions when selecting the starting point for DPO. Our explanation for this is that these SFT checkpoints may not be the best ones for performing a specific task, but they maintain better diversity in responses, i.e., higher entropy. This makes these SFT checkpoints better starting points

More new insights beyond the small learning rate provided by theoretical analysis.

Your insight about empirically using small learning rates to stabilize fine-tuning during post-training is very perceptive. Our theoretical analysis obtains the same methods from a KL perspective, which can serve as a theoretical explanation. Beyond this, we also proposed a series of imitation learning objectives that maintain KL properties in Section 4.3, analyzed the implicit reward properties during SFT in Section 4.4, and attempted to answer the role of SFT in post-training in Section 4.5. We hope these new insights from theoretical analysis can also excite you!

About the unified algorithm combining SFT and DPO

The failure to combine the SFT and DPO processes into one process is regrettable. However, we have made some progress toward this goal in this work, as we have obtained some theoretical guarantees. There are many possible reasons for the failure of multi-objective optimization, and we believe the most likely cause is data quality. We leave this to future work and hope our progress can inspire subsequent insights.

About the average direction in Figure 1

You are absolutely correct. Here, "average direction" means that, assuming there are four data points in a batch, as shown in Figure 1 of our paper, these four data points contribute equally to the estimation of the optimization direction, that is, to the estimation of the reward gradient.

Some more theoretical proof of ORPO

Your questions about other methods are interesting! We provide some extended theoretical analysis here about ORPO. The original optimization objective of ORPO is written as: $L = L_\text{SFT} + \lambda L_{OR}$ The first part is the classic SFT loss, which can be transformed into a reward learning process through the same approach. The second part can be expanded and written as: $L_\text{OR} = \log \sigma(\log \frac{\frac{P(y_w|x)}{1-P(y_w|x)}}{\frac{P(y_l|x)}{1-P(y_l|x)}} )$ Consider the term inside the $\log \sigma$ $\log \frac{\frac{P(y_w|x)}{1-P(y_w|x)}}{\frac{P(y_l|x)}{1-P(y_l|x)}}$ $= \log \frac{P(y_w|x)}{1-P(y_w|x)} - \log \frac{P(y_l|x)}{1-P(y_l|x)}$ $= \log \frac{P(y_w|x)}{P_\text{ref}(y_w|x)} - \log \frac{P(y_w|x)}{P_\text{ref}(y_l|x)} + \log \frac{1-P(y_l|x)}{1-P(y_w|x)} + \log \frac{P_\text{ref}(y_w|x)}{P_\text{ref}(y_l|x)}$ $= \log \frac{P(y_w|x)}{P_\text{ref}(y_w|x)} - \log \frac{P(y_w|x)}{P_\text{ref}(y_l|x)} + \log \frac{\Sigma_yP(y|x) - P(y_l|x)}{\Sigma_y P(y|x) - P(y_w|x)}+ \log \frac{P_\text{ref}(y_w|x)}{P_\text{ref}(y_l|x)}$ $= \log \frac{P(y_w|x)}{P_\text{ref}(y_w|x)} - \log \frac{P(y_w|x)}{P_\text{ref}(y_l|x)} + \log \frac{\Sigma_y\frac{P_\text{ref}(y|x)}{P_\text{ref}(y_l|x)}\frac{P(y|x)}{P_\text{ref}(y|x)} - \frac{P(y_l|x)}{P_\text{ref}(y_l|x)}}{\Sigma_y \frac{P_\text{ref}(y|x)}{P_\text{ref}(y_w|x)}\frac{P(y|x)}{P_\text{ref}(y|x)} - \frac{P(y_w|x)}{P_\text{ref}(y_w|x)}}$ $=r_w - r_l + \log \frac{\Sigma_y\frac{P_\text{ref}(y|x)}{P_\text{ref}(y_l|x)}\exp(r(x, y)) - \exp(r(x, y_l))}{\Sigma_y \frac{P_\text{ref}(y|x)}{P_\text{ref}(y_w|x)}\exp(r(x, y)) - \exp(r(x, y_w))}$

The $r_w - r_l$ is the classic BT model, aiming to widen the difference in reward between the chosen sample and the rejected sample. The latter part takes the total amount of reward into account. After dividing both the numerator and denominator by $\Sigma_y P_\text{ref}(y|x) \exp(r(x, y))$, we obtain: $r_w - r_l + \log \frac{\frac{1}{P_\text{ref}(y_l|x)} - \frac{\exp(r(x, y_l))}{\Sigma_y P_\text{ref}(y|x) \exp(r(x, y))}}{\frac{1}{P_\text{ref}(y_w|x)} - \frac{\exp(r(x, y_l))}{\Sigma_y P_\text{ref}(y|x) \exp(r(x, y))}}$ $=r_w -r_l + lop \frac{c_1 - \text{ratio of } r_l}{c_2 - \text{ratio of } r_w}$ This can be viewed as a correction term that represents, beyond increasing the reward difference between win and loss samples, further consideration of the total reward allocation for other possible responses. It considers the relationship between the current sample's reward allocation and the total reward allocation, requiring that the win portion accounts for a higher proportion of the total reward allocation than the loss portion, thereby further constraining the reward learning process.

Choosing KL-SFT checkpoint as the reference model in DPO is better

You reach an absolutely correct conclusion about the reference model selection during the DPO process. To the best of our knowledge, current DPO implementations have already begun using the model after the SFT process as the reference model instead of the base model. In our work, we further showed that the KL-constrained SFT checkpoint is a better choice for the starting point of DPO. This phenomenon maintains high consistency with our theoretical analysis.

We hope the above response finds you well. Thank you again for your time for our work.

We apologize for the interruption. We would like to know if our reply has addressed your questions or if you need any further clarification from us.