Improving Reasoning Ability of Large Language Models via Iterative Uncertainty-based Preference Optimization

Dear Reviewer 6qtw,

Thank you for your valuable comments and thoughtful questions. We sincerely appreciate the time and effort you have invested in reviewing our work. We have implemented several key revisions in the draft in response to your and other reviewers’ constructive feedback. Below, we provide detailed responses to each of your comments to the best of our ability.

W1. Thanks for your suggestions. In fact, we conduct experiments with full parameter training at first, but we find that the trend is consistent with the LoRA training. Therefore, due to the resource limitations, we chose LoRA training finally.

W2. Since our method focuses on preference dataset construction and the failures of DPO in complex reasoning tasks, we have only compared it with a similar method DPOP. However, your comments make us realize that this is insufficient, thus we compare our method with more variant methods in Appendix C.1. The results can be also seen as follows:

We find that KTO outperforms the vanilla DPO method and gains performance from iteration but underperforms our IUPO method. We think that our approach is more suitable for these scenarios where the preferred and dispreferred examples have a small edit distance.

Additionally, we want to claim that the motivations and objectives of KTO and our IUPO are different. KTO applies prospect theory to directly maximize the utility of generations instead of maximizing the log-likelihood of preferences. In the specific implementation, KTO does not require paired preference data. In contrast, we introduce an approach to collect preference pairs through iterative sampling and execution feedback, and on the other hand, we propose an iterative uncertainty-based preference optimization method that addresses the failures of DPO in complex reasoning scenarios.

We highly appreciate your valuable input, and we are committed to integrating these insights into our upcoming experiments and the refined version of our work.

We hope these updates adequately address your concerns and kindly encourage you to reconsider our review score in light of these clarifications. Your contribution significantly enhances the quality of our research, and we thank you for your thoughtful feedback.

Response to Questions:

Q1. Iterations can be more than 3. We conduct the analysis experiments of the iterations and model performance in Figure 7. We find that an increase in the number of iterations gives diminishing improvements to the model. We selected 3 iterations because they can balance performance gain and computation well. Furthermore, other works[3][4] also come to similar conclusions as well as the choice of iteration numbers.

References:

[1] From r to Q∗: Your Language Model is Secretly a Q-Function

[2] PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases

[3] Self-Rewarding Language Models.

[4] Iterative Reasoning Preference Optimization.

Dear Reviewer yKKv,

Thank you for your valuable comments and thoughtful questions. We sincerely appreciate the time and effort you have invested in reviewing our work. We have implemented several key revisions in the draft in response to your and other reviewers’ constructive feedback. Below, we provide detailed responses to each of your comments to the best of our ability.

Response to Weakness:

W1. We are sorry for the confusion of our method. We have updated the equations in Section 3.3 in the revised draft version. We define the uncertainty measure \delta of an answer sequence $y$ as $\Delta(y) = \{\Delta(y^t)|t=1,...,|y|\}$, and $\Delta(y^t)$ refers to the uncertainty of token $t$ of $y$, defined as $\Delta(y^t) = p(y^t_1|y^{<t}, x) - p(y^t_2|y^{<t}, x) + \epsilon,\quad \Delta_t \in [\epsilon, 1]$. Then we revised probability of $\pi_\theta$ as follows: $\pi_{\theta,\Delta}(y|x)=\prod_t^{|y|}\pi_\theta(y^t|y^{<t},x)\cdot\Delta(y^t)$

W2. Thank you for your insightful questions and suggestions. Allow me to describe our approach with these two pieces of work:

• [1]: This work provides a theoretical extended analysis for DPO, and demonstrates that it is possible to reframe DPO in the token-level Markov Decision Process (MDP) which enables the credit assignment ability of the method. This work also studies the well-know issue of DPO training that the implicit chosen rewards decline during the raining process. However, this paper does not give a solution to alleviate this issue. Furthermore, although DPO can assign credit to the tokens in sequence, it is still affected by the token uncertainty of the model, especially in scenarios where chosen and rejected examples are minimal contrastive.

• PLUM [2]. PLUM proposes an on-policy preference learning framework that iterative optimizes the policy model via response sampling, which is similar to ours. However, the motivation and focus are different between PLUM and us. The main contribution of PLUM is the test cases automatic generation framework for code tasks to verify the generated responses and create preference data. The subsequent optimization algorithms used DPO and KTO with no changes. In contrast, we collect the preference data iterative sampling and execution feedback and consider the learning state of the policy model. In the code reasoning tasks, we use the data with test cases that do not require additional generation. We find the preference pairs collected in an iterative manner exhibit minimal contrast, and we propose the uncertainty-based preference optimization method to achieve fine-grained control and alleviate the failures of the DPO method as described in our paper.

W3. Thank you for your valuable suggestions. We have updated the comparison experiments between our IUPO, DPO, and KTO in Appendix C.1. The results can be seen below:

We find that KTO outperforms the vanilla DPO method and gains performance from iteration but underperforms our IUPO method. We think that our approach is more suitable for these scenarios where the preferred and dispreferred examples have a small edit distance.

W4. Thanks for your careful reading and helpful reminders. We have fixed the typos you mentioned in the revised draft.

We highly appreciate your valuable input, and we are committed to integrating these insights into our upcoming experiments and the refined version of our work.

We hope these updates adequately address your concerns and kindly encourage you to reconsider our review score in light of these clarifications. Your contribution significantly enhances the quality of our research, and we thank you for your thoughtful feedback.

Dear Reviewer mQmU,

Thank you for your valuable comments and thoughtful questions. We sincerely appreciate the time and effort you have invested in reviewing our work. We have implemented several key revisions in the draft in response to your and other reviewers’ constructive feedback. Below, we provide detailed responses to each of your comments to the best of our ability.

Response to Weakness:

W1. Thank you for your suggestions. We have updated the comparison experiments between our IUPO, DPO, and KTO in Appendix C.1. The results can be seen below:

We find that KTO outperforms the vanilla DPO method and gains performance from iteration but underperforms our IUPO method. We think that our approach is more suitable for these scenarios where the preferred and dispreferred examples have a small edit distance.

W2. We acknowledge that the training cost increases as the iterations increase since it brings more training data and training time. However, the preference data comes from the model itself and we do not introduce external information, thus the performance gains are small but meaningful. Moreover, our method still outperforms other methods with the same resource consumption, as shown in Table 4, where more iterations with updating data outperform one iteration with more data.

W3. Thanks for your careful reading and helpful reminders. We have fixed the typos you mentioned in the revised draft.

We highly appreciate your valuable input, and we are committed to integrating these insights into our upcoming experiments and the refined version of our work.

We hope these updates adequately address your concerns and kindly encourage you to reconsider our review score in light of these clarifications. Your contribution significantly enhances the quality of our research, and we thank you for your thoughtful feedback.

Response to Questions:

Q1. See Re W1.

Q2. We follow [3] to call the $\Delta(y)$ as uncertainty. The value of $\Delta(y)$ reflects the level of confidence the model has in predicting a token. As you said, the actual magnitude of this value is the opposite of what uncertainty means, but we are more concerned with uncertainty than the certainty of the models.

Q3. If the uncertainty measure of a token is below a fixed threshold, we will adjust the confidence level of the tokens in the subsequent window, which in turn affects the calculation of the subsequent loss.

Q4. We are sorry for the confusion. We have updated the equations in Section 3.3 in the revised draft version. The uncertainty measure \delta of an answer sequence $y$ can be defined as $\Delta(y) = \{\Delta(y^t)|t=1,...,|y|\}$, and $\Delta(y^t)$ refers to the uncertainty of token $t$ of $y$, defined as $\Delta(y^t) = p(y^t_1|y^{<t}, x) - p(y^t_2|y^{<t}, x) + \epsilon,\quad \Delta_t \in [\epsilon, 1]$.

Q5. Thanks for your careful reading and helpful reminders. We have fixed the typos you mentioned in the revised draft.

Q6. DPO training for the 70B model is resource intensive, due to the resource limitations, we only experimented on the Text-to-SQL task. Furthermore, we want to try whether the preference data collected with a small model iteratively would be applicable to large models, so we conducted experiments using only IUPO.

Q7. In Figure 5, we averaged the uncertainty measure per dataset at token granularity.

Q8. $\theta$ is the logits matrix output from language models, and \theta_j is the logits of the j-th token. Then \theta will be input to the softmax layer and we obtain the final word probabilities distribution.

Q9. The uncertainty threshold $\tau$ and the uncertainty window K are hyper-parameters in our paper. We set the τ = 0.3 and K = 5 based on the experimental results.

We highly appreciate your valuable input, and we are committed to integrating these insights into our upcoming experiments and the refined version of our work.

We hope these updates adequately address your concerns and kindly encourage you to reconsider our review score in light of these clarifications. Your contribution significantly enhances the quality of our research, and we thank you for your thoughtful feedback.

References:

[1] Self-Rewarding Language Models.

[2] Iterative Reasoning Preference Optimization.

[3] Chain-of-thought reasoning without prompting.

[4] Minimum-margin active learning.

[5] Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Dear Reviewer 9kF7,

Thank you for your valuable comments and thoughtful questions. We sincerely appreciate the time and effort you have invested in reviewing our work. We have implemented several key revisions in the draft in response to your and other reviewers’ constructive feedback. Below, we provide detailed responses to each of your comments to the best of our ability.

Response to Weaknesses:

W1. Measuring uncertainty in language models is a popular area of research, and can be used to assess the hallucinations of models. The common uncertainty measure methods are likelihood-based, prompt-based, sampling-based, training-based, etc. However, in our paper, we focus on token-level uncertainty. In our early experiments, we found the size of the probability that the model predicts a particular token positively correlates with the likelihood of making an error in the code or math reasoning scenarios. Thus we follow [3] and [4] to define the uncertainty of the token in the generating process and alleviate the issues of DPO by refining the loss based on the uncertainty measures.

W2. Our work focuses on complex reasoning tasks including mathematical and code reasoning where DPO is prone to problems and the preference pairs are minimal contrastive. In other reasoning tasks such as commonsense reasoning, we can use reward models to select the preferred and undesirable examples, and then apply the iterative framework to conduct a preference optimization process, just as other work has done [1] [2].

W3. We have described the limitations of DPO in complex reasoning tasks including code and mathematical reasoning in Section 1 and Section 3.2. Compared with DPO, we made many improvements to alleviate these issues. Firstly, we introduce a cheap and efficient preference data collection approach considering the learning state of the policy model. Secondly, we iteratively optimize the policy model based on the proposed framework, making DPO online to alleviate the distribution shift problem. Last, in response to the problem of DPO decreasing the probability of preferred examples, we propose the uncertainty-based preference optimization method and theoretically and experimentally demonstrate its effectiveness.

W4. We are sorry for the confusion. We have updated the following items in the revised draft version.

• We define the uncertainty measure \delta of an answer sequence $y$ as $\Delta(y) = \{\Delta(y^t)|t=1,...,|y|\}$, and $\Delta(y^t)$ refers to the uncertainty of token $t$ of $y$, defined as $\Delta(y^t) = p(y^t_1|y^{<t}, x) - p(y^t_2|y^{<t}, x) + \epsilon,\quad \Delta_t \in [\epsilon, 1]$.
• Sorry for the confusion. We have updated the equations in Section 3.
• We select the tokens with uncertainty measure below a fixed threshold τ and then modify the uncertainty values for the subsequent K tokens. The windows size K denotes the number of subsequent tokens to be modified, and the relative distance refers to how many token gaps there are between the token in the window and the selected token.
• In the formal analysis section, we explicitly state that we follow [5] and further extend it to our approach.
• For issues 1 and 2, our uncertainty-based preference method enables the optimization process with fine-grained token control. And our formal analysis and experimental results demonstrate the introduction of the uncertainty measure can alleviate the preferred probability decrease issue. For issue 3, we introduce IUPO and collect the preference data iteratively using the method proposed in Section 2, thus the IUPO method is an online method that the data and policy updated iteratively.

W5. In table 2, the phrase “IUPO-1” refers to DPO with the uncertainty-based method and without iterative training. We can demonstrate the role of the uncertainty introduce by comparing the performance between IUPO-1 and DPO.

Dear Reviewer oETz,

Thank you for your valuable comments and thoughtful questions. We sincerely appreciate the time and effort you have invested in reviewing our work. We have implemented several key revisions in the draft in response to your and other reviewers’ constructive feedback. Below, we provide detailed responses to each of your comments to the best of our ability.

Response to Weakness:

W1 & W2. The motivation of preference data collection approach. [1] and [2] propose a vanilla iterative preference optimization method via rewards computing and preference pairs selection. We also follow an iterative paradigm but with many improvements and differences in the implementation. Firstly, our method considers the learning state of the models by comparing the sampling responses of the naive model and the policy model. The results in Figure 9 demonstrate the effectiveness of this approach. Secondly, [1] and [2] apply a reward model to give rewards to the model’s responses while we simulate a virtual environment to execute synthetic responses in code and mathematical reasoning tasks.

The motivation of uncertainty-based preference optimization method. In fact, [1] uses the vanilla DPO method to optimize the policy while [2] found it yields moderate gains and even decreases the performance on reasoning tasks and then adds a negative log-likelihood term in DPO. In contrast, our IUPO method is self-constrained with our preferred data collection method, as well as considering the failures of DPOs in complex reasoning tasks. As shown in Table 1, the preference pairs collected in the iterative manner exhibit minimal contrast (low edit distance). However, the vanilla DPO is not applicable to this scenario since it may decrease the probabilities of both the undesirable and preferred [3][4]. Therefore, we propose the uncertainty-based preference optimization method to achieve fine-grained control and demonstrate it can alleviate the DPO issue in formal analysis and experimental results.

W3 & Q1. Thank you for your insightful questions. In my opinion, the iterative direct optimization method has several advantages over than policy gradient optimization method in some instances. One of the keys is direct optimization is more efficient and stable than the policy gradient method such as PPO, especially when the policy model has huge parameters. For example, the llama3 herd of models [5] also conducts iterative DPO training with rejection sampling, and they find that DPO requires less computation for large-scale models and performs better than PPO. [6] compares the online DPO and on-policy RLHF method and finds online DPO is more preferred than the method. Overall, although PPO and other policy gradient methods may theoretically be able to obtain better results, iterative/online DPO method training is efficient and the process is stable.

Response to Questions:

Q2. We are sorry for the confusion. Some of the icons in Figure 4 correspond to Figure 2. Circle 1 and circle N refer to the first and N-th responses generated in Figure 2 “Response Sampling” step. They are sampled in the same iteration, one is preferred and another is dis-preferred. We are sorry for the confusion, and we have refined Figure 2 and Figure 4 in the revised draft.

References:

[1] Self-Rewarding Language Models.

[2] Iterative Reasoning Preference Optimization.

[3] Smaug: Fixing failure modes of preference optimisation with dpo-positive.

[4] Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective.

[5] The Llama3 Herd of Models.

[6] Direct Language Model Alignment from Online AI Feedback

Dear Reviewer 6qtw,

Thank you for your valuable comments and thoughtful questions. We sincerely appreciate the time and effort you have invested in reviewing our work. We have implemented several key revisions in the draft in response to your and other reviewers’ constructive feedback. Below, we provide detailed responses to each of your comments to the best of our ability.

Response to Weaknesses:

W1. We use the answer extractor or executable feedback as verifiers in code or mathematical reasoning tasks since they can guarantee the correctness of the judgment and efficiency. For other reasoning tasks such as commonsense reasoning, we can use reward models to select the preferred and undesirable examples, and then apply the iterative framework to conduct a preference optimization process.

W2. The answer extractor is applied in the mathematical reasoning tasks. In this scenario, the responses of LLMs follow a fixed response format as shown in Table 7, i.e. the final part is “The answer is {answer}.” So the accuracy of the answer extractor can be ensured. However, in other scenarios where it uses the reward model as a verifier, the performance of the reward model affects the quality of the preference data and can have an impact on the preference optimization process.

W3. Thanks for your insightful suggestions. However, in our work, we focus on complex reasoning tasks including mathematic and code reasoning tasks, and the preference data only contains information about these scenarios. We do not think that models trained in this way can be generalized well to other scenarios.

Response to Questions:

Q1. The choice of iteration numbers comes mainly from experimental experience. In Section 4.4, we conduct some analysis experiments of the iterations such as model performance within various iterations. As shown in Figure 7, we find that an increase in the number of iterations gives diminishing improvements to the model. We selected 3 iterations because they can balance performance gain and computation well. Furthermore, other works[1][2] also come to similar conclusions as well as the choice of iteration numbers.

Q2. We acknowledge that the training cost increases as the iterations increase since it brings more training data and training time. However, the preference data comes from the model itself and we do not introduce external information, thus the performance gains are small but meaningful. Moreover, our method still outperforms other methods with the same resource consumption, as shown in Table 4, where more iterations with updating data outperform one iteration with more data.

We highly appreciate your valuable input, and we are committed to integrating these insights into our upcoming experiments and the refined version of our work.

We hope these updates adequately address your concerns and kindly encourage you to reconsider our review score in light of these clarifications. Your contribution significantly enhances the quality of our research, and we thank you for your thoughtful feedback.

References

[1] Self-Rewarding Language Models.

[2] Iterative Reasoning Preference Optimization.

Dear Reviewers and AC,

We would like to express our sincere gratitude for your invaluable time, thoughtful feedback, and constructive engagement throughout the review process. Your insights have significantly enhanced the quality of our work, and we are deeply appreciative of your efforts.

As the discussion period draws to a close very soon, we would like to gently remind Reviewers 6qtw, oETz, and D59P that further discussion and a re-review of our response. Your valuable insights contribute significantly to the refinement of our work, and we would be most appreciative of any updates to your ratings, should you feel it is appropriate.

Thank you very much for your time and consideration. We look forward to hearing from you soon.

Best regards,

Authors