Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation

We sincerely appreciate your thoughtful critiques. Below, we respond to each of your concerns.

---

Comment 1: Novelty and Contributions

We are grateful for the opportunity to elaborate on the innovations and empirical findings introduced in our study. While we acknowledge that DPO and its iterative variants have been previously explored, our work introduces several novel insights and empirical contributions that meaningfully extend the current literature:

• Iterative Co-Evolution of Generator and Verifier
  We propose a multi-round training framework where the generator and verifier (PRM) are jointly optimized. Unlike prior works that fix the reward model, we explicitly demonstrate mutual enhancement across iterations, revealing the benefits of feedback-driven co-evolution, which has not been systematically explored in earlier DPO studies.

• Resource-Efficient Verifiable Preference Optimization
  Our work focuses on multi-round DPO with verifiable rewards, showing that it can rapidly enhance reasoning ability in Qwen2.5-series models using limited computational resources. While the benefits of RL in LLMs have been recognized since the release of ChatGPT in 2022, it was not until the emergence of DeepSeek-R1 in 2025 that its effectiveness on verifiable reasoning tasks was demonstrated at scale. We empirically show that DPO can achieve comparable improvements in verifiable reasoning with significantly lower cost, offering a practical alternative to RL-based pipelines.

• Comprehensive Empirical Study Across Strong Baselines
  We benchmark against strong baselines including Simple-RL-Zero, PURE-VR, LIMO, and S1. During rebuttal, we further expanded experiments to include:

  • Generalization to code and RC tasks,
  • Robustness under label noise,
  • Ablations on sampling temperature and training dynamics.

These analyses provide a thorough and well-rounded evaluation of the DPO-VP framework.

---

Comment 2: Comparison with Online RL on the Same Data

Thank you for raising this important point. We clarify that all our main experiments are conducted on the same 8K MATH training dataset, following the setup of SimpleRL-Zero [1].

In Table 2, models such as Qwen2.5-7B-Simple-RL-Zero and PURE-VR are all trained on the same data as our DPO variants.

To ensure completeness and up-to-date comparison, we downloaded and evaluated the latest official checkpoints of Simple-RL-Zero and Simple-RL-Zoo, both trained with PPO-style RL from reasoning trajectories. We also implemented and evaluated a GRPO-Zero variant using the same training data and base model. These are compared to our DPO-VP model under identical evaluation settings. The results are:

These results show that DPO-VP achieves competitive or superior performance compared to RL-based methods, while incurring significantly lower training cost.

[1] https://github.com/hkust-nlp/simpleRL-reason

---

Comment 3: Self-Reflection Analysis

We agree that “Let’s” alone may not fully capture self-reflective behavior. We initially used it as a proxy for explicit reasoning, as it's prevalent in mathematical CoT prompting.

To address your concern, we extended the analysis to include “wait”, “double-check”, and “re-evaluate”:

• “Wait” and “double-check” were rarely observed, likely because such introspective cues are not prominent in the Qwen2.5 base model, which lacks long-chain reflective training. These may be more common in R1-style models distilled from long-chain traces.
• We did observe “re-evaluate”, whose usage closely mirrors “Let’s” — appearing more in harder examples and correlating with correctness. However, its overall count is limited, reducing its statistical utility.

---

General Response Summary

Thank you again for your constructive feedback. Your comments helped clarify our key contributions in terms of:

• Methodological novelty via generator–verifier co-evolution and comprehensive empirical investigation,
• Efficiency over online RL methods, validated through consistent-scale comparisons,
• Broadened self-reflection analysis using multiple linguistic markers.

We will revise the camera-ready version to include these findings and provide clearer comparisons to related works you cited, highlighting both conceptual and empirical distinctions.

We hope this response addresses your concerns and would be grateful if you would consider a more favorable score in light of the clarified contributions and new evidence provided.

Thank you again for your thoughtful feedback on our submission. We have carefully addressed your comments during the rebuttal phase, including clarifications and additional experiments tailored to your suggestions.

We understand this is a busy review period, but we would greatly appreciate it if you could take a moment to review our responses and, if possible, share any further thoughts or updates to your evaluation.

Your input is extremely valuable to us. Thank you again for your time and consideration!

Dear Reviewer AnMH,

We sincerely appreciate the time and effort you've already dedicated to reviewing our work. As the rebuttal phase is nearing its end, we would be truly grateful if you could take a moment to review our responses and the additional experiments we conducted based on your comments.

If you have any further suggestions or feedback, we would be very happy to discuss them.

Thank you again for your time and consideration! We kindly ask for your feedback at your earliest convenience, as the rebuttal deadline is approaching.

We sincerely thank you for your valuable feedback. Below, we address each of your main concerns.

---

Comment 1: Novelty and Contributions

We appreciate the opportunity to clarify the key innovations and empirical findings of our study. Our work offers several new insights that meaningfully extend existing literature:

• Iterative Co-Evolution of Generator and Verifier

  We introduce a multi-round framework where the generator and verifier (PRM) are jointly optimized. Unlike prior works with fixed reward models, we explicitly demonstrate their mutual enhancement across iterations, highlighting the value of feedback-driven co-evolution, it is a direction not systematically studied in earlier DPO research.

• Resource-Efficient Preference Optimization

  We show that multi-round DPO with verifiable rewards can effectively enhance reasoning in Qwen2.5 models with low computational cost. While RL's value for LLMs has been known since ChatGPT (2022), its efficacy in verifiable reasoning only became clear with DeepSeek-R1 (2025). Our findings show that DPO can match such gains at a fraction of the cost, offering a practical RL alternative.

• Extensive Empirical Evaluation

  We compare against strong baselines like Simple-RL-Zero, PURE-VR, LIMO, and S1. During rebuttal, we further extended our study with:

  • Generalization to code and RC tasks,
  • Robustness tests under label noise,
  • Ablations on temperature and training dynamics.

These results provide a comprehensive and up-to-date evaluation of the DPO-VP framework.

---

Comment 2: Generality

We acknowledge that our primary experiments center on mathematical reasoning. To address concerns about generality, we conducted additional evaluations on non-math tasks, including code, scientific reasoning, and reading comprehension.

The results are summarized below:

These results demonstrate that DPO-VP though trained on math data, exhibits strong generalization to non-math reasoning domains, often outperforming strong baselines.

We will clarify this in the camera-ready version to ensure our reasoning claims are properly scoped and not misleading.

---

Comment 3: Concerns about "Coarse" Preference Filtering

Thank you for raising this important point. We clarify that our so-called “coarse” methods are primarily used for trajectory selection, not for collecting external data. As such, our pipeline retains control over data consistency and distribution, even if the filtering logic is simple.

To assess whether such filtering might degrade preference quality, we conducted a controlled label noise ablation under the PRM with offset setting in Table 1. Specifically, for each positive-negative response pair, we applied a random flipping probability, reversing the preference direction with probability p ∈ [0, 1]. The results are shown below:

These results suggest that while DPO is robust to mild noise, its performance degrades under high label corruption, validating the importance of having a reasonably accurate verifier signal. In our case, even coarse filtering based on verifier scores appears sufficient to maintain high-quality training supervision.

---

Comment 4: Impact of Dataset Quality on SFT and Self-Improvement

Thank you for the question. In our self-improvement pipeline, we only retain the questions from those datasets and generate new responses by the model itself. These self-generated responses are then verified and filtered before use.

Therefore, while the original SFT data quality is low, it does not affect our self-improvement pipeline, which is built on newly generated and verified content.

---

General Response Summary

Thank you again for your constructive feedback. Your comments helped clarify our key contributions in terms of:

• Methodological novelty via generator–verifier co-evolution and comprehensive empirical investigation,
• Extend our evaluation to non-math tasks, supporting the claim of general reasoning enhancement,
• Explain the role and limitation of coarse filtering, and the decoupling between SFT data and self-improvement quality.

We will revise the writing in the camera-ready version to reflect these points, cite all the missing related works, and differentiate our framework accordingly.

We hope our clarifications address your concerns and would appreciate your consideration for a more favorable assessment.

Thank you again for your thoughtful feedback on our submission. We have carefully addressed your comments during the rebuttal phase, including clarifications and additional experiments tailored to your suggestions.

We understand this is a busy review period, but we would greatly appreciate it if you could take a moment to review our responses and, if possible, share any further thoughts or updates to your evaluation.

Your input is extremely valuable to us. Thank you again for your time and consideration!

Response to Comment 1: Domain Generalization

We thanks for the insightful question regarding generalization beyond mathematical reasoning.

In Appendix E, we evaluate our DPO-VP model on two non-math domains:

• CMATH (Chinese mathematical QA)
• MMLU-STEM (covering physics, chemistry, biology)

The results show that the model trained solely on math can still generalize to other reasoning-heavy domains.

Furthermore, we conducted additional experiments on the following benchmarks:

• HumanEval and LiveCodeBench for code generation
• RACE for reading comprehension

The results are summarized below:

DPO-VP shows consistent performance across multiple out-of-domain tasks.

In addition, we note that the DPO framework is not inherently math-specific. In fact, its original paper [1] demonstrated strong results on dialogue, summarization, and sentiment alignment tasks. We believe that as long as dense labels can be converted into binary preferences, our iterative DPO method can be directly applied to NLI, QA, and other tasks. Moreover, we think step-wise preference construction for multi-step reasoning is also feasible, though beyond the scope of this current study, just like the reference [2] shows.

[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.

[2] Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs. arXiv:2406.18629.

---

Response to Comment 2: Comparison to Strong SFT Baselines

We thank you for pointing out the relevance of recent SFT-based approaches like LIMO and S1. As these methods were originally evaluated only on 32B models, we reproduced them on our 7B base model to provide a fair and direct comparison.

Specifically, we fine-tuned Qwen2.5-Math-7B on the released datasets from LIMO and S1 for 15 epochs, and evaluated using a 32K context length, consistent with their original setup. In contrast, our DPO-VP model uses a maximum context length of only 2K tokens during both training and evaluation.

The results are shown below:

DPO-VP significantly outperforms S1 and LIMO across nearly all benchmarks.

This result suggests that while both LIMO and S1 yield moderate gains over the base model, their improvements are limited when applied to 7B models. One possible explanation is that SFT with small-scale, long-chain data may be less effective at smaller scales or the SFT data should be meticulously prepared for different models, whereas our iterative DPO framework achieves better alignment and reasoning performance via self-generated preference pairs for different models(Qwen2.5-3B/7B), even under resource-constrained settings.

---

Response to Comment 3: Label Noise Ablation & PRM Robustness

We appreciate your concern regarding the robustness of our DPO framework under noisy preference labels.

To investigate this, we conducted a controlled label noise ablation under the PRM with offset setting in Table 1. Specifically, for each positive-negative response pair, we applied a random flipping probability — referred to as noisy ratio — that reverses the preference direction with probability p ∈ [0, 1]. For example, a noisy level of 0.5 means fully random pairwise supervision, while 1.0 implies completely reversed preferences.

The results below show pass@1 accuracy after a single DPO round on GSM8K and MATH500 using Qwen2.5-7B:

These results confirm that model performance degrades as preference supervision becomes less accurate. While DPO is somewhat tolerant to mild noise (e.g., ≤0.2), higher noise levels lead to noticeable drops. This empirically validates the importance of maintaining reasonably accurate verifier signals for effective preference optimization.

Response to Comment 4: Lack of Explicit Exploration Mechanism

We agree with you that exploration may play an important role in iterative self-improvement.

In our early attempts, we applied fixed sampling temperature (t = 0.7) across all DPO rounds. However, we observed performance saturation after the third epoch. To introduce greater sampling diversity, we increased the temperature in later epochs: t = 1.0 for epochs 4–5 and t = 1.2 for epoch 6-7.

This approach increased response diversity in later rounds and yielded substantial gains. The results below compare fixed vs. increased temperature on the average performance across five benchmarks:

These results demonstrate that even simple temperature annealing can enhance DPO by encouraging more diverse preference pairs, leading to stronger downstream performance.

While our method does not include formal entropy-based or diversity-penalized objectives, we believe that lightweight exploration strategies can meaningfully complement preference-based training. We acknowledge this as a valuable direction for future work, particularly inspired by recent and cutting-edge advances in exploration mechanisms within reinforcement learning [2–4].

[2] Entropy-based Exploration Conduction for Multi-step Reasoning. airxiv:2503.15848

[3] DRA-GRPO: Exploring Diversity-Aware Reward Adjustment for R1-Zero-Like Training of Large Language Models. airxiv:2505.09655

[4] DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management. arXiv:2505.12951

---

General Response Summary

Thank you for the insightful and valuable comments. In response, we conducted a series of new experiments and ablation studies to thoroughly address each concern:

• We demonstrate that our DPO framework generalizes well beyond math, with strong results on code generation (HumanEval, LiveCodeBench) and reading comprehension (RACE).
• We include direct comparisons with recent strong SFT baselines (LIMO, S1) under the same 7B setting, showing that DPO-VP outperforms them across multiple benchmarks. More critically, while strong performance can be achieved with limited SFT data, obtaining such effective data demands large-scale experimentation with meticulously selected samples for different models. On the contrary, DPO-VP makes it easier via self-generated preference pairs.
• We provide a detailed label noise ablation to verify the robustness of our framework under noisy preference signals.
• We explore lightweight sampling-based exploration strategies, and show that performance can be further improved via simple temperature scaling.

We appreciate your suggestions, which have helped us further validate the scalability, robustness, and extensibility of our approach. In the camera-ready version, we will integrate the additional experimental results and expand our discussions accordingly.

We hope our response has satisfactorily clarified the key points, and we kindly ask you to consider a more favorable score based on the additional evidence provided.

Thank you again for your thoughtful feedback on our submission. We have carefully addressed your comments during the rebuttal phase, including clarifications and additional experiments tailored to your suggestions.

We understand this is a busy review period, but we would greatly appreciate it if you could take a moment to review our responses and, if possible, share any further thoughts or updates to your evaluation.

Your input is extremely valuable to us. Thank you again for your time and consideration!

We thank the reviewer for the thoughtful critiques. We address each concern below.

---

Comment 1: Novelty and Contributions

We are grateful for the opportunity to elaborate on the innovations and empirical findings introduced in our study. While we acknowledge that DPO and its iterative variants have been previously explored, our work introduces several novel insights and empirical contributions that meaningfully extend the current literature:

• Iterative Co-Evolution of Generator and Verifier
  We propose a multi-round training framework where the generator and verifier (PRM) are jointly optimized. Unlike prior works that fix the reward model, we explicitly demonstrate mutual enhancement across iterations, revealing the benefits of feedback-driven co-evolution, which has not been systematically explored in earlier DPO studies.

• Resource-Efficient Verifiable Preference Optimization
  Our work focuses on multi-round DPO with verifiable rewards, showing that it can rapidly enhance reasoning ability in Qwen2.5-series models using limited computational resources. While the benefits of RL in LLMs have been recognized since the release of ChatGPT in 2022, it was not until the emergence of DeepSeek-R1 in 2025 that its effectiveness on verifiable reasoning tasks was demonstrated at scale. We empirically show that DPO can achieve comparable improvements in verifiable reasoning with significantly lower cost, offering a practical alternative to RL-based pipelines.

• Comprehensive Empirical Study Across Strong Baselines
  We benchmark against strong baselines including Simple-RL-Zero, PURE-VR, LIMO, and S1. During rebuttal, we further expanded experiments to include:

  • Generalization to code and RC tasks,
  • Robustness under label noise,
  • Ablations on sampling temperature and training dynamics.

These analyses provide a thorough and well-rounded evaluation of the DPO-VP framework.

---

Comment 2: Variance in Table 1 Results

We appreciate this suggestion. The performance difference may indeed fall within variance margins. In the camera-ready version, we will re-run key experiments and report averaged results over multiple seeds to improve statistical robustness.

That said, we emphasize that the main purpose of Table 1 is to demonstrate that even a single round of coarse filtering can substantially enhance the base Qwen model’s reasoning performance. We believe that a single-run result is sufficient to validate this clear performance gain and justify the effectiveness of our filtering strategy.

---

Comment 3: Justification for PRM Construction without Monte Carlo Unrolling

Our choice to train the PRM using outputs labeled by a stronger verifier (e.g., DeepSeek-V3) was primarily driven by efficiency and quality:

• It allows us to bootstrap a high-quality PRM without costly Monte Carlo unrolling or reinforcement learning.
• During DPO iterations, although the PRM architecture is fixed, it is continuously trained with new actor outputs, so its supervision signal gradually aligns better with the evolving generator—forming a form of implicit on-policy adaptation.

While Monte Carlo-based methods such as [2,3] are appealing, they often incur significantly higher computational costs and rely on fine-grained outcome annotations. In our setting, such annotations are not readily available and do not align well with the co-evolutionary training scheme of our generator and PRM, which emphasizes efficient and scalable online supervision.

---

General Response Summary

We sincerely thank the reviewer for the thoughtful and constructive feedback. Your comments have helped us further clarify the novelty and practical implications of our work. Specifically, we have:

• Clarified our core contribution as a systematic study of generator–verifier co-evolution, an aspect underexplored in prior DPO research;
• Reaffirmed the robustness and efficiency of our framework through new results on generalization, label noise tolerance, and ablations;
• Explained our choice of PRM training strategy and its advantages in terms of practical scalability and alignment with the evolving generator;
• Committed to re-running key experiments in Table 1 with multiple seeds in the final version to ensure statistical confidence.

We hope our response has sufficiently addressed your concerns and clarified both the novelty and empirical rigor of our work. If these clarifications and additional results help resolve earlier doubts, we would greatly appreciate your consideration of a more favorable evaluation.