VeriThinker: Learning to Verify Makes Reasoning Model Efficient

Thanks for the thoughtful and valuable feedback. Please find our responses below, we hope they adequately address your concerns. If you have any further questions or need additional clarification, we’d be happy to discuss them with you. Thanks again for your time and efforts.

Q1: The paper mentions that the SFT baseline uses "synthesized concise CoT chains as targets", but I feel there's insufficient detail on how these concise CoTs were generated.

A: Thanks for the insightful comments. We would like to clarify this point.

To synthesize concise reasoning chains for the SFT baseline, we follow CoT-Valve [1] by merging the base and reasoning models at a 0.2:0.8 ratio to generate concise reasoning chains on PRM12K and GSM8K. Only correct CoT solutions are retained, and the average length of these concise chains is about 60% of the original. Both SFT and our VeriThinker method are finetuned using LoRA with same configuration.

To ensure a fair comparison, we also created a small verification dataset using only PRM12K and GSM8K problems and trained our SVFT method with this restricted data. As shown in Tables 1 and 2, our approach still outperforms SFT, especially on the more difficult AIME benchmark.

It’s also important to highlight that generating data for SFT is much more expensive than for our method. SFT relies on a merged long-CoT model to produce training data. In contrast, our method only needs a short-CoT model to generate solutions for verification, making the process much more efficient. For example, using the vLLM framework on the same 4 × A5000 GPUs, a 7B short-CoT model achieves 10× higher throughput than the merged long-CoT model used for SFT.

Table 1 Performance comparison with SFT on the same training dataset (R1-Distill-7B).

Table 2 Performance comparison with SFT on the same training dataset (R1-Distill-14B).

Reference

[1] CoT-Valve: Length-Compressible Chain-of-Thought Tuning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics.

Thanks for the insightful and valuable feedback. We have provided detailed responses below and hope they address your concerns. If you have any further questions or require additional clarification, we would be glad to continue the discussion. We really appreciate your time and consideration.

Q1: The models used in the experiments are very similar. All of them are DeepDeek-R1-Distill models.

A: Thanks for the valuable suggestions. To better demonstrate the effectiveness of our method on different LLMs, we also applied our proposed VeriThinker approach to Phi-4-mini-reasoning-3.8B [1] (produced by Microsoft), which has a different architecture and parameter size. As shown in Table 1, our method significantly reduces the token usage of Phi-4-mini-reasoning-3.8B while maintaining its original reasoning accuracy.

Table 1 Performance comparison on other LLM.

Q2: Very little is explained about the self-constructed dataset used for training the model.

A: Thanks for the thoughtful comments. We would like to clarify that all details of our dataset construction process are provided in Section B of appendix.pdf in the supplementary.zip submitted with our paper. We have confirmed that appendix.pdf is included in the supplementary.zip as downloaded from OpenReview. If you are unable to access it, this may be due to a technical issue on the platform, and we kindly suggest downloading the file again.

Our data construction pipeline includes 4 steps:

• Step 1: Problem Collection. We gather problems from the training sets of four mathematical datasets (PRM12K, GSM8K, LIMO, and Numina-Math) to ensure diversity in topics and difficulty. Specifically, we extract all problems from the training set of PRM12K, GSM8K, and LIMO. For Numina-Math, we only select problems whose solutions are integer-valued math-word-problems, simplifying correctness labeling.
• Step 2: CoT Solution Collection. We generate multiple short-CoT solutions for each problem using five small language models (Qwen-2.5-0.5B/1.5B/7B-Instruct, Qwen-Math-1.5B/7B-Instruct). These solutions are step-by-step and do not include any self-reflection.
• Step 3: Correctness Labeling. We label each solution as correct or incorrect by comparing its final answer to the ground truth, using the Hugging Face math_verify tool for automated labeling.
• Step 4: Verification Data Filtering and Selection. Initially, we discard problems where all five CoT solutions are uniformly correct or incorrect. Such problems lack informative training signals due to being either too trivial or excessively challenging, and this process also helps filter out inherently problematic data. Next, we apply a deduplication strategy using Qwen-Math-1.5B-Instruct as a reference model. Specifically, for problems correctly solved by the reference model, we retain its correct solutions along with incorrect solutions generated by the other models. Conversely, for problems incorrectly solved by the reference model, we retain its incorrect solutions and also incorrect solutions from the other models. This selection strategy ensures each problem contributes at least one correct and one incorrect CoT solution. This deduplication and selection process results in a fine-tuning dataset of 340K instances (150K correct and 190K incorrect solutions), formatted as shown in Appendix Figure 1

This pipeline is highly efficient, the full dataset can be generated in 48 hours using just eight A5000 GPUs.

We hope this clarifies our data construction process. Please refer to the appendix.pdf for full details.

Q3: It is not clear if the dataset was a different dataset with a different verification task, the results would have been similar. Is there any overlap between the datasets used for testing and the fine-tuning dataset?

A: Thanks for the thoughtful comments. We confirm that our training and test sets are completely separated, with no risk of data leakage. To further demonstrate the robustness of VeriThinker, we conducted two additional experiments.

First, we built a small CoT verification dataset using only PRM12K and GSM8K problems for training, and then evaluated the fine-tuned model on the much harder AIME24 and AIME25 datasets. As shown in Table 2, even with training data limited to PRM12K and GSM8K, our method effectively reduced token usage while maintaining or even improving accuracy on these more challenging benchmarks.

Table 2 Finetuned on PRM12K and GSM8K, evaluated on the much more difficult AIME Datasets

Secondly, to further illustrate this point, we evaluated our model (finetuned exclusively on mathematical problems) on the GPQA Diamond dataset. GPQA Diamond is a multiple-choice benchmark covering diverse disciplines (chemistry,physics, biology......) and significantly differs in both format and domain from our training data. As shown in Table 3, our method achieved substantial CoT compression while maintaining accuracy on this challenging benchmark.

Table 3 Cross-Domain Performance on GPQA-Diamond

The above results demonstrate that our method still achieves strong performance even when there is a domain gap between the training and test sets. This cross-domain generalization is also a key strength of our approach.

Q4: redundancy in paper:

A: Thanks for the valuable and insightful comments. We truly appreciate your feedback and will make sure to address this issue thoroughly in the next version of our paper. Your comments are very helpful in improving the quality of our work.

Reference

[1] "Phi-4-mini-reasoning: Exploring the limits of small reasoning language models in math." arXiv preprint arXiv:2504.21233 (2025).

Thanks for the thoughtful and valuable feedback. Please find our responses below, we hope they adequately address your concerns. If you have any further questions or need additional clarification, we’d be happy to discuss them with you. Thanks again for your time and efforts.

Q1: the rationale for using five models and the number of generations is not clearly justified. I would also like to see how the size of the verification dataset affects performance.

A: Thanks for the insightful comments. We would like to clarify this issue.

In our data construction procedure, we first generated CoT solutions for each problem using five different short-CoT models with greedy decoding (one solution per model per problem). Using multiple models of varying reasoning abilities allowed us to produce diverse reasoning paths for the same question, also ensuring almost every problem had both correct and incorrect solutions. Such diverse and contrastive data help the model robustly distinguish between correct and incorrect CoTs, thus benefiting efficient reasoning.

To validate this, we conducted two additional experiments:

• We trained the R1-Distill-7B model on a raw dataset (290k samples) generated using only one short-CoT model (Qwen-math-1.5b-instruct), instead of five.
• We also trained the same model on a smaller dataset (32k samples) constructed solely from PRM12K and GSM8K using our original pipeline.

As shown in Table 1:

• The model trained on the single-model raw dataset performed worse in both token reduction and accuracy retention compared to our carefully constructed five-model dataset (340k samples), confirming the effectiveness of our approach.
• The model trained on the smaller dataset achieved good accuracy but lower token reduction, highlighting the impact of dataset size on our method's performance.

Table 1 Performance comparison using different training data

Q2: the cost of short-cot data pairs generation.

A: Thanks for the valuable comments. Generating large-scale datasets typically relies on inference engines like vLLM [3], which support batch inference. However, inference speed in batch scenarios heavily depends on the longest sequence within each batch. For instance, the Qwen-Math-7B-instruct model produces CoT solutions for the PRM12K dataset with a maximum token length of only 1K tokens, making it 20x faster than the R1-Distill-7B model, which has a maximum token length of 16K tokens. Additionally, short-CoT sequences consume significantly less memory, enabling much larger batch sizes and further accelerating data generation.

To illustrate, using vLLM, our entire CoT verification dataset (initially 1.45M samples, 340K after filtering) was generated in just 48 hours on eight A5000 GPUs. These results clearly demonstrate the convenience and efficiency of our short-CoT data collection process.

Q3: there’s no guarantee that each CoT step is correct, even if the final answer is.

A: Thanks for the insightful comments. All our generated solutions are short-CoT, meaning only have one single reasoning path without any self-reflection or branching. As a result, if a mistake occurs at any step, the model cannot recover and will almost always produce an incorrect final answer.

To demonstrate this, we conducted an additional experiment: We randomly selected 50 correct short-CoT solutions from Qwen-Math-7B-instruct on PRM12K and introduced a calculation error at a random step in each chain. After using the Qwen-Math-7B-instruct model to continue reasoning from that step, every final answer was incorrect. This result clearly supports our point.

Q4: For baselines such as SFT, what data was used?

A: Thanks for the valuable comments. For the SFT baseline, we followed the CoT-Valve [1] to synthesize concise CoT chains: we merged the base and reasoning models at a 0.2:0.8 ratio, then sampled concise CoT responses on the PRM12K and GSM8K datasets. We selected the correct CoT responses as SFT data. These concise CoTs are about 60% the length of the original chains. Both SFT and our VeriThinker method are finetuned using LoRA with same configuration.

To ensure a fair comparison, we also constructed a small verification dataset using only PRM12K and GSM8K, and trained our SVFT method on this data. As shown in Table 2, our method consistently outperforms the SFT baseline, especially on the more challenging AIME datasets, where SFT methods always suffer from a significant drop in accuracy.

Importantly, generating concise CoT data for SFT is much more expensive than collecting verification data for our method

Table 2 Performance comparison with SFT on the same training dataset.

Q5: What does “comparable CoT compression for baselines” mean? Is there any way to control the level of compression in baselines?

A: Thanks for the comments. The compression rate of our VeriThinker cannot be controlled. However, the compression rate of the baseline SFT-based method can be controlled. For SFT, we control the compression rate by adjusting the length of the synthetic concise chains used as training data. The synthetic concise chains we used for SFT are approximately 60% of the original length, which closely matches our VeriThinker's compression rate. This allows us to compare both methods at comparable compression levels, ensuring a fair and meaningful accuracy evaluation.

Q6: Statistical significance is not reported, which is especially important for datasets like AIME 2024 and 2025.

A: Thanks for the valuable suggestions. For the AIME dataset, the accuracy we report is calculated by running inference on each question 16 times and then taking the average. In Table 3 below, we present both the mean accuracy and the standard deviation to demonstrate the statistical significance of our results.

Table 3 Statistical significance on AIME datasets

Q7: Any comparisons that match accuracy and compare compression rates instead?

A: Thanks for the comments. Since SFT-based CoT compression methods inevitably cause a significant drop in accuracy on challenging tasks (AIME) and our method can maintain the accuracy, we conduct an additional comparison with the advanced RL-based method AdaptThink[2] to better match accuracy and compare compression rates. As shown in Table 4, when both methods maintain high accuracy on AIME, our approach achieves lower token usage compared to AdaptThink.

Table 4 Comparing compression rates under similar accuracy

Reference

[1] CoT-Valve: Length-Compressible Chain-of-Thought Tuning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics.

[2] "Adaptthink: Reasoning models can learn when to think." arXiv preprint arXiv:2505.13417 (2025).

[3] "Efficient memory management for large language model serving with pagedattention." Proceedings of the 29th symposium on operating systems principles. 2023.

The authors have provided a rebuttal to your comments, and it's an important part of the review process to give their response careful consideration. Please take a moment to review their rebuttal and provide any follow-up comments. This will help ensure there’s sufficient time for discussion and any necessary follow-up.

Best regards,

AC

Thanks for the detailed rebuttal, and apologies for the delayed response due to some eye issues.

Regarding Q1, I believe the paper would be stronger if it were less sensitive to the dataset. If certain methods require a large amount of diverse reasoning data, is that really a desirable property?

For Q2-Q5, I’m still not fully convinced. The main issue is whether the comparison is fair for SFT. What if we also used different models for SFT, not just DeepSeek-R1, which produces long CoTs, and then applied RFT? Wouldn’t that also lead to lower generation costs? In Table 4, it seems that for the 7B model, SFT reduces token usage significantly in additional experiments.

In short, my concern is about fair and controlled settings. I think the paper would be clearer if you ran Verithinker in a minimal setting: generate synthetic data using the same model on the same dataset for both SFT and Verithinker, then plot a graph with the x-axis as token length and y-axis as accuracy for SFT, and show Verithinker as a point on the same graph (like a Pareto frontier). Right now, the mixture of different datasets and settings makes it harder to clearly interpret the effectiveness of the method.

That said, I understand this may not be something that can be fully addressed in the rebuttal phase. I will likely maintain my score. Still, I consider this a borderline accept, as the focus on verification is quite interesting.

Thanks for the insightful and valuable feedback. We have provided detailed responses below and hope they address your concerns. If you have any further questions or require additional clarification, we would be glad to continue the discussion. We really appreciate your time and consideration.

Q1: The statement of "The prevalence of overthinking stems from insufficient accuracy in this binary classification task." is not supported. I am very skeptical of this claim, as many reasoning models are well capable of judging the correctness of CoTs.

A: Thanks for the insightful comments. Prior work [1,2,3] has also found that reasoning models often struggle to judge the correctness of their own CoT outputs, frequently triggering unnecessary self-reflection even when previous steps are correct. We further investigated this issue by analyzing R1-Distill-7B’s behavior. Specifically, we measured the probability of generating a "reflection" token (such as "Wait") after correct and incorrect reasoning steps on the MATH500 dataset.

As shown in Table 1, even after fully correct steps, large reasoning models initiate self-reflection more than 11% of the time. For incorrect steps, this probability jumps to 36.8%. After fine-tuning with our method, the reflection rate on correct steps drops by nearly threefold, while the rate for incorrect steps increases slightly. These results highlight the core problem and demonstrate how our approach effectively reduces unnecessary self-reflections without sacrificing reasoning accuracy.

Table 1 Reflection probability when facing correct and incorrect steps

Q2: It is unclear why "these methods synthesize a concise target chain approximating the optimal scenario where p(acc | h) ≈ 100%."

A: Thanks for the comments. We apologize for the confusion and would like to clarify this point.

In our paper, $p(\text{acc}|h)$ does not refer to the probability that the generated CoT is correct. Instead, it represents the probability that the model correctly identifies whether reflection is necessary at each reasoning step. Ideally, models should avoid reflecting on correct steps while always reflecting when encountering incorrect ones, corresponding to $p(\text{acc}|h) = 100\%$.

Previous CoT compression methods indirectly improved reflection accuracy by training models to imitate concise chains, which contain minimal, necessary reflections. In contrast, our method directly fine-tunes the model to explicitly distinguish correct from incorrect reasoning steps. This direct training significantly enhances the model’s judgment of when to reflect, as clearly supported by our experimental results in Table 1.

Q3: In Section 3.3, the authors find that, apart from Dataset (5), all other datasets result in maintained or slightly improved accuracy. This result is strange.

A: Thanks for the comments. We appreciate the chance to clarify this further.

In our SVFT method, the model is trained to discriminate between correct and incorrect CoT solutions. Importantly, the model does not know which group represents true correctness; it essentially learns to separate the two categories based on their features. The labels during training serve only as group identifiers and their inherent meaning does not matter. This is why, in Dataset (2), even if the labels are reversed, our method still works. In fact, as shown in paper section 3.3, our method performs just as well if we label the groups as "north" and "south" instead.

After fine-tuning, the model's parameters are adjusted to become more sensitive to tokens that differentiate correct and incorrect CoTs. As a result, when deciding whether to self-reflect during reasoning, the finetuned model can more accurately identify when reflection is truly needed. Consequently, the model can avoid unnecessary self-reflections while keeping the essential ones. Our experimental results in Table 1 also confirm this point.

Q4: The authors need to conduct an ablation study on the training data, as the current experimental setup raises concerns about whether the accuracy improvement comes from the model having seen the correct answers during training (even if not trained on them directly).

A: Thanks for the valuable comments. We conducted two additional experiments to clarify this issue.

First, we constructed a small CoT verification dataset using only questions from PRM12K and GSM8K, and fine-tuned our reasoning model on this restricted data. We then evaluated the fine-tuned model on the more challenging AIME datasets. As shown in Table 2, even with training data limited to PRM12K and GSM8K, our method significantly reduced token usage and maintained accuracy on the AIME benchmarks.

Table 2 Finetuned on PRM12K and GSM8K, evaluated on the much more difficult AIME Datasets

Secondly, to further illustrate this point, we evaluated our model (trained exclusively on mathematical problems) on the GPQA Diamond dataset. GPQA Diamond is a multiple-choice benchmark covering diverse disciplines (chemistry,physics, biology......) and significantly differs in both format and domain from our training data. As shown in Table 3, our method still significantly compressed the length of CoTs while maintaining accuracy.

Table 3 Cross-Domain Performance on GPQA-Diamond

The above results demonstrate that our method still achieves strong performance even when there is a domain gap between the training and test sets.

Q5: the authors need to add more baselines to compare how well a basic LLM performs on the correctness verification task.

A: Thanks for the comments. To further validate our approach, we used Qwen-2.5-1.5B-Instruct to generate CoT solutions on MATH500 and evaluated the verification accuracy of several models.

We compared our VeriThinker-7B with two strong open-source models (Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct) and two advanced closed-source models (OpenAI GPT-4o and GPT-4.1). As shown in Table 4, our method achieves substantially higher verification accuracy than the open-source models, and even outperforms the two large closed-source models from OpenAI.

Table 4 Comparison of CoT correctness verification capability

Q6: The experiments do not include comparisons with recent RL-based methods

A: Thanks for the valuable comments. As shown in Table 5, we compare VeriThinker with two recent RL-based methods (AdaptThink [4] and LC-R1 [5]), both of which use improved GRPO algorithms for CoT compression. VeriThinker outperforms AdaptThink in both token reduction and accuracy. While LC-R1 achieves a higher compression rate, VeriThinker delivers much better accuracy.

Importantly, RL-based methods require much higher computational resources due to the use of rollout strategies and full-model finetuning. For example, AdaptThink needs 32 H100 GPUs (80GB) for 28 hours of training, while VeriThinker achieves strong results with only 8 A5000 GPUs (24GB) and 14 hours of training.

Table 5 Comparison to recent RL-based methods

Q7: Typos and redundancy in paper:

A: Thanks for the thorough and thoughtful review. We are committed to addressing all of these issues comprehensively in the next version. We truly appreciate your valuable suggestions, which will certainly help us improve the quality of our paper.

Reference

[1] Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18468–18489

[2]"No free labels: Limitations of llm-as-a-judge without human grounding." arXiv preprint arXiv:2503.05061 (2025).

[3] "Do not think that much for 2+ 3=? on the overthinking of o1-like llms." arXiv preprint arXiv:2412.21187 (2024).

[4] "Adaptthink: Reasoning models can learn when to think." arXiv preprint arXiv:2505.13417 (2025).

[5] "Optimizing Length Compression in Large Reasoning Models." arXiv preprint arXiv:2506.14755 (2025).