Improving Rationality in the Reasoning Process of Language Models through Self-playing Game

Experimental Designs Or Analyses: While the paper shows improvements on GSM8K and MATH500, one limitation is the lack of explicit testing for generalization to other problem domains or tasks outside of mathematical reasoning. Weakness: While the paper reports significant improvements on tasks related to mathematical reasoning and self-correction, the focus on these domains means the broader applicability of the CDG method remains unclear.

We agree that evaluating generalization beyond GSM8K and MATH is important. To this end, we conduct additional experiments on tasks outside the original training distribution, including diverse reasoning benchmarks (LogiQA, ARC-Easy, and ARC-Challenge) and more difficult mathematical problems (Minerva-MATH). Results are shown below:

These results show that CDG yields consistent gains across different domains, suggesting it enhances general reasoning capabilities. We also train CDG on Dapo-17k [1], a harder dataset. Results on the challenge AIME-24 test set are shown below:

We average over 10 runs due to the small test size. CDG-3 improves performance from 1.67 to 5.67 using only 18k self-generated examples. For comparison, Qwen2.5-1.5B-MATH-Instruct reaches 10.0 after 2.5M CoT-annotated examples and GRPO. This highlights CDG’s data efficiency and effectiveness on low-performing tasks.

Weakness The paper does not provide an extensive discussion on the sensitivity of the model’s performance to hyperparameters (e.g., the thresholds used in the RL setup).

The ReST approach involves very few hyperparameters and does not require a reference or critic model. Unlike PPO or DPO, where improper values of $\beta$ can lead to training collapse (e.g., duplicate or collapsed generations), ReST does not suffer from such instability. As for selecting the threshold $\tau$, we choose it based on data balance considerations and observed performance during training.

Weakness The training process involving multiple self-play rounds and critics is resource-intensive. While the paper demonstrates the effectiveness of CDG, it does not provide a comprehensive analysis of the computational cost or the training time involved. It would be helpful to include a comparison of the cost-benefit tradeoff between CDG and other methods, such as instruction tuning or preference optimization.

We appreciate the reviewer’s concern regarding computational efficiency. We compare our method with traditional RL approaches such as Expert Iteration, from both inference (rollout) and training perspectives.

During the rollout stage, the prover first generate an initial solution. Based on this solution, the critic then generates multiple critiques. The prover subsequently produces multiple second-round responses conditioned on these critiques. The overall rollout cost increases by approximately 20% compared to Expert Iteration to produce the same amount of training data.

During training, the additional computational cost compared to Expert Iteration comes from training the critic. The amount of training tokens for the critic is about 25% of that for the prover.

Overall, while our approach involves more LLM instances, the practical computational overhead remains moderate due to reuse of generated content. In addition, our training does not require the reference model and critic model commonly used in many RL algorithms.

In the experiment for Table 3, for Expert Iteration, we report the best result under equal training budgets. For Step-DPO, we follow the original 3-round setup using 10k GPT-4o-annotated step-level preferences data.

Other Comments Or Suggestions: While the experiments are well-conducted, some aspects of the methodology (e.g., dataset creation, sampling strategies) are not presented in full detail. For example, the paper mentions the use of regular expressions and the SymPy grader for evaluating solutions, but further clarification would improve transparency.

Many details of our dataset construction, sampling strategies, and hyperparameter settings are included in the appendix due to space limitations. Using the SymPy grader to evaluate mathematical answers is a common practice; we follow the same evaluation setup as in qwen2.5-math-instruct [2]. We will also release the detailed evaluation code.

References: [1] Yu Q, Zhang Z, Zhu R, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale[J]. arXiv preprint arXiv:2503.14476, 2025. [2] Yang A, Zhang B, Hui B, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement[J]. arXiv preprint arXiv:2409.12122, 2024.

Thanks for your constructive feedback on our paper. Our response to your questions is as follows:

Claims And Evidence: One limitation is that the mechanism by which CDG improves understanding of reasoning processes is somewhat indirect. It's unclear that if LLMs are truly doing the reasoning and verifications or just pretending to do so.

Indeed, it is difficult to directly assess whether a model reasons in a rational manner. To address this, we evaluate performance across multiple tasks and dimensions to better capture whether the model reasons more rationally.

Our experimental setup is specifically designed for this purpose. For example, the error detection task uses problems the model already solves correctly. Just as human experts who understand a solution can identify and correct their own occasional mistakes, our setup tests whether the model can do the same. Failure in such cases indicates a lack of understanding of its own reasoning rather than a general capability gap.

Methods And Evaluation Criteria: Outdated and easy dataset (GSM8k) and data sizes. The author should try to exploit more challenging mathematical reasoning dataset such as AIME.

Yes, with the rapid progress of reasoning models, GSM8K and MATH are no longer sufficient for evaluating mathematical capabilities. To address this, we train CDG on Dapo-17k [1], a more challenging dataset. The results on the AIME-24 test set are shown below:

We average the results over 10 runs due to the small test size. CDG-3 improves performance from 1.67 to 5.67 using only 18k self-generated examples. In comparison, Qwen2.5-1.5B-MATH-Instruct reaches 10.0 after training on 2.5M CoT-annotated examples with GRPO. This highlights the data efficiency of CDG and its effectiveness in improving initially low-performing tasks.

We further evaluate the CDG-trained models in the paper on harder math tasks (Minerva-MATH) and diverse reasoning benchmarks (LogiQA, ARC-Easy/Challenge), demonstrating the generality of our method:

Experimental Designs Or Analyses: The computational overhead (efficiency) problems should be discussed when comparing with other RL methods since the use of multiple LLM instances.

We appreciate the reviewer’s concern regarding computational efficiency. We compare our method with traditional RL approaches such as Expert Iteration, from both inference (rollout) and training perspectives.

During the rollout stage, the prover first generate an initial solution. Based on this solution, the critic then generates multiple critiques. The prover subsequently produces multiple second-round responses conditioned on these critiques. The overall rollout cost increases by approximately 20% compared to Expert Iteration to produce the same amount of training data.

During training, the additional computational cost compared to Expert Iteration comes from training the critic. The amount of training tokens for the critic is about 25% of that for the prover.

Overall, while our approach involves more LLM instances, the practical computational overhead remains moderate due to reuse of generated content. In addition, our training does not require the reference model and critic model commonly used in many RL algorithms.

Questions For Authors

A1: Yes, as noted in Methods and Evaluation Criteria, our method significantly improves performance on AIME24, a much more challenging dataset. The average score increased from 1.67 (pre-trained) to 5.67 (CDG-trained), demonstrating stronger capabilities on complex problems. Additionally, Section 4.4 shows clear gains on long-chain CoT tasks with average reasoning lengths exceeding 1000 tokens.

A2: The prompt template for determining whether a critique is helpful or misleading is detailed in the appendix. As specified, if the model deems the critic helpful, it outputs the corrected answer in the boxed section; otherwise, it returns “This critic is not critical.” Rewards are applied accordingly using a rule-based scheme.

A3: Beyond performance, we observe increased self-checking behavior. For instance, the usage of the word “check” increases by 22%, and “verify” by 69%, indicating enhanced internal verification.

A4: For the same question, we sample multiple solutions from the prover. In our setup, the helpful critic only engages with incorrect solutions—where the reasoning must contain errors—while the misleading critic targets correct solutions, attempting to induce mistakes.

References: [1] Yu Q, Zhang Z, Zhu R, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale[J]. arXiv preprint arXiv:2503.14476, 2025.

Thanks for your constructive feedback on our paper. Our responses to your questions are as follows:

Claims And Evidence: The paper claims to improve "rationality in reasoning" but doesn't directly measure this construct beyond task performance.

As mentioned in the introduction, recent studies suggest that LLMs often rely on pattern matching rather than truly understanding their reasoning processes. However, directly assessing whether a model reasons in a rational manner is inherently difficult. To address this, we evaluate performance across multiple tasks and dimensions to better capture whether the model is reasoning more rationally.

Our experimental setup is specifically designed for this purpose. For example, the error detection task is conduct on problems the model has already demonstrated proficiency in solving. Just as human experts who understand a solution can identify and correct their own occasional mistakes, our setup tests whether the model can do the same. Failure in such cases indicates a lack of understanding of its own reasoning, rather than a general capability gap.

Claims And Evidence: The improvements, while consistent, are sometimes modest; the long-term stability of the improvements and potential for overfitting to the game format, given that GSM8K and MATH are somewhat similar tasks. Methods And Evaluation Criteria: Evaluation criteria suffer from limited scope.

We agree that evaluating generalization beyond GSM8K and MATH is important. To this end, we conduct additional experiments on tasks outside the original training distribution, including diverse reasoning benchmarks (LogiQA, ARC-Easy, and ARC-Challenge) and harder mathematical problems (Minerva-MATH). Results are shown below:

These results show that CDG yields consistent gains across different domains, suggesting it enhances general reasoning capabilities.

We further evaluate CDG on more challenging mathematical problems where the base model performs poorly. Specifically, we train CDG on the Dapo-17k dataset [1] and test on the AIME-24benchmark. Results are averaged over 10 runs due to the small test size:

CDG-3 improves Pass@1 from 1.67 to 5.67 using just 18k self-generated examples. In contrast, Qwen2.5-1.5B-MATH-Instruct reaches 10.0 after training on 2.5M CoT-annotated samples with GRPO. This demonstrates CDG’s data efficiency and strong gains.

Experimental Designs Or Analyses: There's insufficient reporting of statistical significance (e.g., standard errors); and the paper lacks proper controls for training computation across compared methods.

To assess statistical significance, we conduct t-tests and report p-values in the caption of Table 1. All four tasks show significant improvements (p < 0.05).

For Expert Iteration, we report the best result under an equal training budget. For Step-DPO, we follow the original 3-round setup using 10k GPT-4o-annotated step-level preference data.

Essential References Not Discussed: There are two particularly essential related works not adequately cited or discussed. Besides, the authors should carefully address the similarities and differences between Kirchner et al. (2024).

Thank you for pointing this out. We acknowledge that Saunders et al. (2022) and McAleese et al. (2024) are relevant, as they also explore LLM-based self-critiquing and error detection. We will cite and discuss them in the revision.

As for Kirchner et al. (2024), while both approaches involve multi-agent interaction, the objectives differs. Their goal is to improve output legibility by training a helpful prover to outperform a sneaky one. In contrast, our focus is on enhancing the model’s understanding of its own reasoning by training a prover to distinguish helpful from misleading feedback—aiming for developing rational reasoning abilities than readability. We will clarify this distinction in the revision.

Questions For Authors: Do you use three separate models or a single model? What do you mean by “both before and after CDG training”?

Q1: We use three separate models during training. Prompting alone results in noticeable style differences between the helpful and misleading critics, making them too easy for the prover to distinguish.

Q2: We compare two settings: (1) finetuning the base model on long-chain reasoning data (distilled from QwQ-32B-Preview), and (2) finetuning the model after CDG training using the exact same long-chain reasoning data.

References: [1] Yu Q, Zhang Z, Zhu R, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale[J]. arXiv preprint arXiv:2503.14476, 2025.

We greatly appreciate your recognition of our idea and your constructive feedback. Here are our responses to your concerns:

Claims And Evidence: The goal of the technique is unclear and the evidence provided to support this is not super convincing. Experimental Designs Or Analyses: The step-wise error detection experiment is done on problems that the model can fully solve.

We discuss these two points together, as our experimental design is closely aligned with the goal of the technique. As mentioned in the introduction, recent studies have shown that LLMs lack a true understanding of their reasoning processes and instead rely primarily on probabilistic pattern matching. Our goal is to alleviate this limitation through self-play training. However, performance on a single task (e.g., math problem solving) is insufficient to determine whether a model truly understands its reasoning process. Therefore, we evaluate the model across multiple dimensions to better assess whether it reasons in a rational manner. As shown in Table 3, our method demonstrates clear and consistent improvements across all tasks, supporting our claim.

We now clarify the motivation behind the stepwise error detection setting. Human experts, when they truly understand a solution, can identify and fix their own mistakes when such mistakes occasionally occur. Similarly, we focus on problems the model can usually solve correctly to control for knowledge or capability gaps. If the model still fails to detect errors in such cases, it suggests a lack of understanding of its own reasoning process rather than a lack of ability. This design is well aligned with our stated goal. In contrast, Section 4.2 (self-correction) makes no such assumption and evaluates the model on general problems, where initial answers may or may not be correct—thus covering more realistic error-correction scenarios.

Methods And Evaluation Criteria: Evaluation is done only on two math tasks, namely gsm8k and MATH. I would be more convinced of the proposed technique if it was applied to tasks where the initial performance is quite low.

Yes, with the rapid progress in reasoning models, GSM8K and MATH may no longer sufficiently evaluate mathematical capabilities. To address this, we train CDG on Dapo-17k [1], a harder dataset. Results on the AIME-24 test set are shown below:

We average over 10 runs due to the small test size. CDG-3 improves performance from 1.67 to 5.67 using only 18K self-generated examples. For comparison, Qwen2.5-1.5B-MATH-Instruct achieves 10.0 after 2.5M CoT-annotated examples and GRPO. This highlights CDG’s data efficiency and effectiveness on low-performing tasks.

We further evaluate the CDG-trained models on harder math (Minerva-MATH) and diverse reasoning tasks (LogiQA, ARC-Easy/Challenge), demonstrating the generality of our method:

The helpful and misleading critics are discarded after training. Have the authors using the same model but with different roles?

Yes, this is a very meaningful point. In fact, our initial design of CDG used a single model to play different roles via prompting, but we encountered two key issues:

1. The model’s responses as helpful and misleading critics differ noticeably in style and length under different prompts, making it easy for the prover to distinguish between them.

2. Training a unified model to generate misleading critiques can cause unintended side effects, such as hallucinations during regular problem solving.

To avoid these issues, we adopt three separate models with non-shared parameters. This setup yields more stable and reliable gains in the prover’s reasoning ability.

How hard was hyperparameter tuning for the ReST approach? Could you elaborate on stability of the training with ReST?

The ReST approach involves very few hyperparameters and does not require a reference or critic model. Unlike PPO or DPO, where improper values of $\beta$ can lead to training collapse (e.g., duplicate or collapsed generations), ReST does not suffer from such instability. As for selecting the threshold $\tau$, we manually select it based on data balance and training performance.

References: [1] Yu Q, Zhang Z, Zhu R, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale[J]. arXiv preprint arXiv:2503.14476, 2025.