Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?

More MiP-Formula Samples

Q: The number of samples in some categories is relatively small, which might limit the statistical power of the findings. For example, on the easiest dataset, the authors only provide 50 examples.

We further added 450 more samples for the MiP-Formula category and made it 500 samples in total for this category. The results are reported in the following table. More experimental results on more models will be included in the next version. The results are consistent with our findings reported in the main paper.

Trade-off between Identifying Unsolvable Cases and Not Giving Up Too Quickly

Q: One thing the authors can elaborate on and experiment with is the trade-off between identifying unsolvable cases and not giving up too quickly. It is not always clear to models (and humans) whether a problem is unsolvable. So, maybe you want to err on the side of caution and not give up too quickly?

We appreciate your discussion regarding the trade-off between identifying unsolvable problems and not giving up too quickly. Such trade-off is possible on hard negatives, i.e., those problems that are suspicious to be MiP but turn out to be solvable after cautious thinking. However, these hard negatives are out of the main scope of this study, as we mainly focus on easy positives for humans and how they can fool the intensely trained reasoning LLMs. As verified by the human evaluation result (please kindly refer to the table in our rebuttal to reviewer teYv), humans' high recall indicates that the MiP problems we crafted are indeed very trivial for humans. Therefore, such a trade-off does not exist for humans on our tasks. From the perspective of the model, since the goal is to evaluate the gap of models on human-level critical thinking ability, it should not perform such trade-offs on our problem sets.

Moreover, the results in Table 2 of our original paper also indicate that such trade-off does not exist on model responses of current models, as longer responses from models often lead to worse abstain rate, rather than a better one. This indicates that, across current models, not giving up early does not result in better identification of the unsolvable cases.

The Definition of Critical Thinking

Q: Suppose you sat for an exam where you are not timed, and one question seems unsolvable to you, but you are not sure. You can give up early within 1 minute, or try to solve it for 10 minutes. Either way, you are not penalized for how much time you spend on the problem. If you were a smart person with critical thinking, what is the rational choice to take? I think my choice would be to continue trying to solve it, since you are not penalized for the time you spend, and you are still unsure about whether the problem can be solved. The authors also mention that sometimes the models realise they are stuck in a loop, but continue to go on. This seems like the rational choice is to continue thinking if the length penalties are low. So it does not seem to be that this reflects a lack of critical thinking.

We appreciate your analogy of answering questions in an exam and the discussion about the definition of critical thinking. Indeed, if not penalized for how much time is spent on the problem, it's a solid and reasonable option to keep exploring it. However, in the MiP-Overthinking phenomenon, as discovered for current reasoning models, the "time" they spend is far from being "reasonable". For example, for a math problem in GSM8K, most of the non-reasoning models can solve it with less than 200 tokens, while the reasoning models spend 3500 to 4000 tokens for the MiP counterpart, which shows a 20x time spent. Moreover, as shown in Table 4 of the original paper, these models can actually begin suspecting the problem might not be solvable at the very beginning, but they just do not make the decision.

We think the capability to question the validity of assumptions and premises is an important part of critical thinking, as supported by numerous quotes from philosophers, educators, and scientists. As defined by the National Council for Excellence in Critical Thinking, 1987:

It [critical thinking] entails the examination of those structures or elements of thought implicit in all reasoning: purpose, problem, or question-at-issue; assumptions; … and frame of reference.

Noam Chomsky, a well-known linguist and philosopher, also emphasized the importance of questioning the assumptions in his work.

think for themselves, to question standard assumptions… Don't take assumptions for granted. Begin by taking a skeptical attitude toward anything that is conventional wisdom… Be willing to ask questions about what is taken for granted.

Dear Reviewer gLKw,

As we are approaching the midpoint of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Your insights are crucial for us!

Thank you once again for your valuable time and insights!

Best,

Authors

Concerns about Statistical Pattern Analysis

Q: The analysis in this paper is limited to pattern analysis of model outputs, lacking an in-depth discussion of the factors contributing to overthinking.

The main contribution of our work is to reveal and measure the phenomenon of models' severe overthinking under MiP questions, and offer insights through statistical analysis of models' response patterns. Locating the exact factors inside the model that result in such a phenomenon is inherently hard due to the blackbox nature of LLMs. In fact, leveraging the analysis of models' response has been a widely accepted method, e.g., [1,2]. Although in-depth discussions on the LLM inner mechanisms are undoubtedly valuable, discovering new failure modes and analyzing the response patterns also have their merits and provide insights for the community to mitigate the exposed problems.

[1] Wei et al. "Emergent Abilities of Large Language Models." TMLR, 2022.

[2] Evans et al. "The “Reversal Curse”: Failure to Infer Symmetric Relations" NeurIPS, 2023

Direct Evidence of the Cause

Q: The authors introduce the viewpoints of "inadequate length constraints" and "format and accuracy reward hacking," supported by related literature. However, the experimental results provide little direct evidence for these claims.

In order to verify our assumption in the limited time, small-scale experiments are conducted. Specifically, we used GRPO to train two models based on DeepSeek-R1-Distill-Qwen-1.5B on a 1500-samples combining MATH and past AIME datasets in a ratio of 3:1. We trained one model with a length constraint and one without the length constraint. Their evaluation results on MiP-GSM8k are presented below:

As shown in the table, adding a length constraint would shorten the response length by more than 20%, mitigating the overthinking issue exacerbated by MiP by a certain extent. Detailed experiments, along with corresponding analysis, will be included in our later version.

Concerns about Distillation Results

Q: the ability to replicate behaviors in smaller models through distillation is well-known in the community, making this result entirely predictable and the motivation questionable.

Section 4.2 is to compare SFT vs. RL trained models on MiP-overthinking and explain why SFT models still suffer from MiP-overthinking even without being trained by RL. Although it is not surprising that distillation would cause the student model to replicate the behavior of the teacher model, we find it still unexpected that only 50 samples would cause this effect on MiP-overthinking. This highlights the risk that even a minority of MiP problems in a training set would deteriorate the thinking behaviors of the student model through distillation. Moreover, we found that some widely used SFT datasets that are believed to have high-quality data also contain some MiP samples, which explains the MiP-overthinking observed on SFT models and supports our concerns.

Here is an example from simplescaling/s1K:

2.2.2. Calculate the numerical value of $\Gamma$.

Dear Reviewer hrUF,

As we are approaching the midpoint of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Your insights are crucial for us!

Thank you once again for your valuable time and insights!

Best,

Authors

Potential Improvements

Q: While the current paper is already pretty good, I think one way to continue pushing forward the research field is to hypothesize/propose some methods to improve the reasoning models, defending them against these MiP, while keeping their original high performances.

From the data perspective, a straightforward strategy we can think of to improve the performance of the model is to mix some MiP questions with abstain answers during the post-training process. From the perspective of training technique, we hypothesize that a more adaptive, MiP-aware length reward or compression method during reinforcement learning would help the model learn to reason efficiently. The length-control experiments for reviewer hrUF partially verified our hypothesis.

More Discussions on What Leads to the Contagious Behavior

Q: I like the discussion into the root causes of MiP-overthinking (section 4.2). This is good but the discussions can go deeper: what aspects of the reasoning model’s MiP outputs lead to this contagious behavior?

Thank you for your appreciation. We hypothesize that this contagious behavior is caused by the pattern-copy issue during supervised fine-tuning. [1,2] mentioned that the SFT is mainly a pattern-copy process from the teacher's responses. Thus, when the MiP-Overthinking responses are used for training, the student model will be forced to remember and output all their redundant thinking patterns without understanding when to use them. What's worse, the large amount of thinking tokens in the training responses further strengthens the bias of SFT models to output these tokens.

[1] Ghosh et al. "A Closer Look at the Limitations of Instruction Tuning." ICML, 2024. [2] Biderman et al. "LoRA Learns Less and Forgets Less." TMLR, 2024.

Dear Reviewer 4FcV,

As we are approaching the midpoint of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Your insights are crucial for us!

Thank you once again for your valuable time and insights!

Best,

Authors

MiP in Other Domains

Q: Experiments focus exclusively on mathematical reasoning. While math is a structured domain, extending evaluations to commonsense QA or code generation would strengthen claims about the universality of MiP-overthinking.

We thank the reviewer for the insights. To evaluate the impact of the MiP phenomenon in broader domains, we constructed a new MiP dataset sourced from different fields in the MMLU dataset, consisting of both commonsense and domain-specific questions, including clinical knowledge, chemistry, and physics. For each sample, we manually removed a premise that contributes to the answer and made sure the question is unsolvable. Due to the limited time for rebuttal, we have currently collected 50 human-verified samples and evaluated 4 representative LLMs (2 non-reasoning and 2 reasoning) on this new dataset. The results are presented:

As shown in the table, the substantial gap in average response length between reasoning and non-reasoning models is consistent with our findings reported in the main paper. This consistency verifies the generalizability of our findings about MiP-Overthinking. As for the domain of code generation, it is hard to construct a MiP-dataset because removing any condition of the original question usually makes it invalid. More data, experiments and corresponding detailed analysis will be included in our later version of the paper.

Human Evaluation

Q: Including human performance on MiP tasks (e.g., response length, abstention rates) would contextualize model failures and successes.

We first followed the same evaluation process as we did to LLMs: we presented a few questions to 3 human evaluators unaware of this project without informing them existence of missing premises and asked them to solve the questions. All participants provided immediate correct feedback about the solvability of the questions, demonstrating the critical thinking abilities of humans. However, continuing the evaluation on the whole dataset would be unfair due to psychological bias, as human participants, once noticing missing premises in a few problems, would adopt a different thinking process for all subsequent questions and intentionally check if each problem is solvable or not in advance.

Thus, to proceed with human evaluation, we mixed the MiP questions with solvable questions (50%-50%) and slightly changed the task for the human evaluators to judge whether the questions are solvable. We created three mixed datasets, i.e., MiP-Formula (Mix), MiP-SVAMP (Mix) and MiP-GSM8K (Mix). Specifically, there are 100 samples per dataset, where 50 are MiP questions. We calculate humans' recall and precision of MiP problem identification and record the time they spent. We compare humans' recall with the abstain rate of models (which equals recall in all-MiP datasets) we reported in the paper. The results are shown in the following table.

As demonstrated in the table, humans achieve near-perfect recall of MiP questions in the first three datasets, which is much higher than the abstain rate of reasoning models in Table 2 of our paper. This indicates that our datasets reflect a significant gap between reasoning LLMs and humans.

Dear Reviewer teYv,

As we are approaching the midpoint of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Your insights are crucial for us!

Thank you once again for your valuable time and insights!

Best,

Authors

Dear Reviewer teYv,

As we are approaching the deadline of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Your insights are crucial for us!

Thank you once again for your valuable time and insights!

Best,

Authors

Dear Reviewer teYv,

As we are approaching the deadline of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. To summarize, we have addressed your concern by extending our findings to other tasks and conducting human evaluations. We would be really grateful if you could kindly re-assess our work based on these updates.

Thank you once again for your valuable time and insights!

Best,

Authors