Reasoning Models Know When They’re Right: Probing Hidden States for Self-Verification

Thanks for the comments and feedback! We address concerns and provide more discussion below.

R1.

We thank the reviewer for these important points. We address both concerns below:

Regarding the explanation of why hidden states predict correctness:

Thanks for the question! We think it might have to do with data used to train reasoning models. Training with long CoT patterns requires the model to learn when it should backtrack, which may incentivize modeling the correctness of intermediate answers. This is somewhat validated in Section 4.4, where we show reasoning models have a better calibrated representation of correctness compared to non-reasoning models. We will add more explanation in Section 4.4. Overall we think how the correctness encoding emerges is an interesting question that is worth further investigation.

Regarding novelty compared to existing work:

We acknowledge that the probing method has already been widely used in existing works to explore model representation. However, we believe it’s worth redoing the experiments on reasoning models given their unique search behavior (See detailed discussion in General Response). The probing results also reveal some interesting and new phenomena that have important implications for understanding reasoning models. Specifically:

• Unique phenomenon: Our work addresses the "overthinking" phenomenon, which is unique to reasoning models (involving multiple reasoning paths via backtrackings) where they continue generating tokens after producing correct answers.
• Different research question: Rather than detecting general hallucinations or uncertainty of a single response, we investigate whether reasoning models know when they're right during their reasoning process.
• Model-specific insights: We demonstrate that reasoning models are not using the correctness information optimally during inference, despite having information encoded implicitly. This provides crucial insights for improving training and inference strategies specifically for reasoning models.
• Remarkably low calibration error: It’s interesting to see the calibration error is remarkably low (especially compared with short CoTs generated by non-reasoning models in Section 4.4), which indicates the reasoning models inherently encode information about answer correctness and suggest that the self-verification ability is enhanced during long CoT supervised training.

R2 & Q1. More non-math dataset

Thanks for the suggestion. We already provide results on KnowLogic (see Figure 2, Table 1) in the paper, which is a logical reasoning dataset in multiple-choice form. Below we provide more results on GPQA (we are continuing running more models). They are consistent with prior results. The training data is GPQA_extended with the test set removed (altogether 348 examples), and the test set is GPQA_diamond. We will add the results to the paper.

R3 & Q2. Statistics of intermediate answers

The existence of multiple intermediate answers is a unique observation from outputs of reasoning models. Figure 1, which is adapted from a real output from reasoning models, shows the long CoT contains multiple intermediate answers as the model keeps re-interpreting the user’s question. We have also provided the detailed statistics in Table 4 in appendix of the number of chunks (=number of intermediate answers) obtained after extracting each intermediate answer for each dataset and model combination. It can be seen that one output contains about 8 intermediate answers on average, and that it also correlates with reasoning model capability and data difficulty.

Q3. Sensitivity of LLM annotation

Thanks for your question. To test the sensitivity of LLM annotation, we compare annotation results from Gemini with GPT-4o-mini with randomly sampled 100 examples from MATH training data. As shown in the table below, the two models well agree with each other with 80% annotations exactly the same. The results show that the annotation task is relatively straightforward for state-of-the-art LLMs.

Thanks for the constructive feedback! We provide more explanations and discussions below.

R1. Baseline of prompting for confidence verbalization

Thanks for the suggestion. Prompting the model to verbalize its confidence is a straightforward method to test its notion of correctness. However, in real implementation, we find it extremely difficult to get reasoning models to follow user instruction. Experimenting with different prompting strategies, we only manage to get the model to produce a verbalized confidence for 66% of the chunks. On these chunks, we conducted the same experiment as in Section 5 and find that probing significantly outperforms self-reported confidence in both accuracy and token efficiency. We provide additional figures to illustrate this result (see Figure 1): external materials .

We also use prompting with designed verbalizer. Specifically, we append “The answer is ” to each chunk, then obtain the probabilities of the next token being exactly "correct" or "incorrect", and re-normalize them to produce a confidence score. However, the confidence obtained in this way is still less calibrated (ECE = 0.25). Using this forced self-reported confidence for early exit also performs much worse than our probe method in terms of both accuracy and token efficiency. See Figure 2 in external materials .

We will include these results in the paper.

R2. More non-math dataset

Thanks for the suggestion. We already provide results on KnowLogic (see Figure 2, Table 1) in the paper, which is a logical reasoning dataset in multiple-choice form. Below we provide more results on GPQA (we are continuing running more models). They are consistent with prior results. The training data is GPQA_extended with the test set removed (altogether 348 examples), and the test set is GPQA_diamond. We will add the results to the paper.

As for coding problems, since there are hardly any intermediate answers during reasoning, they are not suitable for validating our findings.

R3

Thanks for your comments. Our main focus is to explore whether reasoning models encode correctness information. The finding points to potential inference-time intervention to improve efficiency, but we do acknowledge that the method in the paper is more of a post-hoc analysis and a more efficient online answer extractor would be needed if deployed. Thanks for pointing that out. We will add more discussion about it in the discussion and limitation section.

R4

Thanks for the question! The major decrease in performance of non-reasoning models comes from the higher calibration error. We think it might have to do with training data difference. Non-reasoning models are trained with question-response pairs that contain a single correct answer, whereas reasoning models usually undergo long CoT supervision which contains intermediate answers that can be right or wrong. Training with long CoT patterns might lead models to implicitly learn when it should backtrack, which largely corresponds to erroneous intermediate answers.

We thank the reviewer for the comments. We would like to firstly emphasize the distinctions that necessitate our investigation. Existing work that interprets hidden states of short CoTs cannot address our central research question: whether reasoning models know when they're right during backtracking, especially given the existing "overthinking" phenomenon. Our motivation distinguishes our work from previous research that mainly focuses on understanding and improving non-reasoning models [1,2,3]. Please refer to our General Response for detailed motivation and novelty discussion.

R1. Why it’s worth revisiting reasoning models’ representations

Our research question focuses on whether reasoning models know when they're right. The answer to this question is not obvious given the observed “overthinking” phenomenon and the reported discrepancy in model’s inner representations and behavior. The question also has far more significance given that reasoning models are uniquely characterized by search behavior during reasoning, and such representation of correctness becomes crucial to the success of reasoning. Therefore, we believe it is important to revisit hidden states of reasoning models. Our results also show that reasoning models exhibit substantially better calibration, which suggest that further research is essential for understanding what and how reasoning models learn during training. Compared to [1] that adopts layerwise logit lens analysis, the simplicity and proven success of our probing method further proves the importance of posing reasoning models as a unique research target.

R2. Comparison with other verifier/reasoning efficiency works

Thanks for the pointer. We would like to clarify that our primary motivation is to explore a fundamental scientific question rather than propose an efficient inference method or verifier. Our work focuses on understanding whether reasoning models know when they're right.

[2] trains an external verifier that validates the correctness of current reasoning and uses the verifier to assist in beam search decoding to mimic search behavior for non-reasoning models. [3] uses layer-wise decoding consistency to measure model confidence in current answer and boosts performance by re-weighting ensemble answers in inference time. Both [2] and [3] focus on non-reasoning models and far more effort is required to validate answer correctness compared to our probing method. This also indicates the difference between reasoning and non-reasoning models, where reasoning models that naturally exhibit search behavior, and have a better representation of answer correctness that can be easily extracted. We will add more discussion on how our method relates to previous works on non-reasoning models in the paper.

Q1.

Thanks for the question! The critical factor is not model size per se, but whether the model has undergone reasoning-specific training. As we demonstrate in Section 4.4, reasoning models exhibit significantly better-calibrated notions of correctness compared to their non-reasoning counterparts. We think it might have to do with training data difference. Non-reasoning models are trained with question-response pairs that contain a single correct answer, whereas reasoning models usually undergo long CoT supervision which contains intermediate answers that can be right or wrong. This exposure appears to enable reasoning models to implicitly learn rich information about the reasoning process, even without explicit supervised signals for correctness encoding. As shown in Figure 2 and Table 1, even smaller reasoning models like DS-Distill-Qwen-1.5B show remarkably low calibration error along with high accuracy.