Calibrating Reasoning in Language Models with Internal Consistency

Thank you for rating our paper positively! Your efforts in reviewing our paper and providing profound advice are sincerely appreciated. We would like to address your remaining concerns in the following response.

W1: Marginal improvement of SC+IC over SC
Thank you for your constructive feedback. We do not observe significant improvement by exponentiating the internal consistency. Nonetheless, we observe significant improvement by applying tuned weights assigned to intermediate layers (please see our global response for details). As shown in Table 1 of the attached pdf, using a weighted voting of latent predictions can drastically improve performance. Perhaps more interestingly, the tuned weights are transferable across datasets, indicating a universal pattern of internal representations for calibration.

W2: Section 4.4 is unclear
As illustrated in the caption of Figure 5, layers with high attention weights on critical tokens do not align with those promoting certain predictions. Such misalignment can result in discrepancy of behavior at different layers, and leads to internal inconsistency. After applying the logistic regression with respect to the last hidden state and the final output (True/False) of the model (as detailed in Appendix B.5.2), the probe vector is the weight matrix (a vector in this case as the labels are binary) of the logistic regression model.

We apologize for any confusion caused and will polish Section 4.4 following your suggestions in the final version.

W3: Necessary details for the main paper
Thank you for your constructive feedback. We will properly reorganize the details and put some important details forward in the main paper for better reading experience.
Q1: Why against the final layer?
Thank you for your suggestion. We choose the final layer because the final layer gives the natural prediction of the whole model, with minimal modification of the original mechanism of the model. While measuring the consistency between every pair of layers is reasonable, we do not observe improvement in the cost of time complexity.

Q2: Applicability of IC to non-binary answers?
While our analysis in the original paper focuses on datasets with true-or-false answers, the concept of internal consistency can be natually extended to non-binary answers. We add new experiments on SWAG $1$ and MathQA $2$ . Please refer to our global response for details. As shown in Table 1 of the attached pdf, our method consistently outperforms the baselines, especially on the more challenging MathQA dataset. Notably, SC+IC (transfer) demonstrates robust transferability even though its weights are optimized for binary questions in PrOntoQA, confirming the effectiveness of internal representations for reasoning.
Q3-Q5: Writing requiring clarification

Line 210 “decrease”: Actually we mean longer reasoning paths tend to result in lower internal consistency, as illustrated in Section 2.2 and Figure 2.

Line 241-242 and line 248: Thank you for your advice. Your rephrasing is much clearer than our original expression and we will correct them following your advice.

Q6: Confusion about Figure 5's caption

Actually we do not expect or suppose the attention weights and number of top value vectors to align. Section 4.4's analysis seeks to explore how transformer components contribute to internal consistency, with the misalignment between these factors offering insight into why internal inconsistency arises.

$1$ Zellers R, Bisk Y, Schwartz R, et al. Swag: A large-scale adversarial dataset for grounded commonsense inference $J$ . arXiv preprint arXiv:1808.05326, 2018.

$2$ Amini A, Gabriel S, Lin P, et al. Mathqa: Towards interpretable math word problem solving with operation-based formalisms $J$ . arXiv preprint arXiv:1905.13319, 2019.

Thank you for rating our paper positively! We sincerely appreciate your observant review and insightful suggestions and aim to address your concerns in the following response.

W1: Motivation behind internal consistency
We would like to clarify that we do not expect latent predictions to be internally consistent, but to use consistency as a measurement for calibration. More specifically, our study on internal consistency is motivated by our preliminary analysis that inconsistency emerges at middle layers and is exacerbated in CoT reasoning. This leads to an intuition that predictions with greater certainty can converge earlier in latent predictions. Consequently, we emphasize consistency from a comparative perspective: A stronger consistency between the latent predictions at intermediate layers and the final prediction indicates higher certainty and thus can be used to weigh reasoning paths for CoT.

We apologize for any confusion caused by insufficient clarity regarding our motivations and will revise these sections for improved clarity in the final version.

W2: The improvements in Table 1 seem to be limited to some models/datasets & Missing detailed data statistics

We would like to highlight that, although the improvement is relatively small for datasets such as BoolQ, consistent performance enhancements are evident across various models and datasets. The gains are particularly pronounced in more challenging reasoning tasks, such as logical reasoning. Given that IC is an unsupervised method, we consider these results valid in demonstrating the calibration effectiveness of internal consistency.

Our method can achieve further improvements by applying tuned weights assigned to intermediate layers. As shown in Table 1 of the attached pdf, using a weighted voting of latent predictions can drastically improve performance. Perhaps more interestingly, the tuned weights are transferable across datasets, indicating a universal pattern of internal representations for calibration.

Regarding detailed data statistics, we place them in the Appendix B.2 of the original paper, including specific numbers of samples and other details.

W3: Results lack analysis and description
We apologize for any confusion caused by the lack of analysis and are happy to discuss with you on specific contents that you feel unclear.

For Figure 3, the panels sequentially display: a) The boxplot describing the distribution of internal consistency under each prompting setting, highlighting the emergence of inconsistency in CoT setting. b) The distribution of internal consistency level when the model gives correct and wrong predictions, and the discrepancy suggests internal consistency can be an indicator of prediction accuracy. c) The discrepancy of the ratio of data instances where the latent predictions match the final ones across layers, which further illustrates incorrect reasoning paths lead to lower consistency level. d) A plot describing the prediction accuracy under each internal consistency level, shows that internal consistency is a reliable metric for calibration. We will elaborate the description in the final version.
W4-W6: Issues with the figures
Thank you for your constructive feedback! We will reorganize the figures in our final version based on your suggestions.

Regarding Figures 3(b) and 3(d), we calculate internal consistency for each sampled reasoning path and associate these values to the accuracy of the final predictions. The results are presented as histograms, where each bar corresponds to a specific bin.

Thank you for your thoughtful reviews! We appreciate your recognition of our method as “novel and promising”. We aim to address your concerns in the following response.

W1: The proposed techniques could be computationally expensive
The proposed techniques incur minimal computational overhead compared to decoding. To assess internal consistency (IC), we apply the model's unembedding matrix to the hidden states of each intermediate layer associated with the answer token, once per reasoning path. Given that these hidden states are computed during decoding, they can be repurposed for evaluating IC without additional computation.

For a more detailed analysis, we estimate the proportion of computation dedicated to assessing IC in Llama-2-7b. The FLOPS per token during decoding amount to approximately $2P$, where $P$ represents the total number of parameters. For calculating IC, this is bounded by the normalization step prior to unembedding, with a time complexity of $O(dL)$, with $d$ being the hidden size and $L$ being the number of layers. Since $dL \\ll P$, the computation overhead is negligible. It is important to note that because IC is computed only once per reasoning path, the actual proportion of allocated computation is very low (~0.01% of the time required to decode a reasoning path). Below are statistics from one of our setups:

Time complexity comparison between IC calculation and decoding; benchmarked with Llama-2-7B on PrOntoQA.

W2: The overfitting risk
We would like to clarify that our method requires no training process to calculate IC and apply IC in reasoning. Without tuning on specific data, our method consistently achieves better results over baselines on various models and datasets, validating the generalizability of our method.

In response to your interest in potential overfitting risks when optimizing for specific data, we conduct additional experiments on tuning weights assigned to each layer during IC calculation (see a more detailed setup in our global response). Despite the minimal number of parameters ($L$, the number of layers) involved, tuning weights yields substantial enhancements on various datasets and models. Importantly, the optimized weights are also adaptable to different datasets, indicating a universal pattern in internal representations that can be leveraged across multiple tasks.

Q1: Why use lower layers?
The use of lower layers is motivated by the intuition that predictions with greater certainty can converge earlier in latent predictions. This intuition is supported by Figure 3c, which shows that lower layers exhibit a higher agreement rate for correct predictions than for wrong ones. Consequently, we emphasize consistency from a comparative perspective: A stronger consistency between the latent predictions of lower layers and the final prediction indicates higher certainty and thus can be used to weigh reasoning paths.

Indeed, Figure 3c also suggests that lower layers (<10) contain less semantic information and are relatively noisy. We conduct a new experiment to exclude lower layers for IC calculation, which results in an improvement for Llama-2 models. However, we also note that it introduces the need to specify layers to exclude and we plan to discuss it along with the new tuning results in the revised version.

Comparison of calculating IC on all layers versus on layers higher than 10. We report calibrated accuracy averaged over different datasets.

Q2: Removing the last few layers to reduce computational complexity?
Yes, it is possible to reduce computational complexity by implementing an early-exiting strategy based on IC, using a proper stopping criterion like saturation $1$ . Nonetheless, our primary aim is to explore how IC can improve reasoning capabilities rather than accelerate inference, which has been the focus of prior research $1, 2$ .

Q3: Interaction between internal consistency with issues like hallucination or bias in data?
IC serves as a metric for calibration, aiding LLMs in expressing their confidence in the prediction. This is essential for mitigating issues like hallucination or bias, by detecting possible hallucinated or biased predictions with IC and correcting them accordingly $3$ . In this regard, our approach to calibration in reasoning aligns well with hallucination and bias detection. We will provide more detailed discussion on this interaction in the final version.

$1$ Schuster T, Fisch A, Gupta J, et al. Confident adaptive language modeling $J$ . Advances in Neural Information Processing Systems, 2022, 35: 17456-17472.

$2$ Raposo D, Ritter S, Richards B, et al. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models $J$ . arXiv preprint arXiv:2404.02258, 2024.

$3$ Pan L, Saxon M, Xu W, et al. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies $J$ . arXiv preprint arXiv:2308.03188, 2023.

Thanks for the authors' response. Given the additional experiments, generalized results, and answers to my questions, I decided to raise my score.

Thank you for the comprehensive and insightful feedback! We appreciate your effort in reviewing our paper and are committed to addressing your concerns in the following response.

W1: Internal consistency is rather straightforward
Indeed, internal consistency (IC) serves as a straightforward measurement for calibration. We consider simplicity a valuable aspect of our approach, given that it incurs minimal computational overhead (approximately 0.01% additional time as discussed in the response to Reviewer B2Yn) and can be seamlessly integrated into various models without the need for tuning.

To better utilize internal consistency for calibration measurement, we conduct new experiments that tune the weights assigned to intermediate layers during IC calculation. For a comprehensive setup and analysis of the results, please see our global response and our response to W2.

W2: Table 1 does not achieve great improvement
We note that SC+IC is an unsupervised method that is not tuned on specific data. While the improvements are not substantial, we observe consistent performance gains over SC across models and datasets, especially on tasks that require more complex reasoning.

The improvements can be lifted by applying tuned weights assigned to intermediate layers. As shown in Table 1 of the attached pdf, using a weighted voting of latent predictions can drastically improve performance. Perhaps more interestingly, the tuned weights are transferable across datasets, indicating a universal pattern of internal representations for calibration.

W3: True-or-false questions may not fully represent the diversity of reasoning tasks
Following a similar analytical protocol in previous work $1, 2$ , we focus on the true-or-false questions for evaluation in the original paper. To increase the diversity of reasoning tasks, we add new experiments on SWAG $3$ and MathQA $4$ . As shown in Table 1 of the attached pdf, our method consistently outperforms the baselines, especially on the more challenging MathQA dataset. Notably, SC+IC (transfer) demonstrates robust transferability even though its weights are optimized for binary questions in PrOntoQA, confirming the effectiveness of internal representations for reasoning.

W4: Paper organization
Thank you for your constructive feedback!

The background section on the Transformer architecture aims to establish notations and introduce readers who are unfamiliar with it. However, we acknowledge that this section is overly lengthy and contains redundant information. In the final version, we will reorganize the structure to enhance clarity and presentation.

Q1: More probing results regarding the suitability of lower layers
We apply probing in our preliminary analysis of internal consistency. While the pattern observed is valid, we find that probing cannot reliably indicate subtle differences in lower layers–a limitation inherent to probing $5$ . Therefore, we instead focus on CoT experiments for evaluation.

Regarding the suitability of lower layers, please refer to our detailed response to Q1 from Reviewer B2Yn, where we present new experimental results and discuss the use of lower layers.

Q2: Results using other approaches for reasoning
We conduct new experiments using least-to-most (L2M) prompting $6$ , a representative method other than chain-of-thought reasoning to elicit reasoning. We sample 4 instances from each dataset (8 for coinflip) as in-context learning examples, following the setup in our original paper, and use GPT-4o to generate subproblems and answer to each subproblem. We manually check the prompts to ensure they are correct. The results below indicate that integrating IC into L2M prompting consistently improves reasoning performance, confirming the efficacy of our approach.

Results of least-to-most prompting. We report calibrated accuracy averaged over four datasets.

$1$ Li K, Patel O, Viégas F, et al. Inference-time intervention: Eliciting truthful answers from a language model $J$ . Advances in Neural Information Processing Systems, 2024, 36.

$2$ Halawi D, Denain J S, Steinhardt J. Overthinking the truth: Understanding how language models process false demonstrations $J$ . arXiv preprint arXiv:2307.09476, 2023.

$3$ Zellers R, Bisk Y, Schwartz R, et al. Swag: A large-scale adversarial dataset for grounded commonsense inference $J$ . arXiv preprint arXiv:1808.05326, 2018.

$4$ Amini A, Gabriel S, Lin P, et al. Mathqa: Towards interpretable math word problem solving with operation-based formalisms $J$ . arXiv preprint arXiv:1905.13319, 2019.

$5$ Alain G, Bengio Y. Understanding intermediate layers using linear classifier probes $J$ . arXiv preprint arXiv:1610.01644, 2016.

$6$ Zhou D, Schärli N, Hou L, et al. Least-to-most prompting enables complex reasoning in large language models $J$ . arXiv preprint arXiv:2205.10625, 2022.