Understanding Chain-of-Thought in LLMs Through Information Theory

For math problems, the primitive tasks are intuitive to define. Have you tested your framework with other tasks, e.g. Blocksworld, or commonsense QA, where the definition of primitive tasks is less straightforward? Will different primitive task affect the conclusion?

This is indeed a very interesting suggestion, and we did consider this direction. However, given the scope of this paper, we focused primarily on the foundational aspects of our framework, particularly in the context of mathematical reasoning, as these settings are easier to control and analyze systematically.

For Blocksworld, the main task is "planning," which itself can be decomposed into sub-tasks comprising specific sequences or tactics of primitive actions (e.g., "stack," "unstack," "move"). In this setting, our information-gain methodology could be applied to evaluate LLM's performance in planning these sub-tasks. By examining the information gain for each sub-task, we could assess where the reasoning process succeeds or fails.

Similarly, in Commonsense QA, the CoT steps could be categorized into distinct reasoning types, such as causal reasoning (identifying cause-and-effect relationships), temporal reasoning (understanding the sequence and timing of events), or spatial reasoning (comprehending physical arrangements and relationships between objects). These categories align naturally with our framework, enabling an evaluation of the LLM's reasoning steps within each category.

While these tasks lend themselves well to our framework, we acknowledge that they may introduce additional nuances or challenges, particularly concerning the assumptions underlying our methodology. Exploring such tasks in detail is indeed an exciting direction which we leave for future work.

[...] a common failure for CoT to get a correct answer but with incorrect intermediate steps?

We thank the reviewer for pointing out this special case that we mentioned in the limitations of our paper. However, we would like to clarify a few things regarding the setup where the LLMs get the correct answer with incorrect intermediate steps.

Recall that our method only estimates the relevant information for the correct final answer added by each successive step. Therefore, in cases where an intermediate step is incorrect, it should add no useful information regarding the correct final answer and therefore the information gain at that step should still be 0 (regardless of whether the final output is correct or not). In contrast, in cases where the model indeed consistently commits errors at intermediate steps which then consistently lead to the correct answer, the baselines such as ORM and Math Shepherd would fail to identify such errors as these methods mainly rely on the correctness of the final answer.

Nevertheless, your question did spark some interest in us and hence we went back to our datasets to check how often this indeed happens and report that in our arithmetic experiments in Section 5.2 only 1.2% of the samples indeed exhibit this behaviour where an intermediate step is wrong, but the final answer is correct. For the synthetic toy experiment, this never happens as we have control over the generation to some degree. Hence, we would argue that for our settings, these are relatively rare and would not affect the outcomes of our framework. We thank the reviewer again for their insightful comment to improve our paper and will definitely add a discussion on this in the final version of our paper.

Also in section 5.1.1, when discussing about pitfall of the baselines, for ORM, you stated that the classifier becomes confident that the final output will be wrong right after λ2, even though the error occurs at λ3. But this is what we expect from the error detection, right? I understand you want to show the classifier will overfit on learning the false correlation between z2[2]>150 and the final error as in section 5.2. But this statement/setting doesn't seem correct to me.

In Section 5.1.1 we aimed to understand if the respective methods were able to correctly identify if a rule $\lambda_i$ had been incorrectly applied. We agree that this behaviour is expected as ORM will learn the correlational relationship between step 2 and the final answer, i.e., the ORM model will become confident that the output is wrong as soon as the sub-task $\lambda_2$ is applied. Therefore, if we were using the ORM model's confidence to debug which sub-task is applied incorrectly, we would be misled into concluding that step $\lambda_2$ is applied incorrectly, even though the error arises at step $\lambda_3$. We will ensure that this is properly discussed in our final version of the paper.

Lastly, we would like to thank the reviewer again for their insightful comments and thorough reading to improve our paper. We hope that the above has addressed any outstanding questions and that the reviewer would consider raising their score if all the questions have been appropriately answered.

First of all, we would like to thank the reviewer for their thorough feedback on our paper. We would also like to thank the reviewer for praising our proposed framework as a "novel approach to evaluating CoT reasoning by using information theory", being "well-structured, and the theoretical assumptions are systematically translated into a practical algorithm." Below, we will address any comments that were raised by the reviewer.

In section 3.3, while the information-theoretic approach is innovative, the need to train a separate supervisor model to estimate information gain might introduce complexity. The paper states that the framework doesn't require annotated data, but the supervisor also requires finetune to predict the ground truth final output.

We would like to clarify that the data required in our setting comprises only the prompts as well as corresponding correct final answers and hence do not require any annotations of intermediate steps (as with standard Process reward models). In our case, the supervisor model takes the CoT steps and is only finetuned to predict the final answer from each step. Note that this means that we do not require any human annotations for the correctness of each intermediate step, rather only a decomposition which is natural in CoT. We will clarify this in the final version of our manuscript.

In section 2.2, The framework relies on the decomposition of tasks into primitive sub-tasks and the assumption that each primitive task contributes incrementally to the information gain. For complex reasoning tasks with intertwined dependencies between sub-tasks, the information gain for each step will be more complex, and this assumption might not always hold.

We thank the reviewer for raising this interesting point. Firstly, we would like to clarify that it is not a theoretical assumption in our framework that each intermediate CoT step contributes incrementally to the information gain. In fact, this follows directly from Proposition 3.4, which establishes this property theoretically without requiring any additional assumptions.

However, we agree with the reviewer that in practice in complex settings where the CoT may involve a large number of intertwined intermediate steps, estimating the information gain accurately can become more challenging due to the increased complexity of the dataset and task structure. In such scenarios, this difficulty could be mitigated by obtaining additional CoT data for training the supervisor model, thereby providing better coverage of the complex dependencies.

Additionally, using larger and more expressive supervisor models may further enhance the framework's ability to process such intricate data effectively. Note that in our experiments, we found that the relatively small GPT-2 models worked well as supervisor models, and we only required relatively small amounts of training data for fine-tuning these models (e.g., roughly 2000 samples were used to fine-tune our supervisor model for GSM-8K experiments). We appreciate the reviewer’s suggestion, and we will include a discussion of this point in the final version of our paper to address these considerations more explicitly.

There are some errors in the paper, for example, in appendix B.1.1, the annotation for task λ2, if I understand correctly, should be z1[0],z1[0]+z1[1],z1[0]+z1[1]+z1[2].

Thanks! We thank the reviewer for having gone through our paper so thoroughly and will correct this typo in the final version of this paper.

We appreciate the reviewer’s recognition of the clarity of our presentation and the thoroughness of our theoretical framework, as well as their constructive comments. Below, we address each of the points raised in the review.

The gap between the proposed framework and the claimed application scenarios remains big, which affects the potential impact of this work.

We thank the reviewer for their suggestions. We would like to emphasize that our work is intended as a foundational investigation into CoT reasoning in LLMs, drawing inspiration from foundational studies such as the Physics of Large Language Models series [e.g., Allen-Zhu et al., 2024; Ye et al., 2024]. Similar to these studies, our aim is to build a rigorous framework that could serve as a basis for future investigations across various reasoning tasks.

While our toy and controlled GSM experiments may involve semi-synthetic settings, we would like to highlight that our arithmetic operations experiments on Llama-3-8b (in Section 5.2) are based on real-world data generated by the Llama-3-8b model without any additional fine-tuning or training. This experiment effectively illustrates that CoT errors in LLMs often involve spurious correlations — for instance, LLMs are prone to failures on mathematical problems with large numbers [Razeghi et al., 2022].

Our experiments demonstrate this phenomenon in real-world datasets, and show that even in the simplest of settings, baseline methods such as ORM and Math Shepherd fail to detect CoT errors accurately, while our information-theoretic approach remains robust and accurate. We believe these findings are valuable and significant on their own, as they point toward the possibility of constructing more robust process reward models without relying on annotated CoT datasets. Furthermore, and more importantly, they highlight the limitations of existing baseline methods which rely on the correctness of final answers, as these methods are prone to spurious correlations in model's CoT reasoning, and consequently may not detect the true cause of an error.

While we agree that additional large-scale experiments could further demonstrate the broader applicability of our framework, we view this as an interesting direction for future work.

References

Zeyuan Allen-Zhu, Yuanzhi Li. Physics of Language Models: Part 1, Learning Hierarchical Language Structures. 2024

Tian Ye, Zicheng Xu, Yuanzhi Li, Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. 2024

Tian Ye, Zicheng Xu, Yuanzhi Li, Zeyuan Allen-Zhu. Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems. 2024

Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning, 2022.

Section 2.1. Do we need to, and how to define the set of lambda in a realistic task?

In practice, solving complex problems often requires breaking them down into sub-operations that a model can directly perform without further decomposition. Our theoretical framework uses tasks (denoted by $\lambda$) to categorize these sub-operations into a finite set of groups (e.g., addition, multiplication), allowing practitioners to evaluate a model's CoT reasoning on any specific sub-task category.

The task of defining $\lambda$ is up to the practitioner and depends on the category they are interested in evaluating. If a practitioner wishes to use our framework to assess a model's performance on sub-operations of a particular type (e.g., addition in the GSM8K dataset or inductive reasoning in the Atlas Reasoning dataset), they would need to choose the relevant category and filter for the sub-operations within this type. They can then calculate the information gain across these filtered steps.

Thus, the categories (or tasks) are domain-dependent, and, in practice, it is only necessary to define the category of interest rather than all possible categories. Alternatively, if the goal is to identify incorrect steps at a sample-wise level within a given CoT trajectory, defining specific categories is unnecessary. This can be achieved by computing sample-wise information gain at each step of the CoT trajectory, as outlined at the end of Section 3 in our paper.

We hope this clarifies the practical definition and use of $\lambda$ in realistic tasks, and we would be happy to discuss further if any questions remain.

Firstly, we sincerely thank the reviewer for their thoughtful and positive feedback. We greatly appreciate their recognition of our framework's novelty, theoretical contributions, and the clarity of our presentation. Below, we address the concern raised regarding sample-wise information gain.

I have only one concern about this paper, namely, I question if the sample-wise information gain (Equation 5) is a valid measure. Note that although the standard mutual information (or "distribution-wise mutual information") is well-known to be non-negative and zero mutual information gives independence, the "sample-wise mutual information" (namely, the term without expectation) can be negative in general. The "intuitive" argument the authors give (lines 299 to 303) to justify the use of sample-wise mutual information is not convincing to me. -- I feel this is the main weakness of an otherwise very elegant development. I encourage the authors to resolve this issue.

Thank you for highlighting this point and we appreciate the opportunity to clarify our sample-wise information gain approach.

1. Sample-Wise Information Gain as a Heuristic Measure

We propose sample-wise information gain as a heuristic measure to assess sub-task correctness at a sample-specific level, intended as a signal to detect incorrect steps within a given CoT sample. While it is true that sample-wise information gain can be negative, this quantity intuitively reflects the change in the model's confidence about the ground truth answer $Y$ being correct. If the sample-wise information gain is positive, it indicates an increase in the model’s confidence about the correct answer (i.e. $p(Y\mid X^M_t) > p(Y\mid X^M_{t-1})$) , suggesting that the model is proceeding along the correct reasoning path. Conversely, if the sample-wise information gain is negative, it indicates a decrease in confidence about the correct answer, signalling a potential error in the reasoning step.

2. Precedent in Prior Work

While this approach does not test the conditional independence in Theorem 3.3 in our paper, there is precedent for using sample-wise mutual information in prior work to make inferences at the individual sample level. For instance, Durme et al., (2009), Ethayarajh et al. (2022), Nandwani et al., (2023) leverage sample-wise mutual information for making inferences on samples. Inspired by these methods, we leverage sample-wise information gain as a practical heuristic to evaluate reasoning steps within individual CoT trajectories.

3. Empirical Evidence Supporting Sample-Wise Information Gain

Importantly, our experimental results (Tables 1–3) demonstrate that sample-wise information gain effectively captures information about the correctness of CoT steps. These results show that this metric can detect CoT errors with higher accuracy than the baselines considered, underscoring its practical utility in real-world settings. While we agree that further theoretical grounding would strengthen its justification, empirical evidence supports its efficacy in detecting reasoning errors. We hope this explanation addresses your concern and highlights the rationale behind the use of sample-wise information gain, and we will further clarify this in the final version of our paper.

Thank you again for your thoughtful feedback and for recognizing the broader contributions of our framework.

References

Benjamin Durme, Ashwin Lall. Streaming Pointwise Mutual Information, 2009.

Kawin Ethayarajh, Yejin Choi, Swabha Swayamdipta. Understanding Dataset Difficulty with V-Usable Information, 2022.

Yatin Nandwani, Vineet Kumar, Dinesh Raghu, Sachindra Joshi, Luis A. Lastras. Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs, 2023.

First of all, we would like to thank the reviewer for their time and thoughtful comments on our paper. We greatly appreciate the positive feedback on the novelty of using information gain for detecting reasoning errors, the clarity of our presentation, and the robustness of our experiments. Below we address the questions raised by the reviewer:

A significant weakness of the paper is that it is unclear how applicable it is to real-world reasoning tasks. More experiments involving more challenging tasks such as MMLU or MATH would be more convincing.

We thank the reviewer for their suggestions. We would like to emphasize that our work is intended as a foundational investigation into CoT reasoning in LLMs, drawing inspiration from foundational studies such as the "Physics of Large Language Models series" [e.g., Allen-Zhu et al., 2024; Ye et al., 2024]. Similar to these studies, our aim is to build a rigorous framework that could serve as a basis for future investigations across various reasoning tasks.

While our toy and controlled GSM experiments may involve semi-synthetic settings, we would like to highlight that our arithmetic operations experiments on Llama-3-8b (in Section 5.2) are based on real-world data generated by the Llama-3-8b model without any additional fine-tuning or training. This experiment effectively illustrates that CoT errors in LLMs often involve spurious correlations — for instance, LLMs are prone to failures on mathematical problems with large numbers [Razeghi et al., 2022]. Our experiments demonstrate this phenomenon in real-world datasets and show that even in the simplest of settings, baseline methods such as ORM and Math Shepherd fail to detect CoT errors accurately, while our information-theoretic approach remains robust and accurate. We believe these findings are valuable and significant on their own, as they indicate the possibility of constructing more robust process reward models without relying on annotated CoT datasets.

Furthermore, and more importantly, they highlight the limitations of existing baseline methods which rely on the correctness of final answers, as these methods are prone to spurious correlations in model's CoT reasoning, and consequently may not detect the true cause of an error. While we agree that additional large-scale experiments could further demonstrate the broader applicability of our framework, we view this as an interesting direction for future work.

References

Zeyuan Allen-Zhu, Yuanzhi Li. Physics of Language Models: Part 1, Learning Hierarchical Language Structures. 2024

Tian Ye, Zicheng Xu, Yuanzhi Li, Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. 2024

Tian Ye, Zicheng Xu, Yuanzhi Li, Zeyuan Allen-Zhu. Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems. 2024

Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning, 2022.

The assumption of needing to annotate the task involved in each step is too restrictive. In reality, such a one-to-one mapping is neither possible nor necessary. A single reasoning step could use more than a single primitive task. It could also use no primitive task at all by simply being a filer step.

We would like to clarify that in our approach, the goal is to evaluate the model's CoT performance on sub-tasks of different kinds. To this end, in practice, our framework uses the notion of primitive tasks to define distinct categories for each sub-step, allowing the model's performance on each category to be evaluated.

For example, in the GSM8K dataset, these categories could include basic arithmetic operations such as addition or multiplication. In reasoning datasets (e.g., Atlas-Reasoning, commonsenseQA), categories could correspond to inductive reasoning, deductive reasoning, or quantitative reasoning. The categories are distinct enough that a suitable CoT decomposition should not combine multiple categories into a single sub-task. If such a combination arises, it suggests that the sub-task could be further decomposed into steps aligned with each individual category.

In this case, our framework allows prompting the model to decompose problems in a way that ensures each sub-step aligns with at most one category. For example, when prompting the LLM on gsm-8k, we explicitly instructed the model to ensure that each step should correspond to distinct sub-operations such as addition, multiplication, etc. This approach preserves interpretability and the granularity needed for reliable CoT performance evaluation, even in complex reasoning tasks.

Moreover, our framework does not require a strict one-to-one mapping of each reasoning step to a specific category. Instead, we need each CoT step to correspond to at most one category and our method remains applicable in cases where a step is simply a filler step and therefore does not belong to any category. In such cases, while the information gain for the filler step may be 0, the information gain for other categories should still convey the same information about model performance.

The distinction between error steps and irrelevant steps is mentioned but it is unclear how the IG method handles them separately. More clarity is needed. From the description, it looks like they will be treated equally. However, irrelevant steps are not as harmful as error steps.

Thank you for raising this important point. Generally speaking, in our framework it is indeed possible to distinguish between irrelevant steps and error steps through the Information Gain (IG) metric. If a step is irrelevant, the information gain at that step may be zero, but as long as the model’s CoT reasoning remains on track toward the correct solution, the information gain will increase at future steps. In contrast, if a step is incorrect and causes the model’s CoT to deviate from the correct path, the information gain will remain zero in all subsequent steps.

Thus, generally by analyzing the information gain in future steps, our framework can effectively differentiate between steps that are irrelevant and those that are genuinely erroneous, capturing their differing impacts on the model's reasoning path.

The paper can benefit from grammar checks. Line 055 detect is repeated twice. Line 164 tasks is supposed to be task. Line 295, which step went wrong is using past tense while most of the paper is using present tense. Same for chose in line 429.

We thank the reviewer for pointing out these typos. We will correct them in the final version of our manuscript.

The proposition 3.4 is problematic as Equation (3) is asymmetric with respect to Y and Xj, while we know that mutual information should be symmetric.

We thank the reviewer for pointing out this potential source of confusion. We would like to clarify that the conditional mutual information in Equation (3), $\mathcal{I}(Y; X^M_j \mid X^M_{j-1})$, is indeed symmetric with respect to $Y$ and $X^M_j$. Specifically, this conditional mutual information is defined as $\mathcal{I}(Y; X^M_j \mid X^M_{j-1}) = E\left[\log \frac{p(Y, X^M_j \mid X^M_{j-1})}{p(Y \mid X^M_{j-1}) \, p(X^M_j \mid X^M_{j-1})}\right],$ where both the numerator and the denominator in the expectation above (as well as the expectation itself) are symmetric with respect to $Y$ and $X^M_j$. This notation for conditional mutual information follows standard conventions in literature [Wyner (1978), Fawzi et al., (2015)], and we will clarify this further in the final version of our paper.

We hope our clarifications have addressed the reviewer's concerns, and the reviewer will consider increasing their score.

Reference:

A.D. Wyner, A definition of conditional mutual information for arbitrary ensembles, Information and Control, Volume 38, Issue 1, 1978, Pages 51-59

Omar Fawzi, Renato Renner, Quantum conditional mutual information and approximate Markov chains. 2015

While the formula the authors wrote down in the reply is symmetric, the formula in the first half of Eq.(3) is not symmetric wrt to Y and X_j. So, it is confusing to me how the authors arrive at Eq.(3), which is the foundation of the entire method.

We would like to further clarify that the symmetry property of $Y$ and $X_{j}^M$ is conditional on $X_{j-1}^M$ and in our setting, the first half of Eq. (3) indeed satisfies this symmetry property.

The proof of Proposition 3.4 provided in Appendix A can be used to show this. However, for clarity, we have included explicit and rigorous proof of this statement below.

We prove this as follows:

The first step in our proof involves showing that $p(Y\mid X_j^M) = p(Y\mid X_j^M, X_{j-1}^M)$. To see this, recall that $X_{j}^M$ encapsulates the model's CoT generation for all steps up to step $j$. Specifically, $X_{j}^M$ also includes the previous CoT steps $X_{j-1}^M$. Therefore, conditional on $X_{j}^M$, the state $X_{j-1}^M$ is deterministic and hence, $Y \perp\\!\\!\\!\perp X_{j-1}^M \mid X_{j}^M$. From this, it follows that $p(Y\mid X_j^M) = p(Y\mid X_j^M, X_{j-1}^M)$.

Now using this fact, we get that $E[\log p(Y\mid X_j^M)] - E[\log p(Y \mid X_{j-1}^M)]$ $= E[\log p(Y \mid X_j^M, X_{j-1}^M)] - E[\log p(Y \mid X_{j-1}^M)]$ Next, the definition of conditional probability yields that the above is equivalent to: $= E\left[\log\frac{ p(Y, X_{j}^M \mid X_{j-1}^M)}{p(X_{j}^M \mid X_{j-1}^M)}\right] - E[\log p(Y \mid X_{j-1}^M)]$ $= E\left[\log\frac{ p(Y, X_{j}^M \mid X_{j-1}^M)}{p(X_{j}^M \mid X_{j-1}^M)\\, p(Y \mid X_{j-1}^M)}\right] \left(=: \mathcal{I}(Y; X_{j}^M \mid X_{j-1}^M) \right)$ Where in the expectation above, both the numerator and denominator (as well as the expectation itself) are symmetric w.r.t. the conditional distributions of $Y$ and $X_{j}^M$conditional on $X_{j-1}^M$.

This proves the equality in Eq. (3) in our paper. Given the symmetry property of the conditional mutual information $\mathcal{I}(Y; X_{j}^M \mid X_{j-1}^M)$, it then follows that the LHS also follows the symmetry property.

Possible source of confusion

There seems to be a slight misunderstanding here, which could be the cause of confusion. Specifically, the symmetry holds conditional on $X_{j-1}^M$ (and not independent of $X_{j-1}^M$).

To be rigorous, what this means is that there exists some functional $\mathcal{F}: \mathcal{P} \times \mathcal{P} \times \mathcal{P} \rightarrow \mathbb{R}$ where $\mathcal{P}$ is the space of probability distributions, such that

1. the first half of Eq. (3) can be expressed as: $E[\log p(Y\mid X_j^M)]-E[\log p(Y \mid X_{j-1}^M)] = E\left[\mathcal{F}\left(P_{Y\mid X_{j-1}^M}, P_{X_j^M\mid X_{j-1}^M}, P_{Y, X_j^M\mid X_{j-1}^M}\right)\right],$ where $P_{Z}$ denotes the distribution of the random variable $Z$ and
2. $\mathcal{F}$ is symmetric w.r.t. its first two arguments, i.e. $\mathcal{F}(p, q, r) = \mathcal{F}(q, p, r)$. Note that there is no symmetry requirement w.r.t. the third argument of $\mathcal{F}$ because the joint distribution $P_{Y, X_j^M\mid X_{j-1}^M}$ is already symmetric w.r.t. $Y$ and $X_j^M$ i.e. $P_{Y, X_j^M\mid X_{j-1}^M}= P_{X_j^M, Y\mid X_{j-1}^M}$.

It can be shown using Proposition 3.4, that the functional $\mathcal{F}$ which satisfies the above conditions is $\mathcal{F}(p, q, r) = - E_{X\sim p}[\log p(X)] - E_{X\sim q}[\log q(X)] + E_{X\sim r}[\log r(X)].$ We hope this clarifies the symmetry property of the first half of Eq. (3).

Basically, the framework's sample-wise information gain is designed to detect errors on a per-sample basis. However, there are instances where a model may accidentally reach the correct final answer, even with incorrect intermediate steps. This can lead to "false negatives," where a faulty chain of reasoning is not flagged because the final answer happens to be correct. And what about false negatives, where a faulty chain of reasoning is not flagged because the final answer happens to be correct? This analysis needs to be presented in the paper.

Next, we would like to highlight that, our proposed method is primarily focused on a mutual information level (average) but can also be applied to a "per-sample basis", similar to V-usable information paper [Ethayarajh et al., 2022]. However, we thank the reviewer for pointing out this special case. We would like to clarify a few things regarding the setup where the LLMs get the correct final answer despite incorrect intermediate steps.

Recall that our method only estimates the relevant information for the correct final answer added by each successive step. Therefore, in cases where an intermediate step is incorrect, it should add no useful information regarding the correct final answer and therefore the information gain at that step should still be 0 (regardless of whether the final output is correct or not). In contrast, in cases where the model indeed consistently commits errors at intermediate steps which then consistently lead to the correct answer, the baselines such as ORM and Math Shepherd would fail to identify such errors as these baselines mainly rely on the correctness of the final answer.

Nevertheless, your question did spark some interest in us and hence we went back to our datasets to check how often this indeed happens and report that in our realistic arithmetic experiments in Section 5.2 only 1.2% of the samples indeed exhibit this behaviour where an intermediate step is wrong, but the final answer is correct. For the synthetic toy experiment, this never happens as we have control over the generation to some degree.

Hence, we would argue that for our settings, these are relatively rare and would not affect our framework. We thank the reviewer again for their insightful comment to improve our paper and will definitely add a discussion on this in the final version of our paper.

Reference: Kawin Ethayarajh, Yejin Choi, Swabha Swayamdipta. Understanding Dataset Difficulty with V-Usable Information, 2022.

Similarly, a lot of time CoT reasoning can hurt performance if the initial step is wrong and then it is hard for the model to recover (https://arxiv.org/abs/2311.07945). In such cases, the steps are considered important if the model can predict the answer correctly, which in many cases is not the case if the start is wrong. There is no analysis done in the paper. Also, the advantage of the approach presented in the paper is to understand and find the errors in the intermediate steps compared to predicting the final answer. This analysis is missing in the paper, especially in the areas where the intermediate errors are analyzed in detail.

The reviewer raises an interesting point, which is that of: "What happens if the model fails to get the initial steps right?". Here, we strongly believe that there is a misunderstanding regarding our experimental setup as we have indeed performed a comprehensive ablation study where we consider settings with errors occurring at each possible step (see Figure 3).

For example, in our toy experiment setting, we have a setup, where we are able to control all 5 subtasks which are all necessary to complete the task. One by one, we then analyse the behaviour of information gain as each of the 5 subtasks becomes "unidentifiable"/wrong. Specifically, $LLM_1$ in our toy experiments (Section 5.1) corresponds to the setting where the initial step of CoT reasoning is wrong while the subsequent tasks are executed correctly.

In the experimental results shown in Figure 3, we observe that information gain becomes negligible (or even negative) as soon as we encounter a wrong step. Additionally, this information gain at subsequent steps remains zero even when the subsequent steps are performed correctly, which is also consistent with our theoretical framework for identifiability outlined in Section 3.

First of all, we thank the reviewer for their comprehensive review to improve our paper. In particular, we appreciate the reviewer acknowledging that our method "provides a novel and rigorous method for evaluating the contribution of each step to the final output". Below, we take the opportunity to clarify some of the concerns and misunderstandings stated in the above review.

I am still unsure about the experimental design, which contradicts some claims of the paper. For example, the use of a supervising model requires a lot of data, which is needed to train the supervising model. This requirement adds computational complexity, requiring additional processing power, memory, and time that the authors claim is not needed compared to the process monitoring models in the introduction.

We believe that there might be a misunderstanding in terms of the comparisons we made in the introduction [lines 38-40]. In particular, the introduction aimed to highlight that existing process reward models required human annotations on the correctness of each of the steps as well as training the subsequent reward model. In our case, we aim to remove the burden of expensive human annotations, while the training requirement remains. We apologize for this misunderstanding and will ensure that this aspect is clarified in the final version of this paper.

In addition, using a supervising model to predict the answer given the previous steps in the chain of reasoning as a way to understand whether the previous steps contain information needed to answer a question may not be correct. For example, many questions can be answered directly by the model in an answer-only setup ("The answer is X"). For these questions, no CoT reasoning is needed. But in this framework, this will not be the case and these steps will be listed as important steps. There needs to be an analysis of questions where the model can directly predict the answers and how this framework addresses those questions. Are intermediate steps needed in such cases?

We thank the reviewer for bringing up this interesting aspect of "answer-only" setup in our framework, i.e. "what is 2+2?". However, we believe that there might be some confusion here. If a question indeed does not require CoT, then there will be exactly a single step in our framework (Prompt -> Final Answer). In this setting, our framework still works (by checking if the predicted answer matched the ground truth answer/label) but would be considered a trivial case. Hence, this paper and our framework primarily focuses on the CoT settings, where multiple steps are required and hence each step adds information if correct.

Could the reviewer please elaborate on their statement "But in this framework this will not be the case and these steps will be listed as important steps.". Given that there are no intermediate steps, our framework will not introduce new steps and hence will also not list them as important. We are happy to discuss further if this point remains unanswered.

We thank the reviewer for bringing up this interesting aspect of "answer-only" setup in our framework, i.e. "what is 2+2?".

Thanks for the response. I am not talking about the cases where the solution can be reached in one step like the example question presented above.

Rather I am talking about a MWP like "John has 10 apples, John gave 5 apples to Jack, ...?". In such cases, CoT is a good choice to solve with the intermediate steps. But the model can also directly say the answer in one step. What about those cases? And also those cases where CoT reasoning is incorrect but the final answer is correct?

I understand that the framework will not introduce extra steps but an analysis is needed around "false negatives," where a faulty chain of reasoning is not flagged because the final answer happens to be correct. In this specific dataset, it is 1.2% as the authors say (cases when reasoning is incorrect but the final ans is correct) but in general it is a problem as a lot of models memorize the answers.