Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMs

We thank the reviewer for their appreciation of the writing and experiments. Here we address their concerns

Typos - Thanks for bringing those to our notice, we will definitely fix those in the final version.

concern 1 - Dataset is a bit small, no external datasets used

response - To further provide proof that our method is effective we provide results on the popular Processbench dataset, which has step level annotations done by humans.

Gsm8k (400 examples)

Math (1000 examples)

concern 2 - the baseline performance of these models is missing

response - In our paper we have two tasks - premise identification and error identification. The former is a novel task in itself, hence we don’t explicitly compare against any baseline as such. Instead we compare the aggregative and dyadic approaches as an ablation to how premises can be identified. For the error identification task, the baseline is when the entire context is fed into the LLM as compared to our approach where we only use the premises generated (which is a standard LLM as a judge scenario, please refer to the Table 3 the rows tagged "full context" is the baseline here). We would also like to highlight that existing popular frameworks like Roscoe and Receval assign a chain level scores, and hence are not directly comparable to our use case.

concern 3 - discussion of the additional compute resources, time and costs

response - As we query the judge LLM twice we observe a 2X latency as compared to the baseline. We empirically verified this for the Qwen32b model, across all 4 datasets.

question 1 - Why do longer reasoning chains make it harder to verify individual steps?

response - The longer a reasoning chain gets, the longer the context is for the later steps. This further implies that a lot of irrelevant information (non-premises) is fed into the context, making it hard for the model to reason whether the step was correct or not.

question 2 - If the nodes are the reasoning steps, how can the edges link to premises?

response - In our formulation in the DAG - each node is a step, and directional edges capture dependencies between steps (nodes). More specifically, if a step i is premise to step j, there is a step (i->j)

question 3 - Why are there more PARCs than chains in the dataset? (A2)

response - Indeed it should be 607 and we will correct this in our final paper

We thank the reviewer for their kind remarks on the writing and experimental details of the paper. Here we address their concerns

concern 1 - Study is limited to the four popular mathematical datasets, and three commonly used LLM

response - Please note that we used three popular model families, and for each family we included at least 2 models of different scales, making the number of models=6. Which establishes the robustness of our method across models. Further here we attach some additional experiments done on the recently released processbench dataset for the error identification task, where across model classes we see significant improvements in error identification, further showcasing our method’s reliability

Gsm8k

Math

concern 2 - how large is the "other" group, and what's inside

response - We observed that the frequency of the other error type in our ground truth dataset was quite low, where only 10 steps were marked as Other. Upon manual inspection, it revealed that the other group is manifested in the forms of - “The solution becomes stuck in a repetitive loop” and “Step is incomplete ”.

concern 3 - In particular, it would be interesting to see how each aspect - mathematical error, logical inconsistencies, and accumulation errors - affects final error detection

response - Since we are restricted by 5000 characters, we request the reviewer to kindly see the response to the third concern by reviewer xai3, where we shared the detailed numbers.

concern 4 - Authors highlight the importance of newly introduced accumulation errors, but do not provide evidence in their study

response - A holistic evaluation of LLM reasoning should consider the entire reasoning chain rather than relying solely on a binary correct/incorrect outcome of the final answer. Reasoning chains may contain subtle intermediate errors despite following a globally correct plan, ultimately rendering the solution incorrect—a nuance overlooked by final-answer correctness metrics. Prior works like PRM800K and ProcessBench have typically annotated reasoning chains only up to the first erroneous step, discarding subsequent steps due to ambiguity. To our knowledge, we are the first to formally introduce the concept of accumulation errors, enabling a more comprehensive evaluation of reasoning chains. Similar to how teachers award partial credit for nearly correct answers, evaluation frameworks should recognize when the overall reasoning plan is sound despite minor mistakes, assigning partial credit accordingly. Accumulation errors, where a step is locally correct but built on flawed premises, explicitly capture this subtlety. Identifying accumulation errors highlights how earlier mistakes compromise the reliability of the reasoning chain, making it essential to incorporate these errors into holistic scoring methods.

We sincerely thank the reviewer for their kind remarks on the novelty and thorough experimental setup of our method. Here we try to address their concerns.

concern 1 - Since the premise is based on the step, it would be better to make it clearer what is an intermediate step defined, and how to get the step list from a complete CoT.

response - In our work, we explicitly prompt the generator model to answer in a step by step format, with formatting instructions in the prompt. The generated solution looks like Step 1: … Step 2: … Finally we do a simple regex to extract the steps.

concern 2 - How did the authors ensure or evaluate whether the premise satisfied the three properties in page 3.

response - In our work, we prompt the highly capable O1-preview model to generate the ground truth premises and later manually verify the generated premises on whether they satisfy the verifiability and minimality conditions.

concern 3 - What’s the necessity of the proposed accumulation error?

response - A holistic evaluation of LLM reasoning should consider the entire reasoning chain rather than relying solely on a binary correct/incorrect outcome of the final answer. Reasoning chains may contain subtle intermediate errors despite following a globally correct plan, ultimately rendering the solution incorrect, a nuance overlooked by final-answer correctness metrics. Prior works like PRM800K and ProcessBench have typically annotated reasoning chains only up to the first erroneous step, discarding subsequent steps due to ambiguity. To our knowledge, we are the first to formally introduce the concept of accumulation errors, enabling a more comprehensive evaluation of reasoning chains. Similar to how teachers award partial credit for nearly correct answers, evaluation frameworks should recognize when the overall reasoning plan is sound despite minor mistakes, assigning partial credit accordingly. Accumulation errors, where a step is locally correct but built on flawed premises, explicitly capture this subtlety. Identifying accumulation errors highlights how earlier mistakes compromise the reliability of the reasoning chain, making it essential to incorporate these errors into holistic scoring methods.

concern 4 - it may be problematic to treat the CoT as correct based on the final answer, due to possible step errors as studied in the paper.

response - We completely agree with the reviewer on this - in our work, we observed that even in the positive reasoning chains, there were a few steps that were annotated as error by the O1 model, indeed LLMs suffer from False Positives, which is established from the work - https://arxiv.org/pdf/2502.06217 . In GSM8k - 4 false positives out of 50 In MATH - 4 out of 50 Metamathqa - 5 out of 50 Orcamath - 4 out of 50 When we release our dataset, we will make sure to flag them as false positives.

We thank the reviewer for their kind remarks on the novelty and thorough experimental design. Here we address the concerns

concern 1 - low precision of premise identification and mapping

response - We would like to highlight that our claim is originated by the high recall primarily. In the context of error identification, verifiability (recall) is more important than minimality (precision), since missing even a single premise can compromise the step’s and subsequent steps' verifiability. Further, upon closer inspection we observe that, low precision often results from the ground truth premise set being very small. For example, if step 7 has ground truth premises {1} and the model predicts {0,1}, precision is 50%, even though the context remains well-pruned.

concern 2 - lack of theoretical guarantees

response - We primarily take an empirical route, and as already pointed out by you, we do thorough experiments across model families, scale and datasets to prove robustness of our method. We hope that the released dataset can help the community to foster more research in this direction as well. To further provide proof that our method is effective we provide results on the popular Processbench dataset, which has step level annotations done by humans. Please refer to our response to concern 1 raised by Reviewer kjS7 for the Processbench results.

concern 3 - Mathematical Error and Logical Inconsistency are merged

response - When we say that the boundary between them is “thin”, we imply that even a simple mathematical calculation (or manipulation) could easily be considered as a logical error (due to fault in mathematical logic). However we agree that for completeness, it's essential to have a detailed breakdown per error category, which we present here. ( we will include these in the final version of the paper)

GSM8k

MATH

concern 4 - Graph-of-Thoughts [1] also disc…

response - Thanks for raising this. Graph of thought induces structure at inference time, while we do it after inference, and later use that to improve error identification. But there is definitely a resemblance between them, and we will include them in our camera ready version.