CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective

We sincerely appreciate your recognition of the psychological foundation of our evaluation framework, the extensive and significant experiments, the valuable insights provided by our work, and the well-organization of our paper.

$\bf{Q1}$: If there are no numerical values in the problem (such as in a symbolic computation problem or when the numbers are represented in text), how should the dimension 6 work? and what will the query look like?

$\bf{A1}$ : Thanks for your detailed review and valuable question, and we apologize for causing this confusion. If the problem does not contain numerical values, then our multi-agent system will not generate any inquiries for Dimension 6, as the Inquiry agent cannot produce any reasonable transformation for the problem. As a result, in such cases, as we outline in Appendix C, we omit consideration of that dimension during the evaluation. For example, the problem "If $a<b$, what is the value of $|a-b|+a+b$?" from MATH dataset does not include any numerical values, so it is unnecessary to assess an LLM's ability of numerical calculations on it. In our framework, we can imagine that our multi-agent systems will also fail to provide a reasonable numerical transformation to such a problem. As a result, we would not evaluate Dimension 6 for this problem, which aligns with real-world scenarios and intuition.

Based on your suggestion, we will add more clarifications in Section 3.2 to better explain our evaluation process.

$\bf{Q2}$: This paper highlights potential directions for future improvements in LLMs. Therefore, how do the authors suggest further optimizing each LLM, such as GPT-4?

$\bf{A2}$ : Thanks for your constructive question. Specifically, we can analyze potential optimization strategies for different LLMs based on the conclusions drawn in Sections 4.3 and 4.4.

For example, as shown in Table 2, weaker LLMs (e.g., Llama2-13B) exhibit the lowest Pass Rates in Stage 1 (i.e., Problem Comprehension), whereas more advanced models (e.g., GPT-4, DeepSeek-V2.5) demonstrate relatively stable comprehension abilities but struggle significantly with mastering Stage 2 (i.e., Problem Solving). This suggests that improving comprehension should be a primary focus for weaker models. Further analyzing Figure 2, we observe that Llama2-13B’s main issues in Stage 1 stem from Dimension 1 (Sentence Paraphrasing) and Dimension 4 (Redundant Condition). This insight suggests that incorporating more training data involving synonymous rewrites and redundant conditions could help enhance its comprehension capabilities.

For GPT-4, its main deficiencies in Stage 1 lie in Dimension 2 (Sentence Disruption) and Dimension 3 (Missing Condition), while in Stage 2, it performs poorly in Dimension 7 (Knowledge Redefinition). For Dimensions 2 and 3, as analyzed in Section 4.4, GPT-4's struggles may stem from its tendency to inherently "over-correct" the unsolvable problems into solvable ones. To address this, we could empower it with the critical thinking skills by reinforcement learning techniques, encouraging the model to recognize counterfactual scenarios instead of merely simulating the reasoning process based on a given input. The results in Dimension 7 indicate that GPT4 may tend to treat knowledge as fixed memorization rather than a flexible reasoning process. This suggests the need for more adaptive knowledge learning strategies, such as contrastive learning and retrieval-augmented training, where models are exposed to dynamic and evolving knowledge sources to encourage reasoning beyond static memorization.

Following your comments, we will supplement these discussions to expand the scope of our work in the revised version.

$\bf{Q3}$: Some Typos.

$\bf{A3}$ : Thanks for your meticulous review and pointing out these typos. We will carefully correct them in the revised version.

We sincerely appreciate your recognition of our motivation and solid psychological foundation, the credibility of our experimental results, the significant contributions of our framework, and the good writing of our paper.

$\bf{Q1}$: The decomposition of the reasoning process into three stages could be elaborated further. When humans prove theorems, do the three stages discussed in this paper enough to represent the whole reasoning process?

$\bf{A1}$: Thanks for your constructive suggestion and insightful question. Our decomposition of the reasoning process into three stages is based on psychological research, which points out that humans typically undergo three general stages when reasoning about a mathematical problem: Problem Comprehension, Problem Solving, and Solution Summarization. Building on this, we carefully analyze the objectives associated with each stage in mathematical reasoning and designed corresponding dimensions for evaluation.

Moreover, these three stages represent a general evaluation framework. As discussed in Appendix E, they do not rely on specific problem types or formats and can apply to evaluting LLMs' abilities in a wide range of tasks. For instance, in theorem proving you mentioned, the reasoning process of human learners can also be split into understanding the given conditions (corresponding to our Stage 1), performing the proof (Stage 2), and summarizing the proof (Stage 3). Of course, for specific tasks like theorem proving, additional evaluation dimensions could be introduced. For example, in Stage 2, besides numerical computation, we could add assessments related to symbolic manipulation skills.

In summary, the three stages we utilize in this paper are widely generalizable. Depending on the task at hand, we can also introduce more granular dimensions to conduct more evaluation. Based on your suggestion, we will include these discussions in the revised version.

$\bf{Q2}$: The relationships among the proposed dimensions could be discussed in more detail. The authors seem to consider them independent. Do they try to combine some of them in one query and test LLMs?

$\bf{A2}$ : Thanks for your valuable question. Indeed, in our paper, the design of the different dimensions is relatively independent. However, as mentioned in Section 3, only when a LLM passes all dimensions can we conclude that it has genuinely mastered the problem $P$. Therefore, these dimensions are collectively used to assess whether the model truly masters mathematical reasoning.

We certainly appreciate your suggestion to combine different dimensions in a single query, which could provide us with valuable insights into the model's ability to handle multiple dimensions simultaneously. However, this approach may present challenges in assessment in the cognitive stage level, and it could be difficult to trace which specific dimension the model is struggling with. For this reason, we have chosen to evaluate the dimensions independently in our primary experiments.

However, we believe that your suggestion is very promising, and we are willing to explore this direction in future work.

$\bf{Q3}$: Why do they choose the clause level, rather than directly disrupt the whole problem?

$\bf{A3}$ : Thanks for your thoughtful question. We agree that disrupting the whole problem is also a possible approach. In this paper, we chose to disrupt the problem at the clause level because it allows our inquiries to maintain a closer similarity to the original problem's structure while ensuring that they remain unsolvable. As a result, if the model does rely on semantics for reasoning, our approach makes it easier to observe such behaviors. Based on your concern, we will add further clarification about it in the revised version.

$\bf{Q4}$: Typos and does "MathExam" mean "MExam" in Figure 2?

$\bf{A4}$ : Thanks for your meticulous review and pointing out these typos. We will carefully correct them in the revised version.

We sincerely appreciate your recognition of our framework's novelty, the validity of our conclusions, and the insights of our work.

$\bf{Q1}$: Validity of Dimension 6.

$\bf{A1}$: Thanks for your insightful comments. First, the strategy of changing numbers in problems has been widely adopted in assessing learners' abilities. For example, human educators commonly pose numerical variations in exam questions to assess students[1,2]. Similarly, it is widely employed in evaluating a model's mastery for math problems[3,4]. Thus, our Dimension 6 is meaningful for LLM evaluation.

Second, we do not assume that "LLMs' problem-solving abilities would not be affected by the change in the calculations". Instead, if a model's performance greatly declines after numerical variations, our framework can identify such deficiencies in numerical processing capabilities, which indeed aligns with your opinions that LLMs struggle with fundamental computations.

Third, in Dimension 6, our Inquiry agent does not introduce extreme cases (e.g., a huge number) that might exceed current LLMs’ processing limits (an example is shown in Table 6). Therefore, our dimension will not significantly increase the difficulty of the problem, so we do not expect any noticeable decline in model performance.

[1] How to solve it: A new aspect of mathematical method.

[2] Cognitive diagnostic assessment for education: Theory and applications.

[3] GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers.

[4] Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap.

$\bf{Q2}$: If all words are jumbled, Dimension 2 may not assess memorization accurately?

$\bf{A2}$: Thanks for your thoughtful question. Our goal in Dimension 2 is not to evaluate the memorization capacity of LLMs. Instead, we aim to assess whether the LLM is truly reasoning or merely relying on the semantic cues in the wording. After randomly shuffling the sequence of words, humans can naturally recognize that the problem becomes unsolvable. Therefore, if an LLM truly master reasoning, it should also recognize this. This is why in Section 3.1 and in Table 6, we clarify that an LLM is considered to have "passed" Dimension 2 if it successfully recognizes the inquiry as “unsolvable”, rather than whether its answer is correct (please also note that the shuffled problem does not have a well-defined correct answer!).

$\bf{Q3}$: Dimensions 2 and 3 evaluate memorization only? Strategy of considering answer?

$\bf{A3}$: Thanks for your insightful questions. As we clarified in $\bf{A2}$, Dimension 2 is not designed to evaluate the memorization, so do Dimension 3. Our goal is to assess whether the LLM is engaging in genuine reasoning, regardless of whether it is responding with memorized information or other capabilities.

In both dimensions, an LLM is considered to pass if it correctly identifies the given inquiry (i.e., $q_2$, $q_3$) as “unsolvable”, rather than considering any specific answers. Therefore, our Pass Rate criteria refers to the proportion of cases where the model successfully recognizes such ill-posed inquiries.

$\bf{Q4}$: Length thresholds.

$\bf{A4}$: Indeed, we do not set fixed thresholds. As stated in Section 4.6, we divide all problems into five levels using an equal-frequency binning approach. This means that we sort all problems by length and then divide them into five equal-sized groups. This ensures that each group contains sufficient data and avoids introducing any potential biases, allowing for more reliable analysis.

$\bf{Q5}$: Information about MExam.

$\bf{A5}$: Thanks for your valuable question. In this work, our goal is not to create a new dataset, MExam, but rather to utilize it to validate that the overestimation of LLM capabilities is not merely due to data contamination (discussed in Section 4.2). Thus, we explain in Appendix C how we collected MExam and the number of problems it contains.

To address your concern, we conduct additional statistical analyses. Due to space limit, please refer to our response $\bf{A4}$ to reviewer $\bf{Xd5A}$ for details. We will also make MExam publicly available if this paper is accepted.

$\bf{Q6}$: Suggestions on writing.

$\bf{A6}$: Thanks for your constructive feedback. We will incorporate the related studies and refine the writing in the revised version.

$\bf{Q7}$: Ethics review.

$\bf{A7}$: Thanks for your attention to ethics. In this work, we invited human annotators to evaluate the outputs of our Judge agents and Reference agents. As stated in Section 4.7, the evaluation protocol was approved by the Ethics Review Board, and all annotators were informed of data usage. Besides, our templates (Appendix B) do not collect private information. Thus, our study does not raise ethical concerns.

We greatly appreciate your in-depth reviews and hope our explanations addresses your concerns. We will also include them in the revised version.

We sincerely appreciate your recognition of our framework's soundness, evaluation significance, and great contributions.

$\bf{Q1}$: Adaptability to more problem types.

$\bf{A1}$: Thanks for your insightful question. Our CogMath is easily adaptable because: (1) it is based on the decomposition of human reasoning processes. The three stages, Problem Comprehension, Problem Solving, and Solution Summarization, reflect the general cognitive processes humans use to conduct reasoning. (2) our agents are highly flexible. As shown in Appendix B, their prompts do not impose any specific requirements on the problem types. Besides, their interactions ensure the quality of the generated inquires and reference answers (verified in Section 4.7). (3) our evaluation metrics are reasonable and easy to use, considering both the general "answer correctly" and "unsolvable" in counterfactual scenarios.

$\bf{Q2}$: Evaluation on new LLMs (e.g., DeepSeek-R1).

$\bf{A2}$: Thanks for your constructive suggestion. We supplement results for DeepSeek-R1 below. Due to API speed limitations, here we conduct evaluations on the widely-used public MATH and GSM8K datasets.

First, DeepSeek-R1 achieves the best performance compared with other LLMs in Table 1, both in “Vanilla” and CogMath framework. This reflects its superior mathematical reasoning capabilities. Second, the performance gap (marked as $\Delta$) suggests that DeepSeek-R1 still exhibits a certain degree of overestimation, highlighting the necessity of our proposed evaluation from the human cognitive perspective. Third, similar to other advanced LLMs, DeepSeek-R1 encounters the most challenges in Stage 2 (Problem Solving). Further investigation reveals that its primary weakness lies in Dimension 7 (Knowledge Redefinition), with a Relative Pass Rate (RPR) of 0.617. This supports the conclusion that current LLMs rely on fixed memorization rather than adapting knowledge flexibly. Lastly, DeepSeek-R1 improves significantly in Stage 3 (Solution Summarization) compared to DeepSeek-V2.5, which suggests a deeper understanding of the reasoning process.

$\bf{Q3}$: How to evaluate after exceeding $\delta$ rounds?

$\bf{A3}$: Thanks for your valuable question. As illustrated in Appendix C, if inquiry quality remains insufficient after $\delta$ rounds, we exclude the dimension because it suggests that the problem may not be suitable for evaluation from that dimension. For example, the problem "If $a<b$, what is the value of $|a-b|+a+b$?" from MATH dataset does not include any numerical values, so it is unnecessary to assess an LLM's ability of numerical calculations on it. Our multi-agent system would also fail to generate a valid transformation, so we would not evaluate Dimension 6, aligning with real-world intuition.

$\bf{Q4}$: How to optimize LLMs (e,g, enhance capability in Poblem Comprehension stage) during "pretrain-SFT-RL" training?

$\bf{A4}$: Thanks for your insightful question. Different phases of training correspond to enhancing different cognitive stages/dimensions of LLMs. For example, the pretraining phase focuses on developing the model's text comprehension abilities and the mastery of fundamental knowledge. The SFT phase is more about teaching the model to simulate a given reasoning strategy. The RL stage allows the model to develop more complex abilities, such as backward reasoning, intermediate step explanations, and error identification.

Therefore, in Problem Comprehension stage, we suggest: 1) For Dimension 1 (Sentence Paraphrasing), increase the corpus used during the pretraining phase. 2) For Dimension 4 (Redundant Condition), train the model with question-answer pairs that include redundant information in SFT phase. 3) For Dimensions 2 (Sentence Disruption) and 3 (Missing Condition), to cultivate critical thinking skills, we can allow the model to think more freely and learn to recognize such situations in the RL phase.

Similarly, for the other cognitive stages, improvements can be made through different training processes. For example, knowledge acquisition in the Problem Solving stage can be reinforced during pretraining, while backward reasoning abilities in the Solution Summarization stage are often enhanced through the RL phase. Overall, our framework provides valuable insights into how to optimize LLMs in different stages.

We greatly appreciate your thought-provoking suggestions and will include these experiments and discussions in the revised version.

We sincerely appreciate your affirmation of our framework's novelty, experimental validity, and evaluation significance.

$\bf{Q1}$: Could include more deepseek version.

$\bf{A1}$: Thanks for your valuable suggestion. We will incorporate more references to DeepSeek-related papers. Besides, we conduct experiments on DeepSeek-R1 as follows. Due to API speed limitations, here we conduct evaluations on the widely-used public MATH and GSM8K datasets.

First, compared to Table 1, DeepSeek-R1 achieves the best performance among all evaluated LLMs, both in “Vanilla” and CogMath framework. This shows its superior mathematical reasoning capabilities. Second, based on the performance gap (marked as $\Delta$), DeepSeek-R1 still exhibits a certain degree of overestimation, highlighting the necessity of our proposed evaluation from the human cognitive perspective. Third, similar to other advanced LLMs, DeepSeek-R1 encounters the most challenges in Stage 2 (i.e., Problem Solving). Further analysis reveals that its main weakness lies in Dimension 7 (Knowledge Redefinition), with a Relative Pass Rate (RPR) of 0.617. This supports the conclusion that current LLMs rely on fixed memorization rather than adapting knowledge flexibly. Fourth, compared to DeepSeek-V2.5, DeepSeek-R1 improves significantly in Stage 3 (Solution Summarization), suggesting a deeper understanding of reasoning process.

$\bf{Q2}$: Scalability of CogMath.

$\bf{A2}$: Thanks for your constructive comments. CogMath is highly scalable because: (1) it is designed based on human reasoning processes, which makes the three stages independent of any specific problem types. (2) our multi-agent system is highly flexible. As shown in Appendix B, they do not depend on the dataset or task definition. Besides, their interactions ensures the quality of our inquiries and reference answers (verified in Section 4.7). (3) our evaluation metric is widely applicable, considering both general “answer correctly” and “unsolvable” in counterfactual situations.

$\bf{Q3}$: Definitions of Pass Rate.

$\bf{A3}$: Thanks for your valuable question. As explained in Section 4.1, Pass Rate for Dimensions 1 and 4-9 refers to the accuracy of answering the inquiries correctly. For Dimensions 2 and 3, they are counterfactual evaluation dimensions. For example, Dimension 2 evaluates the LLM’s response after the words of the original problem are randomly shuffled (an example is in Table 6). Ideally, this will render the problem meaningless, and the LLM should not provide the original answer. Thus, in this case, Pass Rate refers to the proportion of cases where the LLM successfully identifies the inquiry as "unsolvable".

$\bf{Q4}$: Statistical analyses on datasets.

$\bf{A4}$: Thanks for your constructive suggestion. We present the number of problems (#P), the average problem length (Avg.P), and the average answer length (Avg.A) of the original dataset, along with the average inquiry length (Avg.$q_i$) and the average answer length (Avg.$a_i$) for each dimension $i$ in our framework. Since the reference answers $a_i$ for Dimensions 1 to 4 are same to the original answers and the inquiry $q_2$ for Dimension 2 is simply a shuffled version of the original problem, no additional statistics are required for these cases.

We observe that the inquiries in Dimensions 4, 7, and 8 exhibit the most significant length differences compared to the original problems. This is expected, as they ntroduce additional conditions, redefine knowledge concepts, or only ask about one specific intermediate step. Furthermore, in most cases, the reference answers are longer than the original answers. Upon further inspection, we find that this is not due to an increase in problem difficulty, but rather stems from our Reference agent providing a more detailed solution, whereas the original dataset answers are more concise.

We sincerely appreciate your thoughtful comments and will incorporate these experiments and discussions in the revised version.