Not All LLM Reasoners Are Created Equal

Thank you for the insightful and valuable comments.

Have you formally summarized the common types of mistakes? Have you analyzed the mistake patterns and behaviors of small versus large models? Including qualitative case analyses on page 10 could provide a more detailed understanding of model mistakes.

Thank you for the suggestion. We examined random subsets of question-responses from Qwen2.5-MATH-IT (7B and 72B) and LLAMA3-IT (8B and 70B) models and categorized mistakes into three groups of Semantic Misunderstanding, Calculation Error and Step-missing error (following Zhong et al. (2024)) either on Q1 or Q2. Here is the figure. We find that smaller models miss more reasoning steps, and make more calculation errors. Larger models make more semantic and logical errors, although the questions are more difficult. We also include an example of such errors in this figure. Additionally, our analysis shows that less than 1% of the total responses have invalid or unextractable final answers across models.

In the setup, did you combine different Q1s with a fixed Q2? It seems that the authors used a one-to-one mapping for constructing compositional test questions.

The selection of Q1 for a given Q2 is designed to ensure that the new final answer of Q2 remains a positive integer close to the original answer. We randomly choose from a pool of suitable Q1s that satisfy this condition.

How is the value of X parsed?

The compositional 8 shot prompt is shown in Appendix E. The value of X is parsed from the phrase “The Q1 answer is” in the outputs.

References:

Zhong et al. (2024), Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems

We appreciate your valuable feedback.

it showed cost-efficient LLMs perform badly on compositional GSM8k, but does not try to probe into the reasons behind this with more detailed error analysis or case studies. Can you perform more detailed error analysis?

Our objective with this study is to use the compositional GSM task as a case study to highlight differences in reasoning capabilities of various LLMs, rather than to focus extensively on the task itself. We aimed to provide a broad perspective to inform future work and effort in this area.

For your convenience, we have included the results of our new analysis here (also in the revision):

Error distribution on Question 1:

Error distribution on Question 2:

We analyzed random subsets of question-response pairs from LLAMA3-IT (8B and 70B) and Qwen2.5-MATH-IT (7B and 72B) models, categorizing errors into three types: Semantic Misunderstanding, Calculation Errors, and Step-Missing Errors, as outlined in Zhong et al. (2024), for both $Q_1$ and $Q_2$. Our findings indicate that smaller models tend to miss more reasoning steps and make more calculation errors. In contrast, larger models are more prone to semantic and logical errors, though the questions they struggle with are typically more complex and not necessarily the same across models.

The overall error rates on Compositional GSM for the models are as follows:

We also include an example of such errors in this figure. Additionally, our analysis shows that less than 1% of the total responses have invalid or unextractable final answers across models.

it fails to discuss other popular math reasoning datasets such as MATH which involve more complex and diverse reasoning. The authors have mentioned this as a future work, but having not discussed MATH weakens the overall contributions of the paper.

We chose GSM8K due to its widespread popularity, and the recent claims that LLMs have mastered grade-school math reasoning. We did highlight that our method is extensible to other tasks and benchmarks, including MATH, or even multimodal reasoning tasks. Our analysis on GSM8K has already revealed lots of new valuable insights into the capabilities and limitations of LLM reasoners. Notably, we find that smaller, cost-efficient, and math-specialized models show larger reasoning gaps (recognized by R-EFFT, R-vgh2), task specific finetuning can lead to task overfitting (recognized by R-EFFT and R-iMpx), and instruction-tuning affects models of different sizes differently (recognized by R-EFFT). If a model struggles with Compositional GSM problems, it's reasonable to expect it will also struggle with Compositional MATH questions given that MATH problems are more difficult.

References:

Zhong et al. (2024), Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems

We hope we've addressed your concerns and kindly ask that you reconsider your assessment. We're happy to continue the discussion if needed.

Thank you for your time and valuable insights; we hope we've addressed your concerns and comments, and just wanted to send a gentle reminder.

Thank you for your thoughtful and insightful comments.

there are already several benchmarks for assessing the robustness of LLMs in math reasoning. Although this paper includes extensive experiments, the conclusion that LLMs struggle with multi-hop reasoning is not particularly surprising to me.

Our analysis on compositional GSM goes beyond simply highlighting LLMs’ challenges with multi-hop reasoning. We uncover notable performance differences among models that show near-identical results on the original GSM8K benchmark. This variation in performance is particularly intriguing, and in many cases, surprising. For example, we find that smaller, more cost-efficient LLMs exhibit distinct differences in their learning dynamics and reasoning shortcomings, despite having nearly identical training setups and comparable results on standard benchmarks. Rather than introducing yet another benchmark, our goal is to provide insights into the nuanced behaviors of different models, fostering deeper exploration and inspiring future research.

The two-hop QA format appears to be a minimal approach for testing mathematical compositional reasoning abilities. I was hoping for a more intricately designed benchmark, similar to SCAN or CFQ.

We selected GSM8K for this study due to its widespread recognition as well as the recent claims suggesting that LLMs have successfully mastered grade-school math problem-solving. Our compositional GSM study offers a more realistic and streamlined approach, and has already revealed lots of new valuable insights. GSM8K questions already require a good amount of intricate reasoning from the model, so a composition of these questions will inherently involve similar complexity and reasoning abilities by design.

The authors argue in Line 477 that "This raises the question of whether small and cost-efficient models are fundamentally limited in their ability to achieve such generalizations" and suggest we should rethink how we assess “reasoning". Can you elaborate on your insights and how we can achieve this?

We noted that mathematical reasoning is inherently contextual and compositional, but existing evaluation methods tend to focus on task performance, which can be a superficial measure of a model’s reasoning abilities. For instance, GPT-4o and GPT-4o mini have very similar GSM8K accuracies (95% vs 92%), but very different compositional GSM performances (88% vs 68%). Our work highlights these limitations and contributes to the deeper exploration of LLMs’ reasoning capabilities. To truly assess reasoning, we must go beyond simple performance scores and consider factors such as the model's ability to handle complexity, context and compositionality. This shift in focus could provide more meaningful insights into the depth and nuance of LLMs’ reasoning capabilities.

We hope that most of your concerns have been addressed and, if so, we would appreciate it if you could reconsider your assessment. We’d be happy to engage in further discussions.

Thank you once again for your valuable feedback on our work. We've carefully worked to address your comments and questions. Are there any further questions or concerns we should discuss?

Thank you for your reply.

The Gemma family is trained using a different approach compared to the LLaMA 3 family, with notable differences not only in training recipes but also in architecture. Key differences in their training recipes include variations in post-training procedures, datasets, reward models for RLHF, and choices between distillation or training from scratch, among other factors. For more detailed information on each family's training methodology, please refer to their respective technical reports (Gemma team et al., 2024; AI@Meta, 2024). These distinct training choices contribute to the differences in performance observed across existing benchmarks, including our compositional GSM task.

I also found [1] studied the robustness of evaluating GSM8K

To further analyze the differences of small and cost-efficient models, we examined random subsets of question-responses from Qwen2.5-MATH-IT (7B and 72B) and LLAMA3-IT (8B and 70B) models and categorized mistakes into three groups of Semantic Misunderstanding, Calculation Error and Step-missing error (following Zhong et al. (2024)) either on Q1 or Q2. Here is the figure.

We find that smaller models tend to miss more reasoning steps, and make more calculation errors. Larger models make more semantic and logical errors, although the questions they fall short on are more difficult, and not necessarily the same questions. We also include an example of such errors in this figure. We will include these figures in the paper as well.

References:

(AI@Meta, 2024) Llama 3 model card. 2024. URL https://github.com/meta-llama/ llama3/blob/main/MODEL_CARD.md

(Gemma team et al., 2024) T. M. Gemma Team, C. Hardin, R. Dadashi, S. Bhupatiraju, L. Sifre, M. Riviere, M. S. Kale, ` J. Love, P. Tafti, L. Hussenot, and et al. Gemma. 2024. doi: 10.34740/KAGGLE/M/3301.

We are happy to hear that all your concerns have been addressed, and we sincerely appreciate you increasing your overall rating.

-- Authors

Thank you for your insightful feedback.

While they show poor performance on the second question even when the first is solved correctly, they don't establish whether this definitively indicates a reasoning gap versus other potential issues like prompt sensitivity, instruction following ability, or other format issues.

Figure 12 compares models’ abilities to answer a question in different contexts. In the setting where Q1 and Q2 are both in the context, but are independent, models show minimal issues. This is an almost identical format to the setting where Q2 depends on Q1. Therefore models do not encounter any formatting issues in these settings. Additionally, our analysis shows that less than 1% of the total responses have invalid or unextractable final answers across models. As noted in Section 3, some models required a preamble before the 8-shot prompt to properly follow instructions. We tested both configurations (with and without the preamble) and reported the best performance for each model.

Even though we see the dropping performance, what does it mean in reality? In what kind of real scenarios, the users would have such evaluation settings like that?

Many real-world use cases involve compositing multiple skills, and if LLMs struggle with even the simplest of these scenarios, they are likely to encounter significant challenges with more complex ones. This evaluation also complements testing on more challenging benchmarks, which, based on our results with math-specialized models, may have contamination issues.

Could you repurpose the GSM8K training set to something like how you constructed the test set? Would such a simple fine-tuning method solve the issue?

Finetuning on specific examples, like our compositional GSM, might help models address this particular issue, but that would miss the border purpose of our work. One could design additional similar tests to uncover other shortcomings. Relying on specific finetuning to cover every possible case is not a feasible approach.

The evaluation was only conducted on GSM8K dataset, which limited the conclusion's validity to other reasoning scenarios.

While this is a reasonable limitation, our work nonetheless reveals important and surprising differences between models that perform similarly on GSM8K. Notably, influential works such as Zhang et al. (2024) and, concurrently, Mirazeh et al. (2024), among many, examine the grade-school mathematical reasoning capabilities of LLMs--an active area of research. If a model struggles with Compositional GSM problems, it is reasonable to expect it will also face difficulties with similar compositional setup of more difficult benchmarks such as MATH.

The paper claims to examine math-specialized models but tests a relatively narrow set (Numina-7B, Mathstral-7B, Qwen2.5). Some more open-source and open-weight math models might be added for a more comprehensive analysis. e.g., deepseek-math, MetaMath, MAmmoTH, etc.

We selected Qwen2.5 Math (as well as Numina & Mathstral) models because they represent the state-of-the-art math specialized performance for their sizes. We’ve extended our analysis to MetaMath-Mistral-7B and MAmmoTH2-8B-Plus both underperforming Qwen2.5 in both settings.
Here are the results:

References:

Zhang et al. (2024), A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Mirazeh et al. (2024), GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

We hope that we've addressed most of your concerns. If so, we kindly ask that you reconsider your assessment. We would be glad to continue the discussion if needed.

Thank you for your time and valuable insights. We hope to have addressed your questions regarding real-world scenarios, and we would be glad to continue the discussion if needed.