GSM-$\infty$: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?

Concern about Practicality of GSM-Infinite

We appreciate the reviewer’s concern about the practicality of GSM-Infinite, as the problem is crucial for understanding why we believe GSM-Infinite is helpful for researchers to use. We want to kindly rebut that first, different from “many synthetic methods for expanding seed questions into long-context reasoning tasks with controllable quality,” GSM-Infinite introduces two important techniques to offer important additional edges over past benchmarks.

First, unlike prior work that expands GSM-8K seeds, our test examples are generated entirely from scratch using graph-based generation—no GSM-8K questions or data are used (Section 3, Pages 3–5). GSM-8K is saturated in both quantity and difficulty (Figure 5(a), Page 4), with only ~10 examples per op>8. In contrast, GSM-Infinite offers >500 test cases per op, scaling up to op=200+, saturating SOTA LLMs.

Second, prior long-context reasoning benchmarks from "expanding seed" can often have noise filtered out by retrieval (RAG), making them poor tests of long-context LLMs (Figure 4(a), Page 4). Our graph-based expansion produces problems where noise is indistinguishable from essential info (Figure 4(b), Page 4), rendering RAG ineffective. GSM-Infinite scales both reasoning and context length while being fully LLM/human-free.

Moreover, we believe that GSM-Infinite offers the reasoning community a new tool that evaluates LLMs solely on reasoning with complex graph traversal, without testing them with memorization of subject-specific knowledge. Recent LLMs often report metrics from MATH or AIME datasets, since prior datasets such as GSM-8K have been saturated. Although these datasets are of strong reasoning complexity, solving test examples convincingly requires a lot of math-related theorem knowledge in LLMs’ memory. We show in (Table 1, page 7) that the powerful reasoning models also see their performance decaying to zero as we gradually increase the ops. GSM-Infinite offers an alternative reasoning benchmark that is drastically less reliant on LLMs’ memory, while evaluating all LLMs based solely on their reasoning ability of traversing a complex computational graph problem with fine-grained difficulty control for the study.

Lacks to infinite dataset

Thank you for this insightful suggestion. Our current implementation, GSM-∞, focuses specifically on modeling the explicit arithmetic operations (+,-,×,÷) and implicit hierarchical relationships found in GSM-8K using computational graphs. The primary goal was to create a scalable benchmark for grade-school math reasoning with controllable complexity and context length.

While the graph representation is potentially applicable to a broader range of problems, our current work does not aim to build a universal "infinite" dataset covering all reasoning types. Extending this methodology to model other complex structures, like code syntax trees or spatial relationships, is a promising direction for future research.

Dataset Quality Inspection and Overview

Questions were constantly evaluated during development via manual inspection and by feeding them to SOTA LLMs to ensure they were natural, solvable, and LLM-understandable. SOTA models demonstrate high accuracy (>88% for o3-mini on Hard op 30) on zero-noise, lower-op problems, indicating fundamental understandability.

Errors observed stemmed from model reasoning limitations (e.g., confusing concepts, hallucination, misinterpreting relationships), not ill-formed prompts. We collect examples in https://docs.google.com/document/d/1WP3ygB67yNUS-iliSYYjFRY4Ls81yIVcRVxpbTVTZ0o/edit?usp=sharing Part 4.

Moreover, we present a table for an overview of the composition of the dataset (submitting version).

The amount of variation permitted in each of our templates is different, richest for "Crazy Zootopia" and slightly more limited for the other two templates. We use an un-even partition of templates for the ensemble of our dataset. We are actively expanding the number of available templates for GSM-Infinite, now made 10 (much more than 3 when submitting), and will be released once published. Please notice that the template proportions are fully reconfigurable, and switching to a different template won’t leads to large difference in performance. For detailed ablation experiments of templates, please refer to (Appendix F, page 19-20).

Additional figures and tables in the link (anonymous): https://docs.google.com/document/d/1WP3ygB67yNUS-iliSYYjFRY4Ls81yIVcRVxpbTVTZ0o/edit?usp=sharing (referred to as Sup)

1. No ablation on different graph structures.

Thank you. We agree studying graph structure impact is valuable. We ran Qwen 2.5 7B IT on 40k Symbolic (op=9) to analyze the relationship of graph depth and accuracy. Depth is the max node distance from the root; mean depth is the average across nodes. We categorized samples by max depth (Depth=2 to 6).

Results confirm deeper graphs are harder. Accuracy decreases as depth increases:

Further analysis shows accuracy decreases as mean depth increases. When mean depth is similar, max depth has little impact, suggesting overall complexity (mean depth) is the primary factor. A plot confirms this trend across depth settings: Sup Part 1.

2. Limited Analysis between models. Below data from Hard subset. Acknowledged. Based on Table 1 (p7):

• Reasoning Models: Show significant gains over non-reasoning counterparts with identical parameters/architecture. |op|DeepSeek R1|DeepSeek V3|Qwen QWQ 32B Preview|Qwen 2.5 32B Instruct (non-reasoning)| |---|---|---|---|---| |10|.95|.79|.64|.61| |20|.94|.60|.60|.36| |30|.95|.36|.26|.12| |40|.92|.16|.13|.04| |AUC|8573.8|2407.9|2846.2|1399.8|
• Model Size: Within families, larger models perform better. |op|Qwen2.5 72B Instruct|Qwen2.5 7B Instruct|Llama 3.1 70B Instruct|Llama 3.1 8B Instruct| |---|---|---|---|---| |5|.95|.59|.73|.52| |10|.82|.29|.64|.40| |15|.76|.18|.49|.19| |20|.64|.05|.33|.07| |25|.49|.03|.26|.06| |30|.37|.00|.04|.02| |AUC|2016.375|618.500|1205.250|606.500|
• Architecture: Current hybrid models (Linear Attention/SSM) lag behind comparable Transformer models. |Models|Architecture|ParameterSize|AUC| |---|---|---|---| |Qwen-2.5-32B-Instruct|Transformer|32B|1405.055| |MiniMax-Text-01|Hybrid (Linear Attention)|456BMoE(45.9B/token)|1178.510| |Jamba-1.5-Large|Hybrid (SSM)|398BMoE(98B/token)|466.400|

3. No cost discussions.

We compute the cost for Llama-3.1-70B below as a reference. We evaluated 100+ examples per op across 3 subsets (use 100). We evaluated up to 40-55 ops per subset (sampling frequency decreases after op 30). Total zero-noise evaluation (~9300 examples) requires ~9.8M input / ~39.2M output tokens. Using competitive (DeepInfra) API pricing, costs are:

Larger/reasoning models cost more. With additional tables and elaboration in Sup Part 2., found using strides (evaluating every N ops) up to 4 has minimal impact (<3% AUC change) on results. Reducing samples per op (e.g., 100 or 50) also yields similar AUC, allowing further cost reduction.

4. Need more detailed graph generation process.

Acknowledged. We will make sure to add greater and finer step-by-step details about the generation process in the revised version of the paper.

5. No Model Error Discussion

We analyzed errors for Llama-3.1-8B and Qwen-2.5-7B on op=5 problems. Examples available in Sup Part 4. Common error types:

6. Noise distribution impact & RAG.

Our ablation (Sec 5.3, Fig 7c/d) shows model performance is robust to noise types, but RAG struggles significantly with our "Spider Topology" noise. Our RAG baseline is strong (all-mpnet-base-v2 retriever, Llama-3.1-70B decoder, 2k token budget, strong benchmark performance). RAG fails because it uses contextual (semantic similarity) search. Spider Topology adds irrelevant nodes semantically close to and connected to core graph nodes. RAG's fixed budget means noise can displace relevant chunks. RAG retrievers evaluate chunk relevance contextually, lacking the deep reasoning needed to identify truly necessary information, which attention layers possess. Contextual info alone is insufficient.

7. Forward vs. Reverse Problems.

The difference isn't arithmetic ops, but implicit relationships (Sec 4.3, Fig 6). Forward problems use class-instance/hierarchical dependencies (concrete -> abstract). Reverse problems provide abstract values to find unknown instance values (abstract -> concrete). Most LLMs perform better on forward problems (App E), hypothesized due to training data bias towards constructive logic.

We fine-tuned Qwen-2.5-500M with 1.3M generated problems (equal forward/reverse). The base model showed forward > reverse initially. Fine-tuning significantly boosted overall performance, with reverse slightly outperforming forward, confirming training data can address this. Detailed data in tables before and after fine-tuning is in Sup Part 3.

Response to other reasoning relationship question

We want to express our great appreciation to the reviewer for your positive feedback on GSM-Infinite. The question raised is also highly insightful and greatly appreciated. We want to clarify that the goal of GSM-Infinite is to model all the relationships that appear in GSM-8K with graph representation, an abstract way that is easier for manipulation and perturbation for scaling up in reasoning complexity and context length. GSM-Infinite model all explicit operations from plus, minus, multiply, and divide, and the implicit operations with hierarchical instance dependency. For example, “pigs” and “cats” are specific instances under the class “animal”, which GSM-Infinite can model well. (Is that the entailment relationship you are talking about?)

On the other hand, GSM-Infinite by no means captures all the relationships possible through graph representations. We believe exploring these alternative complex relationships through a graph is also highly interesting and insightful. We will leave these explorations for further complex graph to natural language mappings to future works.

Concern GSM-Infinite not practical

We appreciate the reviewer’s concern about GSM-Infinite’s practicality. While GSM-8K has been a standard for evaluating LLM math reasoning, its difficulty has become saturated (Figure 5(a), Page 4), which is why newer models have moved on. GSM-Infinite builds on GSM-8K by abstracting its core operations into graph representations and using graph perturbation to scale both reasoning complexity and context length. Our pipeline is entirely LLM- and human-free, generated independently of GSM-8K (Section 3, Pages 3–5). By scaling up the op=200 and above, we saturate SOTA LLMs and capture their full spectrum of reasoning ability for study.

On the other hand, recent LLMs often report metrics from MATH or AIME datasets. Although these datasets are of strong reasoning complexity, solving test examples convincingly requires a lot of math-related theorem knowledge in LLMs’ memory. GSM-Infinite offers an alternative reasoning benchmark that is drastically less reliant on LLMs’ memory, while evaluating all LLMs based solely on their reasoning ability of traversing a complex computational graph problem with fine-grained difficulty control for the study.

Because of space limitations, we hold five examples for op=5 for both Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct in the following link: https://docs.google.com/document/d/1WP3ygB67yNUS-iliSYYjFRY4Ls81yIVcRVxpbTVTZ0o/edit?usp=sharing.

We select a smaller operation setting with a smaller model to keep the complexity of the problem lower for the practicality of human inspection. Below is a summary of error causes and the corresponding index of the example.

Generation Throughput

During generation, we strictly constrain the number range in the solution path and disallow negative or floating-point numbers in arithmetic operations. These constraints make generating the Hard subset of GSM-Infinite significantly slower than Symbolic and Medium, as Hard requires heavy use of aggregation, implicit multiplication and addition when op is large—which quickly inflate numbers and reduce flexibility compared to Symbolic, where operations can be mixed more freely. Despite this, our system achieves far superior throughput on the Hard subset compared to any LLM- or human-in-the-loop approach. Using 16 threads on an M3 Pro, we reach 1.1M tokens/s for examples averaging 1k tokens (op=50), a speed that's hard to match with LLM-based generation.

When going up the op number, the throughput will go down because the chance that the operation will avoid the strict rejection sampling criteria is decreasing, causing some wasted computation. Just for reference, the much more flexible Symbolic subset can generate problems with 30k examples/s with 16 threads on CPU, since we can have more precise control of every operation in the Symbolic benchmark. We think the need for strong rejection sampling criteria is an implementation challenge, and more careful and more efficient graph control implementation is greatly possible and likely to lead to further improvement in throughput.

The above shows throughput for op=20 for increasing context length. The sudden drop in throughput from zero noise to 8k is because when injecting noise to a specific context length requirement, we have to first perform a tokenization step of the zero-noise example and calculate how much noise statement we need. The one extra tokenization causes a slowdown in generation throughput. On the other hand, since we impose far less strict checking on the noise statements, generating noise is highly efficient and parallelizable across multiple threads. We show our generator’s strong context length flexibility.

Three missing citations

Greatly appreciate the comments. We acknowledge that more discussion of previous multi-hop commonsense reasoning benchmarks will be added in the paper later version. But these three papers address different problems from ours.

Firstly, HotpotQA [2] and MusiQue [3] are on short-context 2 to 4-hop commonsense reasoning. Also, huge human labor is required to collect, limiting scaling up on quantity or the reasoning complexity. Also, these two are converted to long-context problems incorporated in RULER ([2]) and LOFT ([2] and [3]), which are cited and heavily studied (Section 2, page 3). The added noise in two can be effectively filtered by RAG, inapt for LC-LLM evaluation.

Secondly, CLUTRR [1], though aired before LLMs, similarly generates problems of family relationship deduction from graphs but with 3 big distinctions. (1) Despite similarity in methods, CLUTRR is heavily tailored to family relation deduction, especially specialized to kinship graphs, severely limited by scope. (2) Human annotation is essential to making CLUTRR, while GSM-Infinite relies completely on software, no LLM nor humans. No guarantee of the absolute correctness of the labels provided by the generator is a crucial drawback, as humans make mistakes. Specifically, we found that a minor part of the dataset is mislabeled. One example below.

Label: sister Question: Given the story, please provide the relationship between two selected people. Story: [James] bought a new dress for his daughter [Lynn]. [Theodore] and his son [Wesley] went to look at cars. [Theodore] ended up buying the Mustang. [James] took his grandson [Theodore] to the baseball game. Question: What is the relationship between 'Wesley' and 'Lynn'?

The corrected label is Wesley is the grand nephew of Lynn.

(3) Having either LLM or Human in the loop also leads to huge difficulty in scaling up the reasoning complexity. Below, we evaluate the 2to10 test set given by CLUTRR with various LLMs.

Lack of scaling prevents the evaluation of the full spectrum of behaviors. GSM-Infinite generates op=200 problems (and more if needed) to capture the full spectrum of SOTA models' reasoning performance.

RAG clarification

We follow the convention of context-level RAG method [1]. We solely use the input context as the data store (no external source). We segment the input text into chunks (roughly a sentence) using NLTK package. The retriever first rates each chunk based on its relevance to the query, and then the top-k scoring chunks are selected as the input to the decoder model. k is determined by the 2k budget from our experiments.

[1] Lewis, P. Retrieval-augmented generation for knowledge-intensive nlp tasks. (2020)

Dataset generation with Graph Clarification

During generation, no human/LLM is in the loop. “One-shot” generation is irrelevant to our settings. Each problem, regardless of context length, is generated based on one computational graph (not multiple graphs). Natural language problems directly mapped from the graph using our developed generator software. For a detailed explanation and reference, please look at (Figure 6, page 6) and (Section 3.1, 3.2, page 3-5).

Forward Reverse Observation

The difference between forward and reverse problems isn’t the arithmetic operations. Referring to (Section 4.3, page 6), the explicit plus, minus, multiply, and divide operations appear in both forward and reverse problems. The difference is the implicit operation/relationships contained. The implicit addition and multiplication are displaying class-instance and hierarchical instance dependency, respectively. Problems in forward logic only contain these two implicit operations - you can notice that in the computation graph (Figure 6, page 6), the more concrete and detailed instances are first computed, and then, using them collectively, more general and summarizing variables are computed. However, reverse problems contain the implicit minus and division operations. Given the value for the abstract variables, the LLM needs to compute the unknown instance variables. Here is a simplified example. Forward Problem - #pigs in Zoo is 2. #sheeps in Zoo is 4. Assume the animals not mentioned are of 0 quantity. What is the #total animals in Zoo? Solution: We go from #pigs in Zoo (instance) -> #sheeps in Zoo (instance) -> total animals (abstract). Reverse Problem - #pigs in Zoo is 2. #Sheep in Zoo is non-zero. #total animals in Zoo is 6. #sheep in Zoo? Solution: We go from total animals (abstract) -> #pigs in Zoo (instance) -> #sheeps in Zoo (instance). We show a detailed study in (Appendix E, page 18). We hypothesize that models' usual strength in forward stems from that humans naturally write in the constructive logic ordering, or forward, more, leading to naturally less training data for reverse logics.