Response to Claims And Evidence: Some datasets also have a systematic hierarchy such as GPQA

The hierarchy in GPQA represents the difficulty of the original problems, whereas the hierarchy we introduce in Re-Imagine defines reasoning complexity through variations of problems from existing, well-established benchmarks.

We believe that Level-3 mutations align with counterfactual reasoning based on Pearl’s ladder of causation—how our belief about the occurrence of event Y changes if event X had value x’ instead of x. In Figure 1, for example, we present: "Janet bakes muffins with 2 eggs... Assume Janet no longer bakes muffins, how much money does she make every day?" Here, the observed outcome (event Y) is Janet’s daily earnings, while the causal factor (event X) is whether she bakes muffins or not. The logical facts are structured as a math problem, and Level-3 mutations modify one of these facts by introducing an additional assumption.

We agree with the reviewer that creating scenarios unlikely to occur in the real world, such as "Assume Janet bakes muffins using 10,000 eggs every day," can be highly interesting. Thanks to the modular plug-and-play design of our pipeline, we encourage the community to implement mutations of their interest to thoroughly assess models' reasoning abilities.

---

Response to Theoretical Claims 2: NL-symbolic-NL process

We recognize that the accuracy of NL-symbolic-NL translation is critical. Therefore, for benchmarks requiring full NL-symbolic-NL translation, such as GSM8K, we employ four methods to ensure the correctness of the mutated QA pairs and report results with quantified noise:

• NL-symbolic: We select Python solutions that not only produce the correct answer but also ensure that all constant variables in the code (root nodes in the computational graph) align with the numbers in the question, and vice versa (see lines 246-249, left column, in the main paper).
• Symbolic-NL: to ensure the accuracy of the symbolic-to-NL translation, we prompt GPT-4o a second time to back-translate the mutated math problem into Python by modifying the original question’s Python solution. The generated code must produce an execution result that matches the ground truth answer of the mutated question (see line 220-224, right column, in the main paper).
• Manual Quality Check: we manually check the quality of the mutated QA pairs and report the error rate in each mutation (see line 227-235, right column, in the main paper).
• Report Results with Quantified Noise: we show the error rate of the mutated data when reporting the model's performance (see Figure 2).

---

Response to Experimental Designs Or Analyses 2: Error analysis of the relationship of questions across levels, and training on level-3 questions.

We emphasize that this study primarily focuses on introducing the reasoning hierarchy, the benchmark mutation pipeline, and evaluating models' zero-shot and few-shot performance on dataset mutations. Model training is beyond the scope of this paper. However, thanks to the scalability of our proposed pipeline, we are expanding training set mutations to investigate whether exposing models to mutated questions during training can enhance their reasoning abilities. However, the results will not be included in this paper.

In Appendix B.2, we present experiments in an in-context learning setting, revealing that (1) simply replacing the original demonstrations in the context with mutated ones does not improve models’ reasoning accuracy. However, (2) models perform significantly better on generated test set variations when provided with both original and mutated examples as demonstrations. These findings offer insights for future model-training experiments.

---

Response to Experimental Designs Or Analyses 3: More popular LLMs should be adopted for analysis, such as Qwen

We conduct additional experiments with a broader range of popular LLMs, and the observations in the paper still hold for these new models.

Loop (mutations Raw, JunkHint, JunkNoHint, readOriginal, WriteOriginal, Xoriginal)

• QwQ-32B: 75.9% 76.7% 66.5% 63.67% 39.59% 36.33%
• R1-Distill-Llama-70B: 75.10% 62.45% 49.80% 60.41% 49.39% 48.57%

CruxEval (mutations Raw, Mutate String (L2), Mutate Value (L2), Redefine Function (L2), Replace Operator (L2), Swap Conditional (L2))

• GPT-4.5: 45.5% 23.55% 32.45% 29.79% 29.14% 32.06%

• GPT-o3-mini: 56.88% 35.95% 59.82% 56.35% 58.52% 50%

GSM8K (mutations Raw, SampleValues, OverWriteValue, UselessInfo, AddDependence, InsertConditional)

• qwq-32b 100 98.2 92.6 99.1 59.6 95.6

• r1-distiall-qwen-32b 98.3 93.5 85.9 97.6 63.2 85.0

• GPT-o3-mini 97.4 90.3 84.02 93.5 77.3 91.6

• GPT-4.5 97.45 89.76 81.28 95.47 61.54 89.28

Additional experiments are ongoing. We will include performance results for qwq, r1-distill-qwen, GPT-o3-mini, and GPT-4.5 across all four benchmarks in the camera-ready, and an error analysis in the Appendix.

We thank the reviewer for the detailed feedback.

---

Response to weakness 1: Novelty and the Influence of the Question Difficulty

(1) Novelty and contribution:

In summary,

• We present the reasoning ladder for LLMs, which systematically defines different levels of reasoning difficulty. This framework establishes a unified reasoning hierarchy that integrates both previously studied mutations and the new mutations introduced in our work. (Please refer to our response to Q1 for further explanation of the reasoning ladder.)
• Alongside the reasoning hierarchy, we introduce— to the best of our knowledge—the first scalable mutation generation pipeline that applies across multiple benchmarks and tasks. This framework enables the creation of an arbitrary number of mutations at each level of the hierarchy for existing benchmark problems. (We elaborate on the scalability further in our response to Weakness 2.)

Compared to the previous studies,

• We pointed out that previous work is primarily limited to Level-2 mutations ([1,2,3,9]), which evaluate a model's ability to generalize beyond existing benchmarks while preserving the original reasoning path of the questions.
• Huyuk et al. [4] explored a level-3 mutation, Bi-Counterfactual. However, like previous studies on level-3 mutations, their approach heavily relies on manually crafted patterns and rule annotations. To the best of our knowledge, no scalable solution has been proposed for generating problem variations across different reasoning levels spanning multiple benchmarks and tasks.

(2) Disentangle the decline in model performance from the question difficulty:

We use GSM8K as an example. We quantitatively define the difficulty of a numerical question reasoning using the number of calculation steps in the code snippet, following iGSM [2]. We compute the average accuracy of each model across examples with varying numbers of calculation steps. Given the large number of models tested, we aggregate their results and present the overall average accuracy in the following table. We included detailed results for each model respectively in the Appendix of the paper.

From the table we make three key observations:

• In nearly all scenarios, even when tested on examples with the same number of calculation steps, models consistently perform worse on the mutated sets compared to the original test set.
• Compared to Level-2, Level-3 mutations present a significantly greater challenge. Especially, the accuracy on Level-3 mutations with just two calculation steps is lower than on Raw test examples with six calculation steps with a significant margin.

---

Response to weakness 2: Generalization of the method

The pipeline requires three types of adapters: Question-to-Symbolic Adapters, Symbolic Representation-to-Mutation Adapters, and Mutation-to-Natural Language Question Adapters. The pipeline is designed in a modular plug-and-play fashion, making all adapters both reusable and customizable. Users can either utilize existing adapters if they meet their needs or modify them to suit their specific dataset or domain. The estimated manual effort required for adapter customization is based on the four domains we implemented:

• Question to Symbolic Adaptor: Prompts Writing.
• Symbolic Representation to Mutation Adapter: Write around 50 lines of code to define the mutation of the symbolic representation.
• Mutation to Natual Language Question Adaptor: Write around 100 lines of code for model prompting to translate the mutation to natural language.

We replaced the word "automatic" with the more precise term "scalable" in the paper.

---

How do the three levels relate to the causal ladder?

The key connection between Pearl's ladder and our framework is the problem's computation graph, which can be understood as a causal model in Pearl's framework. Each problem in a benchmark can be interpreted as a single realization of the graph with specific node values. Experiment associated with different perturbations in such graph can be related to operations in Pearl's ladder of causation. For instance, computing the effect in the outcome of the change in one leave node maps to the definition of a counterfactual. Note, however, that that not all mutations in the three levels have a causal counterpart (like adding an irrelevant piece of information or changing an operation). In this sense our framework can cover a broader definition of reasoning in each level.

Response to Claims And Evidence: Robustness to stochastic synthesis

We believe there may be a misunderstanding about the experimental setup. To clarify:

• All models in the paper are tested on the same generated benchmark instantiation, ensuring fair comparison between models.
• We also report the statistical accuracy of models on GSM8K numerical answer predictions in Figure 7. The bar plot presents the average accuracy and variance for each model, tested on 10 test variations per mutation, sampled using 10 different seeds. Importantly, these 10 test variations are identical across all models in this experiment. The results show the robustness of the metric to stochastic synthesis.

We thank the reviewer for pointing out the unclear instructions on how to evaluate using dynamic benchmarks like Re-Imagine. We will add the following clarification to the paper:

For a fair comparison, all models should be tested using the same data. In practice, to evaluate using dynamic benchmarks like Re-Imagine, we recommend that researchers: (1) sample multiple test variations initially; (2) record and report the random seeds in publications or repositories; (3) report the statistical accuracy on the sampled test variations for both baseline models and proposed approaches.

Response to Methods And Evaluation Criteria: What exact models are used for (1) NL-to-code (2) Computation graph parsing (3) code-to-NL?

We use language models only in the NL-to-code and code-to-NL steps. All mutations to symbolic representations (ie. computation graphs) are performed explicitly by modifying the AST (abstract syntax tree https://docs.python.org/3/library/ast.html) data structure of the code according to user-defineable rules.

For NL-to-code and code-to-NL:

• GSM8K

  • NL-to-code uses Mixtral-8x7B (see line 240-244, left-column in the main paper).
  • Code-to-NL uses GPT-4o (see line 266-270, left-column in the main paper, Figure 13 and 14 in Appendix B).

• CLadder

  • NL-to-code uses the causal engine offered in the original CLadder benchmark (see line 1124 in Appendix C).
  • Code-to-NL uses Meta-Llama70BInstruct (see line 1140 in Appendix C).

Loop and CruxEval start from the symbolic representation, so the NL-to-code and Code-to-NL steps are skipped; thus, applying Re-Imagine to these benchmarks does not require use of any models.

Response to Experimental Designs Or Analyses and Weaknesses: should directly compare against related works methods to show concrete improvements

We highlight that the goal of Re-Imagine is to establish a unified reasoning hierarchy that integrates both previously studied mutations and the new mutations introduced in our work.

We re-implement the two most widely studied mutations from previous work—UselessInfo and SampleValues—while scaling them up to the entire GSM8K test set. Since no prior studies have focused on evaluating models' reasoning abilities through mutations in the other three benchmarks—Loop, CruxEval, and CLadder—there is no meaningful mutations to be included in the framework.

---

Response to Experimental Relation To Broader Scientific Literature: Why Symbolic Representation?

The primary reason for introducing the executable symbolic representation is to obtain ground truth answers for the mutated examples in a deterministic manner. For instance, if the symbolic representation is a Python code snippet, the answer to the mutated question is derived by executing the mutated code. As a result, both the symbolic-to-mutation and mutation-to-answer modules in the pipeline are deterministic. The other two modules—NL-to-symbolic and mutation-to-NL question—rely on LLMs. Given that LLMs have demonstrated relatively high reliability and robustness in NL-to-code and code-to-description tasks, the accuracy of these two modules is generally satisfactory. On top of these, for each LLMs-based module, we conduct verifications to further guarantee the mutations accuracy:

• NL-to-symbolic: see lines 246-249, left column, in the main paper.
• Symbolic-to-NL question: see line 220-224, right column, in the main paper.

In conclusion, executable symbolic representations enable us to break down the mutation process into smaller steps that are either deterministic or easily handled by LLMs, and can be thoroughly verified. In addition, the symbolic representation gives us greater control over the mutation generation process. We can explicitly define and study different types of mutations.

We are unclear about which "simpler mutation methods" the reviewer is referring to. We assume they might be methods that generate mutated QA pairs directly using LLMs without the assistance of symbolic representations. However, as shown in the paper, LLMs struggle with answering mutated questions, and relying on them to generate ground truth answers for mutated questions can be risky. Additionally, there are no reliable validation methods to verify the accuracy of the generated mutations. We are open to further discussion if the reviewer provides more details about the simpler mutation methods they have in mind.

---

Response to Experimental Questions For Authors: Zero-shot Only?

For all four benchmarks, we adopt the standard in-context learning methods commonly used in the original benchmarks. In previous studies, zero-shot learning has been most widely used in CLadder, CruxEval, and Loop, while 8-shot learning has been the most common setup for GSM8K. We replicate this configuration in our experiments. In Appendix B.2, we further extend our experiments to explore the impact of different types of in-context learning examples on answering mutated GSM8K questions.

---

Response to repeated text in pages 2-4

We did not find any repeated paragraphs or quotations as mentioned by the reviewer (any specific line number references to this would be greatly appreciated); however, we appreciate the note and have performed an additional pass editing this section for proofreading and clarity. We acknowledge that the typesetting of the table and caption may have led to some confusion or re-reading of earlier text, and we improve the layout for the camera-ready submission.