Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective

1. Task Complexity Expansion

We agree that further expanding task complexity is essential. In future work, we will:

• Introduce multi-step reasoning tasks that require chaining hypotheses, intermediate conclusions, and final integration.
• Incorporate hierarchical rules with nested, conditional, and conflict resolution components.
• Implement dynamic difficulty adjustment using parameters (e.g., reasoning steps, rule density, interference factors) to generate tasks that evolve with model proficiency.

---

2. Enhancing Rule-Inductive Reasoning

We plan to deepen our investigation into rule induction by:

• Increasing test difficulty—e.g., training with 8-bit binary data and testing with 16-bit numbers—to challenge the model’s generalization.
• Recording and validating the reasoning steps, not just the final answer; automated tools will assess the plausibility of the reasoning chain.
• Including detailed error case analyses, primarily in the Symbolic Math Abstraction (SMA) and Symbolic Reasoning (SR) categories, to illuminate common failure modes.

---

3. Benchmark Evolution and Maintenance

We have a three-stage plan to keep the benchmark state‑of‑the‑art:

• Short-term: Open-source the evaluation framework with detailed examples and error analysis.
• Mid-term: Develop a dynamic task difficulty system through parameterization of reasoning steps and rule complexity.
• Long-term: Build an adaptive challenge mechanism that auto-generates advanced tasks as models improve. We will also create robust feedback channels and community platforms to encourage continuous improvement.

---

4. Detailed Examples and Error Analysis for Rule Induction

We will add:

• Inclusion of Detailed Case Studies: We will include a dedicated section in our open-source GitHub repository that presents detailed case studies, particularly focusing on SMA and SR tasks.
• Error Pattern Analysis: Along with concrete examples, we will provide analyses of common error patterns, explaining the underlying causes and typical failure modes in rule-induced tasks. We thank you for emphasizing the need for more specific examples and error analyses. We agree that these enhancements will provide immediate, valuable insights into model limitations and help guide future improvements.

---

5. Dataset Openness and Community Engagement

To maximize community impact, we will:

• Open-source the complete project on GitHub, including data generation code, symbol mapping, the full dataset, and all evaluation scripts.
• Provide thorough documentation and user guides.

---

We appreciate your detailed feedback. Your recommendations have pinpointed vital areas for improvement. In response, we have developed clear and actionable measures to refine task complexity, enhance rule induction analysis, and ensure the benchmark remains dynamic and accessible. These improvements will help us advance robust abstract reasoning evaluation in LLMs.

Sincerely,
The Author Team

1. Concrete Instances vs. Abstract Features

Our paper distinguishes concrete instances (C)—detailed input strings containing surface-level information—from abstract features (A), which capture only the essential properties required for reasoning. For example, a concrete description of a dog may include breed and color, whereas its abstract feature reduces these details to “dog.” Our mapping f: C->A is designed to force models to look beyond token memorization; an inverse mapping (from A->C) is unnecessary for our task.

---

2. Notation and Example in Definition 3.2

(a) Notation "$A \times R$"

In Definition 3.2, the notation $A \times R$ indicates the set of all ordered pairs (a, r), where each a belongs to the abstract feature set A (Section 3.1) and each r belongs to the rule set R (Section 3.2). The reasoning function Re is defined as Re: $A \times R$→Q, which means that for any input pair (a, r), the function produces a corresponding conclusion q in Q. This formulation follows standard mathematical conventions (e.g., see [Cormen et al., 2009]) for combining elements from two previously defined sets.

(b) The Example Involving Uncertainty

The phrase “This dog is likely to bark” illustrates that typical human reasoning can involve uncertainty. However, in strictly symbolic or arithmetic tasks, the reasoning outcome is deterministic. The example is meant only to show how abstract features and rules combine, with the particular uncertainty reflecting context rather than a general property of all reasoning functions.

---

3. Theoretical Foundation of Theorem 3.7

Theorem 3.7 states that if a large language model achieves an abstract reasoning score Γ on our test set T that meets or exceeds a threshold γ, then for every task (c, r, q) in T the probability of correctly outputting q (via the abstraction mapping f and the reasoning function Re) is at least γ. In essence, a high Γ confirms the model’s robust ability to perform abstract reasoning across T.

This theorem is grounded in our rigorous definitions of f and Re, which provide a clear correspondence among the inputs, the applied rules, and the correct outputs. Although the main text offers only a brief proof, more detailed proofs and explanations are available in the Appendix.

---

4. Score Range Interpretation (Theorem 3.9)

For Theorem 3.9, we define the combined score as
F(Γ, Δ) = w₁·Γ + w₂·(1 − Δ) and, according to Theorem 3.8, a higher threshold γ implies a higher Γ. The weights w1 and w2 are chosen based on task requirements. Even when w1 = w2, the two components remain distinct: Γ measures the reasoning accuracy, while (1 − Δ) reflects the model's robustness against dependence on specific tokens. Thus, the score range interpretation stays relevant as each component captures a distinct aspect of performance.

---

Additional Responses

1. On Supporting Abstract Reasoning Abilities

Our task design is not limited to simple arithmetic operations. Instead, we adopt a symbol remapping strategy that forces models to abandon reliance on surface-level patterns and requires them to extract and apply the underlying abstract rules. For example, in a date calculation task (e.g., days between dates([2024, 07, 29], [2021, 10, 31]) = ?), the model is required not only to perform numerical computations but also to understand the intrinsic logic between time and dates, which manifests a higher demand for abstract reasoning. Even large-scale models did not achieve above 60% accuracy on these tasks, which shows that our benchmark is sufficiently challenging for current models’ abstract reasoning abilities.

2. On the Conciseness of Theorems and Theoretical Framework

We acknowledge that our current exposition of definitions and theorems is brief. The definitions of the abstraction mapping (f) and the reasoning function (Re) serve as the foundational building blocks of our mathematical model, and the corresponding theorems (e.g., Theorem 3.7 and Theorem 3.9) follow standard analytical approaches. Detailed proofs and clarifications are provided in the Appendix.

3. On the Benchmark’s Focus on Arithmetic Tasks

While our benchmark primarily features arithmetic and related symbolic tasks, it is not confined to simple calculations. In addition to basic arithmetic, our benchmark incorporates extended calculation tasks, operations in various number bases, and date calculations. Most tasks are designed using symbol remapping techniques to destabilize simple memorization, thereby revealing models’ inability to generalize abstract rules beyond familiar forms.

---

Because your review was relatively brief, we hope you will raise more questions or concerns to help us further improve the quality of our paper. Your additional feedback will be invaluable as we continue to refine our theoretical framework and experimental design.

Thank you for your review.

Sincerely,
The Author Team

1. Relevant References

Thank you for highlighting these references. Chollet (2019) is already cited (see Introduction). In the revised version, we have added:

• Garcez (2020), Neurosymbolic AI: The 3rd Wave
  This work highlights the limits of connectionist methods and supports our discussion on transitioning from memorization to true abstraction.

• Webb et al. (2023), Emergent Analogical Reasoning in Large Language Models
  This study provides empirical evidence on zero-shot analogical reasoning and contextualizes our findings on performance drops under symbol remapping.

---

2. Analysis of Training Data Configurations

Experimental Design Overview

To further evaluate the influence of different training data configurations on abstract reasoning, we tested two models—Llama-3.1-8B-Instruct (fine-tuned) and GEMINI-2.0-FLASH-THINKING-EXP-01-21—and also tested human participants.

Training Data Configuration:

• Unmapped (*) Configuration: The training data is generated using a random seed to follow the same distribution as the fixed_len_chat_bit_raw_dataset. No symbol mapping is applied.
• Fully Mapped (†) Configuration: The training data is generated using a random seed to follow the same distribution as the fixed_len_chat_bit_dataset. Symbol mapping is applied.

Test Benchmarks:

• We test on two mapped datasets:
  • fixed_len_chat_bit_dataset
  • fixed_len_chat_str_dataset

Training Setup: Training was performed using a single NVIDIA A800 80GB GPU, with 2,000 training samples over 8 epochs, a batch size of 8, cosine learning rate scheduling (with 3% warm-up), and using bfloat16 mixed precision.

Experimental Results and Analysis

Below are our experimental results:

Key Observations and Conclusions:

• Under the unmapped configuration, the performance improvement is marginal.
• For the fully mapped configuration, the Llama-3.1-8B-Instruct model shows significant improvement on the bit dataset (from 0.13 untrained to 0.60 by Epoch 8). In contrast, the Llama-3.1-8B-Instruct cot† shows improvements from 0.05 to 0.11. Analysis of the Chain-of-Thought outputs indicates that the model still largely imitates the training examples rather than explicitly inferring and applying abstract rules.However, the improvement on the string dataset is limited, suggesting that the rules learned under remapping do not generalize well across different symbolic representations.
• Human performance exceeds that of the models, further underscoring that current LLMs heavily rely on memorized associations rather than fully extracting and applying abstract rules.

---

3. Comparison with Human-Level Abstract Reasoning

We conducted a supplementary experiment with four undergraduate computer science students who evaluated the same tasks:

• Bit Dataset: Human participants achieved an accuracy of 0.97.
• String Dataset: The overall human accuracy was 0.47.

This direct comparison clearly demonstrates that even with extensive training, LLMs continue to underperform relative to human abstract reasoning capabilities, reinforcing the need for further improvements in model design and training.

---

4. Detailed Benchmark Implementation

We agree with this recommendation and will revise Appendix A.7 in a future version to include detailed examples demonstrating the symbol remapping process and specific task configurations. Furthermore, we commit to releasing the complete source code, which will comprise:

• A dynamic symbol mapping tool with customizable remapping rules.
• Code templates for generating custom datasets across all six defined task categories.
• An automated evaluation pipeline that computes both Abstract Reasoning Score (Γ) and Memory Dependence Score (∆), ensuring consistent answer matching.

---

Once again, we thank the reviewers for their highly constructive feedback and for helping us identify areas for clarification and improvement. We believe that the revisions and additional experiments described above have significantly strengthened our work, and we appreciate your careful consideration of our manuscript.

Sincerely,
The Author Team

1. Test Set Sampling and Duplicates

1. Test Set Distribution:
   Uniform distribution.

2. Real-World Distribution and Correlation:
   No, it is not sampled from a real-world task-based distribution such as SAT problems. However, our benchmark shows strong correlation with other benchmarks. For example, if we only consider the models tested in our paper (excluding agents):
   gemini-2.0-flash-thinking-exp-01-21* achieved rank 1 with the highest Γ score in our benchmark, and on LiveBench its average reasoning score reached 78.17, outperforming all other models.
   qwq-32B-preview* achieved scores comparable to 72B and API models in our benchmark, with a LiveBench average reasoning score of 57.71.
   Qwen2.5-72B-Instruct consistently outperforms competing models both in our benchmark and on the Korean SAT LLM Leaderboard baseline.

3. Elimination of Duplicates: No, duplicate phrasings have been strictly eliminated.

---

2. Symbol Remapping and Its Rationale

1. Performance Consistency After Remapping:
   Yes. A truly abstract model should maintain comparable performance post-remapping. For instance, GEMINI-2.0-FLASH-THINKING-EXP-01-21 drops by 0.40 on the fixed_len_chat_bit_dataset after remapping, indicating sensitivity to symbolic changes. In contrast, four computer science undergraduates scored nearly identically (1.00 and 0.97).

2. Humans and Non-Decimal Arithmetic:

   No. Although non-decimal bases may pose challenges, humans can perform arithmetic in various numeral systems once abstract rules are applied. In our test of 16 samples from the add_base3_raw_dataset, undergraduates achieved 0.87 accuracy, while GEMINI-2.0-FLASH-THINKING-EXP reached 0.75.

---

3. Validity of the Metric F(Γ, ∆)

1. Properties of a Valid Metric:
   A valid metric should be monotonic (increasing with higher Γ and decreasing with higher ∆, sensitive (proportional to changes), continuous, and normalized within [0, 1].

2. Our Metric:
   Defined as
   F(Γ, ∆) = w1Γ + w2(1 − ∆) it satisfies these properties.

3. Justification:
   Our metric inherits the desirable mathematical properties documented in the literature (e.g., Sokolova and Lapalme).

---

4. Multimodal Considerations

1. ARC work for multi-modal LLMs
   Yes. While ARC is designed for 2D visual inputs and can potentially benefit multimodal LLMs. For example, Align-DS-V—an experimental vision-language model derived from DeepSeek-R1-Distill-Llama-8B—achieved a score of 40.5 on the ARC-Challenge (5-shot), compared to 21.4 from DeepSeek-R1-Distill-Llama-8B.

2. Consider multimodal datasets Our work complements ARC by isolating the text modality to evaluate abstract reasoning in LLMs. While multimodal LLMs are emerging and humans naturally integrate visual and linguistic cues, most LLMs excel at language processing. Focusing exclusively on text allows us to establish a clear baseline for abstract reasoning without the added complexity of visual data.

---

5. Applicability of ConceptARC to LLMs

LLMs are optimized for one-dimensional text, not two-dimensional spatial data. Our experiments show that even when Arabic numerals and simple graphics are array-formatted, LLMs struggle with 2D spatial properties, making ConceptARC less suitable for assessing textual reasoning.

---

6. Minimality in Concept Representations

Our abstraction mapping process as an information compressor that inherently normalizes equivalent expressions without requiring an explicit minimality algorithm.

---

7. Addressing Uncertainty in Reasoning

The phrase “likely to bark” in our cognitive example was intended solely as an illustrative simplification. Our framework currently outputs deterministic results post-abstraction; future work will integrate probabilistic models to handle inherent uncertainties.

---

8. Clarification on Eq. 7's Probability

In Theorem 3.7, the probability is computed over the test set T, where each task is independently sampled from a uniform distribution over the task space.

---

9. Tokenization and Long Symbolic Strings

Sequences like “JJJQQQQQ” may split into multiple tokens by the tokenizer; however, modern LLMs handle these effectively. In our benchmark, the majority of token counts for such symbolic strings are below 5k—well within the 8K-token (or larger) context window of current models.

---

10. Clarification of Dataset Names

• var_len: Variable operand lengths.
• chat: Conversational style prompts.
• bitop: Bit manipulation operations.
• raw: Original symbols without remapping.

Detailed descriptions and examples will be added to Appendix A.6.

---

We appreciate the reviewer’s feedback and hope that these clarifications, along with our proposed modifications, demonstrate the robustness of our theoretical framework and benchmarking strategy. Thank you very much for your consideration.

Sincerely,
The Author Team