Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles

Thank you for the encouraging and constructive feedback! We appreciate your recognition of the Enigmata benchmark, our proposed Enigmata-Model training paradigm, and the model's demonstrated generalization ability.

Here are the responses to your concerns

W1: Inconsistent Data Generation — 6 of 36 tasks rely on fixed pools or manual annotation. Would this undermine the claim of universal scalability and automation?

A1: Thank you for this question. We would like to clarify that the vast majority of Enigmata tasks (30 out of 36) are fully auto-generated with configurable parameters, controllable difficulty, and programmatic verification. These auto-generated environments cover all seven domains and form the core of our training and evaluation pipeline.

For the remaining six tasks that rely on fixed pools or partial manual annotation, we chose to include them because they exhibit unique structural challenges that are currently difficult to procedurally generate with high quality. These tasks still provide valuable diversity and difficulty for evaluating general reasoning, even if not fully automated. Rather than undermining scalability, we believe this design reflects a practical trade-off between automation and task complexity. We are actively working toward expanding auto-generation coverage for these remaining categories.

W2: Multi-Stage RL Training Instability. The paper notes that response length is more volatile under multi-stage RL, suggesting possible instability when optimizing diverse reasoning tasks.

A2: Thank you for raising this point. We would like to clarify that the observed variability in response length is not a sign of training failure, but rather a reflection of the different inductive behaviors of the two training paradigms analyzed in Section 4.3. We compare two strategies under the same total training budget:

• Multi-Stage RL: better at acquiring task-specific skills, especially for complex tasks not seen in the SFT phase, such as ARC-AGI and Enigmata.
• Mix-Training RL: shows superior generalization to OOD tasks, thanks to broader exposure and more diverse reasoning supervision.

W3: Code Utilization Fails to Improve Reasoning. The paper states that code utilization actually hindered performance, and current models fail to use code effectively without execution.

A3: Yes, and thank you for pointing this out. In our experiments, we investigated whether prompting models to write code (without execution feedback) would help in complex symbolic puzzles. We found that this did not improve and sometimes worsened performance, likely because without an actual interpreter or reasoning agent loop, models lacked feedback on correctness. This aligns with our broader hypothesis: current LLMs, when used purely as reasoning models, cannot fully leverage coding prompts without execution. We are actively working on extensions toward agent-based settings, where generated code can be executed, verified, and refined, leading to more meaningful gains. We believe Enigmata can serve as a strong resource for developing such RL-augmented or tool-augmented agents.

We appreciate your valuable feedback and thoughtful comments. Since it is near the end of the discussion period, could you kindly let us know if we have managed to address your concerns? Should you have any further questions, kindly let us know. We are more than happy to have further discussions with you. Thanks

Thank you for the positive and encouraging feedback! We are especially grateful for your recognition of our benchmark design, thorough experimentation, and the potential value of Enigmata to the broader research community.

Here are the responses to your concerns

W1: Benchmark Comparison – Why Enigmata over existing benchmarks like ARC-AGI 2? I'm not deeply familiar with all reasoning benchmarks—could the authors clarify why Enigmata is useful in the context of other (potentially more challenging) test suites?

A1: Thank you for the thoughtful question. We agree that ARC-AGI and similar benchmarks are valuable for evaluating reasoning capabilities. However, they are primarily designed for evaluation, with small and static datasets, minimal training examples, and limited task diversity, making them less suitable for developing or analyzing reasoning capabilities through learning.

In contrast, Enigmata supports both training and evaluation. It provides a broad set of automatically generated, knowledge-free puzzle tasks across diverse reasoning types, with verifiable rewards, difficulty scaling, and RL-compatible interfaces. These features enable scalable training, curriculum learning, and structured generalization analysis.

Importantly, our experiments show that models trained on Enigmata improve on ARC-AGI and other challenging tasks, demonstrating its utility as a foundation for building generalizable reasoning abilities. We will clarify this distinction more explicitly in the revised version.

W2: Statistical Significance and Error Bars. The authors mention significance in Section 5.3, but it only covers ablations. Tables and figures seem to lack error bars or variance estimates.

A2: We appreciate this careful observation. In fact, our evaluation protocol already incorporates repeated sampling to ensure statistical reliability. Specifically, we evaluate each AIME problem 32 times and all other benchmarks 4 times, reporting average performance across these runs. This setup is designed to reduce the effect of stochastic decoding (we use temperature = 0.7) and improve the stability of reported metrics. In response to your helpful suggestion, we have now conducted additional statistical significance tests and included error bars or confidence intervals in key tables and figures in the revised version, making variance across runs more explicit. The following table summarizes the results of one-sample t-tests on several datasets, based on our final model checkpoints. Each test compares the model's mean accuracy against a null hypothesis of 0.5 (random chance):

W3: Citation of ACES (Pourcel et al., NeurIPS 2024). I encourage the authors to cite this related work on the automatic generation of programming puzzles.

A3: Thank you for the pointer! We will cite the ACES paper (Pourcel et al., NeurIPS 2024) and briefly compare it with Enigmata.

Here are the responses to your question

Q1: Low Performance in Graph and Sequential Domains. Can this be solved with more data?

A1: Great question. We observe that both Graph and Sequential domains require multi-step planning, structured memory, and sometimes intermediate tool use—challenges that current models (even SOTA) struggle with. Our current hypothesis is two-fold:

• For Sequential puzzles, we plan to explore multi-turn RL approaches, enabling models to iteratively refine and revise partial plans, rather than relying on single-pass generation. This aligns better with the inherent temporal and decision-state dependencies of these tasks.
• For Graph tasks, we are investigating the use of graph-centric supervised fine-tuning, which may better capture relational structures and multi-hop dependencies. While scaling up training data may provide incremental gains, we believe that improvements in architectural bias and temporal credit assignment will be essential to make substantial progress in these domains.

Q2: What is the main benefit of Enigmata over other reasoning benchmarks?

A2: Thank you for the question. Enigmata stands out by combining scalability, diversity, and trainability in a unified benchmark. Compared to existing puzzle reasoning datasets, it covers the largest number of task categories (7) and provides a substantially larger number of tasks (36), provides automatic instance generation and verification, and uniquely supports both reinforcement learning and curriculum-based training through the RLVR paradigm.

As summarized in Table 2, Enigmata is the only benchmark that simultaneously offers diversity, scalability, verifiable reward signals, and structured training support—making it especially suitable for studying how reasoning capabilities can be acquired and generalized through learning.

Thank you for the detailed feedback and for recognizing the value of the Enigmata benchmark suite, our RL experiments, and the overall clarity of the paper.

Here are the responses to your concerns

W1: The experiment only contains one model of one family and one scale. It is unsure how the results could generalize across different model families.

A1: Thank you for raising this. While our main experiments were conducted using Qwen2.5-32B, we have extended our evaluation to include models with different architectures and scales to assess the generality of our findings. In particular, we conducted additional experiments using a proprietary mixture-of-experts (MoE) model (with 20B activated parameters and 200B total parameters, anonymized during review). This model differs substantially from Qwen in both size and structure. As shown in the table below, adding Enigmata to the MoE model's training corpus leads to consistent and significant gains across both in-domain (Enigmata-Eval) and math and STEM reasoning benchmarks such as AIME and GPQA:

W2: The paper claims AIME as OOD evaluation, though some math data is also used in training, making this not completely OOD.

We would like to clarify that AIME is not presented as an out-of-distribution (OOD) evaluation in our paper. Our actual OOD evaluation is conducted on KORbench, which comprises symbolic reasoning tasks in different tasks and domains, with no overlap with our training data. We will make this clearer in the revision to avoid potential misunderstandings.

W3: Generalization Across Games and Realistic Settings. The paper would benefit from more analysis on (1) generalization across games, and (2) generalization in more realistic settings—e.g., what kinds of games help generalize to which types of OOD tasks.

A3: Thank you for this valuable suggestion. We address both aspects of your question through two complementary experiments:

(1) Cross-Game Generalization within Enigmata: We conducted a domain-level transfer experiment, where models were trained using supervised fine-tuning on only one puzzle domain, and evaluated across the remaining domains. This allows us to quantify how reasoning skills acquired in one domain transfer to others. Among all domains, the Search category produces the strongest average transfer to other domains (+13.65). This suggests that puzzles involving combinatorial exploration, planning, and constraint satisfaction can cultivate general reasoning capabilities applicable across task types. These results also offer practical guidance for curriculum-efficient training, where a subset of domains can be selected to maximize generalization.

(2) Transfer to Realistic OOD Settings: To examine transfer to distributionally different tasks, we trained a model only on the Enigmata puzzle data, without any exposure to ARC-AGI or math-specific data, and then evaluated it on ARC-AGI, KORBench, and AIME 24. The performance is summarized below:

The Enigmata-only model shows non-trivial improvements over baseline on ARC-AGI and KORBench, despite never seeing these benchmarks during training. This indicates that puzzle training alone can yield partial generalization to unseen, structurally different tasks, even in out-of-domain scenarios. The gains observed here affirm Enigmata's value for reasoning models.

W4: Novelty of Enigmata as a Benchmark. While the paper compiles a set of tasks, most can be found in existing resources. The paper does not seem to add new capabilities for testing LLMs.

A4: Thank you for the thoughtful feedback. While many Enigmata tasks are inspired by classic puzzles, our contribution lies in turning them into a unified, diverse, and RL-compatible benchmark. Most existing resources offer only small, static datasets—suitable for evaluation but insufficient for large-scale training or systematic analysis. Enigmata addresses this by supporting automatic generation, rule-based verification, and controllable difficulty across 36 tasks and 7 reasoning domains, enabling reinforcement learning, curriculum design, and generalization studies.

Second, Enigmata enables new ways to test LLMs by going beyond static evaluation. It supports active training, difficulty scaling, and complex reasoning, allowing us to systematically analyze how models learn and generalize across tasks. which together allow systematic analysis of how models learn, adapt, and generalize. Importantly, Enigmata focuses on symbolic puzzle reasoning—which, unlike knowledge-intensive tasks, targets core reasoning skills without relying on prior knowledge or memorization. This makes it a valuable resource for isolating and understanding the fundamental reasoning abilities of LLMs. We will clarify this distinction more explicitly in the revised version.

Here are the responses to your question

Q1: Comparison with Reasoning Gym. The practical value of this benchmark may be overshadowed by more comprehensive collections like Reasoning Gym. The paper might consider analysis to set it apart.

A1: Thank you for the thoughtful comment. Enigmata and Reasoning Gym were developed independently as contemporary works, so our original submission did not include a detailed comparison. We appreciate the suggestion and will add one in the revision.

The two benchmarks differ in focus: Reasoning Gym emphasizes broad cognitive coverage and is primarily designed for supervised evaluation, whereas Enigmata targets knowledge-free, puzzle-style reasoning and supports RL-based training via automatic generation, verifiable rewards, and controllable difficulty. We view the two as complementary in goals and design. Enigmata's unique value lies in enabling scalable training and fine-grained analysis of how LLMs acquire and generalize core reasoning skills.

We appreciate your valuable feedback and thoughtful comments. Since it is near the end of the discussion period, could you kindly let us know if we have managed to address your concerns? Should you have any further questions, kindly let us know. We are more than happy to have further discussions with you. Thanks

Thank you for the insightful feedback! We appreciate your recognition of the model's strong performance across both puzzle and math benchmarks, as well as the clarity of our presentation.

Here are the responses to your concerns

W1: Transferability of Puzzle Training Data Alone. It remains unclear how effective puzzle training data alone is at supporting generalization. A deeper investigation into the standalone impact of puzzle data on reasoning capabilities would be valuable.

A1: Thanks for highlighting this. To better evaluate the standalone effect of puzzle data, we conducted an experiment where the model was trained only on the Enigmata dataset, without any additional math or external reasoning tasks. The results are shown in the table below. We observe that training with Enigmata puzzle data alone not only substantially improves performance on Enigmata-Eval, but also yields strong generalization to out-of-domain puzzle benchmarks such as ARC-AGI 1, ARC-AGI 2, and KORBench during the RL phase.

Furthermore, we observe similar trends at a larger scale using a proprietary MoE model (20B activated parameters, 200B total parameters, anonymized during review). By adding Enigmata to the original full training corpus (MoE-Model+Enigmata), we observe consistent performance improvements across multiple reasoning benchmarks, including AIME and GPQA for math and STEM reasoning.

W2: Cross-Domain Transferability in Puzzle Games. It would be insightful to explore whether training on one puzzle domain facilitates generalization to others. A data efficiency study could help identify compact training curricula.

A2: We appreciate your suggestion. To directly evaluate cross-domain transferability, we conducted a new experiment where models were trained via SFT on only one puzzle domain, and then evaluated on the remaining six. This setup allows us to measure how effectively reasoning skills learned in one domain transfer to others. The results are summarized below:

As shown, training on the Search domain leads to the strongest average transfer (+13.65) across other domains. This suggests that Search puzzles—characterized by long-horizon, combinatorial reasoning—promote general cognitive skills such as constraint satisfaction, backtracking, and planning. These skills appear to transfer well to a variety of symbolic reasoning tasks, making Search a promising candidate for curriculum-efficient training. We believe this analysis provides a concrete foundation for selecting representative domains to construct compact yet generalizable reasoning datasets.

W3: Transfer Across Difficulty Levels. An interesting question is whether models trained only on easy tasks can generalize to harder ones, which was observed in other domains such as SimpleRL (Zeng et al., 2025). Studying this easy-to-hard generalization would provide insight into how task difficulty impacts skill development and transfer.

A3: We appreciate you for raising this insightful point. We conducted targeted experiments to explore easy-to-hard generalization, particularly within the ARC-AGI benchmark. We trained models using ARC-AGI data with two different prompt length limits—4K (representing "easy" short-context training) and 6K (medium-length). We then evaluated performance on test samples grouped by their required context lengths. As shown below, models trained with shorter prompts struggled significantly on longer/harder inputs, demonstrating that generalization across difficulty levels remains limited:

Furthermore, we conducted qualitative analysis of DeepSeek-R1 solving maze puzzles at different difficulty levels. In easy cases, models adopt a greedy, forward-only strategy with no need for revision or backtracking. In contrast, hard instances require global search, recursive planning, and backtracking, occasionally involving code-style planning logic. Below is an excerpted comparison:

// Easy
  - Starting at (1,1). Possible moves: right is blocked (B at (1,2)), down is (2,1). So first move down to (2,1).
  - From (2,1), possible moves: up is back to start, down to (3,1), right to (2,2). Let's go right to (2,2).
  - At (2,2), possible moves: up is (1,2) which is B, down to (3,2), left to (2,1), right to (2,3) which is B. So move down to (3,2).
  - At (3,2), possible moves: up to (2,2), down to (4,2), right to (3,3) which is B. Let's move down to (4,2).
  - At (4,2), possible moves: up to (3,2), down to (5,2), right to (4,3) which is B. So move down to (5,2).
  - Now at (5,2). Possible moves: up to (4,2), right to (5,3), left to (5,1). Let's go right to (5,3).
  - At (5,3), can move right to (5,4), then to (5,5) which is E. So path would be:
  - (1,1) -> (2,1) -> (2,2) -> (3,2) -> (4,2) -> (5,2) -> (5,3) -> (5,4) -> (5,5)

// Hard
  -  ...
  - Neighbors:
  - Up: (0,0) visited.
  - Down: (2,0). maze[2][0] is '.', not visited. Add to queue. Path: [(1,1), (2,1), (3,1)]. Mark visited[2][0] = True.
  - Right: (1,1). maze[1][1] is '.', not visited. Add to queue. Path: [(1,1), (2,1), (2,2)]. Mark visited[1][1] = True.
  - Left: invalid.
  - So queue now has (0,1), (2,0), (1,1).
  - Next, process (0,1) (1,2 in 1-based).
  - Check if end? No.
  - ....
  - Let me retrace.
  - Possible path:
  - Start at (1,1) → (2,1) → (3,1) → (3,2) → (3,3) → (3,4) → (4,4) → (4,5) → (5,5).
  - Wait, (4,4) is array[3][3], which is B. So that's not possible. Hmm.
  - Wait, the third row (array index 2) is ['.', '.', '.', '.', 'B']. So (3,4) is array[2][3], which is '.' (coordinates (3,4)). Then moving down to (4,4) (array[3][3] which is B) is blocked. So that's a dead end.
  - Alternative path from (3,4) could be right to (3,5), but that's B. So can't go there.
  - So maybe from (3,3), move right to (3,4), then up to (2,4) (but array[1][3] is B), so no.
  - Wait, let's think of another approach. Maybe go from (1,1) right to (1,2), then down.

Responses to your question

Q1: VC-PPO vs. Other RL Variants. The choice of VC-PPO over VC-GRPO or VC-DAPO is unclear. A broader comparison would help assess its suitability.

A1: Great question. We do not assume any requirements for the adopted RL algorithms in our work. In early-stage experiments, we did run our experiments on GRPO. However, we ultimately adopted VC-PPO due to its faster convergence, better runtime efficiency, and smoother learning dynamics. Due to limited time and resources, we chose to run our experiments on VC-PPO as a standard algorithm to ablate over data and training strategies.

We appreciate your valuable feedback and thoughtful comments. Since it is near the end of the discussion period, could you kindly let us know if we have managed to address your concerns? Should you have any further questions, kindly let us know. We are more than happy to have further discussions with you. Thanks