Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models

We thank the reviewer for acknowledging our work is timely and important, rigorous and clearly presented, and for providing the detailed comments and constructive feedback! We will try our best to resolve the comments and greatly appreciate it if you could increase the scores if the comments are addressed properly.

Appropriateness for the main instead of the benchmark track.

The Brainteaser benchmark is not only a resource but a scientific lens into the capabilities and limitations of current most advanced LLMs. BRAINTEASER makes two substantive scientific contributions: (1) it offers a rigorously curated benchmark of expert‑authored math and logic puzzles that juxtapose short, insight‑driven solutions with longer brute‑force enumerations, giving researchers a controlled lens on creativity vs. brute‑force reasoning; and (2) it introduces a multi‑stage evaluation protocol—covering narrative‑to‑formula translation, direct and hint‑guided solution generation, informed self‑correction, and step‑level abstraction—which, when applied to five state‑of‑the‑art LLMs, reveals that models (i) often default to brute‑force search despite available creative shortcuts, (ii) struggle to revise answers even after seeing gold solutions, (iii) gain only modest accuracy from narrative-to-formula translation, and (iv) can break down a solution into meaningful steps. Those substantive scientific contributions squarely satisfy—and enrich—the NeurIPS main‑conference mandate,

Quantitative definition of creativity. Creativity versus generalization. Formal criteria for emergent problem-solving behavior.

We agree that this is an important concern. Providing a fully formal, quantitative definition of creativity would indeed be very useful, but we suspect that creating such a definition in a universally agreed-upon way may not be achievable (i.e., creativity may not be a formally-definable or quantifiable topic). Nonetheless, we believe that our operationalization of creativity is sufficient for the purposes of this paper - i.e., even if we can’t define it strictly, we can provide reasonable proxies for deciding whether a particular behavior is creative or not, even if we can’t provide a fully general definition for creativity.

We do provide a definition of creativity in Section 4.2:

"We define a creative solution as an innovative, insight-driven approach that leverages pattern recognition or lateral thinking. Rather than exhaustively testing all possibilities, it reframes the problem or exploits shortcuts to reduce complexity. Such solutions often involve minimal computation and are especially valuable for problems where brute-force search is intractable."

Quantitatively, one can think of creative solutions as solutions that reduce the problem complexity by more than a constant factor (like reducing O(n^2) from O(n log n)). However, unlike algorithmic problems, it is difficult to quantitatively analyze the problem complexity of brainteaser problems, and so we chose to analyze this qualitatively.

Instruction perturbation scope.

Could you please clarify which experiments you are referring to? We did not include analyses of instruction perturbation, but we may be misinterpreting this comment.

Ablations of prompt length/format.

We examine whether the length of a problem—measured by token count in the problem statement—correlates with model correctness. Our analysis reveals a weak to negligible correlation, indicating that longer problems are not necessarily harder for models to solve. This suggests that problem complexity is not well captured by surface-level features like textual length, but rather by the underlying reasoning depth or conceptual challenge. For instance, some short problems require multi-step logical deductions, while some long problems are verbose but procedurally simple. This reinforces the need for deeper structural metrics—such as number of reasoning steps in the solution or presence of creative insights—to assess problem difficulty.

Synthetic OOD tasks.

We did not include synthetic OOD tasks in our benchmark (all of our problems are naturally sourced), so we do not believe that this concern applies here. Please let us know if we are misinterpreting this concern.

Qwen 3 performance

Thank you for this point; below we present results with Qwen 3:

The results show that Qwen3 with "thinking" capabilities significantly outperforms its larger counterpart, Qwen3-32B without "thinking," across both mathematical and logical brainteasers. Specifically, Qwen3 achieves 70.3 on math problems compared to Qwen3-32B’s 65.9, and 60.5 on logic tasks versus 56.4. These consistent gains of around 4 points in both domains suggest that reasoning-augmented prompting—such as step-by-step thinking or chain-of-thought strategies—plays a crucial role in enhancing problem-solving performance. Notably, the benefit is not simply a function of model size, as the smaller "thinking" model outperforms the larger "non-thinking" one. This underscores that cognitive scaffolding mechanisms, rather than brute-force scaling, are key to tackling structured reasoning challenges.

Comparison with previous artifacts in a table.

Thank you for this suggestion; here is such a table:

This table compares five datasets spanning mathematical, logical, and puzzle-based reasoning. The datasets vary in size, problem style, reasoning type, and contextual richness.

As this table highlights, advantages of the Braingle Brainteasers dataset include that it presents multi-faceted coverage across several types of reasoning, it has a high level of diversity and complexity, it has a low knowledge barrier (little background knowledge is needed), it is challenging yet interpretable, and it is complimentary to existing datasets (providing a middle ground between structured logic datasets such as Zebra-Logic and formal mathematical ones such as MATH and AIME-2025).

[1] Measuring mathematical problem solving with the MATH dataset. arXiv: 2103.03874. NeurIPS Datasets and Benchmarks Track, 2021.

[2] Art of Problem Solving. American invitational mathematics examination, n.d. Accessed:2025-05-15.

[3] Zebralogic: On the scaling limits of llms for logical reasoning. arXiv: 2502.01100. 2025.

[4] BRAINTEASER: Lateral thinking puzzles for large language models. arXiv: 2310.05057. EMNLP 2023.

Discussion on how data contamination could affect performance.

Data contamination remains a potential limitation in evaluating model performance, with varying degrees of severity. On the more superficial end, contamination can arise from direct overlaps such as copying people’s names or short n-grams, which may not significantly affect reasoning or creativity. However, a more subtle and challenging form involves contamination of creative insights—where a model may internalize not just the surface form but the underlying reasoning strategies or ideas from training data [1,2]. This type of creativity contamination is difficult to detect and evaluate [1]. To address these concerns, better methods for understanding and auditing training data are necessary. While models like OLMo offer some transparency [3], it remains an open question how critical data contamination is to downstream performance. We will include a potential data contamination statement in the final version.

[1] Quantifying Memorization Across Neural Language Models. arXiv: 2202.07646. ICLR 2023.

[2] Investigating Data Contamination in Modern Benchmarks for Large Language Models. arXiv: 2311.09783. NAACL 2024.

[3] OLMo: Accelerating the Science of Language Models. arXiv: 2402.00838. ACL 2024.

We thank the reviewer for their valuable feedback and great questions. We hope that our rebuttal addresses all the reviewer’s concerns, and we kindly ask the reviewer to potentially upgrade their score if the reviewer is satisfied with our responses. We are also more than happy to answer any further questions that arise.

Dear Reviewer , Could you please take a look at the authors' response and see if their clarifications address your concerns, specifically on generalization and ablation studies?

Best AC

We thank the reviewer for acknowledging that our work is a good contribution, characterizes useful metrics and calls for more diverse reasoning approaches, and for providing the detailed comments and constructive feedback! We will try our best to resolve the comments and greatly appreciate it if you could increase the scores if the comments are addressed properly.

Examples of the dataset

Thank you for the valuable feedback. We agree that including examples earlier in the paper would significantly improve clarity and accessibility. In the final version, we will incorporate one or two representative examples in the main text. In particular, we will replace Figure 1 with an illustrative example from the dataset.

We provide the full problem used in Section 5.1 as follows and will include it at the beginning at Section 5.1.

Title: Rex and Ralph: Mystery Number

Two mathematicians, Rex and Ralph, have an ongoing competition to stump each other. Ralph was impressed by the ingenuity of Rex's last attempt using clues involving prime numbers, but he thinks he's got an even better one for Rex. He tells Rex he's thinking of a 6-digit number. "All of the digits are different. The digital sum matches the number formed by the last two digits in the number. The sum of the first two digits is the same as the sum of the last two digits." "Take the sum of the number, the number rotated one to the left, the number rotated one to the right, the number with the first three and last three digits swapped, the number with the digit pairs rotated to the left, and the number with the digit pairs rotated to the right. The first and last digits of this sum match the last two digits of the number, in some order." Ralph then asks, "If each of the three numbers formed by the digit pairs in the number is prime, then what is the number?" Rex looks confused, and for a moment Ralph thinks he's finally gotten him. Then Rex smiles, scribbles a few things down on a pad of paper and then says, "Very nice, Ralph!" Rex then tells Ralph his number. What did Rex say?

We also apologize for the confusion caused by the empty appendix sections and buried examples. This was an oversight in the submission process. As noted, examples do appear in the Supplementary Material zip (Appendix G.2 and H.2), but we will ensure the final version integrates these directly into the main paper and proper appendices with complete content. Thank you again for your thoughtful and constructive suggestions—they will help us improve both the clarity and quality of the final paper.

License and permission

Thank you for raising this important point. We take issues of data licensing and attribution very seriously. The problems sourced from Braingle were obtained via publicly accessible pages, and we ensured that only a subset of logically self-contained, well-formatted problems were included. We recognize that these problems reflect the creative efforts of the Braingle community, and we agree that clear attribution is warranted. In the final version of the paper and dataset documentation, we will prominently acknowledge Braingle and refer to this portion of the dataset as the Braingle Brainteasers subset, in recognition of the source and its contributors. Regarding distribution rights: while the Braingle site does not include an explicit license on each problem page, we acknowledge that the copyright status of community-contributed content is nuanced. Out of an abundance of caution, we are currently reaching out to Braingle to seek explicit permission for redistribution. If we are unable to obtain permission, we will refrain from releasing the data publicly and release only metadata such as URLs, and code for crawling the examples, allowing users to crawl the content themselves under fair use guidelines for research. We will also update the Ethics Checklist accordingly to transparently reflect these concerns under items 5 and 12, and we will clarify the licensing status of all included content in our documentation. Thank you again for drawing attention to this critical issue—we are committed to resolving it responsibly.

We thank the reviewer for acknowledging our work is well-written, innovative and structured, and for providing the detailed comments and constructive feedback! We will try our best to resolve the comments and greatly appreciate it if you could increase the scores if the comments are addressed properly.

Evaluation method and potential bias: LLM-human agreement is high.

To address your concern, firstly, we conducted a human evaluation with three annotators on 100 o3-generated solutions. Human annotators annotate each solution for correctness and whether it reflected creative or brute-force reasoning. We found that o3's judgments aligned with human labels with 99.3% and 97% average raw agreement for solution correctness and creativity/brute-force distinction respectively, reinforcing its reliability as an automatic evaluator in this setting. We also found that o3’s judgment has higher alignment than Claude-3.7, deepseek-reasoner, Gemini 2.5 Flash and GPT4-o.

For informed self-correction, problem reformulation and step breakdown have been carefully inspected by human annotators, as indicated in section 4.3, 4.4 and 4.2 (details in Appendix F).

Finally, we note that using LLM-as-a-Judge is an established practice, with an extensive number of papers showing that LLMs, especially recent reasoning models have high accuracy at such judgments on reasoning tasks [1,2,3,4,5,6,7,8,9]. Combined with the fact that o3 aligned well with our human annotators, we believe that this choice is well-justified.

References:

[1] J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization. arXiv: 2505.13346.

[2] REWARDBENCH 2: Advancing Reward Model Evaluation. arXiv: 2506.01937.

[3] RewardBench: Evaluating Reward Models for Language Modeling. arXiv: 2403.13787.

[4] PROCESSBENCH: Identifying Process Errors in Mathematical Reasoning. arxiv: 2412.06559. ACL 2025.

[5] Solving Inequality Proofs with Large Language Models. arXiv: 2506.07927.

[6] Evaluating Large Language Models AT Evaluating Instruction Following. arXiv: 2310.07641. ICLR 2024.

[7] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv: 2306.05685. NeurIPS 2023 Track Datasets and Benchmarks.

[8] Self-Alignment WITH Instruction Backtranslation. arXiv: 2308.06259. ICLR 2024.

[9] Self-rewarding language models. arXiv: 2401.10020. ICML 2024.

Our framework can indeed be used to analyze causal and counterfactual reasoning.

Thank you for this point! Our framework can indeed be used to analyze causal and counterfactual reasoning, and our Brainteasers benchmark already includes those types of questions (e.g., Logic 104, Logic 85 in the Appendix). In particular, in causal reasoning, creative solutions can be characterized by the identification of key causal pathways or the application of intervention logic (e.g., do-calculus), while brute-force solutions involve enumerating all possible variable combinations without deeper causal insight. Similarly, in counterfactual reasoning, creative reasoning manifests as recognizing minimal, targeted changes to antecedents that produce a shift in outcomes, while brute-force responses test many hypothetical permutations without considering the underlying causal structure. Tasks framed as "what-if" scenarios—common in logic puzzles, or COPA-style narratives—could indeed serve as benchmarks, with step-level annotation distinguishing insightful counterfactual manipulation from superficial trial-and-error. Exploring these types of problems more systematically would be an interesting direction for future work.

More nuanced analysis of solution steps.

Thank you for raising this point; we agree that it is important to be careful about analyzing step count. We believe that our way of analyzing step count is sufficiently nuanced. Specifically, we defined the step as “something that does not need to be super fine-grained like “1 +1 = 2” or “a -> b, b->c; a->c.” Instead, each step should represent a key component of the solution and the steps sequentially lead to the final answer to form a complete solution.” We have conducted an extensive analysis of our step-counting approach; this analysis can be viewed in Appendix F. While there is probably no perfect way to count steps, we believe that the analyses and discussion that are presented there are enough to establish that the methodology we have chosen is sufficient as a reasonable characterization of the length of solutions that models have arrived at.

We thank the reviewer for their valuable feedback and great questions. We hope that our rebuttal addresses all the reviewer’s concerns, and we kindly ask the reviewer to potentially upgrade their score if the reviewer is satisfied with our responses. We are also more than happy to answer any further questions that arise.

Dear Reviewer,

As the discussion period comes to a close, we would greatly appreciate a brief confirmation that our rebuttal has addressed your earlier concerns. We also hope you’ve had a chance to review the additional results demonstrating the robustness of our creativity/brute-force evaluation—findings that Reviewer NsnA described as illuminating and that we will highlight in the final version.

We’re happy to continue the discussion and add further analysis to address any remaining questions. Thank you again for your time and thoughtful engagement throughout the review process.

We thank the reviewer for acknowledging our work is well-written, insightful and unique, and for providing the detailed comments and constructive feedback! We will try our best to resolve the comments and greatly appreciate it if you could increase the scores if the comments are addressed properly.

Given that o3 is both an evaluator and a model under evaluation, how do you mitigate potential bias, and why was o3 chosen over a more neutral judge? o3 is a neural judge shown by human annotation.

To address your concern, firstly, we conducted a human evaluation with three annotators on 100 o3-generated solutions. Human annotators annotate each solution for correctness and whether it reflected creative or brute-force reasoning. We found that o3's judgments aligned with human labels with 99.3% and 97% average raw agreement for solution correctness and creativity/brute-force distinction respectively, reinforcing its reliability as an automatic evaluator in this setting. We also found that o3’s judgment has higher alignment than Claude-3.7, deepseek-reasoner, Gemini 2.5 Flash and GPT4-o.

For informed self-correction, problem reformulation and step breakdown have been carefully inspected by human annotators, as indicated in section 4.3, 4.4 and 4.2 (details in Appendix F).

Finally, we note that using LLM-as-a-Judge is an established practice, with an extensive number of papers showing that LLMs, especially recent reasoning models have high accuracy at such judgments on reasoning tasks [1,2,3,4,5,6,7,8,9]. Combined with the fact that o3 aligned well with our human annotators, we believe that this choice is well-justified.

References:

[1] J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization. arXiv: 2505.13346.

[2] REWARDBENCH 2: Advancing Reward Model Evaluation. arXiv: 2506.01937.

[3] RewardBench: Evaluating Reward Models for Language Modeling. arXiv: 2403.13787.

[4] PROCESSBENCH: Identifying Process Errors in Mathematical Reasoning. arxiv: 2412.06559. ACL 2025.

[5] Solving Inequality Proofs with Large Language Models. arXiv: 2506.07927.

[6] Evaluating Large Language Models AT Evaluating Instruction Following. arXiv: 2310.07641. ICLR 2024.

[7] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv: 2306.05685. NeurIPS 2023 Track Datasets and Benchmarks.

[8] Self-Alignment WITH Instruction Backtranslation. arXiv: 2308.06259. ICLR 2024.

[9] Self-rewarding language models. arXiv: 2401.10020. ICML 2024.

Are the results, especially for problem reformulation, statistically significant? Yes.

Thank you for noting the importance of statistical significance; we can verify that the results are statistically significant.

Both deepseek-reasoner and OpenAI o3 show statistically significant improvements in correctness after rewriting. For deepseek-reasoner, accuracy increased from 50.0% to 63.3% (p = 0.043), and for OpenAI o3, it rose from 56.7% to 73.3% (p = 0.023). These results indicate that the rewriting process reliably enhanced the strong reasoning models’ ability to produce correct answers on the evaluated examples.

The improvement in correctness after rewriting for DeepSeek-R1-Distill-Llama-70B shows a positive trend, with accuracy increasing from 23.3% to 33.3%. However, the corresponding p-value from the paired t-test is 0.08, which falls above the conventional 0.05 significance threshold. While this result is not statistically significant, it is still suggestive.

To explain why the gain is modest, we believe it could be 1). Only some of the problems are rewritable and the rest is already in the math competition question style format. 2) the problems in math competition style format still contain natural language and are not translated to formal math. We test this to see if language models handle the input format most similar to the problems they are optimized for, where a large portion is math competition style problems. 3). The remaining errors are due to high inherent reasoning complexity, rather than issues with problem comprehension—posing challenges even for the most advanced reasoning models.

How to ensure sufficient diversity in problem complexity and style?

This is a good point; we agree that it would strengthen the paper to discuss the diversity of problem complexity and style. Below we provide some dataset statistics that support these types of diversity; we will highlight these statistics in the paper by noting how they establish the diversity of the dataset.

Style diversity: We manually categorized problems into 21 distinct style subcategories across math and logic, as summarized in Table 1. Detailed definitions and examples for each category are provided in Appendices G and H of the Supplementary Material. Complexity diversity: We ensure complexity through several complementary measures: Human-rated difficulty (scale 0–3): We selected the most challenging 250 problems in each domain, yielding a difficulty range of 2.5–3. Macro-step analysis: Section 3.2 and Table 2 report the average number of solution steps—6.4 ± 2.5 for math and 8.6 ± 5.1 for logic—with a full range from 1 to 16, indicating substantial variation in complexity. Step-level categorization: Using structured prompting and manual inspection (Appendix F), we classify reasoning steps as creative (non-obvious, essential to solving the problem) or rudimentary (straightforward or mechanical). This fine-grained annotation further reveals the heterogeneity in solution paths and the cognitive demands of different problems.

We thank the reviewer for the thoughtful question and hope this comprehensive breakdown demonstrates the diversity we have rigorously curated in both problem style and complexity.

We also thank the reviewer for their valuable feedback. We hope that our rebuttal addresses all the reviewer’s concerns, and we kindly ask the reviewer to potentially upgrade their score if the reviewer is satisfied with our responses. We are also more than happy to answer any further questions that arise.