MathConstruct: Challenging LLM Reasoning with Constructive Proofs

We thank reviewer 6y3x for their review. We are delighted that they recognize MathConstruct as a valuable and novel contribution. We also appreciate their feedback on the clarity of our paper and will incorporate the suggested clarifications. Below, we address their additional questions:

Q.1. Could the authors evaluate more recent models on MathConstruct?
We have evaluated newly released frontier models, including o3-mini and DeepSeek R1, and provide an updated results table below:

We observe that o3-mini significantly outperforms even o1, demonstrating stronger generalization and mathematical reasoning while being considerably more cost-efficient. R1, on the other hand, achieved an accuracy of only 32.3%.

Q.2. How many variants can be generated, and how were the 480 presented variants selected?
Many of our problems, particularly those in the “Find Inf”-category, permit infinitely many variations. However, generating interesting variations is more challenging for other types of problems, as their constructions fail for most values. Additionally, for each problem, we manually defined a range of parameters to ensure two important conditions: (1) the problem is resistant to brute-force solutions, and (2) the resulting output remains within a 4,000-character limit. Most problems allow for at least four variations under these constraints, although a small number permit only two or three. Therefore, we generated up to four variations per problem or fewer if four were not feasible. In practice, many problems could include significantly more variations. This point is illustrated clearly in Figure 7, in which we use 24 variations for each of the ten “Find Inf”-problems discussed in this subsection.

We thank reviewer S8Ey for their review. We appreciate that they found MathConstruct to be an interesting benchmark, and our experimental design detailed. Below, we address their other questions:

Q.1. Does robust accuracy suitably measure the correctness of problem solving for a given question?
Yes, because all included variations require the same underlying solution strategy. Furthermore, we explicitly exclude brute-forceable variations, thereby preventing models from using shortcuts specific to any single problem instantiation. Therefore, if a model correctly applies the solution strategy across all four provided problem variations, it is very likely capable of generalizing this approach to other instantiations.

More fundamentally, the main goal of our benchmark does not depend on these specific metrics. Specifically, our benchmark aims to provide effective means of evaluating models on a new reasoning task, where relative accuracy between models is the primary metric of interest.

Q.2 Is the small dataset size a limitation?
No, this is not a limitation. Although our benchmark is smaller than datasets like MATH [1], it is significantly more challenging, is better verified, and incorporates variations designed to test model generalization. It is worth noting that many widely used benchmarks contain relatively small problem sets. For instance, the 2024/2025 AIME competitions, which are frequently used to evaluate reasoning capabilities, consist of only 30 problems each. Lastly, rigorous benchmarking requires carefully curated problems that are neither erroneous nor overly simplistic. Thus, the importance of high-quality examples outweighs dataset size.

Q.3. Are there ways to reduce the amount of manual labour involved in the construction of the benchmark?
We refer the reviewer to our response to Q.1 of reviewer Aczy.

Q.4. What value do the variations add to the benchmark?
The variations we define have three main purposes:

• Reducing Variance: Including variations effectively increases the benchmark size, thus reducing measurement variance.
• Problem Analysis: We can perform several additional experiments, such as the one presented in Section 4.5, that analyze the effect of model variations on the model solutions.
• Reducing Contamination: Variations guard against memorization, as a model encountering a known problem from its training data must successfully solve all variations.

Q.5. Is there a way to improve the performance of LLMs?
Yes, we already included several improvements designed to enhance model accuracy. Specifically, we implement parser feedback to ensure accurate interpretation of answers and evaluate agent-based approaches. Furthermore, prompted by the reviewer, we ran additional experiments evaluating an agent-based approach that obtains detailed feedback from the ground-truth verification function. These functions log each failure with a specific reason, and therefore allow the model to correct its answer based on this feedback. Using this approach, o3-mini (the best model) improves by an additional 12% (53%->65%). We stress that this experiment should not be considered valid for real-world applications due to lack of access to the verification function at inference time, but it does give an upper bound on what is possible. We will add a discussion about this result in the appendix of our paper.

We did consider an advanced agent-based approach where the model would first implement this verifier function itself and subsequently use it to validate its solution. However, preliminary experiments checking the automation feasibility of our benchmark showed that current models struggle significantly with this task. Thus, we currently consider this out of scope for this benchmark paper.

Q.6. Did the authors apply mitigations against contamination?
First, we emphasize that our benchmark's problems rarely appear verbatim in existing training datasets. Notably, problems beyond the “Find Any”-category underwent substantial revision to become suitable constructive problems, and some were translated from other languages.

Second, we implemented a contamination detection strategy described in Section 4.4. Previous work shows that rewording mathematical problems can influence model performance in the presence of contamination [2]. By doing so, we observed minimal contamination of our benchmark.

Q.7. Can you include references to these additional works?
We appreciate the reviewer’s suggestion and will incorporate the additional references.

[1] https://arxiv.org/abs/2103.03874
[2] https://arxiv.org/abs/2405.16281

We thank the reviewer for the constructive comments, and are happy to read that they find our work novel and relevant to the community, and our framework easy to work with. We address any remaining concerns below.

Q.1. Can any parts of the pipeline be automated? What challenges did the authors encounter when trying to automate them?
Although certain elements of our pipeline can be readily automated, most aspects currently remain beyond the capabilities of state-of-the-art agents and models. We detail the three main components of the benchmark curation process below:

• Problem Curation: One aspect of the pipeline where automation was partially successful involves the curation and filtering of constructive problems. Constructive problems can be identified according to the selection criteria specified in our paper. A small portion of our benchmark questions were automatically selected by GPT-4o-mini from a larger internal dataset, and subsequently processed manually by the authors. While there were still plenty of false positives among the selected problems, this did make the selection procedure easier for these questions.
• Verifier Function: The most challenging part to automate is the creation of verification functions. Although we experimented briefly with automating this task, current models were unable to generate rigorous enough verification functions. Developing a robust verification function requires the identification of potential failure modes in model outputs, careful creation of efficient checkers, and creation of thorough test cases that cover all relevant edge cases. While future LLMs might be able to automate this, this currently remains out of reach for complex questions of MathConstruct.
• Problem Validation: Another critical aspect is ensuring problem quality, which falls between problem curation and verifier function generation in terms of automation feasibility. We automated several aspects of this step. For example, confirming problems could not be solved through brute-force methods using LLMs, as detailed in the paper. However, each problem underwent additional validation through an internal peer-review process among the authors. Manual peer review is essential for catching small issues and ensuring nothing is missed. While automation can help, it currently cannot fully replace this process as one needs to be certain that problems are correct once processed through this review. Therefore, even in a fully automated pipeline, some form of human validation would remain necessary.

In conclusion, while investigating further automation opportunities is valuable future work, we emphasize the importance of prioritizing a smaller yet more rigorously curated benchmark to accurately measure model performance.