Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs

Thank you for your thoughtful review and for highlighting the importance of our work in mathematical reasoning evaluation and benchmark development. For calibration, we'd like to note that ICLR 2025 uses a different rubric than other conferences; for instance, a rating of 8 at ICLR 2025 only indicates a basic "Accept". Furthermore,

• "Accept" is 8 at ICLR 2025 (vs 7 at NeurIPS 2024)
• "Strong Accept" is 10 at ICLR 2025 (vs 9 at NeurIPS 2024)
• ICLR 2025 does not use 7 or 9

We kindly request to take this into account when reviewing our paper.

“Given benchmark is too small, I fully acknowledge the hardness of the problems from Putnam mathematical competition but a dataset of just 236 problems is too small . For instance JEEBench [1] which ...”

We are expanding the dataset (and adding more variations) by incorporating Putnam problems from 1938-1984 using [1][2] beyond the Putnam Archive (1985-2023), effectively doubling its coverage while preserving its rigor and mathematical sophistication. This expansion enables evaluation of LLMs' mathematical reasoning beyond final answers by leveraging our novel TFA-based metric for assessing proof quality, providing deeper insights into models' mathematical capabilities.

[1] Gleason, Andrew M., et al. The William Lowell Putnam Mathematical Competition Problems and Solutions: 1938-1964. Taiwan, Mathematical Association of America, 1980. [2] G.L. Alexanderson, L.F. Klosinski, and L.C. Larson. The William Lowell Putnam Mathematical Competition: Problems and Solutions, 1965-1984. MAA problem books series. Mathematical Association of America, 1985.

“Given that the dataset is the primary contribution of the paper, authors should include a discussion about the dataset construction. process, i.e, how the problems were selected, % of problems selected for each category ..."

Thank you for pointing out this area of potential confusion; we will elaborate further on this point in our revised submission. We submitted problems primarily based on their suitability to yield a final, fixed boxed answer to be compatible with our evaluation method. 'Combinatorics': 58, 'Algebra': 61, 'Geometry': 43, 'Calculus': 75, 'Analysis': 73, 'Number Theory': 95, 'Probability': 21, 'Trigonometry': 17, 'Linear Algebra': 25, 'Complex Numbers': 10, 'Differential Equations': 1. Note that in our classification, we allowed questions to be classified under multiple domains. Based on Putnam’s own difficulty rating (1, 2 = ‘easy’, 3, 4 = ‘medium’, 5, 6 = ‘hard’), we had 99 easy, 78 medium, and 59 hard.

“Authors could have experimented with other prompting methods apart from few shot prompting ... ”

Prompting We are experimenting with different prompting methods. Beyond the few-shot prompt we discussed in our paper, we’re also testing models with zero-shot prompting and Putnam few-shot prompting. Our Putnam few-shot prompt is the same prompt + few shot examples as our few-shot prompt, except we inform the models that they are looking at problems from the Putnam exam.

Currently, we’ve chosen some better performing Open Source models to run these experiments and intend on expanding to GPT and Claude for the final paper. The results are shown in the table below As expected, zero shot prompting resulted in a lower accuracy than few shot. However, what is interesting is that by telling the model they were looking at Putnam problems, we see a differing result across models. This is something we intend on investigating more with more models.

We plan on running a suite of other prompting methods (including Progressive Hint Prompting, CoT, and changing the few shot examples to be Putnam problems) on a larger set of models as soon as we finish expanding our dataset.

Comparative analysis across benchmarks A quick examination shows that Putnam-AXIOM is significantly more challenging than its contemporary benchmarks. GPT-4's performance demonstrates this:

• Putnam-AXIOM: 9.32%
• SciBench math: 44.52%
• JEEBench math: 28%
• ARB symbolic math: 18%

“Is the evaluation process automated or does it require a human with considerable amount of mathematical rigour and expertise to sit down and check the responses of LLMs to see if the answers?”

Thank you for this note; we’ll emphasize this point in our revised paper. One of the key contributions of our benchmark is that it does not require human graders to be evaluated. The entire evaluation process, like MATH, is automated with boxed answers and a built-in equivalence function.

Thank you for your thoughtful review and for highlighting the importance of our work in mathematical reasoning evaluation and benchmark development. For calibration, we'd like to note that ICLR 2025 uses a different rubric than other conferences; for instance, a rating of 8 at ICLR 2025 only indicates a basic "Accept". Furthermore,

• "Accept" is 8 at ICLR 2025 (vs 7 at NeurIPS 2024)
• "Strong Accept" is 10 at ICLR 2025 (vs 9 at NeurIPS 2024)
• ICLR 2025 does not use 7 or 9

We kindly request to take this into account when reviewing our paper.

“The paper is a little bit short on how new problems are generated.”

Thank you for this feedback; we will elaborate further on how new problems are generated in our revised paper. New problems are generated utilizing Python functions and f-strings. We randomly generate variable names and constants (for constant change questions) to input as parameters, and our functions return the strings corresponding to the question statement and solution. We gather these function outputs in our variation dataset.

“How important are the two functional changes that can be made to the problems. That is, variable change and constant change. Intuitively one would say constant change. Have you investigated this?”

On an older version of the dataset, we conducted an analysis of variable change VS constant change. We found that constant changes were significantly more difficult than the variable changes. We currently plan on running similar analyses once we have finished expanding our variation dataset and will report on our results once finished.

“When would you consider the benchmark to be solved?”

The community norm for when a benchmark has become saturated is when models can consistently obtain over 90% accuracy. Following this standard, we'll consider our benchmark solved when SOTA models can reliably achieve 90% accuracy.

“How do you generate/sample new problems? And how to you make sure that you sample problems uniformly form the problem space. It might be that all the problems you sample fall into the same region and that the LLM could pick up on the statistics of this once we have train data contamination. This ties into the question on when you would consider the benchmark solved. As it might be possible that a different sampling strategy might again lead to degrading performance.”

Thank you for pointing out this concern. We have sampled problems evenly from subject area, difficulties, and answer type. Here is the breakdown:

'Combinatorics': 58, 'Algebra': 61, 'Geometry': 43, 'Calculus': 75, 'Analysis': 73, 'Number Theory': 95, 'Probability': 21, 'Trigonometry': 17, 'Linear Algebra': 25, 'Complex Numbers': 10, 'Differential Equations': 1. Note that in our classification, we allowed questions to be classified under multiple domains.

Based on Putnam’s own difficulty rating (1, 2 = ‘easy’, 3, 4 = ‘medium’, 5, 6 = ‘hard’), we had 99 easy, 78 medium, and 59 hard. We had 78 modified boxing questions and 158 original boxed questions.

In particular, we have an even distribution between final-answer type questions and proof-based questions, which we represent using modified boxing. Furthermore, the distribution in subject areas reflects the inherent distribution of problems in the original Putnam exam.

“Is it correct to say that is an LLM can reason it will be able to solve to benchmark but the inverse is not true. I.e. if it solves the benchmark it might not perform reasoning.”

LLM performance on our benchmark would not definitively determine whether LLMs can reason, but due to the creativity and mathematical logic needed to solve questions as difficult as Putnam’s, it provides positive evidence of mathematical reasoning capability.

Thank you for your thoughtful review and for highlighting the importance of our work in mathematical reasoning evaluation and benchmark development. For calibration, we'd like to note that ICLR 2025 uses a different rubric than other conferences; for instance, a rating of 8 at ICLR 2025 only indicates a basic "Accept". Furthermore,

• "Accept" is 8 at ICLR 2025 (vs 7 at NeurIPS 2024)
• "Strong Accept" is 10 at ICLR 2025 (vs 9 at NeurIPS 2024)
• ICLR 2025 does not use 7 or 9

We kindly request to take this into account when reviewing our paper.

“Variants creation is very simple.”

Thank you for pointing out this concern; we’ll emphasize this in our revised submission. Our purpose in creating variations for the Putnam questions is to expose gaps in LLM performance––namely, potential data contamination of Putnam problems. Additional complexity in the variations beyond their existing levels would have been unnecessary.

“How is testing for equivalent answers combined with the teacher forced metrics?”

Thank you for pointing out this area of confusion; we will clarify this in our revised paper. Most current benchmarks––including ours––prompt the model to utilize boxed answers, and then use equivalence functions to compare boxed answers to the ground truth final answer. TFA is a proxy metric that doesn’t solely rely on the comparing equivalent boxed answers but rather considers the entire model’s response holistically and evaluates its reasoning. TFA does not utilize boxed answers or the equivalence function in computing its score for model responses.

“Can you describe the setup for the evaluation of the proxy metrics? Was teacher forcing conducted on the boxed answer or on the entire rationale?”

Thank you for pointing out this potential area of confusion; we will clarify this in our revised paper. As mentioned above, the proxy metrics are run on the entire rationale provided by the model. TFA does not utilize the boxed answers in any way in computing its scores.

Thank you for recognizing the quality and difficulty of our dataset. However, since our previous comment we have made several substantial improvements that we hope further solves your concerns.

Dataset Size and Comprehensiveness Your concern about dataset size was well-taken. We have significantly expanded Putnam-AXIOM:

• Increased the Original benchmark size from 236 to 522 boxed question-answer pairs by incorporating problems from 1938-1984. The updated breakdown for the dataset follows - ‘Combinatorics': 79, 'Algebra': 193, 'Geometry': 125, 'Calculus': 143, 'Analysis': 177, 'Number Theory': 135, 'Probability': 29, 'Trigonometry': 26, 'Linear Algebra': 41, 'Complex Numbers': 29, 'Differential Equations': 20. Note that in our classification, we allowed questions to be classified under multiple domains. Based on Putnam’s own difficulty rating (1, 2 = ‘easy’, 3, 4 = ‘medium’, 5, 6 = ‘hard’), we had 205 easy, 182 medium, and 135 hard.
• Nearly doubled our functional variations from 53 to 100 (representing a ~30% conversion rate to constant + variable changes of our 522 original questions, with ongoing expansion estimating 150 variations)
• This expansion maintains the benchmark's mathematical rigor while providing more comprehensive coverage

We have further expanded our evaluations across more prompting techniques. In our Putnam Context prompt, we include in the prompt that our problems are from the Putnam Competition.

The following accuracies are on the new dataset with 522 questions. :

The rapid pace of these improvements (all accomplished within one week) demonstrates both the scalability of our approach and our team's commitment to continuous enhancement. We are actively working on further expansions and improvements to make Putnam-AXIOM an even more comprehensive benchmark for mathematical reasoning.

Furthermore, to build upon previous results, we're expanding the evaluations to models with larger parameter sizes and architectures with some preliminary results already (currently only few shot but we plan to have all prompting styles ready for the final version):

• Codestral-22B-v0.1: 4.66
• Qwen2-Math-72B: 11.9
• Qwen2.5-Math-72B: 10.6
• Llama-3.1-70B: 4.24
• Mixtral-8x7B-Instruct-v0.1: 5.93
• Mixtral-8x22B-Instruct-v0.1: 8.47
• Mistral-Small-Instruct-2409: 5.93

Thank you for your thoughtful review and for highlighting the importance of our work in mathematical reasoning evaluation and benchmark development. For calibration, ICLR 2025 uses a different rubric than other conferences:

• "Accept" is 8 at ICLR 2025 (vs 7 at NeurIPS 2024)
• "Strong Accept" is 10 at ICLR 2025 (vs 9 at NeurIPS 2024)
• ICLR 2025 does not use 7 or 9

We kindly request to take this into account when reviewing our paper.

“According to the paper, the proposed evaluation metrics cannot be used to evaluate proprietary models ...”

Thank you for pointing this out; we acknowledge this limitation and invite future work to design new and better proxy metrics. We will clarify this point in our revised paper. For proprietary models, it’s not always possible to run TFA. For instance, with the API for GPT 4, 4o, and o1, it’s impossible to prefill the “assistant message,” which is a crucial step for the TFA metric. Like Huang et al. [1], where BPC calculation requires internal token probabilities unavailable through closed APIs, reflecting a common challenge in similar research. However, our ongoing TFA experiments with Claude 3.5 Sonnet show promising results via subsampling, which we'll include in our revision.

[1] Huang, Y., Zhang, J., Shan, Z., & He, J. (2024). Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937.

“The LLMs evaluated are relatively small outside of the few state-of-the-art models ... How does larger models like llama-70b perform?”

Updated numbers:

• Codestral-22B-v0.1: 4.66
• Qwen2-Math-72B: 11.9
• Qwen2.5-Math-72B: 10.6
• Llama-3.1-70B: 4.24
• Mixtral-8x7B-Instruct-v0.1: 5.93
• Mixtral-8x22B-Instruct-v0.1: 8.47
• Mistral-Small-Instruct-2409: 5.93

“The dataset is relatively small, and is not easily extendable since a lot of manual work is required for new problems ...”

Thank you for pointing out this potential issue with the size of our dataset. During this rebuttal period, we have more than doubled the size of the Putnam-AXIOM Original dataset. We also plan on creating more variations for the new questions we’ve added. Unfortunately, manually editing and creating problems may be the best currently-available way to generate consistent, high-quality problems for a dataset. Due to the limited reliability of automated methods like LLMs, automatically generated questions would still require manual verification––also a time-consuming process.

“I am confused why the paper did not present TFA scores for SOTA models in table 1 ...”

Thank you for pointing out this issue with the TFA table sections. As we mentioned above, it’s not always possible to run TFA for closed-source models. For open source models we can efficiently compute teacher forcing on an entire string with a single forward pass similar to how these models are trained. However, frontier model APIs only let us generate tokens autoregressively. Thus each new token is conditioned not on the ground truth solution as in TFA, but on the previously generated solutions. For instance, with the API for GPT 4, 4o, and o1, it’s impossible to prefill the “assistant message,” which is a crucial step for the TFA metric though we are optimistic about Claude3.5 using a subsampling approximation.

“How is correlation calculated for evaluating the TFA metric?”

Thank you for pointing this out. We collect a large list of 20 models. We first compute model accuracy on MATH. We then compute the proxy metric for each model on MATH. We compute the correlation between model accuracy and the value of the proxy metric for each model (i.e. we compute the correlation on 20 data points). We will clarify this in our revision.

“What LLM model is used to produce table 2 to evaluate the metric? Does TFA out perform other metrics using different sized models?”

Thank you for pointing this out. We will make this more clear in our revisions. To create Table 2, we used 20 diverse models ranging from 7B to 72B parameters, including Meta-Llama-3-70B, Mixtral series, Llama-2 variants, Qwen models, deepseek, EleutherAI's llemma, and Google's Gemma models.

“The paper claims that the generated variations of the dataset has the same level of difficulty as the original. However, won’t perturbing the constants can sometimes make a problem more difficult? Would it still be a fair comparison if each LLM is evaluated on different variations?” Thank you for pointing out this issue with potential difficulty inconsistency. We intentionally designed our variation dataset to maintain consistent difficulty through two design principles:

• Variable changes simply rename variables (e.g., 'x+5=6' to 'y+5=6') without affecting complexity.
• Constant changes are limited to problems requiring the solver to derive general-form solutions or those with predictable relationships (like linear scaling) between question constants and solutions

These constraints ensure that variations remain at the same difficulty level as the original problems.

Thank you for your thoughtful review and for highlighting the importance of our work in mathematical reasoning evaluation and benchmark development. For calibration, we'd like to note that ICLR 2025 uses a different rubric than other conferences; for instance, a rating of 8 at ICLR 2025 only indicates a basic "Accept". Furthermore,

• "Accept" is 8 at ICLR 2025 (vs 7 at NeurIPS 2024)
• "Strong Accept" is 10 at ICLR 2025 (vs 9 at NeurIPS 2024)
• ICLR 2025 does not use 7 or 9

We kindly request to take this into account when reviewing our paper.

“ROSCOE metrics are not applied to the evaluation in Putnam-AXIOM and the boxed answer requirement discards intermediate steps to be used in the performance metric, so it doesn’t address the known LLM’s logical mistakes and leaps in the reasoning process, even when getting the right answer, despite being a problem identified by the authors in the beginning of the paper. The proposed TFA proxy metric is not evaluated on proprietary models so the state-of-the-art models are left out of the most comprehensive evaluation.”

Thank you for pointing this out; we will clarify this point in our revised paper. We recognize the limitations of boxed answers––namely, that they don’t address logical leaps and errors in the model’s responses if they lead to the correct answer. Additionally, many problems do not have boxable answers (i.e. proofs or complex formulae) which limits the scope of current benchmarks. That the main unique strength behind using ROSCOE, TFA, and other proxy-metrics for reasoning. In particular, TFA and our other proxy metrics are applied to the entire reasoning trace, not just the boxed answer. We choose the MATH dataset to run our proxy metric evaluations because we can get a wide range of model performances on the dataset (Putnam-AXIOM does not have any high performing models). Additionally, we found a high degree of alignment between correct reasoning and correct final answer for this dataset (In particular, we found only 3 out of 75 correct final answers did not have correct reasoning we exampling a random sample). Thus we believe it is reasonable to compare against boxed accuracy on MATH as a substitute for correct reasoning traces. We discovered that the ROSCOE metrics had a poor correlation with boxed accuracy compared to TFA and our other proxy metrics on the MATH dataset, so we did not include it for Putnam-AXIOM. For proprietary models, it’s not always possible to run TFA. For instance, with the API for GPT 4/4o, and o1, it’s impossible to prefill the “assistant message,” which is a crucial step for the TFA metric. This limitation mirrors challenges faced in similar research, such as Huang et al. [1], where calculating Bits Per Character (BPC) requires access to internal token probabilities inaccessible through closed model APIs. However, we’re currently experimenting with TFA for Claude 3.5 Sonnet and see promising results with a subsampling approximation. We’ll include these in our revised submission.

[1] Huang, Y., Zhang, J., Shan, Z., & He, J. (2024). Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937.

“The 52 variations can give rise to an unlimited set of problems but they were created manually, and their logic can eventually be data contaminated as well which is not addressed. Also it is a too small dataset.”

Thank you for the feedback on this potential issue with our dataset size and contamination. We've already doubled the size of Putnam-AXIOM Original, have 20 new variations created, and are planning to add more to both datasets. Further, our dataset size now aligns with other reasoning benchmarks (JEEBench: 515, ARB: 105, SciBench: 692), while being significantly more challenging (GPT-4 performance):

• Putnam-AXIOM: 9.32%
• SciBench math: 44.52%
• JEEBench math: 28%
• ARB symbolic math: 18%

“Is there a way of automating the creation of new Putnam-AXIOM variations in order to avoid Logic data contamination? Do you agree Logic data contamination can happen, if not, why not?”

Thank you for pointing out this potential issue with logic data contamination and for your suggestion on automating variation creation. We are currently experimenting with fine-tuning models on the corresponding 53 original questions to our variations, then evaluating the models on our variations. The results of this experiment will determine the significance of logic contamination. We also examined the possibility of using LLMs to automate the creation of variations by prompting GPT-4o to perform variable changes on several Putnam questions. While it succeeded in doing so with a small volume of questions, we found that GPT-4o often failed in keeping the mathematical content of the variable changes the same altering constants or equations. Given the unreliability, the variations would have to be human-verified and corrected anyways, which would likely take as much––if not more––than just selecting and creating variations manually.