MathCAMPS: Fine-grained Synthesis of Mathematical Problems From Human Curricula

We thank the reviewer for the positive remarks on our work, and for the detailed comments! We’ve addressed the points for clarification below:

While the proposed method demonstrates a degree of scalability, it is primarily limited to K-8 level mathematics. The experiments conducted in this study show a strong consistency with evaluations from the GSM8K benchmark. However, I don't think as more powerful models, such as OpenAI's o1, emerge, this benchmark will present significant challenges. Therefore, the potential for extending this framework to more complex problems is a key area for improvement in this work.

We agree with the reviewer that frontier models overall perform well on MathCAMPS. However, the results still show large variability in performance when looking at individual skill, demonstrating the importance of fine-grained benchmarks to shed light on specific capabilities and failure modes. Specifically, the granularity from the evaluations in MathCAMPS allows for interpretability analysis in LLM math reasoning skills which isn’t frequently conducted. Moreover, as our analysis with Pythia shows, our dataset provides a unique opportunity to understand training dynamics, and how particular skills are learned, which is interesting even for simple mathematical skills.

Additionally, the paper's heavy reliance on the Common Core (CC) standards results in an inability to generate mathematical problems outside of these guidelines. Consequently, more complex problems (those above K-8) or those that fall within K-8 but are not included in the CC standards cannot be synthesized.

The MathCAMPS pipeline can be expanded to problems outside of K-8. The Common Core covers grades K-12, and other curricula exist for subjects like Calculus and Linear Algebra that would enable the creation of grammars and solvers to support MathCAMPS-Calculus and MathCAMPS-LinearAlgebra versions. The main challenge would be standards within the Common Core that are more open-ended (e.g. interpreting tables), since conceptual questions are difficult to grade.

How does the author ensure the correctness of encoding the CC Standard into rules? Did the author conduct manual cross-validation or sampling tests?

Thanks for the good question! When encoding the standards, we made sure to look at teacher-created worksheets online (several are available on a CC standard basis on websites like https://www.commoncoresheets.com/), to ensure that our problems reflected the content from real examples. Evaluating this step would likely involve a large-scale study with teachers. Having the perspectives of teachers would be especially important for potential educational applications of our work. For instance, as we mention in the paper, we believe there is a potential to use our pipeline to generate custom problems for human students, fixing the mathematical skill (which can follow their curriculum) but varying the story in a customized way. This application of synthetic problems for evaluating or teaching human students is, however, a separate endeavor that is worth doing right — we would need to recruit an expert population of teachers in order to annotate a spectrum of different problems, or recruit students of appropriate grade levels in order to establish psychometric validity of the problems, if they were to complement existing worksheets. While these evaluations would go beyond the scope of the current work, we will expand on our discussion into what these would entail for future work in our conclusion.

Are there any methods to extend this pipeline to more difficult topics?

Yes! The basic premise for MathCAMPS is: (1) randomly sample a symbolic problem, (2) translate the symbolic problem into a word problem, (3) back-translate the word problem into a symbolic problem, (4) check for answer agreement between the two symbolic problems. Steps 1, 2, and 4 rely on the existence of symbolic solvers for a certain subtopic in math (for example, the python pyro library for probabilistic problems). As long as we have access to a symbolic solver for a subtopic and a strong LLM, more problems can be generated. Generally, Computer Algebra Systems like sympy (the one we use) already cover a very wide range of topics (including calculus, linear algebra, geometry, physics, combinatorics, etc), and these all can thus be integrated with the same ideas.

We thank the reviewer again, and we'd be happy to engage further if there are any outstanding questions or concerns!

We thank the reviewer for the thorough evaluation of our work! We're pleased that you found our use of the Common Core standards and the dialogue setting novel, and appreciated the insights from the Pythia training. We respond to concerns and questions below.

This work has gone in a similar direction as many papers, [...] GSM-Symbolic [3], MORE [4] has provided a vast coverage of different variations that probes the robustness of arithmetic reasoning abilities. MORE provides a vast ontology of perturbations and GSM-symbolic provides many useful templates. Also, while GSM-symbolic is fairly recent, MORE has been published many months back. Given the similarity, it should be mentioned and compared with.

We thank the reviewer for pointing out these pieces of related work. We included a discussion of GSM-Symbolic and GSMore in our Related Work section (see Section 2, paragraph "LLM-generated synthetic datasets for LLMs" in the updated paper). For convenience, we also include the discussion here. Both GSM-Symbolic (only released after the ICLR deadline) and GSMore are interesting works that focus on probing the robustness of LLMs by evaluating how their answers change under different semantically-preserving transformations of existing problems. Instead of relying on existing datasets, we note that MathCAMPS generates problems without the need for a seed dataset. Moreover, our focus is on providing an evaluation that is grounded on a human curriculum, which is not a desiderata of either of these works. This is our focus, more than robustness per se, although our follow-up question analysis also shows failures in robustness in most LLMs. We strongly believe that these are complementary analyses of mathematical skills of LLMs.

Unfortunately, the paper therefore is not as novel as it claims to be.

We are happy to revise our claims if the reviewer believes we stated more novelty than is warranted. However, we refer to our stated contributions at the end of the introduction: (1) a synthetic evaluation that is fully grounded on a human curriculum, (2) the cycle consistency method for evaluating faithfulness, providing a fully synthetic pipeline (note that GSMore required manual intervention to filter out errors in generation, showing the challenge of ensuring data quality in a fully automated pipeline. We tackle this challenge here, allowing our pipeline to scale up cheaply), (3) a method to generate follow-up questions, and (4) the analyses of mathematical skills emerging in Pythia checkpoints. We believe these are fair claims in light of where existing work is at.

How faithful is the representation from CC NL guidelines to the formal grammar? Have the authors evaluated this step independently?

When encoding the standards, we made sure to look at teacher-created worksheets online (several are available on a CC standard basis on websites like https://www.commoncoresheets.com/), to make sure that our problems reflected the content from real examples. Evaluating this step would likely involve a large-scale study with teachers. Having the perspectives of teachers would be especially important for potential educational applications of our work. For instance, as we mention in the paper, we believe there is a potential to use our pipeline to generate custom problems for human students, fixing the mathematical skill (which can follow their curriculum) but varying the story in a customized way. This application of synthetic problems for evaluating or teaching human students is, however, a separate endeavor that is worth doing right — we would need to recruit an expert population of teachers in order to annotate a spectrum of different problems, or recruit students of appropriate grade levels in order to establish psychometric validity of the problems, if they were to complement existing worksheets. While these evaluations would go beyond the scope of the current work, we will expand on our discussion into what these would entail for future work in our conclusion.

We thank the reviewer for their encouraging comments and detailed suggestions! We’ve addressed all the specific points below:

The synthesized problems depend on the capabilities of GPT-4. If the questions are particularly difficult, such as those on the AIME, this framework may fail due to insufficient synthesis model capabilities. However, manually designed questions can mitigate this issue.

We note that our framework does not depend on the capability of the generator model to solve the problem. Rather, it only needs to be able to translate the symbolic problem into a word problem (and back, for cycle consistency). This is a much easier task, and in fact we get many problems that GPT-4o does not solve, as our analysis shows, despite it having generated the original problem. This is because we obtain the answer symbolically, rather than relying on the model itself. MathCAMPS can also be expanded to more challenging domains like calculus and linear algebra, using appropriate solvers (sympy, which we already use, can in fact solve many problems in these domains), again not relying on the model to solve them correctly.

MathCAMPS is only compared to GSM8K, which is insufficient, as GSM8K contains only simple elementary algebra problems.

We chose to compare to GSM8K since our benchmark covers skills at the K-8 level, which is somewhat similar to the coverage offered by GSM8K. Benchmarks like MATH are inherently more challenging (given their content), making them an unsuitable point of comparison.

It cannot be guaranteed that the synthesized mathematical problems are always reasonable or that the answers will consistently match.

While there is no formal guarantee, we manually evaluated 245 problems and found that 97.7% of the problems were high quality (i.e. the numerical answer matched the word problem). Notably, human-curated datasets often also have unreasonable questions. [1] shows a few examples of such problems found in GSM8K and SVAMP, suggesting that their noise rate can potentially be higher than what we get with MathCAMPS (2.3% false positives).

[1] https://huggingface.co/datasets/Cleanlab/bad_data_gsm8k_svamp.csv

When sampling different questions, the model's performance is expected to show subtle variations. The authors should sample multiple test sets to assess the variance in the model's performance.

We ran a study on this in Appendix B ("Familiarity Bias"), generating a separate dataset with a separate model. We used Claude Opus to generate a 10% scale dataset, which we evaluated Claude and GPT-4o on, and overall found our results to be robust regardless of the model used to generate the dataset.

The authors should discuss how the framework can be extended when we need to synthesize more complex problems.

We address how the framework can be expanded in the conclusion. Specifically, the framework can be expanded to subtopics in math that have solver libraries available (e.g. calculus and linear algebra). This is because the underlying idea of sampling a symbolic problem, translating it into a word problem, and back translating it back to the symbolic structure is domain-agnostic.

Thanks again for the thorough review! We're happy to engage further if you have any remaining concerns.

We thank the reviewer for their detailed comments and suggestions! We address each point inline below:

While the paper presents extensive evaluations, it often falls short in providing detailed insights or intuitive explanation for the more surprising failures or performance gaps observed in some models.

We appreciate the reviewer’s suggestion to be more specific about the failures and performance gaps we noticed. We have updated the paper with a new sub-section 4.2, including a few of the standard-wise analyses which we found to be the most surprising. We note that our supplementary material has extensive results that we made easy for users to browse and explore, much more than we had space to discuss in the paper.

A potential limitation of the cycle consistency check is whether it is sufficient on its own. It only verifies the reproduction of the rules, but it does not account for intermediate consistency or potential intermediate issues in the generation. The imperfections of this cycle-consistency check could potentially remove good questions.

In our manual examination, we found it extremely rare for the cycle consistency to not catch intermediate consistency issues (see Section 3.2: only 2.3% of false positives). To illustrate why, suppose we have a symbolic problem S, generated from our grammar. We then generate a word problem W from S. In cycle consistency, we generate an S' only from W (but not S), and check that both S and S' have the same answer (which is checked symbolically, without the LM). This ensures that, if there was an intermediate issue in generating W (for example, the model invented constants, or inverted a relationship between variables, or generated an ambiguous problem), it's very hard to reconstruct a problem that is equivalent to S. Conversely, we found our false negative rate to be very low (3.3%, also in Section 3.2), so few good questions are filtered out. Of the 7 correct problems that were discarded, we noticed the following distribution: 6 problems had near-correct back translations into a symbolic structure. The error lied in syntax (e.g., a missing closing bracket, or invalid variable name). Only 1 of the 7 good discarded problems had a genuine mathematical error in its back translation. We agree this is informative for readers, and have included a discussion of these error cases in the appendix (section D).

The framework only covers 44 reasoning behaviours, with some being very basic and "easy" for current models.

We included skills from a range of difficulties to enable true fine-grained measurement of model abilities. So while higher grades do often compose of skills from lower grades, our dataset enables researchers to understand model failures better. For example, if a model is particularly bad at adding fractions, is it because it is bad at addition or fraction comprehension? Knowing what “easy” skills contribute to model failure on more challenging skills allows for a much finer-grained analysis than previous datasets, which only provide aggregate accuracies.

the paper mentions "Prompting the LLM to back-translate the word problem into a symbolic structure and comparing the new answer to the original enables us to eliminate most unfaithful generation errors and maintain high quality." Could you provide some examples of these unfaithful generations? This would help better understand the soundness of the approach.

Sure! We have added multiple examples of unfaithful problem generations to the appendix, and include two of them here. As you can see, both generated natural language problems are unfaithful to the original structure, and this is caught by the newly generated symbolic structure not having the same final answer.

Original symbolic structure: [[var f = (5 - 4)]]\n[[var t = (4 - f)]]\n[[question d = ['t']]]\ntheme: Rocket Generated word problem: In the space exploration research center, Diego has 5 rockets. He launched 1 rocket for testing purposes. After this, he orders 4 new rockets, but doesn't count the one that he just launched. How many rockets does Diego have now? New symbolic structure: [[var initial_rockets = 5]]\n[[var launched_rockets = 1]]\n[[var ordered_rockets = 4]]\n[[var total_rockets = (initial_rockets - launched_rockets + ordered_rockets)]]\n[[question result = ['total_rockets']]]

Original symbolic structure: [[var v = (79.24 * 37.6)]]\n[[question s = ['v']]]\ntheme: Treasure chest Generated word problem: A pirate finds a treasure chest full of golden coins. Each golden coin weighs 79.24 grams. If the total weight of the coins is 37.6 kilograms, how many golden coins are there in the treasure chest? New symbolic structure: [[var weightInGrams = (37.6 * 1000)]]\n[[var n = (weightInGrams / 79.24)]]\n[[question numCoins = ['n']]]