Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation

We thank the reviewer for their positive comments. Below, we answer the main questions raised by the reviewer:

Q1: For question with 2 or 3 answers, how do you handle them?

A1: For the LiveAoPSBench evaluation set, we only retain questions with closed-form answers. If a solution provide multiple valid answers, they are summarized as a list of closed form answers. After this, we remove all questions that show discrepancies—i.e., different closed-form answers—between the community-provided solutions(see Section 3.2 LLM cross-checking).

In the AoPS-Instruct training set, we include all recognized solutions based on forum discussions for two reasons: (1) Most of the Q-A pairs in training set either have no final answer (e.g., proof-based questions) or the answer is free form, making it hard to rely on voting mechanisms to resolve discrepency. (2) Prior works such as Alphacode[1], have successfully used partially correct solutions for training their models. Therefore, we choose not to apply majority voting filters, allowing users to customize the filtering according to their needs.

[1] Li et al., Competition-Level Code Generation with AlphaCode, 2022

We thank the reviewer for their constructive feedback. Below, we answer the main concerns raised by the reviewer:

Q1: The evaluation is limited to the LLMs with fewer than 7B parameters. To strengthen their claims, the authors should extend their experiments to larger-scale LLMs (exceeding 7B parameters) and provide a more comprehensive assessment of the proposed method’s effectiveness.

A1: We do not currently have the resources to fine-tune larger than 7B models, but we believe we provide substantial evidence that we hope motivates the community to push this further. To reiterate on some of the evidence we show in the paper:

• Table 2 presents results for two 7B math-specialized models, alongside Llama 1B and 3B models, all consistently demonstrating improved accuracy across the board.
• Table 6 in the appendix shows that performance gains transfer from a smaller 1.5B rewriting model to a larger 7B model.
• For evaluation, we tested models up to 72B on our LiveAoPSBench, highlighting the benchmark’s difficulty and its correlation with uncontaminated benchmarks (Figure 5a).

Q2: The evaluation set primarily consists of problems with concrete numeric or symbolic final answers. This excludes a large portion of Olympiad problems that require more open-ended proofs, which the pipeline and automated evaluations currently do not fully capture.

A2: Evaluating proof-based questions has remained a long-standing and persistent challenge in the mathematical reasoning community, with no widely accepted solution outside formal languages like Lean—which fall beyond the scope of our work. A key limitation is the lack of large-scale training data for informal proof-based problems. As noted in our paper, approximately 30% of our training dataset consists of proof-based questions. We are optimistic that this dataset will support the development of new methods for evaluating proof-based reasoning.

We thank the reviewer for their constructive feedback. Below, we answer the main concerns raised by the reviewer:

Q1: No conflict resolution protocol for handling discreprancies in community-provided answers (e.g., voting mechanisms).

A1: For the LiveAoPSBench evaluation set, we only keep questions that have a closed-form answer, and we handle discreprancies between community-provided via LLM cross-check.

In the AoPS-Instruct training set, we include all recognized solutions based on forum discussions for two reasons: (1) Most of the Q-A pairs in training set either have no final answer (e.g., proof-based questions) or the answer is free form, making it hard to rely on voting mechanisms to resolve discrepency. (2) Prior works such as Alphacode[1], have successfully used partially correct solutions for training their models. Therefore, we choose not to apply majority voting filters, allowing users to customize the filtering according to their needs.

[1] Li et al., Competition-Level Code Generation with AlphaCode, 2022

Q2: Lack of validation for each step in the dataset construction pipeline. (whether LLM cross-cheching effectively removes incorrect answers.)

A2: In Figure 5.b we show the effectiveness of LLM rewritting. The effectiveness of LLM cross-checking has been qualitatively verified by human annotators. In Section 4.4, a human-annotated error rate of 8% was reported, whereas we empirically observed an error rate of over 20% before applying LLM cross-checking.

Q3: The current approach detects duplicate data via substring matching but fails to identify semantically equivalent problems with different wording or multilingual translation issues (e.g., Chinese-to-English translations). Could semantic similarity be used instead?

A3: N-gram decontamination is a standard practice in math data. We follow the DeepSeek-Math [2] paper, which uses 10-gram decontamination. Qwen-Math also applies decontamination based on 13-grams [3]. [4] Leverages the LLM for semantic based decontamination, but is not widely adpoted due to its inefficiecy when scaling up, and high false-positive rate.

[2] Shao, et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024 [3] Yang, et al., Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement, 2024 [4] Yang, et al. "Rethinking benchmark and contamination for language models with rephrased samples." arXiv preprint arXiv:2311.04850 (2023).

Q4 The paper does not effectively demonstrate that timestamps mitigate contamination. There is no quantitative analysis of new problem duplication rates, such as statistics on overlap between different time periods and previous datasets.

A4: Thank you for the suggestion! We demonstrate the correlation of timestamps vs contamination by decontaminate against the massive Numnia Training set following the decontamination set up of Qwen2.5[3]. Below we report a change of contamination rate w.r.t to timestamp. We report the statistics of overlap against Numina CoT Traing dataset (released July.2024) as below:

As we can see, the rate of potential contamination significantly decreases as timestamps increase. We will add this info to our revised manuscript.

Q5: NuminaMath also sources data from AoPS and rewrites solutions using GPT-4o. How does your approach compare to theirs, and what advantages does it offer?

A5: While Numina source some problems from AoPS contest pages, their problem-solution extraction mechanism is very simple, leading to very small number (30K QA pairs) of extracted solutions compared to our method. Their method only extracts problems from contest page (where the problems are formally written), and their solution extraction method takes the longest post which contains a black box at the end of the proof.

Instead, we explore the entire AoPS forum discussions, where questions and solutions are shared in unstructured posts by community members. This requires extensive use of LLMs for data filtering and solution extraction, allowing us to extract 652K QA pairs (in contrast to Numina's 30K), resulting in a 21-fold increase in extracted data from AoPS. These improvements result in a stronger dataset, leading to higher performance gains as shown in Table 2 of our paper.

We thank the reviewer for their constructive feedback.

Q1: In Table 2, the size of the datasets on which the models have been finetuned have not been specified. It would be important to ensure that the amount of data on which the models are finetuned is same across different datasets to ensure fair comparison

A1: We used the full Numina dataset, decontaminated against our evaluation benchmarks, resulting in 824K data points. In comparison, our dataset contains 647K data points. Despite being smaller, our dataset demonstrates a greater performance improvement than Numina, as highlighted in Table 2. We will add this information to our revised manuscript.

Q2: For the comparison in Table 2, it would be useful to have a comparison with a stronger baseline. Several powerful synthetic datasets (for eg. ScaleQuest [1] and DART-Math [2] are plausible candidates).

A2: ScaleQuest: Thank you for bringing our attention to this very recent work. We will add a citation to this work as a concurrent work.

DART-Math: Thank you for pointing this work out. Below, we take the DeepSeek model from the DART-MATH paper, and compare it with our DeepSeek-fine-tuned model:

We observe that our model outperforms the DART-MATH model on all benchmarks. We will add this table to our paper.

Q3: The number of seeds over which the performances are computed has not been reported.

A3: We measure the Pass@1 metric in all of our tables, and for non-R1 models, the common practice is to set a temperature of zero for performance measurement. Therefore, there is no stochasticity in the metric to repeat for multiple seeds. This is in line with other works in mathematical reasoning (e.g., see [2] and [3]) where there are no standard deviations to report for Pass@1.

Q4: It would be useful to report the performances of some state of the art models such as o1, DeepSeek-R1, DeepSeek-R1 distilled series, etc. on LiveAoPSBench. This would give a better idea of the difficulty level of the benchmark.

A4: We have reported the performance of R1-distilled models in Figure 8 in the Appendix. However, assessing the full R1 and o1 models is both computationally and financially prohibitive due to their high operational costs, which prevents us from including their results.

Q5: Are any measures taken to remove the 5% incorrect and 3% no-answer category questions from the final evaluation set? The presence of these amounts of degenerate questions could cause significantly wrong judgments about the performance of models.

A5: LiveAoPSBench is a dynamically evolving benchmark, and we continuously work to refine the pipeline as we identify failure cases. We should note that current large language models perform well below this error-rate threshold (e.g., DeepSeek distilled models achieve only 52.2% accuracy on our benchmark). Moreover, even the human-labeled GSM8K dataset is known to contain a small percentage of label errors [1], yet it remained a highly discriminative benchmark until state-of-the-art models began to saturate it. Thus, while we are committed to enhancing LiveAoPSBench, we believe it remains valuable and informative in its current state.

Q6: In Step 3 (solution rewriting), how is the solution that is to be re-written selected out of all the solutions provided by different users as extracted from the forum.

A6: We input the whole topic discussion to the LLM, and the LLM must infer correct post indexes that contain correct solution. This is done by analyzing the forum discussion to detect correct answers. We use few-shot chain-of-thougt prompt to extract these (see Figure 12 for the exact prompt, and our code for few-shot examples).

Q7: Why is Llama-3.1-70B used for Question-Answer Extraction as compared to a chat variant of the Qwen-2.5 series.

A7: We apologize for the typo. In fact, we utilize Qwen-2.5 across the entire AoPS-instruct pipeline (as illustrated in Figure 1 and Table 1). We will correct this mistake in our revised submission.

Q8: Another work which attempts to tackle the problem of evaluation data contamination includes Shah et al., which uses an AI-Assisted approach for creating new and difficult mathematical questions.

A8: Thank you for highlighting this relevant work. The work of Shah et al. aims to produce an evaluation dataset that tests compositional generalization by combining pairs of distinct mathematical skills extracted from the MATH dataset. This differs from our methodology, which emphasizes dynamically evolving, community-driven problem-solving data. We appreciate the suggestion and will include a citation to this work in our revised manuscript.

[1] https://gradientscience.org/platinum-benchmarks

[2] Yang et al., Qwen2.5-Math

[3] Shao et al., DeepSeekMath