4. Response to Q6: Could you expand on the offline RL baseline and provide details on the data, algorithm, and setup of this experiment?

We would like to clarify that we include details on the data, algorithm and experimental setups in the paper and contain an anonymous code repo for reproducibility.

The current version of the paper summarizes the key features of the datasets and experimental setups in lines 341--373. Due to page limits, we have included further details in the appendix, specifically:

• Section B: Provides an algorithm diagram.
• Section C: Includes additional details about data statistics and experimental setups.

We also release our data and code in an anonymous repository (linked in line 373: https://anonymous.4open.science/r/AnnonySubmission-35F0), ensuring full transparency and reproducibility.

Regarding the reviewer's suggestion to compare our approach with other RL methods or preference-based approaches, we offer the following clarifications:

• Scope of the Proposed Approach:

  • The proposed EM approach is specifically designed to address challenges in autonomous code integration for mathematical LLMs.
  • It is not intended as a general solution for LLM reasoning, and we do not claim its broad applicability in this area.

• Comparative Analysis with General Methods: Common alternatives for general LLM reasoning have been extensively benchmarked in prior research [1-3], which deviates from the primary goal of this paper. In addition, these research have shown that RL with rewards outperforms preference-based methods when verification-based outcome feedback is available.

We understand that conducting a comprehensive comparison with all potential solutions for AutoCode, as suggested, could strengthen the argument for our approach. However, this requires tremendous efforts, and we will work on this once we have resolved the key weaknesses in the current submission.

References

[1] Shao Z, Wang P, Zhu Q, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models[J]. arXiv preprint arXiv:2402.03300, 2024.

[2] Gulcehre C, Paine T L, Srinivasan S, et al. Reinforced self-training (rest) for language modeling[J]. arXiv preprint arXiv:2308.08998, 2023.

[3] Ahmadian A, Cremer C, Gallé M, et al. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms[J]. arXiv preprint arXiv:2402.14740, 2024.

Analysis.

• Effectiveness in Methodology-Selection. AutoCode Training significantly improves methodology-selection accuracy, outperforming baseline models by over 20% (compare row 3 with row 5). In particular, GPT-4 exhibits low mean accuracy (50.51%) due to its strong bias for CoT. If GPT-4 intelligently selected methodologies, it could achieve an additional 7.62% gain in accuracy for queries requiring code responses.
• Accuracy Improvements and Code Reduction. AutoCode Training enables our Deepseek-based model to achieve a 5.04% accuracy improvement compared to the best dictated inference (code prompting) while reducing code usage by up to 66%. Similar trends are observed with the Qwen2Math base model. In contrast, baseline models experience substantial accuracy drops when attempting autonomous methodology-selection. For example, DeepseekMath loses 11.54% accuracy when self-selecting methodologies (row 3).
• Connection Between Methodology-Selection and Final Accuracy. Baseline models surpass random selection (50%) in methodology-selection accuracy but fail to improve final accuracy over best dictation. This is accredited to* the gap between methodology-selection and final correctness*: better methodology-selection is not always accompanied with correct final response, due to the greedy decoding and mismatch in prompt context. However, our proposed EM-based joint training significantly bridge this gap: it improves Pass@1 accuracy within correct selections, achieving rates as high as 95% (rows 4 and 5). This success is due to training the LLM with complete responses, optimizing both methodology-selection and final correctness jointly.

3. Response to weakness comments on basline limitations, effectiveness of autonomous decisions, Q4 of comparison with baselines in methodology-selection and Q5 of correlations between methodology-selection and final performance.

We sincerely appreciate the constructive suggestions to analyze methodology-selection and its connection to final performance. However, it is important to clarify that analysis of methodology-selection as a standalone policy is trikcy. This is because we do not train a separate policy that maps a math query directly to a methodology, nor do we have a definitive "golden" answer for methodology-selection. Instead, methodology-selection and solution-generation are conceptually factorized in this work to ease the difficulty of autonomous code integration. Despite this factorization, these two components share parameters within the same LLM and are not inherently separable.

---

=> Q4: comparisons with baselines in methodology-selection.

=> Q5: correlations between methodology-selection and final performance.

We fully agree that the suggested analysis helps convince the significance of AutoCode Training in the final accuracy. Despite the challenges in analysis mentioned above, we strive to give a reasonable analysis on the effectiveness of methodology-selection and its connection to the final performance. We will include this analysis in our revised manuscript.

We first detail the baselines and evaluation metrics used to answer the above questions.

Baselines. We compare our approach with several models that natively support both code and Chain-of-Thought (CoT) responses for math queries: GPT-4, Mammoth-70B trained using Hybrid Instruction Tuning, and DeepseekMath-Instruct-7B trained using tool-integrated reasoning annotations.

Evaluation Metrics.

• Final Pass@1 Accuracy of the Complete Response

  • "Autocode": The LLM autonomously decides the methodology. For baselines without AutoCode Training, a four-shot prompt template (Appendix D.3) is used.
  • "Code": The LLM is explicitly prompted to generate a code response. For GPT-4, a four-shot template is applied. For other baselines, we use their native zero-shot templates.
  • "CoT": The LLM is explicitly prompted to use CoT reasoning with native templates.

• Autocode CodeRate and Improvement Over Best Dictation

  • CodeRate reflects the reduction in code usage compared to dictated code prompting.
  • Accuracy Improvement over best performance of either CoT or code dictation reflects the improvement of AutoCode over the native dictated inference.

• Methodology-Selection Accuracy and Its Connection to Final Accuracy

  • Ground-Truth Labels: Methodology-selection is treated as a binary classification task. The classification label is derived by performing 10 Monte Carlo rollouts per query with controlled methodologies. The optimal methodology is chosen based on higher expected correctness.
  • Imbalanced Classification: Since model capabilities differ, reference decisions are imbalanced (e.g., GPT-4 strongly prefers CoT, with only 7.45% of queries requiring coding). We report mean accuracy across CoT-preferred and code-preferred queries (Selection mAcc), along with per-class accuracy (in the last 3 and 4 columns).
  • Connection to Final Accuracy: Correct methodology-selection does not always guarantee a correct response due to prompt context mismatches and greedy decoding. We report Pass@1 accuracy within correct selections to evaluate how proper methodology-selection directly contributes to correct responses.

2. Response to the weakness comments on Efficiency Assumptions, Q2 and Q3.

The reviewer shows doubts on the efficiency assumptions that EM narrows the policy search space, and raises question on how EM improve efficiency in methodology-selection.

We realize that there are misconceptions that may be caused by the misunderstanding of "narrowing policy search space". So below we clarify this misunderstanding and that we do not make assumptions and do not claim EM improves the speed of methodology-selection.

---

(a) Clarifications on "narrowing policy search space".

In line 431, we stated that “the reference methodology-selection strategy helps narrow down the policy search space during training.” By this, we meant that EM prunes the hypothesis set of policy parameters for a better policy within the policy parameter space, according to $1-3$ . This does not imply any assumption about computational load or speed improvements but rather refers to improved training efficiency.

=> Q2: How does EM uniquely contribute to the narrowing of the policy search space?

The proposed EM uniquely "narrows policy search space", because it uniquely factorizes out the critical sub-policy, methodology-selection, and leverage reference decisions to focus policy improvement (i.e., policy search) in regions with higher expected rewards.

• During the E-step, we compute reference decisions that are more likely to lead to a correct response.

• During the M-step, these decisions assist policy search in high-reward regions, and less promising areas of the policy search space are pruned.

This mechanism parallels supervised learning approaches, where supervision accelerates convergence by providing direct guidance. Unlike standard Supervised Fine-Tuning (SFT), which relies on externally collected expert demonstrations, our EM approach generates reference methodology-selection decisions internally by harnessing the capabilities of the LLM itself. This internal guidance is a key differentiator that allows EM to uniquely narrow the policy search space.

---

(b) Addressing Assumptions and Claims about Efficiency and Computational Load in Methodology Selection

=> Q2: Could you provide empirical evidence or a theoretical basis for how EM improves efficiency in selecting between Chain of Thought (CoT) and code-based approaches?

=> Q3: What is the theoretical or experimental basis for assuming that EM will reduce computational load or enhance efficiency in methodology selection?

We regret the misunderstanding of making any assumptions and would like to emphasize that the argument of "narrowing policy search space" is mentioned as a conjecture (c.f., line 430) based on empirical results, rather than an assumption without proof:

• In Section 3.2 (line 430), we explicitly frame the "narrowing policy search space" argument as a conjecture informed by empirical results in Figure 4 and Table 1.

• Figure 4 demonstrates that under identical training conditions, EM achieves superior performance at the same iteration compared to standard RL.

• Based on logical rationales (line 430) and empirical analysis (line 419), we reach at the conjecture for these results.

References

$1$ Deisenroth M P, Neumann G, Peters J. A survey on policy search for robotics $J$ . Foundations and Trends® in Robotics, 2013, 2(1–2): 1-142.

$2$ Levine S, Koltun V. Guided policy search $C$ //International conference on machine learning. PMLR, 2013: 1-9.

$3$ https://icml.cc/2015/tutorials/PolicySearch.pdf

Dear Reviewer #nGHB,

We thank the reviewer for the valuable feedback and constructive suggestions. We have carefully addressed each of your questions and detailed our responses below.

Motivation for Using EM.

1. Response to the weakness comments on Lack of Comparative Analysis and Q1.

In the current paper, the reasons for choosing EM over alternatives are scattered, e.g., line 056, line 122, line 182, line 259. To address this, we have summarized the motivation clearly below and will ensure these reasons are explicitly clarified in the revised manuscript.

=> Q1: Why is EM chosen over alternative methods like standard reinforcement learning (RL) or supervised fine-tuning?

The motivation for our EM framework lies in the need for math LLMs to autonomously develop effective methodology-selection strategies that complement their inherent strengths (line 059). The central challenge is the lack of reliable supervision for methodology-selection (line 182).

• Limitations of SFT: Supervised Fine-Tuning (SFT) relies on external expert annotations. These data fail to dynamically adapt to the model’s specific strengths (line 057).

• Limitations of Standard RL. While standard RL can explore the policy space to find high-reward regions, it is limited to local explorations around the current policy, which is less efficient in policy training. This is supported by our experimental results (Figure 4).

• Advantages of the proposed EM approach.

  • Direct Supervision for Methodology-Selection: We reformulate the problem as maximum likelihood estimation with latent variables, naturally leading to a theoretically sound EM solution. This approach addresses the challenge of lacking reliable supervision by generating reference decisions during the E-step. The following is a figurative but not exactly accurate summarization:

    • In the E-step, reference decisions for methodology selection are generated by exploring the expected utility of all methodologies. The methodology with the highest value is selected as the reference (Equation 7).

    • In the M-step, these reference decisions directly supervise the learning of the methodology-selection strategy.

  • Efficient policy training. EM facilitates efficient training of the methodology-selection sub-policy:

    • Standard RL explores only local regions within the policy space, relying on small policy-gradient updates.

    • EM uses a reference methodology-selection strategy to guide policy improvements, substantially pruning the policy search space.

  • Theoretical Guarantees: EM enjoys convergence guarantees under its theoretical foundation.

Dear Reviewer nGHB,

Thank you for your valuable comments and suggestions.Your feedback has greatly helped us improve our work and prompted us to reflect deeply on the core aspects of our proposed method.

We have carefully addressed your questions and made revisions according to your feedback. The revised manuscript has been uploaded, and we have summarized the refinements in the top-most post.

We understand that this is a particularly busy time, so we sincerely appreciate any time you can spare to review our revisions and provide further feedback on whether our responses address your concerns. If there are any additional comments, we will do our utmost to address them promptly.

Best Regards,

The Authors

Dear Reviewer 3NjD,

Thank you for your valuable feedback, which greatly helped enhance our work. We have carefully addressed your questions and revised the manuscript in line with your insightful suggestions. As the deadline for the public discussion phase approaches, we are eager to hear your further feedback.

We understand this is a particularly busy time, and we sincerely appreciate any time you can dedicate to reviewing our revisions. Your feedback on whether our responses sufficiently address your concerns would be immensely helpful. Should you have additional comments, we will do our utmost to address them promptly.

Thank you once again for your time and effort.

Best regards,

The Authors

Dear Reviewer #nGHB,

We are deeply grateful for your detailed response and have carefully addressed each of your questions:

• We compare our approach with relevant alternatives and highlight its unique strengths.
• We clarify the misconception regarding the "narrowing policy search space."
• We conduct additional experiments to analyze the methodology selection and its connection to the final accuracy.

Importantly, we have made significant improvements to the paper’s presentation. The revised manuscript has been uploaded for your review. We are pleased to note that Reviewer #uEHE and Reviewer #Pg3w have acknowledged the improved presentation and indicated that their concerns have been addressed.

We look forward to your further feedback and would greatly appreciate any additional insights you might provide. We understand this is a busy time and are truly grateful for the time and effort you dedicate to reviewing our revisions. Your expertise is invaluable to us.

Thank you once again for your time and effort.

Best Regards,

The Authors.

Dear Reviewer #3NjD,

We sincerely thank you for your valuable feedback, which has significantly helped us improve our paper and highlighted promising research directions. Below, we address the specific points raised in your review.

1. Response to Weakness Comments on Presentation:

We sincerely appreciate the reviewer's valuable suggestions. We acknowledge that the current organization of Section 2, which follows the technical derivation of the proposed EM approach, may be difficult to comprehend. To enhance the manuscript's readability and coherence, we plan the following improvements:

• Section 2.1: We will focus on presenting the problem statement and emphasize the key challenge faced by AutoCode---namely, the lack of reliable supervision for methodology selection.

• Section 2.2:

  • We will begin with an overview of the high-level ideas addressing this challenge and provide an overarching pipeline closely aligned with Figure 2. This will offer readers a clearer understanding of the proposed method.
  • Next, we will describe the key components of the proposed method, emphasizing their motivations and high-level concepts. Detailed technical derivations will be moved to the appendix for readers seeking further depth.

We have uploaded our revised manuscript and detail our revisions in the top-most post.

2. Response to Q1 on exploring autonomous query generation.

The reviewer has highlighted an exciting research direction: the development of a self-evolving training mechanism where the model generates queries it performs poorly on and reinforces its abilities through targeted training. This idea is indeed compelling, as it could address a significant limitation in RL-based training for mathematical reasoning.

Existing RL training tends to converge on high-reward responses, as dictated by the reward-maximizing objective. When the query set remains fixed during training, the model's improvements plateau, as most training responses already achieve high rewards. Introducing new, challenging queries could mitigate this limitation and drive further advancements.

While this idea holds immense potential, we have not yet delved deeply into its exploration. Autonomous query generation could tackle critical challenges in the field, making it a topic deserving of focused research efforts—efforts that cannot be adequately addressed within the constraints of the rebuttal phase. Specifically:

• Lack of Query-Generation Capabilities: AutoCode Training is designed to produce math experts, not math teachers capable of designing high-quality queries.
• Challenges in Generating High-Quality Queries: The open question of how to generate high-quality and diverse queries remains unresolved. Existing techniques, such as Evol-Instruct, have limited effectiveness because our dataset collected from various public math datasets already incorporates such augmentations, e.g., MetaMath, WizardMath. To this end, developing a specialized teacher LLM capable of devising math queries catering to the student's weaknesses is likely necessary. However, this requires significant additional research, including defining training objectives for the teacher model and coordinating its interactions with the student model. Such efforts fall outside the scope of the rebuttal phase.

Despite these challenges, we agree that this is a promising research direction and is worthwhile to explore in future work. We thank the reviewer for bringing this important idea to our attention.

3. Response to Q2 on code release

We confirm that we have released our data and code, as noted in line 373, via an anonymous repository: https://anonymous.4open.science/r/AnnonySubmission-35F0

4. Response to Q3 on larger model experiments.

We have conducted experiments using the production-scale 72B Qwen2 model. On the MATH benchmark, we observed a 3.45% improvement in accuracy (from 71.2%) and a 47% reduction in code executions compared to the code dictation baseline. However, we did not include these results in the manuscript for the following reasons:

1. Reproducibility: Production-scale training was conducted using the Megatron framework for our own commercial use, which we cannot release. The code we have made available is based on a public codebase adapted with DeepSpeed. This code cannot support training over 13B.
2. Generalizability Across Models: Training 70B models is computationally prohibitive, and we lack the resources to conduct experiments on multiple base models. To demonstrate generalizability, we focused on experiments with specialized 7B math LLMs, which is consistent with the approach taken in most prior work.
3. Performance of Smaller Models: We acknowledge potential concerns regarding the scalability of our method. However, we emphasize that smaller specialized models such as Qwen2-Math and Deepseek-Math perform competitively with general-purpose large models on challenging benchmarks. For instance, DeepseekMath-Instruct-7B lags behind Claude-3 Opus by only 4% on MATH.
4. No 13B open-source LLMs. For latest models like Qwen2-Math, Deepseek-Math, and the LLaMA-3 family, only the 7B and 70B versions are open-sourced. Recent open-source LLMs released in 2024 typically do not include a 13B variant, as they offer similar performance to the 7B models, and training an additional size is not cost-effective. The latest 13B open-source LLM is CodeLLaMA, released in August 2023 (https://huggingface.co/codellama/CodeLlama-13b-hf), more than a year ago.

Additionally, we emphasize that AutoCode training benefits from combining the accuracy of both CoT inference and code-integrated inference. By intelligently selecting a proper methodology according the model's capability, AutoCode-trained models are more likely to combine the correctness of both CoT and code. The key improvement of AutoCode comes from closing the gap between the model's performance and the accuracy of oracle methodology-selection. This gain is size-agnostic as long as there is a significant gap to the oracle-selection upper bound.

Dear Reviewer 3NjD,

Thank you for your valuable comments and suggestions.Your feedback has greatly helped us improve our work and has inspired new research opportunities.

We have carefully addressed your questions and made revisions in line with your suggestions. The revised manuscript has been uploaded, and we have summarized the refinements in the top-most post.

We understand that this is a particularly busy time, so we sincerely appreciate any time you can spare to review our revisions and provide further feedback on whether our responses address your concerns. If there are any additional comments, we will do our utmost to address them promptly.

Best Regards,

The Authors

Dear Reviewer 3NjD,

Thank you for your valuable feedback, which not only helped improve our work but also inspired new research opportunities. We have carefully addressed your questions and revised the manuscript based on your constructive feedback. As the deadline for the public discussion phase approaches, we sincerely seek your further feedback.

We understand this is a particularly busy time, and we sincerely appreciate any time you can dedicate to reviewing our revisions. Your feedback on whether our responses sufficiently address your concerns would be immensely helpful. Should you have additional comments, we will do our utmost to address them promptly.

Thank you once again for your time and effort.

Best regards,

The Authors

Dear Reviewer #3NjD,

We sincerely appreciate your thoughtful and insightful feedback. We have carefully addressed each of your comments as follows:

• Presentation: We have followed your suggestions to restructure Section 2, resulting in improved coherence and readability. We are pleased that Reviewer #uEHE and Reviewer #Pg3w have also acknowledged the enhanced presentation in the revised manuscript.

• Autonomous Query Generation: Your feedback on the open question regarding autonomous query generation has been immensely valuable. It brought this brilliant idea to our attention, and we have elaborated on its potential in addressing a key challenge in this field.

• Code Release and Larger Models: We clarified that the code and data have been released through an anonymous repository. While we have trained on Qwen2-72B, we chose not to include the results in the current manuscript due to the prohibitive computational resources required to comprehensively evaluate multiple 70B models.

We are eager to hear any further feedback you may have. This rebuttal process has been highly rewarding, particularly because our discussions have sparked new ideas for potential research opportunities. However, we understand you may have a demanding schedule and are truly grateful if you can spare some time to review our revisions and rebuttal. Your feedback is invaluable to us.

Thank you once again for your time and effort.

Best regards,

The Authors

Dear Reviewer #Pg3w,

We thank you for your valuable feedback in improving our work. We have carefully addressed each of your questions and detailed our responses below.

1. Response to Comments on Performance Weaknesses

The reviewer notes that AutoCode4Math-Qwen2 does not outperform Qwen2-Math-Instruct. However, we emphasize that this comparison is not conducted on a fair basis due to fundamental differences in their data quality:

• Qwen2-Math-Instruct: This model benefits from private supervised fine-tuning (SFT) data, incorporating annotations from both human experts and GPT-4. Its improvement over Qwen2-Math-Base is driven by distillation from high-quality expert annotations.
• AutoCode4Math-Qwen2: In contrast, our model is trained on Qwen2-Math-Base using only publicly available mathematical queries paired with gold answers. The improvements observed in AutoCode4Math-Qwen2 stem entirely from the proposed algorithm, without leveraging any private data.

To ensure a fair evaluation of the algorithmic contributions, we compare AutoCode4Math-Qwen2 with AutoCode4Math-RL (see Table 1 and Figure 4). Both models use the same public query set and self-generated responses, thereby isolating the effect of expert annotations and focusing solely on the algorithmic enhancements.

We understand the reviewer's concern on whether the proposed method can improve on top of SFT-ed model. However, we emphasize that we have validated the effectiveness of our proposed method on instruction-tuned models by experimenting with Deepseek-Math-Instruct (refer to lines 400--408), and witnessed significant improvement. This demonstrates the joint effect of the proposed method and high-quality private data.

In the current paper, we do not train on top of Qwen2-Math-Instruct, because it does not natively support math coding abilities (see line 363).

• Unlike Deepseek-Math-Instruct, which natively support code integration for math reasoning, Qwen2-Math-Instruct does not incorporate Program-of-Thought or Tool-Integrated-Reasoning data in its training. To address this, additional fine-tuning would be necessary to enable math-coding capabilities. However, this process is constrained by the limitations of available public data, which, as demonstrated, degrades performance. For example, while fine-tuning with public data imparts some coding capabilities, it significantly reduces the performance of chain-of-thought (CoT) reasoning by more than 20% on MATH (see the comparison of bold and italic numbers in the table below). | Models | Prompt | GSM8K | MATH | |--------------------------------|--------|----------|----------| | Qwen2-Math-Instruct | CoT | 89.9 | 75.1 | | Qwen2-Math-Instruct-FurtherSFT | CoT | 82.79 | 52.98 | | Qwen2-Math-Instruct-FurtherSFT | Code | 85.02 | 58.98 |

• In addition, the instruction-tuning math query set used for Qwen2-Math-Instruct is proprietary. Publicly available data falls significantly short in quality, limiting its effectiveness in fine-tuning.

2. Response to weakness comments on presentation

We regret the misunderstanding regarding "allowing the model to perform some degree of inference-time search between two actions" in the paper summary. To clarify, *our proposed model decodes as standard LLMs do, without any search during inference. *Instead, our approach can be seen as distilling search-based, high-reward trajectories into math LLMs during training. This enables the model to employ a more intelligent methodology-selection strategy during inference for better performance, but without the cost of increasing inference-time compute.

We sincerely appreciate the reviewer's insightful suggestions and acknowledge that the current organization of Section 2, which follows a bottom-up technical derivation of the proposed EM approach, might hurt readability. To improve the manuscript's clarity and structure, we plan the following enhancements:

• Section 2.1: We will focus on clearly presenting the problem statement and emphasizing the key challenge faced by AutoCode -- specifically, the lack of reliable supervision for methodology selection.

• Section 2.2: We will adopt a top-down structure to present our approach:

  • We will start with a high-level overview of our proposed solution to this challenge and introduce a comprehensive pipeline aligned closely with Figure 2.

  • Following this, we will delve into the key components of our method, emphasizing their motivations and overarching concepts in the beginning. In particular, we will detail:

    • How the E-step generates the training data, and how we perform data synthesis.
    • How the M-step trains the model using this data in an efficient offline manner.

We will update our revised manuscript and detail our revisions in a separate comment.

3. Response to Q1 on extending AutoCode Training.

In discussing the performance growth of AutoCode Training, we would like to highlight an important aspect: the proposed approach leverages the "free lunch" of combining the accuracy of both CoT (Chain-of-Thought) reasoning and code-driven inference. Initially, certain queries are distinctively CoT-preferred or code-preferred due to the model's inherent capabilities: they cannot be solved by the alternative method. However, after AutoCode Training, the model learns to select the appropriate methodology for different queries. This enables the model to cover the correctness of both CoT-preferred queries and code-preferred queries. By developing a smarter methodology-selection strategy, AutoCode Training effectively bridges the gap to the performance of oracle selection.

Thus, the convergence of AutoCode Training largely depends on the gap between the current model's performance and the upper bound of oracle selection. To illustrate this, we present three key metrics in the tables below:

• Pass@1: The accuracy of greedy decoding.
• Oracle Pass@1: The combined accuracy of CoT and code-driven inference. A query is considered correct if either CoT or code successfully solves it.
• Selection mAcc: The average accuracy of the learned methodology-selection strategy compared to the oracle selection strategy.

Table 1. Performance Growth of AutoCode Training on MATH

Table 2. Performance Growth of AutoCode Training on GSM8K

As shown in the above tables,

• AutoCode Training yields minor improvements at iteration 3: +1.1% on MATH and +0.2% on GSM8K.
• AutoCode Training yields continuous performance growth because the selection accuracy continues to grow, implying that the model learns continuously better methodology-selection.
• This gain starts to converge because the upper bound of oracle selection converges. For example, Oracle Pass@1 67.44->67.98->68.58 on MATH, and the model performance has approached 66.38% leaving very small room for potential improvement. If we have access to queries with better quality, the potential room for improvement can be further opened up.

4. Response to Q2 on Figure 4 Initial Accuracy.

We have verified that the initial accuracy in Figure 4 is correct, reflecting the CoT performance of the supervised fine-tuned (SFT-ed) model. The confusion may have arisen because Figure 3 reports only the code performance for Code4Math-Qwen2, rather than the CoT result, leading to a perceived misalignment.

We clarify the numbers and explain how Figure 4 and Figure 3 align as follows:

• Deepseek-Math-Instruct achieves 81.27% on GSM8K and 45.32% on MATH with CoT, consistent with the result reported in Figure 3 (line 400).
• Code4Math-Qwen2, fine-tuned from Qwen2Math-Base using public math coding data, achieves 81.58% on GSM8K and 52.68% on MATH with CoT, and does not degrade CoT performance compared to its base model's performance of 80.74% on GSM8K and 51.82% on MATH. The reason why we do not train on top of Qwen2-Math-Instruct has been discussed in Response 1.

We apologize for any confusion regarding Figure 4. For clarity, we will include CoT results in the revised manuscript.

5. Response to Q3 on code release

We confirm that our released code is accessible through the anonymous repository, as mentioned in line 373: https://anonymous.4open.science/r/AnnonySubmission-35F0

4. Response to Q3: Clarification on "the standard RL" baseline

In our work, "standard RL" refers to reinforcement learning with correctness feedback in the context of math reasoning. "RLHF" typically refer to reinforcement learning with human feedback to reinforce the general LLM generation. Therefore, "the standard RL" baseline essentially refers to using the same RL technique in "RLHF" but with correctness feedback in the context of math reasoning.

5. Response to Q4: Evidence for the conjecture why EM outperforms RL

We refer to narrowing policy search space as reducing the set of promising hypotheses of better policy within the policy space. For instance, learning from expert demonstrations narrows the policy search space compared to random exploration because the expert demonstrations provide direct guidance on improving the current policy, thus constraining the set of promising hypotheses.

As mentioned above, the M-step in the proposed EM approach involves supervised learning for methodology-selection, which resembles learning from expert demonstrations. However, a key distinction is that the demonstrations—the reference decisions—are derived from the model's internal exploration of its unique strengths rather than relying on externally dictated decisions. Logically, this supervised procedure takes similar effect to learning from demonstrations, so we conjecture that the proposed EM narrows the policy search space.

We present empirical evidence from two perspectives:*

• Policy Improvement: EM achieves a better policy under identical training conditions and at the same iteration compared to RL. This is evident from the convergence curve shown in Figure 4.
• Methodology-Selection Alignment: EM demonstrates better alignment between its improved methodology-selection strategy and the optimal strategy.

The table below summarizes these findings. It compares EM and RL training with the base SFT-ed LLM policy they are initialized from. We also include the conceptual oracle strategy for reference, which selects the optimal methodology for each query assuming fixed solution-generation (regardless of the improvement on solution-generation). We evaluate two metrics on the MATH benchmark: (a) Pass@1 Accuracy: The accuracy of greedy decoding the whole response. (b) Alignment Accuracy: The alignment of the methodology-selection strategy with the oracle selection strategy.

As shown in the above table, the methodology-selection strategy trained using EM aligns significantly better than using RL, across different base models: +7% for DeepseekMath, and +4.6% for Qwen2Math. This drives the improvement on pass@1 accuracy of the whole response: +5% for DeepseekMath, and +2.6% for Qwen2Math. These results show that EM better aligns the methodology-selection with the oracle strategy, implying that EM finds a better policy in the policy search space using the reference decisions. Notably, EM achieves closer gap with the oracle pass@1 accuracy: 5.6% gap for DeepseekMath, and 3.5% gap for Qwen2Math. Therefore, we conclude that EM outperforms RL because it narrows the policy search space.

References:

[1] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in neural information processing systems 35 (2022): 27730-27744.

[2] Gulcehre, Caglar, et al. "Reinforced self-training (rest) for language modeling." arXiv preprint arXiv:2308.08998 (2023).

[3] Anthony, Thomas, Zheng Tian, and David Barber. "Thinking fast and slow with deep learning and tree search." Advances in neural information processing systems 30 (2017).

2. Response to Q1: Comparison with Relevant Alternatives and Its Advantages

We first summarize the proposed method and alternatives that facilitate self-exploration and can replace the simple SFT.

• The proposed EM approach alternates between two steps. In the E-step, we generate reference decisions for methodology-selection. In the M-step, we leverage the reference decision to supervised the methodology-selection strategy, and reinforce the solution-generation policy with exploration.

• RLHF/RL. This type of algorithms reinforce the LLM language generation policy with exploration, based on the correctness feedback in the context of math reasoning. The exploration is typically done on-policy in standard RLHF methods such as instructGPT[1], which uses PPO, and ReST[2], which uses off-policy RL.

• Expert Iteration[3] (ExIT) alternates between two steps: expert improvement and policy distillation. During expert improvement, expert demonstrations are sampled by improving the current policy through some search procedures, such as monte-carlo tree search used in ExIT. During policy distillation, the expert demonstrations are used to supervised learning the current policy.

The following table highlights the key differences between the EM framework, RLHF, and ExIT:

3. Response to Q2: Benefits of each component of the proposed approach

We summarize the benefits of each component of the proposed EM approach below.

• The E-step benefits from a proven way to optimally generate reference decisions for methodology-selection (Equation 4), which provides effective guidance for improving the current methodology-selection strategy. This directly addresses the challenge of lacking reliable supervision for methodology-selection: (a) the reference decisions are used to supervise the learning of methodology-selection in the M-step; (b) the reference decisions are provably reliable, because they maximizes the expected utility of the current solution-generation policy, as shown in Equation 4. In practice, these reference decisions are generated by exploring the model's inherent capability. In simple (but not accurate) words, we try all decisions, observe the outcomes, and set the decision that yields highest value as the reference decision.
• The M-step benefits from the supervised training of methodology-selection with the reference decisions and reinforcement learning with exploration to improve the solution-generation policy.
• Efficient Joint Training benefits training methodology-selection and solution-generation in a joint and efficient manner. Indeed, methodology-selection and solution-generation share the parameters in a single LLM policy, so it is not reasonable to separate the training of methodology-selection and solution-generation. To align the training condition of solution-generation with the supervised training of methodology-selection, we thus adopt off-policy RL, which allows us to RL with an offline curated exploration data. The algorithm is efficient as the LLM policy is trained in a fully offline manner, compared to on-policy RL methods.

Dear reviewer #uEHE,

We sincerely appreciate your valuable feedback and suggestions, which have significantly helped us improve the clarity and presentation of our work. We have carefully addressed each of your questions and detailed our responses below.

1. Response to weaknesses: Why is the EM Approach Necessary?

The motivation for our EM framework lies in the need for math LLMs to autonomously develop effective methodology-selection strategies that complement their inherent strengths (line 059). The central challenge is the lack of reliable supervision for methodology-selection (line 182). Supervision based on externally dictated decisions fall short because it fails to dynamically adapt to the model's unique capabilities.

We address this challenge by reformulating the problem as a maximum likelihood estimation with latent variables, naturally giving rise to an theoretically-sounded EM solution. To make it easier to understand why the proposed EM approach directly addresses the challenge of lacking reliable supervision, here we give a figurative but not exactly accurate summarization of the proposed EM approach. The EM alternates between two steps:

• E-step: Generate reference decisions for methodology selection by exploring all methodologies and choosing the one with the highest value (Equation 7).
• M-step: Use these reference decisions for supervised fine-tuning (SFT) of methodology selection while reinforcing the solution-generation policy through offline exploration.

Unlike SFT used in previous methods that resort to externally dictated methodology decisions, the reference decisions are obtained internally by monte carlo rollouts of the current solution-generation policy, which reflects the current model's unique strengths.

Additionally, in response to the reviewer’s comment on the complexity of the proposed method, we emphasize that our approach presents a streamlined and efficient implementation of the EM framework (as summarized in the algorithm diagram in Section B). This makes our method highly practical and advantageous in real applications. We regret any misunderstanding that may have arisen and will elaborate on how to improve the presentation of our method in a separate rebuttal comment.

Thank you for your insightful comments and suggestions. Your feedback has been instrumental in improving our work and has prompted us to reflect deeply on the core aspects of our proposed method.

We have carefully addressed your questions and incorporated revisions based on your feedback. The revised manuscript has been uploaded, and we have summarized the refinements in the top-most post.

We understand that this is a particularly busy time, so we sincerely appreciate any time you can spare to review our revisions and provide further feedback on whether our responses address your concerns. If there are any additional comments, we will do our utmost to address them promptly.

Best Regards,

The Authors

Dear Reviewer uEHE,

Thank you for your valuable feedback, which greatly helped us improve our work. We have carefully addressed your questions and revised the manuscript based on your constructive suggestions. As the deadline for the public discussion phase approaches, we are eager to hear your further feedback.

We understand this is a particularly busy time, and we sincerely appreciate any time you can dedicate to reviewing our revisions. Your feedback on whether our responses sufficiently address your concerns would be immensely helpful. Should you have additional comments, we will do our utmost to address them promptly.

Thank you once again for your time and effort.

Best regards,

The Authors