Agentic RL Scaling Law: Spontaneous Code Execution for Mathematical Problem Solving

Thank you very much for your time and careful review. You have raised an interesting point, and we hope to continue discussing this with you in the next phase of the discussion. We will address your concerns point by point as follows:

1. Absolute performance
   We trained for more steps and released ZTRL‑0727, which shows further gains on all datasets.
   ZTRL‑0727 widens the gap on AIME25 and MATH500.

Table 1: The results of the ZTRL model on the Greedy.

In the newly released ZTRL‑0727‑7B, we also compare with much larger models. Results show we are consistently stronger; data and checkpoints are linked in GitHub and will be added to the paper.
On the avg@32 metric, ZTRL‑0727‑7B surpasses qwen3‑238b‑a22b‑non‑thinking by 5.3 pp on AIME25 and 12.08 pp on HMMT25, and approaches or exceeds the closed‑source SOTA claude optus4‑non‑thinking.

Table 2: The results of the ZTRL model on the avg@32.

2. “Cheat” or take shortcuts
   Our model is only 7 B, a small backbone.

3. Tool use is itself part of strategic learning: human mathematicians also rely on calculators or code. RL aims to maximise task success and efficiency, not aesthetic elegance.

4. Not every problem is brute‑forced. We classify three solution modes (analytical, hybrid, brute enumeration). The latter two together are below 35 %. Many tasks still need symbolic reasoning to write correct or efficient code. Gemini, with larger pre‑training and math alignment, can solve elegantly without code; our setting is a smaller base, zero‑supervision RL, and limited resources. The two routes are not contradictory: the model can choose analysis when possible and code when computation is needed.

5. Differences in methodology between our approach and TORL
   Paradigm: both works belong to the “zero‑RL + code tool” family and were done independently in the same period.
   • We study the gap between a general base model and a math‑specialised base, and propose algorithmic changes needed during agent RL. TORL forces code execution; our training is spontaneous and only prompts “You can use code to solve your problem.” RL is used to let the model decide by itself. With longer training, the model shifts to code even without compulsion.
   • TORL uses a math‑base model (effectively an SFT‑tuned backbone). Math‑base already underwent heavy TIR continued pre‑training, so its code‑ratio and pass‑ratio start around 0.2 and 0.8. The general base begins at 0.1 and 0.05, meaning far weaker TIR ability (though still better than llama3‑base).
   • Performance: AIME24 contains more computational, small‑scale, sparse‑reward tasks, so tool calling gives a larger jump; MATH500 and AMC23 have many direct‑derivation or template problems, where TORL’s prompt is already well‑tailored. With longer training our ZTRL‑0727 further improves.

6. “Laziness” as a potential concern
   Math tasks often need exact arithmetic, enumeration, and combinatorial search. Learning to call external compute is crucial for real agents.
   Mitigation:

7. We impose a cost: each call adds tokens and potential failure.

8. As RL progresses, calls become more frequent but more efficient: over 90 % of correct answers with code need only one call; we do not observe blind multi‑call brute force.

9. Future work will add an explicit reward for reasoning quality or minimising calls to encourage “elegant” solutions.

Once again, thank you for your time. We hope to continue discussing this with you in the next phase of the discussion. If your concerns are alleviated, we would greatly appreciate it if you could consider raising your score. We sincerely thank you for your consideration.

Thank you for the considerable effort you put into your review.

1. We trained for more steps and released ZTRL‑0727, which shows further gains on all datasets.

2. We share your curiosity about whether code execution encourages “lazy” solutions. Tool use is itself part of strategic learning: human mathematicians also rely on calculators or code. RL aims to maximise task success and efficiency, not aesthetic elegance. We classify three solution modes (analytical, hybrid, brute enumeration). The latter two together are below 35 %. Many tasks still need symbolic reasoning to write correct or efficient code. Gemini, with larger pre‑training and math alignment, can solve elegantly without code; our setting is a smaller base, zero‑supervision RL, and limited resources. The two routes are not contradictory: the model can choose analysis when possible and code when computation is needed.

3. We claim the differences in methodology between our approach and TORL.

We would be glad to delve deeper into these analyses or run additional case studies that might assist you in reassessing the paper’s contribution. We would value any further questions you have, and if the new analyses alleviate your reservations, we kindly ask you to consider raising your score.

Thank you again for your detailed review.

Thank you for the time and care you have devoted to evaluating our submission. With fewer than 48 hours remaining in the extended author–reviewer discussion period, we would like to ensure that all of your questions and concerns have been fully addressed. If anything remains unclear, we would be grateful for the opportunity to clarify it promptly.

Should you have no additional questions, we would kindly ask you to consider revisiting your score in light of the revisions and new analyses we have provided.

Thank you again for your thoughtful review and for helping us strengthen the paper.

Thank you very much for your time and careful review. Your review is very important to us. It helped us correct many writing problems. Thank you very much for your effort. We will address your concerns point by point as follows:

1. log-scaled x-axis
   Your suggestion is correct. In fact, we have released the checkpoint of ZTRL-0727. This model substantially surpasses previous ones, and because it was trained for more steps, it exhibits a more pronounced scaling law phenomenon. We will add the log-log fitted curve and the coefficient of determination (R²) for ZTRL-0727 in the appendix.

2. Spearman Correlation
   We agree with your suggestion. We will include statistical tests in the appendix. First, we use Spearman’s rank correlation coefficient to quantify the monotonic relationship between tool usage rate and success rate, analyzing the monotonic relationship between rule_rewards (accuracy) and code_ratio (code proportion) during training. The mean of rule_rewards is 0.3759, the standard deviation is 0.0677; the mean of code_ratio is 0.6629, the standard deviation is 0.3678. The result is ρ = 0.686, p = 7.06e-71, n = 500, indicating a strong correlation and high statistical significance (p < 0.001). We will further add other statistical testing methods in the appendix.

3. performance
   3.1 Our setting differs from rStar-Math. rStar-Math uses a much larger supervised/distillation/multi-stage RL pipeline and relies on Monte Carlo Tree Search. Our goal is to study the scalability of “Zero-RL + spontaneous tool use”. rStar-Math uses a Process Preference Model for step-level verification and a math-specialized base model as the backbone, while we only use the general-base model as both backbone and ORM, and rely on RL to help the model learn to use tools to solve complex math problems. The starting points differ and may not be suitable for direct comparison.
   3.2 ZTRL-0727 further widens the gap on AIME25 and MATH500.

Table 1: The results of the ZTRL model on the Greedy.

4. which tokens correspond to actions, and which do not?
   Action tokens are the special markers and parameter segments that trigger the external compilation/execution environment, including:

• All tokens between the start and end markers of <code_call> inserted in our template;
• Special control tokens parsed by the system as tool/action calls.
  Non-action tokens are normal natural language reasoning/explanatory content.

5. set \lambda to 0.
   This is a typographical error; it should be set \lambda to 1. Our expression was unclear: the actual reason is that rewards are extremely sparse and concentrated at the end of the sequence, and in our implementation we already perform “per-step equal distribution + group-wise normalization” of the advantage, so the temporal smoothing effect of GAE is almost negligible. Therefore, the results of \lambda → 0.95 and \lambda = 1.0 are nearly identical. Experiments verify this idea. Under this setting, the adv of environment feedback tokens equals 0, effectively adding an environment mask.

6. call the compiler again?
   Strictly speaking, it cannot be guaranteed 100% (an LLM might bypass the prompt), but since the correct answer is not provided, it will not be rewarded; such samples are penalized. This does not affect the training outcome.

7. mention several metrics
   This was our oversight; we will remove it from the main text.

8. Only Qwen base, beyond Qwen math
   We will revise this.

9. substantiate it with a plot
   We agree. We will add a line/scatter plot showing the relationship between N_{\text{max}} and accuracy, and report Spearman ρ.

10. Table 4.3
    This is a typesetting error; it should be Table 3.

11. the font size
    We will increase the font size and provide a high-resolution version in the appendix. We have also added a composite figure: tool usage rate and success rate are overlaid in the same chart (dual y-axis or normalized on the same axis) to illustrate the correlation more intuitively.

Once again, thank you for your time. If your concerns are alleviated, we would greatly appreciate it if you could consider raising your score. We sincerely thank you for your consideration.

Thank you for highlighting the need for stricter statistical evidence and clearer visuals. We have:

1. Reported Spearman correlation ( ρ = 0.686, p = 7.06e-71) between tool usage and accuracy over 500 checkpoints. We agree with your suggestion. We will include statistical tests in the appendix.

2. Additional experiments demonstrate the performance of our model. ZTRL-0727 further widens the gap on AIME25 and MATH500. This model substantially surpasses previous ones, and because it was trained for more steps, it exhibits a more pronounced scaling law phenomenon. We will add the log-log fitted curve and the coefficient of determination (R²) for ZTRL-0727 in the appendix.

We would value any further questions you have, and if the new analyses alleviate your reservations, we kindly ask you to consider raising your score.

Thank you again for your detailed review.

Thank you for your follow-up on point 6, and for questioning why multiple compiler calls may still fail to produce correct answers—a valid point we’re glad to clarify.

1. “the correct answer is not provided” is because, once the model reaches the allotted maximum number of Python-tool calls, it automatically appends that sentence to the last Python execution output. From that point on, every subsequent response—whether it attempts another Python call or not—is cut off because the overall turn limit has been hit. When the model is still trying to call Python under these conditions, no \boxed{} expression can be extracted, so the response yields no reward.

2. We would be happy to share our thoughts on tool usage and accuracy. Spearman correlation analysis across 500 checkpoints (ρ = 0.686, p = 7.06e-71) shows a positive association between tool usage and accuracy. However, models’ inherent reasoning limitations with hard problems can lead them to adopt a brute-force approach of excessive compiler invocations, which rarely improves real-world results.

We also analyzed average compiler calls for correct vs. incorrect solutions:

Notably, incorrect solutions involve significantly more compiler calls, indirectly indicating that models’ reasoning deficits persist even with increased invocations, leaving hard problems unresolved.

We would value any further questions you have, and if the new analyses alleviate your reservations, we kindly ask you to consider raising your score.

Thank you again for your detailed review.

Thank you very much for your time and careful review. Your review comments are very meaningful to us. We will address your concerns point by point as follows:

1. for others settings the gain is a lot less
   I understand your question as follows: we achieved a large improvement (+30–35%) on AIME25, but the gains on AIME24 and MATH500 are less obvious. First, based on Table 1, since our ZTRL model is trained from the Qwen2.5 Base model, the fair comparison baseline should be Qwen2.5 Instruct (whose training data volume is orders of magnitude larger than ours). Compared with Qwen2.5 Instruct, we obtain +26.7% on AIME24 and +3.8% on MATH500. In our newly released ZTRL-0727-7B, in order to broaden the comparison to more models, we compared against models much larger than ours. Experiments show that we consistently outperform models trained by other methods. The data and checkpoints can be found in the GitHub link in the main text for verification, and we will add these results to the paper. ZTRL-0727-7B exceeds qwen3-238b-a22b-non-thinking by 5.3% and 12.08% on AIME25 and HMMT25 avg@32, respectively, and is close to or surpasses the closed-source sota model claude optus4-non-thinking.

Table 1: The results of the ZTRL model on the avg@32.

Table 2: The results of the ZTRL model on the Greedy.

2. Section 4.3 and 4.4 specifically to be a bit difficult to parse
   In Sections 4.3 and 4.4, the core insights we intended to present are these three points:
   (1) Code usage dips at first (the model fumbles), then rises as it learns the tool’s value. “Code-in-correct” and reward curves climb together: effective code use drives success. Response length grows (more code/output) but does not perfectly track reward.
   (2) Bigger + faster matters, but data and entropy still rule. Larger models (32B > 7B > 1.5B) perform better. Reinforce++ reaches the same peak as PPO about 300 steps sooner. Proof-heavy data (DeepMath) boosts best single-shot scores (Max), while diverse contest data (Orz-57k) boosts consensus metrics (Maj/Avg). Higher decoding entropy (top-p) trades higher Max for lower Maj, so dataset choice and entropy must be tuned together.
   (3) Interaction cap helps—up to a point. Letting the model call tools more times (N_{\text{max}} from 0→4→20) raises accuracy, but gains flatten after about 4 calls. Bigger models still end up using only 1–2 calls per problem because early multi-call attempts are low quality and get low reward.
   We will consider your suggestions and add more explanations, analyze in detail why these trends appear, provide more discussion, and revise the figures.

3. bolded in table
   Yes, that is our mistake. We will correct it in the main text. Since the newly released ZTRL-0727 achieved a significantly improved new sota, detailed results can be found at the GitHub link in our paper. We will revise the tables as a whole.

4. language agnostic code execution or other environment
   In fact, during implementation, we decoupled the environment from the agent training framework. By swapping the env service to a specific environment, cross-environment agent RL training can be achieved (e.g., for appworld, webshop benchmarks). The main infrastructure change needed is to wrap the corresponding benchmark as a server that conforms to our current interface standard, after which training can proceed. I understand this as a decoupled approach. We will update the code repository to better support related training.

5. the number of task instances per dataset
   The average score is the mean of percentages across benchmarks. This is the standard practice in math evaluation, as seen in papers such as Qwen2.5 Math (Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement). To align with other papers, we adopt the mean of percentages across benchmarks [1][2].

6. Minor edits
   We will modify the first sentence at Line 190, add the full phrase for ZTRL, and clarify that code ratio represents the proportion of responses containing code among all responses. We will add this explanation in the figure captions. We will reconsider the title “Agent RL Scaling Law.” In fact, we hope to define standard interfaces that adapt to most environments in the future, and we have already implemented preliminary functionality. Thank you very much for your careful review.

Once again, thank you for your time. If your concerns are alleviated, we would greatly appreciate it if you could consider raising your score. We sincerely thank you for your consideration.

References:
[1] TORL: Scaling Tool-Integrated RL
[2] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild

Thank you for the time you have already devoted to evaluating our submission.

1. We appreciate your positive assessment of the spontaneous-tool-use angle and have acted on all clarity suggestions. Sections 4.3–4.4 will be rewritten to highlight three take-aways (learning curve inflection, model-size effect, and interaction-budget saturation) and Fig. 4’s font size/legend will be enlarged. Per your question on infrastructure portability, we will add Appendix to demonstrates a plug-and-play wrapper that lets the same agent train on different environment and task. We hope the improved presentation makes the contributions clearer.

2. In our newly released ZTRL-0727-7B, in order to broaden the comparison to more models, we compared against models much larger than ours. Experiments show that we consistently outperform models trained by other methods. The data and checkpoints can be found in the GitHub link in the main text for verification, and we will add these results to the paper. ZTRL-0727-7B exceeds qwen3-238b-a22b-non-thinking by 5.3% and 12.08% on AIME25 and HMMT25 avg@32, respectively, and is close to or surpasses the closed-source sota model claude optus4-non-thinking.

We hope these additions address your concerns; if so, we would be grateful if you could reconsider the current score.

Thank you again for your detailed review.

Thank you very much for your time and careful review. Your opinions are very valuable to us. We address your concerns point by point below.

First, ZeroTIR means Training the base model to integrate tool usage via reinforcement learning (RL), relying exclusively on output-based rewards, yields experimentally non-trivial results. The ZeroTIR setting is not an obvious conclusion like DeepSeek R1 to reach for the following reasons:

a. Unlike the methodology in ToRL or Qwen2.5 Math, which explicitly enforces code-based problem solving, our approach leaves open a critical question: can the model gradually acquire tool-utilization capability for mathematical problem solving when guided only by outcome rewards on base model without cold-start? A significant risk remains that the model may fall back on text-based chain-of-thought as its primary strategy.

b. In contrast to ToRL, our experiments start from the \textt{qwen25-general-base} model, whose initial code-usage ratio and pass rate at step 0 are merely 10 percent—far lower than the 30 percent reported in ToRL. With such a baseline, applying RL with only an Output Reward Model (ORM) offers no guarantee that the model will learn to exploit tools or deliver meaningful gains. We argue that this discrepancy deserves deeper investigation.

Table: ACC and CODE-usage

As can be seen from the above table, the code ratio initially dropped from 0.14 to 0.002, then gradually increased, and finally reached a very high proportion of code usage. This phenomenon is probably hard to anticipate before any exploration is conducted since we only use ORM to guide the training process of RL.

1. Comparison with RL-after-SFT and state-of-the-art SFT-based TIR

1.1 We begin training from a base model rather than an SFT-tuned model. This choice is fairer: SFT quality depends on format, scale, and data construction, so validating RL algorithms directly on bases is common practice (SimpleRL [1], DAPO [2], ToRL [3]).

1.2 The strongest SFT-tuned model currently available is Qwen2.5 Math TIR, which already reaches state-of-the-art math performance via massive SFT data. Other SFT-tuned models, including those based on LLaMA or Mistral, lag far behind Qwen2.5 Math TIR [4].

1.3 To widen comparisons, we released ZTRL-0727-7B. Despite its smaller size, it consistently surpasses larger competitors; checkpoints and data are linked in the paper. On avg@32, ZTRL-0727-7B exceeds qwen3-238b-a22b-non-thinking by 5.3 pp on AIME25 and 12.08 pp on HMMT25, and is close to—or better than—the closed-source SOTA claude optus4-non-thinking.

Table 1: The results of the ZTRL model on the avg@32.

Table 2: The results of the ZTRL model on the Greedy.

2. Other open-source foundation models

Tool-calling competence is a prerequisite. Other open-source bases (e.g.\ LLaMA, Mistral) show weak tool performance and lack TIR training; recent studies [5,6] confirm that community models tend to over-call tools and cannot admit failure. Qwen2.5 models excel precisely because TIR data were injected during pre-training. For example, LLaMA 3.1 8B scores an F1 of 16.6 with 67 percent tool hallucination, whereas Qwen2.5 7B reaches F1 32.0 with 21 percent hallucination. Recent mid-training work [7,8] also reports that LLaMA bases are ill-suited to zero-RL, performing far below the Qwen family. For these reasons we chose Qwen2.5 for RL training.

3. Longer training steps

Our open-source framework trains stably beyond 1,000 steps; extended results are shown below.

Table 3: The val results of the ZTRL-0727-7B beyond 1,000 steps. avg@32

Once again, thank you for your time. If these clarifications resolve your concerns, we would be grateful if you could consider raising your score.

References

[1] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
[2] DAPO: An Open-Source LLM Reinforcement Learning System at Scale
[3] ToRL: Scaling Tool-Integrated RL
[4] Qwen2.5-Math Technical Report: Toward a Mathematical Expert Model via Self-Improvement
[5] When2Call: When (not) to Call Tools
[6] T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step
[7] Behavior Injection: Preparing Language Models for Reinforcement Learning
[8] OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling

Thank you for the time you have already devoted to evaluating our submission.

We have carefully addressed each of your comments in our rebuttal, providing additional experiments (Tables 1, 2 and 3) and clarifying the methodological choices that underpin our main contributions.

1. On avg@32, ZTRL-0727-7B exceeds qwen3-238b-a22b-non-thinking by 5.3 pp on AIME25 and 12.08 pp on HMMT25, and is close to—or better than—the closed-source SOTA claude optus4-non-thinking. This demonstrates the robustness of our approach, even against models with significantly larger parameters, different bases, and different training methods.

2. Fig. 4 is now extended to 1000 steps and shows no collapse.

If any aspect of our response remains unclear—or if further evidence would help you assess the paper more favorably—we would be delighted to supply it during the discussion window. Your insights are invaluable to improving this work, and we greatly appreciate any additional questions or suggestions you might share. We hope these additions address your concerns about novelty and breadth; if so, we would be grateful if you could reconsider the current score.

Thank you again for your detailed review.

Thank you for your continued engagement. The author–reviewer discussion period now has fewer than 48 hours remaining, so we want to be sure we have responded to every concern you raised.

During the rebuttal time, we extended the ZeroTIR training run to 1 500 steps again and recorded avg@32 accuracy on all three benchmarks. The complete Training-Steps and its Validation set results is shown below.

Key observations:

1. Accuracy continues to improve or stabilise up to 1 500 steps; no sign of collapse is observed.
2. AIME24 and HMMT25 both display late-stage gains, confirming that the agent benefits from additional compute without overfitting to early exploration patterns.

We hope these new results clarify the long-horizon behaviour you asked about. We look forward to further discussions and exchanges with you. If any aspect of our response remains unclear—or if further evidence would help you assess the paper more favorably—we would be delighted to supply it during the discussion window. Your insights are invaluable to improving this work, and we greatly appreciate any additional questions or suggestions you might share. We hope these additions address your concerns about novelty and breadth; if so, we would be grateful if you could reconsider the current score.

Thank you again for your time and constructive feedback.

Thank you for the time and care you have devoted to evaluating our submission. With fewer than 48 hours remaining in the extended author–reviewer discussion period, we would like to ensure that all of your questions and concerns have been fully addressed. If anything remains unclear, we would be grateful for the opportunity to clarify it promptly.

Should you have no additional questions, we would kindly ask you to consider revisiting your score in light of the revisions and new analyses we have provided.

Thank you again for your thoughtful review and for helping us strengthen the paper.