Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning

We appreciate your comments and suggestions and we address each question and provide further clarifications in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.

Q1: How do the multi-staged training experiments integrate within Sparkle framework? They appear to involve distinct objectives being presented within a single research paper.

Thank you for the thoughtful question. The dual focus is intentional and essential to our study. Our work aims to understand mathematical reasoning under RL. When applying RL to a set of problems, the key knobs are (i) the components necessary to solve a problem (planning, knowledge, etc.)—captured through the SPARKLE analysis framework—and (ii) the raw difficulty of the problems, which motivates our use of multi-stage RL training. A full characterization of RL for reasoning is not possible without addressing both of these elements.

To unify these aspects, we adopt a curriculum-style two-stage RL training setup, inspired by prior works (Sec. 2), and analyze its effects of RL using the SPARKLE framework. Specifically, Stage 1 establishes a strong RL-tuned model using diverse math problems, while Stage 2 explores three strategies for fine-tuning on a subset of the most challenging problems. We explore three data strategies (L184-188) for the second-stage training, from naive RL training with no augmentation, to including partial-solution augmentation. Our results show that without augmentation, outcome-based RL struggles to learn from difficult problems due to sparse or absent reward signals (Table 1). In contrast, partial-solution augmentation emerges as a simple yet effective method for enabling RL to extract training signal from these challenging cases.

The SPARKLE framework is applied consistently across models trained at each stage, allowing us to systematically analyze how different training strategies influence specific aspects of reasoning. Rather than representing distinct objectives, the RL experiments and SPARKLE framework are tightly coupled to provide a unified and comprehensive analysis of RL’s impact on LLM reasoning.

Q2.1: Given that, when complementary information is used as input, a decline in performance is observed in SFT models. Does this not suggest that the instructions are of poor quality?

We would like to clarify that our paper does not evaluate or include SFT models. To avoid confounding effects from supervised fine-tuning, we focus exclusively on comparing base models with RL-tuned models. This design complements prior studies that analyze the interplay between SFT and RL [1,2] (l.194–195).

All results involving planning and knowledge interventions (e.g., Sec. 6.3 and 6.4) are based on this Base vs. RL-tuned comparison. (summarized in l.220–237).

In Sec. 6.3, we find that providing external plans can degrade performance—particularly on the most challenging benchmark (AIME24). However, this is not due to poor instruction quality, but rather reflects a misalignment between externally provided plans and the model's preferred internal plans.

Moreover, the base model itself exhibits some instruction-following ability: for example, its performance improves with planning on GSM8K (Fig. 3, l.279–284), and with sufficient sampling (Fig. 2), it can match RL-tuned model performance.

Q2.2&Q3: The complementary information is generated by LLMs, which may contain toxic and distracting elements...Does the absence of human verification not undermine the effectiveness of your framework?

Human verification is an integral part in the benchmark construction process and has been included in this work.

We would like to clarify that we utilize a web agent to "extract a planning skeleton and summarizing key reasoning steps" (l.145-146) from existing step-by-step solutions in the test sets, instead of prompting LLM to generate a plan without reference. Moreover, we address this risk through a rigorous multi-stage annotation and verification pipeline, as described in Sec. 3.2 (l.149–153). Specifically:

• A second verification agent checks each generated annotation for correctness, coherence, completeness, and pedagogical soundness. If any aspect fails, the annotation is regenerated.

• Final validation is conducted by graduate students with advanced mathematics backgrounds to ensure that each annotation faithfully captures the required reasoning and knowledge.

In addition, to curate the training set for the second-stage RL tuning, we also involved "human verification to further filter out items with flawed solutions" (l.207-208). We believe this step is important for mitigating the potential brittleness of the GPT-4.1-based web agent.

We hope this clarifies that human verification is not only present but plays a critical role in ensuring the reliability and effectiveness of our framework. We would appreciate if you could re-evaluate our paper in light of the human verification measures we have in place.

Q4: How do you justify the inclusion of partial solutions in RL? Partial solutions may train models to rely on step-by-step hints, reducing autonomy—contradicting RL’s goal of fostering self-generated strategies.

Thank you for raising this point. Our motivation is grounded in prior work showing that curriculum-based approaches can significantly enhance the effectiveness of RL training for LLMs (see Sec. 2). However, outcome-based RL methods such as GRPO often struggle to learn from the most challenging problems due to the sparsity of positive reward signals (i.e., most sampled trajectories in a group result in incorrect answers).

This raises the question: can we still leverage these hard problems in a way that improves performance without compromising the capabilities already learned by a well-tuned RL model?

We agree that overusing partial solutions may reduce autonomy if applied indiscriminately. However, our approach is targeted and carefully scoped. Partial-solution augmentation is not used as a general strategy for RL training (i.e., Stage 1), which could encourage dependency on hints. Instead, we introduce partial-solution scaffolding only in Stage 2, and only on the hardest problems that the Stage 1 RL-tuned model consistently fails to solve.

Therefore, We would like to clarify that partial solution augmentation is not used as a general approach for RL training on all problems (Stage 1), which "may train models to rely on step-by-step hints". We only perform partial solution scaffolding in the second stage, on the hardest problems the first-stage RL-tuned model cannot solve, to further boost the reasoning performance.

Specifically, given an RL-trained model, we have explored three data strategies for the second-stage RL training (l.184-188):

• (1) Hard-Only: Training on difficult problems without any augmentation.
• (2) Mixed Difficulty: Combining easy and hard problems for exposure diversity.
• (3) Hard-Augmented: Introducing partial solutions to help the model navigate complex reasoning paths.

Our results (Table 1) show that the Hard-Augmented strategy significantly outperforms the other two, demonstrating that partial-solution scaffolding can effectively improve reasoning when the model is otherwise unable to learn from sparse feedback. In this context, partial augmentation serves as a practical curriculum tool to extend the learning frontier—rather than undermining reasoning autonomy.

Q5: Why not SFT instead if you have the "gold standard" solution?

We agree that a natural baseline would be to apply SFT using available solution traces. However, there are both practical limitations and empirical drawbacks that make SFT sub-optimal in our setting.

• The training set used in Stage 1 contains over 40K math problems, for which "gold standard" solutions are largely unreliable: many solution traces are noisy or incomplete. These traces vary in quality---from simplistic chain-of-thought outputs to human-written final answers. While obtaining new traces by distilling from competitive LLMs is possible, it would require substantial compute and additional human validation to ensure correctness and clarity.
• Empirically, we observe that applying SFT on noisy traces does not lead to improved performance. To evaluate this, we conducted a controlled comparison between three approaches:

As shown above, SFT on hard problems from the SparkleRL-Stage 1 model leads to significant performance degradation across all benchmarks. In contrast, our Stage 2-aug method yields consistent improvement.

The key distinction in Stage 2-aug (details in l.202–216) is that instead of fine-tuning on full traces that are noisy, we augment the hardest unsolved examples from Stage 1 with partial solutions in the input prompt. These partial solutions does not disrupt agent's on-policy learning, which is important for effective RL training. This preserves the benefits of RL, which encourages models to develop robust, self-directed reasoning strategies—given hints (partial solutions) from the challenging problems.

In contrast, SFT fine-tuning on noisy or overly prescriptive solutions can misalign with the model's learned heuristics, disrupting the internal reasoning behaviors acquired during RL training.

[1] Chu et al., Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025

[2] Yeo et al., Demystifying long chain-of-thought reasoning in llms, 2025.

We sincerely appreciate your thoughtful comments, detailed feedback, and support of our work! Due to word limit, we provide concise responses below. We are more than happy to elaborate further in the discussion phase.

Q1: Could the drop in base model performance stem from poor instruction-following? Please include Qwen-2.5-Math-7B-Instruct to isolate instruction tuning from reasoning.

We did not include Qwen-2.5-Math-7B-Instruct in this work because, despite its name, it is not a pure instruction-tuned model. As noted in [1], it was trained by SFT and GRPO, incorporating additional CoT and tool-use supervision. This makes it difficult to disentangle instruction tuning from more general improvements in reasoning capability.

Our focus is on comparing base pretrained vs. RL-tuned models to isolate the effect of RL on reasoning behavior. Notably, even the base model shows substantial instruction-following ability. In Fig. 2, scaling up the number of samples allows the base model to match RL-tuned model performance. On GSM8K, performance improves when planning information is provided—demonstrating plan-following capability even without SFT or RL (l.279–284, Fig.3).

We will clarify this distinction in the final version and are happy to include results for Qwen-2.5-Math-7B-Instruct if the reviewer would find it helpful.

Q2&O1: Claims like “RL-tuned models achieve superior performance...” need statistical backing. Include error bars in Table 1 and figures.

Thank you for this important suggestion. We have conducted detailed statistical analyses and updated the paper accordingly.

To support the specific claim in l.289—that “RL-tuned models achieve superior performance when permitted to develop autonomous planning strategies”—we conducted a Welch’s t-test comparing SparkleRL Stage 2-aug with vs. without planning on AIME24. Results show: t = 3.38, p = 0.0052 (< 0.05), Cohen’s d = −1.69 (large effect size). This confirms the performance difference is statistically significant, supporting our conclusion.

In addition, we now include mean ± standard deviation in Table 1 and have added error bars throughout the revised figures:

We also ran Welch’s t-tests comparing Stage 1 vs. Stage 2-aug across all datasets:

These results confirm performance improvements from Stage 1 to Stage 2-aug are statistically significant and robust across tasks. We will incorporate these findings and the requested error bars in the final version.

Q3: Why not attribute the gains to improved sampling efficiency alone?

We agree sampling efficiency can contribute to performance gains (as seen in Fig. 2). However, our goal is to explore what additional capabilities RL may confer. Key evidence:

• Plan handling: RL-tuned models exhibit greater robustness and lower variance across planning variants (Sec.6.3, Fig.3).
• Knowledge use: RL-tuned models can leverage external knowledge more effectively than base models (Sec.6.4).

These differences suggest that RL affects model behavior, not just sampling variance. This motivates our SPARKLE analysis, which decomposes reasoning across plan following, knowledge use, and subproblem solving.

Q4: How do you know the model is following the plan rather than ignoring it?

We define plan-following flexibility as the model’s ability to interpret, adapt, or selectively use planning information. All models receive identical prompts (see Fig.9), including: “Consider the following planning skeleton... you may adapt or extend this outline.”

Under this setup, RL-tuned models are more robust and consistent (Fig. 3), implying they can integrate or disregard plans as appropriate. Base models show higher variance, often underperforming when given plans—suggesting lower flexibility or misalignment.

Q5: How are difficulty levels assigned? Are there enough high-difficulty samples to draw conclusions?

We assign difficulty levels using AoPS Competition Ratings [2], which aligns with U.S. math competitions (e.g., AMC, AIME, USAMO, IMO). To avoid inductive bias, we deliberately do not use pass@k to assign difficulty--ensuring objective and interpretable levels.

Since SPARKLE enriches existing benchmarks without creating any new problems, our dataset inherits their natural skew--only 4 problems are level 9–10. We avoid drawing conclusions from the tail and focus on levels 1–8, where sample sizes are more reliable (e.g., 40–41 examples at levels 7–8).

At these levels:

• Knowledge augmentation improves RL-tuned performance by +4.9% (level 7) and +42.5% (level 8).
• planning causes a −4.9% drop (level 7) and a +15% gain (level 8).
• Weighted across levels, knowledge yields an average gain of +4.53% vs. +2.50% from planning.

These trends support our claim in l.334 that knowledge integration is key to solving harder problems. We will include difficulty histograms and clarify this in the revision.

Q6: Some claims are too broad given only one model was tested.

We will revise the language to clarify this. We also include results for Qwen-2.5-32B in App. D.2, which exhibit consistent trends. We experimented with other open models (Mistral-7B, LLaMA-3.1-8B), but they fail to achieve non-trivial performance even after RL tuning (e.g., avg@32 0.2% on AIME24), making meaningful reasoning analysis infeasible. This further motivates our focus on models with strong mathematical priors.

Q7: What prompts are used for plan/knowledge/decomposition? How do you ensure correctness?

SPARKLE is constructed using a pipeline (Sec. 3.2) where GPT-4.1 (with Internet access) is given: the original question, ground-truth answer, and expert step-by-step solution.

We prompt it to generate:

• Planning skeleton: 2–3 sentence outline of the reasoning path
• Subproblem decomposition: step-by-step breakdown into well-posed, answerable subquestions
• Knowledge extraction: key definitions, facts, or theorems (retrieved or inferred)

Each annotation is then checked by a second GPT-4.1 verifier agent (prompted to assess correctness, completeness, coherence, pedagogical value). Human validators (math grad students) perform final verification.

We will incorporate more of these implementation details in the revised appendix.

Q8: Please clarify how partial solution guidance works. How's it different from [3]?

Our partial solution guidance augments selected training problems with intermediate solution chunks.

Given a solution (excluding the final answer), we heuristically split it into 4 segments (e.g., by sentence) and create variants:

• V1: question only
• V2: question + chunk 1
• V3: question + chunks 1–2
• V4: question + chunks 1–3
• V5: question + chunks 1-4

This differs from [3] in two key ways:

• We do not enforce an easy-to-hard progression or assume the model can solve subgoals in isolation.
• We train in a single stage (Stage 2) with mixed supervision levels, rather than gradually increasing difficulty.

Our method provides a clear and practical way for incorporating hard problems during RL. Even when the traces are human-written and misaligned with the model’s internal reasoning, RL-tuned models still learn useful strategies and show improved performance.

Q9&Q10: Is Fig 6 averaged over all tasks? Can you add an “All” group to Fig 3–5?

Yes, Figure 6 reports results over the entire SPARKLE benchmark—we’ll clarify this in the caption and text. We will also add an “All Tasks” average to Figures 3–5 and include summary plots in the appendix.

Q11: What’s the subproblem-level solve rate? How does it relate to [4]?

Thank you for this suggestion and we present the subproblem solve rate with average difficulty level for each task below:

We find that subproblem solve rate drops as difficulty increases, showing that decomposition alone is not sufficient to solve complex tasks—models still struggle with intermediate steps, especially under higher cognitive load. Most problems decompose into 4–6 subproblems; we’ll include histograms in the appendix.

Compared to [4], our decompositions:

• Are semantically aligned, derived from expert step-by-step solutions.
• Span harder benchmarks beyond GSM8K (e.g., AIME, Olympiads).
• Are studied under RL, which show that solving full problems does not imply solving subproblems (a gap unseen in prior work). This shows RL-tuned models may solve holistically, but are brittle under explicit decomposition.

Q13: Couldn’t your results suggest we need more compositional training data?

Great question—we believe both directions are valid. However, our evidence suggests that not all decompositions are helpful. Rigid, human-written breakdowns can hurt model performance (e.g., −4.9% on Level 7), likely due to mismatch with internal solution paths.

Thus, we argue for RL methods that emphasize strategic flexibility—the ability to adapt to guidance, retrieve knowledge, and reason through difficult tasks. Compositional training remains promising, but is suggested to align with model reasoning rather than impose rigid scaffolds. We’ll clarify this in the revision.

[1] https://arxiv.org/abs/2409.12122

[2] https://artofproblemsolving.com/wiki/index.php/AoPS_Wiki:Competition_ratings

[3] https://arxiv.org/pdf/2402.05808

[4] https://arxiv.org/pdf/2410.01748v1

Q11-Q13: Subproblems

That's very interesting, thanks for providing that analysis! Are those numbers for SparkleRL-Stage 2?

Re: citation [4], I mentioned it not as a suggestion to show how your results are different (it's clearly a different setup along multiple dimensions), but more that their findings could possibly help explain/augment your results too. They found that questions are harder to solve in composition rather than independently, which might also be an explanation for why asking the model to solve subproblems might make the performance worse in your case—even if the model could solve each subproblem independently, it might struggle to compose them. I suppose we can't fully identify the "independent" solve rate in your case but we can roughly compute $p$ as the average subproblem solve rate (what you've reported above) and then $p^n$ where $n$ is the number of subproblems, and compare that to the actual compositional solve rate (what was already in the paper). So based on the numbers you reported above:

So from this rough analysis we can see that on AMC23, MATH500, and GSM8K you get the same results as [4]—the compositional solve rate is lower than what you'd expect than if you were solving each problem independently, perhaps suggesting something about the difficulty of solving the subproblems in composition. But interestingly on AIME24 and OlympiadBench the results go the other way! These are the hardest problems, and have probably been overfit to less in pretraining, so perhaps in this case the decomposition could help, if the model were better at solving the individual subproblems (which it seems quite poor at).

Q5, Q7-Q10

Thanks for the clarifications, these address my concerns.

We appreciate your comments and suggestions and we address each question and provide further clarifications in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.

W1: The dual focus of analyzing reasoning behaviors and improving learning from hard problems is disjointed.

The dual aspect here is intentional; in fact, we believe it is crucial for our overall findings. Our work is centered on analyzing mathematical reasoning under reinforcement learning (RL). When applying RL to a set of problems, the key knobs are (i) the components necessary to solve a problem (planning, knowledge, etc.,) and (ii) the raw difficulty of the problems. A full characterization of RL for reasoning is not possible without addressing both of these elements.

Our Hard-Augmented Stage 2 setting is not proposed as a general solution to the reasoning challenges observed, but rather as a complementary contribution: it demonstrates a practical method for leveraging hard problems in RL training—an area prior work has found difficult due to absent reward signals. Together, our dual contributions—(1) analysis framework and benchmark of reasoning behaviors across planning and execution, knowledge utilization, and chain of subproblems; and (2) a method to incorporate hard examples—provide stronger empirical evidence for understanding the effect of RL. The availability of two distinct RL-tuned models (SparkleRL-Stage 1 and SparkleRL-Stage 2-aug) allows us to conduct more fine-grained analysis than is typically possible in existing work.

W2: The proposed framework is also not comprehensive, as it fails to account for aspects such as the faithfulness of reasoning.

We appreciate this observation. Our work focuses on three core aspects of mathematical reasoning captured by the SPARKLE framework: plan-following, knowledge utilization, and subproblem decomposition. While faithfulness is indeed an important dimension, it is orthogonal to the goals of our current study. We believe that a fine-grained and dedicated analysis of these three components is equally valuable, as they represent foundational building blocks of mathematical reasoning.

Our goal is not to exhaustively cover all aspects of reasoning—necessary trade-offs must be made to support a focused investigation with a clear central theme. We hope our work can serve as a cornerstone for future research to extend the framework, including incorporating dimensions such as faithfulness.

W3: While the paper reveals that problem decomposition is not useful, this unintuitive result warrants further investigation, or could even motivate an approach to making better use of problem decomposition.

We would like to clarify that our paper does not claim that problem decomposition is not useful. Instead, our findings reveal a more nuanced perspective on how decomposition interacts with RL-tuned models.

In our analysis of plan-following (Sec. 6.3), we observe that RL-tuned models exhibit greater flexibility in generating and executing their own plans. Interestingly, even valid human-written plans do not always yield improved performance—particularly on harder benchmarks like AIME24—because high-level plans may overlook critical edge cases. However, for simpler datasets such as GSM8K (difficulty level 1 out of 10), explicit plans consistently improve performance for both base and RL-tuned models. In these cases, planning helps reduce cognitive load by structuring the reasoning into manageable steps.

In our subproblem decomposition analysis (Sec. 6.5), we consider a problem unsolved if any subproblem is answered incorrectly. Under this strict criterion, RL-tuned models often struggle, especially when the decomposition is overly granular. Combined with our plan-following results, this suggests that fine-grained decompositions may disrupt the model’s internal reasoning flow, rather than supporting it.

We view this as an valuable direction for future research: rather than relying on human-authored decompositions, it could more effective to design model-aligned decomposition strategies that are better matched to the model’s inductive biases. However, proposing a new algorithm that leverages this phenomenon would require dedicated study and is beyond the current scope of our work.

W4: The impact of additional training is not accounted for; comparisons to SFT and other offline methods such as DPO would be necessary to robustly show that it is RL training specifically that reinforces these behaviors.

Thanks for bringing this up! We would like to clarify that our paper does not evaluate or include SFT or DPO methods by design. To avoid introducing confounding effects, we deliberately focus on comparing base pretrained models and RL-tuned models, isolating the impact of RL. This design choice complements prior studies that specifically analyze the interplay between SFT and RL [1,2] (see l.194–195).

Therefore, it is not our objective to claim that RL, rather than SFT or DPO, uniquely reinforces these behaviors. Instead, our goal is to provide a focused, fine-grained analysis of reasoning behaviors that emerge specifically from RL training.

To ensure a clean comparison, we initialize both the base and RL models from the same pretrained checkpoint (Qwen-2.5-Math-7B) and apply RL training only, without any intermediate supervised fine-tuning (SFT) or preference optimization (e.g., DPO). This setup allows us to attribute the behavioral differences directly to RL, without contamination from other training paradigms.

While comparisons to SFT or DPO are indeed important in the broader landscape, they are outside the scope of this work. Our goal is not to establish whether RL is superior to SFT or DPO, but rather to provide a fine-grained analysis of how RL shapes reasoning behaviors—an area that remains underexplored despite RL's widespread use in modern LLM pipelines.

We hope this clarifies our motivation and experimental design. We believe that narrowing the focus to RL is not a limitation but a necessary step toward understanding its unique contributions.

[1] Chu et al., Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025 [2] Yeo et al., Demystifying long chain-of-thought reasoning in llms, 2025.

Q: Why does performance by difficulty level have a U-shape with knowledge augmentation?

Thank you for this thoughtful question. The observed U-shaped trend in Fig. 11 primarily reflects the natural imbalance in difficulty levels within existing math benchmarks. Our framework is designed to augment existing problems—not to construct new ones or rebalance the dataset—so it inherits a skewed distribution, with only 4 total problems at levels 9–10. This sparsity at the top makes accuracy volatile and can visually exaggerate trends like the U-shape. We will clarify this in the revised manuscript.

However, this pattern does not contradict our central finding: knowledge augmentation is more beneficial than planning, especially for harder problems. For example:

• On level 8 (40 problems), knowledge augmentation improves performance by +42.5%, compared to a +15% gain from planning.
• On level 7 (41 problems), knowledge gives a +4.9% gain, while planning causes a −4.9% drop.
• When weighted averaged across all levels, knowledge yields a +4.53% gain, compared to +2.50% from planning. These trends remain robust outside the sparsely populated level 9–10 range.

These trends are robust across sufficiently sampled levels (1–8) and support our conclusion that external knowledge access is a key driver of performance on complex tasks(l.334).

We also emphasize that difficulty levels are assigned based on AoPS Competition Ratings [AoPS], which map to the structure of major U.S. math competitions: Levels 1–2 correspond to AMC 8, early AMC 10 and AMC 12 problems (e.g., standard middle school math), Levels 4–6 span through later AMC 12, AIME and intro level Olympiad-styple problems, and Levels 7–10 include USAMO, IMO, and other hard Olympiad-level problems.

We deliberately avoid using metrics like pass@k of model performance to define difficulty levels, as they may introduce model-induced inductive bias. Instead, we rely on competition-based standards to provide an objective and interpretable standard. While pass@k filtering would allow easier balancing, it introduces model-induced inductive bias, which we explicitly avoid.

Thank you for your valuable feedback, thoughtful comments, and your support of our work! We address each question in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.

Q1: Any qualitative observations from looking at what sorts of human-provided plans (6.3) or subproblems (6.5) are difficult for the model to work with. Do you have any observations on how these plans and subproblems differ from those that the model comes up with and prefers to work with?

Yes, we obtained several qualitative observations while comparing model-generated reasoning with human-provided plans and decompositions.

Planning:

A common pattern is that high-level human plans often lack sufficient granularity, particularly for more difficult problems. This can lead models to overlook important edge cases, especially in mathematically nuanced settings.

For example, in Table 3 of the supplementary materials (AIME24), the provided planning instruction states: "Identify the pattern that arises for losing positions." Following this plan, the model correctly identifies only one of the two correct losing scenarios (n ≡ 2 (mod 5)), but misses the other (n ≡ 0 (mod 5)), leading to an incorrect final answer.

In contrast, when no plan is provided, the same model generates Python code to perform dynamic programming and simulates the game for all n ≤ 2024. This strategy leads to the correct result (809), successfully capturing both losing scenarios.

This example illustrates a broader trend: RL-tuned models often perform better when allowed to explore the problem space using their internal reasoning mechanisms (e.g., generating code or applying strategies they are familiar with), rather than following externally imposed plans. While human-written plans can be helpful for simpler tasks (e.g., GSM8K), they may restrict model flexibility or bias the model toward unfamiliar or suboptimal strategies on more challenging tasks.

Subproblem decomposition:

We observe a similar pattern when studying subproblem decomposition (Sec. 6.5). While human-provided subproblems are often semantically meaningful, breaking a problem into isolated steps can disrupt the model’s ability to reason holistically. In the same AIME24 example, a corresponding subproblem is phrased as:

Q: What is the pattern of losing positions for n tokens?

Despite being logically sound, the model still fails to identify one of the two key scenarios. This suggests that isolated subproblems can inherit the same brittleness as high-level plans if they are not grounded in the model’s natural reasoning strategies.

Q2: What motivates the Hard-Augmented setting and the possiblity of training on these solutions.

Thank you for the thoughtful question! We are motivated by prior work that suggests RL methods with outcome rewards such as GRPO cannot benefit from the hardest problems due to the lack of positive reward signals (i.e., all trajectories in a group yield wrong answers). We explore whether these hard problems could still be leveraged in a way that improves performance—without compromising the capabilities learned by a well-tuned RL model from Stage 1.

We agree that a natural approach would be to train on available solution traces with supervised fine-tuning (SFT). However, SFT does not yield promising results in our setting for the following reasons:

• The training set used in Stage 1 contains over 40K math problems, where many solution traces are noisy (e.g., incomplete solutions). These traces vary in quality---from simplistic chain-of-thought outputs to human-written final answers. While obtaining new traces by distilling from competitive LLMs is possible, it would require substantial compute and additional human validation to ensure correctness and clarity.
• Empirically, we conducted an additional experiment by applying SFT on these noisy traces. The results are shown below:

We observe that SFT on hard problems from the SparkleRL-Stage 1 model leads to significant performance degradation across benchmarks. In contrast, our Stage 2-aug method yields consistent improvement.

The key distinction in Stage 2-aug (see L202–216) is that we augment the hardest unsolved examples from Stage 1 with partial solutions provided in the input prompt, rather than full solutions used for fine-tuning. This design preserves on-policy learning, which is important for effective RL training, thereby further boosting the reasoning performance.

Typos

Thank you! We will carefully revise and correct all typos in the final revision.