Thank you for your kind and constructive feedback. We’re pleased that you found our approach valuable and have addressed your questions and concerns point by point below. We would be grateful to address further questions during the discussion period.

W1: Does BREAD introduce off-policy learning issues?

The expert traces are used to guide the trajectory generation, not to replace or bypass the model’s policy. In detail, BREAD samples rollouts by interpolating between the expert traces and the self-generated traces to adapt difficulty. However, the policy gradient update is always computed using the log-probabilities of the model’s own trace(online part). This ensures that the learning signals remain on-policy, even though the rollout paths may be partially guided by the expert. Therefore, BREAD benefits from expert traces without breaking the assumptions of policy gradient methods.

W2: Please say explicitly in the Policy Update section of Section 3 that the update (1) is the standard GRPO update.

A: Thank you for the suggestion. We will clarify in the next version.

W3,4: The presentation often does not provide all the details of the results.

A: Thank you for the suggestion. We will make this clearer in the next version. For the details of results in Section 3.2, the information is provided in the caption of Table 1: we use Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct as our base SLMs, and the S1K and S1K-1.1 datasets are generated following the procedure in [2]. Additional experimental details can be found in Appendix A.

The hint ratio in the x-axis in Figure 5b refers to the proportion of hint tokens appended to the query during inference, which is distinct from the 40% hint ratio (orange squares) used during RL training. A higher hint ratio means more guidance for the model. Currently, this setting is explained in Lines 180–185 and 198–201. Our results show that conditioning RL training on partially hinted questions not only improves performance on hinted evaluation queries (e.g., those with 40% hints), but also leads to improved accuracy on fully unhinted queries. This highlights the model’s ability to generalize from partial short traces to full long traces.

W5: Compare to other methods in a standardized setting.

A: Thank you for the suggestion. We agree that standardized comparisons are valuable. In our work, we include vanilla GRPO and DeepSeek-Distill as baselines to enable such comparisons. Regarding recent works, we would like to highlight that our focus is on challenging tasks and SLMs where existing methods like DAPO[1] are not specifically designed to perform well. So, these settings are typically not included in the evaluation benchmarks of DAPO and similar methods. For example, DAPO use Qwen2.5-32B instead of smaller models on those challenging tasks,

For example, we train and evaluate DAPO on the same difficult dataset used in Figure 7, under the same settings. We observed that DAPO shows little improvement. This aligns with its design: DAPO’s dynamic sampling strategy tends to filter out hard questions rather than try to solve them, making it less useful for the types of tasks we target with BREAD.

Q1,2: Are we training on the test set?

A: No, we do not train on the test set. In Figure 6, we evaluate using the official test subset provided by the Math/Numina-CoT benchmark. In Figure 7, we create an 80/20 train/test split from the dataset to ensure a clear separation between training and evaluation data. And in Table 2, we train on the training set and evaluate using the test subset provided by the Math/Numina-CoT benchmark.

Q3: Training data for SFT

A: Yes, we use a much lower amount of data. For SFT(full), we use the training subset of MATH(about 7500 samples)/NuminaMath-CoT to do SFT. In contrast, DeepSeek-Distill uses a much larger dataset. They use approximately 800k samples, including both reasoning and non-reasoning samples.

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale

[2] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.

We thank the reviewer for their continued support and thoughtful comments.

Regarding the use of Qwen models, we have conducted additional experiments on the LLaMA3.2-3b-Instruct model. The results are consistent with Qwen experiments. For the final manuscript, we will incorporate at least one more model family such as Phi3 or Phi4 models from Microsoft. To make sure our comparison is fair, we follow the LLaMA3 technical report, which uses a 0-shot CoT prompt. Specifically, because the technical report does not provide detailed prompt, we use a general and intuitive 0-shot CoT prompt and provide it here to make sure our result is reproducible:

messages = [
  {“role”: “system”, “content”: “You are a helpful assistant that reasons step by step to solve complex problems.“},
  {“role”: “user”, “content”: QUERY + “Let’s think step by step.“}
]

The result are for the LLaMA3.2-3b-Instruct model and are as follows:

(1) On the full MATH dataset:

(2) We also identify a hard subset of MATH dataset using similar methodology as described in the paper (Figure 7). Training over this dataset yields:

These experiments support that the method generalizes beyond the Qwen family and is not model-specific, and the effectiveness is more significant when the task is harder. We will also conduct and incorporate more experiments for the final version.

Finally, we will incorporate and provide a discussion of the additional related work cited by other reviewers in the final version. Thank you again for the helpful feedback.

We appreciate your positive evaluation of the manuscript and your thoughtful suggestions. Thank you for acknowledging our approach—please find our responses to your questions and concerns below. We would be grateful to address further questions during the discussion period.

W1: More evidence to conclude that performance degradation with SFT-only arises from model limitations.

A: In addition to empirical results on real datasets, we provide theoretical insights to illustrate the limitations of SFT as presented in Section 3.1. It is known that an L times deeper LLM can internally simulate L chain-of-thought steps of a smaller LLM [1]. So the expert model can generate a dense reasoning trace which necessitates an L times longer simplified trace for the student model to digest via SFT. Translating this to a toy student-expert model over Markov chains, we show how the student model cannot benefit at all from the expert model’s trace. As for more empirical evidence, some recent papers also have similar observations [2,3].

We also would like to clarify that our intent in Section 3.2 is not to claim that SFT-only universally leads to performance degradation. Rather, we aim to show that the limitations of SFT can cause SFT + RL strategies to fail, particularly in complex reasoning tasks involving long traces. We highlight a specific failure mode that can arise even when using widely used SFT datasets, and our results serve to motivate the need for the proposed method BREAD. We also acknowledge the reviewer's concern and will clarify that the strict degradation SFT in Table 1 may stem from the distributional difference between S1K and MATH/GPQA datasets. Note that, we pick the s1k dataset because the s1 paper [8] shows that SFT with s1k achieves very strong results on the 32B model. So it is naturally surprising that our experiments demonstrate that the same training data can in fact hurt the performance of small models. We speculate that this is because the s1k dataset is too hard for small models (besides the distribution shift) but more digestible for large models. Furthermore, NuminaMath evals on Table 2 indicate that SFT can hurt performance even on the same distribution. We will carefully clarify these points in the final revision.

W2: All datasets are all mathematics-related.

A: Thank you for the thoughtful feedback. Mathematical reasoning tasks are widely used in prior works [3-9] as a standard benchmark for evaluating reasoning capabilities. Also, these tasks strike a critical balance, they are sufficiently challenging, yet solvable for SLMs with the correct training strategy. This makes them particularly well-suited for studying GRPO variants like BREAD.

Additionally, the availability of reliable expert traces in this domain allows us to conduct meaningful and controlled experiments, which may not be feasible in other domains.

Therefore, we believe that math benchmarks serve as a strong and appropriate testbed for analyzing the effectiveness of our method. But we really appreciate the reviewer’s concern and would welcome suggestions on alternative reasoning datasets. We are open to expanding our evaluation in future versions.

Q1: Did you observe sparse rewards in RL or performance degradation from SFT with larger models?

A: Yes. The spare reward issue can happen even with Qwen2.5-32B-Instruct, it achieves only 10% accuracy on AIME, which improves to 37% after fine-tuning on the S1K dataset. However, even at 37% accuracy, reward signals in RL remain sparse(19.5% of samples all 8 rollouts fail). BREAD targets the fundamental limitations of SFT and RL that are not constrained by the model size. Although these problems are more noticeable in small models, high-capacity models can also struggle with similar difficulties, particularly when dealing with complex reasoning tasks where high-quality traces are hard to generate and the RL reward is sparse due to low probability of success.

Q2: How many reasoning steps are involved in the expert traces constructed in this paper?

A: We provide statistics on reasoning trace length in Figure 10 (Appendix B.3: Episode Details). For the MATH dataset, the expert traces have an average of 8.42 steps, with a minimum of 1 and a maximum of 228, and a standard deviation of 11.46. For the NuminaMath-CoT dataset, the average is 7.08 steps, with a minimum of 1, maximum of 97, and a standard deviation of 5.40. While reasoning trace length varies widely, BREAD is designed to be robust to this variability. BREAD dynamically creates a learning curriculum by adaptively inserting short expert prefixes/hints.

[1] Saunshi, Nikunj, et al. "Reasoning with latent thoughts: On the power of looped transformers." arXiv preprint arXiv:2502.17416 (2025).

[2] Chu, Tianzhe, et al. "Sft memorizes, rl generalizes: A comparative study of foundation model post-training." arXiv preprint arXiv:2501.17161 (2025).

[3] Li, Yuetai, et al. "Small models struggle to learn from strong reasoners." arXiv preprint arXiv:2502.12143 (2025).

[4] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." arXiv preprint arXiv:2503.14476 (2025).

[5] Qu, Yuxiao, et al. "Optimizing test-time compute via meta reinforcement fine-tuning." arXiv preprint arXiv:2503.07572 (2025).

[6] Cetin, Edoardo, Tianyu Zhao, and Yujin Tang. "Reinforcement Learning Teachers of Test Time Scaling." arXiv preprint arXiv:2506.08388 (2025).

[7] Dong, Qingxiu, et al. "Reinforcement Pre-Training." arXiv preprint arXiv:2506.08007 (2025).

[8] Muennighoff, Niklas, et al. "s1: Simple test-time scaling." arXiv preprint arXiv:2501.19393 (2025).

[9] Aggarwal, Pranjal, and Sean Welleck. "L1: Controlling how long a reasoning model thinks with reinforcement learning." arXiv preprint arXiv:2503.04697 (2025).

Thank you for your detailed and constructive feedback. Below, we address the comments point by point. We would also be grateful to hear any additional feedback or questions.

W1.1 & W3: The issues BREAD target also exist for larger models.

A: Thank you for pointing this out. We agree that the two core issues identified in our paper are not exclusive to small models. However, this is not a weakness of the paper, instead, it highlights the method’s potential applicability across a wide range of settings, making it more broadly useful. Due to limited computational resources, we were not able to run experiments on large models. However, we are open to demonstrate BREAD’s scalability to larger models as a promising future direction.

Meanwhile, we remark that these issues are more acute in small models. Also, BREAD benefits from a strong expert/teacher model to provide high-quality traces (human experts may not readily be available). For this reason, our experiments prioritize SLMs, where the impact of BREAD is most clearly observable. Overall, our focus on SLMs is not a limitation, but a significant contribution in itself for several reasons. SLMs are increasingly important for cost-sensitive deployments, such as on-device reasoning and agentic AI [5]. Most reasoning benchmarks show a large performance gap between SLMs and LLMs. Many pipelines that work well for large models fail to scale down to smaller ones. So, improving reasoning in SLMs remains a central open problem.

W1.2: Novelty concerns and comparison with suggested papers.

A: Thank you for pointing out connections to previous papers. We will cite the papers and include a discussion in our revised version. We will also add R^3 as a baseline. The results and the setting are provided in the response to W4.

We agree that BREAD shares conceptual similarities with [1,3]. All methods aim to mitigate the challenge of sparse rewards by starting training closer to successful endpoints.

But there are several key differences between BREAD and previous papers:

(1) Dynamic vs. fixed hint: Previous works use a manually defined schedule to adjust the prefix (hint) ratio during training, which requires careful tuning. In contrast, BREAD dynamically selects the prefix length for each question at each iteration, using a binary search to automatically adapt the difficulty.

(2) Use expert traces only when needed: Previous works require full expert traces for all training examples, whereas BREAD adaptively applies hints only when needed, significantly reducing the cost of expert trace generation by human experts or large models.

(3) Cold-start capability: R^3 relies on an initial SFT warm-up phase before applying reverse RL, whereas BREAD can improve performance without requiring any prior SFT initialization. We also provide both empirical and theoretical results showing that SFT initialization can fail, and this phase can be expensive.

(4) Theoretical grounding: Complementing our empirical results and previous works, we provide a Markov chain-based theoretical model to explain why small models could struggle with long reasoning traces, why GRPO without hints can perform poorly, and how BREAD mitigates the sparse reward issues.

These differences show the novelty and broader applicability of BREAD, especially in settings where expert traces are costly and model capacity is limited.

W2: Shao et al. raise concerns about the generalizability of the results evaluated on Qwen family models.

A: Thank you for the question. The main message in Shao et al. [4] is not contradictory with ours: They argue that pure RLVR cannot improve the models’ ability, but makes the latent ability in the base models show up. That’s exactly why we should utilize expert traces in BREAD so that new knowledge can be infused during RL training to truly improve the models’ reasoning ability. Notably, [4] states a maximum accuracy of 78.5% on MATH using Qwen2.5-7B, whereas our paper achieves 84.3% with BREAD using the smaller Qwen2.5-3B under the same number of iterations. So it also doesn’t contradict BREAD’s performance benefits, as BREAD goes strictly beyond RLVR with both spurious and true labels.

Additionally, we evaluate BREAD by training LLama3-1B on GSM8k dataset. BREAD achieves meaningful improvements over vanilla GRPO and noticeably outperforms SFT+GRPO, while also reducing the cost of expert trace generation by selectively using hints rather than full traces for every example.

It is worth mentioning that the Qwen model family serves as a well-established testbed for mathematical reasoning tasks and methods, as evidenced by its exclusive use in multiple notable papers [6-10]. However, we are grateful for the reviewer's recommendation to use additional models and will provide additional evaluations in the final version.

W4: Lack of comparison with other methods.

A: Thank you for the suggestion. We ran additional experiments for R^3 as the baseline on MATH. For fair comparison, we deploy the same R^3 on GRPO and train for 300 steps in total for both methods.

BREAD demonstrates clear improvements in both accuracy and sample efficiency. Unlike R^3, which requires full expert traces for every example, BREAD learns effectively using only a small fraction of expert traces. This is important because collecting full reasoning traces is prohibitively expensive. It requires either human labeling or generation by large models such as DeepSeek-R1, whose single forward pass already costs more FLOPs than 25 RL training steps with 8 rollouts. BREAD’s ability to reduce this cost while improving performance highlights its practical value.

[1] Xi, Zhiheng, et al. "Training large language models for reasoning through reverse curriculum reinforcement learning." arXiv preprint arXiv:2402.05808 (2024).

[3] Florensa, Carlos, et al. "Reverse curriculum generation for reinforcement learning." Conference on robot learning. PMLR, 2017.

[4] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arXiv preprint arXiv:2506.10947 (2025).

[5] Belcak, Peter, et al. "Small Language Models are the Future of Agentic AI." arXiv preprint arXiv:2506.02153 (2025).

[6] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[7] Cetin, Edoardo, Tianyu Zhao, and Yujin Tang. "Reinforcement Learning Teachers of Test Time Scaling." arXiv preprint arXiv:2506.08388 (2025).

[8] Dong, Qingxiu, et al. "Reinforcement Pre-Training." arXiv preprint arXiv:2506.08007 (2025).

[9] Muennighoff, Niklas, et al. "s1: Simple test-time scaling." arXiv preprint arXiv:2501.19393 (2025).

[10] Aggarwal, Pranjal, and Sean Welleck. "L1: Controlling how long a reasoning model thinks with reinforcement learning." arXiv preprint arXiv:2503.04697 (2025).

Based on the authors' response, I will raise my score. However, I still recommend that the authors carefully consider these comments.

Thank you for engaging with and acknowledging our response and the additional experiments we have provided.

We fully agree that broader validation is important and are pleased to report that our method remains effective when applied to LLaMA models. In particular, our experiments in previous response with the LLaMA3-1b model show notable gains over baselines. We also conducted additional experiments on the LLaMA3.2-3b-Instruct model. The results are consistent with Qwen experiments. For the final manuscript, we will incorporate at least one more model family, such as Phi3 or Phi4 models from Microsoft. To make sure our comparison is fair, we follow the LLaMA3 technical report, which uses a 0-shot CoT prompt. Specifically, because the technical report does not provide a detailed prompt, we use a general and intuitive 0-shot CoT prompt and provide it here to make sure our result is reproducible:

messages = [
  {“role”: “system”, “content”: “You are a helpful assistant that reasons step by step to solve complex problems.“},
  {“role”: “user”, “content”: QUERY + “Let’s think step by step.“}
]

The result are for the LLaMA3.2-3b-Instruct model and are as follows:

(1) On the full MATH dataset:

(2) We also identify a hard subset of MATH dataset using similar methodology as described in the paper (Figure 7). Training over this dataset yields:

These experiments support that the method generalizes beyond the Qwen family and is not model-specific, and the effectiveness is more significant when the task is harder.

While our focus is on small models due to compute constraints, we are committed to scaling our experiments to larger models and other model families in future work to further validate the general efficacy of our approach.

Thank you again for your constructive feedback.

Thank you for your encouraging feedback on our manuscript. We’re grateful for your recognition of our approach and your constructive comments. Below, we respond to your questions and concerns. We would be grateful to address further questions during the discussion period.

Q1: What motivates modeling the limited capacity of small models by limiting the learnable graphs in the Markov chain?

A: Our motivation comes from three insights: (1) Recent studies[1] show that an L-times deeper LLM can simulate L chain-of-thought steps of a smaller model. This implies that for a student model with limited depth/capacity, a dense reasoning trace from a stronger expert must be broken down into shorter, more manageable segments to enable learning via SFT. (2) Compositional nature of reasoning: Reasoning tasks are inherently compositional and typically require multi-step chain-of-thought processes to arrive at a correct solution [2]. Small models often struggle to learn such long trajectories end-to-end. (3) Markov chains are more amenable to theoretical analysis while being representative of sequential/language data [3-7].

Q2: Can BREAD generalize beyond SLMs?

A: Yes, BREAD is a general method that is not limited by model size. We thank the reviewers for raising this point, and we are open to demonstrate BREAD’s scalability to larger models as a future direction. This highlights the method’s potential applicability across a wide range of settings, making it even more broadly useful.

BREAD targets the fundamental limitations of SFT and RL. SFT often struggles to teach hard-to-imitate traces, and standard RL often suffers from sparse rewards. Although these problems are more noticeable in small models, high-capacity models can also struggle with similar difficulties, particularly when dealing with complex reasoning tasks where high-quality traces are hard to generate and the RL reward is sparse due to low probability of success. For example, Qwen2.5-7B-MATH, the accuracy drops from 13.3% to 6.67% on AIME from 15.1% to 5.6% on GPQA after SFT with S1K dataset. Qwen2.5-32B-Instruct achieves only 10% accuracy on AIME, which improves to 37% after fine-tuning on the S1K dataset. However, even at 37% accuracy, reward signals in RL remain sparse(19.5% of samples all 8 rollouts fail). BREAD addresses this by adapting problem difficulty through a curriculum learning that interpolates between expert (offline) traces and self-generated (online) rollouts. We therefore believe BREAD can help in large-model settings, particularly for tasks involving long reasoning chains or limited supervision. But because of the computational source constraint, it’s hard for us to show results on larger models. We will continue this as future work.

Q3: Does BREAD introduce off-policy learning issues?

A: The expert traces are used to guide the trajectory generation, not to replace or bypass the model’s policy. In detail, BREAD samples rollouts by interpolating between the expert traces and the self-generated traces to adapt difficulty. However, the policy gradient update is always computed using the log-probabilities of the model’s own trace(online part). This ensures that the learning signals remain on-policy, even though the rollout paths may be partially guided by the expert. Therefore, BREAD benefits from expert traces without breaking the assumptions of policy gradient methods.

[1] Saunshi, Nikunj, et al. "Reasoning with latent thoughts: On the power of looped transformers." arXiv preprint arXiv:2502.17416 (2025).

[2] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837.

[3] Zekri, Oussama, et al. "Large language models as markov chains." arXiv preprint arXiv:2410.02724 (2024).

[4] Makkuva, Ashok Vardhan, et al. "Attention with markov: A curious case of single-layer transformers." The Thirteenth International Conference on Learning Representations. 2025.

[5] Edelman, Ezra, et al. "The evolution of statistical induction heads: In-context learning markov chains." Advances in neural information processing systems 37 (2024): 64273-64311.

[6] ldiz, M. Emrullah, et al. "From self-attention to markov models: Unveiling the dynamics of generative transformers." arXiv preprint arXiv:2402.13512 (2024).

[7] Kim, Hyunsu, et al. "Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo." arXiv preprint arXiv:2503.00699 (2025).

Thank you for the helpful suggestion. We plan to include additional ablation studies in the next version to address this concern.

(1) We will evaluate and compare unclipped BREAD and SFT + unclipped GRPO, which will help determine whether clipping has a more significant effect on BREAD or not.

(2) Regarding the off-policy issue, we agree with the reviewer that a "more online" version is unlikely to significantly improve over our current algorithm. We will include a more detailed discussion on this point and provide empirical results for a "more online" variant to further support this claim.

Preliminary experiments confirm this expectation. We tested a more online variant on the difficult subset (same setting as Figure 7) and observed only a marginal difference, just a one-question gap, which may be due to random variation. In this "more online" variant, rather than relying entirely on fixed expert traces, we have declared any successful trace generated by BREAD as the new expert trace. This dynamic update mechanism aims to keep the training more aligned with the current policy, as long as the model continues to make progress by producing successful solutions.

Thank you for the detailed and constructive feedback. Here, we answer the questions and address the concerns. We would be grateful to address further questions during the discussion period.

W1: lack of references of backplay.

A: Thank you for pointing this out. We agree that BREAD shares conceptual similarities with Backplay [1,2]. Both methods aim to mitigate the challenge of sparse rewards by starting training closer to successful endpoints and gradually increasing difficulty. We will cite these works and include a discussion in the Related Work section.

Also, we would like to emphasize several key differences between BREAD and classical Backplay. While Backplay typically replays from fixed intermediate states and often requires delicate tuning, BREAD dynamically selects the hint ratio for each question during training, enabling adaptive difficulty adjustment without additional tuning overhead. This forms a curriculum tailored to each query, and offers significant computational efficiency.

Moreover, unlike Backplay, BREAD does not require expert traces for all training samples. This is important as generating expert traces is often prohibitively expensive for reasoning tasks(e.g., requiring human annotation or large model inference).

W2: What characteristics of a model make BREAD necessary or particularly beneficial?

A: Thank you for this question. We provide a general discussion of this topic in Section 5 (Discussion and Limitations), and we will expand on it further in the revised version.

BREAD is motivated by the observation that certain model characteristics make it challenging to learn complex reasoning behaviors through standard SFT or the model’s initialization has a very small likelihood of success so RL alone fails. In such cases, curriculum-based or backplay-style learning becomes particularly helpful. In particular, we believe BREAD is most beneficial under the following conditions:

1. Limited Model Scale: If the base model is strong enough to generate at least one correct rollout per query, BREAD becomes identical to the vanilla GRPO. However, for smaller or weaker models that cannot do so reliably, BREAD offers a strict advantage. So SLMs, that struggle to imitate or generate correct rollouts, are a key beneficiary.

2. Weak Prior Reasoning Ability: Models that lack training on structured reasoning tasks have difficulty in producing stable and correct reasoning traces during RL exploration. BREAD helps stabilize training by anchoring trajectories to expert traces. For instance, in our experiments, vanilla GRPO based on Qwen2.5-1.5B-Instruct shows limited improvement and is much worse than BREAD (Table 2). Whereas DeepScaleR-1.5B-Preview[5] achieves stronger results using DeepSeek-Distill-1.5b[3], which has the same architecture, but was obtained after careful supervised training. BREAD mitigates the need for this expensive supervised training stage. So BREAD is especially helpful when the base model starts from a weaker reasoning prior.

3. Hint-Responsiveness: BREAD is most effective when the student model can utilize partial hints to guide its reasoning. However, in some extreme cases, a model may fail to benefit from expert traces even when provided in full, due to its inability to understand and follow complex traces. For instance, Llama‑3.2‑1B‑Instruct[6] achieves 6.8% accuracy on MATH‑500, and this improves marginally to 7.3% with hints. Since the model cannot meaningfully use the hints, BREAD provides limited benefit (achieves accuracy 8.9%) in such scenarios.

Q1: Can we use the model's average accuracy on a question to warm-start or guide the branching point (e.g., binary search) more efficiently?

A: Thank you for raising this excellent suggestion. We do expect a correlation: questions with high pass@1 accuracy likely require little or no hinting, while harder questions benefit from longer anchored prefixes (i.e., branching closer to the answer). Our algorithm design reflects this intuition. However, in our current implementation, we prioritize sample efficiency and we only do binary search for queries where the model fails to generate any correct rollout (pass@1 = 0). This helps avoid generating expensive expert traces for easier questions. We discuss how expensive it is to generate the trace in paragraph "BREAD improves sample efficiency during training". But we agree that incorporating accuracy to warm-start the binary search could improve efficiency if we want to binary search on queries with different accuracy, and we see this as a promising direction for future work.

[1] Salimans, Tim, and Richard Chen. "Learning montezuma's revenge from a single demonstration." arXiv preprint arXiv:1812.03381 (2018).

[2] Resnick, Cinjon, et al. "Backplay:" man muss immer umkehren"." arXiv preprint arXiv:1807.06919 (2018).

[3] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[4] Liu, Aixin, et al. "Deepseek-v3 technical report." arXiv preprint arXiv:2412.19437 (2024).

[5] Luo, Michael, et al. DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL. Notion Blog, 2025, https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2.

[6] Meta AI. Llama‑3.2‑1B‑Instruct. Meta Llama 3.2 model card, Hugging Face, 25 Sept. 2024.