Thanks for your effort in reviewing our paper and the insightful comments. We appreciate your recognition of our paper’s elegance and interesting insights. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have:

W1: Validation of the Hypothesis on reasoning for general tasks.

Thanks for the thoughtful review. In the community of RL for reasoning, mathematical reasoning problems are adopted as common practices for evaluation traditions [1,2], so we follow the setting and mainly use mathematical reasoning to test our hypothesis.

To address your concern, we conducted an additional experiment on general reasoning benchmarks, including MMLU and GPQA. Following the experiment setting in Section 2.1, we first calculate the PCR for the Qwen-2.5 series in the following:

We can see that the PCR for other general reasoning tasks (MMLU, GPQA) is also pretty high, with over 90%. This indicates that our current hypothesis holds for reasoning generalization.

We then apply RAST to MMLU and GPQA, and the results are shown in the following:

We can see that the performance brought by RAST is consistent across both datasets, which further strengthens the effectiveness and generalizability of our method.

[1] Wang, Yiping, et al. "Reinforcement learning for reasoning in large language models with one training example." arXiv preprint arXiv:2504.20571 (2025).

[2] Havrilla, Alex, et al. "Teaching large language models to reason with reinforcement learning." arXiv preprint arXiv:2403.04642 (2024).

W2: Refine hypothesis w.r.t. different RL algorithms.

Thanks for raising this point. From a literature perspective, prior work [1] has shown that RL fine-tuning—regardless of the specific algorithm used—tends to result in models that share similar weights with only a small portion of the parameters modified.

From an experimental standpoint, we have also conducted additional analysis using models trained with different RL algorithms from [2] to examine whether similar behavior transfer effects hold in the following table. Note that all the base models here are Qwen-2.5-Math-7B series following [2], and we report the results on MATH500.

PCRs remain pretty high and consistent for RL models trained with a diverse range of algorithms (GRPO/PPO/REINFORCE). Therefore, both conceptually and experimentally, we have sufficient evidence to believe that our hypothesis is generalizable to different RL algorithms.

[1] Mukherjee, Sagnik, et al. "Reinforcement Learning Finetunes Small Subnetworks in Large Language Models." arXiv preprint arXiv:2505.11711 (2025).

[2] Zhu, Xinyu, et al. "The surprising effectiveness of negative reinforcement in LLM reasoning." arXiv preprint arXiv:2506.01347 (2025).

W3: Statistical analysis on different tokens for base and RL models to better support the Hypothesis.

Thanks for your comment and for reading our Appendix. Figure 10 in Appendix C.3 is exactly such a statistical analysis for demonstrating tokens related to reasoning behaviors in base, RL, and RAST models. We first manually curate a token list corresponding to three different reasoning behaviors. Then we use Spacy to statistically calculate the token frequencies in model-generated responses. Specifically, we compute the token appearances for each category (branch-out, backtrack, self-verify) in each math problem’s 32 generated trajectories, with the averaged number of appearances per trajectory as normalized frequency. For example, in Figure 10 (a), 1.101 means that tokens belonging to “self-verify” appear 1.101 times on average per problem. As stated in lines 694-700, RAST steers the model behavior from base models by actually changing the corresponding reasoning behaviors.

Tokens are important signals demonstrating reasoning behaviors from models, and are a widely adopted approach for analysis in the community [1,2]. Therefore, we believe our hypothesis and the main conclusions are well justified through the experiments mentioned above.

[1] Gandhi, Kanishk, et al. "Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars." arXiv preprint arXiv:2503.01307 (2025).

[2] Yeo, Edward, et al. "Demystifying long chain-of-thought reasoning in llms." arXiv preprint arXiv:2502.03373 (2025).

Q1: How does the relative size between the small and large models affect RAST’s effectiveness?

Thank you for the interesting question. In our current setup, we use a 1.5B RL-trained model to guide larger base models ranging from 7B up to 32B, and we consistently observe performance improvements across tasks. This suggests that RAST can be effective even when the small model is an order of magnitude smaller than the target model.

In practice, it is commonly observed that RL training becomes more effective at larger model scales [1], while very small models (e.g., ~100M) often struggle to learn meaningful reasoning behaviors via reinforcement learning. In many such cases, even SFT can outperform RL. As a result, directly using a 100M RL-trained model to guide a 70B model is unlikely to yield strong benefits. In this case, those extremely small models can be trained, e.g., via data distillation from a larger RL expert, offering another lightweight and performant alternative.

That being said, our primary motivation is to avoid the high cost of applying RL to large models, not necessarily to minimize the size of the small model. Training a small-sized model, say a 1.5B RL model, is already much cheaper and more accessible, yet still sufficient to capture reasoning behaviors.

Additionally, in a conceptual view, the $\Delta$ captured is used as the key driver of transfer effectiveness. A larger $\Delta$ intuitively tends to indicate more useful reasoning signals, which in turn improves the impact of the guidance on the target model.

[1] Cao, Yuji, et al. "Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods." IEEE Transactions on Neural Networks and Learning Systems (2024).

Q2: Jointly change for sensitivity analysis.

Thanks for your constructive comment. We have conducted the experiments jointly, and the results are listed in the following (here we only list a subportion due to character limit in rebuttal; row indicates different $\lambda$ while column indicates different temperatures):

We can see that our conclusion still holds: when the choice is within a certain range for decoding temperature and $\lambda$, RAST demonstrates a robust performance and is insensitive to parameter choices.

Thank you for the encouraging review and constructive comments! We are happy that you think RAST is intuitive and well-motivated, with great performance and significant potential for efficiency. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have:

W1 & Q2: Applying RAST between models with different tokenizers.

Thanks for the insightful comment. While we assume the same tokenizers across our experiments, we think that RAST could be applied between models with different tokenizers. The core technique here is tokenizer alignment/mapping across different LLMs, which is another significant research area in the community. So we do not incorporate it in our paper. RAST actually complements those methods, for example, Contextual Dynamic Mapping [1] could be first applied and then we can calculate the distribution for decoding accordingly.

[1] Chen, Yijie, et al. "Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping." ACL Findings 2025.

W2: RAST requires 3 models simultaneously for inference.

Thanks for your comment. We acknowledge that our current setup involves 3 models at inference time, which introduces additional memory and compute overhead during inference. However, the inference usage is in exchange for the computationally costly training. We estimate the training and inference FLOPs based on [1] and training hardware/time in [2]. Specifically, for 14B models, they used 16 H100-80G GPUs for 15 hours; and for 32B models, 64 H100-80G GPUs are used for 36 hours. We assume peak FLOPs for H100 as 1979 TFLOPS with 50% FLOPs utilization during training. The results are presented in the following table:

We can see that the reduced training FLOPs are all worth the increased inference FLOPs. Additionally, the inference stage could be combined with other techniques that can further improve the efficiency, just as you mentioned in Q1, which we will elaborate on later.

[1] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

[2] Zeng, Weihao, et al. "Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild." arXiv preprint arXiv:2503.18892 (2025).

W3 & Presentation: Typos.

Thanks for pointing them out! We have fixed them accordingly.　

Q1: Potential of combining RAST and speculative decoding to improve inference speed.

Thanks for the interesting question. We believe there is great potential for RAST to be integrated with speculative decoding for inference acceleration. We provide some initial thoughts here.

One direct way is to use a draft base model to propose tokens (e.g., 7B base model), and then verify them using the full RAST-composed logits. This may reduce the number of full composite forward passes and decoding steps, especially when the draft is accurate.

Q3: Discussion on RAST-aware training.

Thanks for the insightful question. We believe RAST-aware training holds great potential in boosting model performance. We envision several directions in the following:

(i) RAST-regularized RL training: One promising approach is to convert the logit deltas and the corresponding branching/backtracking signals into explicit regularization terms during training [1]. These signals can penalize or encourage certain decision patterns, enabling small models to internalize the reasoning behaviors by enforcing trajectory exploration/exploitation during the RL training process.

(ii) Adapter-based distillation: RAST outputs/deltas can be used as supervision targets to train task-specific adapters or lightweight finetuning modules (e.g., via LoRA or LoRA Hub-style infrastructure). For the task-specific adapters, speculative decoding [2] would be a good direction to consider. We could also decompose different reasoning behaviors from RAST via LoRA, and achieve controllable and composable reasoning in the style of LoRA-Hub.

(iii) RAST-enhanced data augmentation: RAST can be used to generate multiple high-quality and diverse trajectories. For example, we can use test-time scaling to generate new trajectories by sampling on the position of key reasoning tokens. These trajectories—enriched with exploratory behavior and reasoning patterns—serve as additional supervision signals, effectively augmenting the training dataset for training smaller language models to be generalizable and effective.

Overall, we think RAST provides a new paradigm for training efficiency and provides huge potential for follow-up work. We had some initial discussions on Appendix E, and we will incorporate these new insights into the next version of our paper.

[1] Wang, Shenzhi, et al. "Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning." arXiv preprint arXiv:2506.01939 (2025).

[2] Fu, Tianyu, et al. "R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing." arXiv preprint arXiv:2505.21600 (2025).

Thank you for the positive review and constructive comments! We are grateful for your recognition of our paper’s clear formulation, cornerstone hypothesis with comprehensive experiments and clear analyses. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have:

W1: The related work section lacks concrete comparisons and integration with the main discussion.

Thank you for the feedback. The current related work section is organized based on the unique position of our paper. Our goal is to establish a new paradigm for leveraging small RL-trained models to guide larger models at test time, thus circumventing the high cost of RL training for large models. To the best of our knowledge, there is no existing baseline that directly fits this setup.

Therefore, we choose two subsections of topic-wise — RL training for reasoning models, and technique-wise — decoding-time strategy for behavior steering. We will further refine Section 5 to make these contributions more explicit and will contrast our approach with the literature (e.g., training-free alignment) following your advice.

W2: The relevance and clarity of Figure 5 are unclear.

Thanks for the constructive suggestion. We appreciate the opportunity to clarify the purpose of Figure 5 and its connection to Section 4.2 (Lines 237–245).

Figure 5 serves as a concrete, intuitive illustration of the token-level behavior shift we discuss in Section 4.2. While our primary analysis is quantitative (e.g., high PRC scores, KLD spikes on specific tokens), Figure 5 complements this by showing how these shifts manifest in actual model outputs. Specifically, it demonstrates how the RAST-guided model $\tilde{M}$ introduces reasoning-critical steps—such as proposing, verifying, and ruling out possibilities—that are missing from the base model $M_{base}$.

Lines 237–245 describe this exact difference in reasoning patterns: while $M_{base}$​ proceeds deterministically and makes early errors, $\tilde{M}$ engages in more structured reasoning, including self-verification ("check") and hypothesis testing ("possible"). These moments of divergence correspond precisely to tokens with high KLD, as quantified and highlighted in green in Figure 5. This visual alignment between semantic generation and measured divergence provides strong and interpretable support for the behavior shift hypothesis. Therefore, Figure 5 is meant to enhance interpretability by showing exactly what kinds of reasoning behaviors are being introduced by RAST and where. We will revise the text to make this connection more explicit.

Limitation: Lack of a dedicated discussion outlining the conceptual and experimental limitations of the work.

Thanks for your attention to this point. Due to space limitations, we moved the limitation discussion of RAST to the Appendix. Please refer to Appendix F for a conceptual outline of limitations both method-wise and experiment-wise.

Thanks for reviewing our paper. Below, we provide detailed responses to each of your comments and respectfully hope you can consider adjusting the current rating:

W1: The idea of knowledge transfer from small to larger models is not new.

We respectfully believe that this concern may reflect a misunderstanding of our contribution.

First, we are not claiming to build a brand-new method for simply transferring knowledge from small to large models. Instead, we first reveal an important insight by introducing and verifying our key hypothesis in line 40. Built upon our insight, we propose a simple and intuitive method to activate reasoning capabilities in large models through a smaller RL-trained model. This way, we can bypass the costly training resource by training large RL models.

Technically, our methodology belongs to the research line of decoding-time steering of model behaviors, and we have already discussed this research line in detail in Section 5.2. It differs fundamentally from knowledge distillation or model compression that you mentioned.

W2: RAST is only evaluated on math reasoning tasks and specific tasks.

(i) In the community of RL for reasoning, mathematical reasoning problems are adopted as common practices for evaluation traditions [1,2], so we follow the tradition and mainly use mathematical reasoning to test RAST in a controlled setting.

(ii) Our goal is not to claim a universal solution for all general reasoning problems. We believe that it is unrealistic to expect a single paper to address the full space of general reasoning tasks. Instead, we aim to provide a focused and well-supported demonstration that reasoning activation via test-time RL-guided decoding is effective and efficient—starting with math reasoning as a representative domain.

(iii) We have conducted experiments on domains other than math. In section 3.4, we provide experiments on code reasoning and results in Table 3 demonstrate RAST’s generalizability. During the rebuttal period, we also test on other datasets like MMLU and GPQA. Following the experiment setting in Section 2.1, we first calculate the PCR for the Qwen-2.5 series in the following:

We can see that the PCR for other general reasoning tasks (MMLU, GPQA) is also pretty high, with over 90%. This indicates that our current hypothesis holds for reasoning generalization.

We then apply RAST to MMLU and GPQA, and the results are shown in the following:

We can see that the performance brought by RAST is consistent across both datasets, which further strengthens the effectiveness and generalizability of our method.

[1] Wang, Yiping, et al. "Reinforcement learning for reasoning in large language models with one training example." arXiv preprint arXiv:2504.20571 (2025).

[2] Havrilla, Alex, et al. "Teaching large language models to reason with reinforcement learning." arXiv preprint arXiv:2403.04642 (2024).

W3: RAST lacks a formal theoretical analysis.

Our work is primarily application-driven, aiming to demonstrate a practical and effective method for transferring reasoning behaviors across model scales. That being said, while a formal theoretical analysis would certainly enrich the understanding of the underlying mechanisms, we believe it is not a prerequisite for demonstrating the utility and novelty of our approach in this setting.

Additionally, we actually already provided a bunch of analyses in the paper, trying to offer insights on “why the logit adjustments from smaller RL-trained models can effectively activate reasoning”. The analyses include PCR (Section 2.1), transferability signal (Section 4.1), and token-level behavior shift (Section 4.2).

W4: The performance of RAST is highly dependent on the quality of the small RL-trained models.

We believe this is not a weakness or limitation, but rather an expected and inherent feature — similar to any machine learning system, the quality of the model’s output definitely is affected by the input to the system.

RAST is designed to be modular and model-agnostic: it can leverage a reasonably good small model that encodes helpful reasoning behaviors, without requiring changes to the large model or costly joint training. This flexibility actually enhances RAST’s applicability, as users can plug in different small experts tailored to their domain or task without retraining the large model.

In our experiments, we show that small RL-trained models across different scales can provide meaningful guidance to much larger base models, resulting in consistent performance improvements across diverse reasoning benchmarks. This highlights the robustness of the transfer/activation mechanism.

W5: Lack a detailed analysis of the trade-offs between the computational cost of training the small expert models and the benefits gained.

We actually already have such an analysis in Section 4.4. Results in Table 4 show that with only 50% training resources, RAST can recover more than 80% performance of the trained models. In fact, the efficiency advantage is exactly one of our major claims made in the paper.

We hope the above responses address your concerns. We are also happy to engage in discussions with you if the reviews could be more constructive and detailed.

To reviewer zUV7:

• Please read all reviews and author responses, then post a brief reply as early as possible so there is time for meaningful back-and-forth.
• Join the discussion before clicking “Mandatory Acknowledgement.” Clicking prematurely does not waive this requirement.
• Reviews without discussion may be flagged Insufficient Review under the Responsible Reviewing rules.

Thank you for helping to uphold the highest standards of the conference.

AC

Thank you for your response and for taking the time to review our paper.

We would like to respectfully clarify a point regarding your comment:

RAST lacks a formal theoretical analysis, which does not demonstrate the novelty of the idea.

We believe that the presence or absence of theoretical analysis is not inherently tied to the novelty of a contribution. Novelty is typically assessed based on the originality and distinctiveness of the proposed idea or method, rather than the specific type of analysis presented.

---

1. Novelty of RAST. RAST introduces a novel inference-time framework that transfers RL-induced reasoning behaviors from small to large models via logit residuals. This approach avoids the need for costly RL training on large models while achieving comparable or better performance. Our method is model-size agnostic, works across architectures, and frames reasoning activation as an inference-time steering problem. To our knowledge, this perspective and approach has not been addressed in prior work. We have also empirically validated our fundamental hypothesis through extensive experiments.

If there are specific prior works that you believe closely resemble RAST, we would greatly appreciate it if you could point them out. This would help us better understand your concerns regarding novelty and address them directly.

---

2. RAST Is an Empirical Paper. The NeurIPS Reviewer Guidelines state that claims can be supported by either theoretical analysis or experimental results. Given that RAST is an application-focused contribution, we have provided comprehensive empirical results and in-depth analyses to support our claims. We believe this is consistent with the expectations for submissions in the Applications track.

---

In light of the above, we kindly ask if you could reconsider your rating, or alternatively, provide more specific and constructive feedback regarding the aspects of novelty or analysis that you find lacking. Such input would be invaluable for us to further improve the paper. If there are any other concerns that you feel remain unaddressed, please let us know so we can clarify or provide additional information.

Thank you again for your time and consideration.

Thanks for the constructive comments. We sincerely appreciate your recognition of RAST's clarity, novelty, and core insights, as well as the empirical validation of its effectiveness and scalability across models without additional tuning. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have:

W1 & Q2: The lack of ablations isolating the specific reasoning behaviors being transferred, and the importance of determining which reasoning behaviors are important.

Thanks for the thoughtful comment. We agree that demonstrating specific reasoning behaviors being transferred is an important aspect of our claim. Therefore, we conducted such an analysis in Appendix C.3. In this analysis, we first manually curate a set of tokens that reflects certain reasoning behaviors as shown in Figure 9. We then statistically test the frequencies of those key tokens as an approximation of appearances for reasoning behaviors. The results are shown in Figure 10. We can see that the reasoning behaviors of RAST are much closer to the corresponding large RL model. We believe that this keyword-based quantitative analysis provides concrete evidence beyond accuracy/performance, indicating that RAST is really steering the base model towards improving real reasoning skills.

We agree with Q2 on the importance of determining important reasoning behaviors. However, this question lies somewhat beyond the scope of our current work, which aims at developing a method for reasoning activation to reduce the costly RL training effort. Determining different reasoning behaviors remains a valuable future direction for follow-up works, as discussed in lines 750-755. As an initial trial, our analyses in lines 694-700 reveal the correlation between certain reasoning behaviors with specific datasets. For example, AIME 2024 exhibits a pronounced amount of "branching out". This indicates that the AIME dataset needs careful traversing and more sophisticated inductive reasoning capabilities.

W2: The use of MATH500 in both training and evaluation raises concerns about whether improvements reflect true reasoning transfer.

Thanks for the comment. We respectfully think there is a misunderstanding. We would like to clarify that MATH 500 is not incorporated in the reward data for the small RL model training in SimpleRL. In fact, according to Section 2.1 of the Training dataset in [1], the SimpleRL authors used MATH as the training data, which differs from MATH500. That being said, the training and test datasets do not overlap. So the problem of “re-activating memorized solutions” does not apply in this case.

Despite this, our evaluation is not limited to MATH. Specifically, we evaluate RAST across 5 other diverse mathematical reasoning benchmarks—including GSM8K, AIME 24, AMC23, Minerva, and Olympiad—ensuring that our conclusions are based on a comprehensive and diverse set of tasks, rather than bias or coincidence arising from any single dataset.

W3 & Q1: Comparison and integration of RAST with instruction-tuned models.

Thanks for your constructive comment. We only incorporate base models in our setup because:

• It is common practice in the literature of RLVR [2] to evaluate methods on base models (explained as zero-RL in line 67), and
• The small RL models we used are trained from a base model directly. We intentionally avoid introducing additional instruction-tuned priors to ensure a clean experiment setting for examination of RL-trained models only.

To address your concerns, we additionally conduct experiments on direct inference with instruction-tuned models as a comparison. The results are shown in the following table (we use the instruction-tuned version of Qwen2.5 series):

Together with Table 1 in the paper, we can see that direct inference with instruction-tuned models does not match the performance of RAST, especially for difficult datasets such as AIME and Minerva. This experiment further strengthens the effectiveness of RAST.

We also conducted statistical analysis on reasoning token behaviors and compared the normalized token frequencies between the instruction-tuned model and RAST following Appendix C.3:

We found that RAST is better than an instruction-tuned model in terms of reasoning behavior activation.

W4: Path Coverage Rate is measured on limited, closely related models.

Thanks for your insightful comment. We conducted experiments entailing more datasets, including math and general reasoning (AIME, GPQA, MMLU), and an additional model family (Llama 3.1-8B) for PCR calculation. The results are shown in the following table:

We can see that our conclusion still holds, with over 90 PCRs across all the different model families and different benchmark data points. The results prove the generalization of RAST. We will add these results to Section 2.1 in the newer version of our paper.

Q3: Figure 3 interpretation.

The gray bar in Figure 3 represents the performance by RAST, where the red line indicates the performance of larger RL models. We can see that in all benchmarks except AIME 24, RAST (gray bar) is always above the RL models (red line). On AIME 24, when k grows to 32, RAST chases the performance of larger models.

[1] Zeng, Weihao, et al. "Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild." arXiv preprint arXiv:2503.18892 (2025).

[2] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).