We sincerely thank Reviewer gjwQ for the constructive feedback. We appreciate your valuable recognition of the great significance and thorough experiments of our approach.

[W1, Q2] Typo corrections

W1: In line 24, I believe "hundreds of parameters" is not correct, as you are saying "LLMs with hundreds of parameters (line 124) and SLMs with "few billions of parameters (line 26)"

We apologize for the typos. To clarify, in line124, we wish to say "LLMs with hundred of billions of parameters".

Q2: Can you please clarify what you mean in lines 170-171? I assumed you would rely on the smaller model for generating the majority of the tokens. Why is it said that only 3% of generated tokens are from the smaller model?

In line 171, the correct statement should read: "...while relying on the larger Rl-32B model for only 3% of the generated tokens."

We genuinely thank the reviewer for the careful review. We will carefully proofread the paper to eliminate typos in our writing.

[W2.1, L1] Assumptions on greedy decoding

W2.1: Some assumptions, such as "greedy" and "sentence-level" path-following, might be limiting in some cases and for some tasks, but I understand that it may have been necessary to have such assumptions due to the huge search space.

L1: Also, the greedy decoding may potentially be problematic for tasks where we want to generate more diverse or creative text.

We thank the reviewer for this insightful question. Below, we provide additional analysis and preliminary experiments on how R2R generalizes to sampling-based decoding strategies.

Path-Following Routing Under Sampling

R2R naturally extends to sampling-based decoding by adapting divergence labeling to a probabilistic setting. Specifically, the deterministic continuation sequence pair $(S_s, S_l)$ from Equations 3–5 are replaced by $k$ sampled continuation pairs under a chosen sampling method. Divergence is then evaluated for each sampled pair. The routing decision for each token then depends on whether the overall divergence probability exceeds a predefined threshold:

$$P(V(S_{s_i}, S_{l_i}) = 0|i \in [0,k)) \geq P_{\text{threshold}}$$

If setting $P_{\text{threshold}}$ to the LLM's self-divergence probability $P(V(S_l, S_l)=0)$, it ensures the mixed model, under full path-following routing, maintains the same quality expectation as the LLM alone. We will provide detailed descriptions and theoretical analyses of this method in the final manuscript.

Efficient Data Generation Pipeline

Sampling multiple continuations for divergence labeling is computationally expensive. However, since continuations are sentence-level, we empirically observe that the divergence decision primarily depends on the first differing token between the SLM and LLM samples, rather than the subsequent continuations. To efficiently approximate probabilistic routing, we therefore only sample for next-token generation $y_i(\theta_s|S_{<i})$ and $y_i(\theta_l|S_{<i})$, while leaving continuations deterministic.

Given limited time for rebuttal and the extensive volume of tokens, we set $k=1$ and $P_{\text{threshold}}=0.5$, simplifying the setup. This approximation incurs overhead comparable to our greedy data generation pipeline.

In implementation, our data-generation pipeline adapts to the new setup by using sampling-based generation for both LLM responses (step 0) and SLM prefill (step 1), while keeping other procedures unchanged.

Experiment Results

We adopt DeepSeek-R1’s recommended sampling settings (temperature=0.6, top-p=0.95) for preliminary experiments. We retain the neural router architecture, as the router already accepts embedding of the sampled token.

Under sampling-based decoding, R2R remains Pareto optimal compared to distilled models and query-level routing methods.

Divergence distribution analysis

The average divergence rate under sampling slightly increases (+3.1%) but remains low overall, demonstrating consistent divergence patterns with greedy decoding.

[W2.2, L2] Assumptions on sentence-level path-following routing

W2.2: Some assumptions, such as "greedy" and "sentence-level" path-following, might be limiting in some cases and for some tasks, but I understand that it may have been necessary to have such assumptions due to the huge search space.

L2: For example, the SLM might want to generate the sentences in a paragraph in a different order but still follow a valid reasoning path and achieve the same conclusion as the LLM in some cases

We appreciate the reviewer's thoughtful comment. While mislabeling neutral continuations as divergent typically does not degrade performance, it may occasionally introduce unnecessary computational overhead due to additional LLM activations.

To evaluate mislabeling caused by insufficient continuation, we extend our routing strategy to N-sentence path-following. As N gradually increases from 1 to the total remaining sentences in the LLM-generated response, it transitions smoothly from sentence-level to fully path-following routing.

In experiments on the subset of AIME dataset, we observe that increasing N from 1 to 5 only reduces the divergence rate modestly from 2.91% to 2.66%, corresponding to an average activated parameter reduction of 0.08B. Given this limited improvement, we empirically select N=1 (sentence-level) as our default setting.

[Q1] Combining R2R and query-level routing

What are your thoughts on combining R2R with some query-level routing methods so that you can choose which SLM or LLM to use based on the difficulty of the query, and then using R2R to route tokens to the SLM or LLM for the generation?

Thank you for the insightful suggestion. We investigated combining R2R with query-level routing methods using the Qwen3 model series with varied sizes, evaluated on the AIME benchmark. Specifically, we treated R2R-S (0.6B+8B) and R2R-L (0.6B+32B) as the SLM and LLM, respectively, for query-level routing.

From these experiments, we conclude:

1. R2R models generally serve as superior SLM/LLM candidates for query-level routing compared to original single-model setups, consistently advancing the Pareto frontier.

2. The interaction between medium-sized R2R models and query-level routing involving smaller or larger R2R-LLMs is empirically non-trivial. This observation suggests exciting opportunities for future research into sophisticated routing strategies, which combine the search space of query and token-level routing methods.

We will incorporate these discussions into the revised manuscript, highlighting insights and promising directions for future exploration.

We sincerely thank Reviewer uytF for the valuable feedback. We appreciate your great recognition of the significance, novelty, and strong experimental results of our approach.

[W1, Q2] Verifier depends on LLM, potentially mislabels divergent and neutral tokens.

W1: The data labeling pipeline depends on a separate LLM-based verifier, which may introduce a bias or circular dependency.

Q2: What guarantees does the LLM verifier provide, and could it mislabel semantically divergent tokens as neutral?

To evaluate reliability in classifying divergences and neutral differences, we conducted additional experiments comparing the different LLM verifiers with human experts. We also exame model robustness to mislabeling.

Verifier Capacity and Verification Quality

We constructed ground-truth divergent labels with three independent human expert verifiers [1]. The general divergent tokens are labeled by majority voting among expert answers, taking up 24.8% of differences. The core divergent tokens, a subset of highly divergent tokens with more strict criteria, are labeled with unanimous consent of experts, which takes up 11.9% of differences. he table below compares the chosen verifier (Qwen2.5-72B) against a fourth independent expert. Our verifier achieves a comparable performance over human expert, with high recall on core divergence.

Note that the verifier judges semantic differences between two continuations, which is notably simpler than generating correct responses or evaluating the entire reasoning path. Consequently, even a smaller verifier (Qwen2.5-7B) provides satisfactory performance. However, significantly smaller models (e.g., 3B) may yield less ideal results.

Verification Quality and Response Quality

We evaluated how verifier accuracy impacts response quality using sentence-level path-following routing (Equations 3–5) on AIME questions. It evaluates the influence of verifier on final response quality, without incurring the neural router.

As shown below, even a verifier with moderate recall (e.g., 66% for Qwen2.5-3B) can guide better reasoning paths than the baseline R1-7B model.

Sentence-level path-following routing also connects the verifier prompt to final response quality, enabling direct performance tuning via prompt engineering. The current verifier prompt is optimized for Qwen2.5-72B. Empirically, smaller verifiers can benefit from prompts biased towards classifying ambiguous cases as divergent, enhancing recall. We will include detailed prompt-tuning discussions and extended results in the final manuscript.

Quality-Guaranteed Verifier

We also introduce the quality-guaranteed full path-following routing (L151-158, proof in Appendix D). For scenarios with shorter reasoning paths and sufficient computational resources, it explicitly ensures the final response quality with generated labels.

[1] We recruited four undergraduate engineering students as annotators. They receive identical contexts and instructions as the verifier LLM, covering 1357 different tokens between R1-1.5B to R1-32B on six AIME-2024 questions.

[W2, Q4] Generalizability across domains and tasks

W2: The training data (math, QA, and code) may not generalize well to open-ended tasks.

Q4: Could R2R apply to generative tasks like summarization or dialogue, where path divergence may not always be objectively verifiable?

We appreciate the reviewer’s comments on R2R’s generalizability.

Generalizable Data Labeling Method

Our labeling method, relying on semantic comparison rather than formal verification, easily generalizes across tasks. Unlike methods requiring task-specific external tools (e.g., formal proofs [2], code execution [3]), R2R uses an LLM verifier to identify general semantic divergence in meaning, reasoning, logic, or conclusions (Appendix E.1).

In practice, we apply the identical labeling strategy and verifier prompt across closed-form math, coding tasks, and open-ended QA tasks. This consistency enables our router to naturally generalize to unseen tasks during inference, as demonstrated later.

For tasks with subjective divergence criteria that are challenging to identify semantically within a single sentence, the continuation length can be extended. This adjustment gradually transitions from sentence-level routing to full routing (L151-158), the latter of which only requires overall response quality evaluation—a feasible requirement commonly met for LLM tasks.

Generalizable Router

Our router leverages outputs from the SLM (e.g., logits) to predict token divergence. Benefiting from the inherent generalizability of SLMs, indicators such as logits entropy robustly identify divergent tokens across different tasks.

To validate the router’s generalizability, we directly apply our router, trained for math, QA, and code, to additional benchmarks: Arena-Hard for Dialog [4], and MMLU-Redux-Philosophy.

R2R continues to outperform the 14B model with less than 7B parameter size.

[2] X, Huajian, et al. "Deepseek-prover-v1.5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search." arXiv 2024.

[3] D. Jung, et al. “Code execution as grounded supervision for LLM reasoning.” arXiv 2025

[4] We evaluate the first 100 questions in the benchmark to expedite testing.

[W3, Q1] Sampling method extension

W3: The method is only validated under greedy decoding. It remains unclear how well R2R generalizes to sampling-based strategies

Q1: How does R2R perform under temperature sampling or nucleus sampling? Would the divergence pattern be consistent?

We provide additional analysis and experiments on sampling-based decoding.

Path-Following Routing Under Sampling

R2R naturally extends to sampling-based decoding by adapting divergence labeling to a probabilistic setting. Specifically, the deterministic continuation sequence pair $(S_s, S_l)$ from Equations 3–5 are replaced by $k$ sampled continuation pairs under a chosen sampling method. Divergence is then evaluated for each sampled pair. The routing decision for each token then depends on whether the overall divergence probability exceeds a predefined threshold:

$$P(V(S_{s_i}, S_{l_i}) = 0|i \in [0,k)) \geq P_{\text{threshold}}$$

If setting $P_{\text{threshold}}$ to the LLM's self-divergence probability $P(V(S_l, S_l)=0)$, it ensures the mixed model, under full path-following routing, maintains the same quality expectation as the LLM alone. We will provide detailed descriptions and theoretical analyses of this method in the final manuscript.

Efficient Data Generation

Sampling multiple continuations for divergence labeling is computationally expensive. However, since continuations are sentence-level, we empirically observe that the divergence decision primarily depends on the first differing token between the SLM and LLM samples, rather than the subsequent continuations. To efficiently approximate probabilistic routing, we therefore only sample for next-token generation $y_i(\theta_s|S_{<i})$ and $y_i(\theta_l|S_{<i})$, while leaving continuations deterministic.

Given limited time for rebuttal and the extensive volume of tokens, we set $k=1$ and $P_{\text{threshold}}=0.5$, simplifying the setup. It incurs overhead comparable to our greedy data generation pipeline.

In implementation, our data-generation pipeline adapts to the new setup by using sampling-based generation for both LLM responses (step 0) and SLM prefill (step 1), while keeping other procedures unchanged.

Experiment Results

We adopt DeepSeek-R1’s recommended sampling settings (temperature=0.6, top-p=0.95) for preliminary experiments. We retain the neural router architecture, as the router already accepts embedding of the sampled token.

Under sampling-based decoding, R2R remains Pareto optimal compared to distilled models and query-level routing methods.

Divergence Distribution Analysis

The average divergence rate under sampling slightly increases (+3.1%) but remains low overall, demonstrating consistent divergence patterns with greedy decoding.

[Q3]: Batch routing decisions

Have you considered batch routing decisions (e.g., look-ahead or segment-level) instead of per-token routing?

Although R2R currently routes at the token-level, it can extend to coarser granularities such as segments: we simply aggregate token‑level signals within a segment and issue a single routing decision. This opens up new trade-offs between routing granularity, promptness, and accuracy. We consider this a promising direction for future work.

We sincerely thank Reviewer PS9S for the extensive feedback. We appreciate your valuable recognition of the intuition, quality and impressive empirical results of our approach.

[W1] Discussion on concurrent work (SplitReason)

I understand that [3] is concurrent. But now that the authors mention [3] in Line 227, I think it would be better to discuss properly in the section on related work.

Thank you for the great suggestion. We will include a proper discussion of the concurrent work SplitReason in our revised manuscript. As the SplitReason repository is under maintenance during the rebuttal period, we plan to include the experimental comparison in the final version.

Both R2R and SplitReason explore mixed inference strategies combining small and large language models, but differ in routing granularity and objectives.

SplitReason aims to offload difficult reasoning steps to the LLM. Specifically, it first uses a strong LLM to identify challenging reasoning steps, then trains the SLM to generate a special token (i.e., <bigmodel>) signaling the LLM to take over these difficult steps.

In contrast, R2R addresses token-level divergences rather than step-level difficulty. Different from SplitReason, R2R also targets the subtle scenario where the SLM and LLM may agree on challenging steps, yet diverge unexpectedly on seemingly straightforward tokens. Such divergences can significantly alter the subsequent reasoning path, thus requiring immediate correction.

We believe the two complementary perspectives would offer comprehensive contexts for efficient reasoning, inspiring even more effective mixed inference future works.

[W2] Insufficient discussion of ablation study

I think the ablation study on the routing objective is not sufficiently discussed.

Thank you for the thoughtful questions. We clarify as follows:

I suppose delegate "more" tokens to LLM will not hurt the performance this much.

1. More LLM tokens do not commonly hurt AIME performance.

   Indeed, all 7 AIME questions correctly answered by the SLM are also correctly answered by the LLM, confirming that the LLM consistently outperforms the SLM rather than specializing in different types of math questions.

Is it because the objective change causes the router misses more divergent tokens?

2. Yes, the different objective leads to more missed divergent tokens.

   To investigate this, we evaluate the divergent token recall of different routers. We use ground-truth responses and divergence labels generated with our data pipeline on AIME. Routers trained with different and divergent objectives are marked at Router1 and Router2, respectively.

As demonstrated in the table, the router trained with the different objective shows lower recall, even at higher LLM usage. This is because it wastes LLM calls on neutral differences rather than targeting truly divergent tokens.

Moreover, it is not clear what the authors mean by "it fails to reach the original accuracy within the same amount of LLM usage, ..." Does it imply that the authors manually control the activation of the LLM in this experiment?

3. Yes, we manually adjust the routing threshold to control LLM activation.

   Since R2R allows for flexible trade-offs between LLM usage and performance, each router defines a trade-off curve. For ablation comparison, we adjust thresholds to match or slightly exceed the LLM rate of our main router, and then compare their accuracies. Despite higher LLM usage, the routers with objective or reduced inputs fail to reach comparable accuracy.

We will revise the manuscript to improve the clarity of this ablation discussion.

[Q1] Including MoE in the context

I wonder if MoE can be included in the context here. In my opinion, MoE is another way of utilizing a model with larger capacity. But the difference is that MoE implicitly uses the whole capacity with fixed cost at each token. In contrast, R2R uses small capacity model with low costs most of time but escalate to larger model when needed. It would be great if we can find a way to properly compare the two. I understand that this can be too demanding for the review period. So this is not a request but an open question out of curiosity.

Thank you for raising this interesting perspective. Below, we first present a conceptual comparison between MoE and R2R, followed by empirical comparisons, and finally discuss how the two approaches can complement each other to further improve the efficiency-accuracy trade-off.

Conceptual Comparison

As noted by the reviewer, both MoE and R2R leverage partial activation of larger capacity models, but with distinct granularities, subject sizes, training overhead, supervision and complementary insights. This conceptual distinction motivates their integration, which we explore later.

Empirical Comparison

We evaluate the MoE model Qwen3-30BA3B and the R2R model (combining Qwen3-0.6B and Qwen3-8B) on the AIME benchmark. The 60 questions are split into Easy (28) and Hard (32) subsets based on whether the question ID $\leq$ 7.

These results highlight the strengths of each approach. Thanks to extensive and sophisticated pretraining, state-of-the-art MoE models can match dense model performance (e.g., both Qwen3-30BA3B and Qwen3-32B achieve 75% accuracy on AIME in our evaluation). However, their fixed per-token overhead limits efficiency for simpler tasks.

In contrast, R2R dynamically adjusts computation based on token-level divergence. It achieves lower cost on easier examples by relying more on the SLM, but its small router and much lighter training cannot match MoE performance on harder questions with the same activated parameters.

Given these complementary advantages, we explore the combination of both methods.

Combining MoE and R2R

MoE for R2R

R2R is directly compatible with efficiency-optimized models such as MoE models. We validate R2R performance using the Qwen3-30BA3B MoE as the LLM:

Combining R2R with MoE models significantly improves the Pareto frontier, achieving robust accuracy even at extremely low activated parameters per token.

R2R for MoE

R2R insights can also enhance MoE model designs. Current MoE methods generally use uniform-sized experts. Introducing mixed-sized experts like R2R could enable query-adaptive expert activation. Moreover, supplementing MoE's token prediction objective with explicit divergence or difficulty-based routing supervision from R2R may encourage adaptive usage of larger experts only for critical decoding steps. Although this approach would require retraining MoE models and is beyond the scope of this paper, it opens exciting future research directions.

We will incorporate these discussions into the revised manuscript, providing insights for further exploration.

We sincerely thank Reviewer ZxyV for the extensive feedback. We appreciate your valuable recognition of the novelty, value and strong experimental results of our approach.

[W1, Q1]: Assumes greedy decoding, should extend to other sampling methods

W1: Critical limitation to greedy decoding: Most production systems use temperature-based sampling or other stochastic methods.

W2: Sampling method extension: How fundamental is the greedy decoding limitation? Could the neutral/divergent distinction be meaningfully applied to temperature-based sampling?

We summarize the formulation, data generation pipeline, and experimental results of R2R under sampling-based decoding. Due to word limit, please refer to our response to reviewer gjwQ, [W2.1,L1] for additional details.

Divergent Label and Data Generation Under Sampling

R2R naturally extends to sampling-based divergence by extending the verifier function (Equation 3) under probabilistic setup. The current token is divergent if the divergence probability (under deterministic verification) for SLM and LLM continuation samples exceeds the LLM’s intrinsic self-divergence.

Observing that divergence primarily depends on the first differing token between SLM and LLM samples (rather than subsequent continuations), our data-generation pipeline easily adapts to sampling. Specifically, it uses sampling-based generation for LLM generation (step 0) and SLM prefill (step 1), while keeping other procedures unchanged.

Experiment Results

We adopt DeepSeek-R1’s recommended sampling settings (temperature=0.6, top-p=0.95) for experiments.

Under sampling-based decoding, R2R remains Pareto optimal compared to distilled models and query-level routing methods.

[W2]: Generalizability across model families, architectures, and size pairs

All experiments use only DeepSeek R1 models. No evidence that the approach generalizes across model families, architectures.

We extend our experiments to evaluate generalizability across:

1. Model families (Qwen3 in addition to DeepSeek-R1)

2. Architectures (dense vs. MoE)

3. Size pairs (including 0.6B, 1.7B, 8B, 30B-A-3B, and 32B)

We report accuracy on the AIME benchmark using the same setup as Table 2.

These results demonstrate that R2R generalizes well across model families, architectures, and various size pairs. Notably, R2R constantly outperforms distilled models and query-level routing baselines, while activating fewer parameters ($\leq$ 3.7B on average).

We will extend Figure 5 in the revised manuscript to include the full cost–accuracy trade-offs across these settings, allowing readers to better visualize how R2R's Pareto front shifts under different model combinations and routing thresholds.

[W3, Q4]: Systematic failure analysis

W3: Limited discussion of systematic failure modes or scenarios where neutral/divergent distinction breaks down.

Q4: Can you identify task types or reasoning patterns where the approach consistently fails? Does performance degrade on highly creative or non-linear reasoning tasks?

To identify systematic failure modes, we analyzed the response status of R2R on AIME and GPQA benchmarks. As shown below, the primary failure mode occurs when R2R cannot complete reasoning within the 32K token limit. This typically happens due to repetitive reasoning patterns not encountered during router training. Additionally, R2R tends to invoke the LLM slightly more frequently on the GPQA benchmark, particularly for queries involving rare tokens such as complex protein or chemical compound names.

We will include a more detailed analysis of these failure cases in the final manuscript.

[Q2] Cross-model generalization of the router

Would a router trained on DeepSeek R1-1.5B/32B work with other model pairs? How much retraining would be required?

We wish to clarify that the R2R router is designed to be trained per SLM-LLM pair, requiring about 448 GPU hours for initial data generation and 2 GPU hours for router training. This overhead remains significantly lower than the distillation processes commonly employed by SOTA models. The cross-model generalizability under this setup is evaluated in [W2].

Nevertheless, our empirical results indicate acceptable performance when directly substituting the LLM of a trained router without additional retraining. This can be attributed to the common divergence patterns shared by strong LLMs paired with the same weaker SLM.

[Q3] Verifier robustness analysis

How sensitive is performance to verifier quality? What happens with weaker verifier models? Could you provide human evaluation of neutral/divergent distinctions to validate the verifier's reliability?

Human Evaluation

We conducted a human evaluation to validate the verifier's reliability. Specifically, we recruited four undergraduate students to independently label 1,357 differences as either neutral or divergent. The differences are between R1-1.5B and 32B, on the first six AIME-2024 questions. Three annotators' labels determined ground-truth divergence by unanimous consent (core divergence) and majority voting (general divergence). The fourth annotator's labels were compared with ground truth. Annotators and LLM verifiers received identical contexts and instructions. Our selected verifier (Qwen2.5-72B) closely matches human expert performance:

Verification Quality

To examine sensitivity to verifier accuracy, we applied our sentence-level path-following routing method (Equations 3–5) across verifiers. Our analysis reveals that even moderate-recall verifiers (e.g., Qwen2.5-3B with 66% recall) still significantly improve reasoning path guidance compared to the baseline (R1-7B).

Note that the current verifier prompt is tuned for Qwen2.5-72B verifier. Empirically, smaller verifiers should benefit from prompts biased towards classifying ambiguous cases as divergent, enhancing recall. We will include detailed prompt-tuning procedures and extended results in the final manuscript.

[Q5] Real-world deployment considerations

How would this integrate with existing serving infrastructure? What are the latency implications of token-level routing decisions?

We integrated R2R into the production-level SGLang serving infrastructure, as evidenced by the high absolute throughput results of our method and baselines.

To assess latency implications of R2R, we performed a detailed runtime breakdown of R2R using two A800 GPUs on the AIME benchmark, averaged over 60 questions:

This indicates that token-level routing decisions introduce minimal latency overhead. Additionally, given the small size of the neural router (56MB), we anticipate further runtime improvements through additional system-level optimizations.

[L1] Discuss computational overhead

The authors should also discuss the computational overhead more transparently in the context of alternative approaches.

Following prior work [1, 2], we analyze both FLOPs and memory access overhead on the AIME benchmark across different approaches.

Given the memory-bound nature of decoding, memory access significantly impacts throughput. R2R notably reduces memory access compared to R1-32B with only a modest increase in computation, primarily due to additional KV-cache updates from switching between the LLM and SLM. In contrast, speculative decoding approaches (e.g., Eagle2, HASS) incur substantially higher computational costs from their tree-structured draft and verification processes (60 tokens per cycle). Despite this overhead, they achieve significant throughput improvements through reduced memory access during decoding.

[1] Hoffmann, Jordan, et al. "Training compute-optimal large language models." NeurIPS. 2022.

[2] Yang, Fan, et al. "GLITCHES: GPU-FPGA LLM Inference Through a Collaborative Heterogeneous System." HPEC. 2024.