Scaling Speculative Decoding with Lookahead Reasoning

Thank you for your thoughtful review and the encouraging feedback. Below, we address your questions in detail.

Q1: SpecReason uses QwQ-32B models as the base model and reported a much lower accuracy on AIME for the original model and using reasoning model scoring actually improved model accuracy for them. Is it possible to get some results (i.e. Table 2) using the same model with different verifiers so that the comparison is straight-forward?

A1: For a direct comparison with SpecReason, we evaluated the QwQ-32B model with the deepseek-r1-distill-1.5B draft model, as used in their paper. The results below show that verifiers grounded in semantic equivalence are most effective at preserving the model's original accuracy.

Performance of QwQ-32B with Various Verifiers
Each cell shows: Accuracy% (Acceptance Rate %), Accuracy values are averaged over 16 runs.

Our LLM-as-a-Judge (LLM-J) verifier and the Embedding model with a high similarity threshold (0.95) both successfully maintained the baseline accuracy on the challenging AIME dataset (77.5% and 77.7% vs. 77.1% baseline). While effective, the Embedding model's lower acceptance rate may lead to lower speedups.

In contrast, the score-based verifier used by SpecReason consistently underperformed. While the original SpecReason paper reported an accuracy gain, they used a shortened context (<8k tokens). Our experiments, using a full 32k context, reveal a significant accuracy drop on AIME to as low as 66.9%. This highlights a fundamental limitation: assessing draft "quality" via a score from the original reasoning model does not guarantee alignment with that model’s true output distribution, which is critical for maintaining correctness on complex reasoning tasks.

Q2 & Q3: Can you report the speed-up of SpecReason and SpecReason+SD? Can you report the latency (or end to end speed-up) of using different verifiers?

A2 & A3: Thank you for these important questions. To analyze the accuracy-efficiency trade-off, we report the end-to-end inference speedups for various verifiers on QwQ-32B model with the deepseek-r1-distill-1.5B draft model. These experiments were conducted on an H200 server (with TP=2 for the 32B verifier) and include results with n-gram speculation to show orthogonality. We will add this discussion to our revised manuscript.

Speedup Comparison Across Different Verifiers
Values show speedup ratio (X times faster)

The results reveal key trade-offs between verifier latency, acceptance rate, and final speedup. For instance, the 7B LLM-J achieves greater overall acceleration than both the heavier 32B LLM-J and the more lightweight Embedding model. This is particularly interesting given their isolated per-step latencies: the Embedding model is fastest (0.009s), followed by the 7B LLM-J (0.013s), and the 32B LLM-J (0.025s). We hypothesize that the final speedup is a function of both low latency and high acceptance rate, and the 7B verifier strikes the most effective balance between these two factors.

We analyzed the performance of the SpecReason verifier (Score 0.7). While it achieves a significant 1.57x speedup on the short-context GSM8K dataset (avg. output ~1K tokens), we observed an overall slowdown (0.77x) on the long-context AIME dataset. Upon analyzing the generation traces, we found that SpecReason actually causes a deceleration on individual examples where the output length grows very long (often >10K tokens). We hypothesize this slowdown stems from a lack of system optimizations in SpecReason, where per-step costs like tokenizer overhead become amplified for these long sequences. This finding is consistent with the original SpecReason paper, where experiments on shorter contexts (<8k tokens) would not have revealed this overhead. We also note that for practical applications of large reasoning models, a context limit of 8k tokens can be overly restrictive. We will include these new results and this discussion in our revised manuscript.

Finally, the +n-gram results consistently show a substantial boost across all methods, confirming that both our lookahead reasoning and SpecReason are orthogonal to token-level speculation

Q4: Auto-regressive calls to the base model: If I understand the paper correctly, this work still involves auto-regressively generating from the large base model. This means that there is opportunity to gain further speed-up if we are able to cut down the amount of auto-regressive generation from the target model.

A4: Our framework is designed to reduce sequential autoregressive generation by introducing parallelism at two different levels.

We thank the reviewer for this insightful question and agree that minimizing sequential autoregressive calls is the primary path to achieving significant speedups. Our Lookahead Reasoning framework is designed to do precisely that by fundamentally reducing the amount of step-by-step generation.

At the step level, the core innovation of our approach is to verify multiple independent draft steps in parallel. Instead of generating one token at a time, it processes several candidate steps simultaneously by using them as parallel prefixes (Figure 1a/line 49-60 shows this process). In addition to this coarse-grained parallelism, our framework also can work together with the token level speculative decoding. As demonstrated in our LR+SD experiments, any remaining sequential generation is further accelerated by n-gram speculative decoding. This combination effectively tackles the autoregressive bottleneck at both a coarse (step) and fine (token) granularity.

Furthermore, the reviewer raises an excellent point about exploring other opportunities to reduce generation costs. For example, we can shorten the output length to cut down the amount of auto-regressive generation. We view such potential methods as complementary, not mutually exclusive. Our LR framework is designed to accelerate the generation of any sequence of tokens. If another technique could produce a more concise intermediate output, our method would still be orthogonal and could be applied to accelerate the generation of that compressed sequence. We believe this synergy with other potential optimizations is a key strength of our approach.

Q5: The reviewer understands that the method proposed is orthogonal to token level speculation. The token level speculation method (prompt look-up decoding) used in the experiment is far from SOTA. While not discrediting this work, it would be nice if we can see more recent speculative decoding methods used together with LR and their combined effect on efficiency.

A5: We thank the reviewer for the point regarding our choice of token-level speculation. Our decision was guided by the practical observation that some state-of-the-art methods, such as EAGLE, do not yet seamlessly support today's large reasoning models, especially in long-context scenarios. For instance, the evaluation code in EAGLE's official repository does not readily scale up to 10K tokens for reasoning models. In our preliminary experiments, we also observed a significant degradation in speedups when attempting to apply these methods, making them unsuitable for our study.

However, n-gram speculation is a highly relevant baseline. Firstly, it offers a compelling balance of performance and practicality. The speedup achieved by n-gram is substantial, while its implementation is significantly simpler and more robust for integration. This practicality is underscored by its explicit support in leading inference engines like vLLM, which often prioritize stable, easy-to-integrate solutions over the system-level complexities introduced by draft-model-based approaches. To illustrate its effectiveness, we present a speedup comparison on H200 GPUs, using Qwen3-32B as the target model, Qwen3-1.7B model as the draft, and a max output length of 32K tokens (see table below).

Speedup comparison of different speculation methods on H200 GPUs using Qwen3-32B

As the table shows, n-gram can outperform a standard LLM draft model, particularly in challenging scenarios where draft model performance degrades. More critically for our paper, this allows us to cleanly demonstrate our core thesis: Lookahead Reasoning is orthogonal to and combines effectively with token-level speculation. The cumulative acceleration shown in the n-gram + LR results directly proves this synergy, with consistent improvements across all three datasets.

We appreciate your time and suggestions in reviewing our work. We take this opportunity to clarify our key contributions and address the concerns raised.

Q1: I think in 3.2, the expectation is derived from the connection to a geometric distribution. Could be good to make this connection concrete.

A1: Thank you for the suggestion. You're absolutely right. In the appendix, we provide a detailed derivation of the expectation. Specifically, we define a random variable X as the number of consecutive accepted drafts before the first rejection, which follows a geometric distribution. We then apply the standard expectation formula for the geometric distribution, together with the law of large numbers, to arrive at the final result for the speedup.

Q2: Over which data set did you compute Figure 2? Is it a combination over all data sets? Would it be possible to show uncertainties?

A2: Figure 2 (section 4.2) is computed on the AIME dataset. We appreciate you pointing out this omission and will clarify this in the main text. Regarding uncertainties, we will add error bars in the revised version to provide a more complete picture of the results.

Q3: Are you planning on releasing the code to reproduce results? The community would certainly benefit from such a contribution.

A3: Yes, we are committed to open source and are actively preparing the code for release. We expect to make it publicly available soon to help the community reproduce our results and build upon our work.

Q4: In 3.3, would it be possible to incorporate a dataset's complexity in terms of expected number of reasoning steps? I assume, you could show that with increasing complexity expected speedups increase.

A4: Thank you for the thoughtful suggestion. We can use the acceptance rate to describe the lookahead reasoning’s performance on a given dataset. This rate then directly informs the expected speedup. The acceptance rate is likely correlated with the dataset's complexity, so we can model the relationship between speedup and complexity through their shared connection to the acceptance rate.

Q5: line 159: explores $\rightarrow$ explored

A5: Thank you for catching this typo. We will correct "explores" to "explored" in line 159 in the revised version.

Q6: You mention and compare to n-gram speculative decoding but never formally introduce it. It would be helpful to do so.

A6: Thank you for highlighting this gap. We will clarify this in the related work section. N-gram speculative decoding is a standard, draft-model-free method, notably implemented as one of the speculative decoding strategies in the vLLM framework. This technique operates by maintaining a cache of all historical tokens. At each decoding step, it uses the suffix of the current sequence to perform a string-matching lookup within this cache. Historical n-grams that begin with this suffix are then proposed as speculative drafts. Subsequently, the verification stage proceeds identically to conventional speculative decoding, where the target model validates the entire candidate sequence in a single forward pass.

Q7: What's W for the results in Table 1?

A7: For the results in Table 1, we used W=1. We will add this clarification to the table caption to avoid confusion.

Q8: The authors mention two limitations at the end of Section 6. I wonder if you could also comment on the implementation complexity. Since LR also requires to set parameters such as $W$ and $\gamma$, please mention that as a limitation which may require tuning or some other selection procedure.

A8: Thank you for highlighting these important practical considerations. Regarding implementation complexity, full integration into production serving engines (e.g., SGLang and vLLM) requires non-trivial engineering effort, particularly for handling requests in tree hierarchy. However, implementing Lookahead Reasoning on the client side is relatively straightforward, though it may not achieve the best performance as a deeply integrated solution. We have already developed a working implementation and plan to release it as open source to facilitate further adoption and experimentation.

Besides, we will mention tuning the hyperparameters as a limitation in a revised version.

We thank the reviewer for the insightful comments! We would like to answer your questions and address your concerns in this response:

Q1: Lack of Context on Prior Work The paper repeatedly contrasts its method with “SpecReason” but never explains what SpecReason actually does or why it is a meaningful point of comparison. Without at least a concise description of SpecReason’s algorithm or performance characteristics, readers unfamiliar with that work cannot judge what the novel contribution truly is.

A1: Thank you for this critical feedback. We agree that a clear context for SpecReason is essential for readers. We have already briefly introduced it in the related work (section 5 line 343-345), but we acknowledge that a more detailed description is beneficial for comparison. We will expand on this in the paper to clarify its role as a key baseline and better frame the contributions of Lookahead Reasoning.

We chose SpecReason as a primary baseline because, as a concurrent work, it is one of the few methods that also addresses the challenge of step-level acceleration for large reasoning models, making it a highly relevant point of comparison.

However, the two approaches are founded on fundamentally different principles:

• SpecReason operates on a score-and-threshold framework. First, a draft model generates a complete reasoning step. Then, the large reasoning model is prompted to assign a numerical score to this draft. The draft is accepted only if its score surpasses a predefined threshold; otherwise, it is discarded, and the target model generates the step autoregressively.

• In contrast, Lookahead Reasoning is grounded in the principles of speculative decoding. Its acceptance criterion is not based on a subjective score but on whether the draft step match what the target model would have generated itself. This ensures statistical faithfulness to the original model, avoiding the potential brittleness of SpecReason’s approach, which is highly dependent on the target model's instruction-following capabilities and the quality of handcrafted scoring prompts. Beyond this, our approach incorporates system-level optimizations like concurrent multi-step verification and asynchronous execution to maximize hardware throughput. We will expand this context in the revised manuscript.

These methodological and architectural distinctions lead to crucial differences in the compute budget and performance outcomes. SpecReason introduces the overhead of an additional scoring call to the large target model for every proposed step. As our results demonstrate, this reliance on an external scorer can cause significant accuracy degradation. In contrast, Lookahead Reasoning's adherence to the speculative paradigm preserves the target model's accuracy while achieving substantial speedups. We will add this detailed comparison to the paper to properly contextualize our work.

Q2: Please include a concise description (algorithmic steps, compute budget, prior speedup results) so readers can gauge how Lookahead Reasoning differs and improves upon it.

A2: We will add a dedicated subsection with the following details to clearly differentiate the two methods. It is important to note that Lookahead Reasoning and SpecReason are concurrent works addressing the same challenge. Therefore, our goal is not to present LR as an improvement upon SpecReason, but to clearly distinguish their fundamental approaches and performance characteristics.

Algorithmic Steps: SpecReason's generation process follows these steps at each generation step:

• Draft Generation: A draft model generates a candidate for a complete reasoning step.
• Scoring Prompt: The large reasoning model is prompted with instructions to evaluate the generated draft and assign it a numerical quality score (e.g., on a scale of 1-10).
• Thresholding: The assigned score is compared to a predefined acceptance threshold. If the score meets the threshold (e.g., ≥ 8), the draft is accepted; otherwise, it is discarded, and the target model generates the step autoregressively.

Compute Budget: For every proposed step, it requires a full forward pass of the large target model simply to generate a score. In contrast, Lookahead Reasoning's verifier can be a much smaller model (e.g., a 7B model or a lightweight embedding model), drastically reducing verification overhead. Furthermore, our architecture incorporates system optimizations like concurrent multi-step verification and asynchronous execution to further mitigate the overhead.

Prior Speedup Results: Regarding prior results, the original SpecReason paper reported speedups of approximately 1.4x - 3.0x on various reasoning tasks. However, it is crucial to note that these results were achieved by limiting the maximum output length to 8k tokens—a constraint that can significantly compromise model accuracy on complex tasks. In our fair comparison on the GSM8K and AIME datasets (using the Qwen2-32B / DeepSeek-Distill-1.5B models from their work), we show that without this restrictive limit, the SpecReason method can lead to slowdowns and a notable drop in accuracy in more demanding, long-context scenarios like AIME.

We believe this detailed comparison will effectively contextualize our work and clarify the novelty and practical advantages of Lookahead Reasoning.

Q3: The baseline comparison is inappropriate - only comparing to SpecReason and n-gram SD. Should include LLM-based SD with same compute budget for fair comparison.

A3: Thank you for your valuable feedback regarding our baseline comparisons. We would like to clarify our rationale and present evidence suggesting that n-gram speculative decoding (SD) is not a weak baseline, but rather a strong, highly-optimized, and often more effective benchmark than LLM-driven SD in practical, system-level applications.

The core assumption that an LLM-based draft model is inherently superior is not true once system-level overhead is considered. Our own experiments directly compare the end-to-end speedup of n-gram SD against a Qwen3-1.7B parameter LLM-drafter when using Qwen3-32B as the target model. The results below clearly show that the efficient, draft-model-free n-gram approach achieves a higher effective speedup.

Speedup comparison of different speculation methods on H200 GPUs using Qwen3-32B

The reason for this outcome lies in system-level realities, a conclusion supported by other recent research that also highlights the practical inefficiencies of LLM-driven SD. The core issue is high overhead: LLM-driven speculative decoding introduces the significant cost of running a second neural network and the system complexity of managing two models. In contrast, n-gram speculation is a highly efficient, draft-model-free approach. To clarify its mechanism: this technique operates by maintaining a cache of all historical tokens. At each decoding step, it uses the suffix of the current sequence to perform a string-matching lookup within this cache. Historical n-grams that begin with this suffix are then proposed as speculative drafts. Subsequently, the verification stage proceeds identically to conventional speculative decoding, where the target model validates the entire candidate sequence in a single forward pass. This makes the draft generation cost virtually free compared to running a separate LLM.

This performance trade-off is directly reflected in the design of leading inference frameworks. For instance, the latest version of vLLM (v1.0) has explicitly removed support for external LLM drafters due to these practical challenges, while maintaining robust, optimized support for n-gram speculation. And while this latest vLLM version also supports methods like EAGLE, our preliminary tests confirmed that EAGLE's performance degrades significantly in the long-context scenarios our work focuses on. Therefore, our choice of n-gram as a baseline is a deliberate one, aligned with the design of state-of-the-art systems and empirical evidence. Therefore, we argue that n-gram SD is the most appropriate and challenging baseline for our work. It is a standard, industry-relevant method that our empirical results confirm is highly effective. By demonstrating that our hybrid Lookahead Reasoning + n-gram method outperforms both n-gram SD and the LLM-driven SD we tested, we are showing a clear improvement upon a strong and practical foundation. We will add this detailed discussion to the paper to better justify our choice of baselines.

Q4: Limited Practical Validation While the theoretical analysis is useful, its significance depends on demonstrating practical effectiveness. Otherwise, it does not stand on its own.

A4: We agree that practical validation is essential to substantiate our claims. Our paper already showed the empirical evidence that using both lookahead reasoning and speculative decoding can achieve speedups as in Section 4.1. Specifically, Table 1 demonstrates that our hybrid approach, which integrates Lookahead Reasoning with n-gram speculative decoding, achieves a speedup of up to 2.1X. This result significantly outperforms the use of a single speculative method alone, thereby confirming that our theoretical contributions translate into tangible performance gains in practical scenarios

We thank the reviewer for the insightful and helpful feedback! We would like to address your questions in the below response.

Q1.1: This paper only considers the inference speed of single sequence, while the usage of GPU resources is not well discussed. Take figure 1a as an example: there are 3 draft steps. Does this mean 3 copies of the target model are needed to implement the pipeline? It is good that this algorithm improves the ceiling of the inference speed of single sequence, but it is also critical to discuss whether the GPU consumption is increased or reduced.

A1.1: Thank you for this excellent point. A discussion on resource usage is critical, and we will add a detailed analysis to the paper to clarify this.

To answer your primary question, our method does not require multiple copies of the target model's weights; only a single instance is needed. We achieve this by leveraging the batching capabilities of modern inference engines (e.g., vLLM). The process works as follows: first, a lightweight draft model generates several candidate steps. These candidates are then grouped into a single batch and processed by the target model. This allows the target model to generate its own corresponding "ground truth" step for each candidate simultaneously. Finally, a verification check is performed between the original drafts and the target model's outputs to decide whether to accept the drafts (fast-forwarding) or fall back.

Our method uses more GPU resources to lower latency. We process many candidate steps together in a large parallel batch. This approach increases both the KV cache memory and the computational load. As a result, the GPU is utilized more effectively and does not sit idle. This trade-off is the key to our speedup: we use more of the GPU's power at each moment to finish the entire sequence faster.

Q1.2: Suppose that we have an inference system with LOOKAHEAD REASONING to process lots of user requests. Will LOOKAHEAD improve the system throughput? I agree that this algorithm can improve user experiences when there are IDLE GPUs, but it is worthy discussing how important these scenarios are.

A1.2: Regarding system throughput, we acknowledge that Lookahead Reasoning, like all speculative decoding methods, trades computation for lower latency. While this reduces per-request latency, the computation spent on inevitably failed speculations may decrease overall system throughput. This effect is scenario-dependent: when GPUs are underutilized, our method provides a speedup; when GPUs are fully saturated, it may reduce throughput. This trade-off is not unique to our work but is an inherent characteristic of all speculative algorithms designed to leverage spare compute resources. We anticipate that ongoing hardware advances will make this trade-off even more favorable.

Furthermore, we emphasize that for many online serving systems, metrics like per-request latency and user experience are often more critical than raw throughput. Our work focuses on optimizing for these latency-sensitive applications. We will add a detailed discussion on this trade-off and resource usage in the revised paper.

Q2: There are many question marks in Figure 1a. Are they formatting error?

A2: The question marks in Figure 1a are indeed a rendering issue that occurs with certain web browsers. The symbols display correctly when viewed in a standard PDF reader or most web browsers. We apologize for this formatting issue and will ensure it is corrected in the revised version.