Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding

We thank the reviewer for the insightful feedback and questions. Your suggestions and questions have helped us clarify the novelty and practical considerations of our work. Please find our detailed responses below.

1. Missing Related Works

Thank you for pointing this out. We will add a new subsection in our related works to provide a comprehensive review of self-speculative decoding methods:

1. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding, ACL, 2024.
2. Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting, NeurIPS, 2024.
3. Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding, ACL, 2024.
4. Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity, EMNLP, 2024.
5. SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration, ICLR, 2025.

We are willing to further expand this section in case we are missing important relevant works.

2. Compatibility with Self-SD Methods

This is an excellent point. SubSpec is indeed designed as a modular framework and is highly compatible with other self-speculative methods like layer-skipping. The integration could be straightforward. For example, to combine SubSpec with layer-skipping, one could simply skip a subset of the offloaded layers entirely rather than replacing them with their quantized substitutes. The feasibility of this is supported by prior work (e.g., "Draft on the Fly," "SWIFT"), which shows high acceptance rates (α≈0.98) even when skipping 35-45% of the layers.

Applying layer-skipping methods itself requires about 60% model VRAM usage, while a 4-bit substitute requires only 25%. A hybrid approach that combines layer-skipping and quantization could be even more powerful, potentially creating a draft model with a smaller VRAM footprint and faster generation speed than either method alone. We plan to investigate this direction further in our future work.

3. Proves On Token-Level Alignment

While we are unable to generate a new figure during the rebuttal period, we can provide two forms of evidence for the high alignment between our draft and target models.

Existing Evidence in the Paper. The most direct measure is the average token acceptance length (τ). As shown in Figure 2(a), SubSpec's τ is nearly twice as high as that of a small draft model of the same family, providing strong quantitative proof of superior token-level alignment.

Additional Experimental Evidence. We performed a simple experiment by computing the average cosine similarity of different draft models' output probability distributions against that of the target model, on each of the first four output tokens generated by the target model (Llama-3.1 8B, 5 samples in MT-Bench):

As shown, SubSpec maintains a much higher cosine similarity, confirming its superior token-level alignment compared to a smaller draft model of the same family.

4. Missing Standard Variation or Error Bars

This is an important consideration. Due to the significant computational cost of our experiments, running multiple seeds for every configuration was infeasible.

However, to validate the stability of our findings, we performed 5 independent runs on a key, representative result: SubSpec (8GB VRAM) on MT-Bench, obtaining a low standard deviation of 0.101 tokens/s. We will add this information and a discussion of it to the final version.

Response to Questions

Could you please elaborate on how throughput translates to speedups? Especially Tab 1 is very similar to that of Eagle-2, but Eagle-2 reported speedups. It would be helpful to include speedups in the results.

All of our experiments are conducted with a batch size of 1, as noted in Section 5 (Comparative Methodology and Parameters). Therefore, the speedup is simply the method's throughput divided by the offloading baseline's throughput. We chose to report absolute throughput primarily for transparency and fairness. We applied strong graph and kernel optimizations (torch.compile) to all methods, which significantly strengthened our baseline relative to those in other papers. Reporting only speedups against our optimized baseline could create an unfair comparison with previously published results.

Could you discuss the implementation complexity and system requirements? e.g., incorporating HQQ for quantization, and does the implementation of SubSpec pose any constraints on systems?

The implementation of our method is straightforward, but its performance is dependent on the capabilities of the underlying quantization libraries, which introduce specific system constraints.

Implementation complexity. The main implementation complexity lies in tree-based SD with shared KV-cache, which includes handling KV cache position indices correctly among draft and target models, during both decoding and verification processes. The incorporation of HQQ for quantization and asynchronous offloading only adds minimal implementation complexity.

System requirements and constraints. The primary system constraint is the need for quantization kernels with efficient GEMM support. Tree-based speculative decoding processes multiple tokens in parallel, requiring matrix-matrix (GEMM) operations. Many low-bit (especially <4 bits) quantization libraries are only optimized for single-token autoregressive generation, which only relies on matrix-vector (GEMV) operations. This technical requirement is why our work focuses on HQQ with 4-bit quantization, as it is one of the few libraries with performant GEMM support for quantized models. We anticipate that broader low-bit GEMM support will become available as (tree-based) speculative decoding grows in popularity.

We thank the reviewer for the detailed and constructive feedback. We have conducted new experiments based on your suggestions, which we believe have strengthened the paper. Please find our responses below:

1. Missing Direct Comparison with SpecExec

We agree that a direct comparison with SpecExec is a crucial baseline. While we previously included a discussion in Appendix B.2, we have now performed a direct performance comparison. Under a similar decoding budget (~288 nodes), SubSpec significantly outperforms SpecExec:

SubSpec achieves ~83% higher throughput, driven by a 2.6x higher average token acceptance length (τ). This result validates SubSpec’s design principle: using a highly-aligned draft model combined with system co-design is more effective. We will prominently feature this direct comparison in the revised Appendix B.2.

2. Diminishing Benefit When Scaling to Very Large Models

This is a valid point regarding the method's scope. Our approach is primarily designed for the common and practical scenario where a quantized version of the target model can fit into GPU VRAM. This covers a wide range of models (up to the 30B scale) on modern consumer GPUs. For extreme cases, such as running a 70B+ model on a single constrained device where even the quantized draft will not fit, a different approach (e.g., using a separate, smaller draft model) would indeed be necessary. However, SubSpec remains applicable for these large models when multiple GPUs are available (e.g., via tensor parallelism), offering an efficient and training-free alternative where training a custom draft model might be infeasible.

3. Missing Related Works

Thank you for pointing this out. We will add a new subsection in our related works to provide a comprehensive review of self-speculative decoding methods.

Response to Questions

Under the same budget, quantized model might not have significant tree building capacity compared to using a small draft model and sample a huge tree ... I think there is some value in studying when it is better to use compressed model of the full as the draft model, when using a smaller draft model is more favorable.

Thanks for this insightful question. We investigated this trade-off by testing whether a larger draft tree could compensate for a smaller draft model. We expanded the tree width (k) from 6 to 32 and observed the following:

Our results show that increasing the tree width yields minor gains in average token acceptance length (τ), while harming throughput due to the increased drafting overhead. We attribute this to the highly skewed probability distributions of LLMs, where a small number of top tokens account for the vast majority of the probability mass.

This finding suggests that SubSpec will always be more favorable against less-aligned small models of the family while sampling huge draft trees. We will add this valuable analysis to the final paper. Thank you again for helping us improve our work.

Thank you again for your constructive comments and for pushing us to clarify this critical comparison.

Reply to Point 1 (Fair Comparison & VRAM Constraints)

All experiments in this discussion thread use Llama-3.1 8B as the target model and Llama-3.2 1B as the draft model, running on MT-Bench (greedy decoding, 8GB VRAM).

You are correct that controlling for the number of tree nodes is not ideal when draft models differ. The ideal approach is to find the maximal achievable throughput for each method under the same hardware constraint. However, there is a critical issue when attempting this comparison: the official SpecExec implementation fails to stay within the 8GB VRAM limit, even with small tree sizes and a reduced KV cache of 1024 (which is 2048 in our paper). To find SpecExec's best possible performance regardless of this constraint, we still conducted a full parameter sweep of SpecExec’s budget (its parameter for draft tree size):

SpecExec's peak throughput is 13.77 tokens/s (at a budget of 512). This is much lower than SubSpec's 24.29 tokens/s, and is achieved while already exceeding the 8GB VRAM limit. This comprehensive sweep also reinforces our earlier conclusion: the strategy of using a smaller draft model with a very large (width) tree may yield diminishing returns. Beyond a certain point (budget=512), the overhead of processing the large draft tree during the drafting phase outweighs the gains in acceptance length, causing overall throughput to decrease.

Reply to Point 2 (Scalability to Large Models)

We agree that scalability to very large models is an important consideration. Although we have already addressed this limitation in Appendix A of our submission, we will incorporate the discussion above into this section.

We thank the reviewer for the insightful feedback.

We agree that viewing SubSpec as a form of quantization compensation is a valuable perspective that helps clarify its unique contribution. The core design principle of SubSpec is to achieve acceleration while being lossless, guaranteeing outputs that are bit-for-bit identical to the full-precision (e.g., fp16) model. This fundamentally distinguishes our method from standard quantization techniques, which are inherently lossy and inevitably alter model outputs. This distinction is critical for applications where any deviation from the original model's behavior is unacceptable.

While we acknowledge that modern 4-bit quantization techniques achieve only 1~2% accuracy degradation on multiple tasks [1,2], the accuracy degradation is not always negligible, especially on more challenging benchmarks [2]. To illustrate this, we evaluated several popular 4-bit quantization methods on MMLU-Pro [3], using a random sample of 50 questions per category. The results showed a non-trivial accuracy drop in accuracy compared to the fp16 baseline:

SubSpec avoids this accuracy-capability trade-off entirely. Nonetheless, we agree that a direct comparison against strong quantized baselines is essential. Hence, we now include a direct performance comparison against fully GPU-resident 4-bit models to provide a clearer picture of the efficiency-capability landscape. The table below shows the throughput of Llama-3.1 8B on a single GPU with an 8GB VRAM limit on MT-Bench:

While a standalone 4-bit model offers maximum throughput by accepting the quality trade-offs of quantization, SubSpec is designed for users who require lossless output that is bit-for-bit identical to the original FP16 model.

For this high-fidelity use case, SubSpec makes offloading practical. It accelerates naive offloading from an unusable 2.4 tokens/sec to an acceptable 24 tokens/sec for interactive use. This allows users to achieve full model fidelity at a speed that is comparable to some standalone quantized models, all on consumer-grade hardware. We therefore position SubSpec as a simple, training-free solution for users who cannot compromise on model quality.

Response to Questions

Could the authors provide experimental results comparing different quantization algorithms and target bit-widths?

Different quantization algorithms. We have shown speed results of different quantization methods above. The reason we chose HQQ (int4) for our experiments is primarily due to its training-free nature and its high-performance triton kernels, which support efficient GEMM operations essential for tree-based speculative decoding.

Different target bit-widths. While state-of-the-art low-bit methods like QUIP# [4] and QTIPS [5] (2-bit) show promising results, their current public kernels lack GEMM support, making them incompatible with tree-based speculative decoding methods. We thus consider exploring draft models with lower bitwidths (2/3 bits) as promising future work, contingent on kernel development. Please refer to our reply for Reviewer DA4c for more details about the system requirements.

We have also added experiments using SubSpec to accelerate an 8-bit target model to provide a more comprehensive analysis. This scenario is considered as a different practical trade-off, where the accuracy degradation introduced by 8-bit quantization is acceptable.

Considering running Llama-3.1 8B on MT-Bench under an 8GB VRAM limit, while quantizing the target model to 8-bit still requires offloading, its memory footprint and offloading time are nearly halved compared to the 16-bit version. Our results show this configuration increases throughput by 36%, presenting a compelling trade-off between precision and performance:

Although admittedly challenging, could the authors consider providing a direct comparison with modern quantized models?

Yes. We have compared the throughputs of different quantized models and discussed trade-offs between SubSpec and strong quantized baselines above, hoping to address your concerns.

Reference

1. An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs, Visual Intelligence, 2024
2. Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models, COLM, 2025
3. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark, NeurIPS, 2024
4. QuIP#: even better LLM quantization with hadamard incoherence and lattice codebooks, ICML, 2024
5. QTIP: Quantization with Trellises and Incoherence Processing, NeurIPS, 2024

Thank you to the authors for the comprehensive response. I believe the main concern I raised—specifically, the comparison with quantization methods—has been adequately addressed. In addition, the experiments under different precision settings serve as a valuable supplement. The key logical issues I previously noted in the manuscript have been resolved. I will adjust my score accordingly, with the specific changes to be determined upon further consideration. I wish the authors all the best.

Thank you again for your valuable feedback. We are already working on revising the paper accordingly.

We thank the reviewer for the very thoughtful and crucial review. We will address the weaknesses in the following:

1. Missing Comparison with Standalone Quantized Models

We agree that comparing SubSpec to a standalone quantized model is essential. As you suggested, we now include a direct performance comparison against fully GPU-resident 4-bit models.

The table below shows the throughput of Llama-3.1 8B on a single GPU with an 8GB VRAM limit:

While a standalone 4-bit model offers maximum throughput by accepting the quality trade-offs of quantization, SubSpec is designed for users who require lossless output that is bit-for-bit identical to the original FP16 model.

For this high-fidelity use case, SubSpec makes offloading practical. It accelerates naive offloading from an unusable 2.4 tokens/sec to an acceptable 24 tokens/sec for interactive use. This allows users to achieve full model fidelity at a speed that is comparable to some standalone quantized models, all on consumer-grade hardware. We therefore position SubSpec as a simple, training-free solution for users who cannot compromise on model quality.

2. Untapped Potential in Quantized-to-Quantized Fully GPU-resident Scenarios

We agree it is an interesting direction. However, for a draft with 2-bit substitutes accelerating a 4-bit target, we believe the gains would be modest. A much more efficient draft architecture (e.g., EAGLE) is required to achieve decent speedup on fully GPU-resident cases, as shown in our analysis in Sections 3.1 and 3.2. The effectiveness of speculative decoding hinges on a draft model that is substantially faster than the target. Also, due to hardware and kernel overheads, current 2-bit quantization kernels are not yet fast enough compared to their 4-bit counterparts to create the large speed gap needed for significant improvement.

Inspired by this suggestion, we analyzed a different practical trade-off, where the accuracy degradation introduced by 8-bit quantization is acceptable.

Considering running Llama-3.1 8B on MT-Bench under an 8GB VRAM limit, while quantizing the target model to 8-bit still requires offloading, its memory footprint and offloading time are nearly halved compared to the 16-bit version. Our results show this configuration increases throughput by 36%, presenting a compelling trade-off between precision and performance:

Thank you again for pushing us in these valuable directions. We will incorporate this analysis into the final paper.