AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders

Dear Reviewer 5tif,

Thank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.

W1: Only the vanilla SD is tested, so it is not clear whether AdaSPEC could generalize to more advanced SD methods like Eagle.

R1: Thank you for your insightful comment. To address this, we implemented AdaSPEC with Eagle—a representative SD method featuring tree attention that demonstrates superior performance to Medusa. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench. The results below demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle.

Table R1. Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.

W2: The only baseline is DistillSPEC. The evaluation is limited in scope. The discussion of related work is limited.

R2: We appreciate the reviewer’s feedback and the opportunity to clarify. As of now, DistillSpec is the only existing work that applies model distillation to train the draft model in the speculative decoding (SD) framework, and it is widely regarded as the state-of-the-art method in this line of research. Therefore, we believe our comparison to DistillSpec is both appropriate and meaningful within the scope of distillation-based SD methods.

Moreover, our results demonstrate not only superior decoding efficiency under the standard SD architecture but also stronger generalization across models of different sizes. We have also highlighted that our approach is orthogonal to other classes of SD techniques (e.g.adaptive rejection, or sampling strategies), and can in principle be combined with them to further enhance performance. For example, our experimental results in R1 above demonstrate that our methods can be effectively integrated into the EAGLE framework.

Regarding the discussion of related work, we have included comparisons to both speculative decoding and distillation literature in Section 2. We agree that additional connections—such as to broader acceleration strategies—may be further explored in future iterations of this work.

W3: There is a huge gap between 3-epoch and optimal-epoch performance. A more advanced hyper-parameter selection strategy could have been proposed to optimize the performance.

R3: Please refer to Q3 for this.

Q1: Can AdaSPEC generalize beyond vanilla SD to more advanced methods like Eagle or Medusa?

R1: Yes. Please refer to W1 for the corresponding results.

Q2: Could more relevant baselines be included, like Rho-1 mentioned in the paper?

R2: Though AdaSpec resembles Rho-1 in the token selection stage, the two studies are different. Specifically, Rho-1 focuses on the pretraining of language models, which is not relevant to our study. We have no idea how AdaSpec could be compared with Rho-1.

Q3: Considering the significant gap between 3-epoch and optimal-epoch performance, why wasn’t a more effective hyperparameter selection strategy proposed to improve results?

R3: First, we have already tuned our hyperparameters to their optimal as much as possible within each setting. The hyperparameters can be found in the appendix.

Importantly, the goal of our work is not to maximize absolute performance via exhaustive hyperparameter tuning, but rather to evaluate and compare the effectiveness of AdaSpec across different realistic training scenarios. As discussed in Section 4.2, we deliberately introduce two representative training settings to simulate common use cases:

The 3-epoch setting simulates a realistic case where the draft model has been partially trained and already performs reasonably well on the target domain. This reflects a typical ablation setup, where forgetting is limited and generalization is already strong.

The optimal-epoch setting aims to approximate the best achievable performance on the target domain with further training, serving as an upper bound for model capability.

In addition, the 3-epoch setting also aligns with our mixed-task experiments in Section 4.6, where we train models sequentially on MBPP and then GSM8K (each for 3 epochs) to test AdaSpec’s ability to retain previously acquired knowledge. As shown in Table 6, AdaSpec is effective in preserving the model’s original capabilities under such limited training regimes, further supporting the design choice of the 3-epoch setting.

Finally, we note that AdaSpec consistently improves performance under both settings, demonstrating strong generality. Importantly, these gains are achieved without task-specific hyperparameter tuning, highlighting the robustness and practicality of our approach.

[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

Thank you again for providing detailed constructive comments and for raising the score! We will update the experiment results in subsequent version.

Dear Reviewer qrev,

Thank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.

W1: Twice the distillation compute

R1: While it is true that our approach introduces additional compute during the distillation stage, the core objective of speculative decoding is to accelerate inference. We argue that a moderate increase in training cost is acceptable, especially given the substantial inference speedups achieved. Moreover, compared to pretraining large language models, the overall cost of distillation on downstream tasks is relatively small and typically affordable in practice.

W2: Limited to Speculative Decoding settings

R2: Our method targets the standard speculative decoding (SD) setting, which is a widely adopted and practically relevant scenario for accelerating inference in large language models. In this context, our approach achieves state-of-the-art performance and outperforms DistillSpec in both efficiency and generalization.

To further validate AdaSpec on more advanced approaches beyond vanilla SD, we have now included additional experiments combining our method with Eagle, a popular decoding strategy utilizing tree attention for multi-token speculation. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench.

Table R1. Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.

The results in the table above demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle.

Could you please clarify which specific scenarios are considered missing? We believe the standard SD setting and the additional experiment above already provides a strong and representative benchmark for evaluating speculative decoding methods.

Q1: Is there a possibility to train both reference and draft model at the same time?

R1: Yes. Although the reference model is limited to a roughly pre-finetuned speculative decoder in our study, it can be replaced by any small model that aligns with the target model. One possible approach to train both reference and draft model simultaneously is to dynamically assign the reference model to be an earlier checkpoint of the draft model. We find this idea quite interesting and believe it opens up a new direction for adaptive speculative decoding, which we plan to explore in future work.

Q2: Can you explain the only mild improvements of acceptance scores (often ~1%)?

R2: The reason for modest improvement in our main results is mainly due to the small size gap between draft and target model. In Pythia settings, where the draft/target size ratio is about 1/40, AdaSpec introduces up to 8% acceptance rate improvement. When it comes to some of the Phi settings, whose draft/target ratio is about 1/8, that advantage is weakened to up to 3%. This shows the advantage of our method on extreme draft/target ratios: the larger the size gap is, the more effective our method is.

Q3: Could you add in your discussion of SD alternatives like MoE, CoT and recurrent-depth approaches in l.25ff? What is your opinion on those compared to SD?

R3: We thank the reviewer for the suggestion. However, we would appreciate further clarification on the intended comparison, as the listed approaches seem to target different aspects of model design and usage.

To our understanding:

• MoE (Mixture of Experts) accelerates inference by activating only a subset of model parameters during each forward pass, providing efficiency gains through activation sparsity. In contrast, speculative decoding is model-agnostic and focuses on speeding up generation by parallelizing the decoding process.

• Chain-of-Thought (CoT) and recurrent-depth methods aim to improve reasoning capabilities or model expressivity, often at the cost of dynamically expanding inference computation. These approaches are typically used to enhance output quality rather than inference efficiency.

[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

Thank you again for the valuable review! We really appreciate your recommendation for acceptance, and we would like to make more clarifications regarding Q3, as well as in the subsequent version of the paper.

Compared to MoE, CoT, and recurrent-depth approaches, the biggest advantage of Speculative Decoding is lossless acceleration—that is, the expected accuracy remains entirely consistent with the vanilla counterpart. SD can achieve over a 2x speed improvement with just a little extra memory for inference.

Compared with SD, MoE has always lagged behind its dense counterpart in performance (mainly due to convergence issues difficulties introduced by sparsity). Besides, the CoT paradigm does not consistently enhance model reasoning capabilities, as pointed out by some recent studies such as [2].

Our method demonstrates that through appropriate token level pruning, the performance of SD can be further improved on a lossless basis, offering a simple and effective new approach.

[2] To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning

Dear Reviewer TYdk,

Thank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.

W1: Although consistent, the effect of the method is modest, which reduces the impact of the work.

R1: The reason about modest improvement in our main results is mainly due to small size gap between draft and target model. In Pythia settings, where the draft/target size ratio is about 1/40, AdaSpec introduces up to 8% accuracy improvement. When it comes to the Phi settings, whose draft/target ratio is about 1/8, that advantage is weakened to up to 3%. This shows the advantage of our method on extreme draft/target ratios: the larger the size gap is, the more effective our method is.

Furthermore, we would like to highlight that the advantage of AdaSpec is in inference acceleration but not training. Although AdaSpec introduces minimal overhead to the standard distillation framework, it is actually very effective in inference-heavy scenarios. Compared with the marginal computation cost of distillation, AdaSpec can lead to 10–20% speedup (as shown in Section 4.6), making it highly practical for real-world deployment.

Finally, to further enhance the practical utility and impact of our work, we integrate AdaSpec to Eagle, a frontier method that combines tree attention with SD. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench. The results below demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle.

Table R1. Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.

These results demonstrate that AdaSpec is not only scalable and lightweight, but also delivers substantial practical gains in scenarios where inference cost dominates—underscoring its relevance for efficiency-critical applications.

W2: Although the implementation of token filtering is simple and should not incur a significant computational or memory cost, given that it involves the additional inference of the reference model, I would appreciate a comparison of the impact, given an equal number of tokens incorporated in the GD step.

R2: We argue that the extra computation cost would be marginal and can be ignored.

1. The extra computational cost mainly involves one forward pass of the reference model. Since the reference model and draft model share the same architecture and size, this would introduce about 25% extra computation (the backward pass of the draft model is twice as much as the forward pass, and gradient checkpointing introduces an extra forward pass).
2. In distillation we also need to run the target forward pass, whose computation is tens of the forward pass as the draft model.

Most importantly, the core goal of speculative decoding is to accelerate inference, not to reduce training cost. Our method offers substantial decoding speedup during inference with only a small, one-time overhead during distillation training. We believe this trade-off is both reasonable and practical.

Q1: In L138, you state that the selected tokens are the ones that are easiest for the draft model to align with the target model. However, this motivation is slightly unclear to me. In particular, it seems that tokens maximizing (9) might satisfy a combination of “learnability”, related to your “ease”, but also “not yet learnt”, since tokens that are already predicted by the reference model well will be easy to predict, but will not satisfy (9). Would you agree?

R1: Thank you for raising the question. Your understanding is correct. We will make this clear with more explanation at L138.

Q2: In L182, could you mention in the appendix and reference how you specifically chose the number of epochs?

R2: For GSM8K, the number is chosen according to accuracy on validation set. For the rest of the settings, that number is chosen according to Cross Entropy loss on validation set.

Q3: In L188-189, for reproducibility, it would be important to report how the hyperparameters and most importantly, the filter fraction were chosen. In particular, could you justify in L286 why you sometimes use k=0.8 instead of 0.4 everywhere? Is there any way to understand when we would want a higher k and when a lower one?

R3: We naturally choose some batch size and learning rate that appears reasonable. Alpaca, XSUM and CNN/DailyMail datasets are relatively large, so we modify some of the settings accordingly. We further tune the epochs to get optimal results.

For Alpaca, we first train a set of ablations including different k, learning rate and batch size. Then we select the optimal result which appears to be 0.8. For the rest of the experiments, since 0.4 is a feasible choice for most of the settings, we keep k=0.4 and no longer tune this parameter.

Q4: In L198, could you please provide the respective plots for the other configurations in the appendix, given that these should be readily available already? That would aid in the completeness of the results.

R4: Yes, we will provide more plots. Since NeurIPS 2025 forbids plots in author reviewer discussion, we will update this in the final version of our paper.

S1: In the equation after L140, you mention selecting the top k tokens, but in the appendix pseudo-code, you appear to choose them based on quantiles instead. I would encourage addressing this inconsistency for reproducibility.

R1: Sorry for the misunderstanding. The "top k tokens" refers to a ratio of the selected tokens, for example, top 40% tokens. Here, k $\in$ [0,1]. We will clarify this in the paper.

S2: I would encourage the authors in the Abstract and Introduction to focus more on the observation that simple KL optimization does not effectively maximize the acceptance rate, which is arguably the goal of SD. Therefore, in SD, we aim for high precision or maximum likelihood, not matching the distribution. As you rightly mention in the limitations, your heuristic is just a first approach to use this observation. Still, you could capitalize on this observation as a key contribution, which might enhance the impact of this work.

R2: Thank you for pointing this out. We will emphasize this insight in the introduction and abstract sections.

[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

Thank you so much for the valuable review and for the Accept rating! We will include the related work discussion and additional cost analysis in the subsequent version.

Dear Reviewer ocoD,

Thank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.

W1: Insufficient Support for the Reference Model: While the reference model is claimed to play a "crucial role" in filtering hard tokens, the paper lacks strong theoretical justification for why its perplexity-based filtering reliably identifies tokens beyond the draft model’s capacity. Additionally, there is a lack of direct experimental evidence (e.g., correlation between the reference model’s "hard token" labels and actual draft model errors) to validate its filtering accuracy .

R1: Take GSM8K as an example, we argue that the reference model can actually prune out the hard tokens for the draft model:

1. An observation is that the hard tokens are typically math related tokens, as depicted in Appendix A.3, these tokens are difficult to predict as they involve mathematical logic, whose complexity is often beyond the draft model’s capacity.
2. Case Studies in Section 4.4 reveals that our token selection mechanism can successfully filter out hard tokens, especially those mathematical tokens.

Besides, we need to point out that many researches on large models are driven by empirical results, with theoretical studies or justification lagging far behind empirical research. Our method has demonstrated strong results in practice and empirically supported our statement in how tokens are filtered, which we believe sufficiently substantiates our conclusions.

W2: Limited Comparisons to Recent Methods: The method is only benchmarked against DistillSpec (2023), without comparisons to more recent advancements in speculative decoding or knowledge distillation. This narrow comparison weakens claims of state-of-the-art performance and reduces the work’s timeliness .

R2: As of now, DistillSpec is the only existing work that applies model distillation to train the draft model in the speculative decoding (SD) framework, and it is widely regarded as the state-of-the-art method in this line of research. Therefore, we believe our comparison to DistillSpec is both appropriate and meaningful within the scope of distillation-based SD methods.

Moreover, our results demonstrate not only superior decoding efficiency under the standard SD architecture but also stronger generalization across models of different sizes. We have also highlighted that our approach is orthogonal to other classes of SD techniques (e.g.adaptive rejection, or sampling strategies), and can in principle be combined with them to further enhance performance.

To further validate our proposed method on frontier SD approaches, we implemented AdaSPEC with Eagle—a representative SD method featuring tree attention that demonstrates superior performance to Medusa. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench.

Table R1. Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.

These results demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle, which enhances the practical utility and impact of our work.

W3: Underdeveloped Generalization Analysis: While AdaSPEC shows promise in tested configurations, its generalization to larger model gaps (beyond 40x) or more complex, low-resource tasks is not thoroughly explored. The limitations section only briefly mentions potential improvements to filtering strategies, without addressing how the method might scale or adapt to diverse real-world scenarios .

R3: We would like to clarify that we have already evaluated AdaSpec under a draft/target size ratio of 1/60 in our additional experiments on the Qwen2.5-0.5B → Qwen2.5-32B setting, which goes beyond the 1/40 ratio raised in the question. These results confirm that AdaSpec remains effective even under such extreme scale gaps.

While we agree that analyzing even larger gaps could be interesting, we believe that in general-purpose applications, going beyond a 40× size ratio is rare and often unrealistic—particularly in settings where the draft model must still perform useful computations (e.g., general text generation). In practice, such extreme ratios may only arise in domain-specific or highly constrained deployment scenarios, which are somewhat outside the scope of our current work.

In terms of diversity, we have already included a wide range of domains in our evaluation—spanning mathematical reasoning, instruction following, code generation, text summarization, and multi-task mixtures (as discussed in §4.6). In addition, our dual setting design (3-epoch and optimal-epoch) is intended to simulate realistic training budgets and performance goals, further supporting the robustness of AdaSpec across practical scenarios.

[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

Dear Reviewer ocoD,

Thank you for your time and valuable feedback on our work. We hope that our response and revision have adequately addressed your concerns. As the discussion period nears its end, we would be very grateful if you could reconsider our work and possibly adjust the score accordingly. If you have any additional questions or suggestions, we would be happy to have further discussions.

Best regards,

Submission7301 Authors

Thank you for your follow-up. Below we address your requests regarding token-level distillation outcomes and comparative distillation efficiency, and clarify the practical implications of our method.

Token-Level Distillation Outcomes

Our paper already provides a both qualitative and quantitative token-level analysis to evaluate the impact of AdaSpec.

In our GSM8K case study (Appendix A.3 and Section 4.4), we show that in math domain, AdaSpec tends to select math-related tokens during filtering. These tokens represent structured, learnable reasoning patterns (e.g., numbers, mathematical operators, or logical steps). Our reference model helps focus the distillation process on these parts, enabling the draft model to better capture the reasoning logic through supervision from the target model.

This aligns with our core idea: AdaSpec guides the draft model to allocate capacity where it matters most—on tokens that are important and learnable but may otherwise be under-emphasized in uniform distillation. This observation offers token-level evidence.

In addition to case-level analysis, we also provide statistical token-level evidence across datasets in Figure 2, which supports the effectiveness of AdaSpec’s selective distillation approach:

• Logit Margin Distributions: AdaSpec increases both the frequency and magnitude of positive top-2 logit margins while reducing negative ones, showing the draft model makes more confident and accurate predictions.

• KL Divergence Across Tokens: AdaSpec yields consistently lower token-level KL divergence, with a distribution shift toward zero compared to DistillSpec, indicating tighter alignment with the target model’s outputs.

Together, these metrics offer a multi-faceted view of token-level alignment, confidence, and knowledge transfer quality.

Comparative Distillation Efficiency across Diverse Tasks

In terms of task diversity, we have already included a wide range of domains in our evaluation—spanning mathematical reasoning, instruction following, code generation, text summarization, and multi-task mixtures (as discussed in §4.6).

Moreover, our dual setting design—comparing 3-epoch and optimal-epoch training—serves to simulate varying training budgets and practical constraints, providing further evidence of AdaSpec’s robustness across different real-world use cases.

Importantly, the token-level analyses (e.g., logit margin distributions and KL divergence metrics) and the ablations in §4.5 are also conducted across these diverse tasks. This reinforces that the gains from AdaSpec are not task-specific artifacts, but hold across multiple domains and model behaviors.

Practical Impact and Scalability

We believe AdaSpec offers real-world utility based on the following:

1. Easy integration: It is implementation-friendly and requires no architecture changes.
2. Scalable: We validated it with up to a 60× model size gap (e.g., Qwen2.5-0.5B → Qwen2.5-32B), showing strong performance even in extreme settings. While >70B experiments are out of academic reach, our method poses no intrinsic barrier to industrial-scale deployment.
3. Orthogonal to other methods: As shown with Eagle + AdaSpec, our method can be combined with orthogonal improvements (e.g., adaptive sampling, tree attention) for additional speedup (~10%).
4. Superior to DistillSpec: Compared to DistillSpec—the only other distillation-based speculative decoding baseline—AdaSpec achieves 10–20% faster inference under the same standard speculative decoding framework.

Dear Reviewer fMLq,

Thank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.

W1: It is unclear why the proposed filtering approach (eq 9) evaluates and compares the loss of the reference vs the draft model (Mq), instead of the correctness/distance of the reference (distilled) model to the original (Mp). Since Mq is the model being trained in step2, Mq's perception of easy samples is expected to change overtime, however the proposed approach does not appear to adapt to such changes. In contrast, evaluating the correctness of Mref (w.r.t Mp) may potentially form a more robust and stable proxy of the predictive difficulty of each token for the draft model.

Reply:

As we propose in "introduction" chapter (line 57), our goal here is to identify tokens that are more learnable at the current training stage of the draft model Mq. Compared to the absolute value of per-token loss/accuracy, the relative loss change is more indicative of ease of “learnability”. This is because the potential to learn each token varies, and a high loss/low accuracy does not necessarily equate to ease of learning. Evaluating the difference between Mref and Mp, as suggested, would instead reflect how closely the reference approximates the teacher model, but may not align with the objective of selecting learnable tokens for improving Mq, leading to suboptimal results.

We think that possibly the reviewer might be referring to the initial (untrained) version of Mq when mentioning Mp. Interpreted this way, comparing to the base model could provide an alternative view on token difficulty, and we agree that this is an interesting direction worth exploring in future work. That said, this may be more likely to be viewed as an alternative design choice rather than a clear weakness.

W2: It is unclear how the effectiveness of the proposed approach is affected by the learning capacity of the draft model (and whether the percentage of selected tokens should be scaled accordingly). Although the authors have conducted experiments between 2 different pairs of models, with varying relative parameter counts, more robust insights about the scalability of the proposed method will largely benefit the community and enhance the applicability of the proposed approach.

Reply:

The first question is the capacity of the draft model. Technically, the capacity of the draft model is relative to the target model, which can be depicted by the size gap between them. The two pairs of draft/target size ratios are representative in contemporary research : $\sim$1/40 for Pythia setting, and $\sim$1/8 for Phi-2 setting. This holds practical significance in real-world applications.

When practical applications require larger target and draft models, our method is designed to be implementation-friendly and readily scalable. It can be integrated seamlessly into existing training pipelines without architectural changes, making it suitable for scaling to larger models and datasets.

In Section 4.6, we conducted further GSM8K tests using a combination of Qwen2.5-0.5B and Qwen2.5-32B models ($\sim$1/64 size ratio), which demonstrate that our approach remains effective when scaling up to larger models. We again present those results below.

Table R1. Acceptance Rate of Qwen2.5-0.5B$\rightarrow$32B with 3-epoch GSM8K.

The second question is how to scale k. This hyperparameter appears to be relatively stable with respect to model scale, and we did not find strong evidence that it needs to be adjusted as the model size increases. Thus, we keep it to 0.4 for all shapes of the models, even for the Qwen2.5-32B experiment.

While we acknowledge that experiments on even larger models (e.g., >70B) would be valuable, such experiments are unfortunately beyond the computational affordability of academic research. Nonetheless, our method poses no intrinsic barrier to such scaling, and we believe it can be readily adopted in industrial-scale settings.

The third question is the generalizability of AdaSpec on broader settings and applications. To further validate AdaSpec on more advanced approaches beyond vanilla SD, we have now included additional experiments combining our method with Eagle, a popular decoding strategy utilizing tree attention for multi-token speculation. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench.

Table R2. Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.

The results in the table above demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle.

[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty