SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths

Thank you for your positive feedback on our work! We sincerely appreciate the time and effort you have invested in reviewing our paper and providing valuable feedback. We would like to provide detailed responses below:

---

Comment 1: Missing comparison to baseline adaptive draft length using confidence of small model (e.g., per BiLD).

Response 1: We have discussed the sub-optimality of heuristic methods (e.g. only using entropy or top-K prob as a metric of uncertainty) in Appendix B. Empirically speaking, these heuristics may achieve certain performance gain, as is discussed by Liu et al (2024), Kim et al (2024), and Xu et al (2023) (lines 118-121). But still, these methods are fundamentally flawed.

The goal of our paper is not to demonstrate empirical superiority over these heuristic methods, as we have clearly discussed that these methods can fail in certain cases. Instead, our goal is to develop a robust and theory-backed algorithm that practitioners can trust when they have the demands of optimizing the speculative decoding pipeline.

---

Comment 2: Fundamentally, the learnability of Equation 4.1 depends on how well we can predict the relative likelihood of the large model given information from the small model. Comment about the viability of this task might be useful.

Response 2: It is hard and potentially impossible to give a rigorous argument about the learnability of Equation 4.1. A plausible situation is when the degree of alignment (measured by total variation distance) between the draft distribution and the target distribution correlates with the existing context. In that case, the acceptance prediction head can be trained to utilize the existing tokens to predict the acceptance.

---

Comment 3: One might also imagine leveraging information from the larger model on previously rejected tokens, e.g., per Aishwarya PS, “Tandem Transformers for Inference Efficient LLMs”, ICML 2024

Response 3: Thank you for the pointer! This is indeed a valid operation. We will try to explore this in the future.

---

Comment 4: “take constant time, which is true when we have enough computational resources to” → constant in what? What kind of computational resources – compute or memory?

Response 4: We meant that “the time of the forward passes remains constant when we change the length of the input tokens”. The computational resources include both compute units (e.g., number of tensor cores) and memory (e.g,. HBM). When we increase the sequence length, the forward pass may take a longer time if the longer tokens cannot be processed in parallel using the compute resources. This can include both (1) more loops over the tensor core job scheduling and (2) longer waiting times for HBM communication.

---

Comment 5: “In the typical setting of speculative decoding where the draft model and the target model align reasonably well” → what does “align” mean here? Why is this the typical setting? See also Appendix D.4.

Response 5: By saying “the two models align well”, we mean the distance of the distributions given by the two models is small, which can be formally defined as the expected total variation distance over a distribution of natural text. This is the typical setting because both the draft model and the target model are approximating the same distribution:

• (1) One case is when the draft model and the target model are obtained from a set of pretrained models. These models share the same pretraining corpus and are trained using the same objective --- to approximate the distribution of the pretraining corpus.
• (2) Another case is when a smaller model is distilled from a larger model. In this case, the small model is trained to mimic the distribution of the target model.

---

Comment 6: Algorithm 1, what is the end index of the for loop?

Response 6: In our experiment, we have set an upper limit for the for loop to be 20. (line 285)

---

Comment 7: Similar ideas may have been considered in contemporaneous work (Zhong et al. 2025)

Response 7: Zhong et al.'s method cannot guarantee that the output distribution matches the target distribution exactly. We will properly cite and discuss this in our next revision.

---

We sincerely hope that our responses can address your concerns. Thank you for your positive overall evaluation again!

We sincerely appreciate the time and effort you have invested in reviewing our paper and providing valuable feedback. We would like to provide detailed responses below:

---

Comment 1: This paper makes no claim as to how this proposed method compares empirically to other methods for dynamically adjusting the draft length (Mamou, Jonathan, et al.) [1].

Response 1:

Thank you for the pointer. Compared to [1], our paper has unique advantages of theoretical analysis and clearer, self-contained writing, while the main text of [1] is only 4-page long. We will properly cite [1] and include the following discussion in our next revision:

Discussion: The authors of [1] propose to train an FFN taking several statistics of the probability vector of the draft model to predict the total variance distance $TV(y_i^D, y_i^T)$ of the draft model and the target model, which is highly related to the acceptance probability. Our method differs from [1] in that:

• (1) Our acceptance prediction head is built on top of the final hidden state of the last token, which contains richer information than merely the statistics of probability distribution (top_K and entropy).
• (2) We predict the acceptance of the token $Y_i$ when $Y_i$ is fed into the draft model, i.e., when the distribution of $Y_{i+1}$ is being computed. Instead, [1] predicts the acceptance of $Y_i$ when the actual sampling result of $Y_i$ is undetermined, which may limit the empirical performance as the target is noisier.

---

Comment 2: The experiments are light, with only vanilla speculative sampling considered as a baseline. Although there is a discussion in Appendix B about why heuristics such as token-level draft distribution statistics are not appropriate, these methods are never evaluated.

Response 2:

We have discussed the sub-optimality of heuristic methods (e.g. only using entropy or top-K prob as a metric of uncertainty) in Appendix B. Empirically speaking, these heuristics may achieve certain performance gain, as is discussed by Liu et al (2024), Kim et al (2024), and Xu et al (2023) (lines 118-121). But still, these methods are fundamentally flawed.

The goal of our paper is not to demonstrate empirical superiority over these heuristic methods, as we have clearly discussed that these methods can fail in certain cases. Instead, our goal is to develop a robust and theory-backed algorithm that practitioners can trust when they have the demands of optimizing the speculative decoding pipeline.

---

Comment 3: line 111: "...including block paralleling sampling...", line 130: "...we first generates candidate tokens..."

Response 3: Thank you for your meticulous evaluation of the paper. We have fixed the typos!

---

Comment 4: Appendix D.3 varies the weight on the rejection class, w_{rej} , but does not touch w_{acc} . Was w_{acc} tuned, or was it fixed to 1 in experiments?

Response 4: Thank you for pointing this out. It was fixed to 1 in the experiments. This setting is stated in the main text (line 283). But we agree with you that it should be repeated in Appendix D.3. We have updated our manuscript.

---

Comment 5: How important was tuning $w_{acc}$, $w_{rej}$, and $h$ to achieve acceptance prediction performance above baseline speculative sampling?

Response 5: It is easy to achieve performance better than the baseline speculative decoding. In our experiments, the best throughput achieved by the baseline SpecDec is 17.62 tokens/s, 18.55 tokens/s, and 19.14 tokens/s, for Alpaca, HumanEval, and GSM8k, respectively. Cross-referencing the numbers with Table 3 in Appendix D.3, we can see multiple $w_{rej}$’s lead to higher performance than the baseline SpecDec. The threshold $h$ only requires light tuning (e.g., varying among 0.3, 0.5, 0.7), and can be done efficiently without the need of re-training the prediction heads.

---

We sincerely hope that our responses can resolve the misunderstandings and address your concerns, and we would greatly appreciate it if you would like to re-evaluate our work given the responses.

Thank you for recognizing our efforts and for your positive evaluation of our work! We sincerely appreciate the time and effort you have invested in reviewing our paper and providing valuable feedback. We would like to provide additional explanations below:

Comment 1: In terms of comparison experimental study, the paper lacks comparisons with other speculative decoding paradigms and methods (like medusa, eagle, sequoia, DySpec, or any comparable methods).

Response 1: We have discussed this in the related work section. Specifically:

• (1) The improvement of our method is achieved through algorithmic innovations, and can be plugged into methods like EAGLE. (Lines 104-109)
• (2) For methods like Medusa, as the generated tokens may deviate from the target model’s distribution under the general stochastic sampling setting, we choose not to compare against this line of methods in our paper. (Lines 110-115)
• (3) For token tree methods like Sequoia and DySpec, our method is not directly comparable. However, our work can serve as a starting point towards the problem of optimal token tree construction, as the candidate length K can be viewed as the depth of a token tree with only one branch. Please refer to our discussion in Appendix B (Lines 555-568).

---

Comment 2: Additionally, the experimental setup is relatively simple and does not explore the acceleration of target models and draft models of different scales.

Response 2: Due to page limit, we deferred an additional experiment of Gemma models (2B v.s. 27B) to Appendix D.4, and showed similar speedup of our methods.

---

We sincerely hope that our responses can address your concerns. Thank you for your positive overall evaluation again!