A Drop-In Solution for On-the-Fly Adaptation of Speculative Decoding in Large Language Models

Comment 1: My main question is why focusing only on single-sequence speculation rather than the tree-based speculation structure, which is the current best practice for speculative decoding. More specifically, how does your approach compare with EAGLE-2, which leverages a similar idea (adaptive speculation) but on draft trees rather than a single speculation sequence, which results in a much higher speed-up rate? The simplicity of your solution can be a bonus if it can be generalized to tree-based speculation and achieve a higher speed-up than EAGLE-2..

Response: Our drop-in method dynamically changes window size which maximizes speculative decoding efficiency. It is proposed from a different angle and is complementary to those studies. By adaptively changing the draft tree depth, the on-the-fly adaption of speculative decoding can adjust the draft token sequence length on the fly thus improving decoding performance. We apply our method to the SOTA EAGLE-2 and report the result in Section 6.4. Results show that, on the MT-Bench, we achieve up to $3.56\times$ speedups compared to original autoregressive decoding, and an additional $4.2$% improvement over SOTA.

Comment 2: Minor suggestions: Try formulating the verification latency as a function of the speculation structure from a more systematic perspective. For example, make it a function of the speculation sequence length or speculation tree structure, this may also depend on the underlying hardwares' specifications.

Response: Thank you for your insightful comment. We have observed that LLaMA's verification latency remains relatively consistent. This is largely due to GPUs' massive parallelism capabilities. However, for certain other models, verification latency can vary depending on the speculation length. We agree that a more detailed modeling of verification latency could further enhance our results. This could be achieved by incorporating on-the-fly observations from the beginning, which would allow us to build a simple performance model and integrate it into our current method. We have adjusted our theoretical analysis accordingly. For the empirical results, we will implement these changes and include them in the revised version of our work.

Comment 3: Include some ablation studies to show how different configurations affect the effectiveness of the speculation accuracy estimator. For example, how different (recent history length) values can affect your speculation accuracy prediction for online window size optimization, and how different linear model (different vector lengths or random variables) can affect your speculation accuracy prediction for estimating the speculation accuracy for different draft models.

Response: We included a sensitivity study for the effects of different history length values in Appendix C.5. We will also add a sensitivity study for different linear models in the revised paper.

Comment 1: Baselines seem especially weak and some other baselines are not compared. For example, Medusa and EAGLE, they are currently popular approaches that has been compared in numerous papers; all of these methods utilize speculative decoding techniques and demonstrate strong performance.

Comment 2: Although I find the concept of a dynamic speculation window size intriguing, I am skeptical about its practical application value. Recent studies have already adopted tree attention or tree decoding-related technologies, they do not require a speculation window size and have achieved significant speedup. Could you discuss how your approach compares with, or could potentially be integrated into, tree attention or tree decoding technologies?

Response: Medusa and EAGLE use tree-based decoding to speculate multiple possible token sequences in parallel and then validate each of these sequences to keep the longest validated one. Our drop-in method dynamically changes window size which maximizes speculative decoding efficiency. It is proposed from a different angle and is complementary to those studies. By adaptively changing the draft tree depth, the on-the-fly adaption of speculative decoding can adjust the draft token sequence length on the fly thus improving decoding performance. We apply our method to the SOTA EAGLE-2 and report the result in Section6.4. Results show that, on the MT-Bench, we achieve up to $3.56\times$ speedups compared to original autoregressive decoding, and an additional $4.2$% improvement over SOTA.

Comment 3: In my understanding, Table 2 presents the results using adaptive window size selection, and Table 3 presents the results using draft model selection. Why not conduct an experiment to show the results of using both techniques simultaneously? Additionally, could you specify which draft model was used for the 'w/o draft selection' condition in Table 3?

Response: Thanks for your comment. Table 2 reports the results using adaptive window size selection, and Table 3 already reports the results using draft model selection on top of window size selection. The draft model used for the `w/o draft selction' is the one with the highest speedup compared to the original autoregressive decoding. Specifically, for the target model LLaMA 70B, LLaMA 7B, 7B, and 13B are used as the draft model for alpaca, humaneval, and gsm8k, respectively. For the target model BLOOM 7B, BLOOM 1B1, 560M, and 560M are used as the draft model for alpaca, humaneval, and gsm8k, respectively.

Comment 1: The improvement is somehow marginal. The window size selection method generally achieves improvements of less than 10% and less than 1% in some cases. The proposed draft model choice method leads to less than 1% improvement in many cases.

Response: Thank you for your comment. While the improvement in some of the cases may appear marginal (e.g., less than 1%), it is important to consider the broader context and implications of these optimizations. The choice of speculation window length and draft model is critical for unlocking the potential of speculative decoding. Suboptimal decisions not only diminish the benefits but can also introduce slowdowns. As lightweight optimization tools, our proposed method addresses this challenge by moving away from the trial-and-error approach, enabling adaptability to changes in tasks, models, software, hardware, and runtime conditions.

We also demonstrate the flexibility and scalability of our method by incorporating the adaption to the tree-based speculative decoding methods (e.g., EAGLE, EAGLE-2). Without any profiling, our method adapts the window size and achieves consistent speedups compared to the SOTA EAGLE-2.

Moreover, even small performance gains, such as a 1% improvement, can have a significant impact in large-scale deployments. For example, Google’s infrastructure optimizations have shown that a 1% increase in speed can result in millions of dollars in operational cost savings. It is important to note that all of the gains come for free: Our solution is a drop-in replacement without any extra burden being added to the developers or users, thanks to its on-the-fly adaptive optimizations. Thus, our approach provides meaningful advantages by reducing the dependency on manual tuning and ensuring efficiency improvements in dynamic and resource-constrained environments.

Comment 2: The method is only tested in domain-specific tasks (math/code/finance QA/summarization) and is not evaluated on comprehensive chat datasets like ShareGPT, which raises questions about its applicability to realistic chat scenarios.

Response: We added experiments on ShareGPT on both the 4090 and A100 machines. Results are added into Section 6.4 and Appendix C.3, showing that our drop-in method achieves an average of $1.71\times$ speedups compared to original autoregressive decoding, and an additional $4.9$% improvement over speculative decoding baselines.

Comment 3: What is the value of window size for the standard speculative decoding baseline? Is the window size for the standard speculative decoding baseline in each dataset set as the optimal value obtained by searching through all possible fixed values?

Response: The value of window size for the standard speculative decoding baseline in each dataset is obtained through searching all possible fixed values and finding the best one. For example, for the OPT 13B-125M pair, the window size for the standard speculative decoding baseline is 4, 6, and 5 for the gsm8k, alpaca, and humaneval dataset, respectively.

Comment 4: BLOOM models achieve ~70% throughput improvement in xsum dataset, while having no speed-up compared with default LLMs without speculative decoding. Why is this throughput improvement so high compared with others ($\sim$10%)?

Response: Conventional speculative decoding suffers a significant slowdown in the xsum dataset. This shows one drawback that speculative methods face. Our method can adaptively change the window size, sometimes to zero, and hence avoid slowdowns.

Comment 5: Is it possible to provide results on a comprehensive chat dataset?

Response: We have included new results in Section 6.4.

Comment 6: How is the vector embedding of a prompt calculated (like u in line 321 and b in line 347)?

Response: A prompt is transformed into a feature vector as the concatenation of several features such as prompt length and perplexity. These details are included in the Prompt Feature paragraph in Section 5.

Comment 7: What is online predictive model construction in line 382?

Reponse: This refers to the reinforcement learning model setup.

Comment 1: Scope for Enhanced Comparative Analysis: The paper covers multiple adaptive techniques to optimize speculative decoding, but a broader comparative analysis could provide more clarity. Highlighting conditions where each method performs best would further aid in understanding the practical strengths and limitations of each approach, helping readers assess their applicability across diverse scenarios.

Response: We appreciate your suggestion regarding the scope for enhanced comparative analysis. While the paper does provide an overview of multiple adaptive techniques for optimizing speculative decoding, we agree that a broader comparative analysis, especially one that highlights the conditions under which each method excels, would further clarify their practical strengths and limitations. Using OPT 13B-125M as an example, the online window size optimization gives overall stability and the best speedups. State-based speculation also gives similar speedups when $\gamma_{\max}$ is large. RL-based speculation performs best when $\gamma_{\max}$ is low. Cache-based and state-based speculation performs best with longer and more structured prompts and thus are more suitable for code completion tasks. Similar observations are present for LLaMA 70B-7B.

Comment 2: How scalable are the adaptive methods presented? Further clarification on scalability could help evaluate the practicality of these techniques in diverse deployment scenarios.

Thank you for your insightful comment. We target diverse deployment scenarios, and the proposed methods are designed with scalability in mind, leveraging adaptability to changes in tasks, models, and runtime conditions. For instance, as our evaluation shows, the methods can dynamically adjust to varying workloads (domain-specific QA, summary, code completion, and mathematical reasoning), hardware resources (different hardware with different memory and bandwidth), and target models (model size ranges from 7B to 70B), which makes them suitable for both small-scale deployments and large-scale systems like those in cloud environments. Furthermore, their reliance on real-time adjustments instead of offline trial-and-error searches enhances their scalability by minimizing setup time and resource overhead.

That said, scalability may vary depending on specific factors such as hardware constraints, the complexity of the inference tasks, and the software stack. In the revised work, we provide more evaluation demonstrating scalability, including a new benchmark MT-Bench and performance analyses across a wider range of deployment conditions (tree-based decoding method), to give readers a clearer understanding of their applicability to different scales and contexts.