polybasic Speculative Decoding Through a Theoretical Perspective

Thank you for these insightful questions that help clarify the practical implications of our theoretical framework.

Regarding Theorem 3.2's sensitivity: We recommend measuring the relevant parameter values under identical experimental conditions. While measurement errors may occur, our theoretical framework is robust against minor measurement inaccuracies because when comparing inference time changes after adding models, the errors are relative. This relative nature of measurement ensures that our framework's predictions remain reliable even with some experimental noise.

On implementing multi-model systems: The primary challenge in extending to multi-model systems is KV cache management. This can be addressed using techniques similar to those in MagicDec, such as StreamingLLM technology, which preserves only initial tokens (Attention Sinks) and KV caches within a sliding window, significantly reducing memory requirements. Our theoretical framework guides model selection and is orthogonal to other speculative methods in its approach. We plan to enhance our system in future work.

Stability in long-context scenarios: Theorem 3.3's variance analysis strengthens our theoretical framework by reducing errors from acceptance token length variations during model selection, maintaining stability across all tasks. For long-context tasks, effective KV cache management is clearly essential, and our framework provides a stable foundation upon which these optimizations can be built.

Comparison with recent works: Thank you for your suggestion - we have expanded our experimental comparison to include Hydra, which further validates the effectiveness of our approach across different speculative decoding implementations. ($c$: speed ratio, $μ$: average acceptance length)

In conclusion, we appreciate your recognition of our work and your thorough review. We hope these responses address your questions and highlight the contributions of our theoretical framework to the understanding and optimization of speculative sampling systems.

1. On the Connection Between Theory and Experiments

We appreciate your concern about the gap between our theoretical foundation and experiments. To address this, we've conducted specific experiments that directly validate our theoretical framework:

For Case 1:

• $ith$：vicuna-7b, $new$: vicuna-1b, $(i+1)th$: EAGLE
• $T_{new}/T_i$ = 0.80
• $L_{new }× (1/L_i - 1/L_{i-new})$= -0.117
• Since 0.80 > -0.117, Theorem 3.2 predicts performance decrease

For Case 2:

• $ith$：vicuna-7b, $new$: W4A4 vicuna-7b, $(i+1)th$: EAGLE
• $T_{new}/T_i$ = 0.318
• $L_{new} × (1/L_i - 1/L_{i-new})$= 0.330
• Since 0.318 < 0.330, Theorem 3.2 predicts performance improvement

These results confirm our theoretical framework's ability to guide model selection decisions for speculative decoding systems.

2.Expanded Model Experiments

Following your recommendation, we have extended our experiments to include Vicuna-13B and Llama-2-Chat-70B models, both showing significant acceleration effects.

3. Theoretical Formulations and Symbol Definition Issues

We appreciate your detailed feedback on our theoretical formulations. We will revise the identified errors and provide clearer explanations for the mathematical notations and symbols used throughout the paper.

Equation 2:

In Equation $T = \sum_{i=1}^n F_i \cdot T_i$, we combine prefill and decoding costs into a single term $T_i$ to simplify model selection for speculative decoding. This focuses on comparing efficiency when adding models rather than precisely calculating absolute inference times. Our experimental results (like Vicuna-7B's $4.43\times$ speedup) validate this approach. While separating costs ($T_i = T_i^{\text{prefill}} + T_i^{\text{decode}}$) could add precision, it wouldn't change our core theoretical conclusions. We'll clarify this rationale in the final manuscript.

Line 157 and MagicDec:

Line 157 presents a binary form of Equation (2). We appreciate the reference to MagicDec - our theoretical approach indeed aligns with theirs, though we chose our formulation for conciseness and clearer derivations.

For equivalence:

• MagicDec: Speedup = $\Omega \cdot \frac{T_T}{\gamma T_D + T_V}$
• Ours (dual-model): Speedup = $\frac{L_1 T_T}{T_T + T_D}$

By mapping $\Omega \leftrightarrow L_1$ and $\gamma T_D + T_V \leftrightarrow T_T + T_D$, the formulas are mathematically equivalent.

Some symbols and equations were not clear

Regarding line 169 (the positioning of "new" between i and i+1): As mentioned in our paper's second contribution, our theoretical framework aims to discover conditions under which adding auxiliary models can improve inference speed in speculative sampling systems. Intuitively, we should insert a model with stronger inference capabilities between the draft model and target model to bring the composite draft model's capabilities closer to the target model.

line 174 (the symbol $α$): The paper mentions that acceptance probability $p = 1 - α$, where α represents the rejection probability.

line 191 (the symbol $S$): $S$ is simply an intermediate summation value used for notational convenience, which we didn't explicitly define given space constraints. You also noted that equations on page 4 are unnumbered - we chose not to number equations that are merely part of proofs and not essential to the paper's central theoretical framework.

Theorem 3.3 We realize that our use of "$n$-model" may have caused confusion. In this context, "$n$" refers to "a truncated geometric distribution of maximum $n$ trials" (line 189). We adopted this notation to maintain consistency with SpecDec. In our revised manuscript, we will add a clarifying statement at this location to prevent any misunderstanding.

4. Comparison with Other Cascade Methods

We did not experimentally compare our approach with other cascade methods because the current state-of-the-art in speculative sampling is EAGLE, which significantly outperforms these cascade methods in terms of acceleration. Therefore, we focused our comparison on EAGLE.

5. Other Issues

Regarding Algorithm 1, the note "(continue accumulating tokens)" was intended to indicate continuation of the Draft execution in line 7. We will remove the else branch to ensure the correctness of the pseudocode.

Thank you for your thorough review. We hope our responses clarify your concerns and demonstrate our paper's contributions. We respectfully request an improved score. Maybe it's unclear due to word limit, please communicate with us if you have any further questions.

1. Response on Accuracy Evaluation

We appreciate your highlighting the importance of accuracy evaluation. This is indeed a critical point that deserves attention.

Our method inherently preserves output distribution through the verification mechanism in Algorithm 1. The VERIFY procedure ensures draft tokens are only accepted when they align with the target model's distribution. When verification fails, we fall back to sampling directly from the target model, guaranteeing identical outputs to standard autoregressive generation. This preservation-by-design is why speculative decoding methods typically focus on speedup metrics rather than accuracy. We followed standard performance metrics in this field to maintain consistency with prior work. Thank you for this valuable suggestion.

2. Response to Reviewer Concerns on Model Configuration

Thank you for highlighting these important questions about our paper's consistency and practical focus. We appreciate the opportunity to clarify based on our contributions and motivations:

Theoretical Framework: Our core contribution is establishing a theoretical framework for speculative sampling that enables the construction of multi-model speculative systems. When models satisfy appropriate conditions, we can extend from binary to multi-model configurations.

Three-Model Implementation: The three-model system represents a concrete instantiation of this framework, chosen because it achieves an optimal balance between theoretical complexity and practical feasibility. Naturally, this led us to explore extensions to more models, along with their inherent limitations.

Focus on Single LLM Inference: We fully agree with your point about optimizing inference latency for a single LLM. Speculative sampling works by having draft models generate multiple candidate tokens at once, which the target LLM then verifies in a single forward pass. This reduces the number of target model executions, thereby optimizing overall inference latency. Our discussion of four-model systems aims precisely at further improving LLM inference speed, aligning with your suggestion.

Training Considerations: Our approach has minimal training requirements. We can utilize quantized versions of existing models or off-the-shelf smaller models, making our method orthogonal to and compatible with most existing speculative sampling approaches.

Practical Value: The main value of our work lies in providing a theoretical framework that enables researchers to systematically optimize multi-model speculative decoding systems, with demonstrated practical improvements in reducing target LLM inference latency.

3. Response to Terminology Inconsistency

Thank you for highlighting the inconsistency in our reference to baseline techniques. We apologize for the confusion this has caused.

We acknowledge the error in our notation. Throughout the paper, we intended to reference EAGLE-2 consistently as our baseline, as it represents an improvement over EAGLE-1.

In the revised version, we will:

1. Standardize all references to EAGLE-2 throughout the manuscript
2. Add proper citations to both EAGLE-1 and EAGLE-2
3. Include a brief explanation clarifying that EAGLE-2 is an advancement of EAGLE-1, which is why we selected it as our baseline for comparison

Thank you for your thoughtful review of our paper. We genuinely appreciate both your critical observations and the opportunity to address them.

Based on the clarifications provided, we respectfully request that you reconsider your evaluation of our paper. We remain available to address any additional questions or concerns you might have.

We sincerely thank you for your thoughtful feedback and the positive evaluation of our work.

1. Experiments on Larger Models (13B, 70B)

Following your suggestion, we have conducted additional experiments with Vicuna-13B and LLaMA-2-chat-70B. ($c$: speed ratio, $μ$: average acceptance length)

We initially focused on smaller models to ensure all experiments could be run on a single GPU, facilitating direct comparison with existing methods. This is consistent with the evaluation approach in most speculative decoding literature. However, we agree that demonstrating scalability to larger models strengthens our contributions.

2. Four-Model System Evaluation

Concerning the four-model system evaluation, we have implemented a prototype following our theoretical guidance. Using LLaMA-2-7B as the target model with three draft models of decreasing capacities (4-bit quantized LLaMA-2-7B, 4-bit quantized LLaMA-2-1B, and EAGLE), we observed the following results:

We note that diminishing returns become apparent as more models are added. This is because finding models that satisfy our Theorem 3.2 becomes increasingly challenging. Our future work will focus on quantization, sparsification, and KV cache optimization to reduce model inference time. We appreciate your suggestion, which has further advanced our work.

We thank the reviewer for these valuable suggestions and have incorporated these additional experiments and discussions in the final version of our paper.