Fast Large Language Model Collaborative Decoding via Speculation

We thank the reviewer for their time and insightful comments. Below, we address each concern in detail.

Claims And Evidence 1: compare to PE when model sizes are close

First, we evaluate the speedup of parallel ensemble (PE). However, PE is even slower than the sequential ensemble. For details on the experimental setup, results, and further discussion, please refer to our response to Reviewer hXqY under 'Other Comments or Suggestions'.

Moreover, as you noted, when model sizes are comparable, SE is theoretically expected to perform similarly to or slower than PE. Nonetheless, we maintain that SE remains a promising and broadly applicable approach. Our reasoning is as follows:

1. In practice, PE is limited by its reliance on complex engineering and powerful hardware. It also requires twice the throughput and more GPU memory. Consequently, sequential ensemble is also widely used, as seen in repositories like xydaytoy/EVA, starrYYxuan/UniTE, and cmavro/PackLLM. SE can greatly speedup sequential ensemble without requiring specialized engineering or additional throughput and GPU memory, highlighting its practical potential.

2. Furthermore, if SE is equipped with the same throughput as PE (i.e., double its current throughput), it can be combined with PE to accelerate more models. For example, the 2-model ensemble in Figure 2 can be naturally extended to a 4-model ensemble. The procedural steps for this extension are detailed below, and a schematic illustration is provided in SE with_PE.

   • Step 1: Models $\mathcal{M}_1$ and $\mathcal{M}_2$ are invoked in parallel to produce distributions $p_1^{(1)}(\cdot)$ and $p_1^{(2)}(\cdot)$. A token $x_1^{(1)}$ is sampled from $p_1^{(1)}(\cdot)$ as the proposal, and its score under $\mathcal{M}_2$, given by $p_1^{(2)}(x_1^{(1)})$, is saved.
   • Step 2: Similarly, $\mathcal{M}_3$ and $\mathcal{M}_4$ are invoked in parallel to score $x_1^{(1)}$ and generate bonus distributions $p_2^{(3)}(\cdot)$ and $p_2^{(4)}(\cdot)$. A bonus token $x_2^{(3)}$ is sampled from $p_2^{(3)}(\cdot)$, and its score under $\mathcal{M}_4$, i.e., $p_2^{(4)}(x_2^{(3)})$, is saved.
   • At this stage, $x_1^{(1)}$ has been scored by all 4 models and proceeds to verification. If accepted, $x_1^{(1)}$ is treated as a sample from the ensemble distribution, as illustrated in Step 3.
   • Step 4: The process is repeated with $\mathcal{M}_1$ and $\mathcal{M}_2$ invoked again to score $x_2^{(3)}$ and generate new bonus distributions, mirroring the procedure in Step 2.

   The algorithm retains the properties discussed in the paper, including losslessness, and it is never slower than the 4-model PE.

   Note that the method described here is a preliminary version and can be further optimized. For instance, in Step 1, we can sample two tokens from both $p_1^{(1)}(\cdot)$ and $p_1^{(2)}(\cdot)$, and verify them in parallel using a tree attention to further improve the acceptance rate. We will explore this in future work to further enhance the performance of SE.

Claims And Evidence 2: compare to PE when model sizes differ

When the model sizes differ greatly, speedup is mainly coming from reducing the number of large model invocations.

In the optimal case, PE reduces only the computational time of the small model, while the large model remains a major time bottleneck. In contrast, SE—under setting shown in Figure 2—can halve the number of large model invocations, resulting in a substantially more effective speedup.

On the other hand, the speedup comes from allowing the smaller model focuses on sequential generation, while the larger model focuses on parallel verification. This shares a common acceleration principle with SD.

As noted on Page 5, Line 251 (left column), two hyperparameters, $\gamma_p$ and $\gamma_q$, control the proposal length for each model when acting as the proposer. When model sizes differ greatly, the larger model’s $\gamma$ is set to 1, and the smaller model is assigned a higher value (e.g., 5), as described in Section 4.1, under “Configuration of $\gamma$”. As a result, the smaller model is invoked more frequently—about 5 times as often as the larger model—rather than "half" of the total invocations. This setup ensures that the smaller model focuses on sequential proposal generation, while the larger model conducts parallel verification, echoing the acceleration strategy of vanilla SD. This is further supported by the ablation study in Figure 6(b), which demonstrates that speedup consistently increases with higher $\gamma$ values of the small model.

Experimental Designs Or Analyses:

We also provide a case to better illustrate how SE achieves greater speedup compared to SD; please see our response to Reviewer hXqY under "Questions for Authors".

If you have any questions or concerns, please let us know. We’re committed to addressing them.

We thank the reviewer for their time and insightful comments. Below, we address each concern in detail.

For clarity and brevity, we use the following abbreviations: Experimental Designs Or Analyses (EDOA), Weaknesses (W), and Questions (Q).

W1: Speculative Ensemble (SE) offers two non-trivial improvements over vanilla SD.

Compared to simply adapting vanilla SD to ensemble scenarios, SE offers two non-trivial improvements:

1. Compared to vanilla SD, where one model consistently serves as the proposer and the other as the verifier, we introduce an alternating proposal framework (APF) in Section 3.3, as shown in Figure 2. This method switches the roles of proposer and verifier during decoding, and is specifically designed to enhance efficiency in ensemble settings. We also validate its effectiveness through ablation experiments shown in Ablation of APF. For a case-specific explanation, please refer to our response to reviewer hXqY under 'Questions for Authors'.
2. Unlike the vanilla SD, which employs two models, the proposed SE extends to an n-model scenario to accelerate the n-model ensemble in Section 3.4, as shown in Figure 3 and Figure 7.

Moreover, Reviewer hXqY described our method as “clever, novel, and well-validated,” while Reviewer 3eWy referred to it as a “novel framework,” both highlighting the non-trivial improvements offered by our approach.

W1: concerns about general applicability

Our proposed SE not only accelerates traditional weighted ensemble (WE) methods (Eq 4), but also accelerates ensembles of any form, size, or number of LLMs at the probability or logits level. This includes techniques such as contrastive decoding (CD) [1] (Eq 5) and decoding-time realignment [2], demonstrating its broad applicability. As Reviewer 3eWy noted, SE “can be useful in various application,” further supporting this point.

W2: concerns about theoretical novel insights

While our theoretical analysis of correctness (i.e., losslessness) and speedup follows original SD paper, we introduce two novel theoretical insights tailored to the ensemble scenario:

1. As discussed on page 4, line 205 (left column), in vanilla SD, estimating the expected speedup requires extensive experiments to estimate the acceptance rate $\alpha$. However, in WE setting, the parameter $\lambda$ (Eq 4) naturally serves as a lower bound for $\alpha$. This allows us to estimate the speedup in advance and choose a suitable proposal model, such as those described in Corollary 3.5.
2. Vanilla SD does not guarantee acceleration; in fact, it may result in slower speed when $\alpha$ is small. In contrast, SE is theoretically proven to be no slower than the standard ensemble, even in the worst case, as established in Corollary 3.7.

Q1, 3, 4 & W3, 4: compare with more baselines

Thank you for your valuable suggestions. To better demonstrate SE, we conducted experiments comparing SE with three non-ensemble baselines: Large Model, vanilla SD, and Medusa. The results are shown in Non-ensemble. Note that for Medusa, we only reported the results for Vicuna, because it only provides the pretrained draft model for Vicuna. For Qwen-3B, we did not report results for vanilla SD, because there is no suitable draft model for it.

Additionally, it is important to note that SE, as an acceleration algorithm, is a line of research parallel to SD. It does not focus on finding a quality-speed trade-off. Instead, SE focus on accelerates inference while preserving the performance gains provided by ensemble.

EDOA1 & Q2: results for models larger than 7B and the impact of model size

As described in Section 4.1, "Model Pair Configuration" and in Table 1, we evaluated the CD setting using the model pair (OPT-13B, OPT-125m). The corresponding results are presented in Table 4.

In addition, we also tested a larger model pair (Llama-2-13B, Vicuna-13B) under the WE setting. The results are shown in Larger Model.

Together with the results in Tables 2 and 3, we observe that SE achieves greater speedup as model size increases in both the WE and CD scenarios, with a more pronounced effect in CD. In WE, this trend may be due to improved model performance with larger sizes, which increases similarity between models and leads to a higher acceptance rate. In the CD scenario, the growing speedup may stem from the increasing cost of invoking the larger model.

If you have any further questions or concerns, please feel free to let us know. We are committed to addressing any concerns to the best of our ability.

[1] Contrastive Decoding: Open-ended Text Generation as Optimization

[2] Decoding-time Realignment of Language Models

We thank the reviewer for their time and insightful comments. Below, we address each concern in detail.

For brevity, we use the following abbreviations: Claims and Evidence (CAE), Methods and Evaluation Criteria (MAEC), Theoretical Claims (TC), Experimental Designs or Analyses (EDOA), Weaknesses (W), and Questions for Authors (Q).

CAE & MAEC1 & TC & EDOA1 & W2 & Q1: concerns about generation quality

We apologize for the lack of clarity regarding generation quality. In the SD domain, performance is theoretically well-established, so researchers typically focus on comparing speed rather than performance [1] [2] [3]. Our approach follows this standard practice.

From both theoretical and experimental perspectives, we confirm that SE consistently maintains ensemble quality.

Theoretically, as shown in Appendix A.1, we proved that the generated tokens precisely follow the ensemble distribution—that is, they can be regarded as samples from this distribution. Reviewers hXqY, A7ww, and 3eWy also endorsed the correctness of our proof.

To further validate this, we conducted additional experiments. As shown in Generation Quality, when T=0, the performance is exactly the same (as proved in Section 3.2 of [4]), while when T=1, due to randomness, the performance shows slight differences but remain largely consistent. For the weighted ensemble (WE), we report only T=1 case, as T=0 is uncommon in WE setting, as discussed in Section 4.1 "Ensemble functions and methods".

MAEC1 : concern about Corollary 3.5

Corollary 3.5 states that for any given $\lambda$, there exists a $\gamma$ that guarantees an ensemble speedup. Notably, $\lambda$ is not a hyperparameter of the SE algorithm; instead, the proposal length $\gamma$ is. As an acceleration method, SE aims to speed up the ensemble under a given $\lambda$. The selection of $\lambda$ follows the same procedure as in standard ensembles—that is, it is set to the value that yields the best performance. For example, [5], [6], and [7] respectively set $\lambda$ based on the trade-off between model purification and standard performance, the performance on a development set, and the perplexity.

In addition to accelerating common weighted ensembles (Equation (4)), the proposed SE can also speed up any form of probability- or logits-level ensembles (Equation (3)). In this more general setting, Corollary 3.7 provides a theoretical guarantee of SE's acceleration.

MAEC2 & EDOA3 & W1: compare with non-ensemble baselines

Regarding this, please refer to our response to Reviewer m8L7 under "Q1, 3, 4 & W3, 4".

Q2: the practicality of ensemble small and large models

Regarding this question, please refer to our response to Reviewer A7ww under "MAEC1".

EDOA2: concerns about ablations

First, the quality-speed tradeoff in Appendix C.1 is not a central focus of the paper, please refer to our response to Reviewer A7ww under "MAEC2" for a more detailed explanation.

Second, we did not separate the impact of speculative decoding and ensemble benefits because SE is designed specifically for ensemble scenarios, rather than treating ensemble as a separate component. While it is well known that ensemble can enhance LLM performance, it typically incurs slow inference. SE aims to accelerate inference while preserving these performance gains.

W3 & Q3: concerns about alternate proposal

First, please refer to our response to Reviewer hXqY under "Questions For Authors" for an example illustrating the impact of using the alternate proposal framework (APF). As shown, APF enables the generation of an additional bonus token within the same number of model invocation, thereby enhancing efficiency.

Second, our ablation study confirms the effectiveness of APF. The results are shown in Ablation of APF.

Third, as detailed in Section 4.1, "Ensemble Functions and Methods" and Table 1, we evaluated 4 groups of comparable model configurations (e.g., Llama-2-7B + Vicuna-7B). The results in Table 2 show that APF can still enhance efficiency if the drafter and verifier models are of comparable sizes.

If you have any further questions or concerns, please feel free to let us know. We are committed to addressing any concerns to the best of our ability.

[1] Fast Inference from Transformers via Speculative Decoding

[2] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

[3] GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding

[4] Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation

[5] Purifying Large Language Models by Ensembling a Small Language Model

[6] Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration.

[7] Relative representations enable zero-shot latent space communication

We thank the reviewer for their time and insightful comments. Below, we address each concern in detail.

For clarity and brevity, we use the following abbreviations: Methods and Evaluation Criteria (MAEC), Theoretical Claims (TC), Experimental Designs or Analyses (EDOA).

MAEC1: contrastive decoding is the ensemble function in our setup when model sizes differ

As discussed in Section 4.1 "Ensemble Functions and Methods" and summarized in Table 1, for model pairs like Llama-3, -2, and OPT, where the proposer and verifier differ a lot, we applied contrastive decoding (CD) [1] rather than the traditional weighted ensemble (WE).

CD enhances token generation quality by subtracting the logits of a smaller model from those of a larger one, as defined in Equation (5). When combined with CD, LLMs generally achieve improved performance. To support this, we present results using Llama-3 and -2 with CD, and compare them to the large model baseline, as detailed in CD.

Therefore, when model sizes differ, our method is not intended to trade performance for speed. Instead, our focus is, when an ensemble method (e.g., CD) improves LLM performance, SE can further accelerate the ensemble while preserving these gains.

Additionally, it is important to point out that, as shown in Section 4.1 ("Ensemble Functions and Methods") and Table 1, we also evaluated the speedup of SE under the WE setting when model sizes are similar. The results are presented in Table 2.

MAEC2: about weighted ensemble when model sizes differ in Figure 8

Although this paper primarily focuses on CD when model sizes differ, we also introduce WE with different model sizes as an exploratory approach to the quality-speed tradeoff. This discussion appears on page 4, line 203 (right column), with the corresponding results shown in Figure 8. However, this is not a central focus of the paper; rather, we present it as a potential strategy for achieving faster acceleration in vanilla SD—specifically, by applying WE with the proposal model and using SE.

In this context, $\lambda$ serves as a hyperparameter that governs the quality-speed tradeoff. One possible selection strategy is to choose the largest $\lambda$ such that the ensemble performance exceeds a predefined threshold. As shown in Figure 8, a larger $\lambda$ indicates greater acceleration. Therefore, this selection strategy ensures that the highest speed is achieved while meeting the performance requirements.

MAEC3: concerns about verifying bonus tokens from small model

First, as previously noted, when model sizes differ, our primary focus is on CD. This ensemble approach can outperform a single large model. Therefore, verifying bonus tokens is meaningful.

In the context of WE with different model sizes, as you noted, verifying bonus tokens from the large model typically leads to reduced performance. Therefore, using the bonus tokens directly without verification is a more suitable approach. We appreciate your constructive feedback and will reflect it in the revised version. That said, our design prioritizes interpretability—specifically, ensuring that the distribution of generated tokens remains controllable and consistent. Consequently, in some cases, it remains beneficial to verify bonus tokens produced by the large model. For instance, as mentioned on page 5, line 231 (left column), some studies suggest that appropriately ensembling a smaller model can enhance safety.

TC: about Corollary 3.5

The term "swap" in Corollary 3.5 does not refer to the dynamic role alternation between proposer and verifier in the alternate proposal framework.

Corollary 3.5 is specifically formulated for the Speculative-based Ensemble, which includes only the methods introduced in Section 3.2 and excludes the alternate proposal framework. It states that when the proposer and verifier are fixed, choosing an appropriate proposer model before inference begins can ensure acceleration. For example, when ensembling $\mathcal{M}_p$ and $\mathcal{M}_q$, if using $\mathcal{M}_p$ as the proposer does not guarantee acceleration, then $\mathcal{M}_q$ must. The "swap" referenced in Corollary 3.5 refers to this initial selection at the start of inference and does not occur dynamically during the inference process.

Additionally, Corollary 3.7 guarantees the acceleration of SE under the alternate proposal framework. This corollary provides a more general result that ensures acceleration for any form of ensemble, not just WE.

EDOA & MAEC2: compare to single model SD

Regarding this, please refer to our response to Reviewer m8L7 under "Q1, 3, 4 & W3, 4".

If you have any further questions or concerns, please feel free to let us know. We are committed to addressing any concerns to the best of our ability.

[1] Contrastive Decoding: Open-ended Text Generation as Optimization

We sincerely thank the reviewer for the time and effort spent reviewing our submission and greatly appreciate your insightful comments and constructive suggestions. Below, we have done our best to address each of your concerns in detail.

Methods And Evaluation Criteria: report the "raw speeds"

We sincerely thank you for your insightful suggestions. To better showcase SE's performance, we have reported raw generation speeds, as presented in Raw Speed.

As you correctly noted, SE is the first method proposed to accelerate any form of LLM ensemble. We have extended SD baselines to the ensemble scenario where possible and included them in our comparisons. Other SOTA methods in SD, such as EAGLE [1] and Medusa [2], might also be adapted to ensemble scenario, but doing so would require specific modifications, which are beyond the scope of this work.

Other Comments Or Suggestions: compare with the very strong baseline

Thank you for your valuable suggestion. For clarity, we refer to the strong baseline you identified as the parallel ensemble (PE).

First, we use the popular LLM ensemble repository yaoching0/GaC to test the speed of PE and implement a corresponding sequential ensemble to compute speedup. However, as shown in Parallel Ensemble, PE is even slower than the sequential ensemble.

The inefficiency likely stems from overly frequent communication. In particular, during the generation of each token, PE requires communication between the main node and two GPUs. Because the time to generate a single token is very short (especially when kv cache is enabled) the communication overhead becomes significant, introducing noticeable latency. In contrast, standard sequence-level parallelism avoids this issue, as the time to generate an entire sequence is much longer than the communication time, rendering the overhead negligible. These findings further underscore the importance of SE. Note that the speed reported in the original GaC repository was measured with kvcache disabled. In contrast, our tests were conducted with the kvcache enabled, as it is commonly used in current LLM inference. Therefore, our setup provides a more realistic evaluation.

Additionally, the speedup achieved by PE is closely tied to its implementation and the underlying hardware. With improved engineering and more powerful hardware, PE could potentially attain greater speedup. Nonetheless, we maintain that SE remains a promising approach. For a detailed explanation, please see our response to Reviewer 3eWy under "Claims and Evidence 1".

Questions For Authors: the SD method in Tables 2 and 3

We apologize for the earlier lack of clarity.

We use an example to clarify this SD process: In each cycle, suppose the proposal model $\mathcal{M}_q$ sequentially generates 5 tokens, which are then verified in parallel by the target model $\mathcal{M}_p$, resulting in 6 distributions, where the first 5 ones correspond to the verified distributions of 5 proposal tokens, while the 6-th is the bonus distribution. Since the 6-th bonus distribution lacks a corresponding distribution from $\mathcal{M}_q$ to compute the ensemble distribution, we cannot directly sample the bonus token from the ensemble distribution. To resolve this, we invoke $\mathcal{M}_q$ again to produce the 6-th distribution. Finally, we apply the standard accept-reject criterion in SD.

Compared to this SD baseline, SE addresses this by treating the 6-th token as a proposal from $\mathcal{M}_p$ and verifies it using $\mathcal{M}_q$. This verification not only obtains the 6-th distribution from $\mathcal{M}_q$, but also generates an additional bonus token from $\mathcal{M}_q$. In the next cycle, only 4 invocations to $\mathcal{M}_q$ are needed to form the proposal, thus reducing one invocation of $\mathcal{M}_q$ compared to the SD in Tables 2 and 3.

As described above, the SD in Tables 2 and 3 does not throw away the bonus token; instead, it utilizes the token in the same manner as the standard SD. However, as previously discussed, SE utilizes bonus tokens more efficiently. With the same number of model invocations per cycle, it can generate one additional bonus token, resulting in greater acceleration.

If you have any further questions or concerns, please feel free to let us know. We are committed to addressing any concerns to the best of our ability.

[1] EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

[2] Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads