ParallelSpec: Parallel Drafter for Efficient Speculative Decoding

This is a reminder for the authors to provide a response for the reviews above. There are various weaknesses and questions to be addressed and your response determines whether a raise of the overall score can be conducted. I really like this paper and I would like to hear some answers from the authors.

(Q4) Compatibility with dynamic draft trees.

Thank you for bringing this up. Although the main contribution of this paper is not the token tree verification strategy—our token tree, as mentioned in Appendix A.3, is the same as those used in EAGLE and Medusa—we believe that our approach is fully compatible with the confidence-guided dynamic token tree proposed in EAGLE-2, with no conflicts between the two. We appreciate your understanding that combining ParallelSpec with the dynamic token tree is not feasible during the discussion phase due to the extensive code modifications required. That said, we are glad to report the results of EAGLE-2 on Spec-Bench in Table 1 in the revised version.

We sincerely appreciate your comments, which helped improve our paper, and we hope our responses above helped to address your concerns. Please feel free to leave us more comments if you have any questions, and we are always happy to engage in discussions.

Dear Reviewer YUuo,

We are grateful for your positive recognition and continuous interest in our work. We would be happy to address your concerns as below.

(W1) Competition between Medusa and Medusa+ParallelSpec is not fair

As you mentioned, the improvements our method brings to Medusa with the new drafter structure and the introduction of parallel drafting may obscure the actual performance gains achieved solely through parallel drafting.

However, as Reviewer bvJa pointed out, the Medusa drafting stage could be considered a parallel decoding stage to some extent, as there is no information exchange between the different language model heads when guessing future tokens. Therefore, we think simply adding parallel drafting to Medusa can not be seen as a significant scientific contribution since the only thing needed is to parallelize the inference of multiple language model heads.

We agree that Medusa-ParallelSpec creates confusion that it still preserves the original multi-head drafter design. The difference we would like to highlight is that Medusa-ParallelSpec is trained only with Medusa loss, i.e. token-level classification loss over the vocabulary, while EAGLE-ParallelSpec is trained with EAGLE-specific feature-level regression loss together with classification loss. We will have another name on Medusa-ParallelSpec in the next version to avoid such confusion.

(W2 & Q1) EAGLE adaptation with ParallelSpec is not detailed.

Good question! As you described, for the original EAGLE drafting, the features are autoregressively regressed with the input of last-step embedding and features from target models (i.e., target features). In the design of EAGLE-ParallelSpec, each drafting step uses only the embedding of [MASK] tokens with the target features of [MASK] tokens left empty, i.e. $\tilde{f}\_t, \ldots, \tilde{f}\_{t+K+1}=\pi\left(\left[f\_{<t}, 0, \ldots 0\right]; \left[e_{<t},\left[M A S K\_1\right], \ldots,\left[M A S K\_K\right]\right]\right)$.

The reason behind that is these introduced [MASK] tokens are not in the original vocabulary of the target models; therefore, there is no way to produce meaningful target features for these [MASK] tokens. We only use the trainable embeddings of [MASK] as a signal for the drafter to predict tokens at different future time steps.

We apologize for not describing the inference-time adaptation in the paper, and we think this is crucial for readers to understand EAGLE-Parallel.

(Q2) Inconsistent baseline results with Spec-Bench.

We are also aware of the differences between baseline methods reported in ParallelSpec and Spec-Bench. Reviewer bvJa also raised similar concerns. Here are some efforts we made during the discussion stage:

1. We reached out to the Spec-Bench team for their hardware specifications when reporting results in the Spec-Bench paper. We confirmed that their team used the A100 SXM4 version to perform the evaluation, and our team used the A100 PCIE version. SXM4 version has a significantly higher rated thermal power consumption (400W v.s. 250W in the PCIE version). Also, SXM4 has a different memory bandwidth. We re-examined the evaluation log and confirmed that in our evaluation environment described in L810-L812, the Spec-Bench toolkit works as expected to report faithful results in Table 1.

2. We re-ran the Medusa method on RTX-4090 GPU and reported the results in the revised submission (L330). ParallelSpec still brought significant relative improvement in terms of speedup ratio when evaluating with a different GPU.

3. We appreciate your understanding that, due to budget and practical constraints, we do not have access to the same computational resources as the Spec-Bench team. This challenge is a common issue in the SPD community when reporting speedup results, as the essence of speculative decoding is to exploit redundant computation intensity in hardware accelerators.

4. However, since all the improvements in this paper were obtained using the same hardware environment, our comparisons are relatively fair. We also notice that in the Spec-Bench paper, Table 5 and Table 6 also reported entirely different sets of speedup ratios when the only difference is the GPU used for benchmarking, indicating the hardware specifications are one of the major factors that impact reported speedup. We add explanations in L869 to discuss this issue.

(Q3) Organization of Table 2.

Thanks for suggesting this. We also think separating the original Table 2 will improve the clarity of our paper's presentation. Please see pages 9 and 10 for the revised table.

Dear Reviewer y5aG,

Thank you for your insignificant comments. We believe the submission could greatly benefit from them. We would like to address your concerns below.

Parallel Decoding is not novel. How was our work different from Stern et al. and Xia et al.?

We appreciate your comments on the novelty issue, and Reviewer bvJa raised similar issues. Medusa and Stern et al. initiated the use of multiple language model heads for multi-token prediction, inspiring recent efforts such as “Better & Faster Large Language Models via Multi-token Prediction.” ParallelSpec, on the other hand, adopts a compact transformer model as the drafter, using [MASK] tokens at different positions to guide predictions.

Xia et al. proposed SpecDec earlier in 2022. It has not leveraged significant advancements in the speculative decoding community in recent years, such as speculative sampling, token tree verification from SpecInfer, and feature-level regression in EAGLE. Its drafter and target model were still based on an encoder-decoder architecture. Most importantly, it relaxed the verification criteria, which prevented it from achieving lossless speedup. This is reflected in the evaluation, where BLEU scores are inconsistent with naive greedy decoding. We believe that ParallelSpec is one of the works that achieve parallel decoding using a standalone trainable drafter while maintaining lossless speedup.

Missing results on larger target models, e.g. LLaMA-2-70B.

Thanks for raising the issue regarding larger target models. While we are unable to provide results for ParallelSpec on a 70B model during the discussion phase due to resource and time limitations, we would like to clarify that the issue you mentioned is likely common across most draft-then-verify frameworks.

Specifically, when the target models grow to 70B, it is customary to scale up the drafter accordingly. For instance, in the speculative sampling framework, a LLaMA-2-7B drafter would generally be a better fit for a LLaMA-2-70B target model than a LLaMA-68M drafter considering the trade-off between accuracy and overhead. The same principle applies to parallel speculation frameworks: for a 70B target model, the drafter would typically be scaled up to 1B, a single transformer layer of LLaMA-2-70B. Autoregressively forwarding a 1B drafter still incurs a non-negligible computational cost.

Training ParallelSpec is less efficient than EAGLE.

We must acknowledge that, due to the special training strategy, the training of the parallel drafter is not as fast as that of the autoregressive drafter. In group-wise parallel training, each dynamically adjusted output token embedding alongside multiple [MASK] embeddings must simultaneously be aligned with several future token embeddings, unlike autoregressive training, which only focuses on the next token. This extends the training sequence length and results in more training FLOPs. As a reference, training EAGLE for the same number of epochs on an 8xA100 PCIe 40GB setup took around 10 hours, compared to 13 hours for ParallelSpec. We will explicitly point out this limitation of ParallelSpec in L387 of the revised version.

Updated EAGLE-2 results.

We are glad to report the results of EAGLE-2 on Spec-Bench in Table 1 in the revised version L330. We also kindly refer you to the latest Spec-Bench Leaderboard (https://github.com/hemingkx/Spec-Bench/blob/main/Leaderboard.md) for EAGLE-2 results on Spec-Bench hardware.

Misleading Figure 1 presentation about wall time trace.

We revised the caption to clarify that the saved latency could only be achieved if drafting accuracy is the same for both drafting styles given the prefix sequence.

We sincerely thank you again for your efforts in reviewing which helped a lot in improving our paper. We hope our responses above could address your concern and please feel free to leave any comments if you have. We are always happy and open to discussions.

(To continue)

4. However, since all the improvements in this paper were obtained using the same hardware environment, our comparisons are relatively fair. We also notice that in the Spec-Bench latest benchmark, Table 5 and Table 6 also reported entirely different sets of speedup ratios when the only difference is the GPU used for benchmarking, indicating the hardware specifications are one of the major factors that impact reported speedup. We add explanations in L869 to discuss this issue.

Other action items (Q1-Q8)

We are grateful to the comprehensive comments for our paper. We made the following changes to respond to your suggestions.

1. Figure 1 has been revised to highlight the difference between the two drafting styles.
2. We revise the claims in the abstract and at the end of the introduction to reflect relative improvements for both models.
3. Related Works and Background Sections have been updated to incorporate the suggested references and fix the notation issue.
4. We revise L395-L396 to reflect the details of SpS implementation. In short, we followed Spec-Bench to use Huggingface Transformers assisted_generation module to benchmark the SpS method, which has a heuristic-driven $\gamma$ adjusting approach implemented in https://github.com/huggingface/transformers/blob/4e90b99ed916300b80bac9db793f2a96b2a87122/src/transformers/generation/candidate_generator.py#L238-L262.

Thanks again for your review to make this paper better. We hope our responses above could address your concerns and please free feel to leave more questions if you have. We are always happy to engage in discussions.

Dear Reviewer bvJa,

Thank you for the constructive feedback, especially the detailed correction in Algorithm 1! We would like to introduce our response to your concerns and the status of the action items.

(W1) Novelty Issue

We would like to address your concerns regarding the novelty issue compared with Monea et al， PaSS and Medusa. We view these works as significant contributions to the parallel decoding field. Medusa and Stern et al. initiated the use of multiple language model heads for multi-token prediction, inspiring recent efforts such as “Better & Faster Large Language Models via Multi-token Prediction.” ParallelSpec, on the other hand, adopts a compact transformer model as the drafter, using [MASK] tokens at different positions to guide predictions.

We recognize that this approach may appear similar to PaSS. However, we believe that depending solely on several trainable lookahead tokens without other trainable parameters, as in PaSS, makes it difficult to develop a drafter that is both accurate and efficient. As you noted, our design employs a separate drafter for parallel drafting, supported by a sophisticated training strategy to ensure high training efficiency.

(W2) Reproducibility issue regarding the draft token tree

We apologize for not clearly stating the process of constructing the token tree used in ParallelSpec. In fact, in all experiments of ParallelSpec, we adopt the same static token trees used in Medusa-1 and EAGLE, with the structure illustrated in Figure 6. The pseudo-code for the structure is listed below:

mc_sim_7b_63 = [[0],[1],[2],[3],[0,0],[0,1],[0,2],[1,0],[1,1],[2,0],[2,1],[3,0]
               ,[0,0,0],[0,0,1],[0,0,2],[0,1,0],[0,1,1],[0,2,0],[0,2,1],[1,0,0],
               [0,0,0,0],[0,0,0,1],[0,0,0,2],[0,0,0,0,0],[0,0,0,0,1]]  # depth of 5. in total 25 nodes waiting for verification

We think the primary contribution of this paper does not lie in proposing a novel token tree verification strategy; therefore, using the same token tree ensures that the relative improvements can be fairly attributed to the parallel drafter design. We would also like to report additional results on EAGLE-2 in L330 of the revised version, where we can see even a context-dependent dynamic draft tree provided marginal improvements on Spec-Bench. We revised Appendix A.3 to describe the token tree used explicitly.

(W3) Presentation issues mainly in Algorithm 1

We sincerely appreciate your meticulous review of the algorithm. To ensure Algorithm 1 is presented in its optimal state, we will carefully cross-check the notational issues in Algorithm 1 according to your suggestions after the discussion phase concludes. We are happy to see it did not impact your ratings with our paper.

(W4) Explanation for baseline performance differences with Spec-Bench

We would like to first address your concerns about the significantly worse performance of Medusa baseline compared with the metrics reported in the Spec-Bench paper.

1. It is worth noting that the Spec-Bench team has updated their leaderboard (https://github.com/hemingkx/Spec-Bench/blob/main/Leaderboard.md) to reflect the corrected speedup ratios on both A100 SXM4 80GB GPU and RTX 3090 24GB GPU. The corrected average speedup for Medusa-vicuda-7b-v1.3 is 1.78$\times$ instead of 2.42$\times$ reported in the paper.

2. We reached out to the Spec-Bench team for their hardware specifications when reporting results in the Spec-Bench paper. We confirmed that their team used the A100 SXM4 version to perform the evaluation, and our team used the A100 PCIE version. The SXM4 version has a significantly higher rated power consumption (400W v.s. 250W for the PCIE version). Also, SXM4 has a different memory bandwidth. We re-examined the evaluation log and confirmed that in our evaluation environment described in L810-L812, the Spec-Bench toolkit works as expected to report faithful results in Table 1.

3. We re-ran the Medusa method on RTX-4090 GPU and reported the results in the revised submission (L330). ParallelSpec still brought significant relative improvement (2.31$\times$ v.s. 1.63$\times$) in terms of speedup ratio when evaluating with a different GPU.

4. For EAGLE ~10% difference with Spec-Bench Paper, we think this is mainly due to the different hardware configurations we have. We appreciate your understanding that, due to budget and practical constraints, we do not have access to the identical computational resources as the Spec-Bench team. This challenge is a common issue in the SPD community when reporting speedup results, as the essence of speculative decoding is to exploit redundant computation intensity in hardware accelerators.

Dear Reviewer YbZP,

We appreciate your positive feedback on our work, and we would like to address your concerns regarding incomplete evaluations, reproducibility, the EALGE-Medusa combination, and a detailed latency breakdown of ParallelSpec.

(W1 & W2 & Q1) Incomplete evaluations on Medusa with temperature=1.0

Good catch! When evaluating ParallelSpec performance, we generally follow the EAGLE-2 [1] evaluation protocol on what combinations of speculative decoding (SPD) frameworks and target models we need to choose. The Table 1 caption of EAGLE-2 paper mentioned that SPD methods like Medusa are excluded when temperture=1 because they “relax acceptance conditions under non-greedy settings, which do not guarantee lossless acceleration.” As such, we follow their practice and exclude these methods from Table 1 in our paper. Of course, we would happily provide additional results in the revision Table 1 when temperature=1.0. It is worth noting that since the lossless generation is no longer guaranteed, speedup ratios will be reported by comparing the generation speed (tokens/s) instead of end-to-end latency for these methods. Other missing baselines in the table are mainly due to the absence of corresponding target models, e.g., Medusa does not provide official checkpoints for Llama-2.

(Q2) Code availability

Reproducibility is always our focus. The code will be publicly available upon the completion of the review process.

(Q3) Possibility to combine Medusa and EAGLE

We believe that combining Medusa with an EAGLE-style approach—i.e., using multiple independent transformer decoder layers to predict different token positions—is certainly feasible. However, this approach runs counter to the design philosophy of ParallelSpec. Firstly, the primary motivation behind designing ParallelSpec was to avoid the memory overhead caused by Medusa-style multi-language model heads and to better leverage soft [MASK] tokens to guide the drafter to predict future tokens at different token positions, thereby achieving parameter sharing. Combining Medusa with EAGLE would undermine both of these assumptions, which is why this design was not considered within ParallelSpec. However, we acknowledge that this design could still be a valid alternative, provided the tradeoff between draft accuracy and memory overhead is carefully evaluated.

(Q4) Detailed breakdown of ParallelSpec latency

We would like to first refer you to Appendix 1 for case studies on a comparison between Medusa and Medusa-ParallelSpec, where average draft latency and verification latency are denoted in Figure 5. For other methods, we report the per-token draft latency of Medusa, EAGLE, and ParallelSpec on vicuna-7b-v1.3, assuming the verification latency for these methods remains the same. Note that the draft latency for Medusa-ParallelSpec and EAGLE-ParallelSpec are identical since they share the same drafter architecture.

Thanks again for your review in helping improve our paper, and please feel free to leave us more comments if you have; we are happy to engage in discussions.

[1]: EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (Li et al., EMNLP 2024)

Dear Reviewer yjYC,

Thank you for your insightful comments and your appreciation of the novel design in ParallelSpec and its strong implications for real-world LLM applications. We would like to address your concerns below.

ParallelSpec requires a dedicated training process that limits its immediate applicability.

As a speculative decoding framework that requires an additional training stage to align drafter distribution to its target model, we agree that such an alignment stage prevents the immediate application of ParallelSpec. However, training-free methods such as conventional speculative sampling (SpS) usually have lower speedup ratios compared to those methods that need extra alignment, as indicated in Table 1, where Hydra, Medusa, EAGLE, and ParallelSpec obtain preferable performance. Therefore, we think a dedicated training process could be good for applications that prefer higher speedup to immediate applicability if non-trivial performance improvement can be observed.

Reproducibility Issue

Thanks for raising the reproducibility issue. We describe the experimental setting and hyper-parameters in Section 5.1 and Appendix A.2. The code will be publicly available upon the completion of the review process.

Better explanation of group-wise parallel training

We apologize for not clearly describing the group-wise parallel training strategy. The motivation for such a design is to ensure the model behaves the same in both the alignment training and drafter inference stages. The attention mask in Figure 3 significantly accelerates the training by packing multiple parallel groups into ONE single training step and uses a customized attention mask to prevent information leakage across different groups. The special input token organization and position indices in the right part of Figure 2 ensure identical behaviors between training and inference.

$K$ sensitivity and group-size $K$ selection guideline

We kindly refer you to Figure 4, which illustrates $K$ sensitivity in data from four domains. Following the common practice of how auto-regressive speculative sampling methods determine draft length $\gamma$, we choose the group size $K$ by observing the end-to-end speedup ratio on validation sets in the target domains.

Memory overhead of [MASK] tokens

Good question! We use the Deepspeed Memory Profiler to measure the max memory allocated for Medusa, EAGLE, and EAGLE-Parallel with the same target model vicuna-7b-v1.3 during the benchmark on Spec-Bench, with fp16 precision and batch_size=1.

When benchmarking with the same sequence, the only memory overhead brought by ParallelSpec is the $K$ [MASK] token embeddings and associated intermediate activations, which are negligible compared to the target model and drafter model size. Thanks to the transformer-based drafter model design, ParallelSpec also demonstrates superior memory consumption compared to the Medusa multi-head drafter design.

High-temperature settings speedup impact

While ParallelSpec does not have explicit designs in high-temperature setups, we are glad to discuss some strategies to maintain drafter alignment without significantly impacting speedup. For example, in the knowledge distillation setting, we can consider using moving earth distance to better align the draft with the target model under high temperatures. Specifically, given the p_draft, q_target, and T(randomly sampled from [0.8, 1]), where p and q are logit distributions of draft and target models, we can modify the distillation loss so that the expectation of acceptance rate E_{T,t} where t is the token could be maximized. In this way, the alignment of the two models could be maximized during the distillation process via sampling from higher temperatures from [0.8,1].

We sincerely appreciate your review in helping improve our paper and hope our response will be satisfactory. Please feel free to leave us more comments, and we will be happy to engage in discussions.

Thanks for the detailed response. I'm increasing these scores:

• Soundness: 3: good
• Contribution: 3: good
• Rating: 6: marginally above the acceptance threshold