EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

Thank you for the thorough evaluation and insightful suggestions, which have been instrumental in refining our work.

Q1. As mentioned in the paper, “DeepSeek-v3 inspired the design of EAGLE3”, some discussions and comparisons between the design of deepseek-v3 and that of EAGLE3 could be added to the paper.

A1. The Multi-Token Prediction (MTP) architecture in DeepSeek-V3 shares similarities with EAGLE. The key difference lies in that EAGLE reuses a single network in a loop to predict all future tokens, which requires a feature loss to align the input and output features. In contrast, DeepSeek-V3's MTP uses multiple separate networks to predict tokens at different positions, thereby avoiding the need for feature loss at the cost of increased computational overhead.

This design inspired us to remove the feature loss in EAGLE-3 (similar to DeepSeek-V3), while still using a single network (as in EAGLE), thanks to the simulation mechanism during training.

Q2. The ablation study showed that a mix of low, middle, and high-level features improves the performance, is there any insights or discussions on why the fused features can improve the performance of the draft model? As in deepseek-v3, only the top features is taken as the input of the multi-token prediction decoder, it is recommended to add some discussions on this point.

A2. We discussed this point in the original manuscript, Lines 83–88. The original text is as follows:

EAGLE and speculative sampling methods such as Medusa reuse the top-layer features of the target model, specifically the features immediately before the LM head. For an LM head with a full-rank weight matrix, the top-layer features corresponding to the logits of the next token are unique, ensuring that the information contained in these features aligns directly with the logits of the next token. However, predicting the next-next token based solely on top-layer features—which are inherently limited to the next token—poses a significant challenge.

Q3. In Table 3-5, only 2 out of three algorithms (EAGLE1, EAGLE2, and EAGLE3) are evaluated, it is recommended to evaluate all of them.

A3. The difference between EAGLE-1 and EAGLE-2 lies in the draft tree generation process. However, the high batch size settings used in Tables 3 and 5 are not well-suited for tree-based drafting. This is because tree-based drafting is more computationally intensive, and the redundant GPU resources required for such speculative strategies are scarce under high batch size configurations. As a result, both methods employ chain-based drafts in these experiments, rendering EAGLE-1 and EAGLE-2 functionally equivalent under this setting. Regarding other methods, since SGLang only supports the EAGLE series, we did not include additional baselines to ensure a fair comparison.

We provide additional experimental results for Table 4 below.

Thank you for your valuable suggestions. In what follows, I will address your questions and clarify some misunderstandings.

Q1. The implementation of multi-layer feature fusion is described only at a high level, with no clear mathematical formula and rationale for layer selection. The ablation study only compares with/without fusion, lacking analysis of different layer combinations.

A1. We select the 3rd, $n//2$-th, and $(n{-}3)$-th layers, where $n$ is the total number of layers. Let the corresponding feature vectors be denoted as $l$, $m$, and $h$, respectively. The fused feature is computed as $W \cdot \text{concat}(l, m, h)$, where $\text{concat}(\cdot)$ denotes vector concatenation, and $W \in \mathbb{R}^{h \times 3h}$ is a learnable projection matrix, with $h$ being the hidden dimension of the target model. The following is an ablation study on layer selection. As long as the final layer is not used and the features are fused, the performance improves significantly, while the specific choice of layers has limited impact. Furthermore, increasing the number of layers beyond three does not bring additional gains. Therefore, we opt for the $3$-rd, $n//2$-th, and $(n{-}3)$-th layers in our design.

Q2. Experiments are limited to LLaMA, Vicuna, and similar models (up to 70B). There is no evaluation on more recent or larger models (e.g., Qwen-32B or MoE), which limits claims of generality.

A2. We provide additional results on larger MoE model architectures. The table below shows the acceleration performance of EAGLE-3 within the SGLang framework on LLaMA4 Scout (MoE, 109B) and LLaMA4 Maverick (MoE, 400B). It is important to note that results obtained under the production-grade SGLang framework should not be directly compared with those from standard Huggingface-based environments.

Q3. EAGLE-3 is trained with more data than baselines, which may confound the effect of the method with data scale. There is no controlled comparison under equal data conditions.

A3. Figure 2 compares EAGLE-2 and EAGLE-3 under the same data scale. Futhermore, we have already included a comparison under the same data conditions with HASS—the most similar and strongest-performing baseline to our method—in Appendix B of the original manuscript. Similarly to EAGLE-2, HASS fails to scale up, while EAGLE-3 exhibits rapid performance improvements as more training data become available. EAGLE-3 consistently outperforms HASS across all data scales. For convenience, we paste the results here. Note that Data Scale = 1 refers to using the full amount of ShareGPT data. The evaluation is conducted on the MT-Bench dataset, with LLaMA-Instruct 3.1 8B as the target model.

Dear Reviewer 9fyH,

Thank you once again for your thoughtful and constructive feedback on our paper. We truly appreciate the time and effort you’ve dedicated to the review process.

As a gentle reminder, the discussion period will conclude on August 6 (AoE). We’ve provided detailed responses to your comments in our rebuttal and sincerely hope they address your concerns.

If you have any further questions or additional suggestions, we’d be more than happy to address them.

Thank you again for your support and engagement!

Warm regards,

Authors of Submission 13079

Thank you for your thoughtful review and constructive feedback. We truly value your input and have addressed your concerns in the following responses.

Q1. Limited model architecture coverage, MoE is becoming the mainstream backbone and getting more important.

A1. EAGLE-3 is also applicable to accelerating MoE models. The table below shows the acceleration performance on LLaMA4 Scout (MoE, 109B) and LLaMA4 Maverick (MoE, 400B) within the SGLang framework. It is important to note that results obtained in the production-grade SGLang framework should not be directly compared with those from standard Huggingface-based setups.

Q2. Limited applicability for proprietary models where internal features cannot be accessed, such as openai gpt or claude models.

A2. OpenAI and Anthropic can train EAGLE-3 heads for their proprietary models. For those models, EAGLE-3 can reduce deployment costs for companies by improving generation speed. This, in turn, lowers the cost for end users and enhances the overall user experience.

Thank you for your time and effort in reviewing our manuscript. We sincerely appreciate your valuable comments and suggestions. Below, we provide detailed responses to your concerns.

Q1. While the paper mentions the fusion of three hidden layers (denoted as $l$, $m$, and $h$), it provides no details on how these specific layers are selected.

A1. We select the 3rd, $n//2$-th, and $(n{-}3)$-th layers, where $n$ is the total number of layers. The table below presents experimental results on different layer selections. As long as the final layer is not used and the features are fused, the performance improves significantly, while the specific choice of layers has limited impact. Furthermore, increasing the number of layers beyond three does not bring additional gains. Therefore, we opt for the $3$-rd, $n//2$-th, and $(n{-}3)$-th layers in our design. The evaluation is conducted on the MT-Bench dataset, with LLaMA-Instruct 3.1 8B as the target model.

Q2. The authors do not appear to specify the number of sampling steps used during training, which also undermines the reproducibility of the method.

A2. Thank you for the reminder. We used 7 sampling steps during training, and we will clarify this detail in the next revision.

Q3. Although migrating the implementation to established inference frameworks is a commendable step, the paper lacks a detailed justification for the chosen configuration, specifically, the chain length of 3 and the decision to omit the draft tree.

A3. Speculative sampling utilizes the redundant computational capacity of GPUs. However, under the high batch size setting targeted by SGLang, such redundancy becomes limited. Consequently, we disable the computationally intensive tree-based drafting. Moreover, longer speculative chains require more resources, and our experiments show that a chain length of 3 achieves the best performance across most batch sizes. Therefore, we adopt this configuration in all our experiments.

Q4. Moreover, we observe that even with a batch size of 1 (contrary to the large-batch setting emphasized in the paper), the reported speedup on H100 using sglang falls significantly short of earlier results. This discrepancy suggests that the earlier performance gains may have limited applicability in real-world scenarios. As such, the proposed method has yet to convincingly demonstrate its suitability for large-scale deployment, rather than being merely a toy-scale solution.

A4. On fixed hardware, all acceleration techniques and engineering optimizations aim to approach the hardware’s performance limits. Naturally, SGLang—already equipped with many optimizations—is more difficult to accelerate further compared to an unoptimized Huggingface setup. It is widely acknowledged in the community that production environments are harder to accelerate than experimental ones, and this applies to all acceleration methods. Accordingly, even modest improvements in production settings directly translate to cost savings.

SGLang exemplifies the type of large-scale deployment scenario you referred to. Our experiments show that EAGLE-3 performs effectively in this environment, achieving a 2–3× reduction in latency compared to vanilla inference on SGLang. This highlights that our method extends well beyond toy-scale applications. Moreover, EAGLE-3 has been officially integrated into large-scale inference frameworks such as SGLang and vLLM, further demonstrating its practicality and scalability in real-world production systems.

Q5. However, based on the parameter partitioning strategy described, where full backpropagation is not required, the computational workload appears to be only slightly greater than that of inference and significantly less than that of full model training. The reported training duration thus raises questions and calls for further clarification.

A5. Although EAGLE-3 uses only a single layer, it requires 7 sampling-training cycles, making the overall training cost relatively high (roughly equivalent to training 7 layers in a 70B model with 80 layers in total). Moreover, since our focus is on inference efficiency, we did not apply specific optimizations to the training process. In fact, organizations that require the deployment of 70B models typically have access to abundant computational resources, making this level of overhead acceptable.

Dear Reviewer o9ZR,

Thank you once again for your thoughtful and constructive feedback on our paper. We truly appreciate the time and effort you’ve dedicated to the review process.

As a gentle reminder, the discussion period will conclude on August 6 (AoE). We’ve provided detailed responses to your comments in our rebuttal and sincerely hope they address your concerns.

If you have any further questions or additional suggestions, we’d be more than happy to address them.

Thank you again for your support and engagement!

Warm regards,

Authors of Submission 13079

Thank you to the authors for their detailed responses to my questions. I believe the main weaknesses of the manuscript lie in its casual treatment of key hyperparameters and the performance degradation observed when integrated with the inference framework. That said, the authors have addressed my concerns well, particularly the first one, through additional experiments and clarifications.

Regarding the second issue, while the authors have provided a plausible explanation for the performance drop, the response does not fully resolve the underlying concern.

Overall, based on the revisions and clarifications provided, I will raise my review score. I wish the authors all the best.