Thank you very much for your careful review and constructive feedback! Your insightful questions have provided valuable directions for future improvements and greatly inspired our thinking. Our responses are as follows, and we hope they can resolve your questions and concerns.

Weakness: Model diversity.

We have added experiments for another MoE: Mixtral-8x7B-Instruct-v0.1. We used Eagle's speculation head for Mixtral rather than a standalone draft model to validate our method's generality. Due to the format limitations of this year's rebuttal, we can only present the results in tables as follows. Your advice on model diversity is very valuable and we will add them to the revised paper.

As shown in tables, experimental results on another MoE also conform to our theoretical analysis, where the speculative decoding (SD) speedups on MoE models first increase and then decrease as batch size increases. Since our theoretical analysis places no restrictions on model architectures, this result is as expected and should hold for other MoE models.

Table 1: The SD speedup trend vs. batch size, under different settings for Mixtral-8x7B-Instruct-v0.1

Table 2: The overview of SD speedup on Mixtral-8x7B-Instruct-v0.1

Notation: This table follows the format of Table 1 in the original paper. $\gamma$ represents draft token length, $T_{AR}$ denotes auto-regressive decoding time, $T_{SD}$ denotes speculative decoding time, $\sigma$ is the ratio of actually generated tokens to theoretical maximum, and $\mathbf{x}$ denotes maximal speedup across batch sizes.

Question 1: Consideration of a hybrid metric that combines target efficiency and acceptance rate

A hybrid metric combining target efficiency and acceptance rate would closely approximate the speedup metric itself. As demonstrated in the theoretical derivation in Section 3.1:

$Speedup = \frac{1-\alpha^{\gamma+1}}{1-\alpha} \cdot \frac{1}{\gamma \cdot \frac{T_D(B,1)}{T_T(B,1)} + \frac{T_T(B,\gamma)}{T_T(B,1)} + \frac{T_{reject}}{T_T(B,1)}}$

Where $\alpha$ represents the acceptance rate, $\frac{T_T(B,1)}{T_T(B,\gamma)}$ corresponds to our target efficiency metric, $\gamma$ is the average draft token length (a constant). Both $T_D(B,1)$ (draft model inference time) and $T_{reject}$ (rejection sampling overhead) are significantly smaller than $T_T(B,1)$, contributing minimally compared to $\frac{T_T(B,1)}{T_T(B,\gamma)}$.

Therefore, the hybrid metric you suggested would be nearly identical to the speedup formulation above. To avoid redundancy, we chose to directly use the speedup metric rather than introducing an additional combined metric. However, the introduction of target efficiency is critical as acceptance rate alone cannot fully explain the SD speedup trend.

Question 2: Discussion of chunked batch processing

Thank you for this excellent question. Chunked batch processing is indeed a valuable strategy that can help jointly optimize throughput and latency.

LLM serving systems must satisfy both throughput requirements and latency constraints such as Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT). Generally, there exists a fundamental trade-off between low latency and high throughput. When batch sizes are very large, each request receives limited computational resources, potentially failing to meet latency requirements.

For dense models, SD also follows this trade-off [1,2,3]: when speedup is significant, batch sizes are small (limiting throughput); when batch size increases, SD speedup degrades substantially (worse latency). However, our work demonstrates that for SD with MoE models, there actually exists a regime where speedup increases with batch sizes. This unique property suggests that chunked batch processing could simultaneously enhance both throughput and latency.

We believe this represents a promising direction for future work, particularly in serving environments where satisfying Service Level Objectives (SLOs) is critical.

Question 3: Training a task-specific or sparsity-aware draft model to further improve speedup

Regarding task-specific draft models, this is indeed an active area of research. Previous works [4,5,6] have demonstrated that training specialized draft models can improve acceptance rates for specific tasks. Additionally, recent work [7] has explored an online approach that finetunes the draft model with real-time draft token acceptance patterns during serving. Since acceptance rate optimization has been extensively studied and is not our contribution, we chose to use the basic fixed draft model.

Regarding sparsity-aware draft models, we think this approach only has limited benefits. The acceptance rate fundamentally reflects the alignment between draft and target models on a given task. While task-specific training can indeed improve this alignment, sparsity patterns in the target model don't have a direct impact on this. As long as MoE target models achieve similar performance on a task, they are identical in accepting tokens from the draft model's perspective.

References

[1] Optimizing speculative decoding for serving large language models using goodput.

[2] The synergy of speculative decoding and batching in serving large language models.

[3] Specinfer: Accelerating generative llm serving with speculative inference and token tree verification.

[4] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.

[5] DistillSpec: Improving Speculative Decoding via Knowledge Distillation.

[6] Learning Harmonized Representations for Speculative Sampling.

[7] Online Speculative Decoding.

Thank you for your careful reading and thoughtful comments on our paper! We provide our responses below and hope they address your concerns.

Weakness 1: Clarify the counterintuitive part of our paper's findings.

There seem to be some misunderstandings of our work. We are not arguing that MoE itself favours moderate batch sizes, but rather that SD acceleration for MoE favours moderate batch sizes.

Given that speculative decoding (SD) leverages spare computational resources caused by memory-boundedness, SD acceleration is expected to be more pronounced at small batch sizes, which has been widely observed in previous SD works [1,2,3]. Therefore, your statement that "For a small batch size, MoE is inclined to memory-bound... So a moderate batch size will be efficient for MoE" is incorrect — SD should be more effective under memory-bound conditions.

The counterintuitive part for MoE is that SD is ineffective at small batch sizes, but becomes more effective as batch size increases to moderate level. This is because at small batch sizes, MoE's multi-token verification becomes significantly slower than single-token decoding, leading to degraded target efficiency (the metric we have proposed in paper). Computing multiple tokens often activates more experts, resulting in increased parameter loading and therefore prolonging execution time.

Beyond the preliminary conclusion of SD speedup trend on MoE mentioned above, we further investigate the impact of MoE sparsity on SD speedup. Through theoretical analysis and experiments, we show that SD is actually more effective for sparser MoE models. The table below compares SD speedup between MoE models and dense models (which can be viewed as completely non-sparse MoE models) under identical settings. While MoE initially shows inferior acceleration ratios, it subsequently achieves more obvious acceleration over an extended range.

Table: Comparison of MoE speedup on the MoE model (Qwen2-57B-A14B-Instruct) and the dense model (OPT), corresponding to Figure 2c in the original paper.

The beliefs that SD is unsuitable for MoE and degrades with increasing batch sizes have been long-held in the community[1,2,3,4,5]. Consequently, SD research has focused primarily on dense models for edge deployment, overlooking the potential for recently popular MoE architectures. Our work breaks this conventional wisdom by identifying opportunities for SD in other regimes, thereby helping the community develop a more comprehensive understanding of SD.

In summary, our paper effectively challenges traditional viewpoints and provides counterintuitive findings as follows:

• SD is more suitable for dense models than MoEs [4,5] → We demonstrate that at moderate batch sizes, SD is more effective for sparse MoE models.
• SD effectiveness decreases with increasing batch size [1,2,3] → We demonstrate that for MoE, effectiveness may actually increase with growing batch size.

Weakness 2: Concerns about moderate batch size

We would like to clarify the practical value of our work from both application and model perspectives.

From the application perspective

• Moderate batch sizes are common in private serving scenarios, which are becoming increasingly popular due to their ability to ensure data security. For example, major AI companies now provide internal enterprise chatbot services for their employees.

• When latency requirements are strict, large batch sizes are often not feasible. LLM serving must satisfy not only throughput requirements but also other service level objectives (SLOs) including time-to-first-token (TTFT) and time-per-output-token (TPOT). Excessively large batches limit the computational resources allocated to each request, leading to latency violations. In such cases, moderate batch sizes are necessary.

• From a high-level perspective, our work reveals that applying SD on MoE can relax the trade-off between latency and throughput. In most cases, latency and throughput cannot be optimized simultaneously. For example, when applying SD on dense models, speedup diminishes as batch size increases. However, we have shown that for MoE models, there exists a regime where SD speedup increases (lower latency) with larger batch sizes (larger throughput).

From the model perspective

Compared to dense models, MoE models are actually least efficient in moderate batch sizes. At small batch sizes, MoE models use a small portion of FFN parameters. At large batch sizes, all loaded parameters are provided with abundant computation. However, at moderate batch sizes, all experts are activated while the workload provides insufficient computation, leading to underutilized computational resources. Therefore, optimizing MoE in this regime is critical. Our work demonstrates that SD is a viable solution for such an efficiency gap.

Reference

[1] Optimizing speculative decoding for serving large language models using goodput.

[2] The synergy of speculative decoding and batching in serving large language models.

[3] Specinfer: Accelerating generative llm serving with speculative inference and token tree verification.

[4] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.

[5] Utility-Driven Speculative Decoding for Mixture-of-Experts.

Thank you sincerely for your constructive feedback! We hope our responses below can address your concerns and clarify any misunderstandings about our work.

Weakness 1: Model diversity.

We have added experiments for another MoE: Mixtral-8x7B-Instruct-v0.1. We used Eagle's speculation head for Mixtral rather than a standalone draft model to validate our method's generality. Due to the format limitations of this year's rebuttal, we can only present the results in tables as follows. Your advice on model diversity is very valuable and we will add them to the revised paper.

As shown in tables, experimental results on another MoE also conform to our theoretical analysis, where the speculative decoding (SD) speedups on MoE models first increase and then decrease as batch size increases. Since our theoretical analysis places no restrictions on model architectures, this result is as expected and should hold for other MoE models.

Table 1: The SD speedup trend vs. batch size, under different settings for Mixtral-8x7B-Instruct-v0.1

Table 2: The overview of SD speedup on Mixtral-8x7B-Instruct-v0.1

Notation: This table follows the format of Table 1 in the original paper. $\gamma$ represents draft token length, $T_{AR}$ denotes auto-regressive decoding time, $T_{SD}$ denotes speculative decoding time, $\sigma$ is the ratio of actually generated tokens to theoretical maximum, and $\mathbf{x}$ denotes maximal speedup across batch sizes.

Weakness 2: Exclusion of tree-based speculative decoding.

We would like to point out that while tree-based speculation can achieve higher speedup at batch size = 1, they perform poorly in batched inference. Compared to chain-based methods, although tree-based methods accept more tokens in each iteration, they have significantly lower values of $\frac{\text{accepted tokens count}}{\text{total draft tokens count}}$, since only one path in the tree can be accepted while all other branches are discarded. Specifically:

• At batch size = 1, there are abundant idle computational resources, so verifying a large number of draft tokens does not incur significant latency overhead, making tree-based speculation schemes suitable.
• As batch size increases, there are fewer idle computational resources, and the large number of draft tokens slows down the verification stage. Moreover, most of the verified tokens are destined to be rejected, which wastes computational power.

Additionally, tree-based methods are not engineering-friendly, as they require separate handling of attention masks for each request in the batch. Other works involving batched speculative decoding such as [1,2], also adopt chain-based approaches.

Question 1: Is the belief that speculative decoding cannot effectively accelerate MoEs an untested assumption or a well-established empirical claim?

This belief is supported by experimental results rather than untested assumptions. We have already provided a citation supporting this belief in line 36 of the original paper, which refers to EAGLE [3]. Table 2 and Table 3 of EAGLE shows that SD achieves 2.54× better speedup on dense models than MoE models. Additionally, [4] from Georgia Tech and Nvidia Research also explicitly states in their abstract that "speculation is ineffective for MoEs" and reports 1.5× SD degradation when applied to MoE. We appreciate the reviewer's constructive feedback, and will incorporate these supporting data in the revised version.

Beyond empirical validation, this belief has solid theoretical grounding. Under small batch sizes, systems are memory-bound, meaning execution time is primarily determined by parameter access. For dense models, decoding one token or verifying multiple tokens loads the same amount of parameters, resulting in similar total execution time. However, for MoE models, verifying multiple tokens means activating more experts and loading additional parameters, thereby increasing latency and reducing speculative decoding efficiency. More rigorous analysis can be found in Section 3.1 in the original paper. This theoretical limitation explains why most existing speculative decoding works focus on accelerating dense models.

Question 2: Illustration of the system.

Thank you for your constructive suggestions on improving the clarity of our work. We will include a system diagram and a detailed textual walk-through of Algorithm 1 in the revised manuscript.

As figures are not allowed in this year's rebuttal, we provide a textual description of the MoESD framework here. Based on our analysis of roofline effects and the number of activated experts in MoE, we establish the expression of SD speedup for MoE as shown in line 11 of Algorithm 1. Due to uncertainties in operator implementation and GPU architecture, the expression includes several parameters with clear physical meanings for relaxation. Through profiling a small number of speedup data points and lightweight fitting (as detailed in Appendix B), we can determine the values of these parameters. Applying these parameters to the expression yields a complete speedup model. Our experimental results show that this modeling consistently matches real measurements, thereby validating the correctness of our theoretical analysis. This modeling approach also makes the tradeoffs involved in SD acceleration for MoE more transparent and explainable.

References

[1] MagicDec: Breaking Throughput-Latency Trade-off for Long Context Generation with Speculative Decoding.

[2] BASS: Batched Attention-optimized Speculative Sampling.

[3] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.

[4] Utility-Driven Speculative Decoding for Mixture-of-Experts.

Thank you for your valuable suggestions on diversity, which help to make our conclusion more convincing. We have supplemented experiments, hoping they can address your concerns.

Weakness 1: Model diversity

We have added experiments for another MoE: Mixtral-8x7B-Instruct-v0.1. We used Eagle's speculation head for Mixtral rather than a standalone draft model to validate our method's generality. Due to the format limitations of this year's rebuttal, we can only present the results in tables as follows. Your advice on model diversity is very valuable and we will add them to the revised paper.

As shown in tables, experimental results on another MoE also conform to our theoretical analysis, where the speculative decoding (SD) speedups on MoE models first increase and then decrease as batch size increases. Since our theoretical analysis places no restrictions on model architectures, this result is as expected and should hold for other MoE models.

Table 1: The SD speedup trend vs. batch size, under different settings for Mixtral-8x7B-Instruct-v0.1

Table 2: The overview of SD speedup on Mixtral-8x7B-Instruct-v0.1

Notation: This table follows the format of Table 1 in the original paper. $\gamma$ represents draft token length, $T_{AR}$ denotes auto-regressive decoding time, $T_{SD}$ denotes speculative decoding time, $\sigma$ is the ratio of actually generated tokens to theoretical maximum, and $\mathbf{x}$ denotes maximal speedup across batch sizes.

Weakness 2: Hardware settings diversity

We add experiments for additional hardware settings including more inter-GPU communication and different compute-memory tradeoff as you required. The configurations include: 2×A800 (original setup for comparison), 4×A800 (newly added), and 4×L40 (newly added). Since L40 only has 48GB memory while the Qwen2 MoE model has 57B parameters, we use 4×L40 due to memory constraints.

More inter-GPU Communication: 2×A800 (1st group) and 4×A800 (2nd group) configurations in Table 3 are compared. For absolute timings (i.e., $T_{AR}$ and $T_{SD}$), we observe obvious decreases due to additional devices. For SD speedups, they maintain similar trends but with slight degradation. This occurs because while the large model is parallelized between GPUs, the small draft model remains on a single GPU, relatively increasing its forward pass time compared to the large model's forward time.

Different compute-memory tradeoffs: 4×A800 (2nd group) and 4×L40 (3rd group) configurations in Table 3 are compared. A800-80GB provides 312 TFlops BF16 performance with 1.94 TB/s bandwidth (roofline ridge point ≈161), while L40 offers 181 TFlops with 864 GB/s bandwidth (ridge point ≈214). L40's higher ridge point indicates more available compute capacity for SD verification, enabling higher speedups as confirmed by our experimental results.

Table 3: SD speedup on different hardware settings

Notation: This table follows the format of Table 1 in the original paper. $\gamma$ represents draft token length, $T_{AR}$ denotes auto-regressive decoding time, $T_{SD}$ denotes speculative decoding time, $\sigma$ is the ratio of actually generated tokens to theoretical maximum, and $\mathbf{x}$ denotes maximal speedup across batch sizes.

We also present how speedup varies with batch size in Table 4. As shown, the speedup demonstrates a consistent trend that aligns with the theoretical analysis and experiments reported in the original paper: the speedup initially increases and then decreases, achieving peak performance at moderate batch sizes.

Table 4: SD speedup vs. batch sizes on different hardware settings

In summary, the SD speedups in scenarios with more inter-GPU communication or compute-memory tradeoffs conforms to our theoretical analysis as expected. We will include these results in the revised paper to strengthen the generalizability of our findings.

We sincerely appreciate your careful reading and valuable suggestions for improving our manuscript. In the following response, we first answer your four questions with additional experimental results as supporting evidence, and then respond to the weaknesses you mentioned. We hope these revisions adequately address your concerns.

Question 1: Is there any way to predict what the optimal batch size might be based on your theory, vs through hyper-parameter tuning?

Predicting speedup based purely on theoretical analysis without using any profiling data is challenging. On GPUs, GEMM is indeed predictable because it involves regular and highly optimized operators. However, this becomes difficult for operators like Attention that involve customized kernel optimizations (FlashAttention1/2, eager attention, SDPA attention), fusion (of different types of nonlinear layers or positional encodings like RoPE and its variants), and multi-hierarchy memory access (registers, caches, DRAM), as pointed out by previous works [1,2,3]. Consequently, other system optimization works like NanoFlow also adopt a two-stage strategy: profiling first, then runtime execution.

We take profiling results of Qwen2-57B-A14B (hidden size 3584) and Mixtral-8x7B (hidden size 4096) as an example to illustrate this. For FFN, Qwen takes a shorter time than Mixtral (143μs vs 226μs), aligning with their relative hidden sizes. However, for Attention, Qwen takes a longer time than Mixtral (271μs vs 115μs), contradicting the theoretical expectation based on hidden size scaling.

Therefore, modeling speedup trends with pure analytical methods requires case-by-case analysis for different operator implementations, GPU microarchitectures, and instruction sets. Moreover, many hardware details are not disclosed by GPU vendors, making such predictions even less practical. In contrast, the hyperparameter approach offers a more general paradigm and is easy to use: all parameters have clear physical meanings, only a few profiling data points are needed, and the computational overhead is very low. Our method achieves a balance between effectiveness and practicality: on one hand, it captures the primary factors (i.e. the number of activated experts and roofline trends); on the other hand, it avoids getting lost in system implementation details.

Question 2: End-to-end speculative decoding speedup comparsion between MoE and dense models.

We compare target efficiency in Figure 2c in order to isolate the effect of acceptance rate. Given that the strong correlation between target efficiency and speedup has been demonstrated in Figures 2a and 2b, we believe this comparison provides a reliable assessment.

We also have end-to-end (E2E) speedup comparisons, as shown in the table below. As batch sizes increase (>= 16), speculative decoding (SD) indeed achieves greater speedup on MoE models compared to dense models (i.e., the last row exceeds 1). We appreciate the reviewer's constructive feedback, which helps to make our statements more direct and concrete. We will attach these E2E speedup results in the revised paper.

Table 1 Comparison of E2E SD speedup between MoE and dense models (corresponding to Figure 2c)

Question 3: Addition of error bars.

The table below presents the speedup of five independent runs and their average for Figure 2a. The variance in speedup is minimal, confirming the statistical significance of our findings. Theoretically, the speedup should also be stable since we fixed the random seed to ensure identical workloads across all runs. Your advice is very helpful, and we will add complete error bars in the revised paper.

Table 2 Speedup of five independent runs and their average

Regarding the fluctuations in performance curve, they are attributed to GPU's dimension quantization effects: tensor core utilization changes with matrix dimensions. When dimensions are not divisible by the GPU's native tile sizes, computation performance degrades. This phenomenon is well-documented in NVIDIA's document [4]. Auto-regressive (AR) decoding is more sensitive to this effect, making the time ratio of AR to SD (namely, SD speedup) fluctuate. Despite these local fluctuations, the overall speedup trend follows our theoretical analysis, confirming the validity of our conclusions.

Question 4: Latency and throughput tradeoff.

This question reveals a core difference between MoE and dense models when applying SD. Our findings actually demonstrate that for MoE models, latency and throughput can be jointly optimized, while this is hard for dense models.

For dense models, there exists a trade-off between low latency and high throughput: smaller batch sizes yield higher SD speedups (low latency but limited throughput), while larger batch sizes result in lower SD speedup (increased throughput but high latency). However, for MoEs, speedup can increase with batch size, indicating both latency and throughput are improved. Our work reveals the underlying cause of this counter-intuitive phenomenon and further extends it to MoEs with different sparsities.

Your insightful question provides a valuable perspective that helps readers understand our work from a higher level. We will incorporate this into our revised paper.

Clarification of the novelty and impact of our work.

While our work indeed builds upon established principles, we would like to respectfully highlight several aspects that demonstrate novelty and impact.

Correcting Critical Misconceptions with Quantitative Methodology: Our work challenges two prevalent beliefs that have constrained SD research as follows:

• SD is more suitable for dense models than MoEs [6,7]. → We demonstrate that at moderate batch sizes, SD is more effective for sparse MoE models.
• SD performance degrades with larger batch sizes [8,9,10] → We demonstrate that for MoE, effectiveness may actually increase with growing batch size.

Our work is the first to systematically refute these beliefs, addressing an important gap in current research. Beyond only qualitatively describing SD speedup trends as previous works have done [5,6,10], we further developed a quantitative model making SD more explainable, which is also a first attempt in SD research. Our theoretical analyses align well with experimental results, providing rigorous support for our conclusions.

Practical Impact on MoE Optimization: Beyond theoretical findings, our work has practical impact on MoE optimization. Compared to dense models, MoEs are efficient at both small batches (fewer activated parameters) and large batches (abundant computation), but suboptimal at medium batches where full parameters are loaded without sufficient computation to amortize this overhead. Other MoE optimizations such as expert offloading and caching are also ineffective in this regime due to fully activated experts. In this paper, we show that SD—which has been overlooked by previous works for MoE—is effective, thus providing a novel perspective on MoE's deficiency at moderate batch sizes.

In summary, though based on existing theories, we have developed novel conclusions that help correct misconceptions and optimize MoE models, supported by a quantitative methodology unprecedented in previous SD works. Other reviewers such as kQQx, Xih7 and 8fAG also acknowledge the value of our work on this timely topic of MoE and SD.

Reference

[1] NanoFlow: Towards Optimal Large Language Model Serving Throughput.

[2] Orion: Interference-aware, fine-grained gpu sharing for ml applications.

[3] Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking.

[4] Since links are not allowed in rebuttal, please search for "Matrix Multiplication Background User's Guide NVIDIA".

[5] MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding

[6] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.

[7] Utility-Driven Speculative Decoding for Mixture-of-Experts.

[8] Optimizing speculative decoding for serving large language models using goodput.

[9] The synergy of speculative decoding and batching in serving large language models.

[10] Specinfer: Accelerating generative llm serving with speculative inference and token tree verification.