SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications

Thank you for your positive review and detailed feedback, our response is below.

How will the proposed algorithm impact memory consumption?

Suffix Decoding uses CPU memory on the host during serving, which is usually plentiful and under-utilized. To show Suffix Decoding’s memory consumption and other performance stats, we performed an additional micro-benchmark on its suffix tree, shown below:

As shown, Suffix Decoding can cache 572M tokens using only 6.15 GB of CPU memory, and its memory usage scales linearly. In comparison, optimized serving systems can generate on the order of ~5000 tokens/s for Llama-8B on an A100 GPU $2$ . Typical A100 systems (e.g. AWS p4d.24xlarge) have 144GB of CPU memory per A100 GPU. This means that Suffix Decoding can potentially cache 31 days of generated tokens (see math below) before requiring cache eviction.

Time to fill up GPU memory:

1. Memory per token: $\frac{6.15 GB}{572M tokens} = 1.075 \times 10^{-8} GB/token$
2. Total token capacity: $\frac{144 GB}{(1.075 \times 10^{-8} GB/token)} = 1.34 \times 10^{10} tokens$
3. Time to fill RAM: $\frac{1.34 \times 10^{10} tokens}{5000 tok/s} = 2.68 \times 10^{6} seconds = \boxed{31 days}$

General domain (e.g. MT-Bench) performance.

Suffix Decoding's MT-Bench performance is reflected in SpecBench A.1.2 tables. These tables contain 8 MT-Bench subsets (Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, Humanities). As SpecBench equally samples all 8 categories, overall Suffix Decoding MT-Bench performance can be calculated by averaging these column results.

Additionally, we provide evaluation results that include hybrid Suffix/Spec Decoding (end of Sec 3) using Suffix Decoding with EAGLE-3:

SpecBench - Speedup (x)

AgenticSQL - Speedup (x)

As shown, the hybrid Suffix Decoding and EAGLE-3 method performs similarly to plain Suffix Decoding on the more structured/repetitive tasks (e.g. AgenticSQL), while being faster than plain EAGLE-3 on the more general domain SpecBench. We will include these additional results in updating our paper.

$1$ TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput | Liu et al. 2025 | arXiv:2406.14066

$2$ (Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) | SGLang Team, Jul 25, 2024 | LMSYS Org)

Thank you for your positive review and detailed feedback, our response is below.

Performance on open-ended or less repetitive token sequences.

As the reviewer mentioned, Suffix Decoding can still learn useful patterns even in open-ended workloads and can fall back to the SOTA model-based method. The hybrid suffix/speculative decoding (end of Sec 3) also helps address this scenario. We provide additional evaluations that show the performance of this hybrid method below:

SpecBench - Speedup (x)

As shown, hybrid Suffix Decoding + EAGLE-3 performs similarly to plain Suffix Decoding on more structured/repetitive tasks (e.g. AgenticSQL), while slightly improving upon plain EAGLE-3 on the open-ended tasks (Spec-Bench). We will include these additional results in the updated version of our paper. Given space limitations, we are including here only the wall-clock speedups compared to the baseline. A comprehensive breakdown of results, including per-token latency (TPOT), mean accepted tokens per step (MAT), mean acceptance rate, and speculation time per generated token, will be incorporated into the revised paper.

How to set the number of draft tokens per request within a batch.

We agree that optimizing the speculation per request in a batch is crucial for many practical online deployments. While SuffixDecoding focuses on what tokens to speculate, it is also compatible with existing works that explore how much to speculate per request. For example, TurboSpec $1$ uses a "goodput" metric to balance speculation lengths and batch size, noting that optimal speculation decreases as batch size increases. By dynamically adjusting the number of speculative tokens generated by the SuffixTree per request, SuffixDecoding can be integrated with TurboSpec to maximize the batch-wise goodput metric.

Furthermore, the statistics-based scoring used by Suffix Decoding can potentially better inform methods like TurboSpec. For example, choosing to speculate more from sequences that have a higher marginal score. These are interesting and important directions for future work.

[1] TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput | Liu et al. 2025 | arXiv:2406.14066

Thank you for your detailed review and feedback, our response is below.

Latency and profiling (the paper reports FLOP-based efficiency but omits wall-clock latency and energy measurements on GPU hardware).

We believe there is a misunderstanding regarding the metrics reported in our paper. To clarify, all speedup measurements in our paper are based on wall-clock latency, not FLOP counts. The speedup in Fig 4 is the ratio of real-world wall-clock time per output token measured on H100 GPUs. For example, in SWE-Bench, Suffix Decoding achieves 31.093 ms per token vs vanilla's 49.836 ms per token (see Appendix A.1.2), resulting in a speedup of 49.836 / 31.093 = 1.6x.

FLOPs and energy consumption are less relevant for evaluating speculative decoding methods because speculative decoding typically expends more FLOPs (via speculation) to obtain lower wall-clock latency (via parallel verification).

Memory overhead: How large do the suffix trees grow in practice, and what are the RAM and update-time costs under concurrent agentic requests?

The suffix trees employed by Suffix Decoding are highly time and memory efficient. To illustrate this, we present a microbenchmark below showing the memory footprint and per-token update cost across various tree sizes (using requests from the SWE-Bench dataset). We will include this additional microbenchmark in the revised paper.

Overall, the memory consumption scales linearly with the size of the suffix tree, while the update time remains consistently low. For comparison, optimized inference engines running Llama-8B on an A100 GPU can typically generate ~5000 tokens per second $1$ or ~432M tokens per day. Typical A100 systems (e.g. AWS p4d.24xlarge) have 144GB of CPU memory per A100 GPU. This means that Suffix Decoding can potentially cache 31 days of generated tokens per A100 GPU before hitting CPU memory limits. Beyond this limit, it is also straightforward to incorporate cache eviction (e.g. LRU) into Suffix Decoding to avoid out-of-memory problems.

Generalization beyond Agentic workloads: Have you evaluated SuffixDecoding on less structured or single-turn tasks (e.g., open-ended question answering)?

To clarify, the Spec-Bench dataset used in our evaluation (Sec 4.1, Fig 4) consists of mainly non-agentic, open-ended tasks (e.g. single-turn question-answering). We included this dataset explicitly to stress-test Suffix Decoding in a scenario where we did not expect it to perform well, as evaluating performance under these extreme scenarios is important to understand the limitations and robustness of Suffix Decoding.

The tables in appendix A.1.2 detailed performance results across all 13 Spec-Bench categories. For example, the qa (question answering) column represents open-ended, single-turn questions. Among the 13 categories, qa, rag, math_reasoning, summarization, and translation are single-turn tasks.

Although Suffix Decoding is not specifically designed for these open-ended tasks, it can be combined with other speculative decoding methods in a hybrid mode to achieve the best of both worlds. We have performed additional experiments evaluating this hybrid-mode, which we present below.

SpecBench - Speedup (x)

AgenticSQL - Speedup (x)

Here, we used a fallback threshold of τ=7 for SpecBench, and τ=1 for AgenticSQL. For more open-ended tasks such as QA (question answering), the hybrid approach with Suffix Decoding outperforms the speculative decoding method alone. This indicates that while Suffix Decoding can greatly improve agentic applications, it does not have to come at the expense of other open-ended applications.

Note that all baselines are slightly improved compared to the table in our submission due to performance improvements in the benchmark framework.

Given space limitations, we are presenting only the wall-clock speedups compared to the baseline. A comprehensive breakdown of results, including per-token latency (TPOT), mean accepted tokens per step (MAT), mean acceptance rate, and speculation time per generated token, will be incorporated into the revised paper.

Hybrid Fallback Threshold: Can you characterize its sensitivity across mixed workloads?

We report the results of SpecBench and AgenticSQL experiments using the Hybrid Suffix Decoding method with various thresholds. In the table below, we show the measured wall-clock speedups with respect to the vanilla (no speculation) baseline.

SpecBench - Speedup (x)

AgenticSQL - Speedup (x)

For open-ended generation, a good heuristic is to set the fallback threshold (τ) to a value close to or slightly exceeding the mean number of accepted tokens (MAT) for the model-based speculator used with the suffix-tree. For instance, we used EAGLE-3, which has a MAT of approximately 4.65 tokens/step in SpecBench. The table shows that τ values of 5, 6, and 7 yield the best overall speedups, with the results being quite similar across these values.

Conversely, in agentic workloads like AgenticSQL, the standalone SuffixDecoding implementation (τ=0) is optimal because it can confidently speculate much longer sequences than model-based speculators such as EAGLE-3.

$1$ (Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) | SGLang Team, Jul 25, 2024 | LMSYS Org)

I’m happy to raise my score in light of these clarifications.

Separately, the paper would benefit from citing prior work on independent multi-token heads in its Related Works, for example [1,2,3]:

[1] Better & Faster LLMs via Multi-Token Prediction, Gloeckle et. al., ICML 24.

[2] Accelerating Blockwise Parallel Language Models with Draft Refinement, Kim et. al, NeurIPS 24.

[3] Blockwise Parallel Decoding for Deep Autoregressive Models, Stern et. al, NIPS 18.

Thank you for your positive review and detailed feedback, our response is below.

Scalability of lookup time, speedup and scoring.

For lookup time vs number of records, we provide an additional micro-benchmark of the suffix tree below, where we measure its lookup time, update time, and memory consumption for various tree sizes. We will include it in our final paper.

In short, the total memory consumption scales linearly with the size of the tree, and both the update and lookup times remain fast even with larger trees.

For speculative speedup and accuracy vs number of records, we provided a brief ablation study in Fig 8. Speedup consistently improves as suffix tree size increases, demonstrating that more historical data leads to better performance. Importantly, acceptance rates remain stable (~10-20%) even as tree size varies dramatically, indicating that our adaptive speculation mechanism (MAX_SPEC = αp) effectively manages the quality-quantity tradeoff.

If there are specific questions beyond what’s provided in our ablation study, we are happy to answer during the discussion period.

Scenarios when CPU memory is filled up.

We agree that the CPU memory can be filled up, but we would like to work through a practical scenario. Our micro-benchmark results above show that Suffix Decoding is highly memory efficient, caching 572M tokens using only 6.15 GB of memory, and scaling linearly. At the same time, optimized serving systems can generate on the order of ~5000 tokens/s for Llama-8B on an A100 GPU $1$ . Typical A100 systems (e.g. AWS p4d.24xlarge) have 144GB of CPU memory per A100 GPU. This means that Suffix Decoding can potentially cache 31 days of generated tokens per A100 GPU before hitting CPU memory limits (see math below). Beyond this limit, it is also straightforward to incorporate cache eviction (e.g. LRU) into Suffix Decoding to avoid out-of-memory problems.

Time to fill up GPU memory:

1. Memory per token: $\frac{6.15 GB}{572M tokens} = 1.075 \times 10^{-8} GB/token$
2. Total token capacity: $\frac{144 GB}{(1.075 \times 10^{-8} GB/token)} = 1.34 \times 10^{10} tokens$
3. Time to fill RAM: $\frac{1.34 \times 10^{10} tokens}{5000 tok/s} = 2.68 \times 10^{6} seconds = \boxed{31 days}$

Can you explain the title?

We used the term "extreme" in reference to the significantly longer speculation sequences possible with suffix trees compared to typical speculative decoding, which is also crucial for exploiting repetition in agentic applications such as SWE-Bench and AgenticSQL. We used the term "emerging" in reference to the growing field of agentic AI applications that involve multiple steps in answering a user query, often with highly repetitive LLM calls. We will add additional explanations and a footnote to our introduction to explain these choices.

'Related Works' section is in a bit of a weird spot.

We thank the reviewer for pointing out the non-standard location of the related works section and we will move it closer to the background section as suggested.

$1$ (Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) | SGLang Team, Jul 25, 2024 | LMSYS Org)