TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation

We sincerely thank you for your time and insightful feedback, which has helped us improve our manuscript.

Q: Reformulate Claim

A: We appreciate your suggestions to replace "fail" with "benefit" and will revise accordingly.

1. Dynamic KV Cache Compression (Challenge II) The limitation of baseline methods is not merely budget constraints but their single-compression design for long inputs. For long outputs, however, the KV Cache grows dynamically with generated tokens. In contrast, our method maintains a fixed-size KV Cache regardless of output length (no hard budget), as we continuously compress the growing cache.

2. Memory Limits: Like all speculative decoding methods, we preserve a full KV Cache for lossless generation. If this cache exceeds GPU memory (not a pre-set budget), generation fails—but this is distinct from baselines’ rigid budget-based failures. Note that our focus is on draft-phase optimization.

3. Example: For TriForce (budget=4096):

   • Input=100K, Output=256 → KV Cache=4096+256 (valid).
   • Input=8192, Output=100K → KV Cache=4096+100K (fails).

   Our method fixes the KV Cache size for any input/output length.

Q: Sampling Diversity

A: We sincerely appreciate your insightful comments regarding the diversity-speedup balance.

1. Trade-off In Table 6, our experiments demonstrate that the proposed penalty mechanism enables effective balancing of these two objectives:

   • Without penalty: Achieves 3.58× speedup with baseline diversity
   • With θ=1.2 penalty: Maintains 3.27× speedup (only 0.31x reduction) while significantly improving diversity metrics

   This evidences that our approach attains competitive acceleration without severely compromising diversity.

2. Main Results All experiments in Tables 3-4 already employ θ>1.0 penalty configuration (Table 10). The absence of severe repetition issues in these results further confirms the practical viability of our trade-off strategy.

3. Self-BLEU Self-BLEU provides direct measurement:

   Metric	Self-BLEU-2	Self-BLEU-3	Self-BLEU-4	Self-BLEU-5
   w/o Penalty	0.7899	0.7445	0.7162	0.6966
   w/ Penalty	0.5588	0.4745	0.4115	0.3788

   Consistent 23.11-31.78% relative reduction across n-grams confirms diversity improvement, aligning with Distinct-N trend(Table 8). BERTScore/MAUVE are inapplicable for our reference-free open-generation setting, as they require parallel references.

4. Implementation The adjustable penalty coefficient allows practitioners to prioritize either speed or diversity based on application needs, demonstrating our framework's adaptability.

Q: 250M vs. 68M

A: For a fair comparison, we used 68M for llama2 and 250M for llama3.1 in the TriForce experiment in Table 3, and the parameter settings of the two draft models are exactly the same. Because the vocabulary size of llama3.1 is much larger than that of llama2, the model size becomes larger.Finally, we have open-sourced this model to support future research.

Q: Table 1

A: While Table 1 is briefly referenced in L49–50, we will explicitly add this context in the revised manuscript to ensure transparency.

We reproduced Triforce's results on A100-80G and PG-19 using identical hyperparameters. The only difference (as noted in L49-50) is that their original experiments were conducted on llama2(MHA), while ours used llama3.1(GQA).

The observed performance gap arises because MHA typically requires a KV cache several times larger than GQA, enabling TriForce’s acceleration.

Q: Clarify Statement of Baseline

A: Thank you for your question.

1. Limitations in Prior Work: The values 256 and 64 in Table 1 are just examples. We also optimized TriForce and experimented with generating 100K lengths (Table 3). It does achieve a 2x speedup on MHA, but not on GQA.

2. Length Restrictions in Baselines: As observed in practice, baselines enforce strict limits on generation length: TriForce caps generation at the KV Cache budget size:

   def update_graph_cache(self, kv_cache=None):
       self.value_cache[:,:,self.max_budget-(kv_cache.seq_len-self.prefill):self.max_budget] = kv_cache.value_cache[:,:, self.prefill:kv_cache.seq_len].clone()
       self.key_cache[:,:,self.max_budget-(kv_cache.seq_len-self.prefill):self.max_budget] = kv_cache.key_cache[:,:, self.prefill:kv_cache.seq_len].clone()
   

Q: Penalty Value

A: During experiments, we observed that while higher θ (e.g., 1.5) increases diversity, it often leads to incoherent outputs or even garbled text in practice. (Table 8) Thus, we adopted θ=1.2 as a empirically stable default, balancing diversity and quality.

Q: Typos

A: These were typos in our manuscript, and we will correct them in the revised version. To clarify: when θ=1.0, no repetition occurs.

We hope our answers have resolved your concerns.

Thank you very much for your valuable suggestions and acknowledgment of our method. We are encouraged by your recognition of our thorough ablation studies and the practical impact of our method, which achieves up to 3× speedup across diverse architectures. We also appreciate your note on the wider community would benefit from our findings. Below, we will address your concerns:

Q1: Nitpick: The title "FROM Hours to Minutes" might be slightly misleading

A1: We appreciate this observation. We will remove this phrase from the title in the revised version.

We hope our answers have resolved your concerns. If you have any other concerns, please feel free to let us know. Thanks again for your review.

We sincerely thank you for your time and constructive feedback. We are encouraged by your positive assessment that our experiments convincingly demonstrate a 3× speedup, the sensible design of our method for ultra-long sequence generation, and the rigorous ablation studies on sampling methods, temperature, and penalty windows. Below, we address their concerns point-by-point.

Q1: though “minimal training cost” lacks detailed quantification beyond training three linear layers (§3.1), slightly weakening its evidential strength.

A1: Thank you for highlighting this point. We will provide a detailed breakdown of the quantized training cost in the revised manuscript. Specifically, for any large language model, our approach requires only γ（Number of tokens generated in parallel） × hidden_size × hidden_size trainable parameters. The specific training time is shown in Appendix I. For example, the training time for models 8B does not exceed two hours. The concrete results for the LLMs used in our experiments are summarized in the table below:

Q2: However, the resource cost of maintaining full KV cache for verification isn’t quantified, which could affect scalability claims on resource-constrained settings—an area for minor clarification.

A2: Thank you for your suggestion. The full KV Cache memory overhead can indeed be derived from the model configuration. Taking Llama3.1-8B (GQA) as an example (batch size=1, bfloat16), the peak memory usage is calculated as: 2 × Layer Num × Batch Size × Seq Len × KV Heads × Head Dim × Bytes = 2 × 32 × 1 × 102,400 × 8 × 128 × 2 bytes ≈ 13.4 GB. The table below summarizes the peak memory requirements for other LLMs in our study when maintaining full KV Cache:

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question! Thanks again for your review.