Thank you for your valuable feedback in helping refine our work.

• Note: Before reading further, we kindly ask you to check the Author Rebuttal by Authors and the attached PDF for detailed explanations and additional results.

W1. Limited Experimental Scope

We acknowledge the limitation of our initial experiments. To address this, we include additional results on more open LLMs with different sizes and more datasets. Specifically, we conducted experiments with Vicuna 7B and 13B with 4 block draft heads in the attached PDF. These new results also show that our approach scales effectively and consistently improves performance across different model sizes, achieving an additional speedup of over ~20% and ~300% speedup relative to vanilla decoding. Here, the blockwise parallel LMs extended from Vicuna 7B and 13B are trained with alignment stages.

W2. Difference with Existing Work

We believe our contributions, including rescoring methods as well as findings, are both valuable and innovative compared to existing works. Here’s why:

Medusa

• Similarity: Both approaches explore potential gains from TopK predictions of blockwise parallel LM heads.
• Technical Differences:
  • Medusa searches for the best candidate in block drafts without altering the original logits
  • Our method involves rescoring the block drafts to obtain the best candidate.
• Exploratory Differences
  • Medusa does not explicitly address the reasons behind its decoding method.
  • Our work is the first to explicitly address issues such as (1) consecutive repetition, (2) confidence of different heads, and (3) top-k oracle block efficiency.
• Our contributions are orthogonal and different from Medusa, and our method integrated with Medusa shows even better results, detailed in the attached PDF.

Speculative Decoding

First of all, BPD (presented in 2018) is a predecessor and an instance of speculative decoding.

• Similarity: Both approaches use small language models for efficient LLM inference.

• Technical Differences

  • Speculative decoding uses a small drafter to predict the next multiple tokens
  • Our method uses the small drafter (for both local and global rescoring) to rescore the logits of block drafts with a top-k mask, which are then used for speculative inference.

• TL;DR

  • Speculative decoding uses an independent module for drafting
  • Our method corrects the drafts from a dependent speculative module (e.g., BPD and Medusa), which is totally different from the use of the small LM in speculative decoding.

W2-2. Regarding the Contribution of the Global Rescoring with N-gram Models

We believe the global rescoring with n-gram models is valuable and innovative. Here’s why:

1. Efficiency: The n-gram rescoring is highly efficient, taking only 1.6 ms per lattice via OpenFST, making it practical for real-time applications (Table12 & Table13 in Appendix H).
2. Effectiveness: Despite being a classical approach, n-gram models remain effective. Our results show significant improvements in block efficiency and speedup when integrated with BPD.

The assertion that n-gram models lack novelty is not supported by our evidence. Instead, our results clearly demonstrate their practical value and effectiveness.

Q1. Clarity on How Observations Contributes to Algorithm Design

• Section 6.2 (Confidence across multiple heads): This section examines the confidence levels across different prediction heads in BPD, providing insights into how well the parallel heads are trained. These observations, while not directly informing the current design of our rescoring algorithm, are included to align with our paper's goal of "Exploring and Improving Multi-Token Prediction (block draft)." They suggest potential future improvements, such as using variable interpolation weights based on head confidence levels. We will refine the manuscript to clarify this point.

• Section 6.3 (Top-k oracle block efficiency): The concept of top-k oracle block efficiency serves as a theoretical upper bound for the potential improvements achievable with rescoring. By quantifying the maximum possible efficiency, we gain a benchmark against which we can measure the actual performance of our algorithms. This observation led to the realization that current methods, such as BPD and Medusa, have significant room for improvement. As a result, we developed our rescoring methods to approach this upper bound more closely, thereby optimizing block acceptance rates and reducing redundancy.

Q2. Potential Rescoring Issues

To clarify, the original prediction is used for the next-token prediction, but rescoring happens after that (from the 2nd position future token onward) (You can find it in Section 7 Algorithm2. We will make it clearer in the camera-ready version). The rescored top-1 token can differ from the original top-1 token, which is intentional. The original top-1 token for future positions often fails to be accepted for speculative inference, but our findings show that rescoring improves this. Evidence is provided by improvements in block efficiency and speed up.

Additionally, BPD follows a draft-verify-accept structure. Incorrect predictions in the draft phase do not pass verification, ensuring that BPD and Medusa's predictions are identical to vanilla decoding. We focused on the number of tokens accepted, using block efficiency as a hardware-agnostic metric. Based on reviewer feedback, we have also included latency experiments in the attached PDF. Appendix H provides a detailed discussion on LLM inference speed improvements, covering low-level aspects such as KV-Cache and Parameter I/O memory bandwidth in TPU/GPU experiments.

We believe these responses and additional results in the attached PDF address your concerns comprehensively and highlight the novelty of our work.

Dear Reviewer vMaK

In response to the feedback, we enhance the depth and robustness of our work with detailed explanation and additional experimental results. This includes:

• Evaluations with open-sourced 7B & 13B LLMs [Figure A and Table A-D in authore-response pdf ]
• 5 additional datasets (MT-Bench, Natural Questions, GSM8K, RAG, Summarization) [Figure A and Table A-D in authore-response pdf ]
• Clarification on the difference from existing works
• Comparing with recent efficient llm inference method [Table E in author-rebuttal]
• Integration with Medusa decoding (recent extended version of BPD; presented @ ICML2024) [Figure A and Table A-D in authore-response pdf and Table E in author-rebuttal]
• Additional experiments with temperature sampling (T=0.7, 1.0), beyond greedy decoding [Table A-D in authore-response pdf ]

Given the tight timeline, with the discussion phase concluding on Aug 13, we kindly request you to review our responses. We believe our detailed responses provide clarity on the concerns raised. Your feedback is pivotal to the quality of our work, and we earnestly await your thoughts, especially since we have less than 2 days remaining.

Thank you for your efforts in this paper.

Best regards,

Authors

Thank you for your valuable feedback in helping refine our work.

• Note: Before reading further, we kindly ask you to check the Author Rebuttal by Authors and the attached PDF for detailed explanations and additional results.

W1. Comparison with Other Latency Reduction Approaches

We appreciate the suggestion to include comparisons with other latency reduction approaches, such as quantization or model pruning. Our paper's primary focus is "Exploring and Improving Multi-Token Prediction (Block Draft)", which aligns more closely with speculative inference methods. While speculative inference methods like ours are orthogonal to pruning and quantization, we recognize the importance of providing a thorough comparison for a comprehensive understanding.

Recent works such as Medusa [1,2] (presented @ ICML 2024, last month; extended version of BPD) are examples of speculative inference techniques similar to ours. Although our paper was submitted before many of these studies were published or open-sourced, we have now included a comparative analysis in the Table E (see Author Rebuttal by Authors) and the attached pdf. This analysis demonstrates that our method consistently enhances the efficiency of blockwise parallel LMs (BPD/Medusa), outperforming other speculative inference approaches (achieving an additional speedup of over ~20% and ~300% speed-up relative to vanilla decoding). This inclusion aims to provide a more comprehensive understanding of our method's effectiveness relative to other approaches in the field.

[1] Tianle Cai, et al. "Medusa: Simple LLM inference acceleration framework with multiple decoding heads." ICML 2024.

[2] Ankner, Zachary, et al. "Hydra: Sequentially-dependent draft heads for medusa decoding." arXiv (2024).

W2. Generalization to Larger Models and Computational Cost

We understand the need to evaluate our methods on larger, state-of-the-art models and to discuss the computational costs associated with our rescoring methods. To address this, we have conducted additional experiments with larger models, specifically Vicuna 7B and 13B with 4 block draft heads, as detailed in the attached PDF. These results demonstrate that our approach scales effectively, maintaining its performance gains across different model sizes.

Regarding computational costs, we would like to emphasize that we've already covered a detailed discussion in Appendix H. Appendix H provides a detailed discussion on LLM inference speed improvements, covering low-level aspects such as KV-Cache and Parameter I/O memory bandwidth in TPU/GPU experiments. Our findings show that the efficiency gains from our rescoring methods outweigh the increased computational requirements. The attached PDF supports this statement in terms of the wall clock speedups relative to the standard autoregressive decoding.

W3. Variability in Block Efficiency Improvements

We have extended our experiments to include a wider range of datasets and tasks, as detailed in the attached PDF. While most tasks show significant improvements, we acknowledge that not all tasks benefit equally.

Achieving consistent improvements across all tasks and datasets is a challenge faced by many LLM and LLM studies, not just our own. Various factors, including the characteristics of specific datasets, can influence the results. This is not necessarily a limitation of our method but rather a common challenge in the field [3].

However, our results demonstrate that our method consistently enhances performance, even as model sizes increase (e.g., 7B and 13B parameters). The improvements are consistent for larger models, as highlighted in the attached PDF.

[3] Xia, Heming, et al. "Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding." ACL Findings (2024).

We hope this mitigates your concern and demonstrates the robustness and effectiveness of our proposed methods.

Dear Reviewer rNoJ

In response to the feedback, we enhance the depth and robustness of our work with additional experimental results. This includes:

• Evaluations with open-sourced 7B & 13B LLMs [Figure A and Table A-D in authore-response pdf ]
• Comparing with recent efficient llm inference method [Table E in author-rebuttal]
• 5 additional datasets (MT-Bench, Natural Questions, GSM8K, RAG, Summarization) [Figure A and Table A-D in authore-response pdf ]
• Integration with Medusa decoding (recent extended version of BPD; presented @ ICML2024) [Figure A and Table A-D in authore-response pdf and Table E in author-rebuttal]
• Additional experiments with temperature sampling (T=0.7, 1.0), beyond greedy decoding [Table A-D in authore-response pdf ]

Given the tight timeline, with the discussion phase concluding on Aug 13, we kindly request you to review our responses. We believe our detailed responses provide clarity on the concerns raised. Your feedback is pivotal to the quality of our work, and we earnestly await your thoughts, especially since we have less than 2 days remaining.

Thank you for your efforts in this paper.

Best regards,

Authors

Thank you for your valuable feedback. We understand your concern regarding the use of different models for the rescoring phase and its potential impact on final performance due to varying token generation distributions.

• Note: Before reading further, we kindly ask you to check the Author Rebuttal by Authors and the attached PDF for detailed explanations and additional results.

W1. Additional experiments on different models

To address this, we conducted additional experiments demonstrating that local neural rescoring methods are robust and consistently improve performance across different target models (Instruction-tuned openLLM Vicuna 7B and 13B with 4 block draft heads). Specifically, we have shown that:

1. Our approach significantly speeds up BPD even further for existing, open-sourced, instruction-tuned LLMs.

2. Local rescoring also accelerates decoding for different model architectures, including the very recent Medusa model [1] presented at ICML 2024, which employs tree-attention to enhance performance in blockwise parallel LMs, achieving an additional speedup of over ~20% relative to BPD/Medusa and ~300% speed-up relative to vanilla decoding.

In the attached PDF, we provide detailed results showing consistent performance improvements when our model is applied on top of Medusa decoding. These results highlight the versatility and robustness of our rescoring approach in the field regarding multi-token predictions, confirming that it effectively enhances performance regardless of the underlying model used for token generation.

We hope this addresses your concern and illustrates the robustness and effectiveness of our proposed methods.

[1] Tianle Cai, et al. "Medusa: Simple LLM inference acceleration framework with multiple decoding heads", ICML, 2024.

Dear Reviewer FLJ6

In response to the feedback, we enhance the depth and robustness of our work with additional experimental results. This includes:

• Evaluations with open-sourced 7B & 13B LLMs [Figure A and Table A-D in authore-response pdf ]
• 5 additional datasets (MT-Bench, Natural Questions, GSM8K, RAG, Summarization) [Figure A and Table A-D in authore-response pdf ]
• Comparing with recent efficient llm inference method [Table E in author-rebuttal]
• Integration with Medusa decoding (recent extended version of BPD; presented @ ICML2024) [Figure A and Table A-D in authore-response pdf and Table E in author-rebuttal]

Given the tight timeline, with the discussion phase concluding on Aug 13, we kindly request you to review our responses. We believe our detailed responses provide clarity on the concerns raised. Your feedback is pivotal to the quality of our work, and we earnestly await your thoughts, especially since we have less than 2 days remaining.

Thank you for your efforts in this paper.

Best regards,

Authors