CASD: Enhancing Generation Accuracy via Context-Aware Speculative Decoding

1. Overall the method is a minor modification of REST, simply replacing the external database with the input context. It is only a new way to build the retrieval database.
2. The modification is less general than the original REST algorithm, since it only applies to tasks whose prompts are long and the answers are likely to appear in the prompts. When the input prompts are short, the modification will not work.
3. Some important experiment results are missing. No comparison with traditional speculative decoding (with a small LM) on speedup ratios is provided. To ablate the effect of conditional verification with the threshold, the method should be compared with the acceptance method of REST, not top-k verification.

1. Limited novelty: This paper primarily focuses on draft retrieval and employs a probability threshold to accept or reject tokens, which I consider as an incremental contribution.

2. Limited soundness: Unlike original Speculative Decoding, CASD uses the original model as the draft model to compute probabilities, which seems like a backward step. The draft model in Speculative Decoding plays a crucial role in speeding up the method; relying on the original model for probabilities may yield only minor speed gains. Additionally, CASD may violate the traditional autoregressive decoding style since it accepts the longest draft, which may cause unwanted text behavior.

3. Limited baseline comparison: The paper does not compare CASD with other prompting baselines such as the original SD. In addition, its experiments are conducted with only LLama.

4. Threshold sensitivity: The results indicate that performance varies significantly with different threshold settings.

5. Limited writing: The writing can be significantly improved. For example, the details of prompt compression are limited. It's unclear why using LLMLingua-2 and it's unclear why it works.

1. The difference between the proposed method and REST [1] is minimal, same exact retrieval algorithm is used; the general database in REST could also be replaced by the input context.
2. Training-free / model-free speculate decoding baselines (e.g. [2]) is not compared.
3. Different templates is need for different datasets. It's not clear whether CASD is suitable for the most common chatbots scenarios.
4. This method tends to make the output distribution more inclined to restate the input context, which is advantageous for extractive QA tasks where the answer is present in the input. However, its impact on more general scenarios remains unclear.

[1] REST: Retrieval-Based Speculative Decoding [2] Inference with Reference: Lossless Acceleration of Large Language Models

• One limitation of CASD is its generalizability to all tasks. While it excels in tasks that benefit from direct context, its effectiveness may be limited in tasks requiring abstractive or creative generations.

• As stated in the paper, CASD conflicts with common sampling techniques in autoregressive models, which makes it less practical for many scenarios that require multiple generations per sample.

• Although the experiments are conducted on multiple datasets, the baseline is too limited, which is vanilla decoding of LlaMa3.1-8B-instruct model only. While it’s understandable to skip decoding methods that require additional training or draft models, there are other training-free speculative decoding methods available to compare with. Comparisons with these alternative approaches, such as REST, are crucial for a more comprehensive evaluation.

• The choice of the probability threshold for accepting draft tokens during verification significantly impacts CASD's performance. The sources acknowledge that a lower threshold can enhance accuracy with reliable contexts but might compromise fluency. However, the experiments primarily rely on fixed or manually tuned thresholds, which lack generalizability and require prior knowledge of the dataset characteristics. It would be better if the threshold could be determined automatically by a generalizable heuristic rule.

• The setting of the vanilla decoding is unclear anywhere in this paper, i.e., temperature, sampling sizes, maximum context length and maximum generation token length. These can affect the model's performance and thus should be described in the experiments section.