ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding

We are grateful for your insightful questions and suggestions for improvement.

1. Comparison with existing speculative decoding methods for VLMs

We acknowledge and have cited these valuable prior works in our paper. However, as their source code is not publicly available, we encountered difficulties in reproducing their results for a direct, controlled comparison. Therefore, we directly cite the performance data reported in their original papers and apply our method to the same LLaVA-1.5 7B model and datasets they used to ensure a fair comparison. The results are as follows:

As shown in the table, our method significantly outperforms the previous state-of-the-art works across all common benchmarks.

2. Detailed breakdown of vision adaptor overheads

Theoretically, the image adaptor increases the parameter count of the draft model but reduces its computational load during prefilling by processing fewer tokens. Since the draft model is already very small and efficient, we observed no statistically significant impact on its overall speed. The analysis of the prefilling stage on the COCO Captions dataset is presented below:

And the token rate for the decoding stage:

We attribute these minor speed variations primarily to measurement noise.

3. Relationship between output length and speedup ratio

The VLMs we tested tend to generate text of varying lengths across different datasets. We present the relationship between the average number of generated tokens and the speedup below:

Generally, longer text outputs tend to yield a higher speedup ratio, as there are more opportunities for the draft model to make correct predictions. Nevertheless, our method maintains stable and robust performance across all datasets, even those with shorter response lengths.

4. Performance on text-only inputs

We evaluate the speedup on the text-only MT-Bench [3] dataset using the LLaVA-1.6 7B model. We compare the performance of the original text-only draft model with our draft model that has been fine-tuned on multimodal data. The speedup after our multimodal fine-tuning is 0.96x of the speedup achieved before fine-tuning. This demonstrates that our proposed method has a negligible impact on performance for text-only tasks, preserving the model's utility as a general-purpose assistant.

[1] M. Gagrani, et al. "On speculative decoding for multimodal large language models." 2024.

[2] M. Lee, et al. "In-batch Ensemble Drafting: Toward Fast and Robust Speculative Decoding for Multimodal Language Models." 2024.

[3] L. Zheng, et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS, 2023.

We sincerely thank you for your insightful feedback and the opportunity to clarify these important aspects of our work.

1. Only reporting speedup metrics

Our work, ViSpec, is based on the principle of speculative decoding. The token compression and drafting process ONLY involves a small, auxiliary draft model. The final output is always verified by the original, unmodified target model. The speculative decoding process has been mathematically proven to be lossless, meaning it produces an output distribution that is identical to that of standard autoregressive decoding from the target model alone [1, 2].

Because the final output is provably identical, the model's accuracy and any other performance metrics remain unchanged. Consequently, it is standard practice in the speculative decoding literature to report only the acceleration (i.e., speedup), as performance metrics are not affected. Our paper follows this established convention.

2. Experiment on benchmarks containing high-resolution images

We conduct experiments on the suggested benchmarks, and our method continues to demonstrate strong performance. The results are presented below, comparing ViSpec against the EAGLE-2 baseline for both LLaVA-1.6 7B and Qwen2.5-VL 7B.

Notably, Qwen-VL does not limit its input image token count, leading to a significantly longer prefilling time on high-resolution benchmarks. Since speculative decoding accelerates only the decoding stage, this extended prefilling time results in a lower overall speedup ratio. However, ViSpec’s mean acceptance length remains robust, indicating that the acceleration of the decoding phase itself is effective. Interestingly, the baseline method actually decelerates the Qwen-VL model on these benchmarks (i.e., speedup < 1.0x). This finding further validates our core argument from Section 4.1: a shallow draft model’s performance inherently degrades when processing the long and redundant token sequences from high-resolution images, highlighting the critical need for the compression strategy that ViSpec introduces. We will include this analysis in the final version of our paper.

3. Comparison with FastV and MiniCPM-V

Our method is fundamentally different from approaches like FastV and MiniCPM-V. The core distinction lies in the nature of the acceleration:

• FastV & MiniCPM-V (Lossy Compression & Prefill Focus): These methods are lossy model compression techniques. Their main goal is to accelerate the computationally intensive prefilling stage by processing fewer visual tokens. For instance, FastV reduces the number of tokens fed into the transformer, which cuts down the FLOPs for prefilling. However, this approach offers minimal benefit for the decoding stage. Autoregressive decoding is typically memory-bound, where the bottleneck is loading model weights from memory for each forward pass, not the computation itself. Since these methods still require the full model weights to be loaded for every single token generation, they do not alleviate this memory bandwidth bottleneck, and the actual wall-clock speedup during decoding is often negligible.

• ViSpec (Lossless Acceleration & Decode Focus): Our method is a speculative decoding framework that accelerates an existing, unmodified target model. It is mathematically guaranteed to be lossless, meaning it does not alter the model's final output quality in any way. Our focus is purely on reducing the decoding latency. In stark contrast to FastV & MiniCPM-V, ViSpec is designed specifically to accelerate the memory-bound decoding phase by reducing the number of sequential steps required to generate the full output.

Our method is, in fact, orthogonal to approaches like FastV and MiniCPM-V. FastV or MiniCPM-V can be used to compress the target model itself, accelerating the computationally-intensive prefill stage. Following that, our ViSpec framework can be applied to this already-optimized model to accelerate its memory-bound decoding stage. Therefore, these methods are not in conflict; combining them could lead to a cumulative effect, further enhancing overall inference efficiency by optimizing both prefill and decoding stages.

[1] Y. Leviathan, et al. "Fast Inference from Transformers via Speculative Decoding." ICML Oral, 2023.

[2] C. Chen, et al. "Accelerating Large Language Model Decoding with Speculative Sampling." DeepMind, 2023.

We sincerely thank you for your detailed feedback and constructive suggestions. We address your points below.

1. Writing and sectioning issues

We will refine the writing and consolidate these subsections in the final version. If you have more specific suggestions for improvement, we would be very grateful to hear them.

2. Architecturally analogous “Global Visual Feature Integration”

The concept of a "global feature" is widely applied in deep learning. Its core idea is not only adopted in ContextNet [1] but also well-established in numerous other influential works, such as Squeeze-and-Excitation Networks for channel-wise feature recalibration [2], the [CLS] token in BERT for sentence-level representation [3], and the class token in Vision Transformers for image-level classification [4]. However, the mere presence of a global feature is not where the novelty lies. The crucial aspects are the specific motivation for its use and more importantly the method by which it is generated and integrated. In this regard, our Global Visual Feature Integration differs significantly from ContextNet.

• Motivation: ContextNet introduces the global context vector to address the limited receptive field of CNNs, which struggle with long-range dependencies in speech signals. Authors of ContextNet note that this is NOT considered a problem for architectures like RNNs or Transformers. In contrast, as we argue theoretically in Section 4.1, the core issue we address is the inherent deficiency of shallow Transformers (draft models) in processing long and redundant sequences, particularly a large number of image tokens. To the best of our knowledge, we are the first to identify this specific challenge. This insight leads to our proposal to compress image tokens. Consequently, the reduced number of image tokens relative to text tokens introduces the risk of the draft model "forgetting" the visual context. The Global Visual Feature Integration is therefore needed to mitigate this risk.

• Method: ContextNet generates a global context vector from a convolutional block's output using a Squeeze-and-Excitation (SE) module. This vector is then used to perform feature recalibration by multiplicatively rescaling the local feature maps. The global feature, therefore, modulates the local information. We derive the global visual feature vector directly from our vision adaptor module, which uses learnable queries to summarize the image. We then integrate this feature by additively injecting it into the hidden state of every subsequent text token via a learned projection. This ensures that a constant, global representation of the image is available at every step of the generation, acting as a continuous anchor to the visual input.

In summary, while both approaches use a "global feature," they do so for different reasons (CNN limitations vs. draft model forgetting) and with different mechanisms (SE module vs. vision adaptor) tailored to their unique problems. Moreover, as we demonstrate in our ablation study in Section 5.3, our framework significantly outperforms prior work even without Global Visual Feature Integration, highlighting the contribution of our vision-aware token compression strategy.

3. Recognizing fine-grained objects with compressed embeddings

A key principle of speculative decoding is that the performance of the draft model affects only the acceleration speed, NOT the final output quality. If the draft model fails to recognize a fine-grained object, its prediction is simply rejected and corrected by the target model. The output distribution is guaranteed to be identical to that of the target model decoding alone.

In Table 2 of our main paper, we have already presented experiments on three diverse datasets (COCO Captions, GQA, and MME), varying the number of compressed image embeddings from 1 to 64. These results consistently show that increasing the number of embeddings beyond 1 does not yield meaningful gains in speedup, which demonstrates that a single compressed image embedding is often sufficient for answering common visual questions, including relatively fine-grained ones such as, “What is the short person holding?” (an example from GQA) .

To further address the reviewer’s concern about fine-grained tasks, we conduct additional experiments on the more fine-grained OCR dataset SynthDoG EN [5]. The results below show the performance of LLaVA-1.6 7B with greedy decoding as we vary the number of compressed image embeddings.

We observe a performance drop on this benchmark compared to less fine-grained tasks like GQA (from 2.22x down to 1.97x), but our method still significantly outperforms the EAGLE-2 baseline, which uses no compression (last row). While increasing the number of image embeddings from 1 to 4 slightly increases the mean acceptance length (τ), this gain is offset by the additional computational cost, resulting in a decreased end-to-end speedup.

We believe it is counterproductive to force a shallow draft model to capture every fine-grained detail. Its purpose is to generate plausible token sequences for the majority of cases to maximize acceleration. The more difficult cases are efficiently handled by the powerful target model, which is the essence of speculative decoding.

[1] W. Han, et al. "ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context." Proc. Interspeech, 2020.

[2] J. Hu, L. Shen, and G. Sun. "Squeeze-and-Excitation Networks." CVPR, 2018.

[3] J. Devlin, M.W. Chang, K. Lee, and K. Toutanova. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." NAACL, 2019.

[4] A. Dosovitskiy, et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." ICLR, 2021.

[5] G. Kim, et al. " Donut: OCR-Free Document Understanding Transformer." ECCV, 2022.

We appreciate you taking the time to review our paper and providing this valuable feedback. We address your specific concerns below.

1. Novelty of this work against BLIP-2 and EAGLE

Our primary contribution is the identification and analysis of a fundamental bottleneck in applying speculative decoding to Vision-Language Models, an issue unaddressed by prior work like BLIP-2 and EAGLE. We subsequently introduce a comprehensive solution to tackle this challenge: a lightweight Vision Adapter to compress redundant visual information and a Global Feature Injection mechanism to combat the lost-in-the-middle effect. These are complemented by novel dataset generation and training strategies designed to produce high-quality, long-form training data and to train a robust, effective draft model.

In Section 4.1, we provide a theoretical analysis from the unique perspective of speculative decoding. We demonstrate how the performance of a shallow draft model degrades when processing the long and highly redundant sequences typical of image patch embeddings. Our analysis reveals that as the number of redundant visual tokens increases, a shallow Transformer's attention mechanism becomes saturated, effectively averaging over the redundant inputs while ignoring unique and critical information. This leads to our central argument: for speculative decoding to be effective in a multimodal context, this redundant visual information should be compressed before it is processed by the draft model. This core insight directly motivates our structural solution: the vision adaptor for compression and the global feature injection for maintaining context.

2. Performance gap between COCO Captions and TextVQA

The difference in speedup between COCO Captions (3.22x) and TextVQA (2.90x) likely stems from the differing granularity of these two tasks. COCO Captions require a holistic, overall description of the image, a task for which our compressed visual representation is highly effective for the draft model. In contrast, TextVQA is a fine-grained task that demands focusing on specific text within the image, token compression may reduce the draft model's ability to predict these fine-grained details, lowering the acceptance rate. This does not affect the final output quality, as the target model makes corrections, but it does logically result in a lower speedup ratio.

Another factor that can contribute to varying speedups across different benchmarks is the length of the generated output. This occurs because speculative decoding accelerates only the decoding stage, not the prefilling stage. Since the overall speedup calculation includes the fixed prefilling time, tasks that produce longer sequences see a greater impact from acceleration. For instance, on the GQA dataset, where the average response is short (46 tokens), we observe a 2.22x speedup. In contrast, on MME, with longer responses (115 tokens), the speedup increases to 2.55x. For longer outputs, the accelerated decoding phase constitutes a larger proportion of the total inference time, naturally leading to a higher end-to-end speedup ratio.

3. Experiment with Temporal Data

In principle, our method could be even more effective for video inputs, as video data contains additional temporal redundancy compared with static images. Since video inputs are typically processed as a sequence of embeddings, the problem is not fundamentally different from handling image patches. To test this hypothesis, we apply our draft model, which was trained purely on static image data, directly to video tasks. For a preliminary experiment, we compress each video frame into a single embedding, average their corresponding global features, and evaluate the Qwen2.5-VL 7B model on the MSVD-QA [1] and MVBench [2] datasets. The MSVD-QA dataset is a video question-answering task where the model must answer natural language questions based on the content of a video. MVBench is a benchmark specifically designed to evaluate temporal understanding in videos, featuring 20 different tasks that require reasoning about actions, their sequence, and temporal relationships. We limit the maximum number of frames to 32, as processing more frames would only lengthen the prefilling time, diminishing the speedup gained from accelerating the decoding stage. Results are presented below:

The results demonstrate that our ViSpec method achieves a notable speedup even without any video-specific training, whereas the baseline method actually decelerates inference on MVBench (0.83x speed). Developing a dedicated framework optimized for video remains a promising direction for future work.

4. Speedup on multimodal benchmarks such as MME

As we have described in Appendix A, "Implementation Details," following [3], we use modified prompts to instruct the model to generate long-form, explanatory answers, rather than just single-word responses. For example, we ask the model to "Provide a detailed description of the given image." This allows us to rigorously evaluate the decoding acceleration on these benchmarks.

5. Experiment in high-concurrency settings

We conduct experiments on the LLaVA-1.6 7B model using the SQA dataset, with the batch size varying from 1 to 8. We compare ViSpec to the standard batched decoding inference in the Hugging Face Transformers library. The results are as follows:

We still observe a substantial gain when using batched decoding. The principle of speculative decoding is to lower the number of memory access cycles to accelerate memory-bound decoding. Therefore, it is expected that the relative speedup over standard decoding will decrease as the batch size increases, since higher concurrency better utilizes the computation units. Nevertheless, our method maintains a significant speedup across all tested batch sizes. Implementation optimizations could further boost this performance.

[1] D. Xu, et al. "Video question answering via gradually refined attention over appearance and motion." ACM-MM, 2017.

[2] K. Li, et al. "MVBench: A comprehensive multi-modal video understanding benchmark." CVPR, 2024.

[3] Gagrani, Mukul, et al. "On speculative decoding for multimodal large language models." 2024.