SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference

We sincerely thank the reviewer SmbH for the effort in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

---

1. The discussion examines how effectively SparseVLM performs on highly complex tasks.

Thank you for your attention to our performance on more complicated tasks. Firstly, due to adaptive sparsification, our method can determine whether the vision tokens in the current layer are redundant for the prompt to avoid losing critical information. Besides, our token recycling mechanism can retain as much information as possible in a cost-effective manner. To validate the effectiveness, we perform experiments on MMMU Pro, MMBench (Attribute Reasoning, AR), and MMBench (Logical Reasoning, LR) with LLaVA-7B. Our performance on MMMU Pro consistently approaches the average accuracy reported in Table 1 across all token settings, demonstrating our method's robustness in highly complex tasks. Furthermore, even with a severe reduction in the number of tokens from 576 to 192, the MMBench (LR) performance surpasses that of the baseline. The reason is that SparseVLM progressively sparsifies tokens, enabling the model to gradually focus, transitioning from the image background to the objects within the image and then to the details of those objects, and achieve outstanding performance. In summary, our SparseVLM can well handle highly complex tasks while maintaining vital information.

---

2. The analysis of the impact of token recycling in more complex, multi-step reasoning tasks.

We appreciate the insightful suggestion. To further explore the effectiveness of our token recycling strategy in more complex and multi-step reasoning tasks, we conduct ablation experiments on MMBench (LR) and MMMU Pro. Across multiple sparsity ratios (64, 128, 192), our algorithm achieves a significant average performance improvement of $0.8$% and $1.0$% on MMBench (LR) and MMMU Pro, respectively. Notably, as the number of pruned vision tokens increases, the benefit brought by our recycling method increases. For instance, when pruning from 192 to 64 tokens, pruned token recycling significantly boosts accuracy from $0.2$% to $1.6$% on MMMU Pro. In short, compared to common tasks (e.g., GQA in Table 4), our token recycling mechanism proves its higher value in complex, multi-step reasoning tasks.

---

3. Further experiments on other transformer-based VLM architectures.

We sincerely thank you for your advice. We have conducted experiments across various transformer-based architectures, including different vision encoders (e.g., CLIP in LLaVA and CLIP+ConvNeXt in Mini-Gemini) and different LLM decoders (e.g., LLaMA in LLaVA and Qwen in Qwen2-VL). To further validate compatibility, we also tested our approach on Cambrian-1 13B (576 tokens setting), which is another transformer-based VLM architecture. Shown in the table below, under the 192-token setting, our method achieves a $2.3$% lower performance drop than PDrop; meanwhile, under the 64-token setting, our approach maintains an accuracy loss below $8.0$%. Therefore, our method is fully compatible with other transformer-based architectures.

---

We sincerely appreciate your thorough review and valuable suggestions to improve our work. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.

We sincerely thank the reviewer bpD8 for the effort in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

---

1. The analysis of the generality of SparseVLM to other general multimodal tasks.

We sincerely appreciate your valuable suggestion. To investigate how SparseVLM sparsifies visual tokens when dealing with questions similar to "Describe this photo in detail.", we further visualized the token sparsification results from different layers of SparseVLM when handling these questions. It was surprising that when confronted with such questions, SparseVLM first shifts the model's attention to the background information of the image. Subsequently, it sparsifies the tokens corresponding to this part and focuses on the information of the main objects in the image (e.g., the people in the image). Finally, it pays attention to the detailed information of the objects (e.g., the nose, mouth, and eyes of a person), enabling the model to describe the image smoothly and progressively. Therefore, our method follows a global-to-local understanding process, yet still effectively captures the key information in captioning tasks. For a visualization example, please refer to the anonymous link: https://imgur.com/a/lbp1qkI.

---

2. The performance of SparseVLM on other general multimodal tasks.

(1) Qualitative analysis

Here we take the captioning task as an example and conduct a qualitative analysis. Specifically, we ask the model with the question_id of 1 in LLaVA-Bench (In-the-Wild), where the prompt is "Describe this photo in detail".

The GPT-4 answer is: This photo features an aerial view of Diamond Head, located in the Hawaiian Islands. The volcanic tuff cone appears prominent and impressive, with its ridges and slopes covered in lush green vegetation. The crater of Diamond Head is clearly visible, surrounded by the steep walls of the structure. In the background, you can see the vast expanse of the ocean with varying shades of blue, showcasing the pristine waters surrounding the island.

The original LLaVA answer is: The image showcases a breathtaking view of a large, rocky island with a mountainous terrain. The island is surrounded by a beautiful blue ocean, and it appears to be a tropical paradise. The island is covered in lush greenery, and there are several houses scattered across the landscape. In the distance, a city can be seen, adding to the picturesque scene. The combination of the island's natural beauty and the city's presence creates a unique and captivating view.

Our method (192 tokens) answer is: The image showcases a breathtaking aerial view of a large, rocky island with mountainous terrain. The island is surrounded by a beautiful blue ocean, and the shoreline is dotted with houses scattered across the landscape. The houses vary in size and are positioned at different distances from the shore, creating a lively and picturesque scene.

From the answers, it can be seen that our method successfully preserves the general style, atmosphere, and scene composition of images. The observed minor omissions in specific details (such as 'lush greenery') represent a conscious trade-off, as we intentionally maintain only $33.3$% (192/576) of visual tokens to optimize efficiency.

(2) Quantitative analysis

We further performed experiments on LLaVA-Bench (In-the Wild) and LLaVA-Bench (COCO), and the results are shown in the table below. It can be observed that SparseVLM achieves significant performance in the task of general visual dialogue. When the number of tokens is sparsified to 192 tokens, it may even lead to a slight performance improvement, which demonstrates the effectiveness of our algorithm. The reason is that SparseVLM is capable of progressively sparsifying tokens, enabling the model's attention to gradually focus, transitioning from the image background to the objects within the image and then to the details of those objects. This allows the model to understand the images in a step-by-step manner, and achieve outstanding performance in tasks such as general visual dialogue.

---

We sincerely appreciate your thorough review and valuable suggestions to improve our work. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.

---

1. References on the computation of the rank of the matrix.

The rank of a matrix is the maximum number of linearly independent rows, determined by singular value decomposition. We will add a reference to the corresponding supplementary part.

---

2. Explanation of the selection via the rank method and its threshold.

Mathematically, matrix rank quantifies non-redundant information via its linearly independent vectors. The motivation of Rank(P) is to adaptively determine dynamic thresholds to identify redundant visual tokens per layer for sparsification, eliminating heuristic sparsity ratios requiring manual tuning in other methods. Experiments scale Rank(P) solely for fair comparisons under matching sparsity to demonstrate algorithm superiority. Practical deployment requires no scaling adjustments or tuning.

---

3. Ablation study on determining the number of deletions.

We conducted comparative experiments with min-$k$ (top-$k$) selection strategy. Compared to the min-$k$ method, our rank method outperforms across all token settings. The results suggest that the rank-based method is more advantageous in information-constrained scenarios.

---

4. Adjustment of the position IDs.

Token pruning in SparseVLM creates layer-varying KV token counts, typically below the original sequence length. Using unpruned position IDs misaligns RoPE, as IDs mismatch the pruned KV cache. We solve this via dynamic position ID recalibration: (1) Track retained tokens from prior steps via LLaVA’s KV cache. (2) Reassign position IDs to preserved/recycled tokens. (3) Apply RoPE with updated IDs. Our method updates positional encodings during token additions/removals, ensuring alignment with pruned sequences and spatial consistency across encoding schemes.

---

5. Performance of SparseVLM on more challenging benchmarks.

(1) Image tasks

We tested on MMBench (LR) and MMMU Pro. Our performance on MMMU Pro approaches the average accuracy reported in Table 1 across all token settings. Besides, SparseVLM progressive token sparsification even improves MMBench (LR) performance at 192 tokens by shifting focus from global to local.

(2) Video tasks

We tested our method on VideoMME and LVBench. Our method prunes $90$% of vision tokens, retaining only 227, while preserving around $99$% of the original accuracy. It outperforms FastV by $6.8$% on VideoMME and $2.2$% on LVBench.

---

6. Claim about "first attempt to explore text-aware guidance" in vision token sparsification.

(1) VideoLLM-MoD uses a trained linear projection to predict the importance of vision tokens. The basis is from page 5 of their paper: "The LayerExpert determines the importance score μ for a given vision token using a linear projection." Reference: arXiv:2408.16730.

(2) FreeVideoLLM uses the prompt feature for temporal and spatial samplings. The evidence is from page 5 of their paper: "We calculate the relation score (i.e., cosine similarity) between the frame features and the prompt feature." Reference: arXiv:2410.10441. We consider it as concurrent work with ours since it is not accepted yet. Still, we will update it in our related work.

(3) Our approach builds solely on text tokens and filters the visual-relevant text raters to improve performance, which has a fundamental difference.

---

7. Failure cases for sparsification.

"n281241.jpg" in GQA. Q: "What is the picture hanging above?" The label is "Chair," but the VLM predicts "Wall." The question asks what is above the object, not where the picture is hung. Misguided attention to background tokens causes spatial misinterpretation, resulting in incorrect answers.

"n494677.jpg" in GQA. Q: "Is the weather cloudy?" The label is "Yes," but the VLM predicts "No." Despite sparse vision tokens accurately capturing the sky, the model still errs, indicating reasoning limitations rather than sparsification flaws.

For visualizations, please refer to the anonymous link: https://imgur.com/a/Hq0qYYa.

---

We appreciate your thorough suggestions. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.

We sincerely thank the reviewer TyrH for the effort in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

---

1. The further explanation of our claim about "first attempt to explore text-aware guidance" in vision token sparsification.

Our method is the first to explicitly utilize the guidance of text for the sparsification of VLMs. The detailed discussions are as follows.

(1) LLaVA-PruMerge only leverages the class tokens to predict the importance of spatial visual tokens. The basis for this derives from the statement on page 5 of their paper: "Note that we use the class attention value from the penultimate layer for this calculation." Reference: https://arxiv.org/pdf/2403.15388.

(2) FastV directly utilizes all the tokens, including text tokens, vision tokens themselves, and system tokens to evaluate the significance of vision tokens. The evidence is derived from the following statement on page 8 of their paper: "We simply compute the average attention-score one token received from all other tokens as the criteria in our experiment." Reference: https://arxiv.org/pdf/2403.06764.

(3) In contrast, our approach builds solely on text tokens and further filters the visual-relevant text raters to improve performance, which has a fundamental difference from LLaVA-PruMerge and FastV. Our effectiveness is also validated in Table 1, where our method shows superiority.

---

2. Further explanation on how the correlation among attention vectors relates to visual redundancy.

We identified several relevant studies that also investigated the correlation between matrix rank, model training, and the informativeness of visual tokens.

The first study [1] reveals a positive correlation between the rank of attention matrices and the performance of Transformer models, suggesting that the rank of the attention matrix influences both the representational capacity and learning effectiveness of the attention mechanism. A higher rank generally indicates a greater ability to encode diverse and informative token interactions. On the other hand, lower ranks often indicate linearly dependent attention vectors, which correspond to overlapping or redundant visual information.

Moreover, some methods leverage singular value decomposition (SVD) to prune the attention output matrix, illustrating that low-rank structures often correspond to redundant information. For example, in [2], the authors use SVD to quantify each token’s information contribution. Combined with attention scores, this allows them to prioritize and retain the most informative tokens. This approach closely aligns with our use of rank, as both rely on analyzing the redundancy of visual tokens in attention matrices.

For the mathematical definition of rank, the rank captures the number of linearly independent directions in the attention space—lower rank implies that multiple tokens share similar attention patterns, justifying the removal of redundant ones. So, based on these findings and the mathematical definition of rank, we use Rank(P) to quantify redundancy among visual tokens.

We also validate our approach experimentally by analyzing attention maps in various vision tasks. Results confirm that attention matrices often contain redundant structure, and pruning based on Rank(P) effectively reduces computational overhead with minimal impact on performance.

[1] Min, Zeping, and Zhong Li. "On the Limitation and Redundancy of Transformers: A Rank Perspective."

[2] Tan, Xudong, et al. "TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models."

---

We sincerely appreciate your thorough review and valuable suggestions to improve our work. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.

We sincerely thank the reviewer PtfS for the effort in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

---

1. The analysis of the efficiency of input image resizing and vision token pruning in VLM, which supports dynamic image resolutions.

Firstly, while dynamic resolution capability in VLMs requires additional training compared to fixed-resolution ones, our method is entirely training-free. We still conducted comparative experiments on Qwen2-VL to evaluate our effectiveness. The table below shows that SparseVLM incurs a comparatively minimal performance degradation compared with the resizing manner. Secondly, our method is orthogonal to dynamic resolution and can further reduce the number of tokens on resized images while maintaining nearly lossless accuracy.

---

2. Extending SparseVLM to Hierarchical Vision Encoders VLM architecture.

Since FastVLM has not yet open-sourced its weights and model architecture, we conducted experiments on ConvLLaVA-7B (256-token setting). The table below shows that our results own near-lossless accuracy ($0.6$%) on ConvNeXt vision encoders under the 192 tokens setting. Besides, under extreme compression (64 tokens setting), the performance of PDrop drops by $19.0$%, while SparseVLM only drops by $8.8$%.

---

3. Extending SparseVLM to Multiple Vision Encoders VLM architecture.

Our existing experiments (Figure 4) cover MiniGemini, a VLM architecture employing multiple vision encoders (CLIP and ConvNeXt). Following the reviewer's advice, we further conducted experiments on Cambrian-1 13B. Shown in the table below, our method is well-suited for multiple vision encoders. Under the 192-token setting, our method achieves a $2.3$% lower performance drop than PDrop; meanwhile, under the 64-token setting, our approach maintains an accuracy loss below $8.0$%.

In summary, the experiments on MiniGemini in the paper, along with the above Cambrian-1 and ConvLLaVA experiments, demonstrate that our method can match well with various visual encoder architectures. The reason is that our method is applied to the decoder of VLM, where the visual features are pre-aligned by the projection layers, ensuring the robustness of selecting appropriate text raters.

---

4. The potential implications of SparseVLM in multi-turn conversations.

Currently, most vision token pruning methods (e.g., FastV [ECCV 2024] and PDrop [CVPR 2025]) fail to effectively maintain compatibility with multi-turn conversations. Here, we propose a potentially viable approach, where we can integrate DyCoke-like [CVPR 2025] dynamic pruning: (1) KV Cache Preservation: Pruned tokens' KV Cache is stored via dynamic pruning (DP-KV Cache). (2) On-demand Updates: New prompts trigger DP-KV Cache retrieval for retained tokens and updates for pruned ones. This ensures multi-round consistency by maintaining historical context while adapting to evolving queries.

---

We sincerely appreciate your valuable review and suggestions. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.