SparseVLM: Visual Token Sparsification for Efficient Vision Language Models Inference

We sincerely thank the reviewer Rbrk for the efforts in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

1. In Table 1, even the setting with the least acceleration, "Retain 192 Tokens," exhibits substantial performance drops across multiple benchmarks.

Thank you for your insightful comments.

First, we want to clarify that there is an efficiency-performance trade-off in token sparsification. Our method achieves state-of-the-art performance at comparable efficiency levels with other methods. While "retain 192 tokens" is a more aggressive setting that may slightly impact performance, we can achieve marginal drops with more moderate sparsification ratios.

As shown in the additional experiments in the table below, retaining mid-range token counts of 448, 384, and 320 allows us to maintain performance while realizing significant efficiency gains. For example, with 22% of the vision tokens deleted (resulting in 448 tokens), the average performance drop is only 0.5%.

Note that this trade-off is adaptively controllable by our proposed sparsification level adaptation, which means that one can easily set the sparsification ratio to achieve the desired efficiency and performance.

2. Why was the unusual number of 142 image tokens chosen? Settings retaining different proportions of image tokens should be tested.

(1) Reason for 142 image tokens: The 142 tokens are adaptively selected by the scaling factor $\lambda$ in equation (8), which is designed for fine-grained sparsifications in each layer.

(2) Experiments on various numbers of tokens remained: We added experiments to test LLaVA with 320, 192, and 64 remaining tokens. As shown in the table below, our method with a rater selection mechanism obtains the optimal results, particularly in POPE (e.g., a gain of 2.2 is noted at the 192 tokens setting). Furthermore, we gain significant improvements compared to the approach in FastV [1] (using all tokens). This indicates that having more raters does not necessarily lead to better outcomes, while selecting raters that are relevant to the visual context is crucial.

Through the above table, we observe that our rater selection mechanism is optimal, particularly obvious in the POPE (e.g., a gain of 2.2 is noted at the 192 tokens setting). Furthermore, we verify that contrary to the approach of FastV [1], having more raters does not necessarily lead to better outcomes; rather, selecting raters that are relevant to the visual context is crucial.

[1] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." European Conference on Computer Vision. Springer, Cham, 2025.

3. The number of tokens is adaptively determined by calculating N in the method, but in the experiments, the number is set to specific values (e.g., 192), how?

(1) In fact, N directly determines the degree of sparsification, but the scaling factor $\lambda$ in equation (8) influences N. Therefore, we can delete a specific number of irrelevant vision tokens via $\lambda$. For instance, if you want to sparsify the number of vision tokens to 192 on MME, the $\lambda$ should be set to 13.5; if you wish to sparsify the number of vision tokens to 64 on TextVQA, the $\lambda$ should be set to 0.8.

(2) The result of N in a decoder layer cannot be 0, because when the rank equals $L_v$, it indicates that P is full rank, and we will skip the sparsification for that layer as said in line 244.

12. 5.1 section "3 settings (using all tokens, only text tokens, and only text raters we select)". explain the settings in detail.

In this ablation experiment, we evaluate vision tokens with three different types of raters:

(1) All tokens: this manner is the same as FastV, where all the tokens (text tokens, vision tokens, and the system token) function as raters. The pseudocode is as follows.

relation_vis_text = self_attn_weights[:, :, v_token_start: v_token_start+v_token_num]

(2) All text tokens: this manner means only all the text tokens function as raters. The pseudocode is as follows.

relation_vis_text = self_attn_weights[:, text_token_start:v_token_start, v_token_start: v_token_start + v_token_num]

(3) Text raters we select: based on the manner (2), we filter out the visual-relevant text tokens to evaluate vision tokens. The pseudocode is as follows.

relation_vis_text = self_attn_weights[:, t_token_idx, v_token_start: v_token_start+v_token_num]

13. Please give a clear explanation of your sparsification process, which is not stated in the paper.

(1) Yes, we do the pruning and recycling process during the prefilling stage, which occurs before the VLM begins its autoregressive token-by-token generation process.

(2) In the prefilling stage, we start to prune and recycle in specific sparse layers (e.g., 2, 6, 15, 19), and have no operations in the left normal layers. During the generation process, we do not prune vision tokens anymore. Therefore, the specific token numbers in experiments (e.g., 64, 128, 192) are computed in the prefilling stage.

latency-vs.-average accuracy

TFLOPs-vs.-average accuracy

In summary, the above experiments fully demonstrate the effectiveness of our method in reducing latency and computational complexity.

8. The explanation of our claim "it is the first attempt to explore the potential of text-aware guidance for efficient inference of VLMs.

Actually, our method indeed is the first work to explore the potential of text-aware guidance for the sparsification of VLMs. FastV [1] just simply utilizes all the tokens, including text tokens, vision tokens themselves, and system tokens to evaluate vision tokens. In contrast, our approach builds solely on text tokens and further filters the visual-aware text raters to improve performance. This effectiveness is also validated in Q2, where our method shows superiority.

[1] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." European Conference on Computer Vision. Springer, Cham, 2025.

9. The description of the ToMe method in the Related Work section is inaccurate.

Thank you for your observation regarding the description of the ToMe [1] method in the Related Work section. We appreciate your attention to detail. We will revise the description to reflect the mechanism employed by ToMe accurately. Specifically, we have revised to "ToMe (Bolya et al., 2022) merge similar visual patches in Transformer blocks and speed up the match process through the Bipartite Soft Matching (BSM) algorithm".

[1] Bolya D, Fu C Y, Dai X, et al. Token merging: Your vit but faster[J]. arXiv preprint arXiv:2210.09461, 2022.

10. In the introduction, the calculation of the number of image tokens seems incorrect.

Thank you for your careful review of related work. We acknowledge that the claim stating "a 672 × 672 image in LLaVA-HD yields 2304 vision tokens" is inappropriate. The accurate calculation should reflect that a 672 × 672 image results in 576 * 5 tokens, based on the configuration of four sub-images plus one resized original image, leading to a total of 2880 tokens. We corrected this in the revised manuscript and thank you for bringing this to our attention.

11. In Table 1, for the experimental results under the settings "Retain 192/128/64 Tokens," what exactly do these settings mean? For FastV, does this mean that only this number of image tokens is retained across all layers?

(1) Clarification of Settings: These settings are established to demonstrate the generality and robustness of our method under fewer tokens count. We select 64, 128, and 192 tokens at regular intervals (64) to evaluate performance across different token counts. (2) Settings for FastV: For FastV, we employ its pruning algorithm in the first layer, and this implies that only this specified number of image tokens is retained across all layers of the LLM decoder.

3. Designation of V Matrix

In order to return the mean value of the attention scores for the selected text raters directly in Flash Attention, we need to design a special V matrix.

Here, $V$ is an $n \times d$ matrix, $n$ is the total number of rows in the matrix, $i$ is the row index, $1 \leq i \leq n$, $S = \\{ i \mid R[i] \geq m, \, i \in \\{1, 2, \dots, L_t\\} \\}$ define the text raters which we selected in Section 3.2.

4. Incremental Accumulation

Rather than storing $P$ explicitly, the result is directly accumulated into the output using:

The final result is obtained by concatenating all blocks:

5. Streaming Softmax

When combining multiple blocks, an incremental softmax computation ensures that normalization is maintained across the entire sequence:

This avoids global dependencies and enables efficient block-wise computation.

6. Top-k selection for vision tokens

The top-k selection can be expressed as:

where $O = \text{Concat}(O_1, O_2, \ldots, O_B)$ is the output array of second Flash Attention, $O_v$ is the vision tokens part of $O$, $\text{rank}(x_i, O_v)$ represents the position of $x_i$ in $O_v$ when sorted in descending order.

The corresponding indices of the top-k elements are:

Summary Formula

The complete process of SparseVLM Flash Attention can be summarized as:

Here, each block $B$ is processed independently, and the results are combined using incremental normalization.

[1] Dao, T. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems.

[2] Dao, T. (2023). Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.

Comparsion

Here, we utilized the mentioned SparseVLM FlashAttn in Part 2 compared with the baseline matched with FlashAttn. We conducted comprehensive experiments on LLaVA and MGM across three benchmarks: POPE, TextVQA, and MME. The results are shown in the following table. Besides, we compared our method with the random sparse matched with FlashAttn, and we observed that our method has significant improvement, when under a similar CUDA time.

7. The additional efficiency experiments and explanation about latency and FLOPs on image understanding tasks.

Thanks for the suggestion. We have complemented more experiments and revised the paper accordingly.

(1) More Trade-off Comparison: Thank you for your detailed suggestions. We have included latency-vs.-accuracy and FLOPs-vs.-Accuracy tradeoffs for SparseVLM applied to LLaVA and MGM across three benchmarks: POPE, TextVQA, and MME. Figures 9 and 10 in Appendix A.7 now include a total of 12 sub-figures illustrating these tradeoffs on the image understanding tasks.

(2) Efficiency Summary: The following table presents the latency-vs.-average accuracy and FLOPs-vs.-Average accuracy results for individual datasets, providing a comprehensive overview of our method's efficiency.

4. Why the linear correlation among attention vectors would relate to visual redundancy?

We identified two relevant studies on the rank of the attention matrix. The first paper [1] shows a positive correlation between attention rank and Transformer performance, indicating that higher ranks lead to better model effectiveness until a saturation point is reached.

The second paper [2] explores the limitations and redundancies of attention mechanisms, demonstrating that variations in attention matrix rank correlate strongly with visual feature redundancy.

Building on these insights, we apply the concept of attention matrix rank to LLM decoder layers. The rank quantifies the linearly independent information within the matrix, reflecting relationships among tokens. Linearly dependent rows suggest redundancy, allowing us to retain one token while pruning others.

We calculate attention matrix ranks for various tasks and adaptively prune tokens based on these ranks. Our extensive experiments show that attention matrices exhibit redundancy, enabling effective pruning without significant performance loss.

[1] Min, Zeping, and Zhong Li. "On the Efficiency of Transformers: The Effect of Attention Rank."

[2] OpenReview. "On the Limitation and Redundancy of Transformers: A Rank Perspective."

5. Figure 1 shows that the patches selected by fastv are identical under different questions, which is unreasonable.

(1) We sincerely apologize for this oversight. While the visualization in Figure 1 is correct, the caption contains an error and should be revised to 'VocoLLaMA [1].' Its sparsification is unrelated to the text.

(2) Our method has a fundamental difference from FastV [2]. FastV does indeed evaluate vision tokens using text tokens, but it also incorporates the vision tokens themselves and the opinions of system tokens. Our approach, on the other hand, builds solely on text tokens and further filters the text raters to ensure their relevance to visual information. This effectiveness was also validated in Q2, where our method showed superiority.

[1] Ye, Xubing, et al. "VoCo-LLaMA: Towards Vision Compression with Large Language Models." arXiv preprint arXiv:2406.12275 (2024).

[2] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." European Conference on Computer Vision. Springer, Cham, 2025.

6. SparseVLM compatibility with Flash attention and their comparison.

Our SparseVLM can be compatible with FlashAttn. The following is our solution.

SparseVLM Flash Attention:

To ensure that SparseVLM remains compatible with Flash Attention, we devised a method to extract the mean value of the processed attention map without explicitly obtaining the full attention map. In normal decoder layers that do not require pruning, we use the original Flash Attention directly. For layers where pruning is necessary, we implemented an additional Flash Attention-based operation to directly obtain the mean attention scores w.r.t. the text raters, which is lightweight and also enjoys the efficiency in Flash Attention.

Specifically, the first forward pass operates identically to the original Flash Attention, generating the hidden states for all tokens before pruning. In the second forward pass, we introduce a specially designed V matrix. In this matrix, for the rows corresponding to the text raters we wish to analyze, we set the values to $1 / \text {len(text raters)}$. This configuration allows the inner product between the attention map and the V matrix to return the mean value of the attention scores for the selected text raters directly in Flash Attention.

Using this mean value, we perform a top-k selection to identify the vision tokens to retain. Tokens that are excluded during this process are converted into masks, which are then applied to the hidden states produced by the first Flash Attention pass to complete the pruning operation. This method enables efficient integration of pruning with Flash Attention while preserving compatibility and computational efficiency.

Core Principles and Calculation of SparseVLM Flash Attention

1. Attention Score Calculation

For each block $B$, compute the scaled dot-product attention scores:

Here, $S_B$ is the attention score matrix computed within the block.

2. Block-wise Softmax

To ensure numerical stability, the softmax is computed in a stable manner using the log-sum-exp trick:

(1) Subtract the maximum value for numerical stability:

(2) Normalize:

We sincerely thank you for your efforts in reviewing our paper. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether your concerns have been addresses or not. Please let us know if you still have any unclear part of our work.

Best, Authors

Dear reviewer Rbrk,

I hope this message finds you well. We greatly appreciate the valuable feedback and suggestions you have provided so far. As the deadline approaches, we are eager to receive your feedback on our response and revisions. If possible, we kindly request an update on the progress of the review to ensure we can address any further comments or revisions promptly.

Should you require any additional information or assistance from our end to help facilitate the review process, please do not hesitate to let us know. Your insights are highly valuable to us, and we genuinely appreciate your time and effort in reviewing our paper.

Thank you for your patience and cooperation. We are looking forward to hearing from you soon.

Warm regards,

Submission402 Authors.

Thank you for your response.

Firstly, your rating of "5" in the previous review was determined by the initial 10 weaknesses, most of which we have addressed significantly. Could you please specify any obvious issues that have emerged leading to a lower evaluation from you? We are very open to receiving more detailed feedback from you, pinpointing the exact issues that remain unresolved. This feedback would greatly assist us in our future revisions.

Regarding the Raters used in FastV. FastV works differently between with and without KV cache. We carefully examined the source code and found that it does indeed evaluate vision tokens using the last text token only when the KV cache is applied. Besides, the statement in the text, 'compute the average attention score one token received from all other tokens as the criteria,' has proved it utilizes all the tokens.

The correction to FlashAttention. In comparison to the baseline method Flash Attention, our approach demonstrates significant advantages in terms of FLOPs and CUDA Time. We are curious about how you arrived at "doubts about the fundamental understanding of this field.

4. Why use the features from the visual encoder and text embeddings to select raters?

Following the Pre-training stage of the VLM, the visual and textual modalities are effectively aligned, enabling direct similarity computations to select raters. Furthermore, we choose the selection before entering the LLM, rather than conducting it within the LLM sparse layers, to save computational resources.

5. The performance of logical reasoning tasks.

We conducted comprehensive experiments on four logical reasoning benchmarks, including MMMU, MMMU Pro, MMBench Attribute Reasoning (AR), and MMBench Logical Reasoning (LR). The results of our method are listed in the following table. We are surprised to observe that even with a severe reduction in the number of tokens from 576 to 192, the performance on MMMU and MMBench (AR) surpasses that of the baseline. Furthermore, in the 128-token setting, the average accuracy decreased by less than 2%, which serves to validate the efficacy of our approach. Notably, SparseLVM demonstrates superior performance in logical reasoning tasks compared to the overall results presented in Table 1. We attribute this enhancement to our method’s capability to eliminate extraneous redundant information, thereby enabling the LLM to concentrate more effectively on relevant visual information, which in turn enhances its logical reasoning abilities. Even in scenarios with limited visual information, such as the 64-token setting, our method still achieves not bad performance by aggregating useful information.

Summary Formula

The complete process of SparseVLM Flash Attention can be summarized as:

Here, each block $B$ is processed independently, and the results are combined using incremental normalization.

[1] Dao, T. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems.

[2] Dao, T. (2023). Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.

(2) Attention mechanism used in the baseline: In our original implementation, we utilized the attention operation implemented by the official PyTorch, including our method and baseline.

(3) Speed experiments w & w/o FlashAttention:

As shown in the table below, we conducted speed experiments on LLaVA and MGM to compare SparseVLM with FlashAttention. Our method owns less latency while maintaining comparable accuracy. Besides, we compared our method with the random sparse matched with FlashAttention, and we observed that our method has significant improvement, when under a similar CUDA time. These findings, from both comparisons, demonstrate that SparseVLM not only achieves high accuracy but also provides an explicit speed advantage.

3. How to deal with RoPE for the sparsified visual tokens?

As is widely known, RoPE (Rotary Position Embedding) requires computation based on the kv_seq_len (key/value sequence length) and the position ID. In the original LLaVA, position IDs are assigned according to the sequence length of the current input to the decoder. In this setup, without token pruning, the position ID for each layer of the decoder remains fixed. Additionally, the key and value for each layer are stored in the KV cache, whose length matches the sequence length. During inference, after LLaVA generates the first token, it uses the KV cache to compute subsequent tokens by combining the current token with the previously stored key and value. Consequently, the position ID for each new token corresponds to the sequence length (with IDs starting from 0).

However, in our SparseVLM setting, tokens are pruned, and the number of tokens retained in each layer of the KV cache can vary, with the length being less than or equal to the original sequence length. If we assign the position ID of a newly generated token based on the original sequence length, an error will occur during RoPE computation. This is because the position ID of the current token will not align with the pruned key/value sequence length retained in the KV cache.

To address this issue, we propose a solution: we utilize LLaVA's KV cache to extract the length of the pruned key/value sequence retained in the current layer during the previous computation. Using this extracted length, we dynamically compute the position ID of the newly generated token. Subsequently, we calculate RoPE using the pruned key/value sequence length and the updated position ID. Whenever irrelevant tokens are removed or new tokens are inserted, our method reconstructs the positional encodings to ensure that the spatial relationships remain unchanged, regardless of the positional encoding method. This approach ensures compatibility between the position ID and the pruned sequence length, maintaining accurate computation and preserving the original positional relationships.

We sincerely thank the reviewer 7sbX for the efforts in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

1. Our SparseVLM performance on multi-turn conversations.

Thanks for the comment. The compatibility with multi-turn dialogues is actually an advantage of our method over existing prompt-agnostic methods. The LLM in our method takes as inputs all the vision tokens and adaptively sparsify them according to the language prompt. When dealing with new questions, our SparseVLM can sparsify the vision tokens differently and is thus compatible with multi-turn dialogues. However, existing prompt-agnostic methods learn fixed visual compression regardless of the texts and cannot handle subsequent questions in multi-turn conversations.

2. SparseVLM compatibility with Flash attention and their comparison.

(1) Clarification of FlashAttn Compatibility: Our SparseVLM can be compatible with FlashAttn. The following is our solution.

SparseVLM Flash Attention:

To ensure that SparseVLM remains compatible with Flash Attention, we devised a method to extract the mean value of the processed attention map without explicitly obtaining the full attention map. In normal decoder layers that do not require pruning, we use the original Flash Attention directly. For layers where pruning is necessary, we implemented an additional Flash Attention-based operation to directly obtain the mean attention scores w.r.t. the text raters, which is lightweight and also enjoys the efficiency in Flash Attention.

Specifically, the first forward pass operates identically to the original Flash Attention, generating the hidden states for all tokens before pruning. In the second forward pass, we introduce a specially designed V matrix. In this matrix, for the rows corresponding to the text raters we wish to analyze, we set the values to $1 / \text {len(text raters)}$. This configuration allows the inner product between the attention map and the V matrix to return the mean value of the attention scores for the selected text raters directly in Flash Attention.

Using this mean value, we perform a top-k selection to identify the vision tokens to retain. Tokens that are excluded during this process are converted into masks, which are then applied to the hidden states produced by the first Flash Attention pass to complete the pruning operation. This method enables efficient integration of pruning with Flash Attention while preserving compatibility and computational efficiency.

Core Principles and Calculation of SparseVLM Flash Attention

1. Attention Score Calculation

For each block $B$, compute the scaled dot-product attention scores:

Here, $S_B$ is the attention score matrix computed within the block.

2. Block-wise Softmax

To ensure numerical stability, the softmax is computed in a stable manner using the log-sum-exp trick:

(1) Subtract the maximum value for numerical stability:

(2) Normalize:

3. Designation of V Matrix

In order to return the mean value of the attention scores for the selected text raters directly in Flash Attention, we need to design a special V matrix.

Here, $V$ is an $n \times d$ matrix, $n$ is the total number of rows in the matrix, $i$ is the row index, $1 \leq i \leq n$, $S = \\{ i \mid R[i] \geq m, \, i \in \\{1, 2, \dots, L_t\\} \\}$ define the text raters which we selected in Section 3.2.

4. Incremental Accumulation

Rather than storing $P$ explicitly, the result is directly accumulated into the output using:

The final result is obtained by concatenating all blocks:

5. Streaming Softmax

When combining multiple blocks, an incremental softmax computation ensures that normalization is maintained across the entire sequence:

This avoids global dependencies and enables efficient block-wise computation.

6. Top-k selection for vision tokens

The top-k selection can be expressed as:

where $O = \text{Concat}(O_1, O_2, \ldots, O_B)$ is the output array of second Flash Attention, $O_v$ is the vision tokens part of $O$, $\text{rank}(x_i, O_v)$ represents the position of $x_i$ in $O_v$ when sorted in descending order.

The corresponding indices of the top-k elements are:

We sincerely thank you for your efforts in reviewing our paper. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether your concerns have been addresses or not. Please let us know if you still have any unclear part of our work.

Best, Authors

Dear reviewer 7sbX,

I hope this message finds you well. We greatly appreciate the valuable feedback and suggestions you have provided so far. As the deadline approaches, we are eager to receive your feedback on our response and revisions. If possible, we kindly request an update on the progress of the review to ensure we can address any further comments or revisions promptly.

Should you require any additional information or assistance from our end to help facilitate the review process, please do not hesitate to let us know. Your insights are highly valuable to us, and we genuinely appreciate your time and effort in reviewing our paper.

Thank you for your patience and cooperation. We are looking forward to hearing from you soon.

Warm regards,

Submission402 Authors.

3. The additional experiments about VLM architecture different from LLaVA.

Thanks for your suggestion. Since the vanilla Flamingo is a prior work that is not as competitive as recent VLMs, we adopted our method on Qwen-VL on AI2D, TextVQA and MME to validate our generalizability across different architectures, as shown in the following tables:

AI2D_TEST Performance Comparison

TextVQA Performance Comparison

MME Performance Comparison

In summary, our approach achieved strong performance on the Qwen-VL architecture. Even after pruning approximately 42% of the tokens on the TextVQA, MME, and AI2D benchmarks, we retained at least 95% accuracy of the original accuracy. Additionally, our method demonstrated significant computational efficiency, reducing CUDA times and TFLOPs by approximately 17.8% to 41.4%.

Here, we utilized the mentioned SparseVLM FlashAttn in Part 1 compared with the baseline matched with FlashAttn. We conducted comprehensive experiments on LLaVA and MGM across three benchmarks: POPE, TextVQA, and MME. The results are shown in the following table. Besides, we compared our method with the random sparse matched with FlashAttn, and we observed that our method has significant improvement, when under a similar CUDA time.

Memory consumption issues

The research in FastV shows that the attention allocation in the shallow layers is more balanced than that in the deeper layers [1]. Consequently, we do not perform sparsification on the first two layers. This leads to the fact that our experiments start sparsification from 576 tokens (taking LLaVA as an example). Therefore, peak memory consumption is similar to that of the approach without using the sparsification method. However, by sparsifying tokens, our method can significantly reduce the length of the KV Cache after the sparsified layers in the subsequent steps, thereby effectively reducing memory occupation and being suitable for running on devices such as mobile phones. Moreover, we calculate the memory occupied by vision tokens based on LLaVA, and the results are shown in the following table:

[1] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." European Conference on Computer Vision. Springer, Cham, 2025.

2. The comparison of state-of-the-art baselines.

Thank you for your valuable suggestion. Firstly, we need to clarify that our method is training-free, requiring no training resources. It only necessitates loading pre-trained weights for inference, which is simple and efficient.

(1) Victor [1] was published on the arXiv platform in October 2024 and is concurrent with our work. In addition, Victor contains learnable parameters and is the same type of work as Voco-llama and llama-vid, while our SparseVLM is training-free, so it is unfair to compare it with Victor.

(2) UPop [2] attains a high compression ratio, by progressively searching and retraining the subnet. However, the method needs to train trainable masks during the architecture search, which is inconsistent with our training-free SparseVLM. Therefore, it is unfair to compare it with UPop.

Despite this, we still tried to find a state-of-the-art and training-free method PDrop [3] for comparison to demonstrate the superiority of our method, summarized as follows.

[1] Wen, Y., Cao, Q., Fu, Q., Mehta, S., & Najibi, M. (2024). Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers. arXiv preprint arXiv:2410.14072.

[2] Shi, D., Tao, C., Jin, Y., Yang, Z., Yuan, C., & Wang, J. (2023, July). Upop: Unified and progressive pruning for compressing vision-language transformers. In International Conference on Machine Learning (pp. 31292-31311). PMLR.

[3] Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y., ... & Lin, D. (2024). Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247.

We sincerely thank the reviewer YHTW for the efforts in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

1. SparseVLM compatibility with Flash attention and memory consumption issues.

Thank you for your kind suggestion! To eliminate your concern about SparseVLM are not compatible with FlashAttention, we proposed an algorithm in detail below:

SparseVLM Flash Attention:

In normal decoder layers that do not require pruning, we use the original Flash Attention directly. For layers where pruning is necessary, we implemented an additional Flash Attention-based operation to directly obtain the mean attention scores w.r.t. the text raters, which is lightweight and also enjoys the efficiency in Flash Attention.

Specifically, the first forward pass operates identically to the original Flash Attention, generating the hidden states for all tokens before pruning. In the second forward pass, we introduce a specially designed V matrix. In this matrix, for the rows corresponding to the text raters we wish to analyze, we set the values to $1 / \text {len(text raters)}$. This configuration allows the inner product between the attention map and the V matrix to return the mean value of the attention scores for the selected text raters directly in Flash Attention.

Using this mean value, we perform a top-k selection to identify the vision tokens to retain. Tokens that are excluded during this process are converted into masks, which are then applied to the hidden states produced by the first Flash Attention pass to complete the pruning operation. This method enables efficient integration of pruning with Flash Attention while preserving compatibility and computational efficiency.

Core Principles and Calculation of SparseVLM Flash Attention

1. Attention Score Calculation

For each block $B$, compute the scaled dot-product attention scores:

Here, $S_B$ is the attention score matrix computed within the block.

2. Block-wise Softmax

To ensure numerical stability, the softmax is computed in a stable manner using the log-sum-exp trick:

(1) Subtract the maximum value for numerical stability:

(2) Normalize:

3. Designation of V Matrix

In order to return the mean value of the attention scores for the selected text raters directly in Flash Attention, we need to design a special V matrix.

Here, $V$ is an $n \times d$ matrix, $n$ is the total number of rows in the matrix, $i$ is the row index, $1 \leq i \leq n$, $S = \\{ i \mid R[i] \geq m, \, i \in \\{1, 2, \dots, L_t\\} \\}$ define the text raters which we selected in Section 3.2.

4. Incremental Accumulation

Rather than storing $P$ explicitly, the result is directly accumulated into the output using:

The final result is obtained by concatenating all blocks:

5. Streaming Softmax

When combining multiple blocks, an incremental softmax computation ensures that normalization is maintained across the entire sequence:

This avoids global dependencies and enables efficient block-wise computation.

6. Top-k selection for vision tokens

The top-k selection can be expressed as:

where $O = \text{Concat}(O_1, O_2, \ldots, O_B)$ is the output array of second Flash Attention, $O_v$ is the vision tokens part of $O$, $\text{rank}(x_i, O_v)$ represents the position of $x_i$ in $O_v$ when sorted in descending order.

The corresponding indices of the top-k elements are:

Summary Formula

The complete process of SparseVLM Flash Attention can be summarized as:

Here, each block $B$ is processed independently, and the results are combined using incremental normalization.

[1] Dao, T. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems.

[2] Dao, T. (2023). Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.

We sincerely thank you for your efforts in reviewing our paper. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether your concerns have been addresses or not. Please let us know if you still have any unclear part of our work.

Best, Authors

Dear reviewer YHTW,

I hope this message finds you well. We greatly appreciate the valuable feedback and suggestions you have provided so far. As the deadline approaches, we are eager to receive your feedback on our response and revisions. If possible, we kindly request an update on the progress of the review to ensure we can address any further comments or revisions promptly.

Should you require any additional information or assistance from our end to help facilitate the review process, please do not hesitate to let us know. Your insights are highly valuable to us, and we genuinely appreciate your time and effort in reviewing our paper.

Thank you for your patience and cooperation. We are looking forward to hearing from you soon.

Warm regards,

Submission402 Authors.

2. More explanations and discussions about the performance of SparseVLM.

Thank you for your insightful comments. Firstly, we want to clarify that there is an efficiency-performance trade-off in token sparsification, and under the same efficiency, our method already obtains state-of-the-art performance compared to other methods. Moreover, the "retain 192 tokens" is too aggressive to have a minor impact on performance; on the other hand, one can obtain marginal performance drops by setting moderate sparsification ratios. As demonstrated by the additional experiments in the table below, retaining mid-range 448, 384, and 320 tokens can preserve the performance while still having obvious efficiency gains. For instance, with 22% of the vision tokens deleted (448 tokens in the table), the average performance drop is only 0.5%.

Note that this trade-off is adaptively controllable by our proposed sparsification level adaptation, which means that one can easily set the sparsification ratio to achieve the desired efficiency and performance.

3. The comparison of our SparseVLM with baseline matched with FlashAttn.

Thank you for your detailed suggestions. Here, we utilized the mentioned SparseVLM FlashAttn in Part 1 compared with the baseline matched with FlashAttn. We conducted comprehensive experiments on LLaVA and MGM across three benchmarks: POPE, TextVQA, and MME. The results are shown in the following table. Besides, we compared our method with the random sparse matched with FlashAttn, and we observed that our method has significant improvement, when under a similar CUDA time.

4. The report of Qwen2-VL with SparseVLM on high-resolution image understanding benchmarks.

Thank you for your valuable suggestion. We applied our SparseVLM to Qwen2-VL and conducted comprehensive experiments on InfoVQA, and AI2D. The results are shown in the following table. We found that our method still shows superiority on high-resolution image understanding benchmarks. For instance, on the AI2D, when the sparsification ratio increases from 43.74% (768 tokens) to 81.25% (256 tokens), SparseLLaVA only drops the accuracy by 5% without any additional training, while reducing the CUDA Times and TFLOPs by 42.75% and 75.09% respectively. This demonstrates that high-resolution image understanding benchmarks like AI2D also contain a significant amount of token redundancy. Due to the limitations of computing power and time, we did not conduct experiments on DocVQA for the time being (takes two to three hours), and this will be accomplished in our revised version.

AI2D_TEST Performance Comparison

InfoVQA_Test Performance Comparison

5. The explanation of some hyperparameters.

Thank you for your valuable comments, and we explain them in detail.

(1) The m in equation (6) is the mean of the $R$ matrix, a dynamical number determined by $R$ instead of a specific and constant one.

(2) $\lambda$ in equation (8) is a scaling factor to determine the number of deletions, and it is up to your needs. For instance, if you want to sparsify the number of vision tokens to 192 on MME, the $\lambda$ should be set to 13.5; if you want to sparsify the number of vision tokens to 64 on TextVQA, the $\lambda$ should be set to 0.8.

(3) The $\tau$ in equation (9) is a recycling ratio to decide the number of reconstruction candidates, which is a constant number. In all our experiments, we set it to 30%. Although it is not well-designed, our method is still effective and reserves more information with fewer slots.

6. How does SparseLVM insert these recycled tokens into the preserved tokens?

Firstly, whenever irrelevant tokens are removed or new tokens are inserted, our method reconstructs the positional encodings to ensure that the spatial relationships remain unchanged. Specifically, the recycled tokens are added at the end to preserve the original positional relationships. Furthermore, previous methods like [1][2] also adopt a similar core idea to reconstruct positional encodings for the insertion of new tokens, while still keeping the performance of their methods. Finally, these recycled tokens can be seen as another form of system or [cls] tokens, which contain the semantic information of multiple tokens.

[1] Shang, Yuzhang, et al. "Llava-prumerge: Adaptive token reduction for efficient large multimodal models." arXiv preprint arXiv:2403.15388 (2024).

[2] Zeng, Wang, et al. "Not all tokens are equal: Human-centric visual analysis via token clustering transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

We sincerely thank the reviewer hiZv for the efforts in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

1. Clarification of FlashAttn Compatibility

Our SparseVLM can be compatible with FlashAttn. The following is our solution:

SparseVLM Flash Attention:

To ensure that SparseVLM remains compatible with Flash Attention, we devised a method to extract the mean value of the processed attention map without explicitly obtaining the full attention map. In normal decoder layers that do not require pruning, we use the original Flash Attention directly. For layers where pruning is necessary, we implemented an additional Flash Attention-based operation to directly obtain the mean attention scores w.r.t. the text raters, which is lightweight and also enjoys the efficiency in Flash Attention.

Specifically, the first forward pass operates identically to the original Flash Attention, generating the hidden states for all tokens before pruning. In the second forward pass, we introduce a specially designed V matrix. In this matrix, for the rows corresponding to the text raters we wish to analyze, we set the values to $1 / \text {len(text raters)}$. This configuration allows the inner product between the attention map and the V matrix to return the mean value of the attention scores for the selected text raters directly in Flash Attention.

Using this mean value, we perform a top-k selection to identify the vision tokens to retain. Tokens that are excluded during this process are converted into masks, which are then applied to the hidden states produced by the first Flash Attention pass to complete the pruning operation. This method enables efficient integration of pruning with Flash Attention while preserving compatibility and computational efficiency.

Core Principles and Calculation of SparseVLM Flash Attention

1. Attention Score Calculation

For each block $B$, compute the scaled dot-product attention scores:

Here, $S_B$ is the attention score matrix computed within the block.

2. Block-wise Softmax

To ensure numerical stability, the softmax is computed in a stable manner using the log-sum-exp trick:

(1) Subtract the maximum value for numerical stability:

(2) Normalize:

3. Designation of V Matrix

In order to return the mean value of the attention scores for the selected text raters directly in Flash Attention, we need to design a special V matrix.

Here, $V$ is an $n \times d$ matrix, $n$ is the total number of rows in the matrix, $i$ is the row index, $1 \leq i \leq n$, $S = \\{ i \mid R[i] \geq m, \, i \in \\{1, 2, \dots, L_t\\} \\}$ define the text raters which we selected in Section 3.2.

4. Incremental Accumulation

Rather than storing $P$ explicitly, the result is directly accumulated into the output using:

The final result is obtained by concatenating all blocks:

5. Streaming Softmax

When combining multiple blocks, an incremental softmax computation ensures that normalization is maintained across the entire sequence:

This avoids global dependencies and enables efficient block-wise computation.

6. Top-k selection for vision tokens

The top-k selection can be expressed as:

where $O = \text{Concat}(O_1, O_2, \ldots, O_B)$ is the output array of second Flash Attention, $O_v$ is the vision tokens part of $O$, $\text{rank}(x_i, O_v)$ represents the position of $x_i$ in $O_v$ when sorted in descending order.

The corresponding indices of the top-k elements are:

Summary Formula

The complete process of SparseVLM Flash Attention can be summarized as:

Here, each block $B$ is processed independently, and the results are combined using incremental normalization.

[1] Dao, T. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems.

[2] Dao, T. (2023). Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.

We sincerely thank you for your efforts in reviewing our paper. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether your concerns have been addresses or not. Please let us know if you still have any unclear part of our work.

Best, Authors

Dear reviewer hiZv,

I hope this message finds you well. We greatly appreciate the valuable feedback and suggestions you have provided so far. As the deadline approaches, we are eager to receive your feedback on our response and revisions. If possible, we kindly request an update on the progress of the review to ensure we can address any further comments or revisions promptly.

Should you require any additional information or assistance from our end to help facilitate the review process, please do not hesitate to let us know. Your insights are highly valuable to us, and we genuinely appreciate your time and effort in reviewing our paper.

Thank you for your patience and cooperation. We are looking forward to hearing from you soon.

Warm regards,

Submission402 Authors.

3. The explanation for the calculation of $P$.

We appreciate your valuable comments regarding the definition of $P$. (1) $P$ is a subset of the attention matrix selected by vision and text indexes, derived after applying softmax. The reference to 'logits' in line 183 is indeed inappropriate, as logits refer to unnormalized values. We have fixed it in the revised version. (2) Once $P$ is obtained, we do not perform normalization again, as it has already undergone softmax.

4. The explanation for the eq. (7).

Thank you for bringing this to our attention and pointing out the error in equation (7). We have revised the notation as follows:

$R = \\frac{1}{L_v} \\sum_{j=1}^{L_v} (\text{Softmax} (H_v {H_q}^T))[j,:],$

Where is $H_vH_q^{T} \in \mathbb{R}^{L_v \times L_q}$. We first take the mean over the vision dimension, leaving only the text dimension, where it is $R \in \mathbb{R}^{L_q}$. Then, we take the mean over the text dimension to generate the threshold of raters $m$.

5. The explanation for text token select threshold.

Thank you for your insightful comments and advice. In our method, we do not adopt the complex design for the threshold, and only utilize an average tool to generate it, which is simple but effective. Furthermore, based on your suggestion, we conducted additional ablation experiments on LLaVA to compare the accuracy differences between using the mean and the topK ($k$=8) on various benchmarks, and the results are as follows:

Although topK selection has a minimal improvement on MME, we argue that adaptive selection by the mean is general and robust to various prompts compared with specific topK selection.

6. The discussion of the number of vision tokens to prune is based on the rank of $P$.

Thank you for the insightful question and suggestion!

(1) Clarification of extreme cases: The situation mentioned above hardly occurs during the actual inference process, because the values in the attention matrix after applying softmax will reduce the difference between larger and smaller values, resulting in a smoother distribution of the matrix.

(2) Reproduction of your suggestion: First of all, thank you for your valuable suggestion, and we applied it to SparseVLM for experiments on TextVQA, MME, and POPE. As shown in the table below, when the number of singular values with higher contributions identified after Singular Value Decomposition (SVD) matches the number of vision tokens retained after the Rank(P) operation, the accuracy of both methods remains largely similar. However, we note that the CUDA execution time for our method is slightly faster than that of SVD.

(3) Optimization for extreme cases: To summarize, the differences in accuracy between the two methods are minimal, suggesting that the numerical accuracy issue you mentioned rarely arises during actual reasoning. Besides, to completely avoid the situation you mentioned, we will set values smaller than $10^{-5}$ to zero in advance. Subsequently, the number of non-zero singular values will be used as the rank for selecting vision tokens.

7. Add the memory used by the baseline in Table 4.

Thank you for your valuable suggestion regarding the memory usage for the baseline in Table 4, whose value is 302.4. We have updated it in the revised version.

We sincerely thank the reviewer RuY4 for the efforts in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

1. The additional efficiency experiments and explanation about latency and FLOPs on image understanding tasks.

Thanks for the suggestion. We have complemented more experiments and revised the paper accordingly.

(1) Baseline Storage Memory: The revised version of the manuscript updates the baseline value to 302.4.

(2) More Trade-off Figures: Thank you for your detailed suggestions. We have included latency-vs.-accuracy and FLOPs-vs.-Accuracy tradeoffs for SparseVLM applied to LLaVA and MGM across three benchmarks: POPE, TextVQA, and MME. Figures 9 and 10 in Appendix A.7 now include a total of 12 sub-figures illustrating these tradeoffs on the image understanding tasks.

(3) Efficiency Summary: The following table presents the latency-vs.-average accuracy and FLOPs-vs.-Average accuracy results for individual datasets, providing a comprehensive overview of our method's efficiency.

latency-vs.-average accuracy

TFLOPs-vs.-average accuracy

In summary, the above experiments fully demonstrate the effectiveness of our method in reducing latency and computational complexity.

2. The additional efficiency experiments and explanation about latency and FLOPs on video understanding tasks.

Thank you for your valuable feedback regarding video understanding tasks.

(1) Clarification of Token Budget: The 135 is the equivalent number of tokens after our pruning and recycling. Specifically, the original 2056 tokens are pruned in the second layer to a single token. For the equivalent calculation: the first two layers without pruning, have 2 × 2056 tokens, while the subsequent 30 layers, after pruning, contribute 30 × 1 token. With our SparseVLM performing token recycling, the average number of tokens increases slightly to 135. To ensure a fair comparison, we align the token count in FastV with this average.

(2) More Trade-off Figures: In our revised manuscript, we have included additional figure (Figure 11 in Appendix A.7) that demonstrate both the speed-versus-accuracy and token budget-versus-accuracy trade-offs across multiple token budgets on TGIF and MSVD datasets, as summarized in the table below. This will provide a clearer understanding of our method's performance and its implications.

Video-LLaVA Trade-offs

Dear Reviewer RuY4,

We sincerely thank you for your efforts in reviewing our paper. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether your concerns have been addresses or not. Please let us know if you still have any unclear part of our work.

Best, Authors

We greatly value your feedback!

To sparstify the numerous vision tokens in video understanding tasks, we inquire about the necessity of sparse operations for each layer. This leads to multiple rank operations being executed, consequently increasing the time consumption. The manner differs from the strategy FastV employs, where sparse operations are only conducted in a single layer (Layer 1). To address this, aligned with the FastV, we have optimized the sparsification strategy in video understanding tasks and conducted sparsification in only three layers (Layers 1, 4, 15), and performed swift validation experiments by randomly sampling subsets from TGIF and MSVD datasets. The results are listed as follows and visualization has been updated in the Appenidx. We observed that our method has significant improvement over FastV under comparable CUDA time constraints. For instance, our approach achieves even higher accuracy than FastV in the 1092-token setting and also demonstrates a significant speed advantage, especially in the MSVD dataset using the 700-token setting.