On Exploring Visual Attention Shrinking for Accelerating VLMs for Video Understanding

Thank you for taking the time to review our paper and providing insightful suggestions. We address your concerns point by point below.

W1: Lack of sufficiently significant performance difference compared to other methods.

It is worth noting that previous methods often test on a single model or models with similar strong language model backbones, which may not be applicable in all scenarios. However, our approach aims to generalize across different models and succeed. Additionally, since it is an untuned method, the performance improvements are limited. In the case of LongVA, the LongVA model primarily focuses on enhancing long-context capabilities through textual data during training and does not train on long video data, which may explain its poorer performance when the number of visual tokens is reduced significantly. This could also affect the ability of certain layers to extract key information through attention mechanisms. In contrast, ToMe does not utilize attention mechanisms but rather employs cosine similarity. In its application, we adopted a relatively conservative approach, retaining over 2000 visual tokens even in the final layer. Furthermore, ToMe evidently performs poorly on other models. Nevertheless, for fairness, we have included the corresponding results.

We also conduct additional experiments with LongVA-7B, and apply a more conservative removal strategy on MLVU task, we achieve a 1.59X speedup using 63.3% of the visual tokens, with an accuracy of 56.7% (which is higher than ToMe's 55.8%, which using 65.5% of the visual tokens to achieve a 1.61X speedup). Just adjusting on this task, we can achieve an better average performance compared to ToMe. (51.6 vs 51.4)

W2: Explanations for instances of "Failed" in Table 1.

Related to the points mentioned in W1, previous methods often did not test on VLMs with different language model backbones, leading to failures in some scenarios. The most common examples of failure outputs are meaningless repetition or an immediate end of the response. The failure of FastV on LongVA is also related to the training strategy discussed in W1. LongVA primarily trains on textual data to enhance long-context capabilities. When half of the visual tokens are removed in early layers, the data diverges significantly from what was used during training, adversely affecting performance.

W3: Comparisons of prior work.

Thanks for your advice and we will supplement the relevant section later. It is worth to mention that FastV does not employ a gradual token reduction strategy. It removes R% of visual tokens at once at one selected layer. In contrast, ToMe gradually merges tokens using cosine similarity. Additionally, in practical applications, we only leverage the attention of textual information to visual information, while FastV utilizes global attention. When applying scaled dot-product attention or flash attention commonly used in video understanding models, global attention cannot be directed obtained, which is a significant drawback. In contrast to other methods, our method emphasizes retrieving and refining information from one modality based on another, and is applied specifically in video scenarios.

W4: Suggestions regarding related work.

We greatly appreciate your suggestions regarding related work, as well as pointing out inaccuracies in our descriptions. These works focus more on proposing a new efficient LMM, rather than being a tuning-free and plug-and-play approach, which is different from ours. They achieve more efficient LMMs by performing token merging through lightweight operations before the visual information enters the large language model. We have similar ultimate goals, and thanks for your advice and we think these works to be inspiring and meaningful. We will cite your recommendations and make revisions in the relevant sections.

W5: Analysis of long video sequences.

In most current video understanding models, the final length of the video sequences is related to the number of sampled frames set by the model than to the actual length of the video itself. In our experiments, we observed that with a high number of sampled frames, the first and last frames tend to receive more attention. This may cause the model to focus more on the beginning and end of a long video, resulting in less accurate understanding of the content in the middle section. This may also explain why long video tasks are more challenging. In our work, we follow the model's default settings, where different models have different numbers of sampled frames and visual tokens. Additionally, we have conducted supplementary tests on the MovieChat-1K and LongVideoBench datasets as you suggested, and the results are as follows:

Thanks for your constructive comments and positive feedback. Your deep understanding of the relevant field has highlighted several points we overlooked. We will need some time to revise and supplement the relevant sections of the paper based on your suggestions. Once again, thank you for taking the time to read our paper thoroughly and provide valuable suggestions.

Thank you for the time to review our work. We are glad you think our research meaningful and comprehensive. We address your concerns point by point below.

W1: There is a higher drop in performance in some settings.

It is worth noting that previous methods used in VLMs often test on one model or models with similar strong LLM backbones, which may not be applicable in all scenarios. Our method selects three video VLMs PLLaVA, VILA, LongVA, whose LLM backbone and model design are significant different. The good performance on the three VLMs demonstrates the generalization of our methods. However, other methods fail to work on at least one model.

Moreover, we discuss the performance in certain settings. LongVA focuses on enhancing the model's long-context capabilities through textual data during training and does not train on long video data, which may explain its poorer performance when the number of visual tokens is reduced greatly. Our method is only slightly worse than the ToMe method, which is because our method has a higher removal ratio. We conduct additional experiments on MLVU dataset and apply a more conservative removal strategy, we achieve a 1.59X speedup using 63.3% of the visual tokens, with an accuracy of 56.7% (which is close to the base's 56.9% and higher than ToMe's 55.8%, which using 65.5% of the visual tokens to achieve a 1.61X speedup). Only with this, our method delivers the best overall performance.

For VILA1.5-40B, our intention was to validate that our method can achieve greater acceleration on models with more layers, hence we adopted a more aggressive token reduction strategy to achieve a higher speed-up ratio. We can also make a trade-off, balancing between speedup and accuracy. We conduct additional experiments, applying a more conservative strategy on VILA1.5-40B and comparing it with the FastV method. The results are as follows. As seen, our method outperforms FastV and offers greater flexibility.

W2: Design specific to video.

From a broader perspective, our approach utilizes the model's inherent capabilities and attention mechanisms to retrieve and refine information from one modality based on another, allowing us to extract more important content. However, this does not mean that there is no design specific to video. For example, considering both temporal and spatial dimensions. To process attention more efficiently and perform token reduction, we aggregate attention scores across both spatial and temporal dimensions. Additionally, we compare the relative importance between frames, which can not be considered in VLMs with one image input.

Q1: Confusion regarding Algorithm 1.

Your understanding is correct, the shape of T is m. Apologize for any confusion caused by mistakes in the pseudocode. We will review the paper again and make the necessary corrections and clarifications to avoid ambiguity. Thank you for your careful reading!

Q2: About multi-turn conversations.

There seems to be limited examination of the multi-turn conversational capabilities of video understanding models. If you are referring to single video input, as mentioned in Section 3.3, Line 302, our method is compatible with the KV cache, allowing it to be directly used in multi-turn conversations. If you are considering multiple video inputs, this appears to be infrequently explored before. In this case, we might need to maintain a cache to record some indexes of each video along with the corresponding text tokens. Alternatively, concatenating multiple videos into one might facilitate processing.

Thank you again for your feedback. We hope our responses help clarify some of your concerns.

We understand your concerns. To be precise, in multi-turn dialogues, when we choose to use the KV cache for subsequent rounds, the expected acceleration benefit will not be affected. In theory, to achieve the best performance, we should redo VAS at each Q&A round and select the important and relevant visual tokens, but doing so would impact the acceleration benefit. As mentioned in Section 4.2.1, for some descriptive tasks, text queries are generally short and generic, which can impact performance, but it doesn't mean it's unworkable. We can still retain relatively important visual tokens. Given that our goal is to achieve acceleration rather than pursuing higher performance. We are more inclined to use the KV cache in multi-turn dialogues.

When we apply VAS solely in the spatial or temporal dimension, the results are 46.1 and 45.6 respectively. The accuracy is higher than that of the toy method, and the speedup ratios are slightly lower than the original VAS. In general, there is higher redundancy in the spatial dimension. It is worth noting that although the performance of 'Spatial average' is only slightly lower, its speedup ratio is only 1.22x, which is far inferior to the 1.96x achieved by VAS.

We hope our reclarifications and explanations can resolve your concerns. If our response has addressed your concerns, we hope for your reconsideration in raising your score.

If you have any additional questions or suggestions, we would be happy to have further discussions.

Dear reviewer,

Thank you for your time in reviewing our work. We understand you may be quite busy rencently, but the revised deadline for the discussion is approaching. We kindly inquire whether our responses have addressed most of your concerns and hope you might reconsider adjusting your score. If there are any remaining issues, we would be happy to address them promptly.

Best regards,

The Authors

Thanks for your positive feedback and valuable suggestions! Below, we address the weaknesses and questions you proposed.

W: Suggestions and Explanation for Performance on Some Datasets

Thank you for your suggestion. We provide the visualization of text-to-visual attention maps w/ or w/o reduction for comparison in the Appendix. Since the total number of tokens can reach thousands with a video input, we aggregate the attention for visual tokens from the same frame during visualization. The gray areas represent the removed parts, and it can be seen that when we remove the potentially unimportant visual tokens, the attention distribution of the remaining tokens is largely unaffected.

Regarding the poor performance on some tasks, we first discuss the Egoschema task. we think what you mean is that the performance on the PLLaVA-7B model drops significantly compared to the base model, but this does not affect our method is the best, as we still achieve an accuracy of 34.9%, while other methods do not exceed 30%. The underlying reason for this may be when the number of visual tokens is reduced, PLLaVA-7B tends to output fixed option under the default prompts. We tried to mitigate this issue, but it still hasn't been addressed completely. This may be due to the inherent bias the model has retained during training for specific inputs.

For the MLVU task, it is inherently a challenging long-video dataset, and the main issue lies with the performance on the LongVA model. Our method is only slightly worse than the ToMe method, which is due to our adoption of a higher removal ratio. We conduct additional experiments, and when a more conservative removal strategy is applied, we achieve a 1.59X speedup using 63.3% of the visual tokens, with an accuracy of 56.7% (which is higher than ToMe's 55.8%, which using 65.5% of the visual tokens to achieve a 1.61X speedup). Furthermore, understanding long videos remains difficult for models, and LongVA focuses on enhancing the model's long-context capabilities through textual data during training and does not train on long video data, meaning that some layers' attention mechanisms cannot extract key information effectively. The FastV method, which also uses attention mechanisms, performs poorly and is even almost not applicable to it.

Q: Some Application Details and Extensibility

The relevant details are described in Section 3.3. In brief, we begin by applying token reduction to the current layer when we detect that a proportion of visual token attention has dropped in both the previous two layers. However, this may lead to infinite waiting, so we also record and apply token reduction if no token reduction has been made several layers.

Essentially, our method functions like a retrieval process, where one modality's information is used to retrieve key content from another modality. The motivation here is to leverage the model's inherent ability and use the attention mechanism of textual informations to extract key visual information. Generally, models are most capable when handling textual information. Additionally, if some models have a unified representation across different modalities, this strategy can be extended to other modalities as well. How to process other modalities specifically can be explored in future work.

Once again, thank you for your constructive feedback. It has provided us with valuable insights, and hope you are satisfied with our response.

Dear reviewer,

Thank you for your time in reviewing our work. We understand you may be quite busy rencently, but the revised deadline for the discussion is approaching. We kindly inquire whether our responses have addressed most of your concerns and look forward to your feedback. If there are any remaining issues, we would be happy to address them promptly.

Best regards,

The Authors

We sincerely thank you for taking the time to review our paper. We are glad you think our work can have a broad adoption. We address your concerns point by point below.

W1: Lack of Sufficient Experimental Support.

Thanks for your advice. The datasets we select in fact already include specialized long video datasets such as MLVU [1], in which the longest video lasts up to two hours. Additionally, we also use datasets like VideoMME [2], which include short, medium, and long videos spanning 6 primary visual domains with 30 subfields to ensure broad scenario generalizability. As shown in Table 1 of the main paper, our method outperforms other methods on these datasets. To further validate the effectiveness of our method, we conducted additional experiments on the LongVideoBench [3] and MovieChat-1K [4] datasets, and our method still works well. The results are as follows:

W2: Limited Novelty.

It is true that using attention scores for further analysis in models is a common practice. However, previous works like StreamingLLM [5] usually did not consider targeted design for inputs with both visual and textual information. Also, as we mention in the Introduction and Section 3.2, the important tokens should vary depending on the text query. Unlike the work of PruMerge [6] et al., we focus primarily on the attention from the text modality to the visual modality, as the visual information that needs to be emphasized should vary depending on the text query.

Moreover, our method involves multiple dynamic pruning steps different from FastV [7]. The underlying principle is to leverage the model's inherent reasoning ability during the forward process to extract key information. We specifically consider both the temporal and spatial dimensions for video data to achieve efficient removal, which was not considered in previous methods for image VLMs. These different designs can potentially be transferred to other modalities in future works.

Additionally, we truly appreciate your careful reading of the paper. We have corrected relevant typos and will later conduct a thorough review of the entire paper to revise possible remaining issues. Thank you again for your helpful suggestions and assistance.

[1] Zhou J, Shu Y, Zhao B, et al. MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding. arXiv preprint arXiv:2406.04264, 2024.

[2] Fu, Chaoyou, et al. "Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis." arXiv preprint arXiv:2405.21075 (2024).

[3] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." arXiv preprint arXiv:2407.15754 (2024).

[4] Song, Enxin, et al. "Moviechat: From dense token to sparse memory for long video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Xiao, Guangxuan, et al. "Efficient streaming language models with attention sinks." arXiv preprint arXiv:2309.17453 (2023).

[6] Shang, Yuzhang, et al. "Llava-prumerge: Adaptive token reduction for efficient large multimodal models." arXiv preprint arXiv:2403.15388 (2024).

[7] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." European Conference on Computer Vision. Springer, Cham, 2025.