MMInference: Accelerating Pre-filling for Long-Context Visual Language Models via Modality-Aware Permutation Sparse Attention

We greatly appreciate the reviewer's recognition and thoughtful, constructive feedback. Below, we address each of the comments and concerns in detail.

1. "How stable are the identified patterns across different models and datasets"

Thank you for the suggestion. We have tested these patterns across various VLMs and a wide range of datasets including open-domain question answering, multiple-choice question answering, video captioning, and video information retrieval. Additionally, we provide further pattern analysis on two recent VLMs, Qwen2.5-VL [1] and VideoChat-Flash [2], to demonstrate that the attention patterns leveraged in our method are consistently observed across different models and tasks, see https://anonymous.4open.science/r/complementary-D5B2/. While we also observe other patterns, such as local window attention, these are in fact special cases of our existing patterns and can be effectively captured by our method.

2. "Typo issue"

Thank you for your careful review. We will fix this issue in the next version.

[1] Qwen2.5-VL Technical Report, 2024.
[2] VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling, 2024.

We greatly appreciate the reviewer’s thoughtful and constructive feedback. We respond to each of the comments and concerns below.

1. "it's better to report average latency and memory consumption in Table 1"

Thank you for the suggestion. We have updated Table 1 to include a dedicated column for end-to-end latency. Regarding memory usage, since all baselines consume similar memory except for VisionZip, we did not include a separate memory column. Below, we provide a table comparing the latency and average performance of MAPSparse with other baselines. The number of frames is fixed at 256, and all tests are conducted on a single A100 GPU. As shown in Section 4.5, our method achieves greater speedup with longer video inputs and more frames.

RTable 1. Performance (%) of different methods on video understanding tasks evaluated.

2. "longer frames"

Thanks for the suggestion. Here we show the results of MAPSparse and the full attention baselines with various numbers of frames from 32 to 256. The results show that more frames can consistently improve the performance on video understanding tasks. MAPSparse improves system throughput by supporting more frames and higher QPS video inputs within the same latency constraints.

RTable 2. Performance (%) of different frame on video understanding tasks evaluated using LongVILA.

3. "meaning and how the recall rates are calculated"

The recall rate reflects how much of the original attention score is retained in the top-k sparse attention. It is computed by calculating the softmax scores over the top-k key vectors retrieved for each query vector, i.e., $\text{softmax}(qK_{\text{topk}}^T)$. We will update the paper to include a formal definition of the recall rate in the next revision.

We greatly appreciate the reviewer’s thoughtful and constructive feedback. We respond to each of the comments and concerns below.

1. "Generalization of the patterns"

We evaluate our method across a broad range of multimodal tasks and VLMs, including open-domain question answering, multiple-choice question answering, video captioning, and video information retrieval. The models tested include Llava-Video and LongVILA, and our method consistently demonstrates strong performance across all these scenarios.

We additionally provide pattern analysis (shown in https://anonymous.4open.science/r/complementary-D5B2/) and experimental results (RTable.1) on the latest VLMs, Qwen2.5-VL [1] and VideoChat-Flash [2], further demonstrating the generalizability of both the observed patterns and our method across different models and tasks.

2. "Patterns in ViT"

Optimizing tokens in ViT can help reduce prefilling latency, but ViT is not the primary bottleneck in long video processing for VLMs. As shown in Figure 1(a), ViT accounts for less than 2% of the total latency, while attention operations in VLMs dominate with over 95%. Nevertheless, MAPSparse is orthogonal to ViT optimization techniques such as VisionZip (shown in Table 1) and VideoChat-Flash [2]. We include an additional experiment based on VideoChat-Flash, and the results indicate that MAPSparse integrates well with token compression methods.

RTable 1. Performance of different methods on video understanding tasks using VideoChat-Flash.

3. "Generalization to other modalities"

Thank you for the suggestion. We conducted additional experiments on the text-only long-context benchmark SCBench [3], comparing MAPSparse with full attention, as shown in RTable 2. The results demonstrate that MAPSparse generalizes well across modalities and tasks. We will include the corresponding results in the next version.

RTable 2. Performance of different methods on long-context benchmark SCBench using Llama-3.1-8B in 128K.

4. "Impact Statement"

We apologize for the misunderstanding. We followed the ICML [4] guidelines when writing the impact statement. Since our method does not alter the original VLMs outputs, it does not introduce any additional risks.

[1] Qwen2.5-VL Technical Report, 2024.
[2] VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling, 2024.
[3] SCBench: A KV Cache-Centric Analysis of Long-Context Methods, ICLR 2025.
[4] https://icml.cc/Conferences/2025/CallForPapers