LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding

(7) The authors claim, "we pioneer to leverage both DINOv2 and SigLIP in video understanding tasks." In fact, several similar works already exist.

This comment may misrepresent our original intent. In this sentence (Line 207), we want to emphasize that we utilize both DINO and SigLIP for MLLM in in video understanding tasks. To the best of our knowledge, Eyes Wide Shut [1] and Cambrian [2] focus only on image modality and haven't explored the benefits of DINO for video understanding tasks in terms of temporal aspects. We will adjust the wording in our revision to avoid any confusion.

[1] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

[2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

W2: Discussion on inference time

Thanks for raising this concern. Please kindly refer to general response for the inference time discussion.

W3: The proposed approach is transferable to existing MLLMs in a training-free manner

We applied our compression method to LLaVA-OneVision in a training-free manner and observed performance improvements on EgoSchema, MVBench, and VideoMME, but a performance drop on MLVU. We hypothesize that this is due to the significant reduction in tokens for long video inputs, which differs from the original training distribution of LLaVA-OneVision. While our compression method demonstrates the potential to enhance other MLLMs, further fine-tuning could help bridge the training-inference distribution gap and achieve better results.

We sincerely thank you for your detailed review. We would like to provide clarifications to all the points you raised by responding to each of your questions below.

W1: Detail explanation

(1) Line 52: The statement "the resampling module causes important visual information loss" needs further explanation. The resampling module is widely used in MLLMs, such as in the latest Qwen2-VL and MiniCPMV-2.6.

Thanks for sharing those newly released works. We will include this in our related work discussion. However, both of the mentioned papers did not use a resampling module for video understanding. MiniCPM-V [1] (technical report for MiniCPM-V 2.5) conducts image partition to divide image into multiple slices, each slice will be encoded into 1024 tokens. They use 64 learnable queries to pass through a one-layer cross-attention module to compress the tokens for each slice. Therefore, they propose a learnable query to compress the image, and there is no video resampling module. The reviewer mentioned MiniCPM-V 2.6 (a MLLM supports video input) did not provide any technical report, and by checking the repo, we assume it follows the same model architecture as MiniCPM-V 2.5, which only has an image resampler.

Qwen2-VL [2] modifies the ViT for dynamic resolution input, and use a simple MLP layer to compress adjacent 2 × 2 tokens into a single token, which does not contain any resampling module. In addition, Qwen2-VL is also token consuming, for example, it uses 2208 tokens to represent a 16-second video resolution 336x644.

This comment might distort our original meaning. In line 52 our original expression is that some approaches "employ intensive resampling modules leading to a considerable loss of essential visual information".

[1] MiniCPM-V: A GPT-4V Level MLLM on Your Phone

[2] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

(2) Why not L_max in line 242?

Thanks for pointing this out. There is a typo in line 242, where $L_q$ should be $L_{max}$. We have fixed this notation in our revised PDF.

(3) Why is J set to 8?

$J$ represents the size of the sliding window, a minor hyperparameter used to divide the video into chunks. Below, we present ablation results demonstrating the effects of different values for $J$. Adjusting the sliding window size has minimal impact on the benchmarks.

(4)(5) Hyperparameters chosen for frame reduction

We empirically observe DINO similarities with several videos and set the DINO threshold as 0.83 for all the benchmarks. We also ablate the DINO threshold. The lower the DINO threshold, the fewer frames are reduced. For instance, in MLVU, as the threshold decreases, performance deteriorates due to the retention of more redundant frames within a given context length.

(6) In Table 4, the STC strategy is not always effective. What is the reason for this?

Table 4 presents the ablation results for each subtask of MLVU. Although there is a slight performance drop in 3 out of the 7 subtasks, this can be attributed to the compression potentially affecting questions that might rely heavily on fine visual details. However, our default setting (DINO+Query+STC) achieves an average overall improvement of 0.39% compared with DINO+Query, delivering the best results.

In response to running time. We believe there may be a misunderstanding referring to “inter-frame similarity calculation” and “frame-question similarity calculation” as “dense computations will undoubtedly significantly increase inference time”.

Below, we provide further clarification and supporting evidence that these two phrases are factually inaccurate.

If we naively apply the image-based model Cambrian (SigLIP+DINO) for video inference, it requires 22.2 seconds for DINO feature extraction and 20.9 for SigLIP feature extraction for all the video frames. However, by incorporating inter-frame similarity calculation (DINO similarity), we effectively reduce frame-level redundancy, adding only an additional 1.05 seconds to the process. This process significantly decreases the computation required for the remaining frames in subsequent steps for SigLIP feature extraction, from 20.9 (for all frames) to 4.32 (remaining frames), which is 4.83x faster. As a result, inter-frame similarity requires minimal computation time but enables a 1.56x overall speedup in feature extraction time compared to the naive approach.

The query dependent component takes only a total of 0.45 seconds including 0.27 seconds for the frame-question similarity and 0.18 seconds for STC. As such, the added computation results in only a slight increase in total computation time.

When compared with other video baselines, our method demonstrates faster performance compared to the token compression approach of Chat-UniVi, and resampler-based method VideoChat2. Our approach is slightly slower than LLaVA-OneVision, requiring 1.27x the processing time, however, our model improves the performance by 5.2% on average across all the benchmarks shown in Table 1 in the main paper.

In processing hour-long videos

One significant advantage of our method is that the DINO-based frame reduction step substantially decreases the computation required for the remaining frames in subsequent steps. Therefore, while all baseline models encounter OOM issues when processing video longer than 30 minutes at a 1 FPS sampling rate, our model can process hour-long videos with an inference time of 83.41 seconds, which is 43x times faster than real time.

Dear Reviewer kRvH,

Thank you for your continued feedback. We believe there may have been a misunderstanding regarding the runtime and novelty concerns, as our structured explanations and data suggest a limited impact on runtime and highlight contributions that are unique to our design. We hope this clarifies any remaining doubts and would be happy to discuss further if needed. Thank you again for your time and valuable input.

We sincerely appreciate your positive and constructive comments. Below, we address the potential concern you raised.

W1: Table 7 indicates that the model performs less effectively on short videos.

Not quite. In Table 7, our model performs slightly worse than LLaVA-OneVision on the VideoMME short split but surpasses other runner-up models, such as VideoChat2 and LongVA. This is because the VideoMME short video split consists of videos less than 2 minutes, where most questions can be easily answered by analyzing a single frame. Our model outperforms LLaVA-OneVision on average for VideoMME, especially the long video split. In addition, our model outperforms LLaVA-OneVision on other short video benchmarks, such as MVBench (around 16 seconds) by 10.2% and EgoSchema (179.8 seconds) by 7.5%. These benchmarks involve more fine-grained temporal frames, requiring a deeper understanding of temporal dependencies to answer questions accurately.

Q1: Does the proposed compression approach hurt performance on short videos? Table 3 lacks results on MVbench.

Thanks for raising this concern. We did not report the MVBench result in Table 3 since it is a very short benchmark and there is only a slight improvement with our spatiotemporal compression on MVBench.

Therefore, there is no negative influence for our spatiotemporal compression approach on short video benchmarks like MVBench.

Thank you for providing detailed feedback to enhance the clarity of our paper and for your valuable suggestions to further improve its quality.

W1: Clarification

(1) Is there a specific DINO threshold

We empirically observe DINO similarities with several videos and set the DINO threshold as 0.83 for all the benchmarks. We ablate the effect of the DINO threshold. The lower the DINO threshold, the fewer frames are reduced. For instance, in MLVU, as the threshold decreases, performance drops due to the retention of more redundant frames within a given context length.

(2) The authors do not explain how tokens exceeding the LLM context are handled.

Thank you for pointing this out. In our implementation, if the number of tokens still exceeds the LLM's context length after all compression steps, we apply forced truncation to the tokens within each sliding window to ensure they fit within the LLM's context window. We will include the explanation in the implementation details of our revision to ensure clarity and prevent any potential confusion.

(3) Is the cross-attention trained end-to-end during the video SFT stage?

Yes, the cross-modal query is trained end-to-end during the video SFT stage, enabling it to selectively reduce frame tokens based on the user's query.

W2: Context length & computation

For uniformly sampling baselines, due to its sparse sampling nature, taking more frames as input (with increased context length correspondingly) will lead to better performance. But we already argue that uniform sampling is not an optimal solution for video understanding since it has large intervals and loses many visual details.

We chose 8K context length as our setting:

(i) Most of the baselines use 8K context length, since the commonly used MLLM is 8K. Hence we conduct for fair comparison.

(ii) We have tried our model inference in 12K/16K context length, the result is similar to our 8K setting, which means we can effectively reduce the video redundant tokens by our spatiotemporal compression approach.

(iii) Currently, there is a scarcity of long videos with high-quality annotations that can support scaling our methods to longer context lengths. Most videos in our training dataset, VideoChat2-IT, are limited to 3 minutes. Below, we provide an overview of the video lengths for each dataset in VideoChat2-IT.

We have also provided the inference computation compared with other baselines in our general response.

W3: Typos

Thanks for pointing this out. The result for the line of GPT4-o is misaligned, we have fixed the mentioned typos in line 46 and Table 1 in our revised PDF.

Q2: Inference time consumed by DINO similarity and token similarity

Please refer the inference time comparison to general response. As shown in the table below, the primary computation is from frame feature extraction, which, in real-world applications, can be preprocessed offline. Notably, our proposed compression components contribute only a small portion to the overall inference overhead.

Dear Reviewer jb4v,

Thanks for your reply. We have integrated all the results in our revised appendix and we will organize them in our final version.

We would like to clarify that, although the DINO extraction process accounted for the majority of our inference time, it should not be attributed to the computational overhead introduced by our compression method. This is because all models require the first step to extract frame features, and the feature extraction time is comparable across them.

Please kindly notice that in our general response, we provide a comparison of the total inference time compared to baselines.

As the table shows, our method demonstrates faster performance compared to the token compression approach of Chat-UniVi, which relies on a KNN-based strategy to merge similar tokens. Furthermore, it is more efficient than the resampler-based method VideoChat2, which compresses video inputs using learnable queries in Q-Former. When compared to methods without compression, such as LLaVA-OneVision, our approach is slightly slower, requiring 1.27x the processing time, however, our model improves the performance by 5.2% on average across all the benchmarks shown in Table 1 in the main paper.

In addition, the feature extraction process is a necessary step that applies to any input videos for any models, regardless of whether the videos are 'new, unseen' as you mentioned.

Please let us know if this clarification resolves the misunderstanding. We are happy to address any additional concerns you may have.

Best, Authors

We sincerely thank you for your thorough review of our paper. Your valuable suggestions are instrumental in enhancing the quality and clarity of our work, and we deeply appreciate your thoughtful feedback.

W1: Lacks of innovation, e.g., DINOv2 has been explored

Although Cambrian explores DINOv2 benefits in low-level image understanding tasks, however, we are the first to utilize DINOv2 similarity to reduce temporal redundancy (Sec 3.1) in video understanding tasks, different from previous literatures that only work on image modality. Furthermore, proper design is critical for modeling the spatiotemporal nature of video, as demonstrated in our proposed query-based feature selection (Sec 3.2) adaptively select the frames most relevant to the query at the highest spatial resolution permitted by the model’s capacity and spatial token compression (Sec 3.3) where visual tokens are compressed based on temporal dependency.

W2: Inference time comparison

Please kindly refer the inference time comparison to general response. Our method demonstrates more efficient performance compared to the token compression approach of Chat-UniVi, and resampler-based method VideoChat2. Compared to non-compression methods like LLaVA-OneVision, our approach is 1.27x slower but delivers a 5.2% average performance improvement across all benchmarks in Table 1 of the main paper.

Q1: Sliding window size of K=8

$K$ represents the size of the sliding window, a hyperparameter used to divide the frame sequence into chunks for spatial token compression. Below, we show the ablation results for different values of $K$. Adjusting the sliding window size will affect the MLVU performance, while having minimal impact for other benchmarks. In STC, each subsequent frame is compared with a first frame as anchor within the sliding window. We assume that a larger context window can weaken the strong temporal dependency when the subsequent frame is far from the initial frame in the window.

Q2: Analysis of failed examples

Our method spatiotemporally reduces video frames/tokens and concatenates tokens all together to form the overall video representation. However, this approach does not encode the temporal location of individual frames. While we experimented with frame-level positional embeddings to alleviate this drawback, the model still struggles with tasks like temporal grounding, meaningly identifying the precise start and end times of events.

We greatly appreciate your valuable comments. Below, we address your questions, hoping to resolve your concerns.

---

W1: Computational complexity and scalability, trade-offs with compression ratio and context length

Thanks for your concerns. We have provided the inference complexity comparison in the general response. Our proposed compression method reduces computational overhead compared to other approaches, such as Chat-UniVi and VideoChat2.

We conducted an ablation study to compare the impact of different context lengths on our method. By default, we use an 8K context length, as it aligns with most baselines and is the standard for commonly used MLLMs, ensuring a fair comparison. When reducing the context length to 6K, performance suffers due to the restricted context window. Conversely, testing our model with extended context lengths of 12K and 16K yielded results comparable to the 8K setting. This demonstrates that our spatiotemporal compression approach effectively minimizes redundant video tokens, maintaining performance even with longer contexts. In addition, currently, there is a scarcity of long videos with high-quality annotations that can support scaling our methods to longer context lengths. Most videos in our training dataset, VideoChat2-IT, are limited to 3 minutes.

We investigate the impact of DINO threshold selection on performance. A lower DINO threshold results in fewer frames being discarded (lower compression rate). For example, in MLVU, decreasing the threshold leads to performance drop as more redundant frames are retained in a given context length.

We analyze the impact of STC threshold selection on performance and observe a trade-off with the compression rate. Increasing the threshold, meaningly reducing the compression rate, leads to a drop in performance, particularly on the long-video benchmarks MLVU and VideoMME. Conversely, lowering the threshold results in more aggressive compression, which also negatively impacts performance.