LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding

1. The paper claims that STC helps compress video tokens, but it does not provide quantitative details on how many tokens are actually reduced.

Please refer to Figure 4(b), where we illustrate the reduction in the number of tokens after applying STC across different video durations. On average, we observe a 40.4% reduction in tokens during STC. The primary goal of STC is to reduce tokens while preserving visual information, rather than achieving a significant performance improvement. The table below shows the retained token ratio before and after STC, demonstrating that while the number of token is significantly reduced, the model’s performance is effectively preserved.

2. Limited Task Scope. Other video understanding tasks like: Video captioning.

Our video training data, sourced from VideoChat2, includes extensive captioning annotations, as shown in Table 6. Our model is capable of generating detailed video captions, as demonstrated in the qualitative results presented in Figure 3.

Recent long video benchmarks adopt the VideoQA format for easier and more stable evaluation, since automatic caption evaluation is challenging, and GPT-4 based evaluation is not cost friendly. MLVU offers subsets like Video Summarization and Sub-Scene Captioning for captioning tasks. Below, we report MLVU video captioning results, with G-Avg representing the average performance across generation tasks.

3. Lack of list of main contributions.

We believe we developed notable contributions that have been thoughtfully developed and carefully integrated to achieve strong performance. We are clarifying the contributions here.

• Vision-centric visual encoders for video understanding. We identify a key insight: vision-centric encoders trained with feature similarity objectives, e.g., DINOv2, excel at frame reduction in the visual space, while CLIP-based features, optimized for vision-language alignment, are suboptimal for this task. To our knowledge, this has not been explored before for video token reduction.

• Context-aware dynamic token compression. Our method adaptively compresses the video both spatially and temporally by dynamically adjusting the number of tokens based on the video's inherent visual complexity and redundancy, considering both spatiotemporal nature of video and semantic relevance to the user’s query.

• We conducted extensive experiments across several video understanding benchmarks, including EgoSchema, MVBench, VideoMME, and MLVU. Our LongVU method significantly outperforms multiple recent open-source video LLM models and also demonstrates strong effectiveness in smaller models (3B).

1. The core idea of Query and STC lacks substantial novelty. Cross-modal query are known to reduce sequence length long time ago[1], and sparsity in MLLM has already been explored by FastV [2]. This paper differs from FastV in that it compresses tokens before feeding into LLM, but the authors did not demonstrate its advantages either.

FastV is a token reduction method designed for image understanding in MLLMs, operating after the second layer of MLLMs by reranking visual tokens based on attention scores from remaining tokens. In DyCoke [3], FastV was compared as a baseline for long video understanding tasks. The results show that while FastV effectively compresses tokens to speed up inference, it sacrifices performance in long-video comprehension compared to the base model.

As a plug-and-play approach, we applied FastV to our LongVU model to compare its compression with our method.

For our case, the limitation of FastV becomes more pronounced. Our approach performs dense sampling and reduces tokens before input to LLM, ensuring they fit within the LLM’s context length with lower complexity. In contrast, FastV still requires processing all video tokens (100K) in the first two layers, leading to a significant exceeding beyond the LLM's context size (8K). This discrepancy affects attention reliability for extremely long contexts that extend beyond the model’s training distribution. In addition, its attention-based token selection fails to account for the temporal nature of video.

FastV is an effective token compression approach for image understanding, but not designed optimal for long video compression. In contrast, our method effectively compresses video tokens by considering the spatiotemporal dependencies inherent in video data, which not only reduces the number of tokens but also improves performance in long-video understanding tasks.

[3] DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models, CVPR 2025

2. Improvements of the STC module are not significant. STC harms fine-grained temporal understanding abilities.

We propose that STC is not primarily aimed at improving performance but at reducing video redundancy to minimize the number of tokens input into the LLM.

By applying STC, we achieve a 40.4% reduction in tokens, ultimately retaining only 14.2% of the original tokens. It demonstrates that while the number of tokens is significantly reduced, the model’s performance is effectively preserved.

1. LLama-VID OOM issue.

Thanks for your concern. We encountered an OOM issue while running on an 80GB A100 GPU for long videos, using the same settings as other comparison models. To address this, we need to precompute the video features in advance and then load the entire model for inference, as shown in LLaMA-VID (long-video-inference). The inference time for 20-minute videos is 55.3 seconds, which will be updated in Table 11.

2. The main contribution has already been proposed in prior work for either image VLMs or other Video VLMs.

2.1. For example, both [1] and [2] use DINO features alongside the CLIP features.

While MoF [1] and Cambrian-1 [2] utilize both DINO and CLIP features, neither explores their combination for video training and video token compression—where careful design is crucial for practical training and good performance. Simply applying both vision features to all video frames is computationally demanding, making long-video training impractical.

Our work goes beyond merely using both encoders for video understanding—we introduce a novel strategy to reduce computation demand and conduct adaptive video token compression for MLLMs. We identify a key insight: vision-centric encoders trained with feature similarity objectives, e.g., DINOv2, excel at frame reduction in the visual space, while CLIP-based features, optimized for vision-language alignment, are suboptimal for this task. To our knowledge, this has not been explored before for video token reduction. In addition, by leveraging DINOv2 to selectively filter redundant frames before extracting SigLIP features, we significantly reduce computational costs and make long video training feasible.

2.2. Using text query to select optimal tokens is also covered in [3, 4] that the authors do not seem to discuss.

The claim that Goldfish uses text query to select optimal tokens is inaccurate, as its selection mechanism does not operate on visual representations. Instead, it retrieves text descriptions based on query similarity. As we have already discussed in lines 151-154, Goldfish chunks long videos into short clips, generates descriptions with VideoLLM, and encodes them using OpenAI’s text-embedding-3-small. Then it retrieves most relevant descriptions based on similarity between the query's embedding and clip embeddings for the final answer. In contrast, whereas our method selects video tokens, making its approach fundamentally different from ours.

In [3], the Text-Conditioned Resampler (TCR) employs learnable queries to resample a long sequence of video frame features into a fixed-length sequence, which is then fed into a language model. This approach is quite similar to QFormer [5] but is applied in the video domain. However, TCR enforces long video compression into a limited and fixed token space, limiting its ability to represent complex visual information and ignoring temporal variations—resulting in redundancy in static scenes and loss of detail in complex ones.

In contrast to [3], our method dynamically compresses video tokens based on intrinsic dynamics, adapting to each video's complexity. We will integrate the above discussion in our revision.

[5] Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

3. The unique contribution that is different from previous work?

(1) Vision-centric visual encoders for video understanding. To the best of our knowledge, no previous work has explored leveraging a vision-centric visual encoder like DINOv2 for frame reduction. While methods like MovieChat and Chat-UniVi reduce redundancy by calculating frame-level feature similarities, they rely on CLIP features. We argue that language-supervised methods like CLIP fail to capture the nuanced differences between frames.

(2) Context-aware dynamic token compression. Many token compression methods, like TCR, VideoChat, and VideoChat2, rely on QFormer-based modules, which limit video representation by transforming it into a fixed number of trainable queries. These approaches overlook the video's dynamic and informational richness, regardless of their complexity. In contrast, our method adaptively compresses the video both spatially and temporally, dynamically adjusting the number of tokens based on the video's inherent visual complexity and redundancy.

(3) Stronger performance with lower cost. Compared to Chat-UniVi, which extends ToMe-based dynamic token merging for visual features in MLLMs, our method is both more efficient and more effective. As shown in Table 11, it processes over 1K video frames in 32.96 seconds, compared to Chat-UniVi’s 49.06 seconds, while achieving a substantial 17.7% accuracy gain in VideoMME long.

1. The drawbacks of uniform sample and dense sampling.

Numerous studies, including LLaVA-OneVision, LongVA, and SlowFast-LLaVA, have explored the trade-offs between these approaches. Below, we present results of the baseline LongVA using either uniform sampling or dense sampling (1fps, with truncation at the end). LongVA extends the context length to 224K, yet it still cannot fully process long videos at 1 fps, requiring truncation. Performance peaks around 128-frame sampling and then decreases.

2. Lacks a thorough discussion on the visual encoders, and analysis on DINOv2/SigLIP solely for feature extraction.

We appreciate the reviewer's concern regarding the discussion on visual encoders. However, we emphasize that our approach primarily focuses on the adaptive video token reduction, rather than an exhaustive exploration of visual encoders.

In the initial stage, we explored various vision encoders, including SAM, DINOv2, and SigLIP. Due to space constraints, we present the ablation studies conducted during the image pretraining phase, where we compare the performance of SigLIP alone with that of a SigLIP and DINOv2 combination.

The results clearly demonstrate that combining both vision encoders leads to superior performance, resulting in a more robust image understanding model. SigLIP alone ranked as the runner-up. In contrast, DINOv2 and SAM, trained without language supervision, consistently showed an average accuracy more than 5% lower than that of SigLIP.

3. Comparing with training-free token compression methods.

Thank you for sharing these methods. We reported the results in the table below and will incorporate them into our revision.

4. Quantitative statistics on the number of tokens and the compression ratio.

We already illustrated the number of tokens before and after reduction for different video durations in Figure 4. Below we provide the average compression ratio for the VideoMME dataset below for each compression step.

At the temporal frame reduction stage, 45.9% of tokens remain. In the query-based token reduction, 52.1% remain, and at the STC stage, 59.6% are retained, yielding the final retain ratio of 14.2%. This breakdown shows the adaptive nature of our approach and the significant compression achieved throughout the process.

5. The order of experiments within the table does not align with the sequence in the paper.

The last three rows of Table 3 outline the sequence of our method: first, DINOv2 for temporal reduction, followed by query-based selection, and finally, STC reduction. The table also includes ablation studies on context length, tokens per frame, and the impact of using SigLIP or DINOv2 for temporal reduction. We apologize for any confusion caused by the table’s condensed format. To enhance clarity, we will reorganize it in the revision—grouping “Uniform” together, renaming "DINO" to "Temporal Reduction (DINO)," and "SigLIP" to "Temporal Reduction (SigLIP)."

6. The impact statement is missing.

Our work introduces a spatiotemporal adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. This innovation paves the way for future research in video compression tailored for MLLM-based applications, enabling more effective long-video, media, and streaming video understanding. Additionally, we contribute to the open-source community by providing datasets, benchmarks, and models to support the advancement of AI-driven video analysis. We envision a future where MLLMs can directly process compressed video formats aligned with VLLMs that account for spatiotemporal redundancy.

We will add an impact statement section in our revision.