FlexSelect: Flexible Token Selection for Efficient Long Video Understanding

We appreciate your valuable comments. We address your concerns below:

Q1: Narrative should be more clear

We acknowledge the narrative could be clearer. Our analysis reveals that specific intermediate layers provide faithful semantic alignment, while early and late layers show poor alignment. However, existing methods fail to identify these optimal layers: DyCoke [29] prunes tokens at layer 3, while FrameFusion [12] operates at the initial layers (0, 1, 2). Based on our analysis, attention scores in these early layers are highly unreliable for semantic relevance estimation. As discussed in the fourth paragraph, we observe that semantic alignment between textual queries and visual tokens gradually strengthens with deeper layers, peaking at an intermediate layer (e.g., layer 19 for LLaVA-Video-7B). This finding serves as the key motivation for FlexSelect and directly explains why prior attention-based methods suffer from performance degradation.

Q2: Lack of comparison with the referenced attention-based token importance methods Framefusion [12] and Dycoke [29]

We appreciate this suggestion and will include direct comparisons in the revision. We compared FlexSelect with FrameFusion [12] on LLaVA-Video-7B and DyCoke [29] on LLaVA-OV-7B, using the same number of sampled frames and pruning ratio. Here are our results.

The results show that FlexSelect achieves better performance than DyCoKe and FrameFusion under the same number of input frames and retain ratio. Additionally, FlexSelect supports 512 input frames due to its Frame Set Partition mechanism, obtaining more performance improvements.

Q3: Potential overfitting to a single layer.

Thanks for the insightful question. We followed your suggestion and conducted additional experiments to investigate potential overfitting to a single layer. We identify layers with high Recall@K scores from Fig. 2 (layers 14, 16, 19, and 21 for LLaVA-Video-7B). We conduct training-free token selection using averaged attention scores and evaluate on the VideoMME benchmark. The results are below:

The results show that multi-layer aggregation yields comparable performance to single-layer selection, with only marginal differences. This suggests two key insights: (1) layers with high Recall@K tend to focus on similar visual tokens with comparable rankings, making aggregation effects minimal, and (2) our single-layer approach is not overfitting but rather efficiently capturing the semantic alignment already present across multiple high-performing layers.

We will add this multi-layer analysis to demonstrate the robustness of our approach.

We are truly appreciated for your valuable comments. In the following, we provide responses to the concerns.

Q1: The proposed compression method only enables local compression, failing to consider global compression

Regarding the concern about local vs. global compression, FlexSelect performs global token selection through our uniform frame set partition strategy (Sec 3.2). Rather than simple chunk-by-chunk segmentation, we partition frames uniformly across the entire video:

For example, if we have 512 frames and want to split them into 8 segments, segment 1 contains frames [0, 8, 16,..., 504]; segment 2 contains frames [1,9, 17,..., 505]; segment 8 contains frames [7, 15, 31,..., 511].

This ensures each segment has a global view of the entire video, allowing relevant tokens from any temporal position (including only the end of the video) to be selected.

We compare our uniform sampling method with the chunk-by-chunk sampling using LLaVA-Video-7B. The results are below:

This demonstrates our method's effectiveness in global compression while maintaining computational efficiency.

Q2: The motivations for segmenting long videos for token reduction and using attention scores from deep LLM layers have been previously proposed in VideoChat-Flash

We acknowledge VideoChat-Flash as excellent related work and will add proper citations. Here we clarify our key differences:

1. The segmentation strategy is different. VideoChat-Flash segments videos into 4-frame clips chunk by chunk and extracts visual features using a video encoder, then employs an MLP to further compress each clip's features into a more condensed representation. This reduces each frame to just 16 tokens, dramatically decreasing the total token count for long videos and improving efficiency. The motivation behind the segmentation is that the substantial redundant and repetitive information, such as backgrounds and objects present between adjacent frames in natural videos.

   FlexSelect, however, segments videos to ensure the context length of each segment within the model’s maximum context limit, enabling existing VideoLLMs to select relevant visual tokens from hundreds of sampled frames. Meanwhile, as clarified in Q1, FlexSelect employs a uniform segmentation to preserve a global view of the entire video within each video clip, which is different from the chunk-by-chunk segmentation in VideoChat-Flash.

2. The token pruning strategy is different. VideoChat-Flash employs a progressive visual token dropout strategy based on the observation that when an LLM processes long video contexts, it attends to the entire video in shallow layers while focusing on local details in deeper layers. It uniformly drops video tokens in shallow layers and selectively drops them based on attention scores in deeper layers, further improving efficiency and slightly enhancing performance.

   However, FlexSelect analyzes the semantic alignment between text and visual tokens across different layers, identifying an intermediate layer that aligns these two modalities the best. FlexSelect simply averages attention scores from this layer and drops low-scoring tokens, achieving significant performance improvement.

Q3: Traditional video model OR LLM layers for token selection? And comparison with VisionZip

Thank you for the insightful discussion. Both LVAgent and LongVU employ traditional visual encoders to extract frame features and select relevant video regions. LVAgent introduces a novel multi-round framework that enables collaboration between multiple VideoLLM agents, achieving excellent performance in long video understanding tasks. In its perception stage, LVAgent finetunes a CLIP-ASP visual encoder to extract frame features and calculate clip scores to selecte important video segments. Similarly, LongVU uses DINOv2 to extract visual features from sampled frames and reduces temporal redundancy by computing cosine similarity between features to select relevant frames.

LongVU’s core contribution lies in its efficient training on video data—its five-stage training strategy balances efficiency and robust video understanding. LVAgent’s strength comes from its Selection-Perception-Action-Feedback mechanism, where multi-round reasoning and multi-agent collaboration enable effective long video processing. In contrast, FlexSelect focuses on training-freely selecting query-related tokens. We will add these discussions in the revision.

To compare whether LLM layers or traditional visual encoders are more suitable for token selection, we evaluate VisionZip, which also training-freely selects top-k tokens based on attention scores from the visual encoder as dominant tokens and merges the remaining tokens as contextual tokens. While VisionZip did not report results on any video benchmarks, its official GitHub repository provides an implementation for Qwen2.5VL. We compare VisionZip and FlexSelect using Qwen2.5VL-7B under 64 sampled frames.

With retaining the same number of visual tokens, FlexSelect continues achieve better performance than VisionZip, indicating that for token-level visual token selection, attention scores from LLM layers is more effective than traditional video models.

Q4: In what aspects do the key innovations and insights provided in this paper differ from previous studies?

Thanks for the question. We have discussed the difference between FlexSelect and VideoChat-Flash, LongVU and LVAgent in Q2 and Q3. Here we clarify our innovations and insights compared with the FastV and VisionZip.

FastV identifies and analyzes the inefficient visual attention phenomena in MLLMs from the perspective of visual tokens' attention proportion relative to all tokens. They find that after the second layer, image tokens garner an average attention score that amounts to only 0.21% of the score attributed to system prompts, while in the initial two layers, this figure reaches 50%. Thus, they drop visual tokens based on the second layer's attention scores, despite efficiency gains, leading to performance degradation. In contrast, FlexSelect analyzes attention scores from the perspective of semantic alignment between visual tokens and user query tokens through our proposed Recall@K experiment, locating the optimal intermediate layer in the LLM. Using the attention scores from this layer to select query-related visual regions achieves better performance.

VisionZip finds that in visual encoder(CLIP or SigCLIP) of MLLMs, only few visual tokens hold higher attention weights while most visual tokens receive very low attention and add significant redundancy. So they locate the dominate visual tokens with high attention scores in visual encoder and merge the other visual tokens. Different from this, FlexSelect analysis the attention scores from LLM layers.

In summary, FlexSelect investigates the cross-modal interaction patterns between text and visual tokens across different layers in Video-LLMs. The results demonstrate that semantic information in visual tokens is not fully resolved in shallow layers but gradually emerges as depth increases, with this phenomenon being most prominent at an intermediate layer. Using the attention scores from this specific layer enables effective selection of query-relevant visual tokens, significantly improving both the efficiency of long-video processing and overall model performance in Video-LLMs.

We appreciate your valuable comments. Below are our responses:

Q1: Lack of comparison to other token dropping methods

We appreciate this suggestion and will include direct comparisons in the revision. We compare the VideoMME result with BOLT, DyCoke and FastV on LLaVA-OV-7B.

For comparison with BOLT, we sample at 1 fps (max to 512 frames) and select 32 * 196 tokens per video, which is the same as BOLT. The result shows that FlexSelect with fine-grained token-level selection significantly surpass BOLT that with frame-level selection.

For comparison with DyCoke and FastV, we maintained their experimental settings, sampling 32 frames per video. FlexSelect consistently outperforms both DyCoke and FastV across different retain ratios.

The key difference from FastV is our systematic identification of the optimal reference layer through semantic alignment analysis. FastV considers the proportion of visual tokens' attention scores in the whole attention map, suggesting that layers with higher proportions are more suitable for token pruning. It points out that in MLLM, visual tokens garner an average attention score that amounts to only 0.21% of the score attributed to system prompts, but this figure reaches 50% in the initial two layers. Therefore, they choose to prune tokens at layer 2. FlexSelect considers the semantic alignment between the vision and language modalities, suggesting that layers with attention scores that better locate query-related visual tokens are more suitable for token pruning. Thus, we prune tokens at the reference layer. The experiment results show that our method significantly surpasses FastV.

Q2: Beyond the NIAH-Type queries

Thanks for your question.

Our method generalizes beyond NIAH-Type scenarios. It performs well for both NIAH-type and holistic queries. We evaluate FlexSelect on four well-designed long video benchmarks that include diverse types of questions, such as complex reasoning, spatial temporal perception. Our results show that FlexSelect consistently enhances the performance across these tasks, no matter the query is NIAH-type or holistic-type. For example, MLVU benchmark contains Holistic Task that consists of Topic Reasoning (TP) and Anomaly Recognition (AR). These questions require the models to watch the entire video to answer correctly.

Q3: Concerns about contradictions with the results of DyCoke

Thanks for your question. The experimental analysis in Table 5 of DyCoke [12] shows that cross-attention based pruning (w/o DP) performs worse than the original model, which seems to contradict with our results but actually not. There are crucial differences that explain our divergent findings:

1. Attention type is different. DyCoke uses output-to-visual attention while we use query-to-visual attention. This is fundamentally different - we analyze attention during the prefilling stage before generation begins, while they analyze attention during the decoding process.
2. Layer selection strategy is different. DyCoke performs pruning at layer 3, while we systematically identify and use the optimal reference layer (e.g., layer 19 for LLaVA-Video-7B).

We compared the results of using these two types of attention scores for token pruning at layers 3 and 19, and the results are below. The experiment is conducted under LLaVA-OV-7B with 32 sampled frames. We found that (1) Query-to-visual attention contains more explicit semantic information and consistently outperforms output-to-visual attention. (2) Regardless of the attention type, the performance at layer 3 is worse than the original model, which aligns with FlexSelect's analysis and is consistent with the results in DyCoke. (3) Our reference layer selection is crucial for achieving performance gains. The dramatic performance difference between layer 3 and layer 19 pruning (56.7→60.4 for query-to-visual) underscores the importance of our systematic approach to identifying the optimal reference layer.

We ensure that our results are reproducible, and will open source all the code related to the results in the future.

Q4: Caption Task Result

Thanks for the question, we compare results of VideoDC on LLaVA-OV-7B with 32 sampled frames, and will include it in our revision. Here are the results.

The result shows that FlexSelect achieves comparable performance with the original model. We find that when models are prompted with queries like "Describe the video in detail", the reference layer's attention patterns still exhibit meaningful selectivity, focusing on important elements like main subjects and actions while deemphasizing unnecessary background. The cross-modal attention mechanism inherently adjusts its selection strategy based on query type—focusing narrowly for specific questions while maintaining broader coverage for descriptive tasks. This indicates that FlexSelect's semantic relevance scoring effectively handles various video understanding scenarios.

Q5: Overfitting and generalization of token selector

Thanks for the discussion. Our token selector is trained on a randomly sampled subset of LLaVA-Video-178K, which contains diverse task types including QA and captioning. The results in Table. 1 show that our token selector (FlexSelect-Lite) can generalize well to different question types.

Regarding model scaling (Table 4), larger token selectors do achieve marginally better performance (67.2→67.4), suggesting improved generalization capacity. However, this comes at significant computational cost (3× response time for 3B vs 0.5B models). Given the minimal performance gains, the 0.5B model offers the optimal efficiency-performance trade-off for practical deployment.

Q6: Missing baseline: Majority Voting

We compare majority voting with FlexSelect on LLaVA-Video-7B and will include it in our revision. Here are our results:

For majority voting, we divided each video into 64 temporal bins, randomly sampled one frame per bin, and repeated this process 8 times. The final answer was determined by the most frequent prediction.

FlexSelect significantly outperforms majority voting while being more efficient. It enables the model to reason over semantically relevant tokens from the entire video simultaneously, rather than combining isolated predictions.

Q7: Dynamic token selection for tasks

Thanks for your insightful advice. Actually, the number of necessary visual tokens varies by task, with some tasks relying on a few key tokens and others requiring a thorough visual analysis. As shown in table below, for caption tasks (VideoDC), performance improves as more tokens are selected, but this trend does not apply to VideoMME. While adaptively choosing the number of tokens based on user queries is a promising research direction, it poses challenges due to the complexity of the task and the difficulty of the video content. We leave this as the future work.

Q8: How Eq. 1 averaging scores across question tokens

We average the attention scores between each visual token and all user query tokens across head dimension:

$r_i = \frac{1}{H \cdot L_t} \sum_{h=1}^{H} \sum_{j=1}^{L_t} attn[h,j,i]$

where H is the number of attention heads; $L_t$ is the number of query tokens; attn is the attention score between the j-th query token and the i-th visual token.

Q9: Temporal selection only

Thanks for this discussion. We explored attention based frame selection by averaging the scores (derived from reference layers) across the frame dimension and inference only with the top-k highest-scoring frames. Here are our results on VideoMME using LLaVA-Video-7B.

The results show that frame selection slightly improves performance but is less effective than token selection when using the same number of visual tokens. We find that this is because most attention scores are nearly uniform and minimal, and some top-scoring visual tokens are irrelevant and act as noise. As a result, the average attention scores at the frame level become smooth, making it hard to identify key frames.

Q10: "No training" in Table 3

Thanks for the question. In the "No training" configuration, we directly utilize attention scores from the final layer of the tiny LLM to rank visual tokens without training. Owing to the pre-existing visual knowledge, the tiny LLM could produce a reasonable rank.

Q11: Were larger models undertrained?

Thanks for this question. The larger token selector in Table 4 was well-trained. We attempted to train the larger token selector on a larger-scale dataset and found that the loss had already converged, and the performance remained basically unchanged when training with more data.

We appreciate your valuable comments. In the following, we provide responses to the concerns.

Q1: The Needle-In-Haystack design is not ideal. The data distribution is different from the natural videos.

Thanks for the question. The Needle-In-Haystack task is a well-established benchmark in video understanding. This task assesses whether a Video-LLM can identify visual cues from the extensive visual tokens based on the query, and has been used in recent works such as the NQA (Needle QA) test in MLVU, the V-NIAH experiment in LongVA, and Multi-Hop V-NIAH in VideoChat-Flash. We choose this task because it allows us to more clearly and easily observe the alignment between text and visual tokens across different layers.

To address your concern, we conducted additional experiments on natural videos using the temporal grounding task. We randomly sampled 128 videos from Charades-STA, where each video is paired with a ground-truth temporal range relevant to one question. We treated tokens within the ground-truth range as semantically relevant and others as irrelevant, then computed the layer-wise Recall@K for LLaVA-Video-7B. Here are our results:

The results on natural videos show a consistent pattern with our synthetic experiments: Recall@K increases through early layers, peaks at intermediate layers (layers 16, 19, 22), and declines in deeper layers. This validates that our reference layer selection methodology generalizes to real-world video data.

Q2: The semantic salient tokens alone are not sufficient for all questions. What about the result of caption tasks.

Thanks for your question. We acknowledge that different task types may require varying levels of visual information. Our experiments demonstrate that FlexSelect adapts effectively across diverse tasks, including captioning.

In Table. 1, we evaluated FlexSelect on four long video benchmarks (VideoMME, LongVideoBench, MLVU, and LVBench) that include diverse types of questions, such as complex reasoning, spatial temporal perception. Our results demonstrate that FlexSelect consistently enhances the performance of VideoLLM across these tasks.

To specifically address captioning, we evaluated FlexSelect on the VideoDetailCaption (VDC) benchmark using LLaVA-OV-7B with 32 sampled frames. Retain ratio refers to the keep ratio of visual tokens.

The results reveal that captioning tasks benefit from higher token retention ratios. With 23.25% retention, FlexSelect achieves comparable performance (3.29 vs 3.30) to the baseline while also improves efficiency a lot.

Interestingly, even for open-ended queries like "Describe the video in detail" the reference layer's attention patterns still exhibit meaningful selectivity. The model naturally focuses on salient elements (e.g., main subjects, significant actions) while deemphasizing redundant background information. For example, when describing the video in Fig. 3, the model attends to relevant elements like the commentator and Christmas decorations while filtering repetitive backgrounds. The cross-modal attention mechanism inherently adjusts its selection strategy based on query type—focusing narrowly for specific questions while maintaining broader coverage for descriptive tasks. This demonstrates that FlexSelect's semantic relevance scoring is sufficiently flexible to handle diverse video understanding scenarios.

Q3: The detailed comparison on the number of selected frames, input tokens, performance and inference latency is desired

Thank you for your advice. The effects of the input frame number and selected token number on performance and time cost have been analyzed in Table 2 and Figure 5. Here we provide a more detailed comparison. The tables below show the VideoMME results and time cost at different input frames and selected tokens.

Our experiments show that using 512 input frames with 6,720 selected tokens (6.25% retention) achieves 68.9% accuracy on VideoMME, improving from the original model's 64.4%. This supports our hypothesis that semantic token selection enhances efficiency and accuracy in long-form video understanding.

Additionally, FlexSelect enables processing up to to 1,024 frames—avoiding OOM errors in the baseline—while maintaining a competitive 68.1% accuracy, offering crucial scalability for real-world applications needing extended temporal coverage.

Q4: The performance comparison on models with large context window is preferred, e.g., both sampling 512 frames w/ and w/o FlexSelect without exceeding the context window.

Thanks for the question. Qwen2.5VL series models support long window context. Following the official repository, we use YARN with scaling factor = 4.0 to expand the context window length from 32k to 128K. The result of Qwen2.5VL (original) we reported in the main table was tested under 768 sampled frames (about 107k tokens). Here are the results:

The results show that FlexSelect with just 7,010 tokens outperforms the baseline with 107k tokens, achieving higher scores on all four benchmarks. By focusing on semantically relevant tokens, FlexSelect reduces the noise and redundancy inherent in dense video representations and improves performance, independent of the model's context window length.

Q5: Did the authors try soft token merging or adaptive region resize instead of hard token selection through the learned token selector?

Thanks for the insight discussion. We explored both soft token merging and adaptive region resizing as alternatives to hard token selection, though neither achieved comparable results to FlexSelect.

For soft token merging, we implemented a LinVT-inspired approach on InternVL2-2B, which trains an adapter to compress visual tokens into fixed number of token sets before they are fed into LLMs. The adapter consists of a Spatio-temporal Visual token Refiner module (SVR) to distill the video information into a concise set of visual tokens, and a Text-conditioned Token Aggregator (TTA) module to refine and integrate the visual tokens with the textual information. In our implementation, we use 4 full transformer layers as TTA and 8 spatial-temporal transformer layers as SVR.

The parameters of the adapter are randomly initialized and we trained it in two stages on video instruction data. The first stage only trains the adapter and the second stage trains both the adapter and the LLM. We finally found that such training degraded the performance. Here are our results:

We hypothesize that learning-based compression introduces information loss that cannot be recovered, whereas FlexSelect preserves the critic information well by selecting semantically relevant visual tokens.

For adaptive region resizing, we explore a similar training free method on InternVL2.5-8B. The visual encoder of InterVL2.5-8B supports both of 224 * 224 and 448 * 448 resolution inputs. We first resize the video frames into the small resolution and locate the important frames based on the query-to-visual relevance scores. And then we resize the important frames to the higher resolution while remains the other frames as low resolution. Here are our results:

Results showed consistent underperformance: with 64 frames (16 high-res), accuracy dropped from 64.2% to 62.4%; even with 128 frames (32 high-res), we only matched baseline performance while using 14,336 tokens, while FlexSelect's 2458 tokens achieving 65.6%.

Q6:More visualizations on the selected tokens are required.

Thank you for this suggestion. In our revised version, we will include more comprehensive visualizations to fully demonstrate FlexSelect's robustness and adaptability across different tasks.

Besides the visualizations in Appendix A.6 (Figure 9), we observe consistent patterns in different question types: (1) Object-specific queries focus token selection around relevant objects; (2) Action queries select tokens across motion sequences with temporal coherence; (3) Counting queries distribute selection to capture all instances. These visualizations show our cross-modal attention mechanism effectively adapts to query semantics.

References:

1. LinVT: Empower Your Image-level Large Language Model to Understand Videos