Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation

Thank you for your valuable feedback. Here is our reply to your main concern:

Can the authors shed some lights of its possible extension to a streaming framework?

We sincerely appreciate the reviewer’s insightful question. While our proposed DKS and DFM processes significantly reduce token counts, enabling ViLAMP to handle long video inputs, the current architecture does not yet support streaming videos where input length increases incrementally over time. However, we believe our method can be adapted to streaming scenarios through some adjustments. One feasible solution involves:

• Keyframe Candidate Pool: A candidate pool for keyframes is maintained. When a new frame arrives, it is identified as a keyframe based on two criteria: (1) Its similarity to the query exceeds a threshold. (2) Its similarity to existing keyframes in the pool is below a threshold. If both conditions are met, the frame is added to the pool, and an existing keyframe is evicted based on criteria such as (a) query similarity, (b) similarity to the newly inserted frame, or (c) the frame’s retention duration in the pool.

While this approach theoretically supports streaming scenarios, several implementation challenges require further exploration. A critical issue lies in kv-cache management: Modifying historical information (e.g., updating keyframes) currently does not update the kv-cache, leading to an approximate rather than exact kv-cache during inference. This approximation may impact performance, and rigorous empirical validation is necessary to quantify its effects. We plan to address these nuances in future work, with a focus on efficient cache updates and experimental validation.

Although this method's effectiveness has been verified on video question answering benchmarks for multimodal large language models, it is still unclear whether this method can truly preserve necessary information for more temporal-sensitive tasks like action recognition, video grounding or temporal reasoning. I suggest the authors to add some experiments and anslyses on this aspect.

There have been many existing frame sampling research in video understanding domain such as [1, 2] that can serve as reasonable baselines for comparative analyses. I suggest the authors to add some comparisons with these frame sampling methods.

Thank you for your valuable feedback. We have conducted additional experiments in response to your suggestions, as detailed below:

For the Video Grounding task, we performed experiments on the widely used QVHighlights[1] benchmark (267 citations). This dataset provides annotations for videos, marking the timestamps of query-relevant segments and the positions of keyframes. We evaluated ViLAMP using the Hit@k metric, which measures the probability of the ground-truth keyframe appearing in the top-k keyframes selected by our method. The results are as follows:

As shown, ViLAMP achieves 60% accuracy for Hit@10 and 83% for Hit@30, providing strong empirical support for its Dynamic Keyframe Selection (DKS) process. The current method effectively captures query-relevant video clips.

For the action recognition task, we incorporated the two baselines suggested: Adaframe and MGSampler, and evaluated them alongside ViLAMP on three benchmarks: ActivityNet[2], Something-Something V2 (Sth-V2)[3], and UCF-101[4]. Since ViLAMP is a generative model, we reformulated the original classification task as a multiple-choice problem during testing. The results are as follows:

The results for Adaframe and MGSampler are sourced from their papers. Notably, although ViLAMP was not trained on these datasets, it demonstrates robust performance, significantly outperforming both baselines. This may be attributed to the generalizable knowledge acquired by the large-scale pretrained model, enabling effective zero-shot transfer to action recognition tasks.

We will incorporate these results into our revised draft. Please let us know if you have any further questions or suggestions.

[1] Lei J, Berg T L, Bansal M. Detecting moments and highlights in videos via natural language queries[J].

[2] Caba Heilbron F, Escorcia V, Ghanem B, et al. Activitynet: A large-scale video benchmark for human activity understanding[C]

[3] Goyal R, Ebrahimi Kahou S, Michalski V, et al. The" something something" video database for learning and evaluating visual common sense[C]

[4] Soomro K, Zamir A R, Shah M. UCF101: A dataset of 101 human actions classes from videos in the wild[J].

Thank you for your time and effort, here are our replies:

Q1: Do the authors consider comparing LongVU with some training-free token compression methods[1-4]?

Since VisionZip[1], FasterV[2], and FastV[4] are based on the LLaVA-NeXT model, which is optimized for single image scenarios, they cannot be directly applied to video input. Due to time constraints, we only reproduce DyCoke[3], and the results are as follows:

We will include these results in our revised draft.

Q2: The DKS presets a fixed number of keyframes K, which may not be suitable for all videos. Intuitively, videos with less content variation may require fewer keyframes, while more dynamic and content-rich videos would benefit from selecting more keyframes.

Thank you for your valuable feedback. We would like to clarify that the parameter K serves as an upper boundary for the number of keyframes rather than a required quantity to achieve. In practical applications, when processing videos with less content variation, the DKS algorithm may automatically select fewer keyframes than the specified K value, as outlined in line 4 of Algorithm 1 (Page 5).

As demonstrated in Figure 6, which illustrates the impact of K on model performance, we observe that the model's performance tends to saturate when K exceeds 32. Taking into account both the model's performance and the length constraints of the base model, we ultimately set K to 32.

Q3: VILAMP requires model finetuning, which may limit its portability and practical value.

Indeed, ViLAMP requires finetuning. This is primarily for performance considerations. Training-free methods such as FastV, PruMerge, and DyCoke can only achieve performance comparable to the base model. We finetune ViLAMP with additional data to attain superior results, enabling it to better adapt to compressed video inputs.

Q4: The hierarchical differential distillation relies on multiple hyperparameters, and as shown in Table 4 of the supplementary material, these hyperparameters significantly impact the model’s performance.

As mentioned in Q3, our method does involve finetuning that introduces certain hyperparameters. We fully acknowledge the potential impact of hyperparameter selection on model performance and have conducted systematic experiments and corresponding analyses to offer practical guidance for users. The results demonstrate that the model maintains stable performance with slight deviations from the preset optimal values, and significant performance degradation only occurs under extreme parameter configurations.

Q5: Why does DFM choose to compress non-keyframes into a single token? Would increasing the number of retained tokens during DFM improve performance?

This is an important question. In our pilot study, we explored various configurations by compressing non-keyframes into 1 token, 3 tokens, and 9 tokens, respectively.

The result shows that using more tokens to represent non-keyframes do not yield performance improvements. We attribute this observation to two main reasons: (1) Useful patches are sparse in frames. As shown in Table 4, DFM’s parameter $\alpha$ achieves the best performance when set to $10^{-2}$. This indicates that the patch scores are amplified by 100 before the Softmax operation (see Eq. 10). Such sharpening concentrates the weights on a small subset of patches, suggesting that focusing on only a few critical patches is sufficient for achieving desirable results. (2) Keyframes already contain sufficient details, while non-keyframes primarily serve to complement missing information. In this case, increasing the number of tokens for non-keyframes may not lead to noticeable gains.

Therefore, for efficiency considerations, we ultimately compressed non-keyframes into a single token. We will include the discussion ablve in our revised draft.

Q6: Is it possible to adaptively select the number of keyframes?

As mentioned in Q2, our current DKS keyframe selection algorithm employs a dynamic selection mechanism, though it currently operates with a fixed upper limit K for the maximum number of keyframes. To enhance model adaptability across diverse video inputs, we plan to develop a dedicated module in subsequent experiments. This module will dynamically calculate the optimal K value by considering inter-frame relevance and variation, and video duration, thereby enabling more intelligent adaptation.