Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior

We sincerely appreciate your insightful comments. All suggestions will be incorporated into the paper, and typos will be fixed. Our responses are as follows.

Response to Weakness 1 & Question 1(b, c) (Concern about Cross-Architecture Transferability)

To better demonstrate DyToK's transferability, we conduct experiments on VideoMME, using assistant and primary models with distinctly different architectures under varying retention ratios.

To maximize the differences in architectures and thus rigorously test DyToK's robustness, we select LLaVA-OneVision (OV) and Qwen2.5-VL (Qwen), two open-source frameworks exhibiting significant architectural differences.

Preliminary results are shown below, with OV using 0.5B/7B and Qwen using 3B/7B as assistant and primary models.

Results show that DyToK remains robust even with cross-architecture assistant-primary model pairs. In some cases, using an assistant from a different architecture even outperforms one from the same series under certain compression ratios.

We attribute this to the assistant’s key role in estimating frame importance. As long as this estimation is accurate, architectural alignment is not essential. Additionally, when the models share similar architectures or training recipes, aligned attention distributions may further boost performance.

Response to Weakness 3 & Question 2 (More Experimental Settings)

(1) Inference speed-up with fixed input frames

Using LLaVA-OneVision (7B) on VideoMME with 32-frame inputs, we have evaluated DyToK across retention ratios from 10% to 75%. All statistics include both the assistant and primary models. Results show substantial GPU memory and latency reductions with minimal accuracy loss. At 50% retention, DyToK achieves 2.1× speed-up, 22.1% memory savings, and 98.5% accuracy. Even at an aggressive 10% retention, it retains 94.3% accuracy with 4.5× speed-up.

Please note: The extent of memory savings largely depends on the model size.

(2) Performance gains under fixed computational budgets

We evaluate DyToK’s capability to support longer video sequences under the same token budget as the uncompressed baseline, using Qwen2.5-VL (7B) on MLVU (token budget alignment details in Appendix H.2).

Token Budget Alignment

We attribute the accuracy gains to DyToK’s token compression, which reduces tokens per frame and enables more frames under a fixed token budget, improving temporal granularity.

To address efficiency concerns from the assistant model, we also evaluate DyToK under a fixed FLOPs budget matching the uncompressed baseline, ensuring fair computational comparison. As Qwen2.5-VL yields variable token counts due to dynamic resolution, we use LLaVA-OneVision (7B) for consistent FLOPs estimation.

FLOPs Budget Alignment

Response to Question 1(a) & Question 3 (More Visualization)

We appreciate your constructive feedback regarding visualization. Due to rebuttal format limits, we cannot include visuals now but will add the following in the final version:

• Attention distribution across different assistant model architectures
• Performance trends across varying token retention ratios
• Selected keyframe visualizations moved from Appendix (Figs. 7-14) to the main text
• Comparative frame weight distributions from all query tokens vs. only the last query token
• Attention distributions for general description queries

Please feel free to suggest additional visualizations.

Response to Question 4(a) (Additional Analysis on Textual Token Selection)

We agree that relying solely on the last textual token may not fully capture the sequence’s semantics. To explore this, we conduct ablations on multiple long video benchmarks using LLaVA-OneVision (7B) under various retention ratios.

As shown above, at higher retention ratios (e.g., 75%, 50%), using only the last query token performs better. At more aggressive ratios (e.g., 25%, 15%), using all query tokens yields further gains. We hypothesize that the last token acts as a precise but narrow selector, while all tokens offer broader coverage with higher recall. When visual evidence is abundant, precision reduces noise; when it is scarce, broader semantic coverage becomes essential.

Response to Question 4(b) (Concern about General Descriptive Queries)

Quantitative Analysis. To assess DyToK’s ability to select keyframes for general descriptive queries, we conduct captioning experiments on VideoChatGPT using LLaVA-OneVision (7B), which includes prompts like “What is happening in the video?” Following Dynamic-VLM[1], we use GPT-3.5-Turbo-0613 for quantitative scoring.

Qualitative Analysis. In visualization experiments, we have observed that DyToK effectively identifies both “semantic keyframes” (relevant to query semantics) and “content keyframes” (capturing key scene or event transitions), making it well-suited for general descriptive queries. This demonstrates an advantage over CLIP-based methods that rely solely on semantic matching. While visualizations cannot be included due to rebuttal constraints, they will appear in the final version along with detailed results.

To support our claim during the rebuttal phase, we replace DyToK’s cross-modal attention-based frame weight estimation with CLIP-based semantic similarity. Implementation details follow AKS[2]. Results remain consistent.

[1] Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM

[2] Adaptive Keyframe Sampling for Long Video Understanding

Dear Reviewer bZfF,

We sincerely appreciate the time and effort you devoted to reviewing our manuscript. In response to your thoughtful feedback, we have submitted a rebuttal with extensive experimental results and clarifications addressing your concerns, which includes the following key points:

• Cross-Architecture Transferability. Extensive experiments show that DyToK remains robust with cross-architecture assistant–primary pairs, sometimes even outperforming same-series models under certain compression ratios.
• Fixed Input Frames / Computational Budget. We assess DyToK’s efficiency (latency, FLOPs, memory) with 32-frame input, and evaluate its performance gains under fixed token and FLOPs budgets.
• Analysis on Textual Token Selection. Ablations show that the last query token excels at higher retention, while all tokens perform better under aggressive compression, suggesting a precision–recall trade-off.
• Analysis on General Descriptive Queries. DyToK effectively identifies both semantic and content keyframes, outperforming semantic matching in qualitative and quantitative evaluations, with CLIP-based ablations further supporting the results.

We hope that these clarifications and additional experiments can address your concerns. Thank you again for your constructive review, and we welcome any further feedback or questions.

Sincerely,

Authors of Paper #19575

We sincerely appreciate your comments. All suggestions will be incorporated into the paper. Our responses are as follows.

Response to Weakness 1 (Additional Analysis on Textual Token Selection)

Thank you for the insightful suggestion. We agree that relying solely on the last textual token may not fully capture the sequence’s semantics. To explore this, we conduct ablations on multiple long video benchmarks using LLaVA-OneVision (7B) under various retention ratios.

As shown above, at higher retention ratios (e.g., 75%, 50%), using only the last query token performs better. At more aggressive ratios (e.g., 25%, 15%), using all query tokens yields further gains. We hypothesize that the last token acts as a precise but narrow selector, while all tokens offer broader coverage with higher recall. When visual evidence is abundant, precision reduces noise; when it is scarce, broader semantic coverage becomes essential.

We will incorporate these analyses into the final version and add visualizations comparing the distributions of frame weights derived from both strategies.

Response to Weakness 2 (Concern on Cross-Architecture Transferability and Weaker Backbone Adaptability)

We sincerely appreciate your insightful suggestion. We agree that validating DyToK's transferability across different model architectures is crucial to demonstrate its generalizability and accessibility. To address this, we conduct experiments on VideoMME, using assistant and primary models with distinctly different architectures under varying retention ratios.

To maximize the differences in architectures and thus rigorously test DyToK's robustness, we select LLaVA-OneVision (OV) and Qwen2.5-VL (Qwen), two open-source frameworks exhibiting significant architectural differences. We believe that strong performance under such settings provides meaningful evidence of generalizability.

Preliminary results are shown below, with OV using 0.5B/7B and Qwen using 3B/7B as assistant and primary models.

The results show that DyToK remains robust even with cross-architecture assistant-primary model pairs. In some cases, using an assistant from a different architecture even outperforms one from the same series under certain compression ratios.

We attribute this to the assistant’s key role in estimating frame importance. As long as this estimation is accurate, architectural alignment is not essential. Additionally, when the models share similar architectures or training recipes, aligned attention distributions may further boost performance.

These comprehensive experimental findings broaden DyToK's practical applicability in various scenarios:

• No lightweight version available: A lightweight variant from another model, such as LLaVA-OneVision 0.5B, can serve effectively as an assistant model, given its proven efficacy and extremely small parameter count.
• Inadequate capacity of lightweight models: If a lightweight model lacks sufficient capability to generate accurate frame weights, a lightweight variant from another pre-trained video large language model can readily serve as the assistant model.

We hope this addresses your concerns regarding DyToK’s cross-architecture transferability and robustness with weaker backbones. Given the importance of this issue to DyToK’s versatility, we will elaborate on this in the main text and add visualizations in the appendix to illustrate attention distribution differences across assistant architectures.

Response to Weakness 3 (Additional Analysis on Deep Layer Selection)

As stated in Section 3.2 (line 154), we define deep layers as the last 25% of the assistant model’s layers to ensure generalizability without architecture-specific tuning. This proportional strategy adapts well across models like LLaVA-OneVision and Qwen2.5-VL, making it unnecessary to repeat Table 3 experiments for each new model.

The ablation in Table 3 serves to validate our key observation from Section 2.2 (line 113) that “deep attention layers provide good keyframe priors.” Appendix A.4 (Table 5) further analyzes individual layers and combinations divided into quarters and sixths, confirming the benefits of aggregating deep layers.

These results support, but do not affect, our practical choice of using the last 25% layers. We appreciate the reviewer’s suggestion and will clarify the distinction between our empirical analyses and adopted strategy in Section 3.2 to avoid misunderstanding.

To further illustrate the transferability of our approach, we also conduct additional layer analyses on LLaVA-OneVision and Qwen2.5-VL.

LLaVA-OneVision

Qwen2.5-VL

The experimental results demonstrate that our layer selection strategy is model- and architecture-agnostic, exhibiting generalizability by eliminating the need for repeated layer analyses when adapting to new models.

Response to Weakness 4 (Inference Time Clarification)

The “prefilling time” reported in Appendix Table 7 reflects the combined inference time of the assistant and primary models during the generation of the first token in DyToK. We focus on this metric because DyToK’s additional computation occurs entirely during prefilling. In contrast, decoding involves only the primary model operating on pruned tokens, leading to consistent speedups. Measuring total generation time would understate DyToK’s actual overhead, as the assistant’s cost becomes negligible over longer sequences. Therefore, prefilling time offers a fair and transparent efficiency measure.

We further validate this choice through additional experiments on long-sequence captioning with VideoChatGPT, which confirm our analysis.

We sincerely appreciate your positive feedback and constructive review. If you have any additional questions, we look forward to further discussions.

We sincerely appreciate your comments. All suggestions will be incorporated into the paper. Our responses are as follows.

Response to Weakness 1 (More Discussion with RAG-Based Methods)

Thank you for your insightful suggestion. Video-RAG primarily focuses on enhancing video comprehension through auxiliary textual information extracted by external speech recognition, OCR, and object detection modules. In contrast, our method specifically targets efficient long-video reasoning and extended video support within a given computational budget, making a direct comparison challenging. A commonality between the two methods is that Video-RAG employs CLIP-based keyframe extraction in its object detection stage.

To facilitate a clearer comparison regarding the handling of keyframes, we incorporate Video-RAG’s keyframe extraction module into DyToK’s temporal importance estimation stage. Specifically, we utilize CLIP’s cross-modal semantic similarity scores to determine frame weights, dynamically allocating token retention ratios for each frame. We present experimental results on VideoMME using LLaVA-OneVision as the base model under different retention ratios below. We will incorporate these discussions and experimental results into our paper.

Response to Weakness 2 (Relative FLOPs Saving Calculation)

We appreciate your valuable feedback on presenting the relative computational savings. To intuitively demonstrate the practical efficiency gains achieved with minimal performance trade-offs, we conduct experiments on VideoMME using LLaVA-OneVision (7B) as the base model, evaluating GPU inference latency, FLOPs, and memory usage across a broad retention ratio range (10%-75%) with 32-frame input. The summarized results are presented in the following table, with detailed efficiency analysis provided in Appendix D. All statistics include both the assistant and primary models.

Notably, the extent of memory savings significantly depends on the model size.

From the table above, DyToK achieves an excellent trade-off between accuracy and efficiency:

• GPU Memory and Inference Savings: DyToK achieves a 2.1× inference acceleration at a 50% retention ratio, and this acceleration increases to 4.3× at a 25% retention ratio. Starting from a 50% retention ratio, GPU memory usage remains around 17.3 GB, which essentially represents the minimum memory footprint determined solely by the model weights.
• Accuracy-Efficiency Trade-Off: At a moderate retention ratio of 50%, DyToK retains 98.5% of the original accuracy, reduces inference time by 2.1×, and decreases GPU memory usage by 22.1%. At a 25% retention ratio, DyToK maintains 97.8% accuracy with a 4.3× speedup. Even at the extreme retention ratio of 10% (only ~14 tokens per frame), DyToK maintains 94.3% accuracy.

To more comprehensively evaluate DyToK’s efficiency, we further conduct long-sequence captioning experiments on VideoChatGPT. Following Dynamic-VLM[1], we use GPT-3.5-Turbo-0613 for quantitative scoring. All statistics include both the assistant and primary models.

[1] Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM

Response to Weakness 3(a) (Broader Model Size)

Thank you for your valuable suggestion. Beyond the 0.5B guiding 7B experiments using LLaVA-OneVision already presented in the main text, we conduct additional extensive experiments using Qwen2.5-VL on the MLVU benchmark, which covers a wide range of commonly used model sizes (3B, 7B, and 32B), as shown below.

These experimental results provide two further insights:

1. Video models inherently possess keyframe priors regardless of model size.
2. Such priors are not constrained by model scale, as even lightweight models can exhibit keyframe perception capabilities comparable to those of larger models. Notably, at a 20% retention ratio, the 3B Qwen2.5-VL assistant model surpasses the 7B variant by a substantial margin, further validating the feasibility of using a lightweight assistant model to provide keyframe priors in DyToK.

Response to Weakness 3(b) (More Analysis on Assistant Model Misprediction Propagation)

Thank you for raising this critical point. In the DyToK framework, the assistant model’s role is strictly confined to extracting frame weights based on keyframe priors, guiding dynamic token retention without directly passing predictions to the primary model.

As illustrated in Figure 1 of the main text, we have empirically found that even when the assistant model’s predictions are incorrect, its cross-modal attention remains accurately aligned with keyframes. Consequently, the transferred frame weights remain generally reliable, reducing the risk of error propagation. This robustness underlies DyToK’s consistent effectiveness across various base models, benchmarks, and retention ratios.

Following your suggestion, we will expand discussions on misprediction propagation in the main text and provide further visual analysis beyond the current visualizations in Appendix Figures 7-14.

Response to Question (Efficiency Improvement Confirmation)

Yes. To ensure fairness, efficiency gains reported in our study (Appendix D) include the forward passes of both assistant and primary models.

Dear Reviewer nbsC,

We sincerely appreciate the time and effort you devoted to reviewing our manuscript. In response to your thoughtful feedback, we have submitted a rebuttal with extensive experimental results and clarifications addressing your concerns, which includes the following key points:

• Discussion with RAG-Based Methods. We compare DyToK with CLIP-based keyframe extraction in Video-RAG through detailed analysis and experiments.
• Relative FLOPs Saving Calculation. DyToK is evaluated on VideoMME across retention ratios in terms of latency, FLOPs, and memory, and further validated on VideoChatGPT for long-sequence captioning, demonstrating strong accuracy-efficiency trade-offs.
• Broader Model Size. Experiments on 3B, 7B, and 32B models reveal that video large language models inherently possess keyframe priors, regardless of model size.
• Analysis on Assistant Model Misprediction Propagation. Experiments show that even with incorrect predictions, the assistant model maintains accurate cross-modal attention to keyframes, which underlies DyToK’s robustness.

We hope that these clarifications and additional experiments can address your concerns. Thank you again for your constructive review, and we welcome any further feedback or questions.

Sincerely,

Authors of Paper #19575

We sincerely appreciate your comments. All suggestions will be incorporated into the paper, and typos will be fixed. Our responses are as follows.

Response to Weakness 1 (Comparison with SOTA Methods)

As described in Figure 2 (page 4) of the manuscript, existing token compression methods can generally be categorized into two paradigms: encoder feature-based and LLM attention-based. Accordingly, we have conducted comparisons with VisionZip[1] (state-of-the-art encoder feature-based method) and DyCoke[2] (state-of-the-art LLM attention-based method), both recently published at CVPR 2025.

Under identical token budgets and across a comprehensive range of retention ratios (10%, 15%, 20%, 25%, 50%, 75%), DyToK has consistently achieved superior performance on diverse benchmarks for long video understanding (VideoMME, LongVideoBench, MLVU). Detailed comparative radar plots and complete experimental results are provided in Figure 1 (Appendix, page 2) and Tables 1 and 2 (Appendix, pages 18–19). If there are specific works that you believe we should additionally compare with, please kindly let us know, and we will do our best to include the comparisons in the discussion phase.

[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models (CVPR 2025)

[2] DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models (CVPR 2025)

Response to Weakness 2 (Evaluation Against Structured Compression Baselines)

1. Training-Free vs. Training-Based: DyToK is designed as a training-free dynamic token compression method; therefore, our evaluation primarily targets other training-free methods. As the structured compression approaches you have mentioned require training, the comparison with them is beyond the scope of our current comparative framework.
2. Inference Acceleration vs. Training Acceleration: DyToK is a training-free method that aims at accelerating inference in video large language models through dynamic token compression, whereas methods such as LoRA or parameter-efficient fine-tuning primarily target reductions in training resource consumption. Due to these distinct goals, a direct comparison is inherently challenging.

Response to Weakness 3 & Question 3 (End-to-End Latency on Real Hardware)

To demonstrate the real-world efficiency gains provided by our method with minimal performance compromise, we have evaluated DyToK using LLaVA-OneVision (7B) on VideoMME with 32-frame inputs. Below we summarize GPU inference latency, FLOPs, and GPU memory consumption across various token retention ratios (10% to 75%). More comprehensive efficiency analyses and experimental results can be found in Appendix D.

Note: The extent of memory savings largely depends on model size.

As demonstrated, DyToK provides an excellent trade-off between accuracy and efficiency:

• GPU Memory and Inference Savings: DyToK achieves a 2.1× inference acceleration at a 50% retention ratio, and this acceleration increases to 4.3× at a 25% retention ratio. Starting from a 50% retention ratio, GPU memory usage remains around 17.3 GB, which essentially represents the minimum memory footprint determined solely by the model weights.
• Accuracy-Efficiency Trade-Off: At a moderate retention ratio of 50%, DyToK retains 98.5% of the original accuracy, reduces inference time by 2.1×, and decreases GPU memory usage by 22.1%. At a 25% retention ratio, DyToK maintains 97.8% accuracy with a 4.3× speedup. Even at the extreme retention ratio of 10% (only ~14 tokens per frame), DyToK maintains 94.3% accuracy.

To more comprehensively evaluate DyToK’s efficiency, we further conduct long-sequence captioning experiments on VideoChatGPT. Following Dynamic-VLM[1], we use GPT-3.5-Turbo-0613 for quantitative scoring. All statistics include both the assistant and primary models.

[1] Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM

Response to Weakness 4 (More Justification for the Attention-Based Importance Estimation)

Thank you for highlighting this issue. We fully agree that different importance estimation strategies warrant further investigation. Following your suggestions, we conduct additional ablation studies by comparing our cross-modal attention-based frame weighting method (Temporal Importance Estimation stage) with entropy-based and magnitude-based approaches:

• Attention Entropy-Based: For each frame, compute the attention weights of the last query token over all visual tokens. Calculate the entropy of these attention weights and normalize the results to obtain the frame weights.
• Feature Entropy-Based: Compute the information entropy of visual tokens along the feature dimension. For each frame, average the entropies of all tokens and normalize the values to derive the frame weights.
• Feature Magnitude-Based: Calculate the L2 norm of visual tokens for each frame. Average the norms and normalize the results to obtain the frame weights.

It can be observed that DyToK consistently outperforms entropy and magnitude-based methods across different retention ratios. We appreciate your valuable feedback and will include further analysis and discussion in the revised paper.

Response to Question 2 (Definition of Keyframe and Adaptive Rate)

• Definition of Keyframe: A keyframe is defined as a crucial frame that requires particular attention for accurate video understanding. For a comprehensive explanation, please refer to the Preliminary section (lines 94–101) and the cited literature there.
• Mechanism of Adaptive Token Allocation: DyToK allocates initial token quantities per frame by multiplying frame weights from the temporal importance estimation stage with the specified token budget. It subsequently trims tokens based on an upper limit to mitigate temporal attention outliers (see Appendix B.1) and redistributes remaining tokens according to frame weights in descending order. Implementation details are described in Algorithm 1 of the Methodology section.

We have completed additional visualizations that provide a clearer explanation of our method. As image attachments are not permitted in the rebuttal phase, these visualizations will be included in the revised paper.

Dear Reviewer HiJ7,

We sincerely appreciate the time and effort you devoted to reviewing our manuscript. In response to your thoughtful feedback, we have submitted a rebuttal with extensive experimental results and clarifications addressing your concerns, which includes the following key points:

• Comparison with SOTA Methods. DyToK consistently outperforms recent CVPR 2025 SOTA methods, VisionZip and DyCoke, across six retention ratios and three long-video understanding benchmarks.
• Evaluation Against Structured Compression Baselines. DyToK is a training-free token compression method for inference acceleration, thus complementary rather than competitive with techniques like LoRA or quantization that reduce training cost or require retraining.
• End-to-End Latency on Real Hardware. DyToK is evaluated on VideoMME across retention ratios in terms of latency, FLOPs, and memory, and further validated on VideoChatGPT for long-sequence captioning, demonstrating strong accuracy-efficiency trade-offs.
• Justification for Attention-Based Importance Estimation. Extensive ablations comparing DyToK’s cross-modal attention-based weighting with attention entropy-, feature entropy-, and feature magnitude-based methods confirm its superiority.
• Definition of Keyframe and Adaptive Rate. We further clarify key concepts and provide additional visualizations for a clearer explanation of our method.

We hope that these clarifications and additional experiments can address your concerns. Thank you again for your constructive review, and we welcome any further feedback or questions.

Sincerely,

Authors of Paper #19575

Thank you for your responses and your acknowledgement. We would like to further clarify the following two aspects of our work:

Comparison with the state-of-the-art methods

We have conducted extensive experiments using VisionZip and DyCoke, two representative and recently accepted methods at CVPR 2025 whose code and models are publicly available, enabling fair and reproducible comparisons. These baselines were selected because they were the most relevant accessible works accepted by CVPR 2025, the latest top-tier conference that had released its accepted list before the NeurIPS 2025 submission deadline (May 11, 2025).

Impact and potential for broader applicability

We have conducted experiments with VisionZip and DyCoke on mainstream VLLMs (LLaVA-OneVision and Qwen2.5-VL) across six compression ratios (10%–75%), three long-video benchmarks, varying input frame lengths, and both encoder-feature and LLM-attention token compression paradigms. The results consistently demonstrate that our approach improves VLLMs’ inference efficiency in a training-free and model-agnostic manner, with broad applicability (see Appendix Tables 1–4, pp. 18–21 for full results).

Moreover, as shown in discussions with other reviewers, our method exhibits:

• Cross-architecture transferability (W2 of Reviewer mobj, W1/Q1 of Reviewer bZfF);
• Scalability across model sizes (W3(a) of Reviewer nbsC);
• Strong generalization to different usage scenarios, including fixed compute budgets, fixed input frames, and general descriptive queries (W1, Q2, Q4 of Reviewer bZfF). Importantly, the generalization capability of our designs can be supported by cross-model statistical analysis instead of relying on the empirical tuning (see our response to Reviewer bZfF’s follow-up), ensuring the method’s robustness in different settings.

Given the above facts and analysis, we believe that the proposed DyTok is a simple, effective and general method. However, its simplicity does not mean limited novelty. To the best of our knowledge, no prior work has explored dynamic temporal budget allocation for accelerating the inference of VLLMs. DyTok introduces a plug-and-play mechanism that consistently improves inference efficiency across different existing models, without any post-training overhead.

We hope this clarification highlights the method’s impact, broad applicability and novelty, and we sincerely appreciate your consideration.