One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

Thank you for your thoughtful and constructive review. We are glad that you found our paper well written, appreciated the clarity of our figures and motivation, and recognized the effectiveness of our method on long-video understanding tasks. We especially appreciate your detailed comments on the strengths of our approach and your insightful suggestions regarding generalization to other models and consistency in ablation studies. We address these points in detail below and have taken concrete steps to further strengthen our work accordingly.

1. Experiment on Other Models

The main weakness of this work which is highlighted by the authors in the limitation part is the reliance on only VideoChat-Flash which is not sufficient to see wether this method can also work with other models or not.

Thank you for this question and caring about the generalization of our method! We hope to clarify this comprehensively for you.

First, our method is by-design applicable to any transformer-based video-language model and is not dependent on any designs from VideoChatFlash. Specifically, our major innovation is (1) LP-Comp supervises LLM layers to conduct learnable and progress token compression, and (2) QC-Comp splits the video into individual and parallel segments for efficient token selection.

Second, we select VideoChatFlash as it was the state-of-the-art on long video understanding at the time of submission, which curates an effective mixture of training data. Moreover, VideoChatFlash is unique in utilizing heuristic token compression methods for long video understanding. Thus, it is both a proper baseline and strong counterpart for us to validate our enhanced token compression algorithm.

Thrid, we acknowledge that including more baselines would better demonstrate the strengths of our method. Unfortunately, our computational resources are highly limited: we could only afford to train on 2.5% of VideoChatFlash's data, yet achieved the improvement on long videos (Table 1).

Despite this, we understand the good intentions of the reviewer and is actively implementing and running experiments on the alternative models, such as LLaVA-Next-Video, and will include the results in the revisions.

2. Ablation Studies

Also the ablation studies are not consistently performed across the same benchmarks so it's harder to parse the importance of each of the components. Why didn't you performed the ablation of Table 4 and 5 on LVBench?

We did not have the sufficient computation to comprehensively conduct the ablations on LVBench, as it is the longest video benchmark. According to your request, we have consistently completed the ablation studies on LongVideoBench and LVBench during the rebuttal. We summarize the results below ("Y" denotes "yes," and "N" denotes "No") and conclude that all of our LP-Comp, QC-Comp, and their accompanied design choices are effective consistently.

After reading the other reviews, and the rebuttals, I am satisfied with the discussion and do not have any more concerns on this paper. Thanks to the authors for running the additional experiment during the rebuttal, it's really appreciated. I will increase my score to 5.

Thank you for your thoughtful and constructive review. We are pleased that you find our proposed method OTPF to be a lightweight, modular, and effective solution for efficient model deployment across diverse tasks. We appreciate your recognition of its versatility, strong performance under extreme token reduction, and ease of integration. Your insightful comments regarding early-layer representations, semantic complexity, and reliance on heuristics have helped us clarify and further articulate the motivations, design choices, and future directions of our work. We address your concerns and suggestions in detail below.

1. Early Layer Representations

OTPF’s effectiveness depends heavily on the quality of early-layer representations. If the backbone lacks strong discriminative power, selected tokens may be suboptimal.

This is a very insightful observation, which is exactly our motivation for developing learnable and progressive compression, the heuristic-based methods in prior works. Our insight is that OTPF's token compression relies less on the quality of early-layer representations via optimizing the LLMs to conduct token compression and adapt to the early-layer representations.

• Considering the heuristic-based approaches, e.g., utilizing the correlation between text and image tokens or even uniform drop, they rely on the assumption that the early representations have to be sufficiently informative to mitigate the token drop. Instead, our learnable compression jointly optimizes the LLM and early-layer representation so that the LLMs can adapt to token compression.
• More importantly, our compression is progressive and features a smooth reduction in the token number, which regulates the information loss happening at the ealy layers.

2. Token Reduction and Semantic Complexity

OTPF compresses every input into a fixed number of tokens, regardless of the semantic richness or spatial/temporal complexity. This static compression may underrepresent informative regions in complex scenes and over-represent irrelevant parts in simple ones.

We agree that this is indeed a common confusion regarding token compression. Our design of XComp has attempted to address this problem from both aspects of inter-frame and intra-frame compression:

• Regarding inter-frame compression, our method does not disregard semantic complexity. Instead, our Question-Conditioned Compression (QC-Comp) is proposed to identify the informative frames for emphasis and decrease the influence of irrelevant frames.
• Regarding intra-frame compression, our learnable compression requires the LLMs to aggregate the information into a single token, which supervises the LLMs to select and emphasize the informative regions into the final token, otherwise, it cannot answer the questions correctly. We suggest that such a data-driven way is more adaptive than previous works' heuristics of localizing informative regions.

As a side note, our paper emphasize one token per frame because of its landmark significance for token compression research in long videos and simplicity for implementation. We hope our work can inpire more adaptive and efficient token compression methods in future works.

3. Hand-crafted Heuristics in Token Selection

The token selection relies on hand-crafted heuristics (e.g., activation magnitude, channel max), which may not always correlate with semantic importance—especially in domain-shifted or adversarial settings.

Thank you for this insightful question! The mismatch between heuristically selected tokens and desired information is a long-standing challenge for token compression in long videos, especially as previous token compression entirely rely on heuristics. Similar to the section above, we discuss from intra-frame and inter-frame token compression about our improvement in decreasing heuristics and directions for future work.

• From an intra-frame level, our LP-Comp (learnable and progressive compression) is exactly proposed for LLMs to learn the token compression via end-to-end optimization without relying on manually defined heuristics.
• From an inter-frame level, we acknowledge that our QC-Comp (question-conditioned frame compression) still relies on the heuristics of attention scores similar to prior works. However, the main novelty of QC-Comp is to enhance the efficiency of inter-frame token compression and lost-in-the-middle with our segmented local attention.
• Connected with your insightful feedbacks, we envision an intriguing future work: extending our intra-frame learnable compression (LP-Comp) into the inter-frame compression QC-Comp to provide a heuristic-free token compression method. Despite its excitement, our submission did not attempt this as our LP-Comp was already a notable step towards heuristic-free compression, and extending the LLM to handle inter-frame compression would require significantly more computation resources for supervised fine-tuning. Therefore, we will gladly include these discussions into our future works.

4. Dense Output Tasks

Have the authors tried OTPF in such settings, or do they envision ways to extend it to support dense output tasks?

Thank you for this constructive question! We hope to answer the question comprehensively to you.

First, while OTPF primarily targets multiple-choice tasks on long videos, it is fully capable of handling dense output tasks such as video captioning. The primary reason we focused on multiple-choice tasks in the paper is due to the current landscape of long-video benchmarks, which predominantly evaluate models using multiple-choice formats. This is consistent with prior work in the domain of long video understanding, where the lack of dense-output benchmarks for long videos has been a limiting factor.

To illustrate the capability of OTPF on dense-output settings, we conducted an additional experiment during the rebuttal phase on the VDC benchmark (proposed in the work Auroracap), specifically the detailed captioning subset, which requires dense textual descriptions. While this benchmark focuses on short video captioning, and is thus beyond the scope of our main setting, we believe it demonstrates that OTPF can generalize to dense generation tasks.

As shown in the table below, our OTPF achieves nearly identical BLEU scores to VideoChat-Flash-2B:

As for the future direction, we believe the development of dense-output benchmarks tailored to long videos would be a valuable direction, and we appreciate your suggestion in motivating this line of research.

Thank you for your thoughtful and encouraging review. We are glad that you find our proposed compression strategy—LP-Comp and QC-Comp—well-motivated and effective in addressing token inefficiency for long video understanding. We appreciate your recognition of our method’s empirical strength and generalizability across benchmarks.

We are also grateful for your constructive suggestions regarding efficiency analysis and fine-grained benchmark evaluation. In the following, we provide quantitative comparisons on latency and FLOPs, and include new results on MME-VideoOCR and CLEVRER to assess generalization to short-video and fine-grained tasks. We believe these additions further clarify the strengths and limitations of our method.

1) Efficiency vs. VideoChat-Flash

While the proposed method could achieve comparable or better performance on several benchmarks, the efficiency comparison with baseline VideoChatFlash. I would like the authors to give more discussion and quantative results on time efficiency and FLOPs.

Thanks for your suggestion to give more efficiency comparison! Here we would like to show the efficiency comparison on LVBench items (863 text tokens), measured over 10 runs on a single NVIDIA H200. Across 1k–4k frames, XComp reduces LLM TFLOPs by ~53–59% and latency by ~45–58%.

2) Short-video benchmarks (MME-VideoOCR & CLEVRER)

Compression works well on general videoQA. However, the performance on fine-grained video tasks like video OCR (like MME-VideoOCR) and reasoning benchmarks (like CLEVRER) should be considered to verifiy its generalization ability.

We appreciate the suggestions from the reviewer on analyzing the performance of fine-grained perception, which is indeed a challenging scenario for video understanding. However, these two benchmarks mostly focus on short videos (MME-VideoOCR clips are mostly ≤10 s, and CLEVRER videos are 5 s), while our XComp primarily targets the scope of efficiency and effectiveness on long videos (up to hours). However, we report the performance short-video results per the reviewer’s request.

MME-VideoOCR. XComp and VideoChat-Flash-2B perform similarly overall, where XComp shows stronger temporal grounding but slightly weaker spatial grounding.

For subsets less tied to fine-grained spatial details, the two models are almost identical:

CLEVRER. Since CLEVRER training set is part of VideoChat-Flash pre-train data, both models perform strongly; XComp shows a small drop, as we only used ~2% of the original SFT data from VideoChat-Flash. Continued fine-tuning is expected to close this gap.

Conclusions. Although these short-video evaluations are out of our original scope, they reveal the discovery that our token compression only marginally impact the performance on fine-grained understanding. We appreciate the reviewer’s suggestion and will include these experiments and discussion to motivate future work on jointly optimizing long-video compression and fine-grained short-video perception.

Thank you to the authors for providing additional experimental results.

My concerns regarding Q1 have been well addressed. The latency results are impressive.

For Q2, my original question was about fine-grained video tasks, rather than the short-video benchmark as mentioned in the authors’ response. Since understanding fine-grained text or subtle actions in long videos is more challenging, I am interested in exploring the boundary or limitations of the proposed compression methods. Specifically, I would suggest discussion on the performance of VideoOCR Bench on videos longer than 30 seconds. This would be more indicative of long-term understanding ability.

Dear Reviewers,

As we're approaching the end of author-reviewer discussion period, please read the rebuttal and start discussion with the authors as soon as possible. If all your concerns have been addressed, please do tell them so. Please note that submitting mandatory acknowledgement without posting a single sentence to authors in discussions is not permitted. Please also note that non-participating reviewers will receive possible penalties of this year's responsible reviewing initiative and future reviewing invitations.

Thanks,

AC