Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs

We thank the reviewer for their constructive feedback. Below, we clarify the raised points and address the specific questions. We are happy to further discuss any remaining concerns during the discussion phase.

Novelty w.r.t. Video-XL

We thank the reviewer for bringing up Video-XL [1]. While both approaches aim to address long video understanding via recurrent mechanisms, there are key differences in design and applicability:

• Video-XL introduces a custom-trained Video-LLM that learns to compress visual tokens, which are then attended to by future tokens. This approach relies on model-specific training, and its effectiveness is constrained by the compression rates and video lengths seen during training. For example, on Long Video Bench, our base model (LLaVA-OneVision 7B) achieves 56.4%, outperforming Video-XL (7B) at 48.8%. This highlights the limitations of architecture-specific designs, which can quickly become outdated and fail to generalize to longer videos beyond those seen during fine-tuning.

• In contrast, rLiVS is training-free and model-agnostic. It leverages:

  1. LLM-informed token selection via cross-attention,
  2. Recurrent accumulation of selected tokens, and
  3. Text-based retrieval for long-term memory.

This modular design enables plug-and-play scalability to new models without retraining, while maintaining high efficiency.

To directly compare with Video-XL, we have included an evaluation on the task of video summarization using the MLVU validation split, we assessed the holistic video summarization score. Our model, rLiVS, achieved a score of 3.65, outperforming Video-XL (3.4). We will include a more detailed comparison with Video-XL in the revised version.

---

Illustration in Method Section

We kindly refer to Figure 1, which illustrates key components of our method:

• The left side of the figure shows the processing of the second short clip. It includes the historical tokens from the first short clip $S^1$, the full token sequence of the second short clip $X^2_V$, and the instruction $X^I$.
• It also visualizes the cross-attention between the generated caption and the tokens of $X^2_V$, which is used to select the top-K relevant tokens (e.g., $X^2_1$, $X^2_5$). These selected tokens are then accumulated into the historical context and passed forward for processing the third short clip.

The right side of the figure illustrates the question-answering phase, where the similarity between the query and the stored descriptions of each short clip is computed. The most relevant subset is then provided as input to the LLM (within the Video-LLM) for answering the question.

We also refer the reviewer to Algorithm 1, which outlines the main steps of our approach.

If the reviewer finds that Figure 1 and Algorithm 1 are insufficient, we would be happy to include an additional illustration in the revised version to further clarify the method.

---

Efficiency Not Reported?

We thank the reviewer for raising this point. We refer to Table 3, where we report latency, VRAM usage, and KV-cache requirements on both RVS-Ego and RVS-Movie, in line with prior work.

Our method achieves the lowest latency, requires no KV-caching, and uses significantly less VRAM compared to the previous state-of-the-art, ReKV [5], while also delivering superior performance.

---

Clarifying the Broader Applicability of rLiVS

We appreciate the reviewer’s observation regarding the scope of rLiVS. While our current evaluation focuses on long-term streaming video understanding and question answering in a training-free setting, we believe this does not imply a narrow application.

rLiVS is designed to enable streaming, never-ending video understanding, a key capability for real-world deployment of visual assistants and robotic systems. These systems often operate in dynamic environments where continuous perception and reasoning over historical visual context are essential.

We follow established benchmarks and protocols from prior work (e.g., [5]) to ensure fair and meaningful comparisons.

Regarding temporal video grounding, rLiVS can naturally identify the short clip most relevant to a given query. However, pinpointing exact timestamps is beyond the scope of streaming video understanding and not the focus of our current work.

As stated above we included the task of video summarization as an additional benchmark alongside video question answering. Using the MLVU validation split, we assessed the holistic video summarization score. Our model, rLViS, achieved a score of 3.65 compared to the base LLaVA-OneVision score of 3.57. These results demonstrate rLViS generalization capability across tasks.

In this light, we argue that rLiVS addresses a significant and practical challenge in video understanding, with design choices motivated by real-world deployment scenarios.

[1] Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding. Arxiv 2409.14485

We thank the reviewer for their constructive feedback and we are glad to read that the reviewer found our solution comprehensive. Below we answer the reviewer questions and we are happy to elaborate further in the discussion period.

Generalization Across Different Video-LLMs

To support our claim that rLiVS is LLM-agnostic, we have integrated our method with an additional video-language model, namely Qwen2.5 VL. Within the time and compute constraints of this rebuttal, we evaluated rLiVS with Qwen2.5 VL on the RVS-Movie long video understanding benchmark and we achieve 53% whereas the uniform baseline achieves 51% on Qwen2.5 VL. This confirms rLiVS effectiveness and compatibility across architectures.

---

FIFO Memory May suffer Information Drift

Our design intentionally uses FIFO memory as a short-term context buffer, allowing the Video-LLM to process the current clip with awareness of recent visual content. In parallel, we treat the captions of stored short clips as a form of long-term memory, enabling the model to answer questions about earlier events.

We agree that this approach may not fully capture the complete historical context. However, processing the full video history would introduce significant computational and latency overhead, and would require models trained to handle such long visual sequences. Our method offers a training-free, efficient alternative that leverages the LLM’s pretraining on long text to support question answering about past events, based on its understanding at the time each short clip was processed.

---

Possible Information Loss from High Compression Rate

We thank the reviewer for raising this important point regarding potential information loss due to our high compression rate. While a 95% reduction may seem aggressive, our design is motivated by the nature of long videos and the demands of online processing.

Rationale Behind High Compression

• Visual Redundancy in Long Videos:
  Long videos often contain significant redundancy. Our goal is not lossless compression, but rather intelligent filtering of less informative tokens.

• Trade-off in Online Processing:
  In real-time video understanding, there is an inherent trade-off between efficiency and information preservation. Some degree of information loss is expected—and acceptable—when optimizing for speed and responsiveness.

Flexibility of the Framework

• Our method is tunable: the compression ratio can be adjusted based on task requirements.
• For tasks requiring higher accuracy, one can retain more tokens at the cost of increased computation.

Empirical Support

• Appendix.6 Table 4:
  On a dataset with rich temporal and object interactions:
  • Retaining 12% of tokens yields no significant performance drop.
  • Our 6% token selection results in only a ~1% performance drop, despite the aggressive compression.

SportsQA Evaluation

We encountered delays in obtaining the SportsQA dataset, which prevented a full quantitative evaluation within the available time frame. However, we conducted a qualitative comparison between the responses of LLaVA-OneVision using 6% selected visual tokens and the same model using all visual tokens.

The comparison reveals that even with only 6% of selected tokens, the model retains the ability to capture fine-grained details comparable to the full-token baseline. Below are a few illustrative examples of different videos - from the subset of 'aerobic gymnastics":

• Full Prediction: "The video showcases a synchronized gymnastics routine performed by a group of athletes at the 10th European Championships for Aerobic Gymnastics".

  Selected Token Prediction: "The video is about a gymnastics routine performed by the Italian team at the 10th European Championships".

• Full Prediction: "The video showcases a rhythmic gymnastics performance by three athletes".

  Selected Token Prediction: "The video showcases a rhythmic gymnastics routine performed by three athletes".

• Full Prediction: "The video showcases a synchronized gymnastics routine performed by two athletes at the 5th Aerobic Gymnastics Asian Championships".

  Selected Token Prediction: "The video is about two gymnasts performing a synchronized routine at the 5th Aerobic Gymnastics Asian Championships.".

• Full Prediction: "The video is about a rhythmic gymnastics performance by two athletes at the 7th FIG Aerobics World Age Group Competitions in Incheon, Korea".

  Selected Token Prediction: "The video showcases a rhythmic gymnastics routine performed by two athletes.

These examples demonstrate that our method maintains video understanding close to the full-token baseline, even in scenarios characterized by rapid scene changes and multi-object interactions. This suggests that rLViS exhibits strong robustness in dynamic scenes, supporting its generalization capabilities across complex video understanding tasks.

Remarks

An interesting future direction could be adaptive compression, where the ratio is dynamically adjusted based on the visual richness of each short clip. However, this would require additional computation for richness estimation, which may conflict with our goal of efficiency in online settings.

Our fixed but tunable compression strategy strikes a practical balance between performance and efficiency, making it well-suited for real-time video understanding. We appreciate the reviewer’s suggestion and are happy to explore further improvements in future work.

---

Clarification on Figure 1:

We apologize for any lack of clarity in Figure 1. The figure illustrates the processing of the second short clip. It shows how the historical tokens from the first short clip $S^1$ are included in the context, alongside the full token sequence of the second short clip $X^2_V$ and the instruction $X^I$.

We also depict the cross-attention between the generated caption and the tokens of $X^2_V$, which are used to select the top-K relevant tokens (e.g., $X^2_1$, $X^2_5$). These selected tokens are then accumulated into the historical context and will be provided as input for processing the third short clip.

We will revise the figure and its caption to make these steps clearer.

We thank the reviewer for their constructive feedback and insightful suggestions. Below, we address the raised concerns in detail. We are happy to elaborate further during the discussion period.

---

Generalization Across Different Video-LLMs

To support our claim that rLiVS is Video-LLM-agnostic, we have integrated our method with an additional video-language model, namely Qwen2.5 VL. Within the time and compute constraints of this rebuttal, we evaluated rLiVS with Qwen2.5 VL on the RVS-Movie streaming long video understanding benchmark and we achieve 53% whereas the uniform baseline achieves 51% on Qwen2.5 VL. This confirms rLiVS effectiveness and compatibility across architectures.

---

Significance and Novelty of the Token Selection Strategy

We appreciate the reviewer's feedback regarding the foundation of our attention-based token selection strategy. We'd like to provide additional clarification on the mechanism and its task-adaptability.

Core Mechanism

Our method employs a flexible top-down attention mechanism, where the input instruction guides the token selection process. The pipeline operates as follows:

Instruction → Generated Caption/Response → Attention-Based Token Selection

Visual tokens are selected based on their attention scores during text generation, ensuring semantic relevance to the task (see below, Task Adaptability, for further analysis). This mechanism aligns with human memory principles, where attention is directed by task-specific goals.

Prior Work Explored Text-Image Cross Attention Scores

While attention scores have been previously used for token reduction [9, 19, 29], that was only intended to reduce tokens processed in later layers following the attention score of previous layers. Our approach is distinct in several ways:

• We extract cross-layer attention scores, identifying tokens most attended to across the model’s depth, with only a sparse subset (~6%) of tokens is retained.
• These selected tokens are recurrently reused to guide the processing of subsequent short clips, enabling continuity in reasoning with zero training overhead.

This strategy not only reduces computational cost but also maintains semantic coherence across video segments.

Task Adaptability

While our current experiments focus on caption-based selection for visual QA (since these tasks require comprehensive “all-around” video understanding), our framework is designed to be instruction-adaptive, allowing it to generalize across tasks by modifying the input instruction. For example:

• Person Tracking: "Track the movement of the person in the red shirt throughout this video sequence."
• Action Recognition: "Identify and describe the specific athletic movements performed."
• Object Detection: "Locate and describe all vehicles appearing in this scene."

Each instruction type produces distinct textual responses and attention patterns, guiding the selection of task-relevant tokens.

To qualitatively assess this, we compared token selection overlaps under different instructions (one focused on main object and the other on background) for two videos:

• Video 1: A girl with sunglasses is the main foreground.

  • Instruction: "Track the locations of the girl wearing the sunglasses in the video."
  • Overlap with generic prompt: 44%

• Video 2: Two people playing cards.

  • Instruction: "Locate and describe the objects appearing in the background of the video."
  • Overlap with generic prompt: 8%

• Generic Prompt used: "Describe what’s happening in the video." (consistent with what we used in the main experiments for captioning)

These results suggest that our token selection reflects the video-LLM’s attention under different task instructions. We will include these findings along with qualitative visualizations in the updated version to further support this analysis.

---

Caption-Based Retrieval Strategy

We thank the reviewer for this insightful question regarding our retrieval design choices. Below, we clarify both the rationale and limitations of our approach.

Adaptive Caption Generation for Retrieval

As discussed previously (in the context of token selection), the caption—or more generally, text—generation is adaptive based on the input instruction. This allows the model to emphasize semantic aspects that are most relevant for retrieval, depending on the task. We view captions as the LLM’s internal thoughts on what it has observed in a given short clip. When answering a query, we access these thoughts without needing to reprocess the corresponding visual tokens.

Limitations and Justification

We agree with the reviewer that captions inherently compress semantic-rich visual information into textual form. While this abstraction may omit certain details, our design is motivated by the following considerations:

• Current VLM Architecture Constraints
  Existing video-LLMs struggle with effective visual-textual alignment for retrieval tasks (as shown in Figure.1 Appendix.6).

• Computational Efficiency
  Alternative approaches—such as using CLIP features for visual-text alignment—introduce significant computational overhead, which conflicts with our goal of efficient online video understanding.

We acknowledge that incorporating visual information directly into retrieval could offer additional benefits. However, our current text-based approach achieves strong performance within existing architectural and efficiency constraints, setting a solid baseline for future research. We consider this an important direction for future work and appreciate the reviewer’s suggestion.

---

Limited In-Depth Analysis

We would like to highlight several ablation studies and analyses that support the design choices in our method:

• Token Selection Strategy (Table.2):
  We compare our attention-based selection with several alternatives, including Uniform Sampling, Mean Pooling, K-Means, and Full Token Usage. Our method achieves a significant improvement over these baselines, with only a ~1% drop in performance compared to using all tokens, while retaining just 6% of the visual tokens.

• Effect of Recurrency (Table.4):
  We ablate the impact of our recurrent mechanism and observe a 3–4% performance gain, demonstrating its importance for maintaining temporal consistency across short clips.

• Retrieved Modality (Table.5):
  We analyze the effect of the retrieval modality and find that textual retrieval outperforms visual alternatives, supporting our choice of caption-based retrieval.

• Layer Selection Strategy (Appendix.2, Table.1):
  We evaluate the impact of selecting attention scores from different layers. Our cross-layer aggregation strategy provides robust and stable performance, comparable to using all layers. In contrast, selecting from localized regions (e.g., early, middle, or late layers) results in less stable outcomes, and using a single layer leads to significantly degraded performance.

• Token Retention Rate (Appendix.5 Table.4):
  We show that retaining only 12% of the visual tokens maintains performance without degradation, highlighting the efficiency of our selection mechanism.

• Additional Ablations:

  • Retrieval strategies and selection mechanisms are discussed in Appendix.6.
  • Context length is ablated in Appendix.4.
  • Attention aggregation strategies are analyzed in Appendix.3.

We are happy to provide any additional in-depth analysis upon the reviewer’s request.

We appreciate the reviewer's engagement and address the remaining concerns with specific clarification:

1. Theoretical Foundation - Human Memory Principles:

Our approach is grounded in well-established neuroscience principles of human memory and attention:

• Attention-driven memory consolidation: Studies by [1] show that attention selectively consolidates relevant information for future processing, paralleling our selective token retention. Specifically in [1] the authors point that:

"...First, memory has a limited capacity, and thus attention determines what will be encoded...": this is exactly the principle that we aim to emulate by selecting visual tokens based on attention of the model to them, that will serve as memory for further processing.

"...Second, memory from past experience guides what should be attended...": this serves as our basis for the recurrent processing; the selected past visual tokens acting as memory are passed in context along with the current (new) visual tokens, thus letting memory drive what the model will attend from these new visual tokens (current short clip).

• Recurrent processing in visual understanding: [2] demonstrates that visual understanding involves both feedforward and recurrent processing, which directly informs our recurrent architecture design.

---

2. Clarification on Caption Usage and effectiveness:

We first emphasize that captions are used for answering questions, not for video processing. Selected visual tokens serve as past visual information (memory) for recurrent video processing, ensuring continuity during the processing phase.

Regarding caption effectiveness when answering, our empirical validation resulted in the points below:

• Superiority over previous ideas in the literature: Our method outperforms ReKV (which uses model's own KV hidden states) and Flash-VStream (which uses a raw visual memory) across the long streaming video benchmarks.

• Comprehensive ablation studies: We demonstrate that captions consistently outperform visual tokens and caption-visual tokens combinations for question answering and retrieval, validating captions as effective and compact short-clip representations.

• State-of-the-art results: Our SOTA performance on streaming benchmarks RVS-Ego and RVS-Movie provides strong empirical evidence that our caption-based answering strategy is indeed effective for this task domain, while being also the most efficient strategy in terms of latency, a critical aspect of real time video understanding.

[1] https://pubmed.ncbi.nlm.nih.gov/17379501/, [2] https://pubmed.ncbi.nlm.nih.gov/11074267/

---

We thank the reviewer for their continued engagement in this discussion, which has enabled us to provide further clarification on the human-memory principles that inspire our work. We respectfully ask the reviewer to consider these theoretical foundations and empirical validations in their final assessment. We commit to incorporating these clarifications into the final version of our paper to ensure the theoretical grounding is explicitly documented for future readers.

Dear Reviewer,

We have provided supporting evidence from neuroscience, showing the grounding of our claim on inspiration from human memory formation and the interplay between attention and memory (a central aspect of our method). We also provided empirical evidence on how selected tokens affect the processing of future clips (i.e. how memory drives attention and vice-versa). Furthermore, we showed the superiority of caption-based retrieval and answering under our training free setting, resulting in strong performance and efficiency across benchmarks. We kindly ask the reviewer to check our additional input and share with us their thoughts or further questions, as the discussion window is closing.

We thank the reviewer for their constructive feedback and are glad that you found our paper well-written with a significant contribution. Below, we address your request in detail.

---

Experiments with Additional Video-LLM

To support our claim that rLiVS is Video-LLM-agnostic, we have integrated our method with an additional video-language model, namely Qwen2.5 VL. Within the time and compute constraints of this rebuttal, we evaluated rLiVS with Qwen2.5 VL on the RVS-Movie streaming long video understanding benchmark and we achieve 53% whereas the uniform baseline achieves 51% on Qwen2.5 VL. This confirms rLiVS effectiveness and compatibility across architectures.

---

Applicability to Proprietary Models

While rLiVS is designed to be model-agnostic, its current implementation relies on access to internal hidden states, namely the cross layers attention scores. This requirement limits its direct applicability to proprietary models, which typically restrict access to such internals and do not allow manipulation of visual tokens.

---

Additional Analysis

We would like to highlight several ablation studies and analyses that support the design choices in our method:

• Token Selection Strategy (Table.2):
  We compare our attention-based selection with several alternatives, including Uniform Sampling, Mean Pooling, K-Means, and Full Token Usage. Our method achieves a significant improvement over these baselines, with only a ~1% drop in performance compared to using all tokens, while retaining just 6% of the visual tokens.

• Effect of Recurrency (Table.4):
  We ablate the impact of our recurrent mechanism and observe a 3–4% performance gain, demonstrating its importance for maintaining temporal consistency across short clips.

• Retrieved Modality (Table.5):
  We analyze the effect of the retrieval modality and find that textual retrieval outperforms visual alternatives, supporting our choice of caption-based retrieval.

• Layer Selection Strategy (Appendix.2, Table.1):
  We evaluate the impact of selecting attention scores from different layers. Our cross-layer aggregation strategy provides robust and stable performance, comparable to using all layers. In contrast, selecting from localized regions (e.g., early, middle, or late layers) results in less stable outcomes, and using a single layer leads to significantly degraded performance.

• Token Retention Rate (Appendix.5 Table.4):
  We show that retaining only 12% of the visual tokens maintains performance without degradation, highlighting the efficiency of our selection mechanism.

• Additional Ablations:

  • Retrieval strategies and selection mechanisms are discussed in Appendix.6.
  • Context length is ablated in Appendix.4.
  • Attention aggregation strategies are analyzed in Appendix.3.

We are happy to provide any additional analysis upon the reviewer’s request.