StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant

We thank Reviewer neQ8 for the positive evaluation and detailed feedback. We are encouraged by the acknowledgment of our clear motivation, the reasonableness of our modeling choices, and the extensive experimental validation. We address your concerns below.

(1) Clarification on how early frames are compressed.
We appreciate the opportunity to clarify the round-decayed compression mechanism. When the total length of visual tokens exceeds the predefined $**MaxLen**$, we apply recursive average pooling starting from the earliest round. This process ensures that compression proceeds round-by-round, prioritizing the preservation of more recent frames.

To illustrate this, consider a simple example where $**MaxLen**$ = 200, each frame contributes 64 tokens (i.e., tokens-per-frame = 64), and the current memory buffer consists of the following sequence:

Memory buffer: frame$_1$, frame$_2$, frame$_3$, Q$_1$, A$_1$, frame$_4$, frame$_5$, Q$_2$

Here, frame$_1$–frame$_3$ belong to the first round, and frame$_4$–frame$_5$ belong to the second round. The total number of visual tokens is 64 × 5 = 320, which exceeds $**MaxLen**$ = 200. Thus, at least 320 - 200 = 120 tokens must be removed. Since our average pooling reduces frame tokens in units of tokens-per-frame = 64, we round up the target to 128 tokens. To achieve this, we apply average pooling over the first three frames in the first round, collapsing 64 × 3 = 192 tokens into 64 tokens by summing them point-wise and dividing by 3. This reduces the total by 192 - 64 = 128 tokens, bringing the sequence within the $**MaxLen**$ constraint, without compressing visual tokens in the 2nd round.

Now, consider a more constrained case with $**MaxLen**$ = 150. We would then need to remove 320 - 150 = 170 tokens, rounded up to 192. As before, pooling the first round (frame$_1$–frame$_3$) reduces 128 tokens. However, this is insufficient, so we continue to the second round, pooling frame$_4$ and frame$_5$ into a single frame representation, further reducing 64 tokens. This results in a total reduction of 128 + 64 = 192 tokens, satisfying the compression requirement.

This round-wise, progressive pooling strategy ensures that older rounds are compressed more aggressively while preserving recent, fine-grained visual details, aligning with our round-decay design philosophy. This process is also described with the pseudo code in Appendix H, and we will add more explanations in the final version.

(2) Justification for "recent frames are more informative"
We appreciate the reviewer’s insightful question. The design choice to preferentially retain recent frames is primarily motivated by the nature of streaming video understanding, where decisions are made incrementally and based on the most recent context. In such scenarios, recent visual information is often more indicative of the current event or interaction state. For instance, in tasks like action detection or dense captioning, models must react to what is happening now rather than rely on distant historical context.

To validate this empirically, we conducted compression ablation studies as reported in Table 4, where we compared our round-decayed compression with alternative strategies, including uniform pooling and FIFO truncation. The round-decay method, which progressively compresses earlier rounds while retaining fine-grained recent inputs, consistently outperformed other approaches across all streaming tasks. This confirms that recent frames indeed carry more task-relevant information in the streaming setup.

(3) Necessity of Using an MLLM for the Activation Model and the Choice of P%.
Our goal is not merely to identify informative frames, but to decide when the model has accumulated enough multimodal evidence to confidently produce a response. This is inherently a causal, streaming process, where the model incrementally observes interleaved video-text inputs over time and must decide whether sufficient understanding has been achieved at each moment. An MLLM is well-suited for this role, as it is trained to perform causal prediction (i.e., next token prediction) over sequences of image/video and text. This makes it fundamentally different from offline frame importance estimators (e.g., [1]), which do not model sequential decision-making in a streaming context. Additionally, MLLM is pre-trained on large-scale multimodal datasets and thus benefits from strong generalization across diverse visual concepts and task formats. This enables the model to adapt to a wide variety of query types and video domains, including those unseen during activation model fine-tuning.

Regarding the range of P%, we dynamically sample values from 0–50% to determine which portion of the video contains answer-relevant evidence. This strategy encourages the model to generate responses only after observing sufficient context(i.e., the first (1–P)% of the video). On average, this means the model responds after seeing around 75% of the video content. This reflects a realistic streaming behavior, where answers are typically derived from progressively revealed context, and discourages too early activations.

(4) Counting task underperformance.
We agree with the reviewer that the performance drop likely stems from the fact that counting tasks inherently require high temporal fidelity and full preservation of fine-grained visual details across time, while our round-decayed compression strategy is designed to prioritize temporal efficiency and compact historical context, which can occasionally lead to information loss in dense, detail-sensitive tasks like counting. To mitigate this, we plan to augment Stream-IT with more counting-style tasks in future iterations, encouraging the model to retain finer temporal granularity when necessary.

[1] Revisiting the “Video” in Video-Language Understanding, CVPR 2022

We thank Reviewer ySNS for the encouraging feedback. We are pleased that the reviewer found the writing clear, the experimental results looking good, and the effectiveness of our constructed dataset. Below, we address your comments.

(1) Choice of LLaVA-OV-0.5B as the activation model.
We chose LLaVA-OV-0.5B for its lightweight architecture and fast inference speed, making it well-suited as a concurrent decision module that introduces minimal computational overhead. We agree with the reviewer that the activation threshold $\alpha$ plays a role in response frequency. In practice, we empirically set $\alpha$ within the range of [0.3, 0.4], which we found to strike a good balance between precision and responsiveness across most tasks. This design also enables flexible control depending on downstream application requirements.

(2) Performance drop in Table 2 (Offline Tasks).
Thank you for pointing this out. The observed performance drops (e.g., -2.0 on LongVideoBench and -2.6 on MVBench) are isolated cases out of nearly 21 benchmark-model combinations. We will clarify these exceptions and emphasize the broader non-destructive trend. We also commit to using more accurate and objective language in the final version of the paper.

(3) Impact of the P% heuristic in activation model.
We appreciate the reviewer’s thoughtful observation. In our design, we dynamically sample P% between 0–50% to simulate varied activation delays. This range encourages the model to avoid generating premature responses in the very early portion of a video, where insufficient context would make meaningful responses infeasible. The model is thus trained to wait until it has observed at least (1–P)% of the video before considering a response. On average, this means the model typically responds after viewing around 75% of the video, which aligns with the intuition that robust understanding often requires seeing most of the content. Moreover, dynamic sampling introduces beneficial randomness, helping the model avoid learning shortcut patterns, such as always responding at fixed positions and promotes stronger generalization across diverse scenarios.

To further validate this, we compare two settings on ET-Bench: one with a fixed P = 25% and another with P uniformly sampled from 0–50%:

As shown, dynamic sampling of P% leads to improved performance and generalization across tasks, supporting the effectiveness of this design choice.

We thank the reviewer for the thoughtful follow-up.

To further support our efficiency claim, we conducted a breakdown of the runtime and memory usage of the two key components in our framework: (1) the main Video-LLM (Qwen2-VL-7B) and (2) the activation model (LLaVA-OV-0.5B vs. LLaVA-OV-7B). All experiments were performed on the subset of ET-Bench using 1 FPS input with a single NVIDIA A6000 (48GB) GPU. We report the average inference time and memory usage:

As shown, compared with the main Video-LLM (Qwen2-VL-7B), the lightweight activation model (LLaVA-OV-0.5B) only contributes less than 12.5% of the runtime (0.35s vs 2.79s) and 9.4% of the memory usage (1.67GB vs 17.7GB), validating our design choice of using LLaVA-OV-0.5B for efficiency. In contrast, replacing it with a larger backbone (LLaVA-OV-7B) results in approximately a 9.5× increase in memory usage and significantly longer inference time, making it impractical for real-time streaming scenarios.

We appreciate the reviewer’s insightful suggestion, which helped strengthen the justification of our architectural choices. We will incorporate all of these into our paper revision.

We thank Reviewer ioHp for the thoughtful review and valuable comments. We are glad that the reviewer acknowledged StreamBridge's effort to fill a critical gap in adapting offline Video-LLMs to real-time streaming scenarios, and appreciated our extensive benchmark coverage and the introduction of the Stream-IT dataset. We address your suggestions below.

(1) Novelty of memory buffer and round-decayed compression

We clarify that while memory mechanisms are not entirely new (e.g., LSTR), our method is specifically tailored to multi-turn streaming scenarios with a fixed token budget. Unlike LSTR, which maintains separate long- and short-term memories limited to only visual tokens, our memory buffer stores interleaved video-text tokens, enabling richer and more coherent contextual understanding and multi-turn dialogues for multimodal LLMs.

Moreover, our round-decayed token compression is a lightweight and training-free strategy for inference only. In contrast, LSTR introduces an additional LSTR Decoder that requires training to extract information from long-term memories via cross-attention. Our method avoids such complexity, instead merging older visual tokens through simple average pooling while preserving recent context, offering both efficiency and adaptability without additional training overhead.

(2) Assumption that recent frames are more relevant

We acknowledge that this assumption may not always hold in non-streaming scenarios where earlier frames can contain critical information. To address this, our round-decayed compression strategy does not discard older content; instead, it pools earlier video tokens to retain coarse-grained historical context. This design is validated in Table 4 and discussed in Lines 294–303. Furthermore, the memory budget can be adjusted via the MaxLen parameter (defined in Line 137). For instance, when MaxLen = 32k, the model can store up to 15 minutes of video at 1 FPS without requiring additional compression, ensuring full retention of video information in low-latency or short-form scenarios.

Moreover, standard offline video benchmarks such as VideoMME and LongVideoBench, whose questions are posed at the end of the video, inherently include scenarios where "the question is intentionally delayed by varying time intervals, simulating scenarios in which a query relates to a past segment rather than the most recent visual input." As shown in Table 2, our approach maintains comparable performance on these benchmarks, demonstrating StreamBridge's generalization that retains relevant historical information and is not biased toward only recent context.

(3) Inconsistent language in result interpretation

Thank you for pointing this out. We will revise the phrasing to ensure consistency in describing all performance changes across models.

(4) Activation model's ablation

We agree and appreciate the suggestion. In our current setting, we use 16 tokens per frame for the activation model. Below, we provide additional quantitative results on ET-Bench, exploring the impact of token granularity. We report only the F1 score, which reflects the precision of the activation model’s response timing:

Table: Activation Token Granularity on ET-Bench

As shown, increasing the number of tokens per frame does not significantly improve performance. This is expected, as the activation model is designed to perform binary classification (i.e., whether to trigger a response), which requires less fine-grained semantic detail than generative tasks. Our default setting of a small MLLM and 16 tokens per frame achieves a good balance between efficiency and accuracy.

(5) Related work coverage

Thank you for the references. We will expand the related work section to include [1, 2, 3], and incorporate a discussion of them to provide a more complete context for our work.

[1] Qian, Rui, et al. "Streaming long video understanding with large language models." NeurIPS 2024.

[2] Di, Shangzhe, et al. "Streaming video question-answering with in-context video kv-cache retrieval." arXiv:2503.00540, 2025.

[3] Xiong, Haomiao, et al. "Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge." arXiv:2501.13468, 2025.

Impact of using a larger activation model

We appreciate the reviewer’s follow-up. In addition to varying the token granularity, we have also conducted experiments using a larger activation model (LLaVA-OV-7B) to study the effect of model scale. We compare its performance against our default lightweight model (LLaVA-OV-0.5B) on ET-Bench:

While LLaVA-OV-7B offers a performance gain (about 2.1 F1 improvement on average), we find that the benefit does not justify the significantly increased computational cost (1.67GB vs 15.9GB), particularly in streaming scenarios. Hence, we opt for the smaller model as a better trade-off between efficiency and effectiveness.

Discussion of Related Work

Thank you again for the suggestion. Below is a brief discussion on how our work relates to the suggested references:

To address the challenge of long-context understanding in streaming video, several memory and retrieval mechanisms have been proposed. For instance, ReKV [1] introduces a training-free framework that stores and retrieves the Key-Value (KV) caches of processed frames, enabling offline models to answer user queries efficiently by reloading only the most relevant context. Besides, VideoStreaming [2] employs a memory-propagated encoding architecture where a condensed representation of the preceding clip serves as historical context for encoding the next, combined with an adaptive selection of memories for question-answering. Moreover, StreamChat [3] proposes a hierarchical memory system comprising short-term, long-term, and dialogue components to facilitate complex streaming interactions, and also contributes the StreamBench benchmark for evaluating diverse streaming scenarios. While these methods effectively advance long-context retention for reactive question-answering, StreamBridge differs by introducing a round-decayed compression strategy specifically tailored for multi-turn real-time interactions, which efficiently prunes redundant historical tokens while preserving recent context with high fidelity. Moreover, StreamBridge introduces a decoupled, lightweight activation model. This plug-and-play component operates in parallel with the main Video-LLM, enabling continuous proactive responses. These designs, supported by our dedicated Stream-IT dataset, effectively transform general-purpose offline models into versatile and proactive streaming assistants without compromising their core performance.

We will incorporate these discussions in the final version to better position our work within the broader landscape of streaming video understanding.

[1] Di, Shangzhe, et al. "Streaming video question-answering with in-context video kv-cache retrieval."
[2] Qian, Rui, et al. "Streaming long video understanding with large language models."
[3] Xiong, Haomiao, et al. "Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge."

We sincerely thank Reviewer Qdek for the insightful and constructive feedback. We are particularly encouraged by the recognition of our modular and elegant StreamBridge framework, the contribution of the Stream-IT dataset, and the comprehensive empirical evaluation. We address your concerns below.

(1) Generalization of the activation model

We agree that generalization is a critical aspect of the activation model. To encourage robustness, our model is trained on five diverse tasks (e.g., dense captioning, step localization, action detection, etc.) using a variety of prompt templates and a dynamic P% sampling strategy, which enables the model to learn activation across varied temporal patterns.

To further evaluate its generalization ability, we report results on NExT-GQA [1], a grounded VideoQA benchmark characterized by diverse, natural question prompts with annotated timestamps. Unlike template-based tasks like dense video captioning, NExT-GQA requires the model to trigger a response precisely when the visual evidence for answering the question appears, presenting a strong challenge for generalization.

Table: Activation Model Generalization on NExT-GQA

Here in the table, Absolute denotes the average time gap (in seconds) between the activation timestamp and the annotated ground-truth response timestamp. Relative refers to this gap as a percentage of the total video duration. These results demonstrate that the activation model can proactively answer the question within a delay of 3.24 seconds or 7.7% of the video duration on average, and generalize well to unseen videos and diverse, open-ended questions, even in scenarios not encountered during training.

[1] Xiao, Junbin, et al. "Can I trust your answer? Visually grounded video question answering." CVPR 2024

---

(2) Direct efficiency comparison

We appreciate the reviewer’s suggestion and now provide a direct inference latency comparison between Qwen2-VL-StreamBridge (ours), VideoLLM-Online, Flash-VStream, and Dispider. All models are evaluated on the ActivityNet dense video captioning task using the same hardware (a single NVIDIA A6000-48G GPU) with 1 FPS video sampling:

Table: Latency and Accuracy Comparison

From the results, we observe that our framework is more efficient than Dispider, while slower than VideoLLM-Online and Flash-VStream. This trade-off is expected: both VideoLLM-Online and Flash-VStream rely on highly aggressive compression schemes (1–10 tokens per frame), which significantly reduce latency but also lead to substantial information loss. In contrast, StreamBridge maintains a richer representation per frame and achieves significantly stronger results on challenging benchmarks such as OVO-Bench and VideoMME, highlighting a better balance between efficiency and accuracy.