StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

Thank you for the insightful feedback on our paper. Below, we address each of your concerns point-by-point:

Response 1: Understanding on "remembering earlier moments."

• In the Real-Time Visual Understanding (RTVU) tasks, we considered not only "current moments" tasks but also "long dependency" tasks, such as Counting (CT) and Causal Reasoning (CR). These tasks require models to capture information over extended time windows, avoiding the possibility of answering questions based solely on local information. To verify this, we conducted evaluations for all models in the main experiment while restricting the context to only the first 60 seconds preceding the query time. The results, shown in the table below, reveal that many models suffered performance degradation on tasks like Counting and Causal Reasoning, while their performance on "current moments" tasks generally improved. This indicates a trade-off between focusing on "current moments" and addressing "long dependency" in the current models for streaming video understanding. It also demonstrates that the RTVU tasks in StreamingBench are capable of evaluating both types of capabilities simultaneously. In future versions of StreamingBench, we will continue to optimize towards this direction.

Table1 : Comparison between experiments retaining only the last 60 seconds of video context before the query time and the main experiment (retaining all context before the query time). Negative values indicate a performance drop caused by limiting the context to the last 60 seconds.

Response 2: For models that do not support audio input

• In the current Omni-Source Understanding experiments, models that do not support audio input only rely on video information to respond to Omni-Source Understanding tasks. For comparison, we conducted an additional evaluation using Video + ASR information as input. The detailed results are shown in General response 1. In summary, adding ASR information enhances performance but still does not match the capabilities of Gemini 1.5 Pro in Omni-Source Understanding tasks. This highlights the necessity of advancing existing MLLMs to natively support audio input in streaming scenarios.

Response 3: Discussion with LongVideoBench.

• Thank you for your valuable suggestion. The "referring reasoning" part of LongVideoBench is indeed similar to the Omni-Source Understanding tasks in terms of requiring an understanding of concurrent video and audio information. The difference lies in that StreamingBench focuses more on streaming scenarios, whereas LongVideoBench emphasizes long-video scenarios, with differences in the specific types of tasks as well. We will include a discussion on LongVideoBench in the revised version of our paper.

Thank you for your detailed and insightful feedbacks. Below, we respond to each of your points in detail:

Response to Q1: Extension of the benchmark for additional tasks

• Thank you for your valuable suggestions! We recognize that there are also other interesting and important streaming video understanding tasks such as dense streaming video captioning and streaming video grounding. We will explore these tasks in the future and try to add them in the next version of StreamingBench.

Response to Q2: For models that do not support audio input

• In the current Omni-Source Understanding experiments, models that do not support audio input only rely on video information to respond to Omni-Source Understanding tasks. As a comparison, we conducted an additional evaluation using Video + ASR information as input. The detailed results are shown in General response 1. The results indicate that while the models demonstrated improved performance on the Omni-Source Understanding tasks with additional ASR information, they still fell significantly short of the performance achieved by Gemini 1.5 pro in omni-source understanding tasks. This highlights the necessity of advancing existing MLLMs to natively support audio input in streaming scenarios.

Response to Q3: Memory Constraint

• In our current experiments, we adhere to the standard settings for each model. While we recognize that memory constraints are indeed crucial, addressing them is not the primary focus of our work at this stage. However, even though we did not enforce a fixed memory limit across all models, the memory usage for most models remains within a manageable range (e.g., for 7B models, it does not exceed the capacity of a single A100 80G GPU).

• Additionally, as a preliminary effort to evaluate the impact of memory usage on model performance, we sampled 100 QA data and analyzed the average number of input tokens processed by several models under standard settings. From the results, it seems that for current video MLLMs, processing more input tokens does not necessarily lead to better performance. The specific results are as follows:

  Table 1: Average Input Tokens and Model Performance

  Model (w/o ASR)	Avg. Input Token	All(RTVU)	All(OSU)	All(CU)	Overall
  LLaVA-OneVision-7B	6426.91	71.12	38.40	31.63	56.16
  Qwen2-VL-7B	56081.45	69.04	34.90	30.37	53.91
  MiniCPM-V 2.6-8B	12696.77	67.44	35.00	34.21	53.71
  LLaVA-Next-Video-32B	5029.79	66.96	34.90	30.04	52.64
  InternVL-8B	8599.95	63.72	35.80	30.59	51.06
  Kangaroo	8420.45	64.60	34.20	29.32	50.97
  LongVA	18585.91	59.96	35.40	29.34	48.55
  VILA-1.5	3319.45	52.32	33.10	27.24	43.18
  Video-CCAM	1179.99	53.96	29.70	21.83	42.34
  Video-LLaMA2	1124.50	49.52	32.40	22.08	40.43

• With more video MLLMs emerging for streaming video understanding, we will include memory usage and constraints as a consideration in future work. Thank you again for your suggestion!

Response to W1: Impact of model size

• Thank you for your suggestion. We conducted additional evaluations on models of various sizes, including LLaVA-OneVision 0.5B & 72B, Qwen2-VL 2B, InternVL-V2 2B & 72B, and VideoLLaMA2 72B. Overall, stronger LLM backbones proved to be beneficial. Due to resource and time constraints, we were unable to complete evaluations for some models. We will update the results in the paper once these evaluations are completed.

  Table 2: Impact of Model Size

  Models	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	All（RTVU)	ER	SCU	SD	MA	All(OSU)	ACU	MCU	SQA	PO	All(CU)	Overall
  LLaVA-OneVision-0.5B	65.12	61.72	65.3	65.36	56.52	58.57	57.41	53.25	55.52	33.16	58.28	37.20	24.40	29.20	34.00	31.20	29.20	30.40	28.40	9.80	28.09	46.36
  LLaVA-OneVision-7B	80.38	72.22	76.03	80.72	68.32	71.65	67.59	65.45	65.72	45.08	71.12	40.80	37.20	33.60	44.80	38.40	35.60	36.00	27.27	11.76	31.63	56.16
  LLaVA-OneVision-72B	79.56	85.94	82.97	83.28	78.75	74.77	81.48	67.87	73.37	51.3	75.98	44.80	26.80	43.60	55.20	42.60	40.40	44.80	36.40	21.57	39.33	61.39
  Qwen2-VL-2B	61.85	68.75	60.57	63.73	56.52	58.26	60.19	50	50.42	37.82	56.76	36.40	23.20	27.20	36.40	30.80	24.40	23.20	40.80	5.88	27.96	45.36
  Qwen2-VL-7B	75.20	82.81	73.19	77.45	68.32	71.03	72.22	61.19	61.47	46.11	69.04	41.20	22.00	32.80	43.60	34.90	31.20	26.00	39.60	1.96	30.37	53.91
  Qwen2-VL-72B	75.48	82.03	79.18	79.41	74.53	71.03	76.85	64.63	69.69	49.22	72.28	39.20	36.00	33.60	53.60	40.60	31.60	38.40	40.80	9.80	35.21	58.01
  VideoLLaMA2-7B	56.40	57.81	65.30	62.75	64.60	43.61	39.81	42.68	45.61	35.23	49.52	30.40	32.40	30.40	36.00	32.40	24.80	26.80	18.67	1.96	22.08	40.43
  VideoLLaMA2-72B	64.85	75.78	77.29	72.79	66.25	58.88	59.26	53.66	58.07	48.19	63.70	43.20	24.40	35.20	49.60	38.10	38.80	30.00	36.80	13.73	33.83	52.19
  InternVL-V2-2B	56.95	50.78	58.04	64.71	57.14	52.65	50.93	44.72	46.46	47.67	53.52	32.00	30.00	29.20	43.60	33.70	32.40	26.80	37.20	13.73	30.96	44.71
  InternVL-V2-7B	68.12	60.94	69.40	77.12	67.70	62.93	59.26	53.25	54.96	56.48	63.72	37.60	26.40	37.20	42.00	35.80	32.00	31.20	32.32	11.76	30.59	51.06

Response to W2: Adding more models for evaluation

• Thanks for your suggestions. We have evaluated LongVILA 8B, LongLLaVA 9B, and Oryx 7B. The evaluation results are shown in the following table.

  Table 3: Results on more models

  Models	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	All(RTVU)	ER	SCU	SD	MA	All(OSU)	ACU	MCU	SQA	PO	All(CU)	Overall
  LongVILA - 9B	67.85	49.22	60.25	71.57	56.52	56.7	40.74	56.5	57.51	41.97	58.48	37.20	25.20	30.40	40.40	33.30	29.20	28.80	25.60	7.84	26.59	46.60
  Oryx - 7B	51.50	48.44	57.1	52.5	59.63	41.74	37.04	48.78	48.16	13.99	47.20	31.60	23.20	19.20	40.80	28.70	29.60	25.20	33.20	5.88	27.84	39.29
  LongLLaVA - 8B	27.32	43.75	39.87	36.39	30.63	28.35	25.92	29.8	26.35	39.38	32.18	32.40	24.80	25.60	28.00	27.70	24.00	24.40	33.20	7.84	25.97	29.98

Thank you very much for your response. Most of my concerns have been addressed, and your experiments appear more thorough now. I appreciate the effort. However, I’m still curious about your more detailed plans to expand your benchmark to cover a broader range of streaming video understanding, as I genuinely believe this is a highly promising and significant area of research that extends far beyond QA tasks. Of course, there’s no need to provide detailed plans here immediately. Just look forward to seeing your future work.

Thank you for your insightful feedback. Below, we address each of your points in detail:

Response 1: Methodology for collecting 900 videos from YouTube

• Thank you for pointing this out. Briefly, we selected videos based on the predefined task definitions and characteristics of 18 tasks. When selecting videos, we conducted extensive searches on YouTube to find videos suitable for annotating these tasks according to specific attributes and requirements. For instance, when looking for videos suitable for the Proactive Output task, we required the videos to contain abundant explicit condition-triggering information. When searching for videos suitable for the Source Discrimination task, we needed videos featuring multiple people, where more than one person spoke before the question was asked. For videos suitable for the Misleading Contextual Understanding task, we required the videos to focus on a specific scene or event, with the scene and event undergoing highly dynamic changes throughout the video. Additionally, to ensure diversity in video content, we selected videos from various fields and perspectives. To find these 900 videos, we approximately reviewed a total of 6,000 videos, with a detailed screening ratio of about 15%. We will supplement a detailed explanation in the appendix in the revised version of the paper.

Response 2: For SFT dataset design

• Thank you for your valuable suggestions. We fully agree that constructing an SFT dataset specifically designed for streaming video understanding (including the Real-Time Video Understanding and Proactive Output tasks as you mentioned) is highly important. However, the primary focus of this work remains on evaluating the streaming understanding capabilities of models. In future work, we will make efforts to develop such an SFT dataset.

Response 3: For models that do not support audio input

• In the current Omni-Source Understanding experiments, models that do not support audio input only rely on video information to respond to Omni-Source Understanding tasks. As a comparison, we conducted an additional evaluation using Video + ASR information as input. Please refer to the General response 1 for details. The results show that while adding ASR information improves the performance of these models on Omni-Source Understanding tasks, it remains significantly lower than that of Gemini 1.5 Pro. This highlights the importance of advancing current MLLMs to natively support audio input in order to meet the demands of omni-source understanding capabilities for understanding streaming videos.

Response 1: Difference between StreamingBench and offline video benchmarks

• StreamingBench is designed to evaluate the streaming understanding capabilities for all video models, not limited to streaming video models exclusively. Existing standard video LLMs can indeed be evaluated on StreamingBench through certain processing methods, which can also be seen as a way to simulate streaming models with standard video LLMs. As described in Section 4.1, this is how we evaluate the majority of current video LLMs.

• We emphasize that the primary distinction between StreamingBench and existing offline benchmarks is the capabilities being assessed. Existing offline benchmarks cannot comprehensively evaluate the capabilities of the models in streaming scenarios, which is the core design principle of StreamingBench. Specifically, this includes:

  • Traditional offline benchmarks typically pose questions about entire videos, lacking the focus on real-time understanding. In contrast, StreamingBench incorporates tasks designed for Real-Time Visual Understanding questions like, "What words are currently shown?" Here, answers depend on the exact moment of the query, a crucial aspect that offline benchmarks overlook.
  • Offline benchmarks often lack consideration for synchronized visual and audio inputs in streaming contexts. When answering such questions, models must consider their synchronization, such as "What is happening in the video when [sound] is made?". The Omni-Source Understanding tasks in StreamingBench are tailored to this.
  • Offline benchmarks typically fail to evaluate the contextual interaction and understanding capabilities in streaming scenarios. StreamingBench includes dedicated tasks for this purpose. For instance, in the Proactive Output (PO) task, where a model must proactively respond at the precise moment a condition is met (e.g., "When the red diamond nine appears on the table for the first time, output 'Red diamond nine.'"). Similarly, the Sequential Question Answering (SQA) task requires models to integrate interaction history with video content to answer questions like, "What actions did the two individuals mentioned in the previous question just take?" These tasks go beyond the static Q&A format of offline benchmarks.

• In terms of evaluation methods, the first two types of tasks can be directly assessed for standard video LLMs by feeding all video content up to the question's timestamp. For the third type of tasks, however, evaluation can only be conducted through certain compromised adaptations that standard offline models can handle. For example, in the SQA task, the context and Q&A are provided in textual form (rather than interleaved with the corresponding video segments). Similarly, for the PO task, we rely on polling the model to determine whether a response is needed, rather than allowing the offline model to autonomously identify the appropriate moment to respond, as these models lack such capabilities.

• In terms of assessed capabilities, StreamingBench provides a comprehensive evaluation of streaming video understanding, offering significant values compared to existing benchmarks.

Response 2: Compared to redundant information in long-video benchmarks

• In long video benchmarks, redundant information primarily arises from the excessive length of the videos. In contrast, in StreamingBench, redundant information does not stem from video length but rather from the nature of streaming scenarios. The tasks associated with this aspect are Misleading Context Understanding (MCU) and Anomaly Context Understanding (ACU):

  • The MCU task focuses on the interference caused by repetitive scenes in answering questions, where redundant information originates from repeated scenes. For example, in a card game video, answering the question "How many cards are on the table now?" can be challenging because the camera consistently captures the table from a fixed angle, leading to repetitive footage that affects the ability to answer accurately. This is common in streaming scenarios such as video surveillance
  • The ACU task, on the other hand, emphasizes the identification of anomalous events in everyday scenes, where redundant information refers to all content other than the anomalous event. This type of redundancy is commonly encountered in streaming video scenarios, such as recordings of daily life activities.

• By analyzing these tasks, we aim to highlight the types of redundant information more relevant to streaming scenarios and provide insights into how these factors influence models as well as directions for their future development.