SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding

For W1 and Q1: Concerns about data quality and the need for sample data to demonstrate the benchmark's significance

We understand the reviewers' concerns regarding data quality. To ensure the quality of our benchmark, we have implemented the following measures:

• Data Filtering and Cleaning: We conducted rigorous and multiple rounds of filtering and cleaning of the raw video data, removing noise and irrelevant information to ensure high quality and richness in 3.1.
• Manual Annotation and Verification: We combined automated tools with manual annotation to perform multiple rounds of verification, ensuring high-quality data in 3.1.2.
• Example Data: In Appendix E of our paper, we present some example data to demonstrate the quality and diversity of our dataset. Additionally, we will soon open-source the entire dataset and models for further research.

For W2 and Q2: Concerns about METEOR and GPT-4 Score, and the suggestion to use open-source models for evaluation

We have additionally used the open-source model (InternVL2-Llama3-76B) for evaluation and compared its results with METEOR, GPT-4 Score, and human evaluations. The specific experimental results and analysis have been added to Appendix F.1 of the paper. The results indicate:

1. The score variations between human evaluations and GPT-4 are generally consistent across different models, though human scores are more discriminative, suggesting that GPT-4 scores are reasonably reliable.
2. InternVL2-Llama3-76B shows excessive leniency, with more than half of the models scoring above 60 using the same prompt as GPT-4, while METEOR scores are too low, lacking discrimination.

Regarding the use of sentence similarity in QAEgo4D as an important evaluation metric for computing distances in semantic space, we have incorporated this reference and content into our paper in Appendix D.

Thank you so much for your positive review and insightful comments. Sorry for the wait, we've added many experiments recently into the revised version in red.

Weaknesses:

1. Lack of visualization of video length distribution: We have added a figure showing the distribution of video lengths (see Appendix A.5). The results indicate that 95.05% of the videos in the dataset are longer than 1 minute, primarily ranging from 60 to 240 seconds.
2. Similarity between StreamingChat architecture and Video-online, explanation of differences: Both our proposed StreamingChat and Video-online are long-context video understanding models, utilizing interleaved multi-image training methods, which are reflected in their architecture diagrams. However, they differ significantly in model architecture, training objectives, and task definitions:
   • Model Architecture: Our StreamingChat uses InternViT as the vision encoder, InternLM2 as the LLM, and an MLP projector for alignment. In contrast, Video-online uses CLIP ViT-L as the vision encoder, Llama-2-7B-Chat or Llama-3-8B-Instruct as the LLM, and an MLP projector for alignment.
   • Training Objectives: StreamingChat focuses on generating answers autoregressively and accurately, while Video-online also constrains the timing of output answers (e.g., during action or event changes).
   • Task Definition: StreamingChat addresses the fundamental task of streaming video understanding, answering questions in real-time as the video plays, enabling temporal multi-turn dialogue. Video-online, on the other hand, focuses on responding to a predefined question during video playback, deciding whether to answer based on action changes. Therefore, the datasets and data input formats in the architecture diagrams differ between the two models.
3. Lack of evaluation on latency and FPS: We have added figures showing the inference speed and resource consumption of multiple LVLMs in relation to video length (see Appendix F.2).

Questions:

1. Maximum video processing time of StreamingChat: In our Statistical Analysis, we mentioned that the dataset "consists of long videos with an average length exceeding 2 minutes." The 5 to 30 seconds range refers to the duration of each scene retained during filtering to ensure video quality. In fact, 95.05% of the videos in the dataset are longer than 1 minute, primarily ranging from 60 to 240 seconds. During training, due to the context window limitation, we split temporal dialogue paths exceeding 100 frames into multiple segments (using 8 A800 GPUs with 80G each). At 1 FPS, the maximum duration for each data segment is 100 seconds.
2. Testing the latest Flash-VStream and Video-online on SVBench: We have added the results of Flash-VStream in Appendix F.3 (to be incorporated into the main text later). However, as mentioned in W2, Video-online is not suitable for our defined streaming video understanding task, as it focuses on responding to predefined questions during video playback rather than answering questions in real-time (hindering multi-turn QAs). We will consider including it in future work.
3. Comparison of inference speed and resource consumption: We have added figures showing the inference speed and resource consumption of multiple LVLMs in relation to video length (see Appendix F.2). The inference time is generally positively correlated with the video length. For resource consumption, some models show a plateau as the input length increases, due to the saturation of input frames and the memory reaching its preset limit.

Thank you so much for your positive review and insightful comments. We have carefully considered your comments and provided detailed responses. We look forward to your feedback.

Thank you so much for your positive review and insightful comments. We appreciate your feedback and suggestions, which we have incorporated into the revised version in red.

Weaknesses: The evaluation heavily relies on LLMs. Using GPT-4 for both generation and validation creates a circular dependency. Response: We have addressed this by incorporating human involvement in the generation process and human evaluation to ensure consistency with LLM assessments. We have added human evaluations of various LVLMs results (see Appendix F.1). To validate the consistency between human scores and those from open-source and closed-source models, we employed human evaluation (10 professional annotators annotating 200 videos over a week), open-source model evaluation (InternVL2-Llama3-76B), closed-source evaluation model (GPT-4), and traditional evaluation metrics (METEOR). We then plotted a score comparison on Multi-turn QA Evaluation. The results indicate: (1) The score variations between Human and GPT-4 are generally consistent across different models, though human scores are more discriminative, suggesting that GPT-4 scores are reasonably reliable; (2) InternVL2-Llama3-76B shows excessive leniency, with more than half of the models scoring above 60 using the same prompt as GPT-4, while METEOR scores are too low, lacking discrimination.

Regarding the circular dependency issue, we mitigated the reliance on LLMs by incorporating two stages of manual annotations during the construction of annotations. Additionally, the inclusion of open-source and human evaluations enriched the evaluation results, further reducing the dependency on GPT-4 for evaluation.

Questions:

1. How reliable are LLM evaluations? In our evaluation results (see Appendix F.1), the score variations between GPT-4 and Human are generally consistent across different models, suggesting that GPT-4 scores are reasonably reliable.
2. Did you try different LLMs to evaluate the models? Yes, we used an open-source model (InternVL2-Llama3-76B) with the same prompt as GPT-4 for evaluation. The results showed excessive leniency, leading to higher average scores for all models.
3. How did evaluation depend on LLMs hyperparameters like temperature? In fact, under the appropriate setting of averaging multiple runs, adjusting the temperature had little impact on the results. We fixed GPT-4's temperature at 0.7.
4. Does it change if run several times? For LLM evaluations, we averaged the results over 5 runs. For human evaluations, we used the mean scores from ten annotators.
5. How does it change over different prompts? In our experiments, more detailed prompts resulted in greater score discrimination. Initially, our prompts did not specify the scoring details for each score range, leading to Multi-turn QA Evaluation scores clustering between 40 and 50. After we detailed the specific criteria for each score range (see Appendix G), the score discrimination improved slightly.

Thank you so much for your positive review and insightful comments. Sorry for the wait, we've added many experiments recently into the revised version in red.

Weaknesses:

1. Lack of comparison with human performance

   We have included a comparison with human performance, as shown in Table 6 of Appendix F.3. This comparison provides insights into the gap between current models and human capabilities in long-context streaming video understanding. The results indicate that human performance significantly outperforms various open-source and closed-source models across all metrics in both evaluation settings (Dialogue Evaluation and Streaming Evaluation). Additionally, humans excel in Temporal Understanding (TU) but perform relatively weaker in Informational Completeness (IC). We also included LVLMs such as Flash-VStream-7B, Qwen2-VL-7B, and LLaVA-NeXT-Video-7B-DPO.

2. Lack of analysis on the impact of model size on performance

   To ensure a fair comparison, we selected base models with 7B or 8B parameters. Our benchmark is an ongoing project, and we plan to update it with different sizes of mainstream Video-LLMs, maintaining a real-time updated leaderboard. Additionally, we conducted further experiments comparing the performance of models with different LLM sizes, specifically InternVL2-2B, 4B, 8B, 26B, 40B, and 76B, on SVBench. The results indicate that model size significantly impacts performance, with larger models demonstrating better accuracy. However, the computational resource requirements also increase correspondingly. The specific experimental results and analysis have been added to Appendix F.4 of the paper.

3. Lack of comparison with several important models

   Thank you for pointing this out. We have now included the evaluation results for LLaVA-NeXT and QwenVL2 in Appendix F.3 of the paper. We will also add the evaluation results for the remaining models to Table 6 as soon as possible.

Questions:

1. The benchmark videos come from publicly available datasets; how do you ensure that the models used for evaluation have not encountered this data during their training?

   Our dataset specifically focuses on streaming understanding tasks. We have constructed a novel and reasonable annotation pipeline to create unique QA chains, ensuring that the data has not been used for training other models. However, we acknowledge that it is challenging to guarantee that no model has been trained on these videos, as high-quality datasets are often publicly accessible. We have employed a complex filtering process to select videos from multiple large-scale datasets (e.g., YT-Temporal-1B), ensuring that the evaluation models do not cover all the videos we use. In the future, we plan to crawl the latest videos or shoot our own to ensure this.

2. How do QA chains address the issue of information sparsity in videos?

   We have removed sparse or low-density videos during the dataset construction process. Only videos with an average scene duration between 5 and 30 seconds are chosen, ensuring fluidity and rhythm essential for effective streaming data analysis. This ensures that the QA chains can describe temporal changes meaningfully, as constructing meaningful QA chains for long videos without key events is challenging.

3. Do the videos in your benchmark retain their audio tracks? Does audio genuinely assist in answering the questions?

   Our work aims to evaluate Large Vision-Language Models (LVLMs), so we selected baselines that do not consider audio to ensure a fair comparison. Since most of the models we selected cannot incorporate audio information, for models that can, such as VideoLLaMA2, we chose the VideoLLaMA2-7B-16F model instead of the VideoLLaMA2.1-7B-AV model, which can include audio information. Although we have not yet conducted experiments with audio, existing research suggests that retaining audio information can improve overall model performance, particularly in complex tasks requiring the integration of multiple information sources. For instance, the results in Figure 8 of [1] indicate that incorporating audio information generally enhances evaluation results on video QA benchmarks.

[1] Shu F, Zhang L, Jiang H, et al. Audio-visual LLM for video understanding[J]. arXiv preprint arXiv:2312.06720, 2023.

Thank you so much for your responses. Your experiments are now much more comprehensive, and all my concerns have been addressed. Therefore, I'd like to maintain my current rating.

I'm also curious about your future research plans based on your benchmark. I’m also curious about your future research plans based on this benchmark. Do you intend to extend it to encompass additional streaming video understanding tasks, such as streaming grounding or streaming dense captioning?