Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

Q5: The authors test the processing speed of models on two NVIDIA Tesla A800 80GB GPUs, which is not typical.

A5: We acknowledge that evaluating inference speed on two NVIDIA Tesla A800 80GB GPUs is not typical for real-world applications. Our principle for this setup stems from the large number of parameters in StreamChat and the need for efficient execution. Similar to large language model serving systems like vLLM [7] and SGLang [8], StreamChat utilizes tensor parallelism to distribute the computational load and improve throughput. For instance, SGLang employs tensor parallelism to efficiently serve models like LLaVA-NExT-Video-34B. While we recognize that our current two-GPU implementation may not ideal for typical users, it allows us to explore the potential of StreamChat in real-time processing.

Based on your suggestions, we have explained the reasons and details of our deployment in the revised version (in Section 4(L343-344) and Appendix F (L1011-L1018) ) to facilitate more clearly understanding of our approach.

1. System scheduling optimization. Improving VRAM utilization, drawing inspiration from efficient serving technologies like SGLang and vLLM.

2. Model quantization. Employing techniques like AWQ [14] and SmoothQuant [15] to reduce model size and enable single-GPU operation.

3. Efficient memory artecture. Simplifying our memory storage mechanism to further improve efficiency and broaden deployment scenarios.

We hope these explanations address your concerns. Please let us know if you have any further questions.

[1] Videollm-online: Online video largelanguage model for streaming video. CVPR 2024

[2] Flash-vstream: Memory-based real-time understanding for long video streams. arXiv 2024.06

[3] MiniCPM-V: A GPT-4V Level MLLM on Your Phone. arXiv 2024.08

[4] InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output. arXiv 2024.07

[5] VILA: On Pre-training for Visual Language Models. CVPR 2024

[6] InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. CVPR 2024

[7] Efficient memory management for large language model serving with pagedattention. ACM SOSP 2023

[8] SGLang: Efficient Execution of Structured Language Model Programs. github 2024

[9] Moviechat: From dense token to sparse memory for long video understanding. CVPR 2024

[10] MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer. CVPR 2024

[11] IVTP: Instruction-guided Visual Token Pruning for Large Vision-Language Models. ECCV 2024

[12] Raptor: Recursive abstractive processing for tree-organized retrieval. ICLR 2024

[13] Walking down the memory maze: Beyond context limit through interactive reading. arXiv 2023.10

[14] AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. Mlsys 2024

[15] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. Mlr 2023

Thanks for your prompt feedback. We address your concerns in the following statement:

Q1: Why use LLaMA-3 to evaluate predictions?

A1: There are mainly two reasons:

First, LLaMA-3 is not only one of the most powerful language models , but also has a complete open source attribute. This is very conducive to more researchers to make further improvements and reproduce our results under cost-controlled conditions.

Second, according to our experimental results, we found that LLaMA-3 and human judgments on the correctness of answers tend to be consistent. We sampled 20 GPT-4o prediction samples on StreamBench for verification and found that the judgment results of LLaMA-3 were slightly different from those of humans.

The experimental data is shown in the following table:

LLaMA-3 tends to give higher scores, but is on par with human judgement in terms of accuracy. We have also provided some examples for your convenience:

Same score:

{"id": 180, "question": "I can't quite remember, who used the knife to cut the beef?", "answer": "It was the man in the white shirt.", "class": "LM", "pred": "I can't tell who used the knife in the images since I can't identify or describe faces. However, it looks like several cooking activities involving a knife are shown, with people in chef uniforms working together.", "llama_pred": "no", "llama_score": 2, "human_pred": "no", "human_score": 2}

{"id": 120, "question": "Is there any scissors in this video?", "answer": "Yes, there is a pair of scissors above the silver pot.", "class": "CS", "pred": "Yes, there are scissors visible in several parts of the video, hanging on the wall on a kitchen rail behind the person cooking.", "llama_pred": "yes", "llama_score": 5, "human_pred": "yes", "human_score":3 }

{"id": 120, "question": "Is there any scissors in this video?", "answer": "Yes, there is a pair of scissors above the silver pot.", "class": "CS", "pred": "Yes, there are scissors visible in several parts of the video, hanging on the wall on a kitchen rail behind the person cooking.", "llama_pred": "yes", "llama_score": 5, "human_pred": "yes", "human_score":3 }

{"id": 120, "question": "Where is a glass or bowl containing red, thick food with a spoon in it?", "answer": "It may be on the cupboard near the cutting board or it is at corner of the cutting board, next to the woman.", "class": "FT", "pred": "The glass or bowl containing red, thick food with a spoon in it is on the countertop, near where the woman is working on the cutting board.", "llama_pred": "yes", "llama_score": 5, "human_pred": "yes", "human_score":5}

Different score:

{"id": 54, "question": "Can you tell me where in the current room I can find the red fire extinguisher?", "answer": "Yes, it looks like there is a red fire extinguisher mounted on the wall near the refrigerator.", "class": "SM", "pred": "The red fire extinguisher is located on the wall in the kitchen area, near the refrigerator.", "llama_pred": "yes", "llama_score": 5, "human_pred": "yes", "human_score": 4}

{"id": 88, "question": "Where did I place the cup that I just filled with coffee grounds?", "answer": "If I\u2019m not mistaken, you placed the cup on the countertop, next to a clear plastic container and a stainless steel pot lid.", "class": "LM", "pred": "The cup filled with coffee grounds was placed on the countertop next to a dish drainer and a moka pot lid.", "llama_pred": "yes", "llama_score": 4, "human_pred": "yes", "human_score":3}

{"id": 220, "question": "Based on what you see, Did the man in black clothing handle a green shovel?", "answer": "Yes, the man in black handled a green shovel in his hand", "class": "LM", "pred": "Based on the images provided, there is no indication that the man in black clothing handled a green shovel.", "llama_pred": "no", "llama_score": 0, "human_pred": "no", "human_score":2 }

{"id": 88, "question": "Where did I place the cup that I just filled with coffee grounds?", "answer": "If I\u2019m not mistaken, you placed the cup on the countertop, next to a clear plastic container and a stainless steel pot lid.", "class": "LM", "pred": "The cup filled with coffee grounds was placed on the countertop next to a dish drainer and a moka pot lid.", "llama_pred": "yes", "llama_score": 4, "human_pred": "yes", "human_score":3}

Q3: StreamChat seems too complicated. The authors should justify every components used in building the memory system in StreamChat.

A3: In order to make StreamChat more adaptive to dynamic stream video scenarios, we have designed vision-based short-term memory, caption-based long-term memory and dialogue memory. However, in order to make these components work smoothly together, we also designed system scheduling, which decouples the operation of each component into different threads, thus ensuring low latency and real-time video processing.

Our intention was to design a human-like memory method that simulates the Atkinson-Shiffrin model[3]. The design of short-term memory is to focus on what is happening right now. Just as we described in Section 3.1.1 , we utilize the continuously updated buffer to simulate human short-term memory. The long-memory tree simulates the complex and abstract memory of human. We use a clustering algorithm to perform secondary compression on visual features to simulate the fragmented storage of human memories of past events, while also reducing the required storage space. In the meantime, we use captions of each video clip as the index to search for the most related information. Our experiment results in Section 4.4 Table 7 verifies the effectiveness of each components.

According to our experiments, adding long-term memory $M_l$ improved the LM task performance by 6.2%, while the use of short-term memory $M_s$ boosts SM task performance by 3.2%. The results indicate that the model’s performance in each subtask aligns with the inclusion of specific memory attributes. Additionally, we observe that different memory components can complement each other. When both long-term $M_l$ and short-term memory $M_s$ are applied simultaneously, the average accuracy increases by 0.9%.

Q4: Looks more like a RAG system specifically designed for online videos. Authors should compare with more RAG system.

A4: We appreciate the reviewer's suggestion to compare our method with more Retrieval Augmented Generation (RAG) systems. We agree that this comparison would be valuable. We are aware of recent work [4, 5] applying RAG to video understanding and addressing hallucination issues. However, these methods currently lack publicly available code, precluding direct experimental comparison.

We are committed to incorporating these comparisons as soon as the code becomes available, providing a more comprehensive analysis of different approaches on StreamBench. Furthermore, we understand the reviewer's implicit concern about the distinction between our approach and RAG systems. We will revise the paper to more clearly articulate the key differences in motivation and operational mechanisms.

[1] Retrieval-augmented generation for ai-generated content. arXiv 2024.02

[2] Memorybank: Enhancing large language models with long-term memory. AAAI 2024

[3] A proposed system and its control processes. The Psychology of Learning and Motivation 1968

[4] iRAG: Advancing RAG for Videos with an Incremental Approach. CKIM 2024

[5] ViTA: An Efficient Video-to-Text Algorithm using VLM for RAG-based Video Analysis System. CVPR 2024

Q3: Include comparisons with more recent benchmarks for video understanding, such as Seed-Bench, Video-Bench, MVBench, and LVBench.

A3: Thank you for the suggestion to include additional benchmarks. We have investigated evaluating StreamChat on SEED-Bench-2 [3], Video-Bench [4], MVBench [5], and LVBench [6].

On SEED-Bench-2, the limited number of frames (4-8) per video hinders StreamChat's ability to leverage its temporal modeling capabilities, which are designed for streaming video. Consequently, our performance is completely consistent with the LongVA baseline on this benchmark, as shown below.

Regarding the poor performance of our method on SEED-Bench, our analysis results are as follows:

First, since our method pays more attention to the processing ability under lengthy videos, and SEED-Bench only provides us with 4-8 frames of video information, as our method mainly focuses on video streaming scenarios, this is not conducive to our method. We will further improve our method to adapt to more scenarios and user needs.

Second, since the setting of SEED-Bench, we removed all the memory storage and evaluated the performance of LongVA. We found that LongVA also exhibited suboptimal performance. We will continue to explore methods to make up for this defect.

We also made efforts to evaluate our method on Video-Bench, MVBench, and LVBench, but encountered several obstacles. We successfully ran experiments on Video-Bench, but the submission website malfunctioned, preventing us from retrieving our scores. This issue appears to be affecting other researchers as well.

For MVBench, copyright restrictions on some videos hindered our evaluation. We have applied for the necessary usage rights but have not yet received approval. Similarly, access to the YouTube data used in LVBench requires approval, which we have not yet received despite application. We are actively working to address these issues and will provide results on these benchmarks if access becomes available.

Q4: The paper lacks an analysis of factors that impact performance on StreamBench.

A4: We appreciate your feedback regarding the analysis of factors impacting performance on StreamBench. We address the effect of input sequence length and language model size below.

First, regarding input sequence length, Section 4.4 (Design of Long Memory Tree, Figure 7(c)) and the table below explores this impact. $C$ (clustering goal) represents the length of each clustered visual feature, which is related to the input sequence length.
Based on our memory design, the final input vision token length is equal to $C$ x $L$. The results show that increasing the sequence length generally improves performance (Scor, Acc, Cho, RDP) but also increases memory consumption. The performance gains diminish with increasing sequence length. To balance performance and resource consumption, we chose a sequence length of 5 (C=5) for our experiments.

We appreciate your suggestion and have reorganized the description of the experimental setup in Section 4.4 to make clear the impact of the input sequence on model performance.

Second, regarding language model size, we agree that larger models generally lead to improved performance, as demonstrated by scaling laws. However, LongVA provieded us the 7B model and we focused on models with 8B parameters or less to maintain comparable computational requirements and ensure a fair comparison. Exploring larger models with increased data and computational resources is a promising direction for future work.

Q4: Do you plan to make this benchmark publicly available?

A4: Yes, we plan to release our code and benchmark data publicly to benefit the research community. We are currently preparing the code and benchmark for release and will make it available on github as soon as possible.

Q5: What the process of identifying the various categories SteramBench encompasses?

A5: The details of our data collection pipeline in Appendix A (L718-L750) may help you solve this question. Our data sources are EgoSchema[3] and Yotube-8M[4]. The video information from Yotube-8M already comes with the category label corresponding to each video when it is downloaded, so we can use it directly. The videos from EgoSchema do not come with category labels, but the paper has a total category set of all videos, so we use Video-LLaVA[5] to help us determine the category of each ego video.

Q6: Carification on the LLM-based accuracy calculation described in the appendix, asking about the formula's variables, the LLaMA model's output, and the specific prompt used.

A6: We sincerely sorry for our loss. Prompts that are used for the LLaMA-3[6] model can be a very important part of getting right results. The following is the prompt that we use:

{
    "role": "system",
    "content":
    "You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. "
    "Your task is to compare the predicted answer with the correct answer and determine if they match meaningfully. Here's how you can accomplish the task:"
    "------"
    "##INSTRUCTIONS: "
    "- Focus on the meaningful match between the predicted answer and the correct answer.\n"
    "- Consider synonyms or paraphrases as valid matches.\n"
    "- Evaluate the correctness of the prediction compared to the answer."
    },
    {
    "role": "user",
    "content":
    "Please evaluate the following video-based question-answer pair:\n\n"
    f"Question: {question}\n"
    f"Correct Answer: {answer}\n"
    f"Predicted Answer: {prediction}\n\n"
    "Provide your evaluation only as a yes/no and score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. "
    "Please generate the response in the form of a Python dictionary string with keys 'llama_pred' and 'score', where value of 'llama_pred' is  a string of 'yes' or 'no' and value of 'score' is in INTEGER, not STRING."
    "DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. "
    "For example, your response should look like this: {'llama_pred': YOUR_JUDGE, 'score': YOUR_SCORE}."
}

After we feed this prompt to LLaMA-3, we can extract the dict information from the response and we can calculate the final score. According to your suggestion, we add this prompt setting to Appendix D (L894-L909) of the final version of paper.

[1] Long context transfer from language to vision. arXiv 2024.06

[2] Egoschema: A diagnostic benchmark for very long-form video language understanding. NeurIPS 2024

[3] A proposed system and its control processes. The Psychology of Learning and Motivation 1968

[4] Youtube-8m: A large-scale video classification benchmark. arXiv 2016.09

[5] Video-llava: Learning united visual representation by alignment before projection. EMNLP 2024

[6] The llama 3 herd of models. arXiv 2024.07

Q2: Characteristics that a basic model needs to have.

A2: Thank you for your attention to the characteristics of the base model.

Our research indicates that a suitable model should possess the following key attributes:

1. Long-Form Video Understanding: Effective processing of long videos is crucial. While we utilize K-Means for feature compression, the information retrieved by our memory mechanism remains relatively long, requiring a model capable of handling extended sequences.

2. Robustness to Prompt Variations: For accurate and coherent multi-turn conversations, the model must be robust to changes in prompt wording. This robustness is essential to prevent inconsistencies or hallucinations in the model's output when prompts are adjusted to incorporate information from the memory mechanism.

By integrating LongVA with our proposed system, we successfully extend its capabilities to encompass streaming video processing and multi-turn conversations while preserving these critical characteristics.

Inspired by your advice, we conducted experiments using the open source model LongVILA. Here is the parameter setting that we utilized.

According to the experiment result, we found that for LongVILA, our system can further improve 2.2% accuracy. We will continue to upgrade more video understanding models to further improve the versatility of our method.

Q3: Would fine-tuning LongVA with some streaming vision-language data be better?

A3: Thank you for this insightful question about fine-tuning LongVA. Our research focuses on improving performance under resource constraints, and a training-free approach is crucial for achieving this with minimal resource consumption. While existing methods like VideoLLM-online [3], Flash-VStream [4], and VideoStreaming [4] explore fine-tuning with self-constructed instruction data, they face certain limitations:

1. Limited Generalization: Methods like VideoLLM-online, while leveraging LLM-generated training data, often achieve suboptimal performance on general video understanding benchmarks. We hypothesize that fine-tuning large pre-trained LLMs (trained on millions or even billions of data points) can negatively impact their generalization ability , which is essential for adapting to diverse scenarios.

2. Reduced Processing Speed: The inherent structure of existing fine-tuning methods has limited video processing speed (with the fastest achieving 5 FPS), a significant drawback for real-time applications.

While we acknowledge the potential of fine-tuning, as demonstrated by the partial success of VideoLLM-online, these limitations motivated our exploration of a training-free approach, which has shown promising initial results. Our future work will focus on identifying limitations of our training-free approach and further investigating trainable paradigms that address these challenges.

First, exploring high-quality streaming video instruct-tuning data. We believe the quality of data is improtant to improve the generalization of video-llms. More video scenes and more training tasks should be included in the instruct-tuning data. What's more, the data filtering is crucial. Filtering out toxic text and data with low information content can effectively improve the quality of training data. We have accumulated a lot of experience in benchmark data piepline as introduced in Appendix A and are confident that it can be transferred to production of training data.

Second, developing more dynamic model structure. In streaming video scenarios, inference efficiency can also be important. We are exploring more dynamic model structure to overcome the tight budget for time and space resource consumption.