Streaming Long Video Understanding with Large Language Models

We sincerely thank the reviewer for recognizing our novelty, method and paper writing. Below we provide point-to-point responses to all the raised questions:

Q1: The necessity of VideoStreaming on short-term videos.

A1: For short videos, sampling multiple frames and treating it as a multi-image understanding task works well. However, this is not suitable for long videos due to excessive number of frames. As we focus on long-form videos, the short video results are mainly to demonstrate our method also performs well in that setting. Since LLaVA-Next-Video is a concurrent work updated on April 30th, we did not include a comparison.

Q2: The results on medium-length videos.

A2: The results of VideoChat2 Mistral-7B on EgoSchema is updated on May 23rd (in arxiv:2311.17005 v4), which is later than the submission. Technically, the accuracy of 54.4% is obtained by using 2M SFT data including ego-centric QA and more powerful LLM. Since it is difficult to obtain the same training data in a short period of time, we train VideoStreaming with Mistral-7B on our own data and compare the results on three benchmarks in Table 4 in rebuttal PDF.

• On EgoSchema, we achieve 48.1%. The gap to VideoChat2 is majorly due to the lack of ego-centric training data.
• On recent popular MLVU and VideoMME benchmarks consisting of medium and long videos, our model significantly outperforms VideoChat2 by 7.2 on MLVU multi-choice questions, 6.1 and 6.4 on VideoMME medium and long subset respectively.

Q3: More experiments on long videos and explanation of Table 8.

A3: In rebuttal PDF, we provide more ablations on long videos in Table 6 and add explanations for different settings in Table.

In Table 6, we ablate the memory propagation, temporal selection, number of summarization tokens $P$ and number of selected clips $V$ on MovieNet-QA.

• For hour-long videos, the propagated memory is crucial for understanding from all aspects.
• The temporal selection significantly improves detailed plot analysis, and appropriately increasing the number of selected clips can further improve performance.

For the performance variance on MovieChat, the reasons are below.

• For global mode requiring long-term comprehension, the lack of memory will cause the LLM to have very limited temporal receptive fields and lead to dramatic performance drop.
• Besides, as shown in Figure 4 in manuscript, the beginning of the movie contains crucial cues for global understanding. Directly feeding the memories of final 4 clips without temporal selection into LLM also results in information loss.
• For breakpoint mode needing detailed understanding of specific moments, it is important to select the related clips otherwise the LLM cannot catch the necessary details.
• Further, the propagated memory across clips increases tolerance for imperfect temporal selection, as it preserves previous contexts. Without memory, there would be a strict requirement on temporal selection accuracy to avoid completely losing necessary details.

Q4: The promising application like GPT-4o.

A4: Yes, VideoStreaming can disentangle video encoding and question-answering into two asynchronous processes for streaming QA with low time latency. We are working on the deployment to make it a more promising application. We did not report it in the paper due to the lack of suitable benchmarks for evaluation.

Q5: About the example in Figure 5.

A5: Thanks for pointing out this. We find that some specialized words like 'Philippe Petit' and 'Twin Tower' can make LLM recall the text pretraining data. To solve this problem, we replace the words that may cause information leakage and formulate a new question as shown in Figure 1 in rebuttal PDF. The comparison between VideoStreaming and text-only LLM demonstrates that our model can comprehend the high-level expressions in the question and summarize the relative contents in the visual contexts.

Q6: The grounding ability in Figure 5.

A6: We use a similarity-based strategy to select the clips related to specific questions, which relies on some semantic relations to do selection. Even with an implicit question expression, our model can correlate the scenes that contain the protagonist with the question as shown in Figure 1 in rebuttal PDF.

Q7: Use previous layers of Vicuna to replace Phi.

A7: Using previous layers of Vicuna to replace half Phi will cause two problems.

• Our streaming encoder is trained with a prefix task to generate outputs that only reference memory tokens to build a robust memory. This requires first training the entire model for next token prediction, before using partial layers in later stages. Replacing Phi with Vicuna layers forces us to first train the full 7B model, significantly increasing computational cost.
• If we take the previous few layers of Vicuna and maintain the same parameters with Phi, we can only retain a very small number of layers. Table 5 in rebuttal PDF shows the performance drops when replacing Phi with previous Vicuna layers under the same parameters.

Q8: The importance of initial tokens.

A8: Yes, you are correct. It is our prefix task that ensures the information of a sequence is distilled into the last few tokens. We will accordingly modify the presentation.

Q9: Operate on the frames instead of clips.

A9: Yes, our model can be applied to the scenario by sampling frames at a moderate FPS. In Table 7 in rebuttal PDF, we show the comparison between clip-based sampling and frame-based sampling at 1 FPS, and present the average number of sampled frames on each dataset.

• The overall performance is close since the two sampling strategies lead to comparable number of sampled frames.
• On the longer MovieNet-QA videos, the frame-based sampling performs slightly better, as the clip-based approach limits the maximum number of clips, resulting in fewer total sampled frames for very long videos.

Dear Reviewer sqPz,

We sincerely thank you again for your great efforts in reviewing this paper, especially for the precious advice that has helped us improve the quality of this paper significantly!

In response to your suggestions, we have supplemented the experiments and discussions in the rebuttal. If there are any further clarifications or additional experiments you would like us to address, please do not hesitate to let us know.

Best,

Paper 6273 Authors.

A1: ...

The response somehow works for me. Let's focus more on the long-form videos. However, I do think that in a future version, you should provide an ablation study on short-term videos to show whether your method decreases performance compared to the simple pooling strategy.

A2 & A3: ...

Nice & important results. Many thanks for that.

A4: ...

The recommendation is to validate on some online/streaming action detection benchmarks, or follow Streaming Vid2Seq [a]/VideoLLM-online [b] on CVPR 2024.

[a] Streaming Dense Video Captioning. arXiv: 2404.01297.

[b] VideoLLM-online: Online Video Large Language Model for Streaming Video. arXiv: 2406.11816.

A5: ...

Now the example is less misleading. Thanks for that. However, I wonder if there are instances, such as a movie title at the beginning frame, where knowledge could still be leaked. Could you provide non-movie examples? If the rebuttal cannot provide an anonymous link, please include them in a future version.

A6: ...

This does not convince me. I suspect the grounding is cherry-picked and there are many false positives and true negatives. However, I understand that this is what the demo needs.

A7 & A9: ...

Thanks for that. The results are consistent with my estimate.

Overall, I appreciate the hard work of the authors. Most of my concerns have been addressed, and I plan to increase my rating score.

We sincerely thank the reviewer for acknowledging our model's ability to handle long videos with high efficiency. Below we would like to provide point-to-point responses to all the raised questions:

Q1: Related work on processing key clips.

A1: Thanks for pointing out this related work. LGDN is a pioneering work in selecting the key clips that are highly related to the text prompts. It employs a contrastive learning architecture to learn global video-text correspondence and then uses the model to calculate fine-grained clip-text correspondence scores to filter out noisy frames. The key difference is that we can leverage both fine-grained temporal grounding labels and implicit gradient flow from the next token prediction loss to guide the learning of temporal selection. We will add the discussions in the related work.

Q2: The redundancy and diversity of selected clips.

A2: Thanks for the promising suggestion of integrating a regularization term in the key clip selection stage. In the author rebuttal PDF, we provide statistics on the feature similarity and the temporal distribution of the selected clips in Figure 2.

• For feature similarity, we calculate the cosine similarity of the time indicators of the selected clips. We divide the x-axis from 0 to 1 into 20 bins, and it shows the distribution of feature similarity. The average cosine similarity is 0.68.
• For temporal distance, we calculate the time intervals between the selected clips (represented as the ratio of the total video duration) and visualize the distribution of these time distances. The average distance is around 35% of the total video length. The statistics on the feature similarity and temporal distance indicate that the selected clips are not redundant.

We will explore integrating a regularization term in future work.

Q3: The efficiency of key clip extraction with multiple queries.

A3: It is worth noting that in our architecture, the video encoding process and the question-answering process are disentangled. Our model only requires once forward computation to encode a video into memories, which are independent of specific questions. When encountering multiple questions for a video, we only need to calculate the similarity between multiple question indicators and the obtained time indicators to select different key clips. This process requires much fewer computations than the streaming video encoding and is efficient.

Dear Reviewer 9Ye5,

We sincerely thank you again for your great efforts in reviewing this paper, especially for the precious advice that has helped us improve the quality of this paper significantly!

In response to your suggestions, we have supplemented the discussions and clarifications in the rebuttal. If there are any further clarifications or additional experiments you would like us to address, please do not hesitate to let us know.

Best,

Paper 6273 Authors.

We sincerely thank the reviewer for acknowledging the significance of the research problem, the interesting and sound technical design, as well as sufficient experiments. Below we provide point-to-point responses to all the raised questions:

Q1: The use of summarization tokens.

A1: Our motivation is to distill the information in a video clip (TN clip tokens) into a compact set of tokens (TP summarization tokens). These TP tokens are intended to serve as a memory that summarizes the clip content, which can be propagated across different clips for long-term memory formulation.

To achieve this, we initialize TP summarization tokens using pooling and feed them into an autoregressive language model. The causal attention layers then aggregate the clip information into the TP tokens to create a compact memory. To ensure these tokens consolidate useful information, we design a prefix task where the language model generates captions using only the TP tokens, without access to the original clip features. This avoids attention between the TT caption tokens and the TN clip tokens, encouraging the TP tokens to encapsulate the clip information.

Q2: The necessity of language reasoning for visual-dominant dependencies.

A2: We validate the necessity of language reasoning in Table 2 in rebuttal PDF. We reimplement a conventional video memory strategy MC-ViT [7] for comparison. Our method presents dominant advantages and the reasons are two-fold.

• MC-ViT consolidates memory according to handcrafted rules like clustering. In contrast, using language model as the streaming encoder allows us to leverage extensive video caption/QA data to guide memory consolidation in an end-to-end manner. This data-driven strategy results in more comprehensive memories that facilitate general video understanding.
• Since the memory is fed into a subsequent LLM for reasoning, we need to align the memory with the input feature space of the LLM. Compared to the output from ViT backbone, it is easier to align the memory generated by a language model with the feature space of LLM, thus improving performance.

Q3: More efficient alternatives in streaming encoder.

A3: We try fewer layers of Phi model as streaming encoder in Table 2 in rebuttal PDF. It indicates too few layers would lead to performance drop due to insufficient memory consolidation. However, even with fewer parameters (0.3B with 4 layers), the language model-based approach still outperforms MC-ViT, which reinforces the necessity of language reasoning for visual feature processing.

Q4: The difference in memory propagation with MemDPC.

A4: The major difference lies in the memory content, the memory usage and the technical design.

• MemDPC formulates a learnable memory bank shared for the entire dataset. The memory serves as the statistics of the whole dataset. In contrast, our memory is a summarization of a specific video.
• The memory bank in MemDPC is an external knowledge reference that provides multiple future hypotheses for future prediction. Our memory is used to conclude the historical information in a long video to facilitate long-term comprehension.
• MemDPC only uses memory to augment the future prediction, there is no explicit memory propagation process. Conversely, in our architecture, the memory is propagated across different clips to form a global encapsulation of a long video.

Q5: More ablation studies.

A5: Besides Table 8~9, we have presented ablation studies on summarization tokens, temporal supervision, time prompts, and similarity measurements in Table 10~13 in the appendix (Page 15~17).

Q6: More baselines like LLaMA-VID.

A6: We add the results of LLaMA-VID with Vicuna-13B on EgoSchema, Next-QA and MovieChat in Table 3 in rebuttal PDF. For MovieChat breakpoint mode, we only input segments of the video up to the breakpoint timestamp to the model. It is easier than our setting which requires adaptive temporal selection. Nevertheless, our model still significantly outperforms LLaVA-VID on all benchmarks.

Q7: About progressive training.

A7: Sorry for the confusion. The progressive training part is used to clarify the training process rather than serve as an innovation. We will modify this to make it clearer.

Q8: Explanations for significant zero-shot performance on EgoSchema.

A8: The reasons are two-fold.

• The questions in EgoSchema do not have obvious ego-centric domain-specific characteristics, but rather require long-term temporal memory and perception, which is a major strength of our model.
• Compared to MC-ViT, our architecture with LLM has stronger generalization ability. It can effectively generalize to different video scenarios even without ego-centric training pairs. This is also validated by the superior zero-shot performance of LLaMA-VID 13B compared to prior methods like InternVideo and FrozenBiLM (35.5 vs 32.1, 35.5 vs 26.9).

Q9: About grounding in Figure 5.

A9: For grounding, we calculate the similarity between the question and the time indicators of each clip to select the related segments. Below are some explanations.

• The similarity-based strategy relies on some semantic relations between questions and clips to do temporal selection. It can correlate the key components in questions like Tower with the clips containing corresponding elements.
• The grounding intervals are determined by the temporal length of the sampled video clips.
• We only feed the encoded memories of the selected clips to LLM, so the model fully relies on the grounding results to produce responses.
• For quantitative evaluation, Next-GQA requires accurate grounding to produce correct answers. The results in Table 5 in the manuscript validate our method reaches promising grounded question-answering ability, even exceeding the methods with specialized grounding modules. There currently lack hour-long grounding QA benchmarks for further evaluation.

We sincerely thank the reviewer for recognizing our motivation, technical algorithm, and writing. Below we would like to provide point-to-point responses to all the raised questions:

Q1: The comparison on IntentQA dataset.

A1: We compare our model with recent advanced methods in the Table below. We report the zero-shot performance on IntentQA test set. Our method presents a dominant advantage in temporal understanding. And the overall performance is significantly superior to recent works.

Q2: The details on Gumbel-TopK.

A2: Sorry for the confusion and missing the detailed explanation for Gumbel-TopK. Similar to the Gumbel-Softmax technique, Gumbel-TopK is used to allow gradient backpropagation when selecting the Top-K indices. The major difference is that Gumbel-TopK can take multiple indices (i.e., the Top-K largest activations) for each sample, while Gumbel-Softmax only takes the index corresponding to the single largest activation for each sample. This design allows us to select multiple video segments that are relevant to the questions (i.e., the Top-K most related ones) to do comprehensive reasoning. This is not a technical contribution, and we will modify 'develop' into 'adopt' for correction.

Dear Reviewer FUM4,

We sincerely thank you again for your great efforts in reviewing this paper, especially for the precious advice that has helped us improve the quality of this paper significantly!

In response to your suggestions, we have supplemented the experiments and explanations in the rebuttal. If there are any further clarifications or additional experiments you would like us to address, please do not hesitate to let us know.

Best,

Paper 6273 Authors.