Understanding Long Videos with Multimodal Language Models

We thank the reviewer for the encouraging positive comments and all the suggestions to improve our work further. We address the points raised by the reviewer below.

W1,W2: “Typos”

Thank you for pointing these out. We will correct the Figure 1 caption and fix our \cite{} vs \citep{} usage.

W3: “Compare to 2024 papers”

Several recent works such as MoreVQA [1], VideoAgent [2], VideoTree [3], LifelongMemory [4], LangRepo [5], Tarsier [6], InternVideo2 [7], LVNet [8] achieve strong performance on long video understanding tasks by utilizing strong pre-trained LLM and MLM models. We will update Table 2 and Table 3 to include comparison to these works. We will also discuss these in our related work section. We would be highly grateful if the reviewer has pointers for any additional papers we may have missed.

W5: “Performance not good”

Our MVU is a framework that can be integrated with other works. We combine MVU with state-of-the-art LVNet (following their open-source implementation) to achieve additional improvements.

This LVNet + MVU variant achieves strong performance competitive with more recent works. It also highlights how the MVU framework can be easily integrated with other existing works to achieve additional performance.

W5: “How to select the frames”

Frame selection is very important for long videos but somewhat orthogonal to our work. This means that we can integrate our MVU framework with existing frame-selection methods easily. LVNet is one such frame-selection approach. Our LVNet + MVU variant combines their orthogonal strengths to further improve performance.

Secondly, MVU extracts object information from multiple frames. This can be viewed as an alternative in addition to frame selection. This is because our object information extraction indirectly acts as an information bottleneck similar to frame selection. For frames without objects of interest, no information is extracted. For multiple frames containing the same object (identified by our object tracker), the repetitive information is removed. This resembles the idea of selecting useful information from multiple frames. Within MVU, this process acts as a secondary form of information selection from many frames. We provide an ablation to highlight this below:

Note how our lightweight object detector and tracker allows us to increase the frames but maintain good inference speeds.

Finally, we thank the reviewer again for these valuable comments that help improve our paper further. We are working on updating our manuscript with these changes.

References

[1] Min, Juhong, et al. "MoReVQA: Exploring Modular Reasoning Models for Video Question Answering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Wang, Xiaohan, et al. "Videoagent: Long-form video understanding with large language model as agent." European Conference on Computer Vision. Springer, Cham, 2024.

[3] Wang, Ziyang, et al. "VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos."

[4] Ying Wang, et al. "Lifelongmemory: Leveraging llms for answering queries in long-form egocentric videos."

[5] Kahatapitiya, Kumara, et al. "Language repository for long video understanding.".

[6] Wang, Jiawei, et al. "Tarsier: Recipes for training and evaluating large video description models.".

[7] Wang, Yi, et al. "Internvideo2: Scaling video foundation models for multimodal video understanding.".

[8] Park, Jongwoo et al. “Too Many Frames, not all Useful: Efficient Strategies for Long-Form Video QA.”.

We thank the reviewer for their response, but further emphasize the following 3 points about our work:

MVU improves on 20+ datasets

• MVU consistently achieves improved performance across 5 video QnA datasets (EgoSchema, NextQA, IntentQA, LongVideoBench, ActivityNet) as well as 16 robotics domain datasets.
• While the performance gains are minimal in some cases, improvements are consistent across 20 different datasets validating the effectiveness and generality.
• LVNet is a state-of-the-art method for EgoSchema over which we obtain an additional improvement. Multiple very recent prior work like [1] and [2] from CVPR'24, ECCV'24 under perform this method by up to 10% points. We believe it is difficult for random fluctuations to improve performance over such a method. In contrast, MVU improves over LVNet even with a smaller margin and shows such improvement across multiple datasets.

Improvements over LVNet consistent

We further experiment with LVNet on NextQA and IntentQA datasets in addition to EgoSchema where MVU achieves consistent results improvements.

This further validates how MVU is contributing to these improved results.

Core Issues of LLMs for video understanding

Our experiments and prior work suggest that core issues of LLMs are slow inference speed and handling long contexts.

• MVU achieves faster inference for MCQ tasks with likelihood selection
• MVU tackles long contexts with efficient information extraction

The inference speed-up is highlighted in Tables 1, 6, and 7. The better handling of context length is highlighted in Table A.10 and Appendix K.

We apologize if these points were not highlighted clearly earlier and would highly appreciate if the reviewer is able to take these into consideration. We are also updating the manuscript to better highlight these points.

1. Compared to LVNet, the proposed method only increases the baseline by 0.2, which is not convincing to verify the effectiveness. I even suspect that the performance might simply be attributed to random fluctuations. 2. I believe this paper fails to address the core issues of using LLMs for video understanding, which makes its value fall below the acceptance threshold for ICLR. I choose to maintain my original score.

Dear reviewer NaUs,

We are wondering if you were able to review our response to the concerns raised.

Given the main concern being about results, we again highlight the following:

1) MVU improves over SOTA baseline LVNet across 3 datasets

Our LVNet+MVU shows improvements consistent across three datasets now. We also compare against two prior works to highlight how improving over this LVNet baseline is not easy.

These results show the clear benefit of proposed MVU in multi-frame VLM settings with LVNet and how it outperforms SOTA prior works.

2) More multi-frame baselines on LongVideoBench

We integrate MVU with three different multi-frame baselines and evaluate on LongVideoBench dataset. MVU leads to performance improvements in each case.

We look forward to the reviewers response.

Thank you!

We thank the reviewer for the positive comments as well as all the valuable feedback. We address all concerns raised below.

W1: “Likelihood selection focuses on exact match”

Likelihood selection uses a likelihood measure which is the likelihood (probability) of the model generating the given sentence (as opposed to being an exact match). We apologize for any lack of clarity in our original paper regarding this and will highlight the following two points better in our paper.

1. In likelihood selection, we use the likelihood measure which is also used as the training loss when training LLMs. Given how LLMs trained with this loss (i.e. all decoder based LLMs such as LLaMA, Gemini, GPT) are highly effective at handling semantic meaning, it follows that this loss can capture semantic meaning.

2. This likelihood measure is calculated within the LLM latent space. This is equivalent to the probability (or likelihood) of that answer being generated by the LLM conditioned on the input question. We derive this in detail in Appendix F. Relating to the same example, this means that likelihood is an estimate of how likely the model would predict ‘C is washing plates’ as opposed to making that exact match. This means predictions closer to the target such as ‘C is cleaning dishes’ would also gain high likelihood values.

In fact, we validate this second point through a toy example. We provide an LLM with the question "X is cleaning dishes in the kitchen. What is X doing? a) washing plates, b) cleaning laundry, c) painting dishes. The correct choice is:" and calculate the likelihood for each of the 3 responses. The calculated likelihoods are 0.996, 0.006, 0.007 for a, b, c respectively (highest is selected), despite response (a) having no common words with the original statement unlike (b) and (c). This illustrates the ability of likelihood selection to capture semantic meanings.

We apologize for including less detail and lack of clarity regarding this in the main text and will update this more clearly in our final manuscript.

W2: “LLM fails to analyze the relationship between answer candidates”

Thank you for this interesting observation. Our likelihood implementation actually follows prior approaches where all answer candidates are fed together to the language model in addition to organizing the Q-A pairs in a batch dimension. This is illustrated in Table A.4 - we will provide more examples to highlight this better during revision. Taking one simplified toy example, given a question “Where is the dog?” and answers “mat, table, bench”, we use three queries along batch dimension as:

• Where is the dog? Select the correct response from: a) mat, b) table, c) bench. The correct response is a) mat
• Where is the dog? Select the correct response from: a) mat, b) table, c) bench. The correct response is b) table
• Where is the dog? Select the correct response from: a) mat, b) table, c) bench. The correct response is c) bench

In fact, applying likelihood selection without such prompting leads to significantly low performance for some datasets. We show this in Table 6 which we repeat below:

We apologize for any lack of clarity in our paper and will update to highlight this better.

W3a: “Ensure consistent occurrence of objects”

This is an important direction for long video understanding. However, our work is orthogonal to this direction. In fact, taking prior work like LVNet which directly addresses such issues, we can integrate our MVU framework to further improve their performance. We include these results below.

W3b: “Performance on more densely sampled frames”

We run the following ablations focussed on densely sampled frames. Dense sampling improves performance.

In fact, the lightweight object detector and tracker used in MVU allows scaling the number of frames with a lesser increase in inference time.

W4: “Results on Datasets better than EgoSchema”

We evaluate on the LongVideoBench dataset and provide these results below. We select this Phi-3-Vision-Instruct baseline since it is the best performing model we can replicate within our compute budget.

It is clear how MVU combined with this baseline can improve performance.

W5: “Recent state-of-the-art approaches”

We thank the reviewer for pointing these works to us. We are updating Tables 2 and 3 to include these comparisons. We also repeat some results below for quick reference.

Here we apply ours over LVNet given its state-of-the-art performance, open-source implementation, and publicly available pre-computed captions (computing long video frame captions with closed source models can be expensive).

We thank the reviewer again for all valuable feedback helpful for improving our paper further.

References

[1] Min, Juhong, et al. "MoReVQA: Exploring Modular Reasoning Models for Video Question Answering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Wang, Xiaohan, et al. "Videoagent: Long-form video understanding with large language model as agent." European Conference on Computer Vision. Springer, Cham, 2024.

[3] Wang, Ziyang, et al. "VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos."

[4] Ying Wang, et al. "Lifelongmemory: Leveraging llms for answering queries in long-form egocentric videos."

[5] Kahatapitiya, Kumara, et al. "Language repository for long video understanding.".

[6] Wang, Jiawei, et al. "Tarsier: Recipes for training and evaluating large video description models.".

[7] Wang, Yi, et al. "Internvideo2: Scaling video foundation models for multimodal video understanding.".

[8] Park, Jongwoo et al. “Too Many Frames, not all Useful: Efficient Strategies for Long-Form Video QA.”.

We thank the reviewer for their feedback and address all concerns below.

W1a: “Context length limiting input frames not addressed”

It is unclear what context length here means. Assuming it is about LLM context lengths (since we use an LLM in our work), we reiterate that this is specifically a key issue addressed in our work. Instead of naively collecting all information within a frame, our MVU only collects object centric spatial and motion information, allowing to process more frames at a fixed context length. As described in our method and empirically validated in our ablations (see L518-521) our work directly addressed this issue.

In fact, when comparing the average token length for a similarly performing baseline (implemented under identical settings using a common LLM), we use less tokens (context length) to achieve similar results.

This highlights how MVU acts as an information bottleneck to extract useful information - a key design element to support long video processing.

W1b: “Modeling long-range temporal information not addressed”

We rely on the power of the LLM to capture temporal information. We provide our video as a sequence of tokens so the temporal information can be captured by the LLM. This is motivated by prior work such as LLoVi. Fusing our three information types and providing to the LLM such that it can perform all temporal modeling is another key design element enabling long video understanding.

Applying additional techniques for more long-range modeling is orthogonal to our work. In fact, we apply MVU over one such work, LVNet [1] that is capable of processing up to 900 frames. Their extracted information is combined with MVU information to make final predictions, leading to additional performance improvements as shown:

W4: “Table 1 comparisons may be weaker due to the differing training setups and base models”

All numbers in Table 1 are from experiments that we run using identical (pre-trained, publicly available) base models and evaluation settings. There is no training of the pre-trained models in our framework or the baselines used in Table 1. Our reproduced numbers for baselines are also comparable with those in other works [2], validating our implementations. We have already given careful attention to ensure no such discrepancies occur.

W5: “Multi-Frame LLaVa Baseline”

Directly adding more frames to the LLaVA baseline we use (that is trained only on images) does not result in improvements for our tasks.

This reduction of performance by naively adding more information (as frames) is consistent with observations in prior work [3].

W6: “Frame Selection Details” As mentioned in L248, L297, L916, and L955 that we perform uniform frame sampling. We apologize for any clarity issues in Figure 3 and will revise this as shown above.

Updated Manuscript: We have uploaded a new PDF including changes (colored in green) to address all concerns raised by reviewers. Key changes include:

1. Added newer baselines from 2024 and integrating MVU over those baselines (updates to Related Work & Experiments sections, Table 2, Table 3)
2. Clarify nature and role of Likehood Selection (updates to Sec 3.2, Appendix F.2, F.3)
3. Evaluations on datasets containing very long ( > 3 mins) videos (Appendix H)
4. Detailed discussion on LLM context length (Appendix K)
5. Additional ablations and baselines (Appendix I, L)

NOTE: Several of the newly added papers (from 2024) are concurrent work (i.e. published after July 1st, the date set in ICLR guidelines). In fact, some of these works even cite and include results from our paper.