VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation

Many thanks for your insightful feedback and valuable suggestions.

W1: The problem of the indicator.

Thanks for pointing this out. We follow VideoLLM-online and it makes an error. The revised equation is as follows:

L = \frac{1}{N}\sum^N_{j=1}\left(-l_{j+1}\log P_j^{`[Txt$_{j+1`$]}} - \sigma s_j\log P_j^{`[EOS]`}\right),

W2: The reason of marginal improvements over VideoLLM-online and the similar results of full computation.

The online narration benchmark only requires generating simple time-synchronized narrations, such as “You are riding the bike,” without needing detailed descriptions that require fine-grained visual perception. Therefore, the performance improvement with increased resolution is marginal.

However, experiments on other benchmarks, particularly those requiring fine-grained visual perception, demonstrate the necessity and benefit of increased resolution. As shown in Table 5b, for example, Ego Accuracy increased from 40.53% to 44.85% on the EgoExo4D Fine-grained Keystep Recognition benchmarks, indicating the potential of our approach for high-resolution vision tasks.

W3: Some visualizations are redundant, e.g., Figure 1 and Figure 5. There lacks the visualization on the selected visual tokens in different layers. It is necessary to show what is crucial in the learned LayerExpert.

We will reorganize our visualizations in the revised version. We also visualized the selected vision tokens learned by LayerExpert, as shown in Global-Rebuttal Figure 1. LayerExpert effectively focuses on critical vision tokens, such as bike-related tokens in Fig. 1a, tokens related to slicing onions in Fig. 1b, and tokens related to a table saw in Fig. 1c., and the model will attend to different tokens at different layers shown in Fig. 1d.

Q1: Compare with some video LLM works that compress each frame into fewer tokens, e.g., LLaMA-VID [33] in terms of both performance and efficiency.

We compare the performance and efficiency of our approach with LLaMA-VID [1], as shown in Global-Rebuttal Tables 1-4. Our approach achieves comparable performance to LLaMA-VID while requiring only 0.57x FLOPs and 0.2x training time.

We highlight the advantages of our proposed approach over existing efficient vision modeling methods in the LMMs field:

1. Utilizing Fewer, Semantic-Aware Tokens: Semantic-aware token selection typically requires cross-attention-based modeling, which is computationally expensive when processing every frame. As shown in Global-Rebuttal Tables 1-2, LLaMA-VID [1] uses text features extracted from a q-former to select semantic-aware visual tokens via context-attention, resulting in significantly higher training costs (10.5 hours vs. 52.5 hours) with only marginal performance improvement.
2. Efficient Inference: High training costs are a significant issue for LMMs, particularly for video-based LMMs, as video consumes the majority of tokens, as indicated in Figure 7 of our paper. Unlike existing methods [2,3] that focus on efficient inference while still requiring high training costs, we successfully reduce both training and inference costs massively while maintaining LMMs’ performance. Our approach provides a new paradigm for other vision-language tasks, especially in training video-based LLMs, and plays a role in democratizing AI by enabling broader access to trained models without the need for large resources.
3. Offline Spatial-Temporal Token Merging: Online video LLMs must process every incoming visual token in real time and cannot perform frame sampling or offline token merging as Chat-UniVi [4] does. We are the first to explore efficient vision modeling in context rather than merely offline pruning or merging vision tokens.

L1: The authors have discussed the limitations in the lack of experiments on exocentric data.

In addition to extensive experiments on ego-centric datasets, we have conducted experiments on exo-centric COIN benchmarks, as shown in Paper Table 4. To further validate the effectiveness and generalization of our proposed method, we conducted experiments using the same training recipe as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables 1-4. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision-language tasks in the LMMs field.

[1] Yanwei Li et al. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. ECCV 2024.

[2] Liang Chen et al. An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV 2024.

[3] Yuzhang Shang et al. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv: 2403.15388.

[4] Peng Jin et al. Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding. CVPR 2024.

Dear Reviewer x5JZ,

We would like to express our sincere gratitude for the time and effort you spent reviewing our paper. As the author reviewer discussion stage draws to a close, we are eager for your response to ascertain if our detailed response has sufficiently addressed your concerns. We would be honored to address any further questions you may have. We eagerly anticipate and highly value your re-evaluation of our paper.

Thank you once again for your thorough review of our paper.

Best regards,

Authors of Submission 10329

We sincerely appreciate your thoughtful feedback and valuable suggestions.

W1: The technical contribution of this method is relatively limited.

Our technical contributions are summarized as follows:

1. Efficient Vision Modeling in Context: It is non-trivial to reduce computation in context under online video scenarios, as online VideoLLMs must process every incoming visual token without perform frame sampling or offline token merging, as done in Chat-UniVi [1]. We are the first to explore efficient vision modeling in context, rather than merely offline pruning or merging vision tokens.
2. Reduction of both Training and Inference Costs: High training costs are a significant issue for LMMs, particularly for video-based LMMs, since video consumes the majority of tokens, as indicated in Figure 7 of our paper. Unlike existing methods [2,3] that focus on efficient inference while still requiring high training costs, we are the first to successfully reduce both training and inference costs massively while maintaining LMMs’ performance. We believe our approach offers a new paradigm for other vision-language tasks, especially in training video-based LLMs.
3. Generalization: To validate the effectiveness and generalization of our proposed method, we conducted experiments using the same training recipe as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables 1-4. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision-language tasks in the LMMs field.

The topic of sparse vision modeling in context has gained popularity, as evidenced by Google DeepMind’s recent research on MoNE [4], released a few days ago. While they adopt a similar approach, their exploration is limited to traditional vision architectures rather than the popular LMMs field. We believe our idea "the importance of different vision tokens should be considered in context, and can be modeled by computation budget" can provide valuable insights into general LMMs.

W2: The discussion on the choice of LayerExpert, the visualization to prove the claim, the semantic-aware token selection strategies for comparison.

Forcing LayerExpert to select only the important visual tokens across each transformer block can be viewed as encouraging LMMs to focus on the most useful regions, while it increases the learning difficulty. As shown in Global-Rebuttal Figure 1, we visualize the tokens selected by LayerExpert and observe that it indeed focuses on important visual regions, such as bike-related tokens in Fig. 1a, tokens related to slicing onions in Fig. 1b, and tokens related to a table saw in Fig. 1c.

Semantic-aware token selection typically requires cross-attention-based modeling, which is computationally expensive when processing every frame. As shown in Global-Rebuttal Tables 1-2, LLaMA-VID [5] utilizes text features extracted from a q-former to select semantic-aware visual tokens via context-attention, resulting in significantly higher training costs (10.5 hours vs. 52.5 hours) with only marginal performance improvement.

Q1: Why increasing the keep ratio r=0.3 results in worse performance? Also, it would be good to see the performance comparison on more keep ratios.

First, selecting important visual tokens via LayerExpert increases the learning difficulty, leading to better performance with fewer visual tokens. Second, there may be deviations in the LMMs’ training process. We will conduct additional trials in the revised version and report the standard deviation across all experiments.

Further ablation studies on the keep ratio are presented below.

Q2: Why the proposed method targets at online video processing?

Processing video in an online scenario differs from offline processing, as LMMs must handle every incoming frame in real-time without frame sampling or token merging. This results in excessive vision token lengths, as shown in Figure 7 of the paper, and the training costs for LMMs increase exponentially with token length. Moreover, online VideoLLMs like GPT-4o have shown great potential in real-world applications, highlighting the urgent need to explore approaches that reduce both training and inference computational costs in online video scenarios.

To validate the effectiveness and generalization of our proposed method, we conducted experiments using the same training recipe as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables 1-4. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision-language tasks in the LMMs field.

[1] Liang Chen et al. An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV 2024.

[2] Yuzhang Shang et al. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv: 2403.15388.

[3] Peng Jin et al. Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding. CVPR 2024.

[4] Gagan Jain et al. Mixture of Nested Experts: Adaptive Processing of Visual Tokens. arXiv: 2407.19985.

[5] Yanwei Li et al. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. ECCV 2024.

Dear Reviewer qExK,

We would like to express our sincere gratitude for the time and effort you spent reviewing our paper. As the author reviewer discussion stage draws to a close, we are eager for your response to ascertain if our detailed response has sufficiently addressed your concerns. We would be honored to address any further questions you may have.

Thank you once again for your thorough review of our paper.

Best regards,

Authors of Submission 10329

Dear reviewer qExK:

Thank you again for your thoughtful feedback! We hope the rebuttal and additional experiments we provided were helpful. As the rebuttal period draws to a close, please don't hesitate to contact us if you have any problems.

Thank you once again for your thorough review of our paper!

Best regards,

Authors of Paper10329

We sincerely appreciate your time and efforts in reviewing our paper. We will address each of your concerns point by point.

W1/Q1: This novelty and contributions compared to Mixture-of-Depths.

We summarize the differences between our proposed framework and Mixture-of-Depths (MoD) as follows:

1. Validation of Effectiveness on LMMs: It is non-trivial to validate the effectiveness of the proposed approach in Large Language Models(LLMs) especially in Large Multi-modal Models(LMMs). While MoD demonstrates comparable loss scales on language models smaller than 1B parameters, we explore efficient methods to reduce computation in context for 8B-LMMs in both training and inference, particularly under online video scenarios.
2. Causal Operations: While MoD introduces causality through auxiliary loss or auxiliary MLP predictors, we simplify this by using LayerExpert to select vision tokens within individual frames across transformer blocks. The causal attention in the LLM then learns the temporal modeling, seamlessly accommodating our online video scenario.
3. Extensive Experiments: Unlike MoD, which validates its approach on language models smaller than 1B parameters on loss scale, we conducted extensive experiments on vision-language benchmarks, including the Ego4D narration benchmark, Ego4D LTA benchmark, EgoExo4D Fine-grained Keystep Recognition task, and COIN benchmarks. Additionally, we performed experiments on general image/video benchmarks, as shown in Global-Rebuttal Tables 1-4, demonstrating the generalization of our proposed approach to vision-language tasks.

The topic of sparse vision modeling in context has gained popularity, as evidenced by Google DeepMind’s recent research on MoNE [1], released a few days ago. While they adopt a similar approach, their exploration is limited to traditional vision architectures rather than the popular LMMs field. We believe our idea "the importance of different vision tokens should be considered in context, and can be modeled by computation budget" can provide valuable insights into general LMMs.

W2: Experiments are mainly on egocentric and instructional datasets, lack of Diversity.

To validate the effectiveness and generalization of our proposed method, we conducted experiments using the same training recipe as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables1-4. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision-language tasks in the LMMs field.

[1] Gagan Jain et al. Mixture of Nested Experts: Adaptive Processing of Visual Tokens. arXiv: 2407.19985.

Dear Reviewer 2CSB,

We would like to express our sincere gratitude for the time and effort you spent reviewing our paper. As the author reviewer discussion stage draws to a close, we are eager for your response to ascertain if our detailed response has sufficiently addressed your concerns. We would be honored to address any further questions you may have. We eagerly anticipate and highly value your re-evaluation of our paper.

Thank you once again for your thorough review of our paper.

Best regards,

Authors of Submission 10329

Dear reviewer 2CSB:

Thank you again for your thoughtful feedback! We hope the rebuttal and additional experiments we provided were helpful. If any residual concerns remain, we would be glad to discuss further. If no concerns remain, we would appreciate it if you re-evaluate our paper.

Thank you once again for your thorough review of our paper.

Best regards,

Authors of Paper10329

We sincerely appreciate your positive comments on our work and will address each of the issues you mentioned below.

W1: Performing experiments to show how does this MoD approach for making a sparse vision encoder work on standard benchmarks.

To validate the effectiveness and generalization of our proposed method, we conducted experiments using the same data configuration as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables 1-4.

Specifically, for video benchmarks, we trained our VideoLLM-MoD on the ActivityNet and WebVid-2.5m datasets and further evaluated it on several other benchmarks. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision tasks in the LMMs field.

Q1: Code availability.

The project code is included in the supplementary materials. We will release all the code, data, and checkpoints as soon as possible.

L1: How does the author suggest the method can be extended for grounding?

Our proposed method can process more vision tokens within the same computational budget. For spatial grounding tasks, this capability allows for larger spatial resolutions and denser frame representations. More vision tokens facilitate finer-grained representations of images and videos, capturing more detailed visual features. This is crucial for accurately locating and distinguishing small objects or intricate details within complex scenes, thereby improving the model’s precision in such scenarios.

Dear Reviewer HwD3,

We would like to express our sincere gratitude for the time and effort you spent reviewing our paper. As the author reviewer discussion stage draws to a close, we are eager for your response to ascertain if our detailed response has sufficiently addressed your concerns. We would be honored to address any further questions you may have.

Thank you once again for your thorough review of our paper.

Best regards,

Authors of Submission 10329