Unifying Specialized Visual Encoders for Video Language Models

W5: How do you verify that the video-language alignment was not very strong for the MERV (frozen) recipe?
MERV (frozen) does not use stage 1 data, which largely consists of captioning and video description tasks focused on visual alignment. Although our testing benchmarks don’t reveal a stark difference in the results, we do see some performance hits on other benchmarks. In our previous explorations, the distributions of language used in video datasets and benchmarks also seem to sparsely overlap based on their sentence embeddings, which could have impacted our ability to perform well zero-shot on the downstream benchmarks (especially difficult, more OOD ones). Therefore, we make this observation mainly based on the difference in data used to train MERV (frozen) and MERV (full), as our assumption is that more data will help further with this alignment gap. We add this clarification in the revised manuscript (lines 387-390).

Q1: Are single encoder models in Figure the variants of MERV?
In all of the Figures, any single-encoder models are our MERV architecture and recipe except with a single encoder and no feature fusion. The only exception is in Figure 4a, where we plot the models with their full embeddings to also demonstrate the impact of our pre-fusion projectors on performance and efficiency (line 422).

Q2: What is the role of P_e?
Role of P_e is to differentiate between different encoders, where we call it positional embedding for each encoder following the similar notation used in other attention models. We have added them so that the feature fusion module has additional knowledge on which key belongs to which encoder, but we have seen that P_e does not have a strong impact on the performance. Thank you for pointing out, we decided to remove this for clarity.

Q3: Does it take 24 hours with 8 L40-48GB GPUs for MERV (frozen) or MERV (full)?
It takes around 24 hours on MERV (frozen), while MERV (full) takes 56 hours as they are trained using 2 stages with a larger dataset.

Q4: Table 1. Add columns to specify video encoder and LLM used. TGIF missing for Video-ChatGPT.
Thank you for the suggestion, we added the video encoder and LLM used by the model in Table 1 (also in Table 3 and Table 4). Regarding TGIF performance for Video-ChatGPT, we apologize for the confusion. We decided to remove TGIF performance from Video-ChatGPT as they were evaluated on the FrameQA subset of the TGIF instead of the full test set (https://github.com/PKU-YuanGroup/Video-LLaVA/issues/37#issuecomment-1900058137), but did not update the main text correspondingly. We also removed the point about score (line 308 on the previous draft), as now the accuracy and score trends more consistently.

Q5: Does 3D Conv in Table 2 (a) apply a 2D 3x3 convolution?
Thanks for catching this typo, it is a 3D Conv, i.e. 2x3x3 (FxHxW). We have added additional clarifications for this in the beginning of Section 4.2.1 on our pre-fusion projector ablations.

Dear Reviewer RqfQ, we deeply appreciate your valuable reviews. We are encouraged that you appreciate our model's performance that outperforms existing baselines on different benchmarks, extensive ablation studies, and empirical experiments and results that can answer the questions in a controlled and scientific manner. Please see our responses to the weaknesses you pointed out below:

W1: MERV or MERV-full.
Given that MERV (full) is trained on more data, we believe that it will offer better generalization. We have revised the manuscript to make this recommendation (line 299).

W2: ViViT is an outdated video encoder.
Thank you for the suggestion. Given limited timeline, we conducted experiments using one newer SoTA video model Hiera [1], i.e. MViTv3, in a single-encoder setting, four-encoder setting replacing ViViT, and a five-encoder setting. The checkpoint we use was pretrained using the Masked Autoencoder (MAE) self-supervised learning technique on Kinetics 400 (K400), and then finetuned on K400 using supervised learning. On the other hand, ViViT was initialized from an ImageNet-21K/JFT backbone, and trained on K400 using supervised learning, so their training setups are similar.

The results are presented in the table below, with the best result in bold and the second-best in italics. Replacing ViViT with Hiera demonstrates improvements on the fine-grained spatiotemporal reasoning benchmark, Perception Test, with accuracy gain of 1.29%. Similarly, adding Hiera yields an improvement on ActivityNet, achieving a 0.36% increase in accuracy. However, on other benchmarks, the original MERV remains the strongest model. Overall, we observe no significant performance improvement when training with Hiera, which aligns with expectations, since Hiera is under the same paradigm as ViViT, functioning as a temporal expert trained on short videos. We also hypothesize that Hiera is more sensitive to the temporal stride than ViViT, as ViViT can reasonably deduce motion from uniformly sampled frames. We expect performance to improve if we incorporate encoders trained on different paradigms and data sources or process a much greater number of frames simultaneously, which we will leave for future work.

[1] Ryali et. al., Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles, ICML 2023.

W3: Distinctive Differences among Contrastive Models.
We have included qualitative analysis for each of the visual encoders in the appendix. Figure 12 visualizes videos that give the highest attention weight for each of the encoders. By looking at which videos that the feature fusion module assigns high attention weight, we can see what type of videos that each encoder specialize on. We see that SigLIP is preferred when given text-heavy videos, while ViViT is preferred when the video contains a lot of temporal motion. Both LanguageBind and DINOv2 are preferred with static videos, but LanguageBind seems to be more preferred when there are meaningful foreground motions.

W4: Elaborate on how to make feature extraction and projection happen in parallel.
Thank you for the feedback, we’ve discussed our use of FSDP in greater detail in Section 4.2.3 under Training Recipes now. Simply put, we use the default FSDP sharding strategy in PyTorch as it is not currently possible to specify an explicit plan for which modules go where (but this may be possible as FSDP matures). However, we design our code so that no operations block each other in this step, so that FSDP can find an optimal sharding strategy automatically (which we believe it does, as shown by our small overhead in the new Figure 3).

Q1: Runtime difference between single encoder vs multiple encoders?

The following table shows the runtime of individual encoder models:

We provide the detailed runtime numbers on 8xL40-48GB GPUs in the tables here and also in our revised manuscript in Figure 3. We track the step time as we progressively add more encoders in a multi-encoder architecture, with the ordering of DINOv2, LanguageBind, SigLIP, and ViViT. We also compare each multi-encoder model (top row in the table below) with an equivalent single encoder model which has the slowest time (bottom row in the table below). For example, we compare DINOv2+LanguageBind with the time of the LanguageBind-only model, as this provides the most equitable comparison of what a theoretical best runtime would be with perfect parallelization (e.g. hiding the DINOv2 encoder runtime entirely under the LanguageBind). Note that this theoretical best runtime doesn’t take into account the overhead of the feature fusion, which remains the largest source of incurred cost remaining.

Comparing the runtime numbers in the two tables, adding multiple encoders only marginally increases the runtime, thanks to our efficient implementation and a feature fusion strategy that does not increase the token length. (The 1 stdev variance is variable of the machine, which tends to happen even after a few iterations of warmup.)

Q2: Fusion Module Significance. What scenarios does our method perform better than the naive techniques?

We have tested different feature fusion strategies and tabulated their performance in Table 2.c and Table 6.c in the appendix. We find our methods to be better performing in overall performance (17.19 T Flops, accuracy 56.83), while having less computation than sequence-wise concatenation (43.09 T Flops, accuracy 54.45), and on-par with channel-wise concatenation (16.29 T Flops, accuracy 56.64). Additionally, our feature fusion module performed better or on-par with channel-wise concatenation in 3 out of 4 benchmarks that we have tested (Table 6.c), having robust performance towards unseen datasets.

Moreover, similar to Figure 5 (Figure 3 in the original submission), we have looked at the performance difference on WH-words of three open-ended benchmarks.

This shows that our method performs consistently better over all but one the visual tasks of these three benchmarks.

We hypothesize that channel-wise concatenation may be preferable to cross-attention in test scenarios that differ significantly from training and demand intensive, fine-grained spatiotemporal reasoning. This is because concatenation naively retains all information with equal importance, whereas cross-attention selectively learns to prioritize information through soft attention, which may fail to generalize effectively when trained on limited training datasets.

Dear Reviewer ccLB, We deeply appreciate the time and the effort that you have shown on the review. Specifically, we thank you for appreciating our multi-encoder approach, extensive experimental studies for different encoder combinations and training stages, improvements to existing baselines, and lightweight, resource-friendly design. Please see our responses to the weaknesses you pointed out below:

W1: The authors have not provided sufficient reasoning to demonstrate the novelty of extending this approach to video.

We agree that we are not the only ones working in this space of multiple visual encoders, which has very recently become popular. We want to re-emphasize that we have two video models in our mixture, as both ViViT and LanguageBind are trained on and embed videos. Additionally, while our method seems simple, we performed extensive ablations to find that such simple approaches are both enough and necessary for flexibly and efficiently modeling videos using multiple image and video encoders (Section 4.2, Table 2). We conducted a very in-depth analysis of how the behaviors of these image-based models differ from video-native models when processing images (Section 5, Figure 4,5,6,12). Prior VideoLLMs largely rely on image models for visual understanding, a paradigm we aimed to challenge with our work. We showed concrete evidence of how useful video models can be for temporally sensitive tasks such as on Something-Something v2 (Figure 4.b, Figure 6), and we added additional analysis per other reviewer’s requests for patterns to how the feature fusion chooses encoders (see the below discussion in W2). Some of our discussion is enabled by our specific feature fusion design, which departs from other conventions in VideoLLM design (e.g., line 346 to line 350).

However, it is also not straightforward to simply incorporate any video encoder for this to work. As requested by other reviewers, we performed new ablations on using extra video encoders, and found that using a new video encoder with similar knowledge areas to our pre-existing encoders does not help much. For example, adding or substituting ViViT with Hiera [1], a short-video encoder similar to ViViT, results in comparable but overall slightly worse gains over VideoLLaVA than our current MERV (Results are shown below, with the best in bold and the second-best in italics).

[1] Ryali et. al., Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles, ICML 2023.

W2: The proposed feature fusion module lacks a clear advantage. Additional statistical significance testing should be considered.

Thank you for pointing that out. We agree that it is important to show that our method performs better than the concatenation method. We re-ran our method and the concat method using two more random seeds. The mean accuracy and standard deviation are shown in the table below. We find that our method is still better than the concat method on average. We ran a paired t-test which returned $p=0.063$; thus, the difference is trending towards significance, but due to compute limitations and the limited timeline we could only do 3 seeds (we’re working on more). Another additional benefit of cross-attention is having accessible encoder weightings for analysis, so we do not choose channel-wise concatenation as our final design.

To this point, the cross-attention method provides explicit attention weights that we can look at to know which videos the feature fusion tends to weigh the most, and look for trends among those. In this setting, we checked the attention weights of all the videos in 4 of our benchmark datasets (MSRVTT, MSVD, TGIF, and Perception Test). For each encoder, we visualized the videos which give the highest attention weight for that specific encoder. The qualitative results are added in the appendix of the paper (Figure 12). As a summary, we found that ViViT attends to videos with large motion, whereas SigLIP is more towards videos with text data (and often more static videos). Meanwhile DINOv2 and LanguageBind have a bias towards videos with static scenes, but LanguageBind attends more often to videos with foreground motion thanks to its video pretraining. This also demonstrates the benefit of using a cross-attention feature fusion, which allows us to analyze the model in this way.

W2: The only video backbone, ViViT, has a slight influence on the performance while it performs well in SSv2. Unclear extensibility of the proposed method to additional visual backbones.

To answer the first question, we have included two video-encoders, ViViT and LanguageBind, in the original submission. While LanguageBind shows a decent influence on the final performance, ViViT seems to be lacking, as shown in Table 7 (Table 5 in the original PDF). We hypothesize that since ViViT is pre-trained only with videos on a classification task, it might relatively lack the vision-language semantic understanding required for the Video-QA tasks in Table 7. However, as seen in Figure 4b, ViViT demonstrates superior performance on tasks where temporal sensitivity and understanding are critical, thus enhancing MERV's performance on such tasks. While the contribution of ViViT may appear small on some of the Video-QA benchmarks we tested, this may also be due to the relative scarcity of the previously mentioned tasks in these benchmarks (e.g., researchers have found that ImageLLMs, though lacking any explicit temporal modeling or training, could have high performance on benchmarks like MSVD-QA [2][3]). We do not believe that ViViT has a slight influence but rather that it plays a very important role, as it is specialized in skills that other encoders lack.

As for the second question, we found that the additional visual encoder is only helpful if the newly added encoder brings visual expertise that is not covered by the existing choice of encoders. We conducted new experiments using another SoTA video model Hiera-B+ [1], i.e. MViTv3, under the same visual expertise as ViViT (see Table above in W1), specialized in temporal understanding, both trained with Kinetics-400. We see that replacing ViViT with Hiera demonstrates improvements on the fine-grained spatiotemporal reasoning benchmark, Perception Test, with accuracy gain of 1.29%. Similarly, adding Hiera yields an improvement on ActivityNet, achieving a 0.36% increase in accuracy. However, on other benchmarks, the original MERV remains the strongest model. Overall, we observe no significant performance improvement when training with Hiera, which aligns with expectations, since Hiera is under the same paradigm as ViViT, functioning as a temporal expert trained on short videos. We also hypothesize that Hiera is more sensitive to the temporal stride than ViViT, as ViViT can reasonably deduce motion from uniformly sampled frames. We expect performance to improve if we incorporate encoders trained on different paradigms and data sources or process a much greater number of frames simultaneously, which we will leave for future work. Nevertheless, we have demonstrated the easy extensibility of our proposed method to additional visual backbones.

[2] Kim et. al., An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM, 2024.
[3] Xu et. al., SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models, 2024.

c) Finally, we respectfully disagree that increased computational costs directly lead to performance improvements. We provide three points of evidence from our ablations. First, in Table 2(a), we provide a total of six 8-frame projector settings, the first of which (257 tok) is equivalent to the original VideoLLaVA. Three of the remaining five settings led to worse performance, despite using more params and FLOPs than our 2D average, which was unintuitive for us as well given their strong performance for other domains. In fact, the most computationally demanding projectors like a 2D Conv did not perform as well as the zero parameter 2D Average Pool. A similar finding which is a classic issue with video models is ablating the number of tokens used, as using multiple frames quickly exhausts the model’s capacity. We found that 64 tokens was the best performing setting, which was counter to what previous works such as Video-LLaVA did with full token embeddings. Finally, we also performed a new set of ablations on using extra video encoders, and found that using a new video encoder with similar knowledge areas to our pre-existing encoders does not help much. For example, adding or substituting ViViT with Hiera [1], a short-video encoder similar to ViViT, results in comparable but overall slightly worse gains over VideoLLaVA than our current MERV, indicating that adding or choosing encoders naively does not improve performance as effectively as our current approach (Results are shown below, with the best in bold and the second-best in italics.).

[1] Ryali et. al., Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles, ICML 2023.

As reviewer RqfQ has said, there are a flood of VideoLLMs that are trained using their own data mixture and recipes, which makes our paper particularly valuable as we made non-negligible efforts in providing fair comparisons and extensive empirical results answering various questions especially for VideoLLMs using multiple vision encoders. Ours is the first work that establishes the training framework for using multiple encoders for VideoLLM, and we found that some methods which work for VLMs such as the C-Abstractor or even other VideoLLMs like Perceiver Resamplers do not work well for multi-encoder representation learning of videos.

We hope that our attention to detail and extensive analysis to the specific behaviors of our model sets our method apart from the others, as we believe that these types of details matter quite a lot. We believe current video LLM works could provide deeper analysis of their methods and differences, so we went to great lengths to make these comparisons as fair and general as possible so that the community can find broad insights.

Dear Reviewer,

Thank you for giving us your time and energy for the thorough and insightful review of our paper. We are encouraged that you find our motivation reasonable, appreciate our technical details and reproducibility needed to run our many experiments, and the presentation of method and results clear. Please see our responses to the weaknesses you pointed out below:

W1: Lack of novelty for three reasons: a) Similarity to Prismer for ensembling (which is a basic idea), b) Lack of in-depth analysis of feature fusion interactions, and c) Unsurprising performance given increased computational costs.

Thank you for this comprehensive set of points, and we agree that these are important points to address for our method. We will discuss them in detail one by one.

a) We appreciate the reviewer for sharing the Prismer paper; we add it to our related work for relevance to our work (line 142). However, there are a few differences we think are pertinent. First, we would classify Prismer as a multimodal work, i.e. it uses segmentation, depth, normals, etc. as input, whereas ours strictly uses RGB as visual input, which we attempt to quantify as “multi-encoder”. We explore how different objectives and datasets create different visual experts, rather than the task itself. Second, Prismer doesn’t handle video, and their Experts Resampler is effectively a Perceiver Resampler for reducing the multimodal input to a fixed token length. We actually tested this type of approach (line 316), but we found that such approaches which are agnostic to structure lead them to being weaker compared to other alternatives (line 348). This reinforces our belief that a structure-aware spatio-temporal alignment is crucial for multi-encoder models to perform well to deal with videos (see point (c)). Finally, while we find Prismer a pioneering work in this aspect, we do acknowledge that they found great difficulty in improving the performance with more experts. Even with four additional experts, their VQAv2 accuracy improves from 72.17% to 72.79%, a gain of 0.62%, whereas we see average accuracy increases of 2.5% on top of the strongest single-encoder methods. We believe that these points are enough to significantly differentiate our methods and findings from Prismer.

b) With regard to analysis of feature fusion interactions, we found that the model is finding the best weights to perform a general unification of the features, rather than learning large changes in weights for different classes of examples to perform single or dual encoder selection. This is understandable from the objective, but also that individual videos can give rise to many different tasks requiring different skills, and a priori of the task, the model simply tries to learn the best representation for all potential questions.

Nevertheless, we can still look at specific videos for which the feature fusion module tends to rely more heavily on particular encoders, and look for trends among those. In this setting, we checked the attention weights of all of the videos in 4 of our benchmark datasets (MSRVTT, MSVD, TGIF, and Perception Test). For each encoder, we visualized the videos which give the highest attention weight for that specific encoder. The qualitative results are added in the appendix of the paper (Figure 12). As expected, ViViT attention weights are highest on videos with large motion, as ViViT has strong temporal motion understanding. Meanwhile, SigLIP, which is vision-language contrastively trained, attends more to videos that have textual data in the video. DINOv2 and LanguageBind are both preferred by videos with static scenes, but LanguageBind, being trained on videos, is preferred by videos with some foreground motion.

Dear Reviewer, Thank you for giving us your time and energy for the thorough and insightful review of our paper. We are encouraged that you appreciate the importance of our motivation, the in-depth ablation studies across our settings, and the clear presentation of method and results. Please see our responses to the weaknesses you pointed out below:

W1: Does MERV still improve when we use more encoders?

Thank you for the interesting question. In response to both your question and Reviewer RqfQ’s question about using more modern video encoders, we conducted experiments using a newer SoTA video model Hiera-B+ [1], i.e. MViTv3, in both a four-encoder setting replacing ViViT and a five-encoder setting. The checkpoint we use was pretrained using the Masked Autoencoder (MAE) self-supervised learning technique on Kinetics 400 (K400), and then finetuned on K400 using supervised learning. On the other hand, ViViT was initialized from an ImageNet-21K/JFT backbone, and trained on K400 using supervised learning, so their training setups are similar.

The results are presented in the table below, with the best result in bold and the second-best in italics. Replacing ViViT with Hiera demonstrates improvements on the fine-grained spatiotemporal reasoning benchmark, Perception Test, with accuracy gain of 1.29%. Similarly, adding Hiera yields an improvement on ActivityNet, achieving a 0.36% increase in accuracy. However, on other benchmarks, the original MERV remains the strongest model. Overall, we observe no significant performance improvement when training with Hiera, which aligns with expectations, since Hiera is under the same paradigm as ViViT, functioning as a temporal expert trained on short videos. We also hypothesize that Hiera is more sensitive to the temporal stride than ViViT, as ViViT can reasonably deduce motion from uniformly sampled frames. We expect performance to improve if we incorporate encoders trained on different paradigms and data sources or process a much greater number of frames simultaneously, which we will leave for future work.

[1] Ryali et. al., Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles, ICML 2023.

W2: Is there a pattern to how the feature fusion chooses encoders for certain tasks?

This is something we wondered as well, as we were hoping that feature fusion could do a selection of encoders. Our analysis found that the model instead is finding the best weights to perform a general unification of the features, rather than learning large changes in weights for different classes of examples. This is understandable given the objective, but also that individual videos can give rise to many different tasks requiring different skills, and a priori of the task, the model simply tries to learn the best representation for all potential questions.

Nevertheless, we can still look at specific videos for which the feature fusion module tends to rely more heavily on particular encoders, and look for trends among those. In this setting, we checked the attention weights of all the videos in 4 of our benchmark datasets (MSRVTT, MSVD, TGIF, and Perception Test). For each encoder, we visualized the videos which give the highest attention weight for that specific encoder. The qualitative results are added in the appendix of the paper (Figure 12). As expected, ViViT attention weights are highest on videos with large motion, as ViViT has strong temporal motion understanding. Meanwhile, SigLIP, which is vision-language contrastively trained, attends more to videos that have textual data in the video. DINOv2 and LanguageBind are both preferred by videos with static scenes, but LanguageBind, being trained on videos, is preferred by videos with some foreground motion.