LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

Thanks to your valuable comments, we have added a number of experiments to the paper to demonstrate the effectiveness of LanguageBind, these are shown in $\color{red}{red}$ font in revised paper.

Q1: Please provide a more fair comparison experiment in Table 2?

A1: Thank you for your valuable feedback. We greatly appreciate your suggestion, which has allowed us to enhance the quality of our paper. In response to your recommendation, we have made several updates to the comparative model of LanguageBind in Table 2 and Table 3 in the revised version. In Table 3, we have employed the CLIP4CLIP model structure for training on both the 380K Howto100M dataset and the 100K VIDAL-10M dataset. The results clearly demonstrate that VIDAL-10M yields superior experimental outcomes, thereby substantiating the reliability and effectiveness of the data. Regarding Table 2, we acknowledge that the comparison between LanguageBind-Large and ImageBind-Huge may have been slightly unfair due to the novelty of the approach and the different problems in focus. To address this concern, we introduced the InternVideo model in Table 2, which used more data and a well-designed modeling structure. Our findings reveal that LanguageBind achieves remarkable performance surpassing that of InternVideo while utilizing only 1/4 of its data volume.

Q2: How to verify whether the performance improvement comes from the proposed dataset or the new pretraining paradigm?

A2: Thanks for your question. We added more ablation experiment between LanguageBind and full parameter fine-tuning and train from scratch in Table 8(c). Training from scratch exhibits the poorest performance, likely due to the lack of prior knowledge from CLIP pretraining. On multiple datasets such as LLVIP, FLIRv1, and Clotho, we found that full-parameter fine-tuning based on CLIP weights did not perform as well as LanguageBind. Meanwhile, the LoRA method stands out for its advantages in terms of time and memory cost. This indicates that LanguageBind is not only efficient but also effective in learning new knowledge specific to different domains.

Q3: Can the authors provide intuition why why raw caption works best for the Infrared modality?

A3: Thanks for your question. We suppose that for modalities with limited scene diversity, the accurate and brief text may work better, i.e., the hashtags in raw captions for depth and infrared images. According to our research, textual data from different sources often have complementary roeeles. For modalities with diverse scenes, such as video, image, text, etc., a mixture of text from different sources may work best. While the scenerios with limited diversity shows the contrary, as the infrared modality mentioned in the quesiont. In addition, since the data of the downstream task of the infrared modality is relatively small, the fluctuation in training can easily affect the final experimental results. Thank you for your question, which is an interesting one for us to follow up and extend to other modalities we want to understand more deeply.

Thanks to your valuable comments, we have added a number of experiments to the paper to demonstrate the effectiveness of LanguageBind, which are shown in $\color{red}{red}$ font in revised paper.

Q1: Please clarify the fairness of the comparison experiments in Table 2?

A1: Thank you for your valuable feedback. We greatly appreciate your suggestion, which has allowed us to enhance the quality of our paper. In response to your recommendation, we have made several updates to the comparative model of LanguageBind in Table 3 in the revised version. In Table 3, we have employed the CLIP4CLIP model structure for training on both the 380K Howto100M dataset and the 100K VIDAL-10M dataset. The results clearly demonstrate that VIDAL-10M yields superior experimental outcomes, thereby substantiating the reliability and effectiveness of the data. More experimental results on data scaling based on CLIP4Clip are presented in Appendix E.2.

Q2: How to validate that the data can use all available modalities to learn stronger representations?

A2: This is an intriguing question. In Table 6, we guided joint retrieval experiments and demonstrated that the results of Video+Audio->Text, Depth+RGB->Text, and Infrared+RGB->Text outperform the performance of individual modalities. These results validate that LanguageBind has learned a unified semantic space and can utilize all available modalities to acquire stronger representations.

Q3: Can LanguageBind address one of the following problems: a) one of the modalities is corrupted; b) one of the modalities has some weaknesses; c) one of the modality undergoes a domain change while the other doesn't?

A3: LanguageBind learns a unified representation using available modalities, which enables the model to exhibit strong robustness when a modality undergoes a domain change. For instance, in the downstream task of nighttime pedestrian detection sourced from the LLVIP dataset, RGB images captured in the dark are used. In Table 6, we demonstrate that the performance of dark images->text is 62.4. When infrared is incorporated, the performance increases to 79.3. This showcases the advantage of binding multiple modalities into a semantic space, as it demonstrates robustness when another modality undergoes changes a domain change.

Q4: Can authors provide experiments for retrieval tasks where the modalities doesn't contain text?

A4: Yes, to evaluate the quality of the learned unified representation, we present the results of emergent zero-shot retrieval in Table 7. This includes Video -> Audio, Image -> Infrared, Image -> Depth, and the performance when incorporating text embedding. On each benchmark, the performance of emergency zero-shot retrieval achieves significant gains, even approaching results obtained by incorporating textual features. These results suggest that LanguageBind aligns various modalities and implicitly transfers text supervision associated with specific modalities and tasks.

Q4: How the training strategy influence the training results? To be specific, what is the results of full tuning? And what happens if the modality encoders are not initialized with openCLIP checkpoints

A4: Considering that our VIDAL-10M dataset is relatively large, we finally adopted the LoRA technique and the Vit-Large framework instead of Vit-Huge like ImageBind in order to accelerate the training speed and save computational resources. In view of your helpful suggestions, we decide to conduct some full tuning experiments to demonstrate the experimental performance of our method in this aspect. As indicated by Table 8, we compare three different initialization strategies. Training from scratch exhibits the poorest performance, likely due to the lack of prior knowledge from CLIP pretraining. On the other hand, full tuning shows significant improvement compared to training from scratch. This highlights the positive impact of leveraging prior knowledge in the form of pre-trained weights. Meanwhile, the LoRA method stands out for its advantages in terms of time and memory cost. It requires less time and memory resources compared to full tuning. Furthermore, LoRA outperforms full tuning on multiple datasets such as LLVIP, FLIRv1, and Clotho. This indicates that LoRA is not only efficient but also effective in learning new knowledge specific to different domains while better retaining the previously acquired knowledge from the pre-trained OpenCLIP models.

Q5: Why the preformance drops when moving from 0.5 -> 0.3 masking? What happens witho no masking?

A5: In our ablation experiments, we conducted experiments with a mask ratio of 0 and validated them on additional datasets. Interestingly, it is abnormal that performance decreases with an increase in the visible area for human understanding of images. However, from a model learning perspective, particularly in terms of data augmentation and regularization, applying a certain ratio of masking to images is a natural form of data augmentation. It forces the model to extract features from different patch tokens with each instance. A similar example is FLIP, where they achieved better results in image-text retrieval with a mask ratio of 0.5 compared to a mask ratio of 0.0, which aligns with our findings.

We greatly appreciate your valuable feedback, which has led us to include some additional meaningful and interesting experiments in our paper. You can find them highlighted in $\color{red}{red}$ font in the revised version of the paper. We would greatly appreciate it if you could take the time to review the revised document.

Q1: What is the benefit of training all modalities in a joint embedding space and the difference of previous CLIP-based works?

A1: In order to effectively showcase the indirect alignment and multi-modal complementary capabilities of LanguageBind, we have added relevant experimental findings, as depicted in Table 6 and Table 7.

• In Table 6, we guided joint retrieval experiments and demonstrated that the results of Video+Audio->Text, Depth+RGB->Text, and Infrared+RGB->Text outperform the performance of individual modalities.

• In table 7, we present the results of emergent zero-shot retrieval in Table 7. This includes Video -> Audio, Image -> Infrared, Image -> Depth, and their performance when incorporating text embedding. On each benchmark, the performance of emergent zero-shot retrieval achieves significant gains, even approaching results obtained by incorporating textual features.

• These experimental results affirm that LanguageBind not only exhibits commendable indirect alignment proficiency but also demonstrates remarkable modal complementarity. This substantiates the exceptional quality of LanguageBind's alignment within the shared semantic space.

Q2: What is the main contribution of this work?

A2: Our primary objective behind proposing LanguageBind is to establish alignment between various modalities and a Language-centric semantic space. This approach is motivated by the fact that numerous contemporary multimodal downstream tasks are inherently interconnected with Language. In table 6 and table 7, by aligning modalities within a shared semantic space, we can effectively leverage the complementarity and alignment between these modalities, thereby enhancing the performance of individual modalities in downstream tasks, as discussed in the initial question. Furthermore, aligning other modalities to the Language modality offers a potential advantage: LanguageBind has the potential to facilitate the development of LLM-driven full-modal Large Models, following the footsteps of PandaGPT, LLaMa-AdapterV1, NeXtGPT, and similar research initiatives.

Q3: What is the significance and future development of proposed VIDAL dataset, since part of the modalities are automatcally generated?

A3: When it comes to building depth and infrared images using generative models, there are several key considerations that we have taken into account. These considerations, among others, include the following two points:

• Limited data size and categorical labels: Depth and infrared data often exhibit a relatively small-scale dataset with text labels primarily consisting of categorical words. This characteristic may not be ideal for large-scale pre-training.

• Evolving generative models for data generation: The field of generative models is constantly evolving, and we have observed an increasing number of works utilizing generative models for data creation. For instance, BLIP2 employs the BLIP-large captioning model to generate image captions. While the depth and infrared data in VIDAL-10M seems have a long-term impact, the slow progress in both data development and the iteration of generative models in these domains suggests that the relevant data in VIDAL-10M will continue to make significant contributions in the long term. Additionally, we can draw inspiration from the data mixing method employed by BLIP2, where the depth, infrared, and other modality data in VIDAL-10M can be combined with real scene data. This mixing approach has the potential to effectively leverage the strengths of different modalities and enhance their usefulness.

Q6: what is the value of training all modalities in a joint embedding space if all use cases have to do with text only?

A6: We have taken your advice into consideration and made significant improvements to our study. Specifically, we have added indirect alignment and multi-modal complementary experiments in Table 6 and Table 7 to further validate the value of our research. Aligning modalities within a shared language semantic space not only benefits various language-based downstream tasks but also enables us to harness the complementarity and alignment between modalities, resulting in improved performance of individual modalities, as demonstrated in Table 6 and Table 7. For instance, in the downstream task of nighttime pedestrian detection sourced from the LLVIP dataset, RGB images captured in the dark are used. In Table 6, we demonstrate that the performance of dark images->text is 62.4. When infrared is incorporated, the performance increases to 79.3. This showcases the advantage of binding multiple modalities into a semantic space, as it demonstrates robustness when another modality undergoes changes a domain change.

Furthermore, LanguageBind offers additional potential value by facilitating better alignment of other modality encoders with language. This alignment creates a more convenient framework for subsequent researchers to utilize our modal encoder in their explorations of Large Language Models and related research endeavors. We encourage other researchers to delve into similar areas of study or leverage LanguageBind's modal encoder in combination with LLM to develop cross-modal large models, such as NextGPT, PandaGPT, etc., which represents a primary focus of our ongoing research.

We greatly appreciate your valuable feedback. As a result, we have incorporated several additional experiments into the paper to showcase the effectiveness of LanguageBind. These newly added experiments are highlighted in $\color{red}{red}$ font within the revised version of the paper.

Weakness

Weakness 1: Lack of ablations

Answer: We have added more ablation experiments, including more modalities, more datasets, and other aspects, which are highlighted in red in the paper.

Weakness 2: Model release

Answer: We have open-sourced the LanguageBind pre-training model, but due to the rules of this conference, I can't provide you with the address information at this time, thank you for your understanding！

Questions

Q1: How the audio is processed?

A1: Sorry for the confusion, we're here for further clarification.

• The motivation of audio processing. ImageBind is trained only on AudioSet, where all the audio clips are 10 seconds long. Therefore, padding does not have a significant impact for them. However, our dataset consists of various durations. Hence, we first perform repetition. If the remaining duration is not sufficient for repetition, it will be padded with zeros. This approach largely protects the model's learning of audio features from a stable data domain and efficiently utilizes computational resources. Additionally, for audio clips longer than 10 seconds, while ImageBind does not encounter this issue, we aim to preserve the original information of the audio. Therefore, we randomly sample three audio segments. To leverage the pretraining weights of the model, which are trained on RGB images, this is why we use three segments instead of any other number.

• Audio stacking. We apologize for any confusion caused and please allow us to clarify in detail. Now we obtain three audio segments, and these segments are first converted into spectrograms. Assuming each spectrogram has a shape of (num_mel, target_len), we stack the three spectrograms together to obtain the final input with a shape of (3, num_mel, target_len).

Q2: How are the multi-view text annotations used during training?

A2: Multi-view text annotations are randomly sampled at each step. In our abalation experiments, we found that the source of the best performing text data is inconsistent for different downstream tasks in different modalities, e.g., infrared, depth. The OFA caption text performs best on the LLVIP dataset for infrared, the raw caption text performs best on the FLIR dataset for infrared, and the ChatGPT refined text performs best on the NYU dataset for depth and the MSR-VTT dataset for video. This suggests that text from different sources has its own unique strengths, so we end up providing text data with multiple perspectives in our VIDAL-10M dataset. According to our training experience, for modality data such as video with a wide range of scenes, mixing multi-view texts for training gives the best results, while for modal data such as depth and infrared thermal images with limited scenes, there may be uncertainty in the source of the best performing text, but the performance of the mixed multi-view text is also very good. More details can be found in our ablation experiments, where we have added more detail.

Q3: Why collect a new VIDAL rather than use those existing datasets?

A3: we have taken this into consideration when building the dataset. The conditions we considered when selecting the video data include, but are not limited to, the following 1.each video is without a watermark, ideally. 2. show the whole event within 20s, preferably without truncation 3. high video resolution 4. visual diversity. The Webvid-10M dataset is an excellent work, but since every video in this dataset has the "Shutterstock" watermark, it severely affects the generated depth map and infrared. Therefore, the current video dataset does not meet the above conditions, and we reconstruct the video part of VIDAL-10M. We crawl videos from short video platforms, which have full semantics and have been filtered by users, rather than truncated segments from longer videos.

Q4: What method is used to go from text logics of length L to single normalized text vector?

A4: Thanks for reminding, and we have used the [CLS] token here.

Q5: Why is OpenLIP in ImageBind worse than reported OpenCLIP in LLVIP?

A5: Since imagebind uses images as intermediates to align infrared and language, the alignment of thermal to language is indirect, whereas OpenCLIP directly aligns the visual to the language, so even though there is no alignment operation on the thermal data, the advantage of direct alignment to the language semantics instead allows OpenCLIP to outperform ImageBind on the LLVIP dataset. What's more, this is one reason why we constructed the VIDAL-10M infrared-text pairs, as we feel that the current multi-modal downstream task is closely related to language, and that a language-driven shared semantic space may be more suitable for the current multimodal development.