OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces

W1：The necessity of language representation decoupling:

In the newly added Figure 10 of the revised version of our paper, we visualize the representation distribution of text from different domains within different pre-trained models. Textual descriptions of audio and visual content have inherent domain gaps and should occupy different subspaces in any semantically meaningful space (such as the self-supervised language model, sentence-T5[1]). Language decoupling reduces the similarity of language representations between audio-related and visual-related modalities from 8.28 to 5.17, reflecting better discernibility and a more dispersed distribution of representations. Additionally, the performance improvements achieved through language decoupling serve as indirect evidence of the enhanced representation quality.

[1] Ni, Jianmo, et al. "Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models." arXiv preprint arXiv:2108.08877 (2021).

W2："unfair comparison" and "potential label leakage"

Our training data strictly adheres to the main baselines, ImageBind++ and InternVL_IB++, and the performance advantage over these methods underscores the effectiveness of our designs. Additionally, although our data is inherited from previous baselines, we remove all captions from AudioCaps in the text memory and retrain the OmniBind_Large model to fully eliminate concerns about "potential label leakage."

The results show that there is only a negligible drop in the audio-text retrieval and even stronger zero-shot classification accuracy on AudioSet. This confirms that AudioCaps captions are not essential for performance and OmniBind's state-of-the-art results are not derived from "potential label leakage."

W3. Experiments on more and latest datasets.

In general, we used the most widely used benchmarks available for each downstream task. Objaverse is the largest and latest test set for 3D-image and 3D classification. AudioCaps and Clotho are the most widely used audio-text benchmarks, and AudioSet is the largest audio classification benchmark, as far as we know. For the audio-image test, we keep the same to the compared baselines. For the image-text task, COCO and Flickr30K are the most commonly used image-text retrieval benchmarks. For video-text retrieval, we evaluate models on the additional MSVD [2]. We are in the process of collecting extra latest benchmarks and will share further results shortly. We would be very grateful if you could provide more detailed, latest benchmarks for us to perform comparative experiments.

Extra results on ImageNet v2 [3] for zero-shot image classification are updated.

[2] Chen, David, and William B. Dolan. "Collecting highly parallel data for paraphrase evaluation." Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. 2011.

[3] Recht, Benjamin, et al. "Do imagenet classifiers generalize to imagenet?." International conference on machine learning. PMLR, 2019.

W1：Writing.

We have added the explanation for the "item pair {p, a, v, t}" and will continue to refine the writing of the current manuscript.

W2&Q1：Training and inference speeds.

In terms of training speed, our method only requires training the connector that binds spaces and the weight routers, both of which are lightweight MLP projectors. As a result, the training process can be efficiently completed using a single 4090 GPU and converges easily. More specifically, on a single 4090 GPU, binding one space takes 5 hours, and learning the router for 14 models takes an additional 6 hours. Only a few million unpaired data are sufficient to achieve the current advanced performance. In contrast, training a multimodal representation model from scratch typically requires hundreds of times more computational resources and training data, as well as significantly longer training times.

For inference, indeed, integrating multiple pre-trained models increases the inference latency of our models. According to the number of integrated encoders, OmniBind requires 2-5 times longer latency than previous methods in each modality. We propose OmniBind to explore an effective "scaling up" approach for multimodal representations, increasing model size for improved performance. Scaling up models and boosting the current performance upper bound of multimodal representation models is a worthwhile pursuit. For example, large models can act as teacher models in distillation learning, where their knowledge is transferred to smaller models, leading to better models across all sizes, as seen in DINO v2 [1]. Proposing larger and more powerful multimodal representation models, therefore, holds substantial practical significance.

[1] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).

W3& Q2：Expert load balancing.

We intentionally avoid incorporating the expert load balancing mechanism used in the MoE structure of LLM, as the experts in MoE of LLM are generally homogeneous, while our encoders differ with prior biases (such as the text encoders from the audio-to-text model and the image-to-text model are experts in different fields). These inherent differences reduce the likelihood of relying excessively on a small subset of experts. Our experiments also empirically find that using the expert load balancing loss in MoE will lead to a decrease in the overall performance.

Additionally, we observe that our current routers can achieve a reasonable and dynamic balance between inputs from different domains, without using the expert load balancing loss, as shown in the router assign weight to Reviewer Z2z7’s Q3.

W4：Biases between textual descriptions of different modalities.

In the Figure.10 of the revised version of our paper, we visualize the representation distribution of texts from different domains in different spaces. In the well-studied self-supervised text representation space, sentence-T5, textual descriptions of audio and visual content show inherent domain gaps and occupy different subspaces.

More analysis of the effectiveness of language decoupling can be found in the response to Reviewer ViaF’s W1.

Q3：the advantages of item-pair retrieval.

Please refer to the response to Reviewer qoeh’s W1.

W5&Q4：How to integrate models with new modalities.

Please refer to the response to Reviewer Z2z7’s Q1.

Dear Reviewer enUq,

Thank you for your valuable comments and suggestions, which have been very helpful to us. We have carefully addressed the concerns raised in your reviews and included additional experimental results.

As the discussion deadline is approaching, we would greatly appreciate it if you could take some time to provide further feedback on whether our responses adequately address your concerns.

Best regards,

The Authors

Dear Reviewer enUq,

Thank you once again for your valuable suggestions! As the discussion period deadline approaches, we provide a summary of our responses to your concerns.

[1. Training and Inference Time]

We have detailed the training cost for each stage, showcasing the exceptional efficiency of our approach, which can be completed within a limited time with a single NVIDIA 4090 GPU. For inference, we highlight the significance of increasing the model size to push the boundaries of multimodal representation further.

[2. Expert Load Balancing]

We compared the results of applying and not applying the expert load balancing strategy in the MoE structure of LLMs. Based on experimental observations, we discuss why our model does not require additional load balancing mechanisms: different pre-trained expert encoder inherently excels in different input domains. Moreover, we provide detailed router weights for various inputs and analyze the effects of weight routing.

[3. Bias between Captions of Different Modalities]

In the updated manuscript, we include additional feature visualizations to illustrate biases in textual descriptions across different modalities. These distribution gaps of captions for different modalities are evident not only in our model but also in the self-supervised text model, Sentence-T5.

[4. Generalizability to New Modalities]

We have elaborated extensively on how to bind additional new representation models to enable expansion into new modalities under various scenarios.

[5. Advantage of Item-pair Retrieval]

We have provided further comparative experiments to demonstrate the effectiveness of item-pair retrieval over the embedding-pair retrieval used in FreeBind. Moreover, we highlight that the greater value of pseudo-item pairs lies in their ability to support the integration of additional data and representation spaces more flexibly—without requiring re-collection of pseudo-embedding pairs during every sapce binding.

We sincerely thank you for your thoughtful suggestions. We hope these responses address your concerns. If you find our replies satisfactory, we kindly ask if you would consider revising your rating based on your updated evaluation of our work.

Best regards,

The Authors

W1：Core insights of OmniBind.

Our core idea is to explore whether scaling up the binding space can consistently improve multimodal models. These two proposed schemes serve as practical solutions to bring this concept to life. Although we did not incorporate sophisticated designs, we believe the innovative "scaling up" insight, the practical methods enabling it, and the impressive final performance offer significant value to the community.

W2： Ablation study of pseudo item pairs and pseudo embedding aggregation.

We conducted a comparison between pseudo item pairs and pseudo embedding aggregation under the experimental setup of FreeBind. Specifically, we integrate EVA-CLIP-18B, ImageBind, LAION-CLAP (general), and LAION-CLAP (music). The R@1 accuracy on audio-text, audio-image, and image-text retrieval benchmarks are reported as follows:

The results show that pseudo-item pairs generally outperform pseudo-embedding aggregation. Beyond the empirical performance improvement, as you mentioned, the greater value of pseudo-item pairs lies in their ability to more easily support the integration of additional data and representation spaces.

Q1.1：Quality of curated pseudo item-pair:

We provide a visualization of pseudo-pair retrieval in the Figure.6,7,8,9 of the updated manuscript. We can find that the pseudo-pairs obtained from different modalities can effectively capture the unique information of different modalities. In particular, for text descriptions, the text retrieved by different modalities has very obvious different tendencies.

Q1.2: Better performance than "individual best".

Binding the omni representation occasionally outperforms the original "individual best," as observed primarily in 3D classification, 3D-image, and audio-image tasks. Since pseudo items are inherently retrieved using these "individual best" models, we attribute this phenomenon to the enhanced and more discriminative visual and textual representations achieved through the binding process, even when these representations are integrated later into the binding space.

Q2.1：How to integrate new models?

Details of how to integrate new models are discussed in the response to Reviewer Z2z7’s Q1.

Briefly, we bind all representation spaces in parallel to the advanced vision-language model EVA-CLIP-18B, due to its high-quality language and vision representations, and almost all existing multimodal models are either vision- or language-related. When a new space is bound, we only need to learn a new set of lightweight routers incrementally, and existing learned projectors for binding spaces can be re-used.

Q2.2 Iterative binding spaces.

Compared with the current parallel binding then weight routing, iterative binding space will make the pipeline much more complicated (each time a new space is iteratively bound via a connector, a new router should be trained. In contrast, binding all spaces in parallel only requires training routers one time). We have been unable to complete the iterative binding experiment these days, while we believe that subsequent research on how to bind spaces more effectively will be an interesting direction.

Q3: Typos.

Thanks for your reminder, we have rechecked these typos in the updated manuscript.

Dear Reviewer qoeh:

We are continuing to explore iteratively binding space, as it is more complex than our parallel binding approach. This process will take a few days, and we will update the results as soon as possible.

Thank you for your constructive feedback and further response. We are more than happy to address any further concerns or questions you may have.

Best,

The Authors

Q1: How to integrate future models with new modalities or better representations.

Overall, incorporating new models into OmniBind requires only minimal incremental learning without the need for extensive retraining or re-engineering.

We begin by revisiting our implementation, which consists of two main stages:

Stage 1 (Binding Spaces): we bind all representation spaces to the advanced vision-language model EVA-CLIP-18B in parallel. This model is chosen for its high-quality language and vision representations, and most existing multimodal representation models are correlated with either vision or language.

Stage 2 (Routing Spaces): we select a set of aligned spaces from Stage 1 and learn weight routers to linearly combine these spaces, which were previously bound in parallel.

Here are detailed ideas to incorporate new models:

For data, if new models introduce modalities not covered by the current pseudo item pairs, we can utilize overlapping modalities to retrieve additional items in the new modality for the existing pseudo pairs. If no new modalities are introduced, no modifications are required.

For training, in most cases, future models are likely to be correlated with vision or language. In Stage 1, we only need to bind these new models to the basic model, EVA-CLIP-18B. In Stage 2, we train new routers to handle the updated combination of spaces. In rare cases where new models are not directly related to vision or language, we can view the current OmniBind as the basic model. The new models can be integrated by binding them to OmniBind, followed by training routers to weight the new models with the basic model.

In summary, the existing learned projectors that bind spaces in the current OmniBind can be preserved, and only several extra lightweight MLP projectors are needed, which can be obtained by two training processes.

Q2.1: Potential biases in multimodal alignment.

Each pre-trained model inherently reflects specific biases associated with its modality. For instance, considering audio-text and image-text models, the textual data used during their training often originates from vastly different domains (The newly added Figure 10 in the updated manuscript proves this). This substantial domain gap results in each model exhibiting distinct semantic biases when processing data within the same modality. In OmniBind, the weight router strategy is proposed to integrate the different semantic biases, enabling the inputs from different domains to be processed by appropriate encoders. More analysis of weight routing in integrating different pre-trained models’ biases is provided in the response to your Q3.

Q2.2: Reliability and consistency of pseudo pairs.

For the reliability and consistency of the pseudo pairs. First, we use the state-of-the-art retrieval models for each modality pair, which are trained on web-scale data and show impressive robustness. Second, we select each modality as the starting point to retrieve others, respectively, which can effectively expand the diversity of data. The newly added Figure.6,7,8,9 in the updated appendix demonstrates the consistency and diversity of our data. Third, our approach can also accommodate partially-paired real-world data, which may further enhance the reliability in practical usage.

W1： Reliability of pseudo pairs.

First, the pre-trained representation models being integrated together have already been exposed to vast amounts of real-world multimodal data pairs. Our work mainly tries to connect and combine the different representation spaces, and the knowledge of OmniBind is actually derived from these pre-trained models. As demonstrated in [1] and [2], connections between different representation spaces are easy to learn and typically robust, even with limited data. Lightweight projectors and relatively small datasets can successfully learn the alignment between two representation spaces and exhibit generalization capabilities in downstream tasks.

Secondly, our pseudo-paired data distills knowledge from state-of-the-art multimodal models, using their predictions as ground truth. The correlation knowledge within the pseudo-pairs is also derived from these advanced existing models. In Figure 6,7,8,9 of the updated manuscript, we provide visualization examples of the pseudo pairs, illustrating the diverse and rich cross-modal alignment information they provide. More importantly, we evaluate our approach on various large-scale benchmark datasets of each modality (Objaverse for 3D, AudioSet for audio, and ImageNet for images). Across all these benchmarks, our method demonstrates strong generalization performance.

Finally, the primary purpose of our pseudo pairs is to construct pairings across all modalities (3D-audio-text-image). The current implementation, which relies purely on unpaired data, is intended to demonstrate the low data requirements of our approach. In practical usage, we can also incorporate real-world partially paired data. For instance, retrieve corresponding audio and text for real 3D-image pairs. We hope the potential collaboration with real data can further alleviate the concerns about the robustness of training data.

[1] Merullo, Jack, et al. "Linearly Mapping from Image to Text Space." The Eleventh International Conference on Learning Representations.

[2] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

W2: Increasing complexity and computational costs.

Integrating multiple models to achieve better performance can be regarded as an effective method to scale up the multimodal representation model, akin to increasing model size for better performance—a common idea in the fields of computer vision (CV) and natural language processing (NLP).

Admittedly, continuously increasing model size raises inference costs. However, inspired by the success of large-scale models in CV and NLP, we believe scaling up model size and exploring the performance upper bound of multimodal models is a meaningful endeavor, for example, acting as a teacher model for distillation learning (distilling the knowledge of the largest model to small models can foster better models of each size, as DINO v2 [3]). Proposing larger and more powerful multimodal representation models, therefore, holds substantial practical significance.

[3] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).

Dear Reviewer Z2z7,

Thank you for your valuable comments and suggestions, which have been very helpful to us. We have carefully addressed the concerns raised in your reviews and included additional experimental results.

As the discussion deadline is approaching, we would greatly appreciate it if you could take some time to provide further feedback on whether our responses adequately address your concerns.

Best regards,

The Authors

Dear Reviewer Z2z7,

Thank you once again for your valuable suggestions! As the discussion period deadline approaches, we provide a summary of our responses to your main concerns.

[1. Reliance on Pseudo-Pairs]

We first discuss that simple connectors are easy to learn and robust. Next, we provide visualization examples to demonstrate the reliability of pseudo-pairs. Finally, we highlight that our approach does not necessarily require unpaired data; incorporating real datasets can further enhance the reliability of the data in practical applications.

[2. Computational Costs and Latency]

Our core motivation is to explore an efficient and effective way to scale up the multimodal representation model, inspired by the success of scaling approaches in the CV and LLM fields. We highlight the significance of increasing the model size to push the boundaries of multimodal representation further.

[3. Integration for Future Models]

We have elaborated extensively on how to bind future models with better representation or new modalities across various scenarios.

[4. Potential Biases in Multimodal Alignment]

Our weight routing method improves the discrimination of the representation space, enhancing the ability to express different modalities of different information density. Statistical analysis of our router weight assignment also supports this improvement.

[5. Statistics of Routers Assign Weights]

We provide statistics on the weight assignment by the routers. Two main findings emerge:

1. For inputs with significantly different biases (e.g., text), our router can dynamically select the appropriate encoders based on the domains of different inputs, thereby effectively integrating the semantic biases of various models and achieving better results.
2. For inputs without specific biases (e.g., image and audio), our router maintains an overall balance while tending to assign more weight to more powerful representations (such as WavCaps, EVA-CLIP-18B, and ImageBind).

We sincerely thank you for your thoughtful suggestions. We hope these responses address your concerns. If you find our replies satisfactory, we kindly ask if you would consider revising your rating based on your updated evaluation of our work.

Best regards,

The Authors