VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

Thank you for your valuable comments and feedback.

We would like to emphasize that our primary goal was to introduce a unified benchmark for evaluating text-image embedding models, providing a more comprehensive perspective on how current embedding models perform across various tasks and domains. Additionally, we aimed to explore the potential of adapting pretrained VLMs to embedding tasks. As demonstrated in Section 4.4, our scaling experiments reveal that further scaling significantly improves model performance, and we are still far from reaching the limits of this scaling trend. While we are excited to highlight that our current model outperforms previous approaches on MMEB, we also aim to share intriguing findings on effectively tuning VLMs into embedding models. These include insights on LoRA versus full fine-tuning, the impact of scaling, and the interactions between meta-tasks.

We address the raised concerns as follow:

W1.

On Evaluating Traditional Benchmarks.

We acknowledge the importance of comparing performance on well-established zero-shot benchmarks for a broader perspective. Since we included MSCOCO in our training split, we focus here on presenting zero-shot retrieval results on Flickr30K, as shown in the table below. Our VLM2Vec demonstrates strong performance, achieving competitive T2I (Text-to-Image) and I2T (Image-to-Text) scores compared to existing CLIP-like models. It is worth noting that VLM2Vec is specifically designed for instructional text/image representation rather than being optimized solely for retrieval tasks.

We believe that by incorporating additional retrieval-specific training data and techniques such as hard negative mining, the retrieval performance of VLM2Vec can be further enhanced. Thank you for highlighting this, and we will expand on this aspect in the revised version to strengthen comparisons across traditional benchmarks. We also plan to include additional ranking-based metrics, such as MRR and NDCG, when presenting the MMEB leaderboard in the future.

W2

On Out-of-Distribution (OOD) Evaluation.

Thank you for raising the concern regarding potential OOD definition during the evaluation of Phi-3.5-vision. To address this issue, we additionally trained VLM2Vec using the llava-hf/llava-v1.6-mistral-7b-hf backbone, which provides a transparent training recipe. According to its documentation, among our OOD evaluation sets, only RefCOCO was used during Llava training. So the other OOD sets are still valid. We have included the corresponding results in the table below (also will be added to the revised paper).

Both VLM2Vec models were trained under the same settings (LoRA-8, 2k steps, batch size of 1k), with one key difference in image processing. Due to time constraints, all input images for llava-v1.6-mistral training were resized to 336x336 to avoid inconsistencies caused by dynamic cropping and to accelerate training. While this resizing significantly limits the model's visual understanding capabilities, leading to performance drops across tasks, VLM2Vec trained with llava-v1.6-mistral still outperforms baseline models on most tasks.

Overall, VLM2Vec (Llava-1.6) consistently underperforms VLM2Vec (Phi-3.5) on IND and OOD. For example, llava-1.6 has been trained on the training set of our several IND datasets like OKVQA, DocVQA, ChartVQA, its VLM2Vec performance is still lagging behind Phi-3.5 on these sets. This indicate that the gap is unlikely due to the instruction tuning set overlap, but rather from the models' underlying perception and grounding capabilities.

Our goal is to provide the VLM2Vec framework and adapt it to all SOTA VLMs for representation learning in the future.

W3.1

On Experimental Details for Baselines.

The baseline models are trained with different data setups. For example, CLIP is trained on image-text pairs, E5-v is trained using text data, and UniIR is trained on retrieval tasks. One of the key contributions of our paper is training on diverse setups:

1. Diverse combinations of image and text modalities: Both the query and target can be an image, text, or a combination of both.
2. Diverse meta-tasks: We incorporate four meta-tasks and use 20 datasets, which sets our approach apart from models like UniIR that are trained on a single meta-task (retrieval).

Furthermore, to highlight the advantages of using VLM as a backbone for fusing text and image data in representation learning, we retrained CLIP-based models on our in-domain dataset.

For the remaining baseline models, UniIR and MagicLens also utilize a shallow fusion approach based on CLIP models, with their primary contribution being the datasets they were trained on. Therefore, we have not included the fine-tuned versions of these two models in this comparison. E5-v is a contemporary work that also leverages VLM as its backbone but is trained exclusively on text data. Therefore, it is not appropriate to fine-tune their model on our datasets.

W3.2

CLIP w/ instructions

Thanks for your suggestion! We have provided the results of the CLIP model using instructions here. Since the CLIP-family model is not well-suited for instruction-following and does not fuse text and image information as effectively as VLM2Vec, using instructions can lead to a decline in performance.

Q1

Figure 5 details

Figure 5 illustrates the generalizability of the three models on unseen meta-tasks. Taking the first subplot as an example, the legend indicates that the model is trained on only one meta-task. For instance, "VLM2Vec_VQA" means the model is trained exclusively on the VQA meta-task, while "VLM2Vec_RET" indicates training on the retrieval meta-task. The x-axis represents the meta-task being evaluated. For example, "VLM2Vec_VQA" achieves a precision of 35.9 on the classification task. Figure 5 demonstrates that the retrieval task provides the best generalization ability to our model. This result is intuitive since the retrieval task dataset includes more diverse instructions and combinations of text and visual modalities.

The table below provides a detailed numerical breakdown of the same data shown in Figure 5, offering an alternative perspective for detailed comparison and analysis.

Q2

Whether this training method affect the original capabilities of MLLM

This is indeed a very valid question. In our VLM2Vec-Lora version, we only need to load 30MB low-rank adapter weights to activate Phi-3.5's retrieval capabilities. In the generative scenario, we simply unload the low-rank adapter to recover the original checkpoint to maintain its performance. This essentially makes the model very flexible to function as both a retriever and generator. In the multimodal RAG setup, we can load both the phi-3.5 checkpoint and lora in the memory. In the retrieval stage, we will load the LoRA module to retrieve multimodal information. After that, we can unload the LoRA module to read the retrieved information and maintain its generative performance.

It's also worth highlighting that recent research has proposed training strategies enabling a single model to handle both generative and embedding tasks in text-based language problems (e.g., https://arxiv.org/pdf/2402.09906). Exploring such approaches in the multimodal setting is a promising direction, and we see this as a natural extension of our work. We will aim to present results on the benchmarks you mentioned in the near future. Thank you again for the valuable feedback!

We realize that we may have misunderstood the first part of W3, and our previous response might have caused some confusion, so we would like to provide further clarifications here. We used four categories of models (CLIP-family, UniIR, MagicLens, and E5-V) as baselines. In the initial version of our paper, we evaluated these models on the MMEB evaluation datasets without any fine-tuning. By comparing their performance with our VLM2Vec on the 16 OOD datasets, we demonstrated that VLM2Vec outperforms these baselines. The superior performance of VLM2Vec stems from two key contributions of our work:

1. The training data and training strategy.
2. The framework uses VLM as the backbone of the embedding model, enabling a deep fusion of image and text features, as opposed to the shallow fusion used by other CLIP-based models.

As suggested by one of the reviewers, it would be fairer to fine-tune the baseline models on our MMEB training data. We agree with this valid point. Given that the second contribution of VLM2Vec (deep fusion of text & image features through VLM) is pivotal to its strong performance, this comparison (VLM2Vec vs. fine-tuned CLIP-based embedding models) could better highlight the advantage of our framework. In our revised version, we fine-tuned some of the baseline models on the same MMEB training datasets, using exactly the same training setup (e.g., batch size, number of training steps). Below are the reasons why we chose certain baseline models for fine-tuning and why we excluded others.

• For the CLIP-family category, we fine-tuned the two models that achieved the best performance on MMEB eval prior to fine-tuning (CLIP and OpenCLIP).
• UniIR and MagicLens are both CLIP-based models that incorporate a shallow fusion layer to combine text and image information. The primary contributions of these models lie in their training data: UniIR uses diverse types of retrieval data, while MagicLens is trained on millions of web image pairs. Since their architecture is similar to the first category and their main contribution is actually their own training data, we decided not to fine-tune them.

In conclusion, VLM2Vec's strong performance can be attributed to two key factors:

1. The training data and training strategy.
2. The deep fusion of image and text features by leveraging VLMs as backbones.

In the initial version of our work, we did not fine-tune other embedding models on our own data. In our revised version, we added additional fine-tuning experiments to highlight the significance of the second point. Please let us know whether we have adequately addressed your concern.

Dear Reviewer hG4f,

We would like to learn if our response addresses your concerns and questions, and we invite any additional feedback or thoughts for improving our paper. If you feel that our responses resolve the issues raised, we would be grateful if you could consider reflecting this in the evaluation. We would be happy to address any further concerns or questions. Thank you again for your time and effort!

1. Are these results obtained from phi-3.5v ?

Yes, the Flickr30K results are obtained from Phi-3.5v.

2. Why can VLM2Vec still achieve exceptional performance?

We do not think this result is counterintuitive. The strong performance is achieved through generalization from the instructional training setup, consistent with findings from prior works such as Flan [1] and UniIR [2]. We incorporate instructions into Flickr30K queries (the instructions we used are "Retrieve an image that matches the given caption:" and "<|image_1|>" Find an image caption describing the given image"). Additionally, our training dataset includes similar caption-image pairs, such as those in VisualNews (T2I and I2T). The model learns to generalize across tasks by following task-specific instructions, demonstrating its ability to adapt effectively to different retrieval tasks.

[1] Chung, Hyung Won, et al. "Scaling instruction-finetuned language models." JMLR 25.70 (2024): 1-53.

[2] Wei, Cong, et al. "Uniir: Training and benchmarking universal multimodal information retrievers." arXiv preprint arXiv:2311.17136 (2023).

3. Could the authors provide the results of zero-shot classification on ELEVATER?

ELEVATER contains several datasets that overlap with those already included in MMEB-eval (e.g., HatefulMemes, Country-211, VOC2007). Given its size, we will aim to present the results on ELEVATER in the future updates as best as we can.

4. A more appropriate comparison would be against training llava-v1.6 with out-of-distribution (OOD) data ...

Could you please elaborate on this suggestion? To clarify, all VLM2Vec models (including Llava-v1.6) are not trained on any OOD datasets. Sixteen OOD datasets were intentionally excluded from training and reserved for evaluation purposes. Regarding Llava-v1.6 specifically, we have verified its training recipe: it only used RefCOCO as an OOD dataset during its training, which is also included in the MMEB OOD split. Therefore, the VLM2vec-Llava-v1.6 results for the other 15 OOD datasets remain valid, and the model demonstrates strong generalization performance on unseen data.

5. For retrieval tasks, data leakage can lead to unfair comparisons.

Could you clarify where the alleged data leakage originates? It is important to note that data leakage specifically refers to scenarios where the test set or its labels are exposed during training. Similarities between the training set and OOD data in terms of task or domain characteristics do not constitute leakage unless identical data points are present. Without clear evidence, this allegation does not hold.

If you are instead referring to distributional overlap (e.g., the training data shares similar tasks/domains with OOD data but does not contain identical samples), this concern can be addressed. The updated VLM2Vec-Llava-v1.6 results show that the model, which only used RefCOCO during training, performs strongly across the remaining 15 OOD datasets. This demonstrates robust generalization capabilities despite not being trained on those datasets.

Thank you for your feedback, and we look forward to further clarification on these points.

We have trained an improved version of the VLM2Vec-llava-1.6. The primary difference from the previous version is the adoption of higher-resolution images, increasing from 336x336 to 1344x1344. We would like to share the results here. We believe that VLM2Vec-Llava-1.6, which leverages LLava-1.6 with a transparent pre-training data recipe, demonstrates the effectiveness of our VLM2Vec framework and its strong generalization ability on OOD evaluation datasets.

Flickr-30K result

VLM2Vec-Llava-1.6 has demonstrated strong performance on image-text retrieval tasks, despite not being specifically trained for retrieval.

MMEB result

FFT means fully fine-tuned on the MMEB train. The results clearly show that VLM2Vec-Llava-1.6 exhibits strong generalization ability on OOD datasets, further proving the effectiveness of our VLM2Vec framework.

Thank you so much for your constructive feedback to our work! The following is our response:

W1: Relatively straightforward Approach

Thank you for your feedback and for acknowledging the straightforwardness of our proposed method. We agree that our framework is conceptually simple, and this simplicity is an advantage here. Our framework is highly scalable and generalizable to a wide range of VLMs. Our experiments also showcase the ability of VLM2Vec to adapt across diverse meta-tasks and domains, underlining its robustness as a multimodal embedding framework.

W2: Clarify the reasons behind these performance differences

We tested the embedding model on a variety of 36 tasks including classification, VQA, retrieval and grounding. These tasks are sourced from various existing datasets. Due to the task type, their difficulty levels vary significantly. For example, ImageNet-A is an adversarially constructed set, which poses great challenges to all the models. In contrast, VOC2007 is a traditional classification task used to test basic visual perception, where many models can reach very high accuracy. Similarly, retrieval tasks like FashionIQ and CIRR require understanding nuances between different fashion images, which pose great challenges to all the models and lead to relatively low performance.

As part of our future work, we plan to conduct detailed analyses of the model’s performance and behaviors across different domains and complexities. This will include exploring its handling of abstract versus concrete concepts, sensitivity to the granularity of cropping, and its ability to represent multiple objects or process multiple images. These directions will help us gain a deeper understanding of the task-specific strengths and limitations of the proposed framework.

Thank you again for your constructive suggestions, and we will aim to incorporate more detailed task-specific evaluations in future iterations.

We deeply appreciate Reviewer kzgZ’s insightful comments, which guided us to enrich our benchmark setting and add more analysis to show the strength of VLM2Vec model. We respond to Reviewer kzgZ’s comments as follows with more analytical results.

W1

Benchmark design with an increased number of candidates.

We agree that increasing the number of candidates would make the benchmark more realistic and challenging. But we also considered computation cost when designing the benchmark. We were concerned that having as many as a million candidates would lead to very low adoption of the benchmark due to the massive computation cost. Choosing 1000 as the number of candidates is a tradeoff between practicality and computation cost. For our VLM2Vec model, evaluation takes approximately 5 hours on 4 H100 GPUs, which is reasonable for most researchers. Additionally, we plan to release a more challenging version of MMEB with a larger number of candidates and increased difficulty. We investigate the performance of models with different numbers of candidates in a set of (100, 500, 1000, 2000, 5000), and choose 1000 as it strikes a balance between unsaturated benchmark performance and reduced iteration time for users.

(In classification, the number of candidates is fixed, which is the number of classes pre-defined.)

W2

Including original test splits for fair comparison

We agree that retaining the original test splits can facilitate comparisons with existing benchmark scores, and this was indeed considered during the initial draft of our benchmark. One issue we encountered is that some original benchmarks have large amounts of test examples and candidates, which can incur massive computation costs. This is a lesson we learned from existing text embedding benchmarks: MTEB and BEIR’s original test set comes at the expense of lengthy evaluation times—MTEB, for example, can take several weeks of GPU hours to complete. To address this, we adopted downsampling as a tradeoff, allowing us to significantly reduce inference costs while maintaining reasonable quality. This approach is also used in contemporary benchmarks like MMTEB. Since our primary goal is to create a comprehensive testbed that evaluates models across a wide range of meta-tasks and domains. To achieve this, we prioritized task and domain coverage by adopting a unified setup for all MMEB tasks, including a ranking formulation and resampled query/target pairs, to streamline benchmarking efforts for users.

W3

Baseline methods trained on MMEB training sets

We retrain several baseline models from the CLIP family using our in-domain data and present the results here to demonstrate that VLM achieves better fusion of text and image information and potentially offers improved generalization capabilities. For the remaining baseline models, UniIR and MagicLens also utilize a shallow fusion approach based on CLIP models, with their primary contribution being the datasets they were trained on. Therefore, we have not included the fine-tuned versions of these two models in this comparison. E5-v is a contemporary work that also leverages VLM as its backbone but is trained exclusively on text data. Therefore, it is not appropriate to fine-tune their model on our datasets.

W4.

Baseline Evaluation Using an Improved Version of the UniIR Model.

Thank you for the suggestion! We evaluated the best version of the UniIR model (CLIP-SF) and have included the results in the table below. While CLIP-SF performed slightly better than the BLIP-FF version, it still falls significantly behind VLM2Vec across all metrics. We will include these results in our revised paper to ensure a more comprehensive comparison.

W5.

VQA and visual grounding task setup

Thank you for pointing this out. We agree that text generation for VQA and object localization for visual grounding are more natural setups for these tasks. A ranking-based formulation may not be ideal for tasks where the target-side data is complex, such as the long-form answers in ScienceQA.

Our motivation for including VQA and VG tasks in MMEB is to test models’ ability to represent concepts and objects as embedding vectors. We argue that the datasets selected in MMEB are well-suited for this purpose. Specifically, for VQA, we focus on answering tasks involving various visual concepts, supported by the observation that most answers are short-form (with an average length of fewer than 5 words). The objective is to challenge models to represent complex concepts across different domains and contexts. For VG, we evaluate models’ ability to represent the same object from different perspectives. In this setup, the query consists of the original image paired with an instruction pointing to the object of interest, while the target comprises a cropped version of the object, along with several distractor candidates.

For more details and examples of VQA and VG tasks included in MMEB, please refer to Table 7 and 9 in our appendix.

Dear Reviewer kzgZ,

We would like to learn if our response addresses your concerns and questions, and we invite any additional feedback or thoughts for improving our paper. If you feel that our responses resolve the issues raised, we would be grateful if you could consider reflecting this in the evaluation. We would be happy to address any further concerns or questions. Thank you again for your time and effort!

Thank you so much for your constructive feedback to our work! The following is our response:

W1 Novelty of VLM2VEC

Thank you for your thoughtful feedback and for raising concerns about the novelty of VLM2Vec. We fully agree with your suggestion to emphasize the uniqueness of VLM2Vec as a framework exclusively designed for multimodal embedding tasks. Unlike prior works, VLM2Vec is specifically optimized for embedding tasks, such as retrieval, classification, and clustering, with a focus on precise cross-modal alignment and understanding. The novelty of this study can be summarized as follows:

1. MMEB offers a platform to showcase the capability of instructional multi-modal embedding capabilities, systematically testing the effectiveness of embedding models. VLM2Vec demonstrates strong meta-task generalization capabilities, as shown through its evaluation on MMEB—a unified benchmark designed to systematically test embedding models across modalities and diverse tasks.
2. Additionally, VLM2Vec provides a scalable framework for adapting any VLM to an embedding model, enabling seamless integration of pretrained vision models and LLMs for embedding purposes. As demonstrated in Section 4.4, scaling experiments reveal that further scaling significantly improves model performance, and we are far from reaching the limits of this trend. Building on this, we intend to expand the VLM2Vec family by utilizing different backbone models and exploring various scaling factors. Our future work aims to release more models in the VLM2Vec family to further establish the generalizability and flexibility of this framework.

In summary, VLM2Vec represents a unique contribution as a dedicated multimodal embedding framework, which can organically complement with regular generative approaches on tasks requiring accessing large volume of data or very long context understanding. We will revise the manuscript accordingly to better articulate these distinctions. Thank you again for your valuable suggestions!

Q1: About [EOS] token in the learning objective

Thank you for your insightful question regarding the use of the [EOS] token for pooling in VLM2Vec. We chose the [EOS] vector for pooling based on findings from the text embedding work E5-Mistra[1] and NV-Embed [2]. As shown in E5-Mistral [1], using alternative pooling methods instead of the [EOS] token may result in small performance drops. However, we believe that extensive training could lead to a minimum gap between different pooling tokens. While we focused on [EOS] pooling in this study, we appreciate your suggestion and will explore the impact of other pooling methods, such as class tokens or learnable tokens in the revision. [1] E5-mistral: Wang, Liang, et al. "Improving text embeddings with large language models." arXiv preprint arXiv:2401.00368 (2023). [2] NV-Embed: Lee, Chankyu, et al. "NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models." arXiv preprint arXiv:2405.17428 (2024).

Q2: Inference procedure for visual grounding

Thank you for raising this important question regarding the inference procedure for different tasks. For visual grounding (VG), the inference procedure is also based on similarity prediction with task-specific instructions. Specifically: Input Representation: The query combines an instruction (e.g., "Select the portion of the image that isolates the object of the given label: red apple") with the full image. This instruction guides the model to focus on a specific object within the image. Target Representation: Each target corresponds to cropped regions (bounding boxes) of the image, including both the object of interest and distractor regions. Embedding Similarity: The multimodal embedding of the query ([EOS] token representing the full image and instruction) is compared to the embeddings of each cropped region using cosine similarity. The cropped region with the highest similarity score is selected as the predicted grounding location. This approach effectively uses instructions to represent the same object from two perspectives: the full image and its specific cropped regions. You can refer to the visual grounding data examples provided in the appendix (Page 24, Table 9) for additional clarity. Thank you for highlighting this point; we will ensure the revised manuscript includes a clear explanation of the inference procedures. We truly appreciate your valuable feedback!