Thank you for your thoughtful comments and time reviewing our work. We are truly pleased with the positive reception our work has garnered from you and the other reviewers.

• “method gives new SOTA performance…method is simple…should yield richer gradients…” - RM3oh
• “B3 method is rigorously formulated…supported with theoretical bounds on InfoNCE loss approximation…consistent and substantial gains, especially under small batch sizes…graph-based community detection is a novel framing ” - RipHG
• “has clear intuition and solid theoretical justification…Experiments seem solid and extensive…” - Rm62i
• “achieves sota performance despite training at smaller batch size…ablation studies validate the effectiveness of B3” - R4DdK

W1: ... confused about the positive samples at the beginning. The papers uses the standard terminology "(query, positive) pairs" to refer to a "positive pair", but does not explicitly explain that a "positive" can be of a different data type than its corresponding "query" (e.g., an image paired with a text caption). Readers whose main exposure to contrastive learning comes from self-supervised methods... this lack of clarity may lead to a fundamental confusion.

Thank you for highlighting this point. As the reviewer correctly observed, the query and positive can originate from different modalities, and each may also be a combination of modalities. For example, in VQA, the query comprises an image and a text question, while the positive is a text answer. This is an important characteristic of the MMEB training set, which we will clarify in the final draft.

The design of excluding top p ranks from the negative samples is quite ad-hoc, and has to be tuned case by case. Q1: The negative samples exclusion ranking threshold p needs to be tuned differently for various task types (e.g., p=30 for retrieval, p=70 for VQA). This suggests the method is sensitive to the data's domain and structure. Could you provide more insight into the underlying reasons?

Excluding top $p$ ranks serves as a mechanism for excluding false negatives, a strategy previously employed in SFR‑Embed and NV‑Embed. For retrieval, we adopted the same value of $p=30$ as used in SFR‑Embed. NV‑Embed also showed that lower values of $p$ between $10$ and $30$ perform well. However, in the case of VQA training datasets in MMEB, the positive (target) responses were notably short. The table below reports the average target lengths for the VQA tasks.

In contrast, for image2text retrieval tasks, the average positive (target) length was as follows

Due to this reason, we found a significantly higher number of targets which had similar/same meaning as that of the labelled positive. For instance, here is the first example in OK-VQA dataset. As seen, due to the short nature of the answers a higher number of targets/positives of other examples are very similar/same compared to retrieval.

• Image: Image containing a women playing tennis
• Q: What is the hairstyle of the blond called?
• Top ranked targets/positives: [“ponytail”, “ponytail”, “ponytail”, “pony tail”, “pony tail”,....]

Hence, we used a higher value of $p$ for VQA tasks. For computational efficiency, we tuned $p$ for smaller batch sizes and retained these values for larger batch sizes. Here are results for a model trained and tested on VQA datasets only at batch size ($|B|=32$) for different $p$ values.

At lower batch sizes, results improve for higher $p$ values. However, as discussed next, at higher batch sizes, $p$ has a smaller effect on the performance.

The table below reports results for $p=0$ across different batch sizes. At smaller batch sizes, using an elevated value of $p$ proves beneficial, effectively reducing multiple false negatives. However, at larger batch sizes, this effect diminishes; as shown, performance remains comparable between $p=0$ and $p=30$ when using higher batch sizes.

Presented below are additional variations of $p$ and $m$ at a higher batch size (512). As shown, B3 maintains consistent performance across these variations when using a larger batch size.

As shown p=0 works very well at larger batch sizes.

Main Takeaway (from $p$ and $m$ analysis)

• The previous tables indicate that the hyperparameters $p$ and $m$ have a greater impact on performance at smaller batch sizes. While B3 requires tuning of these parameters to achieve optimal results at smaller batch sizes, it shows increased robustness to their variation at larger batch sizes.

To further strengthen our B3 pitch, we have included two additional results as requested by other reviewers.

Generalizability to other domains?

We believe that B3 is a highly generalizable method for contrastive training. To further substantiate our claim, we report the performance of B3, trained on the same NLI data as SimCSE (both with and without annotated hard negatives (HN)), evaluated on the STS benchmark. The corresponding SimCSE models (with/without HN) serve as the teacher models. All models (teacher and student) in the following table are 300M Roberta-Large.

B3 demonstrates notable improvements over SimCSE on the STS benchmark, validating our batch mining approach for text-only tasks. Note that even starting with HN in the teacher, we can still see improvements. In future work, we plan to extend this to larger benchmarks like MMEB.

Need for a strong teacher?

In B3, the teacher’s sole role is to spot potential strong negatives and group them in the same batch. We simply take ranks $p:(p+m)$ from the teacher to build batches with strong in-batch negatives— and even with as few as 8 such negatives ($K=8$ in the Table), performance remains solid. Critically, the teacher’s task is limited: it only needs to pick out some strong negatives, not all of them.

Hence, this doesn’t require a very accurate teacher, as the reviewer suspected. To prove the point, we run B3 with a much weaker teacher (CLIP). Despite CLIP performing far below VLM2Vec on MMEB, using it as a teacher still delivers strong results.

All that said, a strong teacher model can still be trained using only positives and a sufficiently large batch size, without relying on any negatives. VLM2Vec teacher models were trained solely with positives and random batches, without incorporating hard negatives.

We again thank the reviewer for their time and effort. We will incorporate the additional results, analyses, and detailed responses provided here into the final draft to ensure a more comprehensive and rigorous presentation.

Reviewer 4DdK, please engage in the discussion period. I understand that the review period started over a weekend, but we only have a few days remaining in the (now slightly extended) discussion period. The authors have provided a thoughtful response to your review and you are obligated to respond to it. You should share with the authors if they addressed questions or concerns you had, and seek clarification about any questions or concerns that remain. Please post your response as soon as you can so that there is time for the authors to follow up and discussion to progress as needed.

Thank you for your thoughtful comments and time reviewing our work. We are truly pleased with the positive reception our work has garnered from you and the other reviewers.

• “method gives new SOTA performance…method is simple…should yield richer gradients…” - RM3oh
• “B3 method is rigorously formulated…supported with theoretical bounds on InfoNCE loss approximation…consistent and substantial gains, especially under small batch sizes…graph-based community detection is a novel framing ” - RipHG
• “has clear intuition and solid theoretical justification…Experiments seem solid and extensive…” - Rm62i
• “achieves sota performance despite training at smaller batch size…ablation studies validate the effectiveness of B3” - R4DdK

To further strengthen our B3 pitch, we have included two additional results as requested by other reviewers.

Generalizability to other domains?

We believe that B3 is a highly generalizable method for contrastive training. To further substantiate our claim, we report the performance of B3, trained on the same NLI data as SimCSE (both with and without annotated hard negatives (HN)), evaluated on the STS benchmark. The corresponding SimCSE models (with/without HN) serve as the teacher models. All models (teacher and student) in the following table are 300M Roberta-Large.

B3 demonstrates notable improvements over SimCSE on the STS benchmark, validating our batch mining approach for text-only tasks. Note that even starting with HN in the teacher, we can still see improvements. In future work, we plan to extend this to larger benchmarks like MMEB.

Need for a strong teacher?

In B3, the teacher’s sole role is to spot potential strong negatives and group them in the same batch. We simply take ranks $p:(p+m)$ from the teacher to build batches with strong in-batch negatives— and even with as few as 8 such negatives ($K=8$ in the Table), performance remains solid. Critically, the teacher’s task is limited: it only needs to pick out some strong negatives, not all of them.

Hence, this doesn’t require a very accurate teacher, as the reviewer suspected. To prove the point, we run B3 with a much weaker teacher (CLIP). Despite CLIP performing far below VLM2Vec on MMEB, using it as a teacher still delivers strong results.

All that said, a strong teacher model can still be trained using only positives and a sufficiently large batch size, without relying on any negatives. VLM2Vec teacher models were trained solely with positives and random batches, without incorporating hard negatives.

We again thank the reviewer for their time and effort. We will incorporate the additional results, analyses, and detailed responses provided here into the final draft to ensure a more comprehensive and rigorous presentation.

Thank you for your thoughtful comments and time reviewing our work. We are truly pleased with the positive reception our work has garnered from you and the other reviewers.

• “method gives new SOTA performance…method is simple…should yield richer gradients…” - RM3oh
• “B3 method is rigorously formulated…supported with theoretical bounds on InfoNCE loss approximation…consistent and substantial gains, especially under small batch sizes…graph-based community detection is a novel framing ” - RipHG
• “has clear intuition and solid theoretical justification…Experiments seem solid and extensive…” - Rm62i
• “achieves sota performance despite training at smaller batch size…ablation studies validate the effectiveness of B3” - R4DdK

We equally appreciate your constructive comments, and have modified the draft to address the raised concerns.

B3 is more like a training technique that enhances contrastive learning by constructing more reliable negative samples. Can this technique be extended to more general tasks?....

We concur with the reviewer that B3 is a highly generalizable method for contrastive training. To further substantiate our claim, we report the performance of B3, trained on the same NLI data as SimCSE (both with and without annotated hard negatives (HN)), evaluated on the STS benchmark. The corresponding SimCSE models (with/without HN) serve as the teacher models. All models (teacher and student) in the following table are 300M Roberta-Large.

B3 demonstrates notable improvements over SimCSE on the STS benchmark, validating our batch mining approach for text-only tasks. Note that even starting with HN in the teacher, we can still see improvements. In future work, we plan to extend this to larger benchmarks like MMEB.

B3 assumes access to the full training dataset for batch construction. This makes it less suitable for continual learning or privacy-sensitive domains.

• Contrastive learning typically necessitates substantial data volumes and large batch sizes, and is therefore predominantly conducted in an offline setting.
• Every single baseline reported in Table 1 of the draft, as well as those on the MMEB leaderboard, operate in an offline configuration.
• Likewise, most leading models on the MTEB leaderboard, including SFR‑Embed and NV‑Embed, are trained in an offline manner.

We regard this not as a limitation but as a natural consequence of the extensive data and batch requirements inherent to contrastive learning. With regards to privacy, B3 is a training algorithm and requires access to the data like every other training method.

Rui Meng, et al. 2024. Sfr-embedding-mistral: Enhance text retrieval with transfer learning.

Gabriel de Souza et al. 2024, Nv-retriever: Improving text embedding models with effective hard-negative mining.

Can B3 adapt to dynamic or evolving datasets, such as in online or streaming learning scenarios? B3 relies on full-dataset ranking and offline clustering…might online approximation variants be viable?

Online learning comprises two essential components: (1) generating effective gradients from incoming examples and (2) retaining knowledge acquired from previously trained examples.

• For the latter, several approaches have been proposed; for instance, Liu et al. (2025) introduced a technique in which gradients from incoming batches are projected to prevent interference with existing knowledge.
• For the former, B3 can be adapted to operate on a sliding window of incoming examples. Notably, at any given step, we already have a trained model available, which can serve as a teacher—eliminating the need for an additional model—making B3 well-suited for this setting.

Liu, X. et al. (2025). Continual multimodal contrastive learning. arXiv preprint arXiv:2503.14963.

Why was METIS selected as the clustering method, and how does it compare to alternative strategies like K-Means or spectral clustering in this context? …. Please justify the choice of METIS beyond runtime...?

Our choice of METIS algorithm is due to three main reasons.

• Note that we need the cluster size $K$ to be fixed in B3 (as the batch size $|B|$ is fixed). Most clustering algorithms, such as K-Means or Agglomerative, have balanced variants, but these are typically much more computationally expensive, often running in $O(n^3)$. In contrast, METIS operates in $O(n \log⁡ n)$ while producing equal-sized clusters.
• A key idea within B3 cluster construction is to minimize the presence of false negatives (upto rank $p$) in the training clusters. Note that each example in the training dataset has a different set of false negatives. Clustering algorithms like K-Means/Agglomerative would require substantial modifications to achieve this.
• Examples with ranks beyond $p+m$ in B3 are less relevant and should not play a role in clustering. Here again, it is non-trivial to include these constraints in regular clustering algorithms.

To accommodate rank-based constraints, we opted for graph-based clustering algorithms. Furthermore, to satisfy the fixed batch size requirement, we selected the METIS algorithm.

Could you elaborate on the trade-off between cluster size (K) and performance, especially across diverse task types? …Does B3 generalize well with a fixed K across task types, or is task-specific tuning essential?

• Saunshi et al. (2019) theoretically demonstrated that using an excessive number of negatives leads to intra-class collisions, which degrades the effectiveness of contrastive learning in general—not limited to any specific domain such as retrieval or VQA. They further validated this empirically across diverse image and text datasets. Similarly, SFR‑Embed reported that an excessive number of hard negatives can negatively impact performance on MTEB.
• Accordingly, for B3—which yields $K$ strong negatives per example—we tuned the value of $K$ instead of simply selecting the maximum possible value (i.e., $K$=batch size). A single $K$ value was used across all tasks, including retrieval, VQA, classification, and grounding.
• It is important to note that the batch size is a multiple of $K$, which prevents adjusting it individually for different datasets under our fixed‑batch‑size setup. As shown in the Table 1 and 4, a single $K$ value generalized for all datasets.

Saunshi, et al. (2019, May). A theoretical analysis of contrastive unsupervised representation learning. In ICML. PMLR.

The approach assumes that the teacher model produces semantically meaningful ranks. If the teacher is weak or biased,... degrade significantly, affecting downstream performance. ...The success of B3 heavily depends on the accuracy of teacher-based similarity rankings. ...Were the teacher models pretrained on the same modalities and tasks as the downstream evaluation tasks? Would using a mismatched or weak teacher degrade B3’s performance?.

In B3, the teacher’s sole role is to spot potential strong negatives and group them in the same batch. We simply take ranks $p:(p+m)$ from the teacher to build batches with strong in-batch negatives— and even with as few as 8 such negatives ($K=8$ in the Table), performance remains solid. Critically, the teacher’s task is limited: it only needs to pick out some strong negatives, not all of them.

Hence, this doesn’t require a very accurate teacher, as the reviewer suspected. To prove the point, we run B3 with a much weaker teacher (CLIP). Despite CLIP performing far below VLM2Vec on MMEB, using it as a teacher still delivers strong results.

All that said, a strong teacher model can still be trained using only positives and a sufficiently large batch size, without relying on any negatives. VLM2Vec teacher models were trained solely with positives and random batches, without incorporating hard negatives. VLM2Vec models used in the paper were trained with the MMEB train set containing the same modalities as the MMEB eval set.

Finally, we sincerely appreciate the reviewer’s thoughtful and insightful questions, which have helped us clarify and strengthen our work. We will incorporate the additional results, analyses, and detailed responses provided here into the final draft to ensure a more comprehensive and rigorous presentation.

Reviewer ipHG, please engage in the discussion period. I understand that the review period started over a weekend, but we only have a few days remaining in the (now slightly extended) discussion period. The authors have provided a thoughtful response to your review and you are obligated to respond to it. You should share with the authors if they addressed questions or concerns you had, and seek clarification about any questions or concerns that remain. Please post your response as soon as you can so that there is time for the authors to follow up and discussion to progress as needed.

Thank you for your thoughtful comments and time reviewing our work. We are truly pleased with the positive reception our work has garnered from you and the other reviewers.

• “method gives new SOTA performance…method is simple…should yield richer gradients…” - RM3oh
• “B3 method is rigorously formulated…supported with theoretical bounds on InfoNCE loss approximation…consistent and substantial gains, especially under small batch sizes…graph-based community detection is a novel framing ” - RipHG
• “has clear intuition and solid theoretical justification…Experiments seem solid and extensive…” - Rm62i
• “achieves sota performance despite training at smaller batch size…ablation studies validate the effectiveness of B3”-R4DdK

We equally appreciate your constructive comments, and have modified the draft to address the raised concerns.

Regarding,

... reason for why VQA tasks benefit from a larger p?

$p$ serves as a mechanism for excluding false negatives, a strategy previously employed in SFR‑Embed and NV‑Embed. For retrieval, we adopted the same value of $p=30$ as used in SFR‑Embed. NV‑Embed also showed that lower values of $p$ between $10$ and $30$ perform well. However, in the case of VQA training datasets in MMEB, the positive (target) responses were notably short. The table below reports the average target lengths for the VQA tasks.

In contrast, for image2text retrieval tasks, the average positive (target) length was as follows

Due to this reason, we found a significantly higher number of targets which had similar/same meaning as that of the labelled target. For instance, here is the first example in OK-VQA dataset. As seen, due to the short nature of the answers a higher number of targets of other examples are very similar/same compared to retrieval.

• Image: Image containing a women playing tennis
• Q: What is the hairstyle of the blond called?
• Top ranked targets: [ponytail, ponytail, ponytail, pony tail, pony tail,...]

Hence, we used a higher value of $p$ for VQA tasks. For computational efficiency, we tuned $p$ for smaller batch sizes and retained these values for larger batch sizes. Here are results for a model trained and tested on VQA datasets only at batch size ($|B|=32$) for different $p$ values.

At lower batch sizes, results improve for higher $p$ values. However, as discussed next, at higher batch sizes, $p$ has a smaller effect on the performance.

... see some study over a parameter range for both p and m

We hereby present some analysis on parameter $m$. Model trained and tested on retrieval datasets only at batch size 32.

At smaller batch sizes, the performance changes with $m$. $m=100$ as used in SFR-Embed worked well for us. We demonstrate the ablations over $p$ in the next question.

... what happens when p=0.

The table below reports results for $p=0$ across different batch sizes. At smaller batch sizes, using an elevated value of $p$ proves beneficial, effectively reducing multiple false negatives. However, at larger batch sizes, this effect diminishes; as shown, performance remains comparable between $p=0$ and $p=30$ when using higher batch sizes.

Presented below are additional variations of $p$ and $m$ at a higher batch size (512). As shown, B3 maintains consistent performance across these variations when using a larger batch size.

As shown $p=0$ works very well at larger batch sizes.

Main Takeaway (from $p$ and $m$ analysis)

• The previous tables indicate that the hyperparameters $p$ and $m$ have a greater impact on performance at smaller batch sizes. While B3 requires tuning of these parameters to achieve optimal results at smaller batch sizes, it shows increased robustness to their variation at larger batch sizes.

... can the method be reformulated to be completely self-supervised? You could start with a randomly initialized teacher network ...

• Contrastive learning relies on two key signals: positives and negatives.
• Our method specifically focuses on selecting negatives in a self‑supervised manner, given a set of positives.
• Constructing a pipeline of progressively harder negatives, as suggested by the reviewer, is feasible, but we leave this for future exploration.
• Self‑supervising positives is an orthogonal direction, and B3 can be seamlessly integrated with existing approaches such as BYOL, DINOv2, or V‑JEPA that address this aspect.

... How does B3 perform on for example standard CL image classification? … additional credence to the method.

We are not sure what the reviewer means by "standard CL image classification", but here are zero-short ImageNet1k classification results. MMEB benchmark in Table 1 additionally contains 10 different classification tasks.

We are happy to provide any further results the reviewer may wish to see.

The method relies on a pretrained model that accurately assigns similarity to the training dataset. …. Is this something you can comment on?

In B3, the teacher’s sole role is to spot potential strong negatives and group them in the same batch. We simply take ranks $p:(p+m)$ from the teacher to build batches with strong in-batch negatives— and even with as few as 8 such negatives ($K=8$ in Table 4), performance remains solid. Critically, the teacher’s task is limited: it only needs to pick out some strong negatives, not all of them.

Hence, this doesn’t require a very accurate teacher, as the reviewer suspected. To prove the point, we run B3 with a much weaker teacher (CLIP). Despite CLIP performing far below VLM2Vec on MMEB, using it as a teacher still delivers strong results.

All that said, a strong teacher model can still be trained using only positives and a sufficiently large batch size, without relying on any negatives. VLM2Vec teacher models were trained solely with positives and random batches, without incorporating hard negatives.

The method can be applied in various domains, as it enhances the standard contrastive learning paradigm

We concur with the reviewer that B3 is a highly generalizable method for contrastive training. To further substantiate our claim, we report the performance of B3, trained on the same NLI data as SimCSE (both with and without annotated hard negatives (HN)), evaluated on the STS benchmark. The corresponding SimCSE models (with/without HN) serve as the teacher models. All models (teacher and student) in the following table are 300M Roberta-Large.

B3 demonstrates notable improvements over SimCSE on the STS benchmark, validating our batch mining approach for text-only tasks. Note that even starting with HN in the teacher, we can still see improvements. In future work, we plan to extend this to larger benchmarks like MMEB.

Finally, we sincerely appreciate the reviewer’s thoughtful and insightful questions, which have helped us clarify and strengthen our work. We will incorporate the additional results, analyses, and detailed responses provided here into the final draft.