Discovering Global False Negatives On the Fly for Self-supervised Contrastive Learning

Thanks for your review and questions.

Q1: It would have been nice to see other downstream tasks than classification to validate further the quality of the learned representation.

A: Thanks for pointing this out. We agree that the unimodal performance has only been validated in the context of classification. In the bimodal scenario, please note we use the Datacomp Benchmark. This benchmark includes 38 zero-shot downstream tasks, which also include cross-modal image-text retrieval tasks. As suggested, we will modify the contributions to clarify the tasks used for validation.

Q2: Display computational overhead results in the core of the paper.

A: Thanks for your comment. We will move the computational overhead result to the main paper.

Q3: Introduction, for practical CL training on ImageNet, how many FN do we have on average? Also, it would be good to give an idea of the impact these FN have on final results (results on a particular downstream tasks can be shown in an early figure).

A: During training on ImageNet100, the empirical average of FN is 1%, that's around 20k FN per batch with a batch size of 1024 and 325 with a batch size of 128. Thanks for your suggestion. We agree both of these questions would provide a better idea of the importance of this line of work. We will make changes to the introduction to address both.

Q4: GloFND performance for fine-grained classification downstream tasks?

A: Thanks for the question, this is something we could have done a better job pointing out. Table 2 presents the performance on several fine-grained downstream datasets such as Stanford Cars, Oxford 102 Flowers, Oxford-IIIT Pets, Caltech-101, and Food-101.

Q5: About the top $\alpha$% assumption. What proportions of detected FN are actually FN in practice?

A: Table 1 presents "False Negative Identification" metrics, which evaluate what proportion of detected FN are actually FN in practice. As it can be observed, GloFND achieves much better precision, recall, and f1-score compared to FNC. Answering your question, the last epoch mean for GloFND is 48.40% of predicted FN are true FN, compared to 27.57% for FNC.

Thanks for your review and questions.

Q1: About GloFND limitations to a specific contrastive learning (CL) technique and integration on classical contrastive learning algorithms.

A: Thanks for this question, this is something we could have done a better job explaining. To clarify, the computation of $\lambda$ in GloFND is independent of the contrastive loss used, as it relies solely on the embedding similarity of negative pairs. This makes it applicable across different contrastive learning methods. In our experiments, we applied GloFND to unimodal SogCLR, bimodal SogCLR and FastCLIP. As you suggested, we have additionally run unimodal GloFND with SimCLR with results shown in Reviewer tdJR's Q2 answer. However, it is worth noting that prior work [1] has shown that SogCLR outperforms SimCLR. Additionally, SimCLR requires a large batch size, and its performance can be more sensitive to the impact of false negatives as batch size increases.

[1] Yuan, Zhuoning, Yuexin Wu, Zi-Hao Qiu, Xianzhi Du, Lijun Zhang, Denny Zhou, and Tianbao Yang. “Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance.” arXiv, September 20, 2022. http://arxiv.org/abs/2202.12387.

Q2: The novelty of this method is insufficient, and its generalizability does not seem to be well demonstrated in the paper.

A: We appreciate the reviewer’s feedback and would like to clarify the novelty and generalizability of our approach. Formulating false negative discovery as identifying the k-th largest similarity across the entire dataset is, to the best of our knowledge, a novel contribution. This formulation enables a simple yet effective algorithm based on stochastic optimization for efficient computation.

We have conducted experiments to demonstrate its applicability in unimodal CL (semi-supervised and transfer learning) and bimodal CL. Moreover, to further demonstrate generalizability, we have conducted additional experiments with SimCLR, reinforcing its broad applicability (see the previous question). We appreciate the reviewer’s concerns and are open to suggestions that could further strengthen this aspect.

Q3: The experiments in this paper seem to be insufficient. The author focuses on how to find false negative samples, but ignores the generality research in specific comparative learning or self-supervised learning methods.

A: To address your concern, we have conducted additional experiments using SimCLR (see Reviewer tdJR Q2). We have also conducted an experiment fine-tuning OpenAI's CLIP model on CC3M (see Reviewer tdJR Q3).

Q4: The author's focus does not seem to be on downstream task performance.

A: We would like to draw the reviewer’s attention to Tables 2 and 3, which present results on downstream task performance. In Table 2, we pretrain on ImageNet100, after which a logistic regression classifier is trained on top of the frozen embeddings for multiple unimodal downstream datasets. In Table 3, we pretrain on CC3M and evaluate performance on 38 zero-shot downstream tasks using the DataComp benchmark. Notably, GloFND improves downstream performance in most scenarios.

Q5: Addressing computational overhead of computing cosine similarity across the entire dataset when selecting the most similar negative samples to the anchor.

A: Thank you for the question. While GloFND computes a global threshold for the entire dataset, it does not require computing cosine similarity across all data points in the dataset. Instead, all computation is done in the mini-batch. This is what makes GloFND shine.

GloFND frames the problem as a stochastic optimization task, allowing for efficient computation. Equation 4 details how the $\lambda_I$ values are obtained, and as shown, GloFND relies only on mini-batch computations to optimize $\lambda_i$.

Importantly, the pairwise similarities are already computed as part of the contrastive loss, meaning the only additional computational overhead introduced by GloFND is:

1. Updating the $\lambda$ values for samples in the mini-batch, which involves a simple gradient computation (Equation 4).
2. Filtering false negatives, which is also done via matrix operations by comparing similarities against the computed $\lambda$ values and applying masking.

Both can run efficiently on GPUs. Overall, our method consists of basic matrix computations and runs in linear time with respect to the number of pairs in a batch $O(B^2)$, where $B$ is the batch size). This overhead is minimal compared to the cost of cosine similarity computations and the forward/backward passes. For the unimodal case, the per-epoch computation time increases by only 2% (from 427s to 435s), demonstrating the efficiency of our approach.

Thanks for your review and questions.

Q1: Experiment on larger dataset than ImageNet

A: We would like to point the reviewer to Table 3, where we tested GloFND on CC3M, which is larger than ImageNet-1k, with 2.7 million image-text pairs.

Q2: GLOFND applied on other baselines (SimCLR, VICReg, Barlow Twins, etc.)

A: We agree that it would be interesting to apply GloFND to other contrastive losses. To the best of our knowledge, VICReg and Barlow Twins do not use negative pairs, so the issue of false negative detection is not directly applicable. Regarding SimCLR, SogCLR has been previously shown [1] to outperform SimCLR, which requires a large batch size. Nevertheless, we have tried SimCLR using a batch size 512 with results shown below.

[1] Yuan, Zhuoning, Yuexin Wu, Zi-Hao Qiu, Xianzhi Du, Lijun Zhang, Denny Zhou, and Tianbao Yang. “Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance.” arXiv, September 20, 2022. http://arxiv.org/abs/2202.12387.

Q3: Fine-tuning large pre-trained model

A: We have fine-tuned OpenAI's CLIP model on CC3M using GloFND. The results on CC3M's validation set are shown below.

Q4: Comparison with finding global thresholds in the entire dataset

A: We agree such a comparison would be beneficial. Kindly note that computing the global threshold in the entire dataset would need to be done every iteration, which is intractable. Instead, we freeze the encoder network and compare GloFND with the (estimated) global thresholds. You can find this analysis in Section 4.3 (iii).

Q5: Why does the performance decrease after epoch 70?

A: Thanks for this question. We hypothesize the reason is that the total number of training epochs is fixed to 200. That is, there is a trade-off between starting at a later epoch (and having a better "sufficiently" trained network), and the amount of training epochs for GloFND to positively impact. Starting at a later epoch entails less training time with GloFND which reduces the potential improvement of removing false negatives. FNC shows a similar pattern, with its performance dropping after 110.

Q6: Improvements in the bimodal setting are less significant than in the unimodal setting

A: Thanks for pointing this out. We agree with your observation. We hypothesize that this is due to the training time because the bimodal dataset is much larger. While we train GloFND for 130 epochs in the unimodal setting, training for only 22 epochs in the bimodal setting makes the value of $\lambda$ less stable.