Learning Multi-Modal Representation Alignments from Noisy Data-Pairs

Compare to DreamData: We thank the reviewer for the feedback. We contend that our approach diverges from the focus on data filtering, as seen in DataComp [9]. Instead, our method is designed to address the challenge of handling noisy data pairs within the dataset during training. These are distinct problems with different emphases. While we acknowledge the benefits of data filtering in terms of producing a cleaner dataset, we will incorporate discussions in the related works section, recognizing its relevance to distinct problem domains.

Novelty: Our method is taken from the perspective of Bayesian learning, which is novel and not considered by existing works in the domain of self-supervised learning. Estimating the variance of predictions is an inherited advantage of Bayesian models. This is because in Bayesian models, the variables of interest (e.g., the weights of the data pairs) is in the form of a distribution. Given the prior and likelihood, their posterior distributions can be effectively calculated. And given the posterior distribution, its confidence (e.g., in the form of variance) can be directly estimated. We will add more background knowledge of Bayesian inference in the revision.

References: Robust Cross-Modal Representation Learning with Progressive Self-Distillation use teach-student distillation for soft-alignment of data pairs.[1]
RINCE: rank the image-text data pairs into different degrees of similarity for robust learning.[2]
Cross-Modal Retrieval with Partially Mismatched Pairs: utilize negative pairs to tackle situation with mismatched pairs.[3]
OpenCLIP: https://github.com/mlfoundations/open_clip [4]
Understanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning.[5]
CyCLIP: Cyclic Contrastive Language-Image Pretraining.[6]
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. [7]
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. [8]
DataComp: In search of the next generation of multimodal datasets. [9]

Novelty and comparison to DeCL: Thank for for the comments however we want to emphasize that our method distinguishes from DeCL. Please note that DeCL is the first work to introduce Bayesian inference into contrastive learning. However, its motivation is completely different from ours: DeCL aims to resolve the gradient bias issue whereas ours aims to resolve the noisy pair problem. From the technical perspective, although our method is built upon DeCL, the main technical novelty is to introduce the “weight” random variable, which was not considered in DeCL.

Related works: Acknowledging the significance of ALIGN [7] and BLIP[8] in the field, we have already cited ALIGN in our paper and will include references to the BLIP series as well. It is essential to note that these frameworks, while serving as pivotal contributions to general vision-language multi-modal learning, do not specifically address the challenge of mitigating noisy data. As foundation models, both BLIP and ALIGN have been pre-trained on extensive datasets, numbering in the billions. Consequently, a fair comparison with our approach may be challenging due to the differences in dataset scale. To provide a comprehensive overview, we will include more foundational model papers in the related works section.

Experimental results: The accuracy of 76.2% in ALIGN [7] is achieved through pre-training on a substantial dataset of 1.8 billion image-text pairs. In contrast, our pre-training utilizes CC3M, a dataset comprising only 3 million image-text pairs. It is noteworthy that our results align with the state-of-the-art (SOTA) in image-text pre-training methods when using CC3M for pre-training. Specifically, our performance is consistent with reported accuracies in the OpenAI CLIP repository [4], as well as results of 16.72% in [5] and 20.03% in [6].

Add more backbone: We appreciate the suggestion. It is acknowledged that, owing to the substantial computational demands, many state-of-the-art (SOTA) papers often opt for a single vision encoder, such as ResNet, for pre-training. Nevertheless, we will heed the recommendation to incorporate experiments with ViT backbone in the final version.

Linear evaluation datasets: We used ImageNet for zero-shot evaluation thus we did not perform linear probing on the same dataset. We will add the linear probing results on ImageNet into our final version.

Clarification on notation: In Eq.1, we use index i and j to refer data within the minibatch, i.e., i refer to the i-th data in the batch, and $s_{i,j}$ represent the similarity score between the i-th image and the j-th text. We will add more details in the method section to make sure the notations and derivations are more clear and smooth.

References: Robust Cross-Modal Representation Learning with Progressive Self-Distillation use teach-student distillation for soft-alignment of data pairs.[1]
RINCE: rank the image-text data pairs into different degrees of similarity for robust learning.[2]
Cross-Modal Retrieval with Partially Mismatched Pairs: utilize negative pairs to tackle situation with mismatched pairs.[3]
OpenCLIP: https://github.com/mlfoundations/open_clip [4]
Understanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning.[5]
CyCLIP: Cyclic Contrastive Language-Image Pretraining.[6]
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. [7]
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. [8]

Related works: Thank you for the valuable comments and suggestions for related works. We acknowledge the significance of incorporating discussions on these works in both the related works section and the experimental comparison. Specifically, in the case of RINCE [2], it achieves robustness by ranking data pairs into different degrees of positives. However, it is essential to note that RINCE necessitates additional information/ground truth, such as super-classes and sub-classes in the CIFAR dataset, for ranking positives. This requirement may not always be practical in the context of large-scale vision-language datasets. In contrast, our approach, rooted in Bayesian inference, does not rely on any supplementary information to reweight data pairs. While RINCE addresses the same issue of contrastive learning concerning noisy one-to-one mapping data pairs, our method distinguishes itself by its independence from additional information requirements. We will incorporate RINCE as a baseline for comparison in the final version of our work.

Background information: Our model is taken from the perspective of Bayesian learning, which is novel and not considered by existing works in the domain of self-supervised learning. In Bayesian inference, we assign a prior distribution for random variables of interest (the Gamma distribution for w in our case), which stands as our prior belief on the value of the random variables. Together with the likelihood from the loss function, we can approximate the posterior distribution. Bayesian inference is the method that aims to approximate the posterior distribution of the random variables efficiently (in our case, we resort the sampling). Thanks for the comment, we will add sufficient background into our revision and make the paper more clear.

Visualization: Thank you for the recommendation. In Figure.3 we visualize the weight distribution of our approach . We will indeed incorporate your suggestion by extending the visualization to include similarity scores from different approaches. We will provide a more comprehensive illustration of the effectiveness of our method.

References: Robust Cross-Modal Representation Learning with Progressive Self-Distillation use teach-student distillation for soft-alignment of data pairs.[1]
RINCE: rank the image-text data pairs into different degrees of similarity for robust learning.[2]
Cross-Modal Retrieval with Partially Mismatched Pairs: utilize negative pairs to tackle situation with mismatched pairs.[3]
OpenCLIP: https://github.com/mlfoundations/open_clip [4]
Understanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning.[5]
CyCLIP: Cyclic Contrastive Language-Image Pretraining.[6]
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. [7]
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. [8]

Related works: Thank you for the valuable comments and suggestions for related works. We will add the discussion of these works into related works as well as comparison in the experimental section. [1] applies teacher-student network in a self-distillation fashion to generate soft targets for a partition of data pairs in the batch, so that many-to-many mapping can be achieved. Nevertheless, it is crucial to highlight that this method remains dependent on the current model for the generation of soft targets. In contrast, our stochastic approach, rooted in Bayesian inference, is inherently independent of the model. RINCE [2] achieves robustness through ranking the data pairs into different degree of positive; however it requires additional information/groundtruth to rank positives such as super-classes and sub-classes in CIFAR dataset, which is not always feasible in large-scale vision-language datasets. Our approach does not require such additional information to achieve robust models. [3] leverages negative samples as complementary learning to alleviate the impact of partially mismatched pairs. In contrast, our approach involves the reweighting of all data pairs. These baseline methods will be included for comprehensive comparison in the final version of our work.

Experimental results: In Table 2, our method is assessed against distribution shift benchmarks, demonstrating superior performance on three out of the four datasets compared to the state-of-the-art (SOTA). It is important to highlight that across various benchmarks, such as zero-shot evaluations (Table 1) and linear probing assessments (Table 3), our approach consistently outperforms the current state-of-the-art by a significant margin. It is noteworthy that, among the diverse tasks and 18 evaluation datasets encompassed, our method exhibits only a marginal decrease in performance on one dataset within the distribution shift benchmark—an outcome we deem reasonable.

Novelty of approach: Our model is taken from the perspective of Bayesian learning, which is novel and not considered by existing works in the domain of self-supervised learning. In Bayesian inference, we assign a prior distribution for random variables of interest (the Gamma distribution for w in our case), which stands as our prior belief on the value of the random variables. Together with the likelihood from the loss function, we can approximate the posterior distribution. Bayesian inference is the method that aims to approximate the posterior distribution of the random variables efficiently (in our case, we resort the sampling). Estimating the variance of predictions is an inherited advantage of Bayesian models. This is because in Bayesian models, the variables of interest (e.g., the weights of the data pairs) is in the form of a distribution. Given the prior and likelihood, their posterior distributions can be effectively calculated. And given the posterior distribution, its confidence (e.g., in the form of variance) can be directly estimated. We will add more background knowledge of Bayesian inference in the revision.

Increase noise level: Thank you for the suggestion. Indeed we performed experiments with increased noise levels to test the performance. We will include the following results in the supplemental materials:

References: Robust Cross-Modal Representation Learning with Progressive Self-Distillation use teach-student distillation for soft-alignment of data pairs.[1]
RINCE: rank the image-text data pairs into different degrees of similarity for robust learning.[2]
Cross-Modal Retrieval with Partially Mismatched Pairs: utilize negative pairs to tackle situation with mismatched pairs.[3]
OpenCLIP: https://github.com/mlfoundations/open_clip [4]
Understanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning.[5]
CyCLIP: Cyclic Contrastive Language-Image Pretraining.[6]
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. [7]
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. [8]