Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift

We thank the reviewer for acknowledging the importance of the topic and our theoretical contribution, and finding our results interesting.

“Lack of distinction from traditional approaches ....”

We believe the distinction between MMCL and traditional single-modality CL becomes evident upon comprehending the messages conveyed in our paper. Here, we elaborate further. (1) Any unimodal representation learning, whether supervisedCL (SupCon) or Self-Supervised-CL (like SimCLR), learns an encoder rather than a classifier. To enable classification, we still need to perform supervised learning by training a classifier on these representations, which, as demonstrated in the paper, is not robust. In contrast, MMCL can bypass supervised learning through zero-shot classification. This, when coupled with the representation structure learned through intra-class contrasting and inter-class feature sharing (analyzed in the Sec 4), leads to robustness. (2) Additionally,we discuss how the richness in captions can contribute to robustness, a factor that doesn’t exist in single-modality CL. (3) Moreover, we added a new theoretical analysis (Appendix F.2) showing that SupCon maps both core and spurious features into the same direction with similar scales in the representation space. Consequently, the spurious feature significantly impacts the predictions of a classifier trained on the representations. (3) We offer a more comprehensive discussion about comparison between MMCL, SupCon, and Self-Supervised-CL in Appendix F.1. Briefly, the two crucial properties of MMCL—contrasting between individual examples and incorporating multimodality—are both pivotal for robustness, and only MMCL possesses both these aspects. (4) Experimentally, we compare MMCL, SupCon, and Self-Supervised-CL in our new experiments on Conceptual Captions, showing the superior robustness of MMCL. Please refer to the global rebuttal.

Explanation of Intra-class contrasting

We believe the mechanism is thoroughly explained and justified in Section 4.1, further validated in experiments shown in figures 2a and 2c. Here, we briefly elaborate to aid understanding of the high-level idea, using the cow-grass example mentioned at the beginning of Sec 4.1. (1) In SL, the large-variance core feature, like a cow's true characteristics, appears unreliable for label prediction due to the variety of cow’s appearances, making it difficult to learn. On the contrary, the low-variance spurious feature, such as grass, is merely a green surface. Hence, the model easily relies on the spurious feature more than the core feature for prediction.(2) In MMCL, the training objective is to contrast unpaired images and texts (e.g., an image of cow A and a text of cow B). With detailed captions that reflect the variance in the images, learning the large-variance core feature aligns with MMCL’s objective of distinguishing different instances of cows. Consequently, the cow feature is no longer considered challenging for the model to learn. For an explanation that delves into more detailed technical aspects, please refer to the text above Thm 4.4. We would be happy to address any specific questions the reviewer may have.

Limitations of experiments

(1) Many other papers already provided the empirical evidence supporting what we're theoretically proving here, on large-scale real-world data. For example, the CLIP paper (Radford 2021, Fig 14) shows the advantage of CLIP over supervised learning; (An 2023) demonstrates that mentioning additional contextual features in captions improves robustness. Therefore, the focus of our paper is not to replicate observed empirical results, but to provide profound theoretical insights into the underlying mechanisms. (2) We conducted new experiments on Conceptual Captions, showcasing the benefits of rich captions, and providing a comparison between MMCL and different uni-modal algorithms. These additional experiments, alongside our existing results from MSCOCO, provide further support for our theoretical findings.

implications of our findings

We emphasize that our paper aims to demystify the robustness of MMCL by unraveling its underlying mechanisms. This study paves the way for further theoretical exploration in this direction. Beyond the theoretical contributions, our research offers valuable practical insights. For instance, given our findings on intra-class contrasting, future research could seek to design loss functions that further enhance intra-class contrasting to achieve even better robustness. Insights into inter-class feature sharing can guide data collection, suggesting that images with multiple objects could contribute significantly to robustness. Moreover, our analysis of rich captions provides guidance on constructing captions for robustness, suggesting that they should mention variance in core features and the shared features among classes. This insight could be used for future research to design better prompts for image (re-)captioning. These implications underscore the broad significance of our work.

We thank the reviewer for recognizing our theoretical contributions and the thoroughness of our analysis, as well as finding the experiments interesting and solid.

Further explanation of Definition 4.6

The meanings of k,c,alpha and beta can be found in the paragraph below Definition 4.6. Here we provide a more detailed explanation. (1) Introducing notations k and c serves the convenience of our analysis and statements. Alternatively, we can describe the distribution as follows: For each label y, if y is odd, then any example in this class consistently has 1 at the (y+1)//2-th coordinate. If y is even, then any example in this class consistently has -1 at the (y+1)//2-th coordinate. $c$ is just the notation we used for the 1 and -1 here and $k$ is just the notation we used for $(y+1)//2$ here for convenience. Hence, the (y+1)//2-th coordinate serves as the core feature for each class. (2) In the training set, each example in class y also holds a spurious feature at the ((y+1)//2+m)-th coordinate, sharing the sign with the core feature but with a larger magnitude $\alpha>1$. Therefore, $\alpha$ represents the strength of the spurious feature. (3) For two different classes $y_1$ and $y_2$, examples in $y_1$ also hold random values at the (y_2+1)//2 -th coordinate, with magnitude $\beta$, and vice versa. Therefore $\beta$ represents the strength of features shared between classes.

Below is a concrete example with four classes when $\beta=0.5$ and $\alpha=2$.

In the true distribution, examples in each class are given by

Class 1: $[1, \pm 0.5, \pm 2, \pm 1 ]$

Class 2: $[-1, \pm 0.5, \pm 2, \pm 1 ]$

Class 3: $[\pm 0.5, 1, \pm 1, \pm 2 ]$

Class 4: $[\pm 0.5, -1, \pm 1, \pm 2 ]$

In the training distribution, examples in each class are given by

Class 1: $[1, \pm 0.5, 2, \pm 1 ]$

Class 2: $[-1, \pm 0.5, -2, \pm 1 ]$

Class 3: $[\pm 0.5, 1, \pm 1, 2 ]$

Class 4: $[\pm 0.5, -1, \pm 1, -2 ]$

The training dataset (MSCOCO) scale is relatively smaller compared to testing?

We note that the training dataset size is larger than the test dataset size. The MSCOCO dataset comprises around 300k images, whereas ImageNet’s validation set (commonly used for testing) consists of only 50k images, not to say that we sampled a subset of classes that can be mapped to MSCOCO classes (320 out of 1000). We also provide other test dataset with different sizes for reference: ImageNetV2 size = 10k, ImageNetR size=30k, ObjectNet size = 50k, ImageNet-Sketch size = 50k, and ImageNet-A size = 7.5k.

New experiments on Conceptual Captions

We added new experiments on Conceptual Captions to enhance our empirical results. Please see the global rebuttal.

Some related works studying multimodal learning and how each modality could interact with and complement each other could be considered to be included in related works such as:

Thanks for providing these related work, we will incorporate them into the related work section in the revised version.

Variance of spurious features

(1) To theoretically characterize MMCL’s robustness, we need to consider a setting where spurious features are learned by SL in the first place. Generally, the model tends to rely more on the spurious feature when its variance is smaller than that of the core feature. This has been theoretically analyzed in Sagawa 2020. Regarding backgrounds, although some have larger variance, it's those ‘easier’ backgrounds with smaller variances that are more likely to be learned by the model. (2) The specific value of $\sigma_{spu}$ is a technical assumption, consistent with the setting in Sagawa 2020, enabling us to directly invoke their theorem for supervised learning. However, as mentioned in the footnote on page 5, our MMCL analysis extends to more relaxed assumptions where \sigma_{spu} only needs to be smaller or equal to a constant order, i.e., O(1).

Does rich details in the captions correlate with a high variance of features?

There are two facets to the richness of captions: mentioning feature variance and mentioning more features, which we have explored in Theorems 5.2 and 5.4, respectively. In Thm 5.2, the large-variance core feature is well-learned only when captions sufficiently mention its variance. If captions fail to delve into the variance of core features and treat all core features of the same kind as a single entity (e.g., use a single word `cow’ to describe all different cows), MMCL cannot attain robustness. In Thm 5.4, we show that mentioning more features generally informs the model about the irrelevance of spurious features. As a result, the model does not learn them, which enhances robustness.

In the experiments section, how is the different accuracy on In-domain distribution obtained?

As we mentioned in Appendix E.2, the different OOD-ID accuracy pairs are obtained by varying the subset size. We trained models with 25%, 50%, 75% and 100% of the original dataset.

Experiment on MSCOCO

(1) CLIP is trained from scratch on MSCOCO. (2) As described in Appendix E.2, the numbers in Fig 2 are the average over the six variations of ImageNet. (3) The original ImageNet doesn't allow running CLIP as it has no captions. Recently, Fang et al. (2022) collected captions for ImageNet and conducted experiments comparing CLIP with SL. However, they observed that CLIP achieves an equal level of robustness as SL in this scenario. Therefore, we hypothesize that ImageNet with captions might not be an ideal environment for CLIP to demonstrate its strengths, potentially due to two reasons: a) Images in ImageNet are predominantly simple, without many additional components apart from the main object, limiting inter-class feature sharing, which our research identifies as crucial for robustness; b) Given our findings that richer captions lead to better robustness, the captions available for ImageNet may not be sufficiently rich (e.g., the user descriptions on the Flickr website, from where they collected the captions, are sometimes irrelevant or overly concise). One potential future research direction we believe holds value is to explore generative models to enhance the captions and assess if this leads to improved robustness.

In Figure.2b, the accuracy drops a lot as the label approach accuracy increases, does this indicate overfitting on the labels?”

In Fig 2b, the legend 'label' refers to the case where we use the image labels as captions. By comparing this with the standard captions, we aim to validate our findings in Theorems 5.2 and 5.4. These theorems suggest that if the captions lack details in the image, such as variance of features and shared features between classes, MMCL cannot achieve robustness. Intuitively, using labels as captions is akin to supervised learning, where intra-class contrasting and inter-class feature sharing, crucial for robustness, are not enabled.

Typo on page 4

Thanks for pointing out the typo. We fixed it in the updated PDF.

Design of the semi-synthetic experiment

The semi-synthetic experiment is carefully designed to conduct controlled experiments for two goals:(1) to compare MMCL and SL’s robustness, and (2) to validate Theorem 4.7, which says that captions mentioning the core feature’s variance can enhance robustness, while also demonstrating the effect of intra-class contrasting.

This experiment setting mirrors Data Model 1 in Definition 4.1, incorporating feature masking defined in Definition 5.1 to control caption richness. Here's how it works: The digit is the core feature, while the color is the spurious feature. Image labels are based on whether the digit in the image is <5. Both features exhibit variance: the core feature has five variations within each class (0, 1, 2, 3, 4 in one class, and 5, 6, 7, 8, 9 in the other), while the spurious feature has three variations ('light', 'medium', 'dark') for each color ('blue' and 'red'). In the training set, 99.5% of images in the '<5' class have a random blue background, and the rest have a red background. Similarly, 99.5% of images in the '>=5' class have a random red background, and the remainder have a blue background.

To simplify, we simulate captions using a vector, which can be equivalently considered as follows: Given an image, the caption mentions information about both the core feature (digit) and the spurious feature (color), with the specificity controlled by $\pi_{core}$ and $\pi_{spu}$.

• For the core feature, with probability $\pi_{core}$, the caption specifies the exact digit in the image (e.g., 0, 1, 2, etc.). Otherwise, the caption only indicates whether the digit is <5 or >=5: it states '2' for all digits <5 (as 2 represents the mean of 0, 1, 2, 3, 4), and '7' for all digits >=5 (as 7 represents the mean of 5, 6, 7, 8, 9).

• For the spurious feature, with probability $\pi_{spu}$, the caption specifies the exact color in the image (e.g., ‘light blue’, ‘medium red’). Otherwise, it only indicates whether the color is blue or red. We assign each color a number for simplicity.

With this setup, we are able to demonstrate in Fig 2 that MMCL is more robust than SL, and more importantly, that larger $\pi_{core}$ (captions emphasizing the variance in the core feature) is important to achieve such robustness, as predicted by Theorem 5.2.

Reference

Miller, John P., et al. "Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization." International Conference on Machine Learning. PMLR, 2021.

Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

We appreciate the reviewer's acknowledgment of the thoroughness of our analysis, the usefulness of our insights, and their recognition that our findings are well-supported.

Additional Experiments for Enhanced Empirical Results

We added new experiments on Conceptual Captions to enhance our empirical results. Please see the global rebuttal.

Different architectures

(1) Architecture for experiments in Fig 2(b)(c). We used ResNet50 as the image encoder and Transformer as the text encoder. We added this information to Section 6 in the updated PDF (marked in blue).

(2) Existing empirical results have already shown CLIP's superior robustness across architectures, and that different architectures are on the same line in the OOD-ID plot. For instance, the original CLIP paper (Radford 2021)'s Fig 13 shows points obtained by different architectures, clearly showing CLIP (purple) outperforming supervised learning (blue) when architecture is varied. Additionally, studies like (Miller 2021, e.g., Fig 5) have indicated that different architectures are on the same OOD-ID line. Hence, given that our comparisons in experiments are based on the OOD-ID relation, we believe these findings hold across architectures.

(3) We also provide results for different backbone architectures in our new experiments on Conceptual Captions (details in Appendix G). Fig 8 shows that the lines for various architectures remain closely aligned under each algorithm, with ViT exhibiting slightly better performance than the others. Overall, CLIP consistently outperforms SL across architectures.

“Is there a way to qualitatively describe the derived or observed conclusions …"?

The following key messages are extensively discussed in sections 4 and 5 of the main paper. Here we reiterate them and hope this can enhance comprehension.

(1) Intra-class contrasting: This relates to the design of the loss function. It contrasts each image with captions that are not paired with that image, and vice versa. This strategy enables the learning of large-variance features, because these features are useful for contrasting and minimizing the loss (c.f. Thm 4.4 and the text above it).

(2) Inter-class feature sharing: This relates to both the nature of the data and the loss function. Nuanced details within images from one class, coupled with detailed captions, can offer counterexamples to the spurious correlations found in other classes. MMCL can effectively exploit these counterexamples through its loss function to avoid relying on spurious correlations (c.f. Thm 4.8 and the text above it).

(3) Rich captions: rich captions relate to the nature of textual data and serve as a pivotal element enabling (1) and (2). We've identified two crucial aspects: a) Captions need to mention feature variance to enable (1); and b) Captions should mention more features to facilitate (2). These are discussed in Sec 5.

whether the robustness is due to single modality v/s multi-modality or cross-entropy v/s contrastive loss.

This question has actually been comprehensively answered in the paper with strong support for the answer. The conclusion is: both multi-modality and intra-class contrasting (different than the type of contrasting in SupCon) are important in achieving robustness. The paper delves into their complex interactions in Sec 4 and 5. Here, we provide a simplified explanation by considering two scenarios: (1) multimodality w/o intra-class contrasting: this is exactly the setting considered in Thm 5.3 and the experiment in Figure 2b, where we only use labels as captions, meaning that no intra-class contrasting. We have already shown that the model is not robust in this case. (2) intra-class contrasting w/o multimodality (e.g., without raw-text captions): In classification tasks, this essentially reduces to the standard supervised learning (typically using CE loss) analyzed in the paper. This is because: while algorithms like self-supervised CL do involve contrasting, they only learn an encoder, instead of a classifier. Therefore, to use these representations, we eventually need to train a classifier with CE on the representations, which will still spuriously associate features in the representations with labels. In contrast, multimodality enables zero-shot classification, bypassing the need for learning a classifier with supervised CE loss, as we discussed at the end of Sec 3.2.

Details on the experiments

The details, including information about the distribution shift datasets used, were provided in Appendix E.1 and E.2. In the MSCOCO experiment, the six shifted datasets are widely utilized in other studies on distribution shift, such as Taori 2020 and Fang 2022. Should the reviewer have any further specific questions regarding the experimental details, we would be happy to answer them.

Why not consider domain generalization benchmarks such as domainbed [Gulrajani 2020] for distribution shift robustness evaluation?

The dataset referenced by the reviewer cannot be used to run MMCL as the training datasets in Domainbed don’t have captions. Instead we consider MSCOCO, widely used in multimodal learning studies (e.g., Santurkar 2022, Goel 2022, Ren 2023, Bansal 2023), and use shifted versions of Imagenet as evaluation set to include a wide range of natural distribution shifts, following previous works such as the original CLIP paper, Taori 2022, and Fang 2022.

I encourage authors to consider including prior works in the contrastive learning paradigm in the related works section for the readers to get the broader perspective on contrastive learning.

Thanks for the suggestions. We added a paragraph in Sec 2 highlighted in blue to briefly (due to the limited space) discuss more related works.

Reference

Garrido, Quentin, et al. "On the duality between contrastive and non-contrastive self-supervised learning." ICLR 2023.

HaoChen, Jeff Z., et al. "Provable guarantees for self-supervised deep learning with spectral contrastive loss." Advances in Neural Information Processing Systems 34 (2021): 5000-5011.

Liu, Hong, et al. "Self-supervised learning is more robust to dataset imbalance." arXiv preprint arXiv:2110.05025 (2021).

HaoChen, Jeff Z., and Tengyu Ma. "A theoretical study of inductive biases in contrastive learning." arXiv preprint arXiv:2211.14699 (2022).

HaoChen, Jeff Z., et al. "Beyond separability: Analyzing the linear transferability of contrastive representations to related subpopulations." Advances in Neural Information Processing Systems 35 (2022): 26889-26902.

Shen, Kendrick, et al. "Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation." International Conference on Machine Learning. PMLR, 2022.

Xue, Yihao, et al. "Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression." arXiv preprint arXiv:2305.16536 (2023).

Tian, Yuandong. "Understanding deep contrastive learning via coordinate-wise optimization." Advances in Neural Information Processing Systems 35 (2022): 19511-19522.

Santurkar, Shibani, et al. "Is a caption worth a thousand images? a controlled study for representation learning." arXiv preprint arXiv:2207.07635 (2022).

Ren, Yunwei, and Yuanzhi Li. "On the Importance of Contrastive Loss in Multimodal Learning." arXiv preprint arXiv:2304.03717 (2023).

Goel, Shashank, et al. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022): 6704-6719.

Bansal, Hritik, et al. "CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning." arXiv preprint arXiv:2303.03323 (2023).

We thank the reviewer for acknowledging our theoretical contribution and the usefulness of our insights. Below are the answers to their questions.

What about SupCon?

(1) CE loss is inevitably used even for SupCon. We note that, our paper focuses on the classification task, in which context supervised learning typically means the standard definition that refers to using CE loss to train a classifier. It's important to note that SupCon is a supervised representation learning strategy, generating an encoder rather than a classifier. To enable classification, one still needs standard supervised learning, i.e., using the CE loss to train a classifier on the representations. Consequently, the CE loss analyzed in our paper inevitably becomes a part of this process. In a broader context, the CE loss is essentially used after any unimodal representation learning to enable classification. In this sense, our analysis covers a wide range of scenarios, including both classical supervised learning with CE loss, and any unimodal representation learning followed by supervised learning with CE loss. Moreover, the majority of the experiments in the original CLIP paper (Radford 2021) focus on the advantage of CLIP over standard supervised learning with CE loss (which is what we study in our paper).

(2) Our paper provides ample evidence and reasoning that can be used to deduce why SupCon lacks robustness, making it quite evident. Referring to Section 4.1, SupCon doesn't perform intra-class contrasting; it brings all representations within the same class closer together. Referring to Section 4.2, SupCon doesn't utilize inter-class feature sharing as it solely relies on labels, thus ignoring additional information, such as the example we discussed regarding 'non-white trees in images of wolves'. The shortcomings of SupCon are also evident in Section 5, discussing the role of richer captions. Theorem 5.3 suggests that with labels as captions ($\pi$ = 0), the robustness significantly decreases – a setup closely resembling SupCon. This is empirically confirmed in Figure 2b.

(3) To be more precise, in Appendix F.2 in the updated PDF, we offer a new in-depth theoretical analysis for SupCon with both data models 1 and 2, demonstrating its lack of robustness similar to CE. The analysis shows how SupCon maps both core and spurious features to the same direction in representations, making them entangled. Hence, the spurious feature significantly impacts the predictions of a classifier trained on the representations.

(4) We also compare MMCL, SupCon, SimCLR, and SL in our new experiments on Conceptual Captions. The results demonstrate that MMCL exhibits better robustness compared to the uni-modal algorithms. Please refer to the global rebuttal for details.

(5) We offer a more involved and comprehensive discussion about comparison between MMCL, SupCon, and Self-Supervised-CL in Appendix F.1, to provide a deeper understanding and an alternative perspective on the messages delivered in the main paper.

the use of linearized loss in theoretical analysis

(1) Theoretically: As mentioned at the bottom of page 3, the linearized loss and its variations, sharing similar essence (where log exp is replaced by the linear function y=x), are widely utilized in theoretical studies and effectively capture key aspects of various self-supervised losses, including self-supervised CL (Ji 2021, Haochen 2021, Haochen 2022, Shen 2022), MMCL (Nakada 2023), non-contrastive (Liu 2021), and Supervised CL (Xue 2023). Theoretical works by (Tian 2022, Nakada 2023) have proven that when minimizing any contrastive loss using gradient descent, at each step, the direction of GD is the same as minimizing the linearized loss but with reweighted terms, illustrating the pivotal role of the linearized loss. (Garrido 2023) demonstrated through large-scale experiments that many design choices of popular loss functions, including log exp and cos similarity, are unnecessary. All of these suggest that the linearized loss captures the fundamental essence of CL or self-supervised learning in general.

(2) Empirically: Our experiments on real data with standard CLIP loss demonstrate that our theoretical findings indeed hold for practical losses. Moreover, many other papers already provided the empirical evidence supporting what we're theoretically proving here, on large data and non-linear loss. For example, the CLIP paper (Radford 2021, Fig 14) shows the advantage of CLIP over supervised learning; (An 2023) demonstrates that mentioning additional contextual features in captions improves robustness.

Therefore, considering both theoretical and empirical perspectives, we expect that the same results hold for non-linear losses with a more complex analysis.