What to align in multimodal contrastive learning?

Thank you for the accurate analysis of our work!

Reviewer: Minor presentation issue: In section 4.2, the title claims that all tasks in this subsection should be 2-modal, but in lines 356-358 some of the video-affect tasks are 3-modal. It is unclear which 2 modalities out of 3 were used for this experiment, and it can be quite confusing (especially on UR-funny which occurred in both the 2-modal experiments and the 3-modal experiments).

Thank you for raising this point! We will slightly modify the section titles to "Experiments with $n$ modalities on real-world datasets" with $n=2,3$, which we believe are less prone to confusion. We decided to separate these two sections because Section 4.2 is fully comparable to the settings presented in FactorCL. Therefore, we used the same data modalities and backbone networks as FactorCL's authors, namely: vision (image frames) and text for MOSI, UR-FUNNY and MUsTARD; and images and proprioception data for Vision&Touch. In section 4.3, we show that CoMM works with more than two modalities and that more modalities can improve the results, which is a current limitation of FactorCL.

Reviewer: (Not important, just out of curiosity) Why is MM-IMDB not listed as part of MultiBench? (Since MM-IMDB is included in MultiBench)

Indeed, MM-IMDb is included in MultiBench, but we decided to treat it as a separate case because we used raw data in our experiments instead of the pre-processed features proposed by MultiBench. This is written in lines 1177-1179 in the Appendix.

Thank you very much for your careful and critical reading of our work!

Reviewer: Despite introducing some novel ideas, the work is still a continuation of FactorCL, where the use of partial information decomposition theory was also explored to derive a suitable learning objective (even if a different one from the one proposed).

We would like to highlight that the use of Partial Information Decomposition (PID) to derive contrastive learning objectives is, to the best of our knowledge, a novel contribution. FactorCL uses traditional information theory (not PID) to derive its learning objectives, which limits their analysis to uniqueness and redundancy, without integrating synergy. This represents a crucial conceptual difference between FactorCL and our approach. By leveraging PID, we are able to derive a loss function that models uniqueness, redundancy and synergy.

Reviewer: The majority of evaluations presented by the authors are in controlled datasets (Section 4.1) or simplified environments (Section 4.2 - Multibench, Section 4.3), where the high-dimensional raw data are preprocessed into low-dimensional feature vectors (as described by the authors on Section 4.2). As such, it is not clear how the learned representations by the proposed method would work on more complex, real-world tasks (beyond the Multimodal IMDB dataset), for example on the ones proposed in MMTBench.

We first would like to emphasize that CoMM was evaluated on 7 multimodal datasets, on 10 tasks, with a diversity of data domains and included experiments with two or three data modalities. Some of these datasets were indeed pre-processed and used as low-dimensional features for the sake of fair comparisons with competitor methods. For trifeatures, MM-IMDb and Vision&Touch, we used raw features. We believe this proves how versatile CoMM is: we can use frozen encoders for any data modality (equivalent to using low-dimensional features) or train encoders from scratch, and the architecture of the encoders can be adapted to the data modality being considered.

Finally, we agree that testing our representations in MMTBench is a great next step to study how to adapt the multimodal representations obtained through CoMM to generative tasks, but out of the scope of this work.

Reviewer: Given the use of pre-processed, low-dimensional, features, instead of raw data, it might be a stretch that claiming that the authors evaluate CoMM on audio data.

Indeed, in some experiments we have used pre-processed features as proposed from MultiBench, because we needed our results to be comparable to those presented in FactorCL [1], which is the closest prior work. However, we have also included experiments with raw data in three datasets: Trifeatures, MM-IMDb, and Vision&Touch. In particular, running the experiments on MM-IMDb on raw images and text was our own choice, in order to run comparisons with modern vision and language models like CLIP, BLIP-2 and LLaVA-NeXT.

Reviewer: Furthermore, how do you apply label-preserving multimodal transformations to these features?

As detailed in Appendix Section B.2 (lines 940–953), the transformations applied to these pre-processed features—typically time-series data extracted from videos, health recordings, or force-torque readings—consist of a random composition of Gaussian noise and random sequence dropping, with the latter ranging from 0% to 80%.

Reviewer: What is the intuition for the crop augmentation being the key to learning synergy?

The intuition for the crop augmentation being key to learning synergy in the trifeature dataset can be found in Table 5. It is because cropping is crucial to learn the texture of the images, and texture is directly related to the uniqueness tasks and the synergy task: $U_1$ is measured by predicting the texture of the first image, $U_2$ is measured by predicting the texture of the second image, and $S$ is measured by the mapping defined between the texture of the first image and the color of the second one.

Indeed, by looking at lines 3 and 4 of Table 5, we observe that when we remove crop augmentation for the first modality, the model does not learn correctly $U_1$ (i.e. texture for the first modality), and similarly, when we remove crop augmentation for the second modality, the model does not learn correctly $U_2$.

References

[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pp. 8748–8763, 2021.

[2] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), pp. 4904– 4916, 2021.

[3] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. MultiBench: Multiscale benchmarks for multimodal represen- tation learning. In Neural Information Processing Systems (NeurIPS) – Track on Datasets and Benchmarks, volume 1, 2021.

[4] Liang, P. P., Goindani, A., Chafekar, T., Mathur, L., Yu, H., Salakhutdinov, R., & Morency, L. P. (2024). Hemm: Holistic evaluation of multimodal foundation models. arXiv preprint arXiv:2407.03418.

[5] Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling feature interactions: An information decomposition framework. Advances in Neural Information Processing Systems (NeurIPS), 36:27351–27393, 2023a.

[6] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems (NeurIPS), 33:9995–10006, 2020.

[7] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning?,” Advances in neural information processing systems (NeurIPS), vol. 33, pp. 6827–6839, 2020.

[8] R. Shwartz-Ziv and Y. LeCun, “To compress or not to compress–self-supervised learning and infor- mation theory: A review,” arXiv:2304.09355, 2023.

[9] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International Conference on Machine Learning (ICML), pp. 1597–1607, 2020.

[10] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual repre- sentation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729–9738, 2020.

Missing details about the synthetic experiment setups

Reviewer: Please add more details on synthetic experiment setup as previously suggested.

We apologize that this experiment was not clear enough. Indeed, the original Trifeature dataset [6] was proposed as a controlled environment to study how vision neural networks learn different features (texture, shape and color). Since quantifying multimodal interactions in high-dimensional continuous data is a hard problem [5], we decided to leverage the properties of Trifeature to design a controlled environment (yet not too simple) to test the models' capacity to learn different multimodal interactions.

As explained in the Appendix, Section E.1 (lines 1109 - 1119), the original trifeature dataset is composed of images containing one of ten shapes, rendered in one of ten textures, and displayed in one of ten colors. In total, there are 1,000 combinations of these 3 features. To generate these images shapes are rendered within a 128 $\times$ 128 square rotated at an angle drawn between $[-45^{\circ}, 45^{\circ}]$ and placed at a random position within a larger image ($224 \times 224$), such that the shape is fully contained in the image. Then, an independently rotated texture and a color are applied.

To generate our trifeature dataset, we considered the 1,000 combinations of the three features and split them into 800 combinations for training and 200 for evaluation. To have more variety in the training set for training, each combination was generated 3 times (the shape and the texture were randomly rotated), obtaining a training split of 2,400 images.

The bimodal Trifeature dataset used in our experiments was built by considering the trifeature dataset twice (as two separate modalities) and building pairs from these two dataset copies. In total, we get 5,760,000 pairs (2,400 $\times$ 2,400) available for training, and 40,000 (200 $\times 200$) available for evaluation.

To create a controlled environment for evaluation of multimodal interaction learning, we needed to carefully design tasks where the dominant interaction was clearly defined.

1. To measure uniqueness $U_1$ (resp. $U_2$), given a pair of trifeature images, the task is to predict the texture of the first (resp. the second) image. The task is then a 10-class classification problem and chance level is at 10%.

2. To measure redundancy $R$, given a pair of trifeature images with the same shape (but different color and texture), the task is to predict the shape of the pair (therefore, the redundant information that can be extracted either from the first or the second image). The task is then a 10-class classification problem and chance level is at 10%.

3. To measure synergy $S$, the definition of the task was more subtle as it should require information from both modalities simultaneously and should not be possible to perform it from one of the images alone. To achieve this, we defined a mapping $\mathcal{M}$ between the ten textures and the ten colors (e.g. stripes=red, dots=green, etc.). Then, given a pair of trifeature images, the task is to predict whether the pair satisfies the mapping or not. The task is then a binary classification problem and chance level is at 50%.

To evaluate these tasks, we built two versions of the bimodal trifeature dataset:

1. For uniqueness and redundancy, we considered 10,000 image pairs (out of the 5,760,000 pairs) for training and 4,096 for testing, that have the same shape (to measure redundancy) and different texture (to measure uniqueness).

2. For synergy, we considered 10,000 image pairs that respect the mapping $\mathcal{M}$ and used the same test set as before (4,096 image pairs).

Following your suggestion we added a new section in the appendix (Section B.5) entitled Experimental settings in on the bimodal trifeature dataset, which provides details on this experimetal design.

Mysterious number of 50% accuracy for probing synergy

Reviewer: How should a 50% probing accuracy for synergy be interpreted?

Again, we apologize this was not clear enough in the original submission. As explained in the previous reply, yes, 50% is the random chance for the task evaluating synergy. This task was cast as a binary classification problem of a pair of images belonging to a mapping $\mathcal{M}$ or not. Random chance was subtly mentioned in line 342, we have added it to Fig. 4 for more clarity. In section B.5 that we have added to the appendix we detail the evaluation setting for synergy (as in the previous reply).

Indirect evidence of CoMM’s superior learning of synergy:

Reviewer: except Figure 4 (which does not include comparison with the most similar work FactorCL), which is shown on synthetic dataset, the paper does not provide any other direct evidence for showing the claimed superior learning of synergy when evaluating on real-world datasets. For example, a large margin of performance when evaluating on datasets with primary synergistic interaction (and minimal other types of interaction), for example, VQA, will constitute strong evidence to support the claim. Also, it'll be useful to provide estimates of the proportion of different types of interaction within each real-world datasets for evaluation to better understand the advantages of CoMM for modeling different combinations of interactions.

We agree that FactorCL is an important baseline and one of the closest works to CoMM in multimodal representation learning by learning multimodal interactions beyond redundancy. Despite our efforts (a thorough grid search for hyperparameter tuning on learning rate --ranging from $10^{-5}$ to $10^{-2}$ -- and number of epochs --until 150 epochs), FactorCL does perform poorly especially on synergy. This new baseline has been added in Figure 4 for completeness.

Measuring multimodal interactions as defined by PID is challenging since it involves estimating information-theoretic measures [5]. The problem is even harder for high-dimensional data distributions. Liang et al. [5] have provided estimators for PID, and according to these measures, synergy is the dominant interaction in UR-FUNNY and MUsTARD, which is consistent with our results (see Table 1) where CoMM ourperforms FactorCL by 2.6% and 8.1%, respectively.

Very recently, Liang et al. [4] (paper to appear in NeurIPS Datasets and Benchmarks 2024), conducted a human evaluation study over 30 multimodal datasets to determine the dominant interaction. According to this study, for many VQA tasks, redundancy—not synergy—is the predominant interaction. Moreover, the multilabel classification problem MM-IMDb was found to be dominated by synergy, where again CoMM outperforms the second best method by 7.5% and 5.8% on macro and weighted F1-scores, respectively (see Table 2).

Strong assumption of Label-preserving multimodal augmentation

Reviewer: why is the Label-preserving multimodal augmentations (Assumption 1) considered a weaker and more realistic assumption? How can the mentioned mutual information quantity be measured in a realistic setting in order to choose the right augmentations?

We agree with the reviewer that Assumption 1 is still strong, but we argue that it is feasible in practice

Indeed, it has been deeply studied in the self-supervised literature in computer vision [7, 8]. Tian et al. [7] established it as the InfoMin Principle: we should reduce the mutual information between views while keeping task-relevant information intact. In their work, Tian et al. also showed that data augmentation is an effective strategy to meet this InfoMin principle, by seeking stronger data augmentation to reduce mutual information until achieving a sweet spot.

We started digging into this question in our ablation studies (see Section 5, Table 5, lines 470-480). Our results revealed that removing data augmentation on either modality (i.e. by keeping more information than the task-relevant one) prevents the model from learning synergy and its performance is also degraded for uniqueness and redundancy.

To further explore the feasibility of Assumption 1 and inspired by the strategy developed by Tian et al., we evaluate CoMM by progressively increasing the strength of the data augmentations applied. In the image domain, we use random crop as the augmentation to control, mainly for two reasons: first, it is intuitive that by decreasing the level of cropping (keeping less information about the images), we are destroying the task-relevant information; and second, because it has been empirically demonstrated that cropping is a critical transformation in self-supervised learning for vision [9, 10]. For the text domain, we use masking as the augmentation to control.

More specifically, on Trifeatures we randomly crop the original image in the first modality from 0% up to $x$% ($x=20$% is the strongest augmentation while $x=60$% is the lightest); and from $x$% to 100% ($x=0.05$% is the strongest, $x=15$% the lightest). For MM-IMDb, we also use random crop of the image modality from $x$% up to 100% and masking of text with a decreasing probability $x$% from 90% (the strongest) to 20% (the lightest).

The results on the synergy task on the bimodal trifeatures dataset are in the following table:

And the results on MM-IMDb:

Our results, both in the controlled environment of the bimodal Trifeatures dataset and in the real-world application of MM-IMDb, demonstrate that the sweet spot of the InfoMin principle can be reached. By gradually increasing the strength of the applied transformations, we enhance model performance by reducing noisy information, up to an optimal point (the sweet spot) where noise is minimized while task-relevant information is preserved. However, applying overly strong augmentations destroys task-relevant information, leading to a degradation in model performance.

Moreover, our empirical analysis (see Section 4) has shown that CoMM, with a general set of data augmentations allows for state-of-the-art performance across numerous real-life scenarios, including 7 datasets, 10 tasks and diverse data modalities.

We have added a new section on the appendix entitled On the feasibility of Assumption 1 with these results. As the reviewer suggested, this analysis can be a good starting point for practitioners to choose effective augmentations.

Thank you very much for the detailed and critical evaluation of our paper and the precise questions!

Clarifications on the multiview redundancy assumption

Reviewer: How applicable is the Multi-view redundancy assumption in the realistic settings?

The multi-view redundancy assumption can indeed be applicable to realistic settings. It is undeniable what cross-modalities contrastive methods, like CLIP [1] or ALIGN [2] --that primarily focus on extracting redundant information--, have achieved in zero-shot or few-shot transfer learning on visual classification tasks, by training on large-scale datasets of aligned images and captions.

Yet, in this paper we argue that there are real-life applications where different modalities can provide unique information (not present in the others), or where modalities together can provide synergistic information (only present when modalities are simultaneously present). This is the case in healthcare with data coming from different medical sensors; in multimedia with data coming from videos, audio and text; robotics where multiple devices and instruments provide diverse data, just to name a few examples [3].

Liang et al. [4] have recently conducted a human evaluation study over 30 multimodal benchmarks to determine the dominant interaction (redundancy, uniqueness or synergy). Out of the 30 listed datasets, less than half have redundancy as the primary interaction, showing the necessity of modeling uniqueness and synergy as well.

In such applications, only modeling redundancy is insufficient to achieve optimal results.

Reviewer: What makes it necessary to go beyond learning redundancy under the Multi-view redundancy assumption?

We fully agree with the reviewer on this point. If we know beforehand that the only interaction intervening for a certain application is redundancy (i.e., the multiview redundancy assumption holds), then there is no need to go beyond redundancy modeling.

As we previously exposed, the multiview redundancy assumption is not satisfied in many applications. Moreover, our experiments (see Section 4.1) have shown that CoMM is able to successfully learn the three types of interactions --redundancy, uniqueness, and synergy--, without assuming multiview redundancy.

Reviewer: Why are existing contrastive modeling approaches that focus on primary redundancy interaction insufficient?

Contrastive modeling approaches that focus on redundancy will not be able to capture uniqueness or synergy. Therefore, they will not achieve optimal results on tasks requiring unique or synergistic information.

This is brought to light by our experiments on the bimodal trifeature dataset (see Section 4.1, Fig. 4 in lines 298 - 313), where we observe that Cross (method focusing on redundancy) excels at the redundancy task (100% accuracy), but fails at capturing uniqueness (11.6% accuracy, only 1.6% better than chance) and synergy (50% accuracy, equal to chance).

In contrast, CoMM obtains very good results on the redundancy and in the uniqueness tasks (99.9% and 87.8% accuracy, respectively), and is the only model able to capture synergy (71.9% accuracy).

Reviewer: Without the Multi-view redundancy assumption, i.e. for other tasks, do those existing contrastive learning framework still capture primarily redundancy?

Yes. This is shown by our experiments on the bimodal trifeature dataset (see Section 4.1, Fig. 4 in lines 298 - 313), and the answer to the previous question.

Dear Reviewer ZXRV,

We are writing to kindly remind you that the rebuttal phase for ICLR submission is ongoing. We posted our responses and paper revision three days ago.

Regarding weakness 2, we have provided empirical evidence to support Assumption 1, both through experiments conducted in a controlled environment and by analyzing a real-life dataset, as detailed in Section C.4 On the feasibility of Assumption 1. Additionally, our findings are corroborated by relevant literature.

Regarding other weaknesses, we have provided further clarifications on our approach and replied to your questions.

If you have any additional feedback, concerns, or questions regarding our response, we would greatly appreciate hearing from you and would welcome further discussion.

Thank you for your interesting feedback.

Reviewer: Assumption 1 assumes the existence of a multimodal augmentation $t$ such that $I(X;t(X)) = I(X;Y)$. This means that if $X$ contains more information than the label $Y$, has to "reduce" the information in $X$ down to $Y$. This is different from just "label-preserving multimodal augmentation".

Thank you for pointing this out. You are right, Assumption 1 is more than just label-preserving augmentations, and its current name is misleading. For this reason we have changed the name to Minimal label-preserving multimodal augmentations.

Reviewer: The augmentations mentioned for real-world datasets, including cropping, jitter, etc., looks like they certainly would lead to $I(X;t(X)) \geq I(X;Y)$. This would be a direct contradiction of Assumption 1. Why does Assumption 1 hold in the real world?

This hypothesis has been deeply studied in the self-supervised literature in computer vision [1, 2]. Tian et al. [1] established it as the InfoMin Principle: we should reduce the mutual information between views while keeping task-relevant information intact. In their work, Tian et al. also showed that data augmentation is an effective strategy to meet this InfoMin principle, by seeking stronger data augmentation to reduce mutual information towards a sweet spot.

We agree with the reviewer that if $I(X;t(X))$ is much bigger than $I(X;Y)$, then the assumption would be violated and the model would be unable to learn multimodal interactions (especially synergy). However, we argue that the set of augmentations used throughout our experiments allow for the assumption to be approximately satisfied.

We started digging into this question in our ablation studies (see Section 5, Table 5, lines 470-480). Our results reveal that removing data augmentation on either modality (i.e. by keeping more information than the task-relevant one) prevents the model from learning synergy and its performance is also degraded for uniqueness and redundancy.

To further explore the feasibility of Assumption 1 and inspired by the strategy developed by Tian et al., we evaluate CoMM by increasing the strength of the data augmentations applied. In the image domain, we use random crop as the augmentation to control for two reasons: (1) it is intuitive that by decreasing the level of cropping (keeping less information about the images), we are destroying information -- that will be, at some point, task-relevant; and (2) because it has been empirically demonstrated that cropping is a critical transformation in self-supervised learning for vision [3, 4]. For the text domain, we use masking as the augmentation to control. More specifically, on Trifeatures we randomly crop the original image in the first modality from 0% up to $x$% ($x=20$% is the strongest augmentation while $x=60$% is the lightest); and from $x$% to 100% ($x=0.05$% is the strongest, $x=15$% the lightest). For MM-IMDb, we also use random crop of the image modality from $x$% up to 100% and masking of text with a decreasing probability $x$% from 90% (the strongest) to 20% (the lightest).

The results on the synergy task on the bimodal trifeatures dataset are in the following table:

And the results on the MM-IMDb:

Our results, both in the controlled environment of the bimodal Trifeatures dataset and in the real-world application of MM-IMDb, demonstrate that the sweet spot of the InfoMin principle can be reached. By gradually increasing the strength of the applied transformations, we enhance model performance by reducing noisy information, up to an optimal point (the sweet spot) where noise is minimized while task-relevant information is preserved. However, applying overly strong augmentations destroys task-relevant information, leading to a degradation in model performance.

Moreover, our empirical analysis (see Section 4) has shown that CoMM, with a general set of data augmentations allows for state-of-the-art performance across numerous real-life scenarios, including 7 datasets, 10 tasks and diverse data modalities.

We have added a new section on the appendix entitled On the feasibility of Assumption 1 with these results.