Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities

We greatly appreciate your helpful and constructive comments! We are glad that you found our contributions well motivated and clearly explained, and that you consider our solution to be simple and effective. We have addressed your questions regarding suitable baselines for Symile both below and in the global response above, but do let us know if there are any further clarifications we can provide. If we have addressed your main concerns, would you be willing to consider raising your score?

Bai et al. as a baseline: While Symile optimizes only a single term in targeting total correlation (TC), Bai et al. derive TC estimators by recursively decomposing TC into a summation of MI terms, to which variational estimators are applied, using two linear separation paths: $\text{Line-like: } TC(X_{1:i+1})=TC(X_{1:i})+I(X_{1:i};x_{i+1})$ $\text{Tree-like: } TC(X_{i:j})=TC(X_{i:\lfloor(i+j)/2\rfloor})+TC(X_{\lfloor(i+j)/2\rfloor+1:j})+I(X_{i:\lfloor(i+j)/2\rfloor};X_{\lfloor(i+j)/2\rfloor+1:j})$ The authors apply the tree-like TC estimator to multi-augmentation contrastive learning, where a text encoder is trained by maximizing the TC between augmentations of the text. For this experiment, the authors use InfoNCE to estimate the MI terms and find that their estimator does not perform much better than pairwise InfoNCE.

As we discuss in the global response, Bai et al. and others have leveraged TC estimators for contrastive learning on multiple augmentations of the same data, but to our knowledge no one has applied such estimators to more than two distinct modalities. We also explain in the global response why pairwise CLIP is the most suitable baseline for Symile, but we recognize that we could have provided a better framing for how we situate our work in comparison to Bai et al., and have updated Related Work accordingly.

Justification for new dataset: We created the Symile-M3 dataset for two reasons:

1. There should exist a dataset specifically designed to evaluate a model's ability to capture higher-order information between modes.
2. We wanted a dataset comprised of distinct data types for more than two modes.

By incorporating multiple languages, we were able to design a task where two modes (text and audio) were needed to predict the third (image), and where, importantly, neither text nor audio alone would suffice for predicting image. That said, we agree that it is important to run experiments on real-world datasets, which is why we ran the healthcare experiment on the existing MIMIC dataset in Section 5.3.

We considered video data, but found that we could not use datasets such as How2 and HowTo100M because the text is a direct transcription of the audio. When one mode is a function of another, the dataset effectively has only two modes, and total correlation between three modes degenerates to mutual information between two modes.

While we had not originally considered datasets like VQA because they incorporate only two data types, we agree that including results on standard datasets would strengthen our paper. Based on your suggestion, we ran preliminary experiments using available encoders on the VQA dataset where the three modes are 1) an image, 2) a text question (e.g. "Who is wearing glasses?"), and 3) a text answer (e.g. "woman"). On the task of predicting one of 18 possible answers given the image and question, Symile outperformed CLIP with a mean test accuracy of 0.590 vs. 0.558. We intend to run this experiment more extensively, and will add the findings to our paper.

Comparison with Liang et al.: While Liang et al. use contrastive learning objectives to optimize conditional MI terms, their approach is restricted to handling only two modalities.

For modes $X_1$ and $X_2$, Liang et al. propose two contrastive objectives (supervised and self-supervised) that apply lower bounds to maximize task-relevant information and upper bounds to minimize task-irrelevant information. The supervised objective introduces a task $Y$:

where $I_{\text{NCE}}$ is the InfoNCE objective and $I_{\text{NCE-CLUB}}$ is another contrastive objective derived from an upper bound on mutual information that depends on the optimal scoring function from $I_{\text{NCE}}$.

Instead of using explicit task labels $Y$, the self-supervised objective uses augmentations $X_1'$ and $X_2'$ to approximate task-relevant information: $L_{\text{FactorCL-SSL}}=I_{\text{NCE}}(X_1;X_2)-I_{\text{NCE-CLUB}}(X_1;X_2|X_1',X_2')+I_{\text{NCE}}(X_1;X_1')+I_{\text{NCE}}(X_2;X_2')-I_{\text{NCE-CLUB}}(X_1;X_2)+I_{\text{NCE}}(X_1;X_2|X_1',X_2').$ Unlike FactorCL, Symile does not require prior knowledge of some specific downstream task defined by either explicit labels or targeted augmentations. FactorCL seeks to decompose the information in two modes for a specific task; Symile targets all information across any number of modes to learn general representations without any decomposition, resulting in computational benefits.

While Symile optimizes a single objective with which all parameters can be updated at once, training either of the FactorCL objectives involves the minimization of each of the $I_{\text{NCE-CLUB}}$ objectives, which in turn requires the optimal critic $f^∗$ from $I_{\text{NCE}}$. Therefore, within each iteration during FactorCL training, one needs to first obtain the optimal critics for the $I_{\text{NCE-CLUB}}$ terms using the $I_{\text{NCE}}$ objective. Symile is able to capture higher-order information for more than two modes while avoiding such computational complexity.

In summary, FactorCL targets the information in two modes for a specific task, and in support of this target makes use of higher-order information. We have added this to Related Work.

Error bars: We will provide error bars for all experiments.

The reviewer is fully satisfied with the global and specific responses and increased the score. The reviewer recommends that the authors include the VQA results and the Symile-M3 dataset discussions in the final draft.

Thank you for your thoughtful comments and questions! We're thrilled that you found our work valuable and our examples and theoretical contributions effective. Please let us know if we can provide any further clarifications. If our response has resolved your questions, we would greatly appreciate it if you would consider raising your score.

Explanation for multilinear inner product (MIP): We chose the MIP as a scoring function for several reasons. We were drawn to its simplicity: the MIP is one of the simplest possible generalizations of the dot product to more than two modalities in terms of computation. The scoring function also needs to be expressive enough to model any joint statistic, which requires that the vectors be multiplied together. Therefore, the MIP strikes a nice balance between computational simplicity and expressiveness.

We understand your confusion regarding the interpretation of the MIP—the MIP is not a measure of how geometrically similar vectors are. Suppose we have three modalities $\mathbf{x}, \mathbf{y}, \mathbf{z}$ whose Symile representations are $\mathbf{r_x}, \mathbf{r_y}, \mathbf{r_z}$. If the MIP of $\mathbf{r_x}, \mathbf{r_y}, \mathbf{r_z}$ is large, then that indicates that $(\mathbf{x}, \mathbf{y}, \mathbf{z})$ has high probability under the joint likelihood. It says nothing about whether or not $\mathbf{r_x}, \mathbf{r_y}, \mathbf{r_z}$ are equal to one another. In other words, the MIP is a measure of similarity imbued by the joint distributition of the modalities.

We're glad you brought this up, and have included this clarification in the paper, as it will benefit our readers.

Baselines: You'll see that we addressed the issue of baselines and benchmarks in our global response, and in our responses to the other two reviewers. Please let us know if you have any further questions here that we can address.

We appreciate your detailed feedback, and are glad that you found our work conceptually and theoretically compelling. We address your questions below, but please let us know if we can provide any further clarifications. If we have successfully addressed your primary concerns, would you be willing to consider raising your score?

ImageBind as a baseline: As we discuss in the global response above, the baseline that we currently use in our paper is pairwise CLIP, and ImageBind is an instantiation of pairwise CLIP. That said, your feedback indicates that we could have made this clearer in Related Work, which we have now updated.

Accommodating data missingness: It seems there was a misunderstanding here. In our missingness experiment in Section 5.2, we do indeed train Symile with missingness in the training data. On the right-hand side of Fig. 5, we show that Symile outperforms pairwise CLIP on the 2-word subset of Symile-M3 when only 12.5% and 2.7% of the training data has complete triplets! We will update this section so that it's clearer to readers that this experiment was run with missingness in the training data.

We absolutely agree that in order for Symile to be at all useful for practical applications, it needs to be able to accomodate any amount of missingness in the data, which is why we ran the missingness experiments in Section 5.2. Our approach for handling missingness was specifically designed to be easy to implement for practical applications. Please let us know if there is additional information we can provide to help clarify this misunderstanding.

Real-world scenarios where Symile outperforms CLIP: We're glad that you found the XOR example to be clear and illustrative of the types of information that Symile and pairwise CLIP capture, and you are correct that it represents an extreme case that is probably rare in real-world applications. To see why, consider again the total correlation decomposition in line 115 of our paper: $3 \cdot TC(x,y,z) = 2 \cdot [MI(x;y)+MI(y;z)+MI(x;z)] + MI(x;y | z)+MI(y;z | x)+MI(x;z | y).$ In the XOR example, the CLIP target is zero, but as you rightly suggest, most real-world cases will contain some pairwise information.

Based on the decomposition above, Symile will outperform CLIP in situations where the higher-order (conditional) information terms play a role. This is typically seen in applications when two modes are needed to infer the third.

We include one such example in Section 5.3 of our paper on the task of predicting chest X-ray from ECG and labs: there is pairwise information to learn, which explains why CLIP performs better than random guessing, but the presence of conditional information allows Symile to ultimately outperform CLIP. As suggested by Reviewer hrNA, we ran preliminary experiments on the VQA dataset and found that Symile outperforms CLIP, illustrating another instance where conditional information benefits Symile.

Multilinear inner product (Q2): The dot product is only defined for two modes. The coordinate-wise sum of the element-wise product is one of the simplest possible generalizations of the dot product to more than two modes. Please let us know if we misunderstood the question or can clarify further.

Negative sampling performance impact and lower bound connection: The results we report in the paper on the Symile-M3 dataset were run using $O(N)$ negative sampling. We have now also run this experiment on the 2-word subset of Symile-M3 using $O(N^2)$ negative sampling. We find that when trained using $O(N^2)$ negative sampling, Symile demonstrates a marginal improvement in test accuracy over training with $O(N)$ negative sampling (0.938 vs. 0.937). However, Symile trained with $O(N^2)$ negative sampling takes almost twice as long to run 24 epochs (43 vs. 24 hours), but achieves its best validation accuracy in far fewer epochs (4 vs. 16). We have added these results to the Appendix.

As discussed in Section 5.3, we found that training with $O(N^2)$ negative sampling on the Symile-MIMIC dataset helped mitigate overfitting. We hypothesize that this is because more negative samples create a more challenging learning problem, allowing the model to better learn to differentiate between positive and negative samples.

In terms of analyzing the difference between the negative sampling schemes in terms of the derived lower bound, in lines 156-157 we write, "We show in Appendix B.3 that, as $N$ gets larger, the total correlation lower bound closes for the optimal scoring function $g^*$." This implies a computational-statistical trade-off: a larger batch size demands more computation but results in a tighter bound.

Colors in Fig. 1: The colors are meant to loosely represent the information in each of the three random variables. We carefully considered how best to structure this figure: on the one hand, we wanted a high-level and intuitive illustration of the different types of information that Symile and CLIP each capture (and it seems that Reviewer hrNA appreciated this); on the other hand, we understand that this figure is underspecified. Ultimately, we decided to err on the side of building the reader's intuition.

SimCLR vs. CLIP in Related Work: We write in Related Work that CLIP "popularized the use of a contrastive objective for general image-text representation learning," but we agree with you that SimCLR popularized contrastive learning for multiple augmentations for a single modality. We will update this section accordingly.

$Z_n$ typo: Thank you for catching this!