Understanding the Emergence of Multimodal Representation Alignment

We thank the reviewer and are glad that they find our experiments well-designed and motivated. Below we address the reviewer’s comments and questions.

Under “Methods And Evaluation Criteria:”

The KNN-based variant … other alignment measures would be appreciated since properly measuring alignment is challenging.

We report additional results with unbiased CKA with a RBF kernel, Mutual KNN [1], SVCCA [2] with three different sample sizes, and all metrics support our paper’s main claims. See response to reviewer B2rQ for more details.

[1] Huh et al. "The platonic representation hypothesis." (2024). [2] Raghu et al. “SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability” (2017).

Under “Experimental Designs Or Analyses:”

Correlation is measured by the Pearson correlation coefficient. However, rank-based correlation coefficients would be a bitter fit, like Spearman or Kendall’s $\tau$, since they don’t assume a relationship a priori (beyond that the order should matter).

We are happy to explore other correlation metrics. Here are the results using Spearman correlation. We find that the Spearman correlation supports our main claims: it shows a strong negative trend between maximum alignment and uniqueness and that the relation between alignment and performance is weaker or negative when there is greater uniqueness. Also, we note the Pearson correlation might actually be a suitable choice given common observations that linear relations in latent representations tend to emerge after training (e.g., see https://arxiv.org/abs/2007.00810).

Under “Essential References Not Discussed:”

Schrodi et al [4] showed that information imbalance causes the modality gap and object bias. …It’d be good to discuss the similarities and differences to Schrodi et al. in future versions.

Thank you for bringing up this related work. We agree that [4] is related to our work in that it analyzes the effect of information imbalance on the representations learned through contrastive learning, whereas our work focuses on emerging alignment through increased model capacity. We will add a discussion of [4] in our updated paper.

Under “Other Comments Or Suggestions:”

It’d be good to make more explicit how alignment is measured between visual-only models like DINOv2 and the LLMs, as done by Huh et al [1].

The details of the alignment computation are in Appendix B as follows: “Following Huh et al. (2024), we use $k=10$ nearest neighbors over 1024 samples from the Wikipedia caption dataset. For the vision model, the class token of each layer is used, and for the language model, the embeddings of a given layer are average pooled to a single token. $l_2$ normalization is applied to the features and elements in the features that are above the 95-th percentile are truncated.”

We’re happy to add any additional details that the reviewer thinks are missing.

Under “Questions For Authors:”

In the synthetic data, is information always redundant or unique for each sample or can this vary per-sample? E.g., for one sample a factor is part of the data while for another sample it is not.

The proportion of redundant to unique information is constant for all samples.

Why does the first encoder in the synthetic setting only have a single layer?

We provide additional experiment results demonstrating that our results are unchanged when $E_1$ is a higher depth. Here, we change the depth of $E_1$ to 2 and 3 and find that the results are not significantly changed. We hypothesize that because $E_1$ is trained on the untransformed modality, $E_1$ will remain relatively easy to optimize even as the depth increases. We will include these results in our updated paper.

How are the models trained in the synthetic setting? I.e., what type of training method is used? CLIP loss? Captioning loss?

In the synthetic setting, the ground truth labels are available, the models are trained in a supervised manner, with cross-entropy loss.

How are correlations computed? Since there are many points per x-value across all plots, this should lower correlations, right?

In Fig. 4 and 5, the correlation is computed using only the maximum alignment rather than all points. We will clarify this in our updated paper.

What is the effect of the number of task-relevant features? Currently, it is only set to 8. What happens when you set it to 256?

The total number of task-relevant features would not impact our results -- what matters is the proportion of redundant to unique features. If we had 256 task-relevant features, out of which 128 are shared, we would expect to see that the result is similar to $U=4$ in our setting. Our results on real-world data are on much higher dimensions -- for our experiments on Wikipedia Image Text and on MM-IMDb, we use LLMs with high dimensional latent spaces of 1024 or greater.

We thank the reviewer and are glad that they find our experimental evidence convincing. Below we address the reviewer’s comments and questions.

Under “Claims and Evidence:”

It remains open whether and how models trained on larger datasets (consisting of multiple degrees of uniqueness) can express a higher degree of alignment.

We present new results on MM-IMDb [1], a dataset for classifying movie genres with 25k paired images and texts. Our results demonstrate that the relation between alignment and performance varies depending on the classification task (see response to B2rQ for more details), suggesting that the degree of alignment depends significantly on the downstream task.

[1] Arevalo et al. “Gated multimodal units for information fusion” (2017).

Under “Experimental Designs Or Analyses:”

I focused more on synthetic experiments to understand the core message there. … Would it have been sensible to have it also for the first modality?

We acknowledge that there are many ways of defining heterogeneity, however, a benefit of leaving the first modality untransformed is that the representation that $E_1$ learns is an ideal one -- because it has a direct linear relationship with the labels -- aligning with this ideal representation could imply that the model has learned something really universal: a requirement for the Platonic hypothesis. Then if $E_2$’s representation is highly aligned with $E_1$, we can infer that $E_2$ has learned to recover information that is comparable to the untransformed modality. Nevertheless, transforming both modalities may yield insightful results, and we leave the exploration of different types of heterogeneity to future work.

I suggest including random baselines when alignment is measured and plotted (RQ1).

We have run experiments computing alignment between randomly initialized neural networks here. Results confirm that the alignment of these neural networks is constant with respect to uniqueness and that there is no correlation between alignment and performance on average.

Under “Relation To Broader Scientific Literature:”

Understanding the datasets shared information or uniqueness is something relevant also in Continual Learning [1].

Thank you for bringing up these related works. We will include a discussion in our updated paper.

Under “Other Comments Or Suggestions:”

One minor note: I do not entirely understand how the bottom part with triangles and circles should be interpreted.

In Figures 1, the triangles and circles represent data from different modalities. In Figure 2, the triangles (and other shapes) also represent data from different modalities with varying degrees of heterogeneity. We will clarify this in the final version of our paper.

There is a repetition at the beginning of section 3, from line 118 onwards. The same sentence appears in line 65.

Thank you for pointing this out. We will remove the redundancy.

Under “Questions For Authors:” About real-world experiments:

How do you measure uniqueness for MOSEI, MOSI, URFUNNY, etc? Not entirely clear how this is evaluated from human annotation

While we do not rely on exact estimates of uniqueness for the MultiBench datasets, past work [1] has sampled several data points and asked human annotators to rate the redundancy and uniqueness of each example. These ratings are shown to agree with computational estimates of redundancy and uniqueness for various MultiBench datasets.

[1] Liang et al. “Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework” (2024)

How is it the case that perturbations of the input correspond to changing uniqueness? … Is it sensible to expect there, where much information is distorted, to see any alignment at all?

We agree that our method of perturbing the Wikipedia caption dataset is not fully aligned with our definition of uniqueness. Hence, we provide new experiment results on MM-IMDb and improved our experiments on the Wikipedia-Image Text dataset. We use GPT-4 to synthesize text captions with unique information that is not present in the images, ensuring that the resulting datasets retain the semantics of real-world text and images. Our findings support our paper’s key claims (see response to obVJ for more details).

Lines 291-300 are not clear about the upper limit for alignment. How is this tested or referenced?

While the theoretical upper bound for alignment, based on the HSIC metric, is 1, our empirical results (Figures 4 and 5, indicated by the red dot) show that the observed upper limit is significantly lower. We therefore conjecture that the maximum achievable alignment is constrained by the amount of shared information between the two modalities. We acknowledge that this argument is not rigorously formalized, and we will clarify this point in the updated version of the paper.

We thank the reviewer for the review. Below we address their questions and concerns.

Under “Claims and Evidence”:

My major question concerns the heuristic of this paper on modern multimodal model design … which however is less discussed thoughout the paper.

To the best of our knowledge, our analysis of the emergence of alignment across the dimensions of uniqueness and heterogeneity is novel and fills an important gap in the literature on cross-modal alignment. While prior work—such as the Platonic Representation Hypothesis [1]—suggests that alignment tends to emerge with increasing data scale and serves as an indicator of good performance, these claims have not been rigorously examined across key characteristics of multimodal data. In this paper, we critically evaluate these assumptions and argue that while alignment may indeed correlate with performance in settings where modalities share high redundancy (i.e., low uniqueness), this relationship breaks down when the modalities are more distinct. In such scenarios, increased alignment does not necessarily translate to better downstream performance. We believe this insight is not only novel but also practically useful, as it encourages practitioners to reconsider alignment strategies in cases where they may be counterproductive.

We additionally explore the application of alignment-performance correlation for quantifying the information content of downstream tasks. Specifically, we present results on MM-IMDb [1], a dataset for classifying movie genres with image and text modalities. Our results demonstrate that the relation between alignment and performance varies depending on the classification task (see response to B2rQ for more details), which can inform practitioners when aligning modalities is beneficial. [1] Arevalo et al. “Gated multimodal units for information fusion” (2017).

Under Methods And Evaluation Criteria:

This paper is mainly analytical without giving a method. … However, in Fig. 4 and 5, the notation is never introduced before.

Thank you for the feedback. We will update our paper to define Alignment as unbiased CKA and Unique as the number of unique features used in computing the label.

Under Experimental Designs Or Analyses:

I appreciate the designs of the synthetic experiments. The uniqueness assessment and label generation are reasonable. However, the experimental setup for real benchmark seems to be unaligned with the problem settings. … the conclusion from real-world experiments may not be convincing.

We agree that our method of perturbing the Wikipedia caption dataset is not fully aligned with our definition of uniqueness. To ensure that the perturbed dataset retains the semantics of real-world text and images, we provide new experiment results that leverage GPT-4 to synthesize text captions with unique information that is not present in the images. We keep the original image data without any additional noise. We upload our perturbed text data and code for generating the perturbations here. For each (image, text) pair in the original dataset, we prompt GPT-4 to produce 10 captions with increasing levels of uniqueness: 10%, 20%, … 100%, such that the final caption contains only information that is unique to the text. As uniqueness is already introduced in the text, we keep the original images in the Wikipedia caption dataset. Using a pretrained sentence BERT model to quantify semantic similarity between the original caption and the GPT-4 captions, we find that the average semantic similarity monotonically decreases as the level of uniqueness increases. We compute the alignment between various types of vision models and LLMs. Our updated results support both claims: 1) The maximum alignment decrease with increased uniqueness. see figure here and 2) The slope of the fitted line to the alignment and performance scores decreases with increased uniqueness, showing that the relation between alignment and performance weakens. see figure here.

I am also concerned about the setting of unsymmetrical encoders in line 190-195 … while the optimization of $E_2$ can be much harder.

We provide additional experiment results demonstrating that our results are unchanged when $E_1$ is a higher depth. Here, we change the depth of $E_1$ to 2 and 3 and find that the results are not significantly changed. We hypothesize that because $E_1$ is trained on the untransformed modality, $E_1$ will remain relatively easy to optimize even as the depth increases. We will include these results in our updated paper.

Under Other Strengths And Weaknesses:

While the paper is written straight-forward, here are several weaknesses in the writing that should be improved.

Thank you for the feedback. We will revise our paper accordingly.

We thank the reviewer for the constructive criticism and are glad that they find our analysis interesting. Below we address the reviewer’s questions and concerns.

Under “Claims And Evidence”:

Is the used metric, HSIC, sufficient to reflect the alignment quality? … is highly sensitive to the chosen kernel, sample size and other hyper parameters.

We believe that the HSIC metric is sufficient to capture alignment quality, as we use a specific linear kernel consistently across all experiments. Moreover, this kernel has no hyperparameters, making it the simplest choice. To verify the robustness of our results, we also evaluate them using alternative alignment metrics. We perform additional experiments on the synthetic data with additional alignment metrics and with different sample sizes which demonstrate that our findings are robust to hyperparameters and consistent across different metrics. Specifically, we report results with unbiased CKA with a linear kernel (our original alignment metric), unbiased CKA with a RBF kernel, Mutual KNN [2], SVCCA [3] and run all metrics with 256, 512 (our original sample size), 1024 data points. We report our updated results here. For all metrics and batch sizes, the maximum alignment decreases with increasing uniqueness as well as increasing heterogeneity. Additionally, the relation between alignment, performance, and depth are consistent across different batch sizes. Across all alignment metrics, performance and depth are positively correlated over different uniqueness values, whereas alignment and performance as well as alignment and depth correlations can be weak or negative for increased uniqueness, indicating that our findings are robust to different kernels and alignment metrics.

[1] Kornblith et al. “Similarity of Neural Network Representations Revisited” (2019). [2] Huh et al. "The platonic representation hypothesis" (2024). [3] Raghu et al. “SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability” (2017).

Under “Other Strengths and Weaknesses”:

No new method is proposed, which limits the contribution of this paper.

While our paper does not propose a new method, we believe that our contributions are significant -- see our response to reviewer obVJ for a more in-depth discussion.

Most experimental analysis is based on the synthetic datasets … rather than a subset of MultiBench to study Emerge property.

We would like to emphasize that MultiBench datasets (used in Section 6) are real-world, with CMU-MOSEI and UR-FUNNY containing 22k and 16k video snippets respectively. In addition, we provide new experiment results on MM-IMDb and improved our experiments on the Wikipedia-Image Text dataset. We use GPT-4 to synthesize text captions with unique information that is not present in the images, ensuring that the resulting datasets retain the semantics of real-world text and images. Our findings support our paper’s key claims (see response to obVJ for more details).

It would be interesting to see the quantification of the uniqueness level in the real-world dataset.

We agree that quantifying uniqueness is an interesting direction, and our results have shown the potential for alignment-performance correlation to be used for quantification. While different pairs of modalities have varying levels of heterogeneity, which can make it difficult to quantify uniqueness across datasets, we propose that alignment-performance correlations can quantify information content between different downstream tasks within a given multimodal dataset. We present new results on MM-IMDb [1], a dataset for classifying movie genres with image and text modalities. Each movie can be labeled with 1 or more genres, and there are 23 classes. We compute cross-modal alignment using various vision models and language models. To measure performance, we train linear layers on the last layer hidden representations of the language models, resulting in F1-scores for each class. Our results demonstrate that the relation between alignment and performance varies depending on the classification task — we see that the slope of the linear fit to alignment and performance scores is weak or even negative, suggesting that for certain movie genres, there is greater task relevant information that is unique to the language modality.

[1] Arevalo et al. “Gated multimodal units for information fusion” (2017).

[1] proposes that different random initializations could also cause a modality gap. Will this affect the conclusion of this paper?

We would like to clarify that our experiments on synthetic data are run with 5 different seeds, and for our experiments on MultiBench datasets, we compute alignment-performance correlation over 3 seeds. Hence, we believe that our results are robust to initializations.