A Closer Look at Multimodal Representation Collapse

We thank the reviewer for recognizing the novelty and thoroughness of our work, as well as pointing us to important adjoining multimodal learning literature that observe modality collapse. Below, we aim to address their concerns, which we will also incorporate in the final version of the manuscript.

Comparison with generative and contrastive models:

For the result in Section 4.1, we evaluate [4] by applying their proposed contrastive objective to our baseline representation learning setting on MIMIC-IV and report the results below in terms of the lowest achieved training semantic loss below.

As we can see, trends similar to that of our original setting reported in the main manuscript, in the semantic loss gap between the Multimodal Prefix and the Unimodal Baseline, play out when we perform a contrastive objective based fusion as reported in [4]. It further supports the claims in Lemma 1 and Theorem 1 that as the number of modalities increase, the modality undergoing collapse contributes less and less to the downstream representation used to encode the semantics, irrespective of the fusion strategy.

Unfortunately, we could not perform this experiment on the generative models [1-3], as it would require training each of the generative models from scratch for a number of multimodal combinations, which would take several weeks on our available compute resources, and hence, is infeasible during the rebuttal timeline.

However, we were able to perform the rank evaluation in both the contrastive and generative settings, since pretrained models are available in some cases for the latter. Due to time constraints of the rebuttal period, we chose [1] as our representative generative model. Since the objective of generative modelling is somewhat different from the downstream application that we experimented with, to analyze [1], we performed the experiment on their proposed MNIST-SVHN dataset, while for [4], since it is for general representation learning, we applied their proposed contrastive objective to our baseline setting on MIMIC-IV. Below, we report the results of our experiment, where vanilla setting refers to the original model, without KD or EBR.

We can see that in both the generative and contrastive settings, the ranks consistently drop as the strength of the modality undergoing collapse $\beta$ (written as Beta in the table) is increased. The drop is sharp around a critical point, where the rank goes below the unimodal baseline, depicting a form of phase transition, a phenomenon also observed in our original experiments (Sec. 4.2 Observations and Analyses). Finally, the dropping rank can be counteracted by implicit (KD), and even more effectively, by explicit basis reallocation (EBR), which results in a much more stable rank across the range of different values of $\beta$.

We thank the reviewer for noting important gaps in our initial submission. Below, we aim to address them, which will be included in the final version.

CM-AE: We apologize for the confusion caused here and we thank the reviewer for pointing out what was an error in our reporting. While we evaluated all the remaining baselines across the 5 missingness rates (Sec. 4.4), we somehow accidentally added the numbers for CM-AE for different random seeds but no missing modality, which is the same as the first row in Tab. 1 of the MUSE paper. Below, we correct this and report the numbers on CM-AE evaluated in the same missing modality setting as that of the others.

In line with the other baselines, CM-AE, too, performs much worse in the missing modality setting and remains significantly below our proposed EBR.

Polysemanticity: Considering the results in Fig. 5 (a) and (c), Fig. 7, and Sec. 4.3. Denoising Effect of Basis Reallocation, since there is no external source of noise in the fusion head, and encouraging monosemanticity through basis reallocation has a denoising effect, the noise that leads to the observed collapse must come from some cross-modal polysemantic interference.

To provide further evidence, we adapt the definition of polysemanticity based on neural capacity allocation from Scherlis et al., 2022, to measure cross-modal polysemanticity as the amount of uncertainty in the assignment of a neuron to a particular modality. We train a two-layer ReLU network on weights from unimodal models to classify which modality the input models are optimized on. Next, we apply this modality-classifier on the weights of our multimodal fusion head and record the average cross-entropy (CE) in its outputs. Higher values of cross-entropy indicate higher levels of cross-modal polysemanticity, since the probability masses are spread out across multiple modalities. Below, we report the results on bi-modal training:

The sharply lower relative CE for KD and EBR directly indicate the reduced cross-modal polysemantic interference under basis reallocation.

Empirical validation of Thms 1 and 2: We do not claim that Fig. 4, just by itself verifies Thms 1 and 2. Below, we provide a more comprehensive explanation:
Thm 1: Thm 1 can be factorized as: (i) as the number of modalities increase (ii) the predictive value of the weaker modality decreases. (ii) is validated by Fig. 4, where the gap in semantic loss between the multimodal prefix and the unimodal baseline increases as the number of modalities increase. (i) is validated by Section 4.3, Denoising Effect of Basis Reallocation, which shows that this drop in predictive value is indeed caused due to interference from noisy features. Combined, they imply that as the number of modalities increase, predictive features of some modalities increasingly get entangled with noisy features of another, leading to the collapse of the former.
Thm 2: Note that by definition, the lower-rank parameterization predicted by Thm 2, is likely to be polysemantic since it has to fit in more features than the number of available dimensions. Fig 5 (a) and (c) establish the decreasing nature of multimodal rank, and Fig 4 establishes the predictivity degradation implying increased noisy polysemanticity due to the former, i.e., decreased rank. This is further discussed in Sec. 4.2 and L351-355, “The rank of the default multimodal representation being bounded above by that of the unimodal baseline beyond the phase transition around the critical point, is a consequence of the upper-bound presented in Thm 2”.

Beta: We apologize for the ambiguity in the definition of $\beta$. A clearer definition is provided in Section 4.2, stating that $\beta$ “is the strength of the modality that gets eliminated by default”. Specifically, $\beta$ allows for a custom weighting of the modality that would get eliminated by default, and hence, increasing $\beta$, forces the model to incorporate it.

Terminologies and Dataset Details: We thank the reviewer for pointing out these missing details, which we will incorporate in the final version.

$\mathbf{z}$, $\mathbf{z^*}$, and $\mathbf{zz^*}$: $z$ and $z^*$ are latent factors of arbitrary dimensionality, and $zz^*$ refers to the inner product between $z$ and $z^*$ in the space of latent factors. Since we try to keep our results agnostic of the vector space from which the latent factors originate, we did not concretize $zz^*$ any further. The dimensionality of $zz^*$ would depend on the nature of the inner product of the task-specific latent space. However, we agree that it is worth clarifying this point, which we will do in the final version.

We thank the reviewer for taking the time to thoroughly understand our paper and providing important comments, which we believe has helped in significantly solidifying our findings. Below, we provide our response, which we will also incorporate in the final version of our manuscript.

Multicollinearity: We indeed expect to see increased levels of multicollinearity as the number of modalities increase, if the dimensionality of the representation space remains constant. As correctly conjectured by the reviewer, we would expect multicollinearity to be more pronounced in the deeper layers of the fusion head. The reason behind this is that although there may be dependencies among features across modalities, they may not be exactly linear. As they propagate deeper into the fusion head, it is more likely that those non-linear dependencies would be resolved and linearized in the final representation space prior to classification. Theoretically, the bound in Thm 2 is derived based on the AGOP, i.e., $\nabla\varphi_W({x}) \nabla\varphi_W({x})^T$, being a low rank subspace in W (corresponding to an independent set of features), as discussed in Radhakrishnan et al., 2024, which is also required since one-to-one dimension-to-feature mappings needed to detect the presence of multicollinearity may exist in neural networks [a]. This aligns with the condition for regression multicollinearity that $X^TX$ should be not a full rank matrix.

To empirically confirm this, we calculate the variance inflation factor (VIF) with increasing modalities on our trained representation space. We report the average VIF across features below.

With the increasing number of modalities, multicollinearity (VIF) increases in all cases. However, basis reallocation encourages cross-modal features to be encoded independently, with the explicit EBR being more efficient in controlling the level of multicollinearity relative to the implicit KD.

[a] Veaux and Ungar. “Multicollinearity: A tale of two nonparametric regressions.”, Lecture Notes in Statistics, 1994.

Definition 1: We apologize for not clarifying the details. $I(z)$ refers to the mutual information between a feature $z$ and the target label $y$, an abbreviation of $I(z; y)$ for notational minimality. Thm 1 holds in the context of definition 1. The latter is validated through our experiments in Fig. 4, Sec. 4.3. The observations therein necessitate the existence of conjugate pairs $zz^*$ across modalities, which have the capacity to cancel each other out.

Statistical Comparisons: Below we report the resulting p-values of performing the Wilcoxon rank test with Holm–Bonferroni correction (significance level $\alpha$ = 0.05) on the Table 1 results between our proposed EBR and the other baseline methods.

The null hypothesis that the proposed EBR and the other models follow the same distribution of AUC-ROC and AUC-PRCs with the chosen missingness rates, were rejected for the both Mortality and Readmission prediction tasks across all baselines, most often, with significantly low p-values, which in all cases, was lower than 0.01. It further provides evidence in support of the uniqueness of EBR in leveraging basis reallocation to free up rank bottlenecks as a novel mechanism to tackle missing modalities.

Connection between theoretical and empirical results: We provide the connections for Thms 1 and 2 in our response to Reviewer xQ3V (Empirical validation of Thms 1 and 2). Here, we provide the same for Thm 3. In Sec. 4.3, we in addition to the results on rank and representation similarity (Fig. 5), we show the difference in the loss landscape geometry between implicit and explicit basis reallocation. Further, we provide evidence that collapse happens specifically due to cross-modal noisy interference, and that basis reallocation, by freeing up rank bottlenecks, allows the new dimensions to be used for denoising, leading to its effectiveness.

Experimental Details:
$\psi$: 512 -> 256
$h$: 1024 -> 512
$h^{-1}$: 512 -> 1024
# Epochs: 1200
LR: Initially 0.01 decayed at a rate of 0.9 every 100 epochs
We interleave between the optimization of $L_{md}$ and $L_{sem}$ every 10 epochs.

We thank the reviewer for recognizing the theoretical and empirical contributions of our work towards understanding modality collapse and providing valuable feedback. Below, we address their concerns, which we will incorporate in our final version.

Identifiability: Indeed there are practical ways of ensuring identifiability of latent factors by ruling out common symmetries. For instance, Gulrajani & Hashimoto, 2022 show that if the CP (Canonical Polyadic) decomposition of the third moment tensor of the distribution of the underlying latent factors is unique and its rank is equal to the dimensionality of the vector space in which the factors lie, then general linear symmetries can be ruled out, ensuring identifiability, for which they also provide an efficient algorithm. Ahuja et al., 2022 also shows that looking at inductive biases of the causal mechanisms instead of the true underlying latent factors is sufficient from identifiable representation learning up to any equivariances shared by the mechanisms. So, our assumption on identifiability up to required symmetries is indeed a widespread one in the literature, and there are many practical methods to ensure this as well.

Unequal conditional cross-entropy across features: According to the condition $I(x; y|z_1) = I(x; y|z_2) = ... = I(x; y|z_k)$, since basins corresponding to multimodal combinations all lie at the same depth, their empirical risks are essentially the same, and so are the gradients from the ERM term. Now, as a result of modality collapse, we know that one of the basins is steeper than the rest, meaning it has a higher local gradient. Since the empirical risk is constant across all the basins / multimodal combinations, the steepness must come from the rank minimization term in Theorem 4 (Depth-Rank Duality (Sreelatha et al., 2024)). Therefore, the combination with a steep entry must lead to a lower rank solution. When the equality is not met across all features, the low-rank / steepness condition is trivially satisfied by the existence of a lower-dimensional subspace of $z_i$s that has a lower conditional mutual information $I(x; y|z_i)$, and deriving the upper-bound on the rank in terms of the AGOP is no-longer necessary. The rank of the subspace comprising features with lower relative mutual information could act as a reasonable estimate of the rank of the final weights that SGD would converge to. By considering the condition with the equality, we analyze the boundary case that even when such a subspace with low conditional mutual information cannot be identified, it is possible to upper-bound the rank of the weight matrix.