Cross-Modal Alignment via Variational Copula Modelling

Thank you for your comments on our manuscripts. We will respond to your comments as follows:

if the goal is to integrate multiple modalities to improve prediction, accurately modeling the joint distribution of these modalities may not be necessary

Thank you for the comments. We understand that previous multimodal learning methods focus on the empirical alignment of representations (e.g., contrastive learning via similarity) and the notion of joint distribution is rarely mentioned. However, learning the joint distribution is a more advanced way of alignment. These ways can inherently be viewed as learning the joint distribution, since their goals are all learning the fused representations (which is equivalent to the joint distribution). Moreover, due to the limit nature of the tail dependence coefficients, it is difficult to empirically estimate the tail dependence based on observed data (i.e., tail observations are rare), prompting the need for probabilistic methods. As acknowledged by reviewer haPi, using a copula to capture the joint distribution is a simple yet effective way of modelling the cross-modal interactions. Hence, we formulate the multimodal learning problem as the problem of estimating the joint distribution, where by placing a statistical model (e.g., copula), we can model the interaction structures between the modalities and obtain a better alignment result. Furthermore, learning the joint distribution enables us to generate representations for missing modalities, where the learned distribution contains information on the interactions between modalities modelled by copulas.

It is unclear how a Gaussian mixture distribution for each modality generates accurate representations for missing modalities, especially since the authors have not modeled the missing data mechanism.

Imputing representations through posterior distributions is an ideal way of generating missing representations. We follow this strategy and formulate the missing data mechanism as a data generation problem through a probabilistic model (i.e., the joint distribution and its corresponding marginals as generators). Without loss of generality, we adopt the unconditional parameterization of $\boldsymbol \mu$ and $\boldsymbol \Sigma$ where the gradients are backpropagated during training. In this sense, the means and covariance matrices of the GMMs contain not only the label information but also the structural interactions modelled by copulas. Hence, the resulting marginal GMMs can generate accurate representations for missing modalities.

What does $q(\theta)$ represent on line 224? Additionally, the connection between $f_\Theta(x)$ and the copula distribution is not clarified.

Connection between $f_\Theta$ and Copula Distribution.

The notation $\theta$ denotes the copula parameter, which inherently represents the correlation/interactions between the modalities, while $\Theta$ represents the neural network weights. To avoid confusion, we have changed $\theta$ to $\alpha$ in the revised manuscript. We have also revised the notations in the algorithms for better presentation: $\hat{y_i}$ should be just obtained from $f_\Theta(\boldsymbol x_1^{(i)}, \ldots, \boldsymbol x_M^{(i)})$ and there should be no posterior distribution for the neural network weights $\Theta$ (i.e., no $q(\Theta)$)

Typos in the manuscripts.

Thank you for pointing out the typos in the manuscript. The $c_M$ you pointed out should be $u_M$ which indicates the quantile of the $M$-th modality. We have performed a thorough check and corrected the typos in the revised manuscript.

[Q2] Missing Mechanism

I did not see your point that you can use GMM to generate better missing modalities. You did not model missing mechanisms so that you do not know whether the missing modalities were caused by some unknown mechanisms. You overstate what your model can do.

Thank you for clarifying the definition of the missing mechanism. We agree that one may leverage the observed missing mechanism in a specific problem domain (e.g., patients of certain traits have missing CXR images) to tackle the missing data problem. However, as we target general multimodal learning and have no prior knowledge of the missing mechanism, we do not (or cannot) assume a specific pattern/mechanism for the missing modality. Hence, we only adopt the most general assumption of missing modality, i.e., as we specified in line 249 of our manuscript,

 line 249: Without loss of generality, we consider missing modalities with complete labels where only the observations are missing.

Under this assumption, our strategy remains viable as we adopt a middle fusion strategy and perform imputation at the feature level (not on raw inputs $X$ and $Z$). In particular, without assuming the mechanism/pattern of the missing modality, we can still devise an imputation strategy on feature level for the missing modality. This strategy is also adopted by other works (e.g., [2-4]). From a probabilistic perspective, missing modalities can be treated as unknown variables, which can be naturally imputed using the data generation mechanism estimated by the GMM. With the help of copula and the GMM assumption, we can maximize the usage of intra- and inter-modal information to impute the missing observations.

In summary, considering that we only impute the latent representations, we believe that we do not overstate the capability of our imputation module, as it optimally utilizes the available information within the general context of multimodal learning.

[2] Tran, Luan, et al. "Missing modalities imputation via cascaded residual autoencoder." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[3] Ma M, Ren J, Zhao L, et al. Smil: Multimodal learning with severely missing modality[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(3): 2302-2310.

[4] Wang, Hu, et al. "Multi-modal learning with missing modality via shared-specific feature modelling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[Q1] Joint Distribution

Your response does not address my point on the importance of modeling the joint distribution of multi-modalities. To me, it is not necessary. Say, $Y=f(X, Z)$, where X is a set of clinical variables and Z is medical imaging. f(., .) is often approximated by using various deep learning methods, such as RNN. I believe that you do not need to model their joint distribution in order to improve the prediction accuracy.

Thank you for your reply.

We would like to emphasize that we do not claim it is necessary to learn the joint distribution in multimodal learning, while we propose it as a more promising approach to handling multimodal learning problems. Our focus throughout the paper is to demonstrate the effectiveness of copula models in facilitating multimodal learning. Our numerical experiments also demonstrate good performance using such a probabilistic approach with copula.

Back to your suggested example:

Say, $Y=f(X, Z)$, where X is a set of clinical variables and Z is medical imaging. f(., .) is often approximated by using various deep learning methods, such as RNN.

This is a general formulation of multimodal learning, where almost all methods (including ours) follow this standard pipeline. Specifically, we adopt the middle fusion mechanism in this work --- we first encode raw data input $X$ and $Z$ into latent features, say $V_1$ and $V_2$, and then fuse them into a joint/fused latent representation $V$. Finally, we make a prediction through an output layer (e.g., MLP+softmax) to predict $Y$. Under a probabilistic framework, this is equivalent to finding the posterior distribution $Y | X, Z$ or $f_{Y | X, Z}(y|x, z)$ through the use of latent variable $V$ (i.e., we first learn $V | X, Z$ through encoding, and then make predictions through $Y | V$).

During the encoding phase, we adopt the standard architectures (e.g., LSTM) to encode $X$ and $Z$ into latent embeddings. However, when fusing the representations $V_1, \ldots, V_M$ into $V$, the notion of joint distribution would become prominent. As we mentioned, traditional methods use naive concatenations or empirical alignment (e.g., through transformers) to learn the distribution of $V$, which may not approximate the distribution or representation of $V$ well, especially when the data is heavy-tailed (i.e., the tail risk is significant). Hence, we proposed that modeling the joint distribution can largely (if not necessarily) enhance the predictive performance of $Y$ as the joint distribution accounts not only the intra-modal (i.e., through marginals) but also the inter-modal information. As this adopted fundamental pipeline is standard and well-known in multimodal learning, we only mentioned it briefly in the main manuscript and cited the related works to avoid the loss of focus.

We agree with you that there could be other approaches to model data of multi-modalities, where using RNN to model $Y$ is certainly viable. Our proposal is to take a distributional approach to model the joint distribution of latent representations to obtain a better distribution of $V$, such that the prediction $Y|V$ based on $V$ can be more accurate. Because we model both the marginal and joint distributions, our approach can be viewed as a probabilistic method by aligning different modalities at the distribution level [1]. This also renders desirable interpretations with marginal effects (i.e., intra-modal effects) and joint effects (i.e., inter-modal effects). Copula, although well-established in statistics, is still less developed in machine learning, especially multimodal learning. We hope our approach could generate more interest in our research community to explore more potential usage of the copula structures.

[1] Nedelkoski, Sasho, Mihail Bogojeski, and Odej Kao. "Learning more expressive joint distributions in multimodal variational methods." Machine Learning, Optimization, and Data Science: 6th International Conference, LOD 2020, Siena, Italy, July 19–23, 2020, Revised Selected Papers, Part I 6. Springer International Publishing, 2020.

Dear reviewer Zvwa,

Thanks for your efforts and thoughtful reviews. Since the reviewer-author discussion session will end in a few days, it would be great if you could check our responses to your concerns and let us know if there are any further questions or remaining concerns. We will happily respond if so. Also, we would really appreciate it if you could consider increasing the score if we have made satisfactory responses to your questions. Thanks again for your valuable contributions to this process!

Sincerely,

CM$^2$ authors

We greatly appreciate your positive comments on our manuscript and your insightful summary of the impacts and novelty of our work. We take great care in responding to several intriguing discussions you raised as follows:

As part of the claimed contribution in this paper is to tackle the missingness, I think a common scenario in healthcare should be discussed - when certain variables or chunks of time series are missing. How can copula deal with the partially observed EHR tabular data, or time series?

Thank you for the interesting question. Since we adopt a GMM to model the intra-model distribution, each mixture component can capture the specific characteristic of the data. Therefore, even if an intra-modality data point is partially observed (e.g., some chunks are missing), these data can be imputed or recovered by the mixture components capturing the characteristics of those missing partitions. This scenarios would be incorporated in future developments of our method.

More Competitive Baselines.

Thank you for your suggestions. We have incorporated a comparison with two recent baselines, specifically LSMT(2023) [1] and Interleaved(2023) [2]. LSMT is a transformer-based model designed for the multimodal medical context, while Interleaved is a multimodal approach that addresses the irregularity of medical multimodal data and fuses representations from different modalities using cross-modal attention. We provide the results of these additional baselines on the MIMIC-IV dataset below. The results indicate that both baselines perform worse than our method on both the fully matched and partially matched datasets. However, the performance gap narrows on the partially matched dataset, suggesting that these baselines are more sensitive to dataset scale, as the partially matched dataset contains a significantly larger number of observations.

[1] Firas Khader, Gustav Müller-Franzes, Tianci Wang, Tianyu Han, Soroosh Tayebi Arasteh, Christoph Haarburger, Johannes Stegmaier, Keno Bressem, Christiane Kuhl, Sven Nebelung, et al. Multi-modal deep learning for integrating chest radiographs and clinical parameters: a case for transformers. Radiology, 309(1):e230806, 2023

[2] Xinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. Improving medical predictions by irregular multimodal electronic health records modeling. In International Conference on Machine Learning, pp. 41300–41313. PMLR, 2023.13

We thank the reviewer for your valuable comments on our manuscript. We will address your comments on the significant experiments, the GMM assumption, and the novelty of our work as follows.

My main concerns are about the Copula model. First, according to theorem 1, once the marginal distribution of each component $F_i(z)$ is provided (we first assume we can learn it well), there exits a copula C to recover the joint distribution with $C(F_1(z_1),F_2(z_2),...,F_M(z_M))$. That means, we should parameterize $q(z_1,...,z_M)$ as $C(F_1(z_1),F_2(z_2),...,F_M(z_M))$ or $C(z_1,...,z_M)\prod_{m=1}^M F_m(z_m)$. But in your variational family, the form of $q(\boldsymbol z)$ seems inconsistent with the above forms.

We appreciate your in-depth observations and we would like to further clarify to avoid potential misunderstanding. For the variational family, we use the joint density distribution instead of the cumulative distribution to obtain the log-likelihood. In fact, if one takes the partial derivatives of Equation (2) with respect to $x_1, \ldots, x_M$ (or $\boldsymbol z_1, \ldots, \boldsymbol z_M$ in our notation), they can obtain the joint density introduced by $q(\boldsymbol z)$ in the variational family. The variational objective is seemingly defined in [1], where its correctness can be verified.

[1] Tran, Dustin, David Blei, and Edo M. Airoldi. "Copula variational inference." Advances in neural information processing systems 28 (2015).

Second, why do we assume GMM to fit the marginal distribution? Have you performed goodness of test to show that latent z are mixture of Gaussian?

We discussed the reasons of assuming a Gaussian mixture model in the general response. The goodness-of-fit test in high dimensions is still a developing field in statistics and there is little research in this direction due to its formidable challenges. Even for VAE, the validity of assuming a Gaussian prior is still under intense discussion. We adopt a Gaussian mixture model in light of the many empirical successes of assuming a GMM on the latent feature distributions. With the future advances in statistical theory in high-dimensional goodness-of-fit tests, we would attempt to validate the theoretical soundness of the GMM assumptions.

Empirically, how do we choose K in the GMM?

As a convention in statistical modelling, $K$ is set to be small to avoid over-specification. The popular choice of $K$ is 2 to 3 such that the learned mixture distribution will achieve an optimal degree of flexibility while preventing over-specification. In the additional experiments (presented in Section F of the Appendix), we evaluate how the performance changes with the changes of $K$. We observe that the performance is quite robust.

We greatly appreciate your positive comments on the soundness and effectiveness of our method. We take great care in responding to several intriguing discussions raised by you as follows:

The theoretical analysis is not in-depth. Section 4.5 presents an existing theorem to justify the uniqueness of the joint distribution, but its implication to the multimodal learning problem is not analyzed clearly and in-depth.

We agree that theoretical insights or properties would be beneficial in understanding the soundness of applying copula to multimodal learning. The key focus of our work is to methodologically formulate the copula multimodal learning, with most of the discussion on how to incorporate copula to large-scale multimodal learning problems (with variational inference). Based on Sklar's theorem, the copula is a well-established method in statistics to model a joint distribution using marginal distributions (which can be viewed as multiple modalities). However, the copula framework is under-explored in machine learning, although it naturally fits to the paradigm of multimodal learning. The main implication of Sklar's theorem is that it allows us to update the copula parameter and the marginal distributions separately. Under the multimodal learning context, such properties allow us to learn the intra-modal information (through learning the marginals) and inter-modal information (through learning the copula parameter) in parallel, which avoids the collision in information and optimizes the learning outcome. The development of theoretical properties of copula multimodal deep learning requires a significant amount of work which warrants further comprehensive investigation in future works.

The marginals are assumed to be mixtures of Gaussian. This assumption seems to lack justification and no alternatives to GMM is discussed.

We discussed the reasonability of choosing the Gaussian mixture model (GMM) as the marginal distribution in the general response. Since the development of a high dimensional distribution is rare, we select the most common assumption (i.e., GMM) that presents the greatest degree of flexibility in modeling high dimensional data. Assuming more complex distributions other than multivariate Gaussian or GMM poses the risk of mis-specification, as these distributions mostly impose additional assumptions on the feature space.

We present a potential alternative by assuming only a multivariate Gaussian distribution in the revised manuscript (commonly assumed by VAE works). We observe that the performance in general worsens as the assumption is too strong and lacks flexibility compared to GMM.

How to interpret Fig. 2? Why is the Gumbel copula more focused on the positive dependence between the modalities while the Gaussian copula has less weight on modeling tail dependences?

Figure 2 presents the learned joint distribution under different assumptions of the copula family. Overall, we observe that the learned interactions are all positively correlated (i.e., slope > 0), indicating that all families capture a similar pattern of dependence. However, the pattern at tails would be slightly different due to the unique definitions of families. We present more discussions in the Appendix on how the copula distribution changes as the training epoch increases.

I sincerely appreciate the responses from the authors. My questions are mostly clarified and I will keep my rating unchanged.