Cross-Modal Alignment via Variational Copula Modelling

We sincerely thank for your valuable feedback and constructive comments. We take great care in responding to several intriguing discussions raised by you as follows:

1.Applied to generic tasks

Thank you for the suggestion. To support the generality implied by our title, we added results on CMU-MOSI and POM (see Response 1 to Reviewer NgEi). CM² consistently outperforms baselines, confirming its effectiveness beyond healthcare.

2.The use of Sklar’s theorem

Thank you for the insightful comment. We use Sklar’s theorem to motivate the decomposition of multimodal joint distributions into marginals and a dependency structure, not as a guarantee of recovering the true copula. While the learned copula is parametric, it enables modeling diverse interaction types (e.g., tail dependencies). To address identifiability, we introduce priors and gradient-preserving sampling, leading to a unique (up to permutation) MAP solution. This theoretical foundation supports robust estimation in complex multimodal settings and connects classical copula theory with scalable deep learning. We will revise the claim to clarify its scope and avoid overstatement.

3.Discuss with copula-based machine learning method

Thank you for the valuable pointers. Our approach leverages parametric families (e.g., Gumbel) to capture domain-relevant properties such as tail dependence. While neural copulas offer flexibility, they introduce additional complexity and may obscure interpretability—key considerations in healthcare. We will include these works in the related discussion.

4.Significance of the result

Thank you for the comment. We performed two-sample bootstrapped t-tests comparing CM² with baselines; most results show statistically significant gains. Please see Response 2 to Reviewer NgEi for p-values supporting the improvements.

5.The connection of this theorem to the pape

Thank you for the comment. While Sklar’s theorem is classical, our contribution lies in operationalizing it within a scalable deep multimodal framework. We leverage its decomposition to decouple marginals (modeled via GMMs) from dependencies (via parametric copulas), enabling feature-level alignment under missing modalities. Our framework also addresses challenges like marginal non-identifiability and tail dependence through variational inference and gradient-preserving sampling. This structured integration of copula theory into end-to-end multimodal learning constitutes a novel and practical contribution beyond simply restating the theorem.

6.Tail distribution of diffirent copula family

We appreciate the reviewer’s point. The qualitative discussion of copula families is relevant because it highlights how different dependency structures—especially tail behaviors—affect the joint modeling process. In domains like healthcare, modeling extreme events is critical for risk-sensitive tasks. This analysis also guides practitioners in selecting appropriate copula families based on data characteristics, enhancing the interpretability and adaptability of our method.

7.How to ensure that the marginal distributions of the modalities are uniform on the unit interval

We ensure uniform marginals via the probability integral transform. In practice, for each modality we first model the latent feature distribution (typically using a flexible Gaussian mixture model) and then compute its cumulative distribution function (CDF). By transforming the latent variables through their respective CDFs, we obtain variables that are uniformly distributed over [0, 1], which is guaranteed by the properties of the CDF. This transformation is a core component of our framework, aligning the modality-specific representations to a common uniform scale and facilitating the subsequent copula-based joint modeling.

8.How much different would there be if a different copula family is used

Due to the limited number of observations, the performance across copula families tends to be more variable in the matched subset . On the other hand, the observations are more sufficient in the partially matched datasets, leading to relatively stable performance across families. This demonstrates the importance of choosing a correct copula family since the tail risks is more evident as the number of observations decreases.

9.Explicit form of $L_{obj}$

The overall objective loss in our framework is defined as $L_{obj} = L_{task} + \lambda_{cop} \cdot L_{\text{copula}}$, where $L_{task}$ is the task-specific loss (such as cross-entropy for classification tasks) and $L_{\text{copula}}$ is the negative log-likelihood of the joint copula model, with $\lambda_{cop}$ balancing the two terms. We would proofread to correct the notational errors in future versions of the manuscript.

10.Typos

Thanks for pointing out the rendering error. We will fix this in the final version.

We sincerely thank for your valuable feedback and constructive comments. We take great care in responding to several intriguing discussions raised by you as follows:

1.Impact of initial modeling of the marginals via GMM

Thank you for the insightful comment. Our initial marginal modeling, which assumes a feature-level representation, is inherently task-agnostic and thus universally applicable across different downstream tasks. By modeling each modality with a Gaussian mixture model—the most flexible and expressive assumption available in latent space - the framework can robustly capture complex, multimodal distributions. Although any marginal estimation could introduce potential biases, our joint optimization via the ELBO ensures that even minor discrepancies are corrected during copula estimation, allowing the dependency structure between modalities to be accurately aligned.

2.Validation on datasets from other domains

To evaluate generalizability beyond healthcare, we conducted additional experiments on CMU-MOSI and POM. CM² achieves the best performance across all metrics (see table below) compared to other methods, demonstrating its broad applicability. We will include these results in the final version.

3.Theoretical guarantees

We acknowledge that our current theoretical guarantee, primarily rooted in classical results such as Sklar’s theorem, may appear limited in scope. However, our primary focus in this work is methodological and applied, targeting the complex challenges of healthcare multimodal data. Our main contribution lies in demonstrating the practical efficacy of integrating copula-based alignment with flexible Gaussian mixture modeling in real-world settings, which is supported by strong empirical results. We view our current theoretical framework as a solid foundation and plan to actively explore deeper theoretical insights—such as more rigorous guarantees on the learned dependency structure—in future work.

4.The scalability to higher numbers of modalities

We assume a fully connected density model where the multivariate Gumbel copula can be obtained by the Archimedean copula $c(\mathbf{u}) = \psi^{(d)}(t(\mathbf{u})) \prod_{j=1}^d (\psi^{-1})'(u_j)$ where $\varphi(t;\alpha) = (\log t)^\alpha$ for the Gumbel copula Hence the higher number of modalities (i.e., when $M > 3$) can be handled in this case

5.Compare to unsupervised approaches

We compared CM² against unsupervised multimodal VAEs including MoPoE-VAE and MVTCAE on MIMIC4. While these methods excel in synthetic and vision datasets, they are not tailored for downstream predictive tasks. In contrast, CM² integrates copula-based alignment with task supervision, capturing fine-grained dependencies (e.g., tail risks) that are critical in healthcare. As shown in tables below, CM² consistently outperforms these methods on both fully and partially matched IHM/READM settings. We attribute this to its stronger alignment of modality-specific representations under distributional assumptions that go beyond standard VAE objectives.

Totally Matched

Partially Matched

6.Typos

Thanks for pointing out the rendering error. We will fix it in the final version.

We sincerely thank for your valuable feedback and constructive comments. We take great care in responding to several intriguing discussions raised by you as follows:

1.Modality Alignment Tasks & Alignment Losses

In our framework, modality alignment is achieved through the copula loss, which explicitly models and optimizes the dependency structure among modalities in a shared latent space. While we do not define separate alignment tasks per se, we evaluate the effectiveness of alignment by measuring cross-modal prediction performance—i.e., how well one modality can predict or reconstruct another. This serves as an implicit test of alignment quality. The copula loss, computed as the negative log-likelihood of the joint copula distribution, encourages aligned representations by capturing inter-modal dependencies beyond marginal distributions. We will clarify this connection more explicitly in the main text.

2.Title of the Work

Thank you for your suggestions. We would revised the title in future versions of the manuscript to better reflect the contents of our work.

3.Tail dependence and Copola Family Analysis

Tail dependence is critical in our application because it captures the co-occurrence of extreme events across modalities—events that are often indicative of critical outcomes in healthcare, such as severe physiological deterioration or acute anomalies in imaging. We specifically chose the Gumbel copula because its ability to model strong upper tail dependence aligns with our goal of highlighting these "strongest signals" present in the extreme ends of each modality's distribution; these signals, which correspond to rare yet significant observations, are essential for accurate risk prediction and decision-making. In contrast, a Gaussian copula would underrepresent these tail dependencies, potentially diluting the impact of extreme but clinically meaningful observations.

4.How imputation for missing latents is performed

Yes, your assumption is correct—thank you for the accurate summary.

5.Specify the dependence structure dealing with more than two modalities

In our current framework, we specify the dependence structure using symmetric copulas rather than adopting a vine copula structure. Specifically, we assume a fully connected density model where the multivariate Gumbel copula is given by $C(u_1, \dots, u_M) = \exp\{-[(-\log u_1)^\theta + \cdots + (-\log u_M)^\theta ]^{1/\theta}\}$ and a general Archimedean copula is defined as $C(u_1, \dots, u_M) = \phi^{-1}\left( \phi(u_1) + \cdots + \phi(u_M) \right),$ where $\phi$ is the generator function and $\theta$ controls the dependence strength. This symmetric, feature-level assumption makes our approach task-agnostic and computationally tractable, avoiding the quadratic complexity that a vine copula structure would introduce when imputing missing modalities. We plan to investigate more complex dependence models, such as vine copulas with tailored structure selection, in future work.

6.Discuss on Recent Multimodal Matching Work

We both adopt probabilistic assumptions on latent features—while Propensity Score Alignment (PSA) models intra-modal distributions from a causal perspective, our method leverages copula theory to model inter-modal dependencies, particularly capturing complex interactions such as tail dependence. We compare our method with PSA on MIMIC4 in the table below. Our method outperforms PSA in both matched and partially matched settings.

Totally Matched

Partially Matched

7.Calling of ELBO

We use the term "ELBO" in a generalized sense to reflect a variational inference framework that combines both supervised and unsupervised objectives. Specifically, our method models the joint latent distribution via a copula and optimizes a variational lower bound that includes the copula log-likelihood (unsupervised) and a task-specific loss. For classification tasks, this supervised term corresponds to the KL divergence between the predictive label distribution and the true label (i.e., a variational form of cross-entropy), which aligns with the evidence lower bound on $\log p(y)$ under a probabilistic decoder. For unsupervised tasks like clustering, this component can instead represent a metric-based loss (e.g., intra-cluster distances). Thus, our objective remains variational in nature, enabling principled integration of both labeled and unlabeled information, and we will clarify this probabilistic interpretation of $p(y|x_1, \dots, x_M)$ more explicitly for mathematical rigor.

8.Typos

Thanks for pointing out the rendering error. We will fix it in the final version.

We sincerely thank for your valuable feedback and constructive comments. We take great care in responding to several intriguing discussions raised by you as follows:

1.Results on other types of multi-modal datasets

Thank you for the suggestion. To evaluate generalizability beyond healthcare, we conducted additional experiments on CMU-MOSI and POM. CM² achieves the best performance across all metrics (see table below) compared to other methods, demonstrating its broad applicability. We will include these results in the final version.

2.Significance of the result

We thank the reviewer for the thoughtful suggestions. We performed two-sample bootstrapped t-tests to compare CM² against SOTA baselines and ablations. Most comparisons yielded significant p-values, as shown in table below. We will revise Tables 2 and 3 by marking results with significant differences (e.g., using *) for better clarity. Due to space limits, we were unable to include the ROC curves now but will include them in the final version for improved visual interpretation.

3.Choice of encoders

We appreciate the reviewer’s concern. While we followed encoder settings from original papers (e.g., MedFuse, DrFuse), we additionally tested each method with different encoders in Table 10 in Appendix. CM² consistently outperforms all baselines across these settings, confirming its robustness to encoder choices. We will clarify this experimental detail in the final version of the paper.

4.In-depth analysis with regard to the healthcare context

We appreciate this insightful suggestion. Our method is designed to optimize global performance metrics through robust copula-based multimodal alignment, and it inherently relies on probabilistic inference and sampling, which introduce variability at the individual case level. In our framework, each prediction is generated as an expectation over a learned latent distribution that is optimized to maximize overall AUROC and AUPR, rather than to provide deterministic case-specific outputs. This stochastic nature—especially under missing modality scenarios—limits the reliability of direct one-to-one comparisons of individual case predictions across baselines.

5.Deal with categorical features

Thank you for the question. Our GMM assumption applies to latent representations, not raw inputs. Categorical features are embedded and transformed before modelling; thus, the Gaussian assumption holds in the latent space.

6.Impact by potential conflicts among different modalities

We appreciate the reviewer’s point. Our probabilistic framework regularizes modality interactions via distributional assumptions. This helps mitigate the impact of noisy or conflicting modalities by reducing sensitivity to outliers in the joint space.

7.Typos

Thank you for catching this. The reference to Table 11 in Section 4.4 should be corrected to Table 6. We will fix this and another rendering error in the final version.