Multimodal Patient Representation Learning with Missing Modalities and Labels

We would like to thank the reviewer for the positive feedback. We have made the necessary updates to our manuscript. Below we would like to take this opportunity to respond to your questions.

---

Q1. Additional experiments with new datasets and baseline.

A1. In response to the reviewer's suggestion, we have conducted additional experiments on new datasets and baselines.

New datasets.

We have conducted additional experiments on datasets from the MULTIBENCH [1] benchmark encompassing various areas such as Robotics, Finance, HCI, and Multimedia. Due to the time constraint, we selected one dataset from each area and evaluated our proposed MUSE and the two best-performing baselines (GRAPE and M3Care). To simulate the missing modalities and labels scenario, we randomly masked out labels with a probability of 0.5 and modalities with a probability of 0.1. In summary, our results indicate that MUSE performs exceptionally well on a variety of tasks and domains, demonstrating its generalizability.

Table: Additional experiment on the MULTIBENCH benchmark. MUSE achieves the best performance on 3/4 tasks, and MUSE+ further outperforms all baselines by incorporating samples with missing labels into the training process.

New baselines.

We have extended our comparison by including three additional baselines: MedFuse [2], MMIN [3], and MMCL [4]. Notably, MMCL also incorporates samples with missing labels during training, making it a fair comparison with our enhanced method, MUSE+. The table below summarizes the performance comparison, with MUSE and MUSE+ consistently outperforming all baselines.

Table: Comparison between MUSE and three additional baselines in terms of AUC-ROC scores. MUSE and MUSE+ still outperform all baselines.

---

Q2. What if we pre-train MUSE with two contrastive learning terms and then fine-tune it using the classification loss term? What if we regard labels as a subset of features?

A2. We appreciate the thoughtful suggestion from the reviewer. Initially, we employed a two-stage training approach and subsequently merged them into an end-to-end training pipeline to improve efficiency. Empirically, we observed that both training schemes yield comparable performance, but the end-to-end approach significantly reduces training time. Regarding the idea of treating labels as a subset of features, we find it intriguing and believe it presents an interesting avenue for future research. We will include this idea in the future work section of the updated manuscript.

---

Thank you once again for taking the time during the rebuttal phase. We hope that these updates address your questions.

---

Q5. In the Missing Modalities and Labels paragraph, it is unclear what the text means by “fail to fully use inter-patient relationships"?

A5. We appreciate the reviewer's feedback and recognize the need for clarification. Modality imputation approaches usually try to generate the missing modalities based on the available modalities from the same patient. However, we believe that the missing modalities could also be inferred from similar patients, while modality imputation approaches do not explicitly model the relationship between different patients.

---

Q6. Storing and Reusing Similarity Calculations.

A6. In a minibatch containing B samples, we calculate similarities between every pair of patients (resulting in B^2 similarity scores in total). These calculated similarities are reused both in the unsupervised and supervised contrastive loss functions. We will provide additional clarification in our revised manuscript.

---

Q7. It is stated in the Training and Inference Section that for MUSE, the authors only feed the original graph through a GNN, but this contradicts the model diagram, which shows both augmented and original graphs going through a GNN. Can this be explained or clarified in the text?

A7. We apologize for the confusion. The model diagram illustrates the training process, where both the augmented and original graphs pass through the GNN. During inference, the augmentation branch is omitted. We will revise our manuscript to make this distinction explicit.

---

Q8. The implementation details are unclear. “We train all models for 100 epochs on the training set, and select the best model by monitoring the performance on the validation set.” What is meant by “all models”? Does this include the baselines or only differently initialized MUSE models? Are the authors using the validation performance to determine all hyperparameters, and what specific metric is used? Is there early stopping for epochs, or are models trained for the full 100 epochs?

A8. To clarify, we trained both MUSE and the baseline methods for 100 epochs. We saved model weights at the end of each epoch and selected the best-performing model by monitoring its performance on the validation set. The specific metric monitored varies depending on the task: we used the PR-AUC score for mortality and readmission tasks and the one-vs-one macro AUC-ROC score for the Alzheimer's disease prediction task.

---

Q9. The paper cites the Transformer paper (Vaswani et al.) as the backbone encoder for several tabular modalities. The original Transformer paper cited is meant for text and not necessarily tabular data, so were there further adjustments made for this encoder?

A9. We use the Transformer encoder to encode the medical codes (e.g., diagnosis, procedures, and medications). However, they are sequence data instead of tabular. Each patient visit usually has a sequence of medical codes recorded at different timestamps. We will make this clear in the updated manuscript.

---

Thank you for taking the time during the rebuttal phase. We hope that these updates address your questions.

---

[1] SimCSE: Simple Contrastive Learning of Sentence Embeddings. EMNLP 2021.

[2] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.

[3] Handling Missing Data with Graph Neural Networks. NeurIPS 2020.

[4] MULTIBENCH: Multiscale Benchmarks for Multimodal Representation Learning. NeurIPS 2021.

[5] MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images. MLHC 2022.

[6] Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities. ACL 2021.

[7] Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data. AISTATS 2023.

[8] Multitask learning and benchmarking with clinical time series data. Nature Scientific Data 2019.

[9] Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformer. AAAI 2020.

[10] Reproducibility in critical care: a mortality prediction case study. MLHC 2017.

We would like to express our gratitude to the reviewer for their positive feedback. We have made the necessary updates to our manuscript. Below we would like to take this opportunity to respond to the reviewer's questions.

---

Q1. Design Choice & Hyperparameter Tuning.

A1. We would like to note that we did not extensively tune the hyperparameters for the experiments. Specifically, we only tune the learning rate for both MUSE and the baseline methods while keeping the other hyperparameters aligned. We leave extensive hyperparameters tuning for future works. As for design choice, we largely adhered to the settings presented in some of the classic papers, including the use of cosine similarity [1], a 15% edge drop rate [2], and the implementation of Siamese GraphSage [3]. It is important to note that we try to align the experimental settings between MUSE and other baselines as much as possible to ensure a fair comparison.

---

Q2. Additional experiments with new datasets and baseline.

A2. In response to the reviewer's suggestion, we have conducted additional experiments on new datasets and baselines.

New datasets.

We have conducted additional experiments on datasets from the MULTIBENCH [1] benchmark encompassing various areas such as Robotics, Finance, HCI, and Multimedia. Due to the time constraint, we selected one dataset from each area and evaluated our proposed MUSE and the two best-performing baselines (GRAPE and M3Care). To simulate the missing modalities and labels scenario, we randomly masked out labels with a probability of 0.5 and modalities with a probability of 0.1. In summary, our results indicate that MUSE performs exceptionally well on a variety of tasks and domains, demonstrating its generalizability.

Table: Additional experiment on the MULTIBENCH benchmark. MUSE achieves the best performance on 3/4 tasks, and MUSE+ further outperforms all baselines by incorporating samples with missing labels into the training process.

New baselines.

We have extended our comparison by including three additional baselines: MedFuse [2], MMIN [3], and MMCL [4]. Notably, MMCL also incorporates samples with missing labels during training, making it a fair comparison with our enhanced method, MUSE+. The table below summarizes the performance comparison, with MUSE and MUSE+ consistently outperforming all baselines.

Table: Comparison between MUSE and three additional baselines in terms of AUC-ROC scores. MUSE and MUSE+ still outperform all baselines.

---

Q3. Bootstrapping 1000 vs. Different Weight Initialization.

A3. We followed the evaluation protocol established in M3Care, which entails evaluating a trained model 1000 times by bootstrapping (i.e., sampling with replacement) from the test set.

---

Q4. Influence of Label Imbalance in Metrics.

A4. We understand the reviewer's concern regarding label imbalance. It is important to note the difference between accuracy and AUC-ROC scores. In datasets with only 1% positive labels, a classifier predicting all zeros would achieve a 99% accuracy score but only a 0.5 AUC-ROC score. Notably, AUC-ROC and PR-AUC scores are commonly used in mortality and readmission tasks on the MIMIC and eICU datasets [8, 9, 10].

We sincerely appreciate the reviewer's valuable feedback and have made necessary updates to our manuscript. We would like to take this opportunity to address the questions and clarify some points regarding our manuscript.

---

Q1. The use of the supervised contrastive loss and the classification loss seems redundant.

A1. We understand the reviewer's concern about the perceived redundancy of the two supervised losses. We would like to clarify that the supervised contrastive loss and the classification loss are calculated on two different tensors. Specifically, we add a projection layer to the patient embedding before calculating the contrastive loss, while the classification loss is based on the original patient embedding. Our intuition behind this design is that the contrastive losses encourage the model to learn both modality-agnostic and label-decisive features. In contrast, the classification loss leverages model-specific and label-decisive features to enhance prediction performance. To empirically support this, we provide the following additional ablation study:

Table: Additional ablation study on the supervised contrastive loss in terms of the AUC-ROC score.

This ablation study demonstrates the contribution of the supervised contrastive loss to the overall model performance.

---

Q2. Did authors find a rate of missingness that ultimately resulted in the model ignoring certain features?

A2. Yes. In fact, we observed that in the MIMIC-IV dataset, naive multimodal models (such as direct concatenation) frequently relied heavily on the clinical text while largely disregarding other modalities. We have referred to this issue as "Modality Collapse" in the introduction section of our manuscript. This observation was a critical motivator for developing the mutual-consistent graph contrastive learning framework. This framework is designed to facilitate the learning of features that are both modality-agnostic and decisive for accurate predictions.

---

Q3. Could you also provide the average Euclidean distance between representations since GRAPE minimizes MSE loss for continuous values?

A3. In our experiment analyzing the learned representations (Section 4.5), we calculate the average cosine similarity across all patients. We appreciate the suggestion to include the average Euclidean distance between representations. Here is the table with the requested information:

Table: Euclidean distance of the representations from the same patients with different modalities.

---

Thank you for taking the time during the rebuttal phase. We hope that these updates could address your questions.

Q5. Missingness is often informative. What if the labels are correlated with modality missingness?

A5. In this paper, we follow existing works (M3Care and GRAPE) and mainly focus on the missing completely at random setting. The modalities we consider are typically collected routinely for ICU patients, and ideally, they should be available for each patient. Missingness in our context is primarily attributed to administrative issues or time gaps.

To address the reviewer's concern, we simulated a scenario where label missingness is correlated with modality missingness and the true label in the eICU mortality prediction task. In this setting, we found that our proposed method can adapt to the non-random missingness scenario. The table below presents the AUC-ROC scores under this condition. Results show that our proposed method can also adapt to this scenario as the correlation could be learned in a data-driven manner.

Table: Results on the eICU mortality prediction task under the missing not at random setting. We simulated this setting by assigning a higher label-missing rate to deceased patients with missing vital signals while keeping the overall label-missing rate still at 50%.

---

We greatly appreciate your time and consideration during the rebuttal phase. We hope these updates could effectively address your questions and further enhance the quality and comprehensiveness of our manuscript.

---

[1] MULTIBENCH: Multiscale Benchmarks for Multimodal Representation Learning. NeurIPS 2021.

[2] MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images. MLHC 2022.

[3] Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities. ACL 2021.

[4] Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data. AISTATS 2023.

We sincerely appreciate the valuable feedback provided by the reviewer. We have carefully considered the reviewer's comments and have made the necessary revisions to our manuscript. Below, we would like to address the reviewer's questions and provide additional information:

---

Q1. What should a modality encompass for multimodal models to work well?

A1. Generally speaking, we believe that this is still an open research question. I.e., when and why does multimodal outperform unimodal jointly? However, our paper primarily focuses on addressing the challenge of missing modalities and labels rather than delving into the theoretical aspects of modality selection. We designed our experiments based on real-world clinical practices, where various modalities play a role in clinical decision-making. We also empirically verified that multimodal models outperform unimodal methods in our setting during the pilot study. Understanding when and why multimodal models will work is out of the scope of this paper. Instead, we focus on the setting where multimodal models can outperform unimodal models, but the missing modalities and labels limit their improvements. We believe this question should not undermine the practical significance of the problem or the novelty of the proposed method.

---

Q2. Additional experiments on datasets beyond healthcare.

A2. In response to the reviewer's suggestion, we have conducted additional experiments on datasets from the MULTIBENCH [1] benchmark encompassing various areas such as Robotics, Finance, HCI, and Multimedia. Due to the time constraint, we selected one dataset from each area and evaluated our proposed MUSE and the two best-performing baselines (GRAPE and M3Care). To simulate the missing modalities and labels scenario, we randomly masked out labels with a probability of 0.5 and modalities with a probability of 0.1.

Table: Additional experiment on the MULTIBENCH benchmark. MUSE achieves the best performance on 3/4 tasks, and MUSE+ further outperforms all baselines by incorporating samples with missing labels into the training process.

In summary, our results indicate that MUSE performs exceptionally well on a variety of tasks and domains, demonstrating its generalizability.

---

Q3. How does the method handle entirely new patients during testing?

A3. In both the training and inference phases, we create patient graphs independently within each batch. We will clarify this point in the revised manuscript.

---

Q4. Additional experiments on more baselines.

A4. As suggested by the reviewers, we have extended our comparison by including three additional baselines: MedFuse [2], MMIN [3], and MMCL [4]. Notably, MMCL also incorporates samples with missing labels during training, making it a fair comparison with our enhanced method, MUSE+. The table below summarizes the performance comparison, with MUSE and MUSE+ consistently outperforming all baselines.

Table: Comparison between MUSE and three additional baselines in terms of AUC-ROC scores. MUSE and MUSE+ still outperform all baselines.