CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models

W1-2, Q1-2, C1-2: In our previous experiments, both pretraining and multitask learning are performed in Exp. 1. We added experiments below comparing unsupervised pretraining vs supervised multitask learning.

First, we found time series models benefited substantially from unsupervised pretraining. As illustrated in App. Tab. 26, we conducted unsupervised masked autoencoder (MAE) pretraining and compared it with pretraining on target dataset only:

We found pretraining on diverse data effectively improved the downstream outcome on the target dataset. Similarly, for EEG datasets, as shown in App. Table 24, pretraining on a diverse range of data improved the performance for 2/3 datasets and overall:

While combining pretrain and multitask learning achieves the best results, pretraining seems to play a larger role than multitask learning in EEG models.

On the other hand, we showed in Fig. 4 that multitask learning effectively improved the vision encoder’s performance. Surprisingly, vision models did not benefit from further pretraining, and only multitask learning helps:

Here, MAE = Masked Autoencoder, CL = Contrastive Learning and MTL = Multitask Learning. For MAE, we followed the same approach as in ConvNeXTv2. For contrastive learning, we followed the CLIP-style approach as outlined in InternVL. In the above vision experiment, neither MAE nor CL pretraining improved the model’s performance for downstream tasks. One hypothesis is that these models are already heavily pre-trained on massive unlabeled natural image corpora, so an additional masked image or contrastive-style phase on clinical data doesn’t substantially shift or enrich their learned representation. We welcome the community’s contribution to better leveraging the diverse labeled data available in CLIMB.

Q3, C3: We found that datasets with novel tasks, Understudied modalities or from Underrepresented regions, as defined in Lines 1219-1290, benefit the most from multitask learning. We believe this is due to the scarcity of data in these categories, which makes data from other modalities or tasks, albeit not closely related, beneficial. On the other hand, if a dataset has a large number of samples or fairly saturated performance on a specialized diagnostic task, then multitask learning may not add much beyond what single-task training already learns.

Q4: During the construction of CLIMB, we aim to include as many datasets from underrepresented regions as possible. This includes datasets from geographically underrepresented regions, as well as data from developing countries where the data has been historically scarce, as defined in Lines 1274-1290. We included a list of dataset collection locations in the App. Table 8. In summary, 8 out of 30 (26.7%) of the datasets with known source locations come from underrepresented regions, a percentage significantly higher than all datasets available from public sources.

Thanks for your feedback regarding Figures 3.a and 3.b. Both figures illustrate two different experiments we conducted on CLIMB: multitask learning and transfer learning. The figures display example data from CLIMB (such as x-rays, CT scans, etc.) as inputs to the model and display evaluation on the right. For Figure 3.a, we train a single model on multiple clinical tasks across different medical modalities and evaluate it on each individual dataset. The goal is to evaluate whether a multitask pre-train model can generalize across diverse tasks, including understudied ones. This setup helps assess if shared representations from CLIMB improve generalization within each dataset. In Figure 3.b, a multitask pre-trained model is applied to a new task with limited data to see whether the model adapts its learned representations to the new task, despite having few samples available. The goal is to determine if exposure to a broader range of tasks in CLIMB helps compensate for data scarcity in specific datasets.

To improve clarity, we will implement the following revisions: 1) provide a more detailed figure description 2) move labels for "understudied" and “few samples" to the side to avoid implying they apply to the entire column. We will also cite the recent works mentioned in the review.

Regarding the computational resources, as described in Appendix C.4.2, all experiments are performed on a server with 8xH200 GPUs for the best performance. At least 20TB of storage is needed if you would like to train a model on the entire dataset. With that said, our model is small enough to fit on one GPU with 24GB of VRAM, and the entire training on one GPU would take less than a week.

All data used in this work is public and can be accessed easily using our framework, as outlined in our response to Reviewer pTYB. In summary, a user only needs to complete one-time registrations of two accounts and complete one CITI training, which takes less than 4 hours. Our framework will then prompt the user to agree to agreements, and downloading each dataset will take less than 10 seconds of human labor. Our framework will also handle the data cleaning, processing and standardization of formats.

We appreciate reviewer SVNB's feedback. Besides our scale and focus on multimodal, our work distinguishes itself from related research in several ways:

• Our focus extends beyond confirming general pretraining benefits by specifically targeting underrepresented regions and modalities (ultrasound, CT, EEG). These more rare, OOD tasks have demonstrated superior performance gain as compared to tasks that occur often in the datasets. This challenges the traditional idea that pretraining mainly helps with popular tasks while struggling to improve perf. on OOD tasks (https://arxiv.org/abs/2211.08411, https://arxiv.org/abs/2212.10511).

• In addition, we show that traditional vision encoders still perform better than medical VLLMs by a large margin, whereas related works mentioned in the review mainly focus on improving VLLMs.

• We release a unified framework that streamlines downloading, processing, and model training across vision, time series, and graph domains, enabling researchers to rapidly replicate and iterate on methods against diverse clinical tasks. Our work is of a much larger scale than related works, as shown in our response to Reviewer pTYB.

Comment on Figure 4: We define underrepresented regions and understudied modalities in Lines 1219-1290 using a two-step approach that identified geographic and economic gaps in dataset coverage. While 8/30 (26.7%) datasets with known collection sites come from these regions, this representation still lags behind developed countries. We will clarify that our classification of understudied modalities reflects public dataset availability rather than clinical importance.

On dataset construction: We put extensive efforts into standardizing the taxonomy. Standardization is difficult as it requires balancing two competing objectives:

• Merging similar terms to facilitate learning and cross-modality transfer
• Minimizing information loss and avoiding inaccuracies when modifying labels

Our standardization efforts concentrated on ECG and chest X-ray, which offer fine-grained labels with varying terminologies. For ECG, we followed approaches from arXiv:2304.08486 and formulated a mapping here in our anonymous repo.

For Chest X-rays, we developed a new mapping with radiologist input. Following practices from 1, 2 and 3, we consolidated all Chest X-ray labels into the CheXpert 14 categories:

Labels from other modalities with the same name are then merged with these classes. We acknowledge standardization may affect label granularity, particularly for lung opacity and lung lesion classes. CLIMB provides both standard and raw labels, allowing researchers to prioritize either granularity or standardization. All experiments in our work used standard labels.

To evaluate standardization effects, we compared vision encoders trained on raw versus standardized labels:

Standardized labels yielded better performance: 3.6% improvement in overall AUC, 10.3% in sensitivity, and 4.0% in specificity. Notably, standardization benefited other modalities more than Chest X-ray itself, where most relabeling occurred. We believe standardized labels help the model connect concepts across modalities more effectively. We hope this work motivates further efforts in terminology standardization and fine-grained relabeling of public clinical datasets.

We thank Reviewer pTYB for the positive reviews and constructive feedback. We have addressed all typographical errors in our manuscript.

Regarding the definition of OOD datasets (Claim 1), we selected OOD datasets primarily based on task differences and provided a comprehensive list with justifications in the App. Table 9. Of the 10 OOD datasets, 7 (COVID-19, CoronaHack, ISIC-2020, BCSS, BUSI, LNDb, PTB-XL-Finegrained) have zero label overlap with other datasets in the same clinical domain, while the remainder feature distinct task granularity. To quantify this heterogeneity, we conducted a membership experiment using ConvNeXT-v2-base as the backbone to predict dataset membership, following hyperparameter settings in Appendix C.4.2. Results confirm our designated OOD datasets exhibit substantial distinctiveness:

Claim 2: In Table 5, "SoTA encoders" refers to the best pre-trained encoders tested across clinical-specific and general domain models. Specifically, as detailed in Sec. 4.1, we employed ConvNextV2, ECG JEPA, and ClinicalBERT. Our experimental protocol follows FuseMoE, except we implemented regression without binning for LOS to increase granularity, and utilized AUC, sensitivity and specificity for 48 IHM to maintain consistency throughout the paper. We will incorporate a comprehensive results table for experiment 3 in the appendix.

Claim 3: We will remove lines 470-473 from the impact statement. As typical of impact statements, this referred to potential future work. The dataset samples contain demographic information where available, which could facilitate future evaluation of robustness, fairness, and privacy in clinical AI.

Methods 1: Data accessibility was a key consideration in dataset selection, as outlined on page 23 (Dataset Selection Methodology). We specifically selected datasets that do not require lengthy approval processes. Researchers can access 37 of 44 datasets in CLIMB after completing these steps, requiring less than 4 hours total:

• Register for PhysioNet (15 mins) and Kaggle (10 mins) accounts

• Complete CITI certification training (3 hours)

• Acknowledge dataset agreements (10 seconds each)

Our framework then manages all downloads and processing automatically.

Methods 2: CLIMB integrates all metadata, labels and text reports with each sample. We did an analysis of the number of words, the number of QA pairs and the total size of the dataset. Comparing with other multimodal clinical QA datasets:

Our dataset exceeds others by at least an order of magnitude in all three metrics.

Methods 3: We have added accuracy and F-1 score to Table 5. We include a subset here due to space limit and will add the full table in Appendix of final manuscript:

Question 2: In this work, our focus is on vision, time series and graphs, where large pretraining efforts are particularly scarce. We focused less on textual modality since multiple textual LLMs trained on diverse medical data already exist, including ClinicalBERT and BioMistral.

Question 1,3: The EHR data encompasses all textual data available in MIMIC-IV within 48 hours of admission: vital signs, lab measurements, treatments, medications, and demographics. We excluded data without timestamps (e.g., diagnoses) and omitted discharge summaries and radiology reports. We performed no normalization or binning on LOS regression. The text is JSON-formatted and fed directly into the text model for embedding.