Dynamic Modeling of Patients, Modalities and Tasks via Multi-modal Multi-task Mixture of Experts

[1] Liang P P, Cheng Y, Fan X, et al. Quantifying & modeling multimodal interactions: An information decomposition framework[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2] Soenksen L R, Ma Y, Zeng C, et al. Integrated multimodal artificial intelligence framework for healthcare applications[J]. NPJ digital medicine, 2022, 5(1): 149.

[3] Yun, S., Choi, I., Peng, J., Wu, Y., Bao, J., Zhang, Q., ... & Chen, T. (2024). Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts. arXiv preprint arXiv:2410.08245.

[4]Puigcerver, J., Riquelme, C., Mustafa, B., & Houlsby, N. (2023). From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951.

[5] Nakajima, E., Tsunoda, H., Ookura, M., Ban, K., Kawaguchi, Y., Inagaki, M., ... & Ishikawa, T. (2021). Digital breast tomosynthesis complements two-dimensional synthetic mammography for secondary examination of breast cancer. Journal of the Belgian Society of Radiology, 105(1).

[6] Mumin, N. A., Rahmat, K., Fadzli, F., Ramli, M. T., Westerhout, C. J., Ramli, N., ... & Ng, K. H. (2019). Diagnostic efficacy of synthesized 2D digital breast tomosynthesis in multi-ethnic Malaysian population. Scientific Reports, 9(1), 1459.

[7] Brown, A. L., Vijapura, C., Patel, M., De La Cruz, A., & Wahab, R. (2023). Breast cancer in dense breasts: detection challenges and supplemental screening opportunities. RadioGraphics, 43(10), e230024.

[8] Astley, S. M., Harkness, E. F., Sergeant, J. C., Warwick, J., Stavrinos, P., Warren, R., ... & Evans, D. G. (2018). A comparison of five methods of measuring mammographic density: a case-control study. Breast cancer research, 20, 1-13.

[9] Heine, J. J., Fowler, E. E., & Flowers, C. I. (2011). Full field digital mammography and breast density: comparison of calibrated and noncalibrated measurements. Academic radiology, 18(11), 1430-1436.

We thank the reviewer for their valuable feedback. We updated our manuscript with some changes in writing and all additional results in Appendix Section F. Changes are marked in blue.

W1: “Computational complexity"

R1: We report the computational cost of our model (inference time, training time, parameters) in Table 16. Note this analysis is conducted on a training batch size of 32 on the RSNA dataset. In the implementation, we used a single A100 80G GPU with 8 cores CPU to train the model. By comparing with other methods, our approach remains a similar time efficiency without adding latency, while using a similar scale of model parameters as other MoE methods. This is because the backbone of MoE introduces more model parameters than other structures [4].

MoE architectures in our method provide a promising way to scale the model size without paying too much computational cost. Our backbone, the recently introduced soft-MoE, runs at a faster speed than regular ViTs with 10x trainable parameters[4]. Similarly, M4oE can also scale to larger model sizes and datasets without introducing tremendous computational complexity.

Table 16 Complexity

W2: “Missing modality”

R2: We agree that the missing modalities or missing labels could be interesting problems to solve. Although it is out of the scope for this paper, where we mainly focus on showing the proof-of-concept for dynamic modality-task modeling using the proposed method, our approach can flexibly deal with with the missing labels, since the model does not require the label supervision from all the tasks in each training iteration. Additionally, our method can also be extended with missing modalities by easily combining with missing modality techniques like learnable embedding banks [3]. These could both be interesting future works following this paper.

We have added this discussion in the Discussion section 4.3.1 Of the revised manuscript.

Q1: “Computational resource requirements”

R3: We have reported the computational requirements in the revision. Please refer to our response R1 for details.

Q2: “Figure 4 clarification”

R4: The synergies of baseline and our method in Figure 4 implies not only the difference in absolute values, but also different correlated patterns of high-synergy modality pairs, which indicates different modality-task dependency.

Baseline: While Task 1 (density) and Task 2 (risk) synergy for baseline do capture different absolute values, the highest synergy pair still only peaks for M1 and M2 (within two views of FFDM) for both tasks.

Ours: For density, our method peaks at M1-M2. For risk, we peak at not only M1-M2 synergy (within FFDM) but also M1-M4 synergy (between FFDM and 2DS). We also capture the M2-M3 and M3-M4 synergy much better than the baseline. This indicates a stronger modality-task dependency.

Also, this result makes more sense from a clinical perspective. The FFDM (M1, M2) matters more for density predictions (Task 1) [7,8,9], while all four modalities indicate meaningful shared information for risk predictions (Task 2) [5, 6]. We revised Sec. 4.3 to clarify this.

Q3: “Provide more PID theory experiments”

R5: Even though we do not know the ground truth distribution in this dataset, a combination of both FFDM and 2DS has been proven to be highly effective in different populations [10,11].

As suggested by the reviewer, we have also computed the modality-unique information following [1], as shown in the table below. The uniqueness measures how much unique information models derive from each modality given a certain dataset. We expect our model to alleviate the loss of unique information compared to baseline multimodal soft MoE methods.

Compared to baseline MoEs, our M4oE achieves a higher uniqueness value for each modality, indicating that our approach effectively captures more modality-specific information. We add this metric in the revision in Table 15.

Q4: “Figure typo”

R6: Thanks for catching this typo! We have corrected this in the manuscript.

Table 15 Uniqueness Comparison

Dear Reviewer 7L3d,

Thank you very much for your valuable feedback. We have provided comprehensive point-by-point responses to address each of your concerns and questions. As we approach the end of the discussion period, we would greatly appreciate your feedback on whether our responses have adequately addressed your points. We remain available to provide any additional clarification you may need.

Best regards,

Submission 11923 Authors

Dear Reviewer 7L3d,

Thanks very much for your time and valuable comments. We understand you're very busy. As the window for responding and paper revision is closing, would you mind checking our response and confirming whether you have any further questions? We are happy to provide answers and revisions to your additional questions. Many thanks!

Best Regards,

Authors of Submission 11923

Table 10 RSNA results

Table 14 Subgroup Performance on Embeds Risk Prediction

Table 9 Embeds Results

Table 12 GAMMA Results

Table 13 MIMIC Results

We thank the reviewer for their valuable feedback. We updated our manuscript with some changes in writing and all additional results in Appendix Section F. Changes are marked in blue.

W1: “About sample-dynamic modality fusion”

R1: Although we agree in practice there could be different dynamic cases for each patient sample including missing and noisy data, this work mainly focuses on discussing the dynamic modality-task dependency for different patient samples when modeling the multimodal fusion for multi-task prediction, instead of missing or noisy data. This is also why we name it as “sample-dynamic modality fusion”, because exactly as the reviewer already mentioned in the question: “the specific and shared information always varies per sample”.

A similar term is also adopted in an analogous multimodal application in the natural domain for the dynamic fusion of infrared and RGB images [5,6]. For instance, in RGB-IR fusion, the relative importance of each modality naturally varies with environmental conditions - RGB provides better information in daylight while IR excels in low-light conditions. In mammogram screening, the optimal modality depends on patient characteristics: FFDM generally performs better for older patients while 2D synthesized images are more effective for younger patients [7,8]. So we think such a term is generally acceptable by the community in these applications.

To make this definition more clear as well as the scope of the paper, we add these explanations to further clarify it in Sec. 1 Introduction, as highlighted in blue. We also added a limitation section on missing modality in Section 4.3.1.

W2: “Performance metrics”

R2: M4oE outperforms baselines with different performance metrics. We reported Accuracy in classification tasks and DICE score in segmentation tasks. We have now also included AUROC and AUPRC. We have revised this sentence in the updated manuscript to be more clear about this as: “ Results demonstrate superiority over state-of-the-art methods under different metrics of classification and segmentation tasks including Accuracy, AUROC, AUPRC, and DICE.” Please check updated results in Appendix F, table 9-13.

W3: “Claim on multimodality”

R3: In the mammography domain, we are using not only different views of images, but actually different imaging modalities acquired using totally different techniques. Similarly, in the experiments on ophthalmology, the model adopts color fundus photos and optical coherence tomography, which are also two different imaging modalities acquired using different imaging systems. Unlike natural RGB images, medical image modalities can be much more complicated due to the essential difference in the imaging physics so as to capture distinct information in each modality. Therefore, we respectfully argue that the applications with these different imaging modalities should also essentially be multimodal problems. Previous works on the same applications also claim multimodal problems[11,12,13], which are already generally recognized by the community.

Besides, to avoid confusion with other data modalities, we have also pointed out that this work was for “medical imaging”, when we introduced the problem in Line 47 and our method in Line 99.

We agree that it’d be interesting to explore non-imaging modalities using our framework as well. Therefore, following your and Reviewer P7K2’s valuable advice, we performed additional multi-modal multi-task experiments on the MIMIC dataset following the experiment setting in [2][8]. Specifically, we used X-ray images, clinical/radiology notes, and electronic health records (EHR) modalities as inputs, to predict two tasks including in-hospital mortality prediction, and binary binned length of stay. The results are shown in Table 12.

Through comparison with baseline methods, our method can achieve comparable results on mortality prediction and outperforms on length of stay prediction.

W4: “Experimental setup”

R4: Ours: We determined optimal $\alpha$ weight for conditional mutual information loss and the number of experts $n$ based on the validation set conducted on the EMBED dataset. Then we keep using the same hyperparameters for our methods on all the benchmarks.

For baselines, we followed their experimental sections and ablation analysis to set the best-performing hyperparameters. We also keep the learning rate =1e-4 and batch size =32 fixed for all the baseline methods, including our approach, for a fair comparison. The hyperparameters used for baseline methods are:

MModN[7] : We set dropout rate = 0.1 and state representation size = 20.

EVIF[3]: We set the weight $\gamma_1=1,\gamma_2 =0.1, \gamma_3=0.01$ for the task loss, mutual information minimization loss, and mutual information maximization loss respectively.

AIDE[6]: Instead of the original 1e-3 learning rate, we changed it to 1e-4 to keep comparisons fair.

Fuller[5]: We set the weight factor $\alpha$ between modalities in gradient calibration to be 0.1. The gradient momentum hyperparameter m is set to 0.2.

AdaMV-MoE[1]: We set the auxiliary loss weights to be 5e−3 and delta_n to be 2000.

We have added content about hyperparameter tuning and settings in Appendix Sec. C.3.5 to further clarify this clearly. See changes in revision highlighted in blue.

We hope all these additional implementation details can further help to enhance the reproducibility of our work. Besides, we will also share our code publicly upon the paper's acceptance.

We have added the new metrics for our main experiments and reported AUROC and AUPRC in Appendix Tables 9-13. Along with the Accuracy, our proposed method achieves superior performance in different metrics.

W5: “Literature review misses intermediate fusion”

R5: Thanks for the constructive feedback! We agree that the papers about intermediate fusion can be relevant to our work in the perspective of multimodal fusion and feature disentanglement. But our work provides a unique insight for dynamic modality-task modeling parametrized by MoE structure and dynamic sample fusion. We also provide sample-level and population-level interpretability. We have added these discussions with relevant works brought up by Reviewer xATa in our related works section (Sec. 2) as highlighted in the updated revision.

Q1: “Expert-task assignment”

R6: Figure 1 provides a high-level schematic diagram showing only the abstraction of our key motivation: linking modality and tasks through experts. In our method, as shown in Fig. 2 and Sec.3, we have introduced that each expert contributes to every task but with different weight assignments determined by the combined matrix. Thus, the weight assignments are also learned automatically, enabled by the backbone of soft MoE [3].

Q2:“Sample adaptivity”

R7: Our work addresses sample adaptivity in dynamic multimodal medical imaging fusion, similar to well-established dynamic fusion scenarios in computer vision[6]. For instance, in RGB-Infared fusion, the relative importance of each modality naturally varies with environmental conditions - RGB provides better information in daylight while Infrared excels in low-light conditions. Analogously, in mammogram screening, the optimal modality depends on patient characteristics: FFDM generally performs better for older patients [7], while 2D synthesized images are more effective for younger patients [8].

Effective sample adaptivity requires the model to selectively utilize the most relevant features from each modality based on the input. However, naive fusion approaches often result in one modality dominating the joint representation, causing the loss of valuable complementary information from other modalities. Our proposed method addresses this challenge by preserving distinct features from each modality while allowing dynamic, sample-specific fusion. Specifically, the soft MoE backbones for both MSoE and MToE are designed to create input-conditioned data pathways dependent on each input respectively[5]. The MSoE modules preserve modality-specific information, which prevents one modality from being underestimated.

To qualitatively demonstrate the sample adaptivity, we discussed in original paper’s Fig 5 and Sec. 4.3 to show different modality contributions across samples.

We also provide the detailed performance of different subgroups, as shown below and in Table 14.

Q3: Interpretation of modality utilisation

R8: How to interpret global vs local modality contribution: Thanks for the reviewer’s questions. Let’s look at both Fig. 3(c) and Fig.5. Figure 3(c) demonstrates the global distribution of modality importance scores, while Fig. 5 shows both global (population-level) and local (sample-level) modality contributions. From Fig. 3(c), we see that M4oE does not heavily rely on one single modality globally, but captures dynamic dependency across all input modalities. This also aligns well with the average population-level results in Fig. 5, which shows that the modality contribution is close in average, with FFDM leads a bit. However, this does not mean the modality contribution is also purely equivalent for each sample. As shown in the sample-level results in Fig. 5, we can see the modality contribution is quite different for different individual patients. We also provide a clinical interpretation for these two patient cases in Sec. 4.3. This indicates that our model allows for dynamic diverging modality contribution for each patient sample in local modality contribution, while avoiding modality competition statistically in the sense of global modality contribution.

More on modality domination : Different modalities should always carry out some useful information, but due to the limitations of the multimodal fusion modeling, the model may become over-reliant on one modality, making a statistical bias, as shown in the Fig. 3(b) for the case of plain Soft MoE. Our method can address this issue to leverage the useful information from different modalities more effectively. However, in the extreme case as the reviewer mentioned where there exists information bias among different modalities due to data quality, we conjecture the distribution across different modalities may be more different statistically in mean and variance. But this is absolutely not the case in the dataset used here, where we can still see the difference of mean values of different modalities but not diverge a lot (it is not forced to be the same in the model design). This actually aligns with the clinical interpretation in this dataset and application where a combination of both FFDM and 2DS is proven to be highly effective on different populations [10,11].

Figure 1 provides a high-level schematic diagram showing only the abstraction of our key motivation: linking modality and tasks through experts. Each expert contributes to every task but with different weight assignments determined by the combined matrix. Also, in the right half of Figure 1, all experts are linked to all tasks.

[1] Cao B, Sun Y, Zhu P, et al. Multi-modal gated mixture of local-to-global experts for dynamic image fusion[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 23555-23564.

[2] Yun, S., Choi, I., Peng, J., Wu, Y., Bao, J., Zhang, Q., ... & Chen, T. (2024). Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts. arXiv preprint arXiv:2410.08245.

[3] Puigcerver, J., Riquelme, C., Mustafa, B., & Houlsby, N. (2023). From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951.

[4] Han, X., Nguyen, H., Harris, C., Ho, N., & Saria, S. (2024). Fusemoe: Mixture-of-experts transformers for fleximodal fusion. arXiv preprint arXiv:2402.03226.

[5] Cao, B., Sun, Y., Zhu, P., & Hu, Q. (2023). Multi-modal gated mixture of local-to-global experts for dynamic image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 23555-23564).

[6] Zhang, Q., Wei, Y., Han, Z., Fu, H., Peng, X., Deng, C., ... & Zhang, C. (2024). Multimodal fusion on low-quality data: A comprehensive survey. arXiv preprint arXiv:2404.18947.

[7] Destounis, S. V., Santacroce, A., & Arieno, A. (2020). Update on breast density, risk estimation, and supplemental screening. American Journal of Roentgenology, 214(2), 296-305.

[8] McDonald, E. S., McCarthy, A. M., Akhtar, A. L., Synnestvedt, M. B., Schnall, M., & Conant, E. F. (2015). Baseline screening mammography: performance of full-field digital mammography versus digital breast tomosynthesis. American Journal of Roentgenology, 205(5), 1143-1148.

[9] Mumin, N. A., Rahmat, K., Fadzli, F., Ramli, M. T., Westerhout, C. J., Ramli, N., ... & Ng, K. H. (2019). Diagnostic efficacy of synthesized 2D digital breast tomosynthesis in multi-ethnic Malaysian population. Scientific Reports, 9(1), 1459.

[10] Nakajima, E., Tsunoda, H., Ookura, M., Ban, K., Kawaguchi, Y., Inagaki, M., ... & Ishikawa, T. (2021). Digital breast tomosynthesis complements two-dimensional synthetic mammography for secondary examination of breast cancer. Journal of the Belgian Society of Radiology, 105(1).

[11] Khara, Galvin, et al. "Generalisable deep learning method for mammographic density prediction across imaging techniques and self-reported race." Communications Medicine 4.1 (2024): 21.

[12] Cai, Zhiyuan, et al. "Uni4eye++: A general masked image modeling multi-modal pre-training framework for ophthalmic image classification and segmentation." IEEE Transactions on Medical Imaging (2024).

[13] He, Xingxin, et al. "Multi-modal retinal image classification with modality-specific attention network." IEEE transactions on medical imaging 40.6 (2021): 1591-1602.

Dear Reviewer dm8E,

Thank you very much for your valuable feedback. We have provided comprehensive point-by-point responses to address each of your concerns and questions. As we approach the end of the discussion period, we would greatly appreciate your feedback on whether our responses have adequately addressed your points. We remain available to provide any additional clarification you may need.

Best regards,

Submission 11923 Authors

Thank you for quickly incorporating the feedback to my questions and concerns. Some of my concerns have been addressed, so I am happy to lift my score to a 5. However, I still believe that there is a fundamental problem of the paper claiming more than it validates with its experiments, as implied in my initial review. This prevents me from further increasing my score. Additional concerns that remain are:

• W1: the definition of sample-dynamic fusion remains vague in the abstract and the terminology is inconsistent with “sample-adaptive” (L86). It remains unclear to me how this varies from any other multimodal fusion problem.
• W3: Please introduce this in the abstract “[…] many significant medical imaging problems”. The three references you provided [11, 12, 13] even narrow this scope in the title which I believe would be suitable here, too.
• Q3: While in a perfect world “different modalities should always carry some useful information” - in practice, this just isn’t the case. That’s why some experiments about these edge cases would strengthen the argument. This concern has not been adequately addressed in the Author response.

We thank the reviewer for their valuable feedback. We updated our manuscript with some changes in writing and all additional results in Appendix Section F. Changes are marked in blue.

W1: “Discussion of related works”

R1: Thanks for the constructive feedback and suggestions for these papers! We agree that these papers can be relevant to our work in the perspective of multimodal fusion and feature disentangling. But our work provides a unique insight for dynamic modality-task modeling parametrized by MoE structure and dynamic sample fusion. We also provide sample-level and population-level interpretability. We have added these discussions with these mentioned works in our related works section (Sec. 2) as highlighted in the updated revision. We additionally include a recent NIPS’24 work that extends multimodal fusion for generative tasks.

W2: “Ablation study for number of experts”

R2: Yes, in original manuscript, we did conduct an ablation study about the number of experts as shown in Figure 7 as well as Section 4.4. From this ablation study, it shows generally more experts lead to better performance, in almost a linear relationship on two tasks’ performance.

In order to investigate the sensitivity of our method on the number of experts, we also conduct an additional experiment: we removed all new task-related and modality-specific components proposed in our M4oE model to construct a basic version of multimodal MoE for comparison. The results are shown in Table 18. Compared to this basic MoE version, our method is less sensitive in the number of experts.

Table 18:

W3: “Computational cost”

R3: We report the computational cost of our model (inference time, training time, parameters) in Table 16. Note this analysis is conducted on a training batch size of 32 on the RSNA dataset. In the implementation, we used a single A100 80G GPU with 8 cores CPU to train the model. By comparing with other methods, our approach remains a similar time efficiency without adding latency, while using a similar scale of model parameters as other MoE methods. This is because the backbone of MoE introduces more model parameters than other structures [4].

MoE architectures in our method provide a promising way to scale the model size without paying too much computational cost. Our backbone, the recently introduced soft-MoE, runs at a faster speed than regular ViTs with 10x trainable parameters[4]. Similarly, M4oE can also scale to larger model sizes and datasets without introducing tremendous computational complexity.

Table 16

Q1: “Ablation studies on more datasets”

R4: Besides EMBED, we have conducted ablation studies on more datasets including GAMMA, RSNA, and VinDR datasets. The results are shown in Table 17. We observe that: In datasets where the target tasks are more different(GAMMA), MToE and conditional MI loss are slightly more important. In datasets with higher modality discrepancy (MIMIC), MSoE plays a more important role.

Q2: “Theoretical explanations on mitigating gradient conflict”

R5: Gradient conflicts in multi-task learning with MoE often result from assigning counteracting tasks to the same expert [3]. As theoretically analyzed in [1], incorporating sparse task-specific modules helps learn the disentangled features of different tasks. This task specialization can help mitigate the counteracting gradient on experts in the MoE blocks. Therefore, our task MoE module uses separate task slots and task-specific embeddings to encourage separate feature learning.

Many existing MoE works [4,5] apply the well-known load-balancing loss. One outcome of this loss is that it pushes expert usage across tasks and batches to be similar. However, this may not be helpful for multi-task learning since experts may be forced to share among tasks with conflicting gradients. Our loss encourages stronger dependence between experts and tasks. The analysis of task-expert sparsity in Figure 6 (b) and (c) shows that our method achieves sparser expert-task relationships to facilitate task specialization, so that it can help to alleviate the gradient conflict problem.

Q3: Clinical interpretation of Fig. 5

R6:

In the sample-level:

Patient 1: Dense glandular tissue spread throughout the breast. High breast density can obscure lesions in traditional mammography, but 2DS images provide improved lesion visibility in dense breasts [6]. According to [7], women younger than 50 years, who usually have dense breasts, may benefit more from 2DS than from FFDM alone [7]. Therefore, it aligns with our observation that M4oE utilizes 2DS much more than FFDM for risk predictions. This indicates a positive sign that the model does not rely heavily on FFDM and provides a potential way for early risk detection[8].

Patient 2: Primarily fatty tissue with less dense glandular areas makes this patient easier to evaluate for abnormalities. Usually, for this patient subgroup, FFDM alone often suffices for accurate screening [9]. This also matches our observation that the model utilizes FFDM more than 2DS for risk prediction task.

In the population-level:

A combination of both FFDM and 2DS is proven to be highly effective on different populations[10,11]. Our model utilizes all of the modalities well for the predictions, while FFDM has indicated slightly higher importance than 2DS on average. We suspect this may be attributed to the relatively higher age distribution (mean=58.5 y) in the dataset.

[1] Lachapelle S, Deleu T, Mahajan D, et al. Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning[C]//International Conference on Machine Learning. PMLR, 2023: 18171-18206.

[2] Yu T, Kumar S, Gupta A, et al. Gradient surgery for multi-task learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 5824-5836.

[3] Chen Z, Shen Y, Ding M, et al. Mod-squad: Designing mixtures of experts as modular multi-task learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 11828-11837.

[4] Riquelme C, Puigcerver J, Mustafa B, et al. Scaling vision with sparse mixture of experts[J]. Advances in Neural Information Processing Systems, 2021, 34: 8583-8595.

[5] Fan Z, Sarkar R, Jiang Z, et al. M³vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design[J]. Advances in Neural Information Processing Systems, 2022, 35: 28441-28457.

[6] Aujero, M. P., Gavenonis, S. C., Benjamin, R., Zhang, Z., & Holt, J. S. (2017). Clinical performance of synthesized two-dimensional mammography combined with tomosynthesis in a large screening population. Radiology, 283(1), 70-76.

[7] McDonald, E. S., McCarthy, A. M., Akhtar, A. L., Synnestvedt, M. B., Schnall, M., & Conant, E. F. (2015). Baseline screening mammography: performance of full-field digital mammography versus digital breast tomosynthesis. American Journal of Roentgenology, 205(5), 1143-1148.

[8] Harbeck, N., Penault-Llorca, F., Cortes, J., Gnant, M., Houssami, N., Poortmans, P., ... & Cardoso, F. (2019). Breast cancer. Nature reviews Disease primers, 5(1), 66.

[9] Destounis, S. V., Santacroce, A., & Arieno, A. (2020). Update on breast density, risk estimation, and supplemental screening. American Journal of Roentgenology, 214(2), 296-305.

[10] Mumin, N. A., Rahmat, K., Fadzli, F., Ramli, M. T., Westerhout, C. J., Ramli, N., ... & Ng, K. H. (2019). Diagnostic efficacy of synthesized 2D digital breast tomosynthesis in multi-ethnic Malaysian population. Scientific Reports, 9(1), 1459.

[11] Nakajima, E., Tsunoda, H., Ookura, M., Ban, K., Kawaguchi, Y., Inagaki, M., ... & Ishikawa, T. (2021). Digital breast tomosynthesis complements two-dimensional synthetic mammography for secondary examination of breast cancer. Journal of the Belgian Society of Radiology, 105(1).

Dear Reviewer xATa,

Thank you very much for your valuable feedback. We have provided comprehensive point-by-point responses to address each of your concerns and questions. As we approach the end of the discussion period, we would greatly appreciate your feedback on whether our responses have adequately addressed your points. We remain available to provide any additional clarification you may need.

Best regards,

Submission 11923 Authors

We thank the reviewer for their valuable feedback. We updated our manuscript with some changes in writing and all additional results in Appendix Section F. Changes are marked in blue.

W1: “Writing improvement”

R1: Thanks for pointing this out! We have revised the manuscript carefully to address these issues in the updated revision.

W2: “Figure 2”

R2: Figure 2 primarily corresponds to Section 3.2, where we provide a comprehensive overview of the framework and the functionality of each component. A detailed description of the data flow and the explanation of different modules is presented in original paper Section 3.2. Here is a high-level explanation to help with interpretation:

Our framework comprises four components: (1) modality-specific feature embedders, (2) modality-specific Soft MoEs (MSoE), (3) Modality-Task Soft MoEs (MToE), and (4) a final fusion block. Each module is described in more details below. The framework first embeds different input modalities with feature embedders, and then projects m input modalities to embeddings $X_1,..., X_m \in \mathbb{R}^{l \times d}$. This is shown in Fig 2-a) part 1.

1. To mitigate the modality competition phenomenon, for each modality $M_i$ we use MSoE blocks to learn and retain the modality-specific features $Z_i \in \mathbb{R}^{l \times d}$ from $X_i$. Detailed MSoE is in Fig 2-b), and data flow is in Fig 2-a) part 2.

2. To model shared modality information and modality-task dependence, we feed $X_1, \dots, X_m$ into MToE blocks and obtain the task-dependent fused modality features $G_1, \dots, G_p$. MToE uses learnable task embeddings and task slots to facilitate the dynamic connection between modalities and tasks. This also allows for probability modeling with the $\phi$ matrix. Detailed MToE is in Fig 2-c), and data flow is in Fig 2-a) part 2.

3. Finally, MToE outputs from $G_1,...,G_p$, and MSoE outputs $Z_1,...,Z_m$ collectively form a pool $H_1,...,H_p$. For each task, $T_k$, $H_k$ is passed to the task heads after a basic soft MoE block for final predictions. The fusion block structure is Fig 2-d) and data flow in Fig 2-a) part 3. To make it more clear, we modify the caption of Fig. 2 and Sec. 3.2 to add the textual explanation of the functions (highlighted in blue).

W3: “More baseline comparison.”

R3: In our paper, we have conducted an extension baseline comparison, including (1) two baselines targeting each medical applications domain in mammography, ophthalmology, radiology+EHR. They are also adapted to integrate with MToE to construct a stronger model for a fair comparison; (2) we also compare with other 7 multimodal multitask baselines in the natural domain. The comparison in Table 9-13 of the original paper over 5 benchmarks has shown the solid improvement of our proposed approach.

In order to compare against more baselines as suggested, we conduct more experiments and compare with more baselines which are all very recently released works (ICML’24 Multimodal Soft MoE[4], ICCV’23 AdaMV-MoE [1], and NIPS’24 FuseMoE [2]). The experimental results are shown in Table 9-13, added in the revision. These additional results further show that our proposed method consistently performs well.

W4: “Result significance”

R4: As suggested, we report the mean and standard deviation from five-fold cross-validation in the reported results in Table 9-13 on multiple metrics, including Accuracy, AUROC, and AUPRC.

W5: “Experiments on non-imaging modalities.”

R5: Since the proposed method mainly focuses on modeling the dynamic multimodal fusion for multi-task learning, we think our method is able to generalize to other modalities as well under this scheme. To demonstrate this, we performed additional multi-modal multi-task experiments on the MIMIC dataset following the experiment setting in [2][8]. Specifically, we used X-ray images, clinical/radiology notes, and electronic health records (EHR) modalities as inputs, to predict two tasks including in-hospital mortality prediction, and binary binned length of stay. The results are shown in Table 12. Through comparison with baseline methods, our method can achieve comparable results on mortality prediction and outperforms on length of stay prediction. Although this is not the main focus of our paper, we hope these additional experiments can convince that our proposed approach is potentially generalizable to non-imaging modalities as well.

W6: “Discuss the quantitative results.”

R6: We have already discussed the main conclusion from the experiment results in Section 4.2. Due to the page limit, we kept it succinct. As suggested by the reviewer, we have added more discussion based on the quantitative analysis in Sec. 4.2 in revision, as highlighted in blue.

Dear Reviewer P7K2,

Thank you very much for your valuable feedback. We have provided comprehensive point-by-point responses to address each of your concerns and questions. As we approach the end of the discussion period, we would greatly appreciate your feedback on whether our responses have adequately addressed your points. We remain available to provide any additional clarification you may need.

Best regards,

Submission 11923 Authors

Dear Reviewer P7K2,

Thanks very much for your time and valuable comments. We understand you're very busy. As the window for responding and paper revision is closing, would you mind checking our response and confirming whether you have any further questions? We are happy to provide answers and revisions to your additional questions. Many thanks!

Best Regards,

Authors of Submission 11923

Dear P7K2,

Thank you again for your valuable review! We’ve provided detailed responses and results to answer your questions. As we are near the end of the discussion period, we would greatly appreciate it if you could review our responses and share any additional feedback you may have. We remain available to address any further questions or concerns. Thanks a lot!

Best Regards,

Authors of Submission 11923

Thanks authors for the impressive work! This work is very interesting. I had a question for the future work on missing modalities. In page 11 Line 540, authors introduce an embedding bank technique by Flex-MoE [1]. How to adapt that technique to the MoSE, MoTe framework? It would also be valuable if authors can evaluate the performance of Flex-MoE on the tasks given in the paper as a future work. Thanks again!

[1]. Yun, Sukwon, et al. "Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts." The Thirty-eighth Annual Conference on Neural Information Processing Systems.

    if num_modalities > 1:
              missing_embeds = torch.nn.Parameter(torch.randn((2**num_modalities)-1, args.n_full_modalities, args.num_patches, args.hidden_dim, dtype=torch.float, device=device), requires_grad=True)
              params += [missing_embeds] #this is to help make the parameters trainable

    for i, (modality, samples) in enumerate(batch_samples.items()):
        mask = batch_observed[:, modality_dict[modality]]
        encoded_samples = torch.zeros((samples.shape[0], args.num_patches, args.hidden_dim)).to(device)
        if mask.sum() > 0:
            encoded_samples[mask] = encoder_dict[modality](samples[mask])
        if (~mask).sum() > 0:
            encoded_samples[~mask] = missing_embeds[batch_mcs[~mask], modality_dict[modality]]
        fusion_input.append(encoded_samples)