Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing

We thank the reviewer for their detailed and constructive comments. We are especially grateful for the recognition of our contributions, including well-supported claims and experiments that demonstrate a comprehensive understanding of the capabilities and limitations of the AMC method. We address the concerns and questions raised as follows.

[Cons1] Clinical Interpretability Evaluation

We should emphasize that interpretableability is not our main contribution, and our main contributions lie in three points:

• Top-Down Dynamic Fusion Mechanism: A novel top-down dynamic fusion mechanism to address the variation in data quality challenge
• Token Fusion for Heterogeneous Modalities: A novel token fusion approach that can work on heterogeneous modalities and is not limited to modalities with explicit mapping relationships
• Extensive Real-World Validation: Validate our method by extensive experiments on real-world datasets.

Interpretability and Clinical Validation

Although our method is not primarily designed to enhance interpretability, it does offer modality-level insights. Through our top-down fusion framework, we assess the importance of each modality. This helps in understanding each modality's contribution to the final prediction. Unlike traditional methods such as the Shapley value, which focus on feature-level interpretability, our approach provides insights at the modality level. Therefore, we do not directly compare our method with clinically validated medical AI methods. However, we can include specific methods in future comparisons if the reviewer suggest.

Case Study

To further address clinical validation concerns, we propose a case study using images from the TCGA dataset. The image is easier for readers to realize the modality data quality. This study illustrates the variation in modality importance, providing a visual representation of data quality. The results in Figure 2 of Anonymous show that images with lower modality importance scores generally exhibit poor quality, while images with higher scores indicate better quality.

We will revise our introduction for clear clarification about our contribution and add the case study to our appendix.

[Cons2] Empirical Validation for Efficiency.

To address the reviewer’s concern, we report the Mean time per iteration during training, and GFLOPs during testing across baselines and AMC. The result shows our method is computationally efficient.

[Cons3] “situates itself within the existing body of AI and machine learning literature” and has limited novelty.

We respectfully disagree. What we really focus on advocating is the novel and practical angle of medical learning with considering multimodal data quality variance for the first time. Motivated by this fresh perspective, our solution integrates the attention mechanism and contrastive learning into the proposed token fusion framework. This token fusion approach is distinctly developed beyond existing methods. Specifically, to address previous drawbacks in token fusion, our framework offers significant differences and improvements over prior models.

In addition, our novelty is consistently recognized by two reviewers. Our paper was rated as an “innovative idea“ by Reviewer joLr: “The proposed method is novel in how it integrates token-level rebalancing within a transformer-based model.”. Then, our novelty is also highly acknowledged by Reviewer JTwf as innovative ideas.

Finally, our method brings novelty from the following perspectives:

• Novel Motivation: we are motivated by the important observation “in medical tasks, data quality varies across different modalities and patients.”
• Technical Novelty: Novel token fusion framework that does not require explicit mapping between modalities, and a novel token replacement policy when the number of modalities is larger than 2.
• Application Novelty: We propose a unified learning framework that can address different kinds of medical applications: patient survival time prediction, patient mortality prediction, and diagnosis classification prediction.

We thank the reviewer for their detailed and constructive comments. We are grateful for the recognition of our contributions, including the interesting approach of token replacement, solid experiments, and innovative ideas. We address the concerns and questions raised as follows.

[Cons1] Typos

The D in line 163 and line 170 should be N. Meanwhile the nn-th should be replaced with n-th. We will modify these typos in our revision.

[Cons2] Misalignment Issue for the Multimodal Images.

This is a good question. Before tokens are fed into our improved transformer blocks, we use Q-Former to encode each modality. In Q-Former, the query comprises modality-specific learnable embeddings, while the key and value derive from the input modality tokens. Q-Former extracts key information from input images, as discussed in [1], and integrates this into learnable query embeddings. Spatial information is implicitly encoded within these embeddings. Our contrastive loss not only retains the spatial information of modalities, such as images, but also enhances the Q-Former's ability to extract critical information from image tokens.

[1] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

[Cons3] Modality Missing Scenario

This paper primarily examines the contribution of different modalities to final predictions. Although handling missing modalities is not our main focus, it is easy to extend AMC to solve the missing modality problem. In such cases, we treat the importance score and token score of any missing modality as zero. This approach enables AMC to function effectively even when some data is unavailable. To demonstrate AMC's adaptability to the missing modality scenario, we conducted experiments using the ADNI dataset. This dataset is naturally suited for testing missing modality scenarios, providing a robust environment to validate our approach. The results in the following table indicate that AMC's performance decreases only slightly when dealing with missing modalities. This demonstrates that AMC can effectively handle the missing modality problem through simple extensions. We use the same model but add examples with missing modality problems. The similar performance indicates that AMC still keep prediction capability in these additional examples.

[Cons4] How does the importance of different modalities vary across different tasks?

As shown in Figure 1, the important modality varies even for the same task across different datasets. This variation is mainly due to differences in data collection times and locations, which can affect the quality of medical data. To address the reviewer's concern, we further evaluated the single modality task performance across MIMIC-IV and ADNI (Table 1 in Anonymous). This approach helps us assess how the importance of different modalities varies across different tasks.

[Cons5] Are there certain redundant modalities that are deemed necessary?

In this paper, we focus on the effects of different modalities. The necessity of certain redundant modalities is beyond the scope of this work. However, this is an important question that we would like to investigate in the future.

[Cons6] Intuitive Visualizations

Thanks for your reminder. We provide several case studies about how each modality's tokens are replaced by other modalities’ tokens in the Figure 3 of Anonymous.

[Cons7] Comparing and Citing More Works for Novelty Certification.

After comparing our AMC with the mentioned MLLM based methods, we summarize the following similarities and differences. Similarity:

• All methods use the attention score to evaluate token importance. Differences:
• Our approach uses the attention score to calculate modality importance.
• We fuse multimodal information through our proposed token fusion framework, whereas existing MLLM methods use self-attention within the LLM backbone.
• We consider both token-level and modality-level feature importance, unlike previous methods that focus solely on token-level importance.

Compare with More Baselines: The input modality, medical tasks, learning objectives, and fusion framework of AMC differ significantly from those of existing methods, making it challenging to include them as baselines in our comparison.

Novelty Clarification: Please refer to the Cons3 in our response to the Reviewer CN7m for this clarification due to limited space.

We are very thankful for your time and consideration. If you have any further questions, please let us know, and we will be happy to address them fully.

We thank the reviewer for the thoughtful and constructive feedback. We address the key concerns and suggestions below.

[Cons1] Interpretability Analysis in the Medical Setting

For the interpretable aspect, we are not primarily aiming to propose a method that focuses on giving interpretableability. However, AMC can provide modality-level insights. To illustrate this, we present visualizations in Figures 4 and 5. These figures demonstrate our model's ability to identify how each modality differently contributes to the final prediction. To further address reviewer concerns, we conducted a case study on the image modality within the TCGA dataset. Please refer to the Cons1 in our response to the Reviewer CN7m for the case study details due to limited space.

[Cons2] Recent SoTA Fusion Baselines.

For our comparisons, we include classical fusion methods such as MAGGate, MulT, and TF. Additionally, we evaluate against recently proposed baselines: FlexMoE and FuseMoE (both from NeurIPS 2024), and MUSE (from ICLR 2024). The FlexMoE and MUSE are designed for medical tasks. If reviewers have specific baselines they recommend for inclusion, we are open to incorporating them into our analysis.

[Cons3] Theoretical Justification about the Calculation of Modality Importances.

As discussed in Section 5.2, Paragraph 1, we explore two perspectives to calculate modality importance: the statistical perspective (Mean v.s. STD) and the value perspective (Mean v.s. Max) of attention In this paper, we propose a novel fusion mechanism to address the challenge of modality quality variation. Our empirical evidence supports the effectiveness of using Mean for determining modality importance. Although this paper does not include a theoretical analysis, our contributions have significant practical implications. Rigorous proof is valuable, but it is outside the scope of this work.

[Cons4] Dataset in Table 4&5

Thanks for pointing this, the datasets used in Table 4&5 are the MIMIC-IV. We will revise the table captions for better standalone clarity.

[Cons5] Citation about Contrastive Learning

Thanks for the reminder. We will include the related citations [1], [2] for instance level and token level contrastive learning respectively in our revision.

[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation [2] Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition

[Cons6] Novelty Clarification

Please refer to the Cons3 in our response to the Reviewer CN7m for this clarification due to limited space.

[Cons7] Section 3 is Hard to Follow

Section 3 introduces token fusion and discusses its drawbacks. The basic idea of token fusion is to prune single-modal transformers and reuse pruned units for multimodal fusion. Each single-modal transformer is pruned individually, and pruned units are replaced by projected alignment features from other modalities.

The drawbacks include: (1) the requirement for well-aligned modalities with explicit mapping, (2) the need to manually set a threshold $\theta$ for pruning, which requires additional effort to determine a feasible value, and (3) when the number of modalities exceeds two, tokens are randomly selected from other modalities as substitutes, assuming all modalities capture the correct information. We are open to refining this section further if the reviewer provides specific comments.

[Cons8] Typos

Thanks for the feedback. We will correct these typos in our next revision.

[Cons9] The Intuitive Justification and Robustness Clarification of Modality Importance Calculation

Intuitive Justification: For a reliable modality, attention values on each token should vary to capture meaningful information. At the same time, the overall attention value should be high to ensure that useful information is integrated effectively. This suggests that the attention map captures meaningful information across the input data. Both standard deviation (STD) and maximum (Max) can be affected by outliers. In contrast, a higher mean value is more robust.

Robustness Clarification: We conducted an ablation experiment on the ADNI dataset to assess robustness. The F1 scores for STD and Max were 53.93 and 52.99, respectively, which are lower than the F1 score of 54.82 achieved by Mean. This demonstrates that robutness of Mean operation for modality importance.

[Cons10] Missing Modality Scenarios

Please refer to the Cons3 in our response to the Reviewer JTwf for this clarification due to limited space.

We are very thankful for your time and consideration. Please let us know if you have any further questions; we will be happy to address them fully.

We thank the reviewer for their time and thoughtful feedback. We are encouraged that the reviewer finds our method novel in how it integrates token-level rebalancing with reasonable reproducibility and evaluation. Below, we address each concern in turn.

[Cons1] Overstated About the Modality Importance

To respond to the reviewer's concern regarding the importance of modalities, we propose two experiments to clarify the role of modality scores in identifying the significance of each modality.

1. Dataset Level Modality Importance: We assess modality importance at the dataset level using the MIMIC-IV dataset. Each modality is evaluated individually using the same AMC backbone. The results in the following table indicate that the 'Code' modality is less important (the lowest F1, Recall, and Precision), aligning with Figure 4 where 'Code' have a higher frequency of token replacement.

2. Instance Level Modality Measurements: We investigate the effectiveness of our modality score in reducing prediction uncertainty. We compare the uncertainty of predictions ($-\sum_{i=1}^{N}p_i\log(p_i)$, where $p_i$ is the predicted probability of class $i$) using AMC with its original modality scores against AMC with equal modality scores. Our findings show that the modality importance score effectively reduces prediction uncertainty, demonstrating its utility in identifying significant modalities.

These experimental results indicate that our modality importance adequately captures modality contribution for the final prediction.

[Cons2] Clinical Validation of the Interpretableability of AMC

Please refer to the Cons1 in our response to the Reviewer CN7m for this clarification due to limited space.

[Cons3] Extend to Missing Modality Scenario

Please refer to the Cons3 in our response to the Reviewer JTwf for this clarification due to limited space.

[Cons4] The Measurement of Modality Quality Definition

We define the modality quality as the performance of each modality with the same modal backbone. As we show in Figure 1, the bottom part shows the performance variations with the same network backbone but different modalities as the input. These performance differences of each modality across different cancer types show the challenge of data quality variation.

[Cons5] Gradient Conflict between Different Modalities.

We analyze the distribution of cosine distances between training gradients derived from the different modalities in the MIMIC-IV dataset. These gradients are computed from a model configured with experts and a dense MLP, both following the same configuration as in the AMC and Dense Model setups. The gradients are extracted from the final transformer layer. Higher positive cosine distances indicate reduced gradient conflict, which suggests better alignment between the modalities in terms of learning direction. Our result in Figure 1 of Anonymous shows that the SMoE effectively alleviates the gradient conflict between different modalities.

[Cons6] Training Efficiency

Please refer to the Cons2 in our response to the Reviewer CN7m for this clarification due to limited space.

[Cons7] Illustrating the customized token fusion workflow of Figure 3.

Thanks for pointing that out. In our token fusion, we first identify the token score (Equation 3) and modality importance (Equation 4) via the attention distribution. Then we utilize this information to decide which token should be replaced and selected to impute intra-modal tokens (Equation 4). Finally, we use $\{{x}_{m}\}_{m=1}^{M}$ for the final prediction. In the revision, we will revise the figure captions for better standalone clarity.

[Cons8] Fail/Underperform Analysis of AMC

When all modalities possess high quality, determining which token to replace in AMC's token fusion process becomes challenging. This can lead to potential information loss due to token replacement, which could explain why AMC underperforms compared to MulT and MAGGate in the LGG cancer type of the TCGA dataset. Despite this, AMC still achieves a top-3 performance among seven baseline methods in the TCGA LGG dataset. This demonstrates that while AMC may not be the best performer in this specific case, it still maintains competitive performance overall.

We sincerely thank you for your time and consideration. Please let us know if you have any further questions; we are more than happy to address them comprehensively.