$\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts

Thanks for your encouraging feedback. Point-to-point responses below.

Q1. For using triplet margin loss to model uniqueness interactions, what is the margin used in the work, why is the margin loss used here?

We use triplet margin loss to uniqueness interactions, as it naturally aligns with the modeling of uniqueness—there are both positive and negative examples. For example, for uniqueness expert for modality 1, we treat the full input as the anchor, the one input with modality 1 being masked as a negative, and all other perturbed inputs as positives.

Triplet margin loss maximizes the distance between the anchor and a negative and minimizes the distance between an anchor and a positive. We set the margin to 1.0, selected based on validation performance. A sweep across three datasets showed that margin = 1.0 performs best in nearly all cases (see table below).

Q2. The author should discuss the computation overhead for the proposed method, compared with its counterparts, such as I2MoE-MulT vs. MulT

While I2MoE introduces additional forward passes—scaling linearly with the number of modalities—the method remains fully end-to-end and weakly supervised. The fusion overhead increases by approximately (#modalities + 2), corresponding to the number of specialized experts.

To quantify the overhead, we compare I2MoE-MulT to MulT across training time per epoch, inference latency, and parameter count (see table below). All experiments are conducted on a single A100 GPU. Despite moderate increases, we find the added cost justified by the significant gains in interpretability and predictive performance.

Q3. MMoE is a very close work to the proposed method, but it is not well compared in the experiment, are there any special concerns about not comparing with it?

While both MMoE and I2MoE aim to model heterogeneous interactions, they differ substantially in design and applicability, making direct comparison non-trivial:

1. End-to-End vs. Preprocessing Dependency: I2MoE is fully end-to-end, with interaction specialization emerging via weak supervision. MMoE relies on a seperate preprocessing step to cluster training data by interaction type, breaking end-to-end training.
2. Local Interpretability: I2MoE provides instance-level interpretability by quantifying expert contributions per sample. MMoE lacks this capability, limiting its utility in settings requiring explanation of individual predictions.
3. Generalizability to Higher Modalities and Complex Domains: MMoE is tailored to vision-language tasks with pretrained LLM/VLM backbones. Its extension to domains like healthcare—where pretrained models for structured data or multi-way modality combinations are lacking—is unclear. In contrast, I2MoE operates without modality-specific pretraining and supports >2 modalities, as demonstrated on ADNI and MIMIC datasets.

Q4. In section 6.2, how to determine the agreement/disagreement between experts, do the authors use any thresholds?

We define expert agreement based on predicted labels. For single-label classification tasks (i.e., ADNI, MIMIC, ENRICO), each expert outputs a logit vector, and predictions are obtained via argmax. For multi-label classification tasks (i.e., MM-IMDB), predictions are thresholded at 0.5 after sigmoid activation. For the regression task (i.e., CMU-MOSI), each expert outputs a real number the threshold is set at 0. Agreement occurs when all experts predict the same label(s); disagreement is defined as any mismatch between expert outputs.

Thanks for your thoughtful feedback. Point-to-point responses below. All supplementary on GitHub.

Q1. I2MoE lower scores on MM-IMDB?

This is primarily due to differences in experimental setups:

1. Evaluation Setup: Our experiments follow the setup in MultiBench [1]. Within this framework, I2MoE achieves Micro-F1: 61.00 and Macro-F1: 52.38, outperforming the SOTA (Micro: 59.3, Macro: 50.2; see Table 23 in [1]).
2. Reimplement: While [2–4] report higher MM-IMDB scores, they adopt different experimental setups and do not release code for exact reproduction. We reimplemented CentralNet and ViLT under the MultiBench setting and observed performance drops compared to their original reports.
3. I2MoE Improvement: We further evaluated I2MoE as a drop-in framework applied to CentralNet and MulT. As shown below, I2MoE consistently improves performance across both Micro and Macro F1 under the MultiBench setup.

Q2. MM-IMDB result in Figure 5 appears different from Table 1

Thanks for catching this–we forgot to update the MM-IMDB result in Figure 5. Updated figures can be found on GitHub.

Q3. …motivation for using random vectors to mask modalities? ..modality mean vectors..?

We use random vectors to completely remove information from the masked modality so that any observed uniqueness and synergistic interaction from the masked modality cannot be attributed to leakage.

While mean vectors could be less noisy, they are also dynamic in our setting. Since I2MoE is trained end-to-end with modality-specific encoders, the mean vectors evolve across training epochs. To mitigate this instability, we maintain a running average of mean vectors (see GitHub), but this introduces additional complexity and still does not guarantee full information removal.

Further, we compare random / mean / zero vectors across five datasets. As shown below, random vectors consistently outperform mean vectors in nearly all settings.

Q4. Interpretation of Figure 3b

Figure 3b shows a local (sample-level) decomposition for a specific MM-IMDB example, where the language modality contributes less unique information for that instance. This does not reflect global trends in the MM-IMDB dataset. For dataset-level insights, we refer the reviewer to Figure 4, which summarizes expert weights across the entire test set.

Q5. (I2MoE) could significantly increase computational costs

Please see our response to Reviewer aVYo Q2.

Q6. The studied baselines such as VGG and LSTM seem somewhat outdated

We clarify that VGG and LSTM are used only as modality-specific encoders, not as fusion architectures. This setup follows widely adopted practice in multimodal learning benchmarks such as [1], where lightweight encoders are used to ensure a fair comparison of fusion methods. We agree that powerful modality-specific encoders could further boost performance. Due to time constraints, we plan to incorporate them in the final version.

C1. Some related studies might have been overlooked

We note that the two works mentioned focus on generative modeling, but we focus on interpretable discriminative modeling. We consider these lines of research orthogonal but will clarify this distinction and cite these works in our related work section.

C2. Some dataset names appear inconsistent (e.g., "MM-IMDB" vs. "MMIMDB").

We will standardize all occurrences to “IMDB” in the final version.

[1] Liang et al., 2021. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning. NeurIPS.
[2] Pérez-Rúa et al., 2019. MFAS: Multimodal Fusion Architecture Search. CVPR.
[3] Vielzeuf et al., 2018. CentralNet: A Multilayer Approach for Multimodal Fusion. ECCV Workshops.
[4] Ma et al., 2022. Are Multimodal Transformers Robust to Missing Modality?. CVPR.

Thanks for your constructive feedback. Point-to-point responses below.

Q1. The connection between interaction loss and PID

We would appreciate any insights from the reviewer on this point. Below, we attempt to connect each expert trained on the perturbed input views to a distinct PID component. This might form a contrastive approximation to the constrained information projections discussed in [1, 2]:

$

I(T; X_1, X_2) = \mathrm{Red}(T; X_1, X_2) + \mathrm{Unq}(T; X_1 \setminus X_2) + \mathrm{Unq}(T; X_2 \setminus X_1) + \mathrm{Syn}(T; X_1, X_2)
$

In the two-modality scenario, our model learns four experts, each trained to specialize in a PID component using perturbed modality inputs.

Unique Information [1, 3]. Experts $ F_{\text{uni}1} $ and $ F_{\text{uni}2} $ are trained on inputs where the other modality is replaced with noise:

$

\mathcal{L}_{\text{uni}1} = \| F_{\text{uni}1}(X_1, \tilde{X}_2) - T \|, \quad \mathcal{L}_{\text{uni}2} = \| F_{\text{uni}2}(\tilde{X}_1, X_2) - T \|
$

Assuming $ \tilde{X}_i $ contains no task-relevant information, these losses approximate:

$

\mathcal{L}_{\text{uni}1} \propto \mathrm{Unq}(T; X_1 \setminus X_2), \quad \mathcal{L}_{\text{uni}2} \propto \mathrm{Unq}(T; X_2 \setminus X_1)
$

Redundant Information [2, 3]. Expert $ F_{\text{red}} $ is trained to match predictions from either single-modality input:

$

\mathcal{L}_{\text{red}} = \frac{1}{2} \left( \| F_{\text{red}}(X_1, \tilde{X}_2) - T \| + \| F_{\text{red}}(\tilde{X}_1, X_2) - T \| \right)
$

This loss encourages $ F_{\text{red}} $ to extract information shared by both $ X_1 $ and $ X_2 $, approximating:

$

\mathcal{L}_{\text{red}} \propto \mathrm{Red}(T; X_1, X_2)
$

Synergistic Information [2, 4]. Expert $ F_{\text{syn}} $ is trained to rely on both modalities jointly. It is penalized for performing well on any partial view:

$

\mathcal{L}_{\text{syn}} = \frac{1}{2} \left( \| F_{\text{syn}}(X_1, X_2) - T \| - \| F_{\text{syn}}(\tilde{X}_1, X_2) - T \| - \| F_{\text{syn}}(X_1, \tilde{X}_2) - T \| \right)
$

This loss isolates information that emerges only through joint modality interaction:

$

\mathcal{L}_{\text{syn}} \propto \mathrm{Syn}(T; X_1, X_2)
$

Q2. Random vector replacement for modality dropout

Please see our response to Reviewer LfMU Q3.

Q3. Strategies to mitigate accuracy degradation in imbalanced datasets like MIMIC?

In imbalanced settings like MIMIC, threshold-independent metrics such as AUROC provide a more reliable measure of performance. We mitigate class-imbalance by applying class weighting (0.25 for negative, 0.75 for positive) during training across all models.

To clarify performance across classes, we report per-class accuracy, balanced accuracy, and AUROC below. I2MoE-MulT achieves higher balanced accuracy and AUROC than MulT, indicating improved positive class recognition without sacrificing majority class performance.

Q4. A discussion on computational overhead and scalability is encouraged

Please see our response to Reviewer aVYo Q2.

Q5. How does I2MoE scale when extending to more than two modalities or when applied to other multimodal tasks (e.g., video–audio fusion)?

Most real-world multimodal datasets ([5, 6]) involve fewer than four modalities. I2MoE performs well in such settings—for example, ADNI includes imaging, genetics, clinical tests, and biospecimens. We also evaluated I2MoE on video–audio fusion using CMU-MOSI (text, video, audio), where I2MoE improves the performance of four different baseline fusion methods (Table 1 and Table 2).

C1. I2MoE contrasts with conventional fusion methods and recent MoE-based models

Thanks for the comment. We will expand Section 2 to position I2MoE within the broader MoE literature you mentioned more explicitly.

[1] Bertschinger et al., 2014. Quantifying Unique Information. Entropy.
[2] Williams and Beer, 2010. Nonnegative Decomposition of Multivariate Information. arXiv.
[3] Wollstadt et al., 2023. A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition. JMLR.
[4] Wibral et al., 2017. Partial information decomposition as a unified approach to the specification of neural goal functions. Brain and Cognition.
[5] Liang et al., 2021. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning. NeurIPS.
[6] Liang et al., 2022. High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning. arXiv:2203.01311.

Thanks for your positive feedback. Point-to-point responses below.

Q1.The support for "local interpretation" is only backed by qualitative samples from one task

Thanks for suggesting to strengthen the evidence for local interpretability. We conducted a human evaluation with 15 participants on 20 movie examples (300 total ratings), asking how reasonable the model’s assigned interaction expert weights were. Participants chose from five options, ranging from “Makes no sense at all” to “Completely makes sense.”

70.4% of responses were positive (Mostly or Completely makes sense), while only 9% were negative, and just 0.7% selected the lowest rating. These results suggest that the model’s expert weights are broadly viewed as reasonable and interpretable by human evaluators.

Link to the questionnaire: Link

Link to deidentified response: Link

Q2. The font size of the tables and figures are tiny, making them hard to read.

Thank you for the helpful suggestion. We have increased the font sizes in all figures and tables and will include them in the final version.

Updated tables and figures: Link

Q3. Writing down the complete objective as a mathematical expression (or maybe an algorithm block for the entire training objective)

Algorithm block for I2MoE training and inference forward pass: Link

Below we explain the complete objective in math expression: Let $ \{F_i\}_{i=1}^{B} $ denote the $ B = n + 2 $ interaction experts: $ n $ uniqueness experts, one synergy expert, and one redundancy expert. For each expert $ F_i $, we obtain outputs from $ (1 + n) $ forward passes (one full input and one for each modality replaced):

$

[\hat{y}_i^{(0)}, \hat{y}_i^{(1)}, \dots, \hat{y}_i^{(n)}] = F_i\mathrm{.forward\_multiple}(X_1, \dots, X_n)
$

The main prediction is computed as:

$

\hat{y} = \sum_{i=1}^{B} w_i \cdot \hat{y}_i^{(0)}, \quad \text{where } [w_1, \dots, w_B] = \text{MLPReWeight}(X_1, \dots, X_n)
$

The task loss is defined as:

$

\mathcal{L}_{\text{task}} = \ell(\hat{y}, T)
$

We define the expert-specific interaction losses as follows:

Uniqueness loss for each $ F_i $ ($ i = 1, \dots, n $):

$

\mathcal{L}_{\text{int}}^{(i)} = \frac{1}{n - 1} \sum_{j \ne i} \text{TripletLoss}\left( \hat{y}_i^{(0)},\; \hat{y}_i^{(j)},\; \hat{y}_i^{(i)} \right)
$

Synergy loss ($ F_{n+1} $):

$

\mathcal{L}_{\text{int}}^{(n+1)} = \frac{1}{n} \sum_{j=1}^{n} \text{CosSim}\left( \text{normalize}(\hat{y}_{n+1}^{(0)}),\; \text{normalize}(\hat{y}_{n+1}^{(j)}) \right)
$

Redundancy loss ($ F_{n+2} $):

$

\mathcal{L}_{\text{int}}^{(n+2)} = \frac{1}{n} \sum_{j=1}^{n} \left( 1 - \text{CosSim}\left( \text{normalize}(\hat{y}_{n+2}^{(0)}),\; \text{normalize}(\hat{y}_{n+2}^{(j)}) \right) \right)
$

We then average the interaction loss over all experts:

$

\mathcal{L}_{\text{int}} = \frac{1}{B} \sum_{i=1}^{B} \mathcal{L}_{\text{int}}^{(i)}
$

The final training objective is:

$

\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \lambda_{\text{int}} \cdot \mathcal{L}_{\text{int}}
$

Model parameters are updated to minimize $ \mathcal{L}_{\text{total}} $.