Sparse MoE as a New Retriever: Addressing Missing Modality Problem in Incomplete Multimodal Data

[Q2: Feature Retrieval in MIMIC Dataset]

In the MIMIC dataset, the label for each patient is not tied to a specific diagnosis (as in the ADNI dataset) but instead pertains to one-year mortality classification, as mentioned in Appendix A1.2 of the original paper.

To address the reviewer’s inquiry regarding how the retrieval process contributes to performance at the sample (i.e., patient) level, we provide a t-SNE plot in Figure 6 (Appendix 7) using the MIMIC dataset for the one-year mortality prediction task.

The plot can also be accessed via the following link:

• https://anonymous.4open.science/r/moe-retriever-rebuttal-24A8/assets/tsne.png

Specifically, we illustrate patient-level embeddings, focusing on patients with $\mathcal{LC}$ modalities but missing the $\mathcal{N}$ modality. In Figure 6 (a), before retrieving the $\mathcal{N}$ modality, the embeddings of patients labeled as "alive" are more widely dispersed. In contrast, Figure 6 (b), which incorporates both inter-modal context (i.e., $\mathcal{N}$ embeddings derived from other patients with $\mathcal{LNC}$ modalities) and intra-modal context (i.e., modality-specific embeddings from $\mathcal{L}$ and $\mathcal{C}$), shows more condensed embeddings.

By leveraging both intra- and inter-modal contexts, we observe that patient embeddings associated with their respective labels become more similar and compact. This improvement in embedding space positively contributes to the downstream task.

[Q3: Open-Sourcing the Code for Processing]

Thank you for your interest in the CMU-MOSI experiment setup. In the experiment, we simulated missing modalities with a probability of 0.3 for each modality independently (and included additional variants during the rebuttal period).

The code for this experiment is available in the following anonymous GitHub repository:

• Link: https://anonymous.4open.science/r/moe-retriever-rebuttal-24A8/dataset.py

We sincerely appreciate the reviewer nRr5 time and effort in reviewing our paper. If you have any remaining concerns, please feel free to reach out. We would be more than happy to address them and are fully prepared to provide our best responses until the deadline ends.

Best,
Authors

Table 5: AUROC Results of CMU-MOSI Modality A+T

From the above tables, we observe that MoE-Retriever outperforms the baselines in the majority of cases. This aligns with the results presented in the paper using Accuracy and F1 metrics, further demonstrating MoE-Retriever's robust performance, particularly in imbalance scenarios such as the MIMIC dataset.

We have incorporated the above results into our refined version and also included the PRAUC results as suggested by Reviewer 3etg. We kindly ask the reviewer to refer to the Appendix for a more comprehensive evaluation of the results.

[W3: English Expression]

Thank you for pointing out the vague expression in the abstract. What we intended to convey with the phrase "lack of specialized knowledge in embedding candidates" is that current learnable embeddings often lack modality-specific or relevant knowledge that could enhance the imputation process and downstream tasks.

The term "specialized knowledge" is commonly used in SMoE literature, often in the context of experts. However, to make the expression clearer and more reader-friendly, we have revised it in the refined version to "lack of modality-specific knowledge" which we believe is more concrete and straightforward.

[Q1: Advantage of MoE-Retriever on Medical Datasets]

We would like to clarify that our method is designed to be domain-agnostic and is not specifically tailored for the medical domain. However, its benefits are particularly notable on medical datasets due to the inherent challenges of such data, including more diverse modalities and frequent missingness. In the medical context, missing modalities often occur due to the cost of experiments or specific patient-related factors, making an effective retrieval process critical for better performance.

Empirically, as shown in main paper Table 1 (Medical) and Table 2 (General), the performance gap between MoE-Retriever and the baselines is slightly larger in Table 1. This indicates that in scenarios with more diverse and severe missing modalities, MoE-Retriever can actively address these challenges, providing a significant advantage.

However, we would like to emphasize again that our proposed method is not restricted to the medical domain. It is designed to be generalizable across various domains and scenarios, demonstrating robust performance in a wide range of applications.

Therefore, while the fine-grained approach may occasionally enhance performance, the trade-off in terms of computational cost is significant. This underscores that the current version of MoE-Retriever strikes a balanced and attractive position between performance and efficiency. Moreover, these findings highlight that the key factor is not merely careful sampling within a pool but, more importantly, how the sampling pool itself is defined—for instance, the modality-combination-based supporting group proves to be crucial for robust performance.

In summary, MoE-Retriever specifically emphasizes incorporating the SMoE design in the missing modality scenario, functioning as a retrieval process rather than solely focusing on effective fusion. Additionally, its balanced design, achieving both strong performance and computational efficiency, further highlights its practical significance. We respectfully argue that the novelty and contributions of our work should not be considered minor elaborations but rather meaningful advancements in the multimodal learning field.

[1] Fan, Dongyang, Bettina Messmer, and Martin Jaggi. "Towards an empirical understanding of MoE design choices." arXiv preprint arXiv:2402.13089 (2024).
[2] Huang, Quzhe, et al. "Harder Tasks Need More Experts: Dynamic Routing in MoE Models." arXiv preprint arXiv:2403.07652 (2024).
[3] Dai, Damai, et al. "Stablemoe: Stable routing strategy for mixture of experts." arXiv preprint arXiv:2204.08396 (2022).
[4] Liu, Tianlin, et al. "Routers in vision mixture of experts: An empirical study." arXiv preprint arXiv:2401.15969 (2024).

[W2: AUROC Results on CMU-MOSI Dataset]

We appreciate the reviewer for suggesting AUROC as a useful metric to further evaluate the performance of our model. As a response, during the rebuttal period, we conducted additional experiments on the MIMIC and CMU-MOSI datasets, including varying missing rates (noise levels) for CMU-MOSI to provide a more comprehensive evaluation. The results are summarized below:

Table 1: AUROC results of MIMIC

Table 2: AUROC results of CMU-MOSI Modality V+A+T

Table 3: AUROC Results of CMU-MOSI Modality V+A

Table 4: AUROC Results of CMU-MOSI Modality V+T

We sincerely appreciate Reviewer nRr5 for highlighting the strengths of MoE-Retriever, such as addressing "one of the most important concerns of multimodal learning," conducting "extensive and satisfactory experiments," and presenting the work in a "clearly written" manner. Below, we provide detailed point-by-point responses to the remaining concerns.

[W1: Comparison with FuseMoE and More Designs]

We would first like to clarify the main differences between FuseMoE and our work. While both studies employ Sparse Mixture-of-Experts (SMoE), the fundamental differences lie in how SMoE is utilized in each work. These distinctions can be summarized from the following two perspectives:

1. Goal of the Study and Usage of SMoE Specifically, FuseMoE employs SMoE in the fusion layer, aiming to effectively fuse multimodal information into the same latent space. In contrast, MoE-Retriever utilizes SMoE prior to the fusion layer, with the primary goal of imputing missing modalities while leveraging both intra-modal and inter-modal contexts. This approach enables the effective imputation of missing modality information. Therefore, we respectfully claim that MoE-Retriever is not simply an added value on top of FuseMoE, as it does not rely on any backbone design of FuseMoE. The utilization and purpose of SMoE in MoE-Retriever are fundamentally distinct, focusing specifically on imputation.

2. Router Design We clarify that router design is not the main focus of our study, unlike prior works that primarily investigate the router design of SMoE [1, 2, 3, 4]. Instead, our study focuses on how to effectively utilize SMoE for imputation in the context of missing modalities. Nonetheless, in Section 4.4 (Table 3) of the main paper, we include an ablation study comparing router designs with different numbers of routers to handle both intra- and inter-modal contexts. The results demonstrate that using multiple routers does not outperform a single-router setup, reinforcing the effectiveness of the current router design in MoE-Retriever.

   In summary, rather than advancing router design, our contribution lies in effectively adopting the SMoE framework for the missing modality scenario. Specifically:

   1. Input Constitution: Leveraging both intra- and inter-modal contexts as inputs for SMoE.
   2. SMoE Goal: Enabling the router to automatically identify the most relevant experts for imputing missing modality information based on the provided context and modality-specific experts.

Moreover, to facilitate the SMoE design, we focus on refining the inputs to the SMoE, which consist of intra-modal and inter-modal contexts. While the inter-modal context is constructed by simply adding the observed modality-specific embeddings, we have made efforts to further enhance the intra-modal sampling procedure. Specifically, we moved beyond random sampling to explore more sophisticated approaches that could better capture the relevant intra-modal relationships, ultimately improving the retrieval process. Based on similarity rather than randomness, could further improve performance, we implemented a cosine similarity-based sampling procedure.

More specifically:

1. For a sample with observed modalities G and C, aiming to retrieve modality I, we first define a supporting group containing samples that possess G and C as a sufficient condition and also include I as the missing modality.
2. Since all supporting group samples share G and C as common modalities, we concatenate their embeddings and compare them with the embedding of the original sample using cosine similarity.
3. The similarity scores are softmaxed to form a probability distribution, which is then used to guide the sampling process. This transforms the sampling into a multinomial distribution where each trial is weighted by the cosine similarity as a sampling probability.

The results of this similarity-based sampling procedure are shown below:

We observe that while the fine-grained approach brings performance gains in certain cases (e.g., I+G+CI+G+C) due to its more careful sampling process, the overall performance remains more robust in the original version.

One important characteristic to note is that leveraging cosine similarity during the training process significantly increases computational complexity. Since parameter updates are performed batch-wise, the cosine similarity scores used for probability sampling must be recalculated in each batch for all possible samples in the supporting group, resulting in a substantial increase in computation time—approximately 20 times higher than the original mean runtime.

We thank reviewer nRr5's prompt engagement and feedback. We are pleased that our responses have alleviated the reviewer’s concerns regarding the novelty and metrics, and we sincerely appreciate the reviewer’s acknowledgment and consideration of increasing the score.

[Follow up Question: Consistent Performance of Acc and F1 for mmFormer on MIMIC Dataset]

We appreciate the detailed question. The consistent accuracy and F1 scores across modality combinations in mmFormer arise partly from its architecture, which emphasizes modality-invariant global features through the modality-correlated Transformer. While mmFormer also incorporates local features via convolutional layers in its hybrid modality-specific encoder, these local features may not be fully captured under imbalanced settings. The reliance on standard cross-entropy loss and zero imputation during training limits the model’s ability to emphasize local context from more informative modalities when some modalities (e.g., $\mathcal{N}$ in MIMIC dataset) are missing. This imbalance leads to a stronger reliance on global, invariant representations, which can smooth out the impact of individual modalities and result in consistent binary predictions across modality combinations, often outputting the same predicted label regardless of the modality combination.

However, although accuracy and F1 scores appear consistent across combinations in Table 1 in main paper, metrics like AUROC (Table 5 in Appendix A.4) and PRAUC (Table 6 in Appendix A.4) show differences because they measure the ranking and confidence of predicted probabilities, rather than binary decisions. These metrics reveal that the predicted probabilities vary due to differences in how the model balances local and global information across combinations. When key modalities are missing, the model’s reliance on global representations from the modality-correlated encoder leads to shifts in probability distributions, which are captured by AUROC and PRAUC. Other baselines, which may not focus as heavily on global modality-invariance, demonstrate variability across all metrics, suggesting that mmFormer's robustness to missing modalities comes at the cost of reduced sensitivity to local, modality-specific dynamics. In this regard, treating each modality independently, such as employing modality-specific experts responsible for retrieving embeddings for missing modalities, as proposed in MoE-Retriever, can be an attractive remedy in such scenarios.

To clarify this phenomenon further, we will explicitly mention this unique characteristic of mmFormer in the final version to ensure readers fully understand its implications.

Thank you again for your engagement. If you have further concerns, please do not hesitate to let us know.

I thank the authors for their detailed response and extensive additional experiments. I think the response well-addresses my concern about the novelty compared to existing MoE methods, the lack of comparable metrics, and the details of implementations. Therefore I have raised my scores.

Besides, I notice that the accuracies and F1s for mmFormer on MIMIC are exactly the same with zero SD across different modality combinations. Could the author briefly explain why this is the case?

We appreciate Reviewer UWni for acknowledging MoE-Retriever’s motivation of "imputing missing modalities" and the "reasonable usage of SMoE." Below, we address the remaining weaknesses point by point.

[W1: Limited Novelty]

We sincerely thank the reviewer for highlighting the strengths of our work, particularly the intuitive approach to imputing missing modalities using SMoE. Our main contribution lies not in merely applying existing techniques but in demonstrating their proper and relevant usage in the context of missing modalities in multimodal learning.

In prior works within this domain, such as FuseMoE, SMoE has been effectively utilized due to its efficient router-expert design, but predominantly in the fusion layer. Conversely, for imputing missing modalities, existing approaches often rely on simple learnable features or complete modalities, overlooking the nuanced context of the missing modality.

Our work targets this specific gap by emphasizing the importance of carefully leveraging both intra-modal and inter-modal contexts. Given these contexts, SMoE’s design principle enables the selection of the most relevant experts to retrieve the missing modality. This bridges the gap between missing modality imputation and the efficient utilization of SMoE.

To verify the significance of our proposed modules—intra-modal context, inter-modal context, and context-aware routing—we refer to the ablation study in Table 3 of Section 4.4. The results demonstrate the individual importance of each module and their synergistic effect when integrated.

In this regard, we respectfully argue that our work is novel, as it is the first to address missing modality imputation through the lens of SMoE design principles, where its application is both necessary and natural.

[W2: Architecture - Incorporating Similarity Measure]

We thank the reviewer for suggesting this interesting and promising idea to further enhance our current architecture. As mentioned, in our current intra-modal sampling process, we rely on sampling from a modality-combination-based supporting group. The effectiveness of this approach is demonstrated in the ablation study (Table 3, Section 4.4), where eliminating intra-modal context (i.e., "w/o Intra-Modal Context") negatively impacts overall performance. This highlights the importance of including sampled intra-modal examples.

However, as the reviewer pointed out, our current approach involves random sampling of intra-modal examples, under the assumption that samples sharing similar modality combinations and including the missing modality can positively contribute to the retrieval process. To evaluate whether a more fine-grained sampling strategy, based on similarity rather than randomness, could further improve performance, we implemented a cosine similarity-based sampling procedure.

More specifically:

1. For a sample with observed modalities G and C, aiming to retrieve modality I, we first define a supporting group containing samples that possess G and C as a sufficient condition and also include I as the missing modality.
2. Since all supporting group samples share G and C as common modalities, we concatenate their embeddings and compare them with the embedding of the original sample using cosine similarity.
3. The similarity scores are softmaxed to form a probability distribution, which is then used to guide the sampling process. This transforms the sampling into a multinomial distribution where each trial is weighted by the cosine similarity as a sampling probability.

The results of this similarity-based sampling procedure are shown below:

We observe that while the fine-grained approach brings performance gains in certain cases (e.g., I+G+CI+G+C) due to its more careful sampling process, the overall performance remains more robust in the original version.

One important characteristic to note is that leveraging cosine similarity during the training process significantly increases computational complexity. Since parameter updates are performed batch-wise, the cosine similarity scores used for probability sampling must be recalculated in each batch for all possible samples in the supporting group, resulting in a substantial increase in computation time—approximately 20 times higher than the original mean runtime.

Dear reviewer UWni,

We are nearing the end of the discussion period, so please respond to the rebuttal by the authors of submission 11331.

All the other reviewers have weighed in, so we are waiting for your response to make a determination concerning this paper.

Please indicate the extent to which your concerns are addressed and explain your decision to update (or not update) your score.

All the best,

The AC

I appreciate the authors' efforts in clarifying the methodology and providing additional experiment results. I have raised my score.

FYI: regarding W2, I think using some algorithms like approximate nearest neighbor search might improve the efficiency.

From the experiment, we observe that in dense models, while the cosine similarities are higher for the combination of V+A, other combinations, such as V+T and A+T, suffer from relatively low cosine similarities. This imbalance can negatively affect the overall optimization process.

In contrast, using sparse models that leverage SMoE in the fusion layer (e.g., FuseMoE) mitigates this issue by achieving a higher average cosine similarity with lower variance. Furthermore, by incorporating SMoE in the retrieval process of embeddings, which is the core motivation of MoE-Retriever, we achieve the highest average cosine similarities across modalities with the lowest variance. This enables MoE-Retriever to deliver the best performance among all baselines when using three modalities.

These findings underscore the importance of addressing gradient conflicts through SMoE utilization, particularly in its role as a retriever of embeddings, thereby optimizing synergy among modalities. This suggests a promising direction for future research in multimodal learning.

We will include this experiment in the refined version and thank the reviewer for raising this phenomenon, fostering further discussion and progress within the community.

[1] Liu, Kuan, et al. "Learn to combine modalities in multimodal deep learning." arXiv preprint arXiv:1805.11730 (2018).
[2] Peng, Jie, et al. "Sparse moe as a new treatment: Addressing forgetting, fitting, learning issues in multi-modal multi-task learning." (2023).
[3] Yu, Shoubin, Jaehong Yoon, and Mohit Bansal. "Crema: Multimodal compositional video reasoning via efficient modular adaptation and fusion." arXiv preprint arXiv:2402.05889 1 (2024).

[Q4: Computational Efficiency of MoE-Retriever]

In Figure 4, all baseline models experience an increase in computational cost due to the inclusion of modality-specific encoders and their respective fusion layers. However, the reason these increases appear visually similar is the relatively larger computational scale of the baseline models compared to MoE-Retriever.

Below is the actual table used to generate Figure 4:

From the table, we observe that baseline models such as ShaSpec, mmFormer, and M3Care, which appear to have relatively stable computational efficiency, actually experience increases in the number of parameters and GFLOPs as the number of modalities increases. However, due to the inherent complexity of their fusion layers—often involving significant stacking of transformer layers—the initial computational cost and parameter count for these baselines are already high when only two modalities are used. Consequently, the relative increase in computational efficiency appears small as additional modalities are incorporated.

[Q1: Results of PRAUC on the MIMIC Dataset]

As the reviewer noted, the use of PRAUC is indeed a valuable metric, particularly for addressing the label imbalance issue in the MIMIC dataset. We have included the PRAUC results for the baseline models and the MoE-Retriever in the table below:

From the results in the table, we observe that MoE-Retriever outperforms the baseline models in the majority of cases. We attribute this performance gain to MoE-Retriever's ability to effectively combine intra-modal and inter-modal contexts, which helps to supplement minority class data. Additionally, its use of a Sparse Mixture-of-Experts (SMoE) router ensures that only the most relevant experts are activated, leading to robust embeddings and higher PRAUC, even in the presence of class imbalance.

This experiment has been added to the refined version in Appendix A.5.

[Q2: Cause of Performance Drop in the CMU-MOSI Dataset]

We appreciate the reviewer for raising this issue. This is a fairly common challenge observed across other baselines as well. In all baselines, compared to the V+A+T modality combinations, the best performance is often achieved with only two modalities. For instance, models like mmFormer, ShaSpec, and M3Car perform best with A+T, while MUSE and FuseMoE achieve optimal performance with V+T.

We posit that in multimodal scenarios, including more modalities does not always guarantee an improvement in overall performance. While additional modalities may provide more information, they can also introduce unwanted noise and challenges such as modality interference, leading to gradient conflicts. Addressing such interference is an active area of research in multimodal learning [1, 2, 3], but it is not the primary focus of this paper. Our study centers on how to effectively retrieve missing modalities in multimodal scenarios by leveraging intra- and inter-modal contexts with an SMoE design.

Nonetheless, during the rebuttal period, we investigated why adding more modalities does not always improve performance from an optimization perspective. To analyze this, we examined the gradients when all modalities are included. Specifically, we computed the derivative of the loss with respect to each modality and measured the cosine similarities between modality pairs to detect potential gradient conflicts. Higher cosine similarity indicates a positive correlation between gradients, while lower values suggest conflicts.

We tracked the median cosine similarity values during training for the following configurations:

1. Dense Models: Models with a Transformer-based fusion layer, such as mmFormer.
2. Sparse Models: Models with an SMoE-based fusion layer, like MoE-Retriever.

The numerical results are presented below:

For a more comprehensive histogram plot, please refer to the link below or Appendix A.6 in the refined version:

• Link: https://anonymous.4open.science/r/moe-retriever-rebuttal-24A8/assets/image-4.png

We thank Reviewer 3etg for taking the time to review our paper and for providing constructive feedback, highlighting the "clear and convincing motivation" and "reasonable method." Below, we provide detailed point-by-point responses.

[W1 & Q3: Different Missing Ratios of CMU-MOSI & ENRICO]
We thank the reviewer for emphasizing the importance of analyzing diverse missing scenarios, as this would provide a more comprehensive evaluation. Beyond the fixed noise ratio of 0.3 used in the paper, we have included additional scenarios with lower noise (0.1) and higher noise (0.5). The performance results on the CMU-MOSI dataset are shown below:

Table 1: Accuracy of CMU-MOSI Modality V+A

Table 2: Accuracy of CMU-MOSI Modality V+T

Table 3: Accuracy of CMU-MOSI Modality A+T

Table 4: Accuracy of CMU-MOSI Modality V+A+T

From the results in the tables, we observe that in diverse settings (noise ratios: 0.1, 0.3, 0.5) and across various modality combinations (V+A, V+T, A+T, V+A+T), MoE-Retriever demonstrates strong and robust performance in most cases. Furthermore, as the noise ratio increases—indicating more severe missing scenarios—the performance gap between MoE-Retriever and the baselines widens in most cases (as shown in Tables 2, 3, and 4).

This robustness is attributed to the imputation process in MoE-Retriever, which leverages both intra-modal and inter-modal contexts. In contrast, the baselines either rely on simple imputation techniques or focus solely on modality fusion while leveraging only the observed modalities.

[W2: Data Statistics of MIMIC Dataset]

We thank the reviewer for inquiring about the missingness statistics of the MIMIC dataset. Below are the details regarding missingness across the 9,003 patient records:

• Code Modality: This modality combines diagnosis and procedure data. There are 4 records with missing diagnoses and 1,777 records with missing procedures.
• Note Modality: Derived from the "text" column in the original CSV file, there are 108 records with missing notes.
• Lab Modality: This modality includes 2,172 different measurements, presenting a more complex missingness scenario. If all 2,172 measurements are considered, there is technically no missing data since critical measurements, such as heart rate, are consistently collected for each patient. However, when evaluating the proportion of missing values in the 9,003×2,1729,003 \times 2,172 matrix, we find that 94.216% of the entries are NaN.

We hope this clarifies the missingness statistics of the MIMIC dataset. For better transparency, these statistics have been included in Appendix 1.2 of the refined version.

We are pleased to hear that our detailed response has addressed most of reviewer 3etg’s concerns. For the remaining follow-up questions, we would like to provide our responses below.

[Follow-up Q1: Comparison with Flex-MoE]

Thank you for raising this insightful question. We would like to clarify the differences between Flex-MoE and MoE-Retriever from the following two perspectives: (a) focus of work and (b) methodology. Additionally, we explore the potential for (c) positive synergy between the two approaches.

• (a) Focus of Work
  While both works address the challenge of missing modalities in multi-modal learning—an actively explored field—we respectfully highlight that the problem formulation and the proposed solutions are fundamentally different.

  1. Flex-MoE focuses on effectively modeling various modality combinations, particularly in the fusion layer. It achieves this by replacing the FFN layer in a Transformer block with a SMoE layer in the fusion layer. Although it introduces a Missing Modality Bank as a remedy, this component is essentially a learnable bag-of-embeddings approach that does not consider contextual factors, such as which observed samples might benefit the sample with missing modalities or how the observed modalities themselves contribute to imputation.

  2. MoE-Retriever, on the other hand, emphasizes the careful imputation of relevant embeddings for missing modalities by leveraging intra- and inter-modal contexts. These contexts, which are absent in Flex-MoE, allow MoE-Retriever to generate contextually-aware imputed embeddings before proceeding to the fusion step. For the fusion, MoE-Retriever employs a standard Transformer layer to aggregate multi-modal inputs.

  In summary, Flex-MoE prioritizes effective fusion techniques, while MoE-Retriever focuses on robust imputation methods.

• (b) Methodology
  The methodologies of Flex-MoE and MoE-Retriever reflect their distinct objectives:

  1. Flex-MoE:
  • Missing Modality Bank: Provides learnable embeddings derived from observed modality pairs and their corresponding missing modalities.
  • Generalized Router: Utilizes fully observed samples to train modality-combination-based experts in SMoE with generalized knowledge.
  • Specialized Router: Uses few observed samples to specialize experts for each corresponding modality combination.
  2. MoE-Retriever:
  • Intra-Modal Context: Defines a supporting group of samples sharing similar observed modalities and missing modalities, capturing intra-modal similarities.
  • Inter-Modal Context: Leverages observed modalities within each sample to provide personalized semantic information.
  • Context-Aware Routing Policy: Based on intra- and inter-modal inputs, the router selects top-k missing modality-specific experts to retrieve the most relevant embeddings.

  In summary, Flex-MoE prioritizes new routing policies for SMoE to model arbitrary modality combinations, whereas MoE-Retriever designs necessary inputs (intra- and inter-modal contexts) for SMoE to retrieve contextually-relevant features for missing modalities.

• (c) Exploring Synergies
  Beyond comparison, we explored potential synergies between Flex-MoE and MoE-Retriever to investigate whether their approaches can complement each other. Specifically, MoE-Retriever was used to generate imputed embeddings for missing modalities, which were then fed into Flex-MoE's router design for fusion. Below are the results of this combination on the CMU-MOSI dataset: | Modalities | Flex-MoE | MoE-Retriever | MoE-Retriever + Flex-MoE | |:----------:|----------------|-----------------|--------------------------------| | V + A | 46.43 ± 1.58 | 53.12 ± 2.26 | 56.68 ± 2.12 (+6.70%) | | V + T | 62.61 ± 2.70 | 65.74 ± 0.55 | 66.04 ± 1.83 (+0.46%) | | A + T | 61.27 ± 2.79 | 66.13 ± 0.69 | 66.44 ± 1.84 (+0.47%) | | V + A + T | 60.71 ± 2.88 | 65.21 ± 2.72 | 66.76 ± 2.01 (+2.38%) |

  From these results, we observe:

  1. Flex-MoE alone does not outperform MoE-Retriever, emphasizing the importance of careful imputation of missing modalities, particularly by considering intra- and inter-modal contexts.
  2. The combination of MoE-Retriever and Flex-MoE surpasses the performance of either method alone, demonstrating that using MoE-Retriever for imputation followed by Flex-MoE for fusion creates a strong baseline. It not only imputes missing modalities effectively but also leverages available modality combinations through expert design. This shows MoE-Retriever’s generalizability as an imputer capable of complementing advanced fusion techniques.

To enhance clarity for future readers, we will ensure that the comparison and the potential synergy of MoE-Retriever with existing fusion methods being included in the final version.

[Follow-up Q2: Cooperativeness and Performance on V+A]
Thank you for highlighting this important and insightful question. We would like to emphasize a significant claim that the multi-modal learning field should carefully consider:

"Although higher cosine similarity can facilitate faster convergence and improve the optimization process, it does not always guarantee better downstream performance."

This is because, in addition to pairwise relative similarities, the absolute modality-specific information quality and its contribution to the task must also be taken into account. Now, specifically, in the CMU-MOSI dataset [1], preprocessing [2] treats the text (T) modality as the central modality. Specifically, the dataset is preprocessed to ensure temporal alignment among the modalities (vision (V), audio (A), and text (T)) based on word-level granularity. As stated in [2]:

"To reach the temporal alignment between different modalities, we choose the granularity of the input to be at the level of words."

This preprocessing implies that the dataset is heavily centered on the text modality, resulting in an uneven distribution of information across modalities.

The preprocessed feature shapes for each modality are as follows:

• Vision (V): (2199, 50, 20)
• Audio (A): (2199, 50, 5)
• Text (T): (2199, 50, 300)

Here, 2199 is the number of samples, 50 represents the number of timestamps, and 20, 5, and 300 are the number of features for vision, audio, and text, respectively. This shows that the text modality has 15× and 60× more features than the vision and audio modalities, respectively, making it likely to dominate the downstream performance. However, even with fewer features, individual modalities like vision and audio can significantly contribute to the task if their feature-label relevance is high.

To validate this, we calculated the feature-label correlation for each feature (averaged globally across timestamps). The statistics (Max, Mean, Median, Normalized Sum of correlations) for each modality are as follows:

For a more comprehensive view of the correlation distributions for each modality, please refer to the plot linked below:

• Link: https://anonymous.4open.science/r/moe-retriever-rebuttal-24A8/assets/modality_correlation_plot.png

As a result, due to the dominance of the text modality, Table 2 in the main paper shows that when the central modality (T) is missing (i.e., V+A), performance is the lowest. This occurs even though the cosine similarity of gradients is the highest, likely because the smaller feature dimensions in vision and audio result in gradients that align more closely pairwise.

These observations suggest that when evaluating modality contributions, it is crucial to consider not only cosine similarity between gradients but also the modality-specific characteristics, such as feature-label correlation and the amount of information each modality contributes.

[1] Liang, Paul Pu, et al. "Multibench: Multiscale benchmarks for multimodal representation learning." Advances in neural information processing systems 2021.DB1 (2021): 1.

[2] Liang, Paul Pu, Ruslan Salakhutdinov, and Louis-Philippe Morency. "Computational modeling of human multimodal language: The mosei dataset and interpretable dynamic fusion." First Workshop and Grand Challenge on Computational Modeling of Human Multimodal Language. Vol. 113. 2018.

We sincerely appreciate reviewer 3etg’s insightful follow-up feedback and hope that the above responses address the remaining concerns. If there are any further questions or clarifications needed, please do not hesitate to reach out to us before the deadline.

Best,
Authors

We sincerely thank Reviewer L5fQ for acknowledging our work as "significant and worth investigating" and appreciating that "considering both contexts is interesting and important." To address the identified weaknesses, we provide detailed point-by-point responses below.

[W1: Novelty Compared to FuseMoE]

We respectfully disagree with this concern. While both studies employ Sparse Mixture-of-Experts (SMoE), the fundamental differences lie in how SMoE is utilized in each work. These distinctions can be summarized from the following two perspectives:

1. Goal of the Study and Usage of SMoE Specifically, FuseMoE employs SMoE in the fusion layer, aiming to effectively fuse multimodal information into the same latent space. In contrast, MoE-Retriever utilizes SMoE prior to the fusion layer, with the primary goal of imputing missing modalities while leveraging both intra-modal and inter-modal contexts. This approach enables the effective imputation of missing modality information. Therefore, we respectfully claim that MoE-Retriever is not simply an added value on top of FuseMoE, as it does not rely on any backbone design of FuseMoE. The utilization and purpose of SMoE in MoE-Retriever are fundamentally distinct, focusing specifically on imputation.

2. Router Design We clarify that router design is not the main focus of our study, unlike prior works that primarily investigate the router design of SMoE [1, 2, 3, 4]. Instead, our study focuses on how to effectively utilize SMoE for imputation in the context of missing modalities. Nonetheless, in Section 4.4 (Table 3) of the main paper, we include an ablation study comparing router designs with different numbers of routers to handle both intra- and inter-modal contexts. The results demonstrate that using multiple routers does not outperform a single-router setup, reinforcing the effectiveness of the current router design in MoE-Retriever.

   In summary, rather than advancing router design, our contribution lies in effectively adopting the SMoE framework for the missing modality scenario. Specifically:

   1. Input Constitution: Leveraging both intra- and inter-modal contexts as inputs for SMoE.
   2. SMoE Goal: Enabling the router to automatically identify the most relevant experts for imputing missing modality information based on the provided context and modality-specific experts.

[1] Fan, Dongyang, Bettina Messmer, and Martin Jaggi. "Towards an empirical understanding of MoE design choices." arXiv preprint arXiv:2402.13089 (2024).
[2] Huang, Quzhe, et al. "Harder Tasks Need More Experts: Dynamic Routing in MoE Models." arXiv preprint arXiv:2403.07652 (2024).
[3] Dai, Damai, et al. "Stablemoe: Stable routing strategy for mixture of experts." arXiv preprint arXiv:2204.08396 (2022).
[4] Liu, Tianlin, et al. "Routers in vision mixture of experts: An empirical study." arXiv preprint arXiv:2401.15969 (2024).

[W2: Clarity of Intra-Modal Context]

Before diving into Equation (2), we would like to provide a clearer overview. The primary goal is to define a supporting group (G) responsible for selecting intra-modal samples. This supporting group consists of the indices (j) where the modality combination satisfies two conditions:

1. It possesses the observed modality combination.
2. It contains the target (missing) modality, which we aim to impute.

For instance, if a sample has ‘GC’ modalities and we aim to impute the ‘I’ modality (as illustrated in Figure 2), the supporting group should include samples with both ‘GC’ and the missing modality ‘I.’ Consequently, the supporting group consists of samples where the modality combination is ‘IGC’ (e.g., $P_2$) or ‘IGCB’ (e.g., $P_9$).

I appreciate the responses from the authors. Most of my concerns have been alleviated. Particularly, with the updated notations, the equations are now readable and understandable. However, while I agree that the proposed method focuses on addressing the missing modality problem, I think the technical contribution is still a bit incremental. I tend to keep my rating unchanged.

We are glad that reviewer L5fQ’s primary concerns, such as updated notations and equations, have been alleviated. Taking this opportunity, we would like to further clarify the technical contributions of our work, which may not have been fully conveyed previously.

[Follow-up Question: Technical Contribution in This Study]

We respectfully argue that MoE-Retriever’s contribution is not incremental due to its unique characteristics, which we outline as follows:

1. Why MoE-Retriever Matters:

   • Compared to existing works that either do not consider or solely rely on intra-modal or inter-modal contexts during feature retrieval, MoE-Retriever incorporates both contexts simultaneously. Furthermore, while existing approaches typically rely on either a single learnable embedding or all embeddings for retrieval, our method focuses on leveraging only the relevant embeddings, where the Sparse Mixture-of-Experts (SMoE) design becomes essential.

2. How MoE-Retriever Works:

   • MoE-Retriever introduces a novel framework for effective imputation comprising the following three components:
     • Intra-Modal Context: Defines a supporting group that includes samples with observed modalities and a missing modality, serving as a sufficient condition to capture intra-modal similarities across samples.
     • Inter-Modal Context: Utilizes observed modalities from each sample to provide personalized semantic information unique to each sample.
     • Context-Aware Routing Policy: Leverages intra- and inter-modal inputs to dynamically determine the top-k missing modality-specific experts responsible for retrieving the most relevant embeddings for the missing modality.

3. What the Advanced MoE-Retriever Design Brings:
   Through fine-grained design exploration, we found that:

   • Using similarities as probabilities for multinomial sampling (rather than random sampling within the supporting group) did not improve performance, as detailed in [W2: Architecture - Incorporating Similarity Measure] (link).
   • Increasing complexity by introducing intra-modal-specific routers or increasing the number of experts and routers also failed to deliver significant performance gains, as discussed in [W3: Inter-Modal Context] (link) and in Table 3 of the main paper.
   • Replacing the fusion layer with an SMoE layer did not enhance performance.

   These results suggest that the current simple design space of MoE-Retriever is a “sweet spot”—balancing simplicity and effectiveness.

In summary, MoE-Retriever tackles imputation through the lens of SMoE by:

• Modeling both intra- and inter-modal contexts with a context-aware routing policy,
• Utilizing a simple yet effective design that outperforms more complex alternatives,
• Making meaningful contributions to the multi-modal learning field with its unique focus on imputation.

We hope this explanation further clarifies the technical contributions of our work, helping to positively influence reviewer L5fQ’s final evaluation.

Best,
Authors

Dear reviewer L5fQ,

We sincerely thank you once again for your insightful engagement and the constructive feedback you have shared during the review process. Your thorough comments and thoughtful follow-up questions have played a key role in enhancing the clarity and overall quality of our paper. We deeply value the time and effort you have devoted to carefully assessing our submission.

As the review period enters its final hours, we humbly ask if you might kindly consider revisiting your score once more. We deeply value your feedback regarding the technical contributions of our work and have made a concerted effort to address these concerns in our follow-up responses. In particular, as a pioneering use of SMoE for missing modality imputation, we clarified how MoE-Retriever uniquely leverages intra- and inter-modal contexts through a context-aware routing policy, distinguishing it from prior fusion-layer-focused works like FuseMoE.

Borrowing this last moment, given the balanced position of MoE-Retriever among the ratings, we kindly ask if you might consider offering your final support—such as a slight increase—which could meaningfully impact the outcome of this paper. If you find our responses and above revisions satisfactory, even a modest increase in your score would mean a great deal to us at this crucial situation. However, we fully understand and respect your judgment and sincerely appreciate the thoughtful consideration you have already provided.

All in all, we sincerely thank you again for your insights and constructive feedback, which have significantly enhanced our work during this rebuttal period.

Best,
Authors