FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion

Dear Reviewer sZSN,

Thank you very much for your constructive suggestions. We are encouraged by your acknowledgment that our idea is promising, the theoretical proof is rigorous, and the experimental design is comprehensive, containing a large amount of supplementary information. Below, we address your concerns in detail.

W1: We wish to emphasize that all the baselines we compared are well-known and representative methodologies within specific categories: concatenation, Tensor Fusion, Multimodal Adaptation Gate, and cross-attention-based methods. The MoE-based fusion method is a completely new line of work, and the comparison is essentially within the concept of fusion approaches.

To the best of our knowledge, based on the latest baselines we compared with, there is neither new work that opens up a new line of fusion methods nor any work that is far superior to the method and baselines we presented. However, in response to your comments and those of other reviewers, we have added additional experiments and ablation studies on new datasets and settings to further strengthen our results. These can be found in the attached PDF file.

W2: Figure 1 illustrates the example of multimodal electronic health records that possess irregularity and missing modality problems. This illustration reflects the situation of the MIMIC-IV dataset. Detailed task descriptions and data preprocessing workflows are discussed in Appendices A and D. We will add detailed explanations to Fig. 1 in the revised version.

W3: Softmax gating is more commonly used than Gaussian gating in real-world problems involving vision, text, and potential multimodal applications, and it also achieves relatively better performance. This is why we mainly compare our method with Softmax gating. Gaussian gating has been compared with Laplace gating in a different work, which illustrates that our proposed Laplace gating is more sample-efficient than Gaussian gating as shown in Nguyen et al. [1].

First, recall that the general form of the Gaussian gating is given by $h(x)=\mathrm{TopK(-(W-x)^{\top}\Gamma^{-1}(W-x)/2)},$ where $W$ is a mean vector, and $\Gamma$ is a covariance matrix. Then, Nguyen et. al. [1] consider two settings of the mean parameters $W^*_j$. Under the first setting (resp. second setting), Nguyen et. al. [1] argue that due to an interaction between the mean parameters $W^*_j$ and the covariance matrix $\Gamma^*_j$ (resp. the expert parameters $a^*_j$) via the partial differential equation (PDE) in Eq.(8) (resp. Eq.(11)) in [1], the rates for estimating mean parameters $W^*_j$ (resp. $a^*_j$) decrease when the number of their fitted atoms increases, and are no faster than $O(n^{-1/8})$. On the other hand, it can be verified that such parameter interactions do not occur under the FuseMoE. Thus, the estimation rates for the mean parameters $W^*_j$ and the expert parameters $a^*_j$ when using the Laplace gating remain unchanged of order $O(n^{-1/4})$. To strengthen our arguments, we also include a table where we summarize the parameter estimation rates under the Gaussian MoE when using the Laplace gating and the Gaussian gating.

Finally, we explain why the aforementioned PDEs affect the parameter estimation rates. In particular, a key step in our proof is to decompose the density difference $p_{G_n}(Y|X)-p_{G_*}(Y|X)$ into a combination of linearly independent terms using Taylor expansions. However, if those PDEs hold true, then components in that decomposition become linearly dependent, which is undesirable and leads to slow parameter estimation rates.

Summary of parameter estimation rates under the Gaussian MoE with the Gaussian gating and the Laplace gating. The function $\widetilde{r}(\cdot)$ is defined in Eq.(12) in [1], while the function $\bar{r}(\cdot)$ is defined in Eq.(9) in our paper. Note that $\widetilde{r}(\cdot)\leq\bar{r}(\cdot)$ and $\widetilde{r}(2)=4$, $\widetilde{r}(3)=6$. Additionally, $\mathcal{A}^n_j:=\mathcal{A}_j(\widehat{G}_n)$ is a Voronoi cell defined in Eq.(7) in our paper:

Limitations: We will revise our paper to avoid potential information overlaps with prior UTDE work. However, we wish to emphasize that the encoder part was not proposed by UTDE itself; the authors of that paper also utilized prior works (i.e., mTAND, Time2Vec) for demonstration purposes, similar to our approach.

Regarding the gating function and encoders, we appreciate your insights. Indeed, all the ablation studies on encoders presented in Appendix H.2 are based on the Laplace gating function of our FuseMoE framework. Choosing appropriate encoders is beyond the scope of this paper, which is why we defer this discussion to the Appendix. However, as the reviewer points out, it is beneficial to determine whether gating functions or encoders contribute more significantly to overall performance improvement. Therefore, we have added additional experimental results on mutating different gating functions in conjunction with different encoders and compared the differences in results. The figures can be found in the attached PDF file.

Reference

[1] Nguyen et al., (2024). Towards convergence rates for parameter estimation in Gaussian-gated mixture of experts.

Authors

Dear Reviewer 6SUH,

We are grateful for your positive feedback and insightful comments. It is particularly encouraging to hear that you found our manuscript to have strong motivation and address important challenges, with good theoretical analysis and comprehensive experimental results. In the following, we address your major concerns in detail.

Q: The author should clearly distinguish the proposed method and others’ modules. From the paper, the Laplace gating function is proposed as a new one. I am not sure the author made some contributions to the encoder design, router design, and loss design. The author should make more illustrations about the contributions, not just combine other people’s work together.

A: We respectfully disagree with your claim that we are merely combining existing literature. In our work, the MoE fusion layer and missing modalities sections are newly proposed and form important parts of the FuseMoE method. The theoretical insights of the Laplace gating function are another integral and unique aspect of this paper, leading to extensive empirical evaluations under multiple circumstances and applications. Most importantly, the new setting and motivation can potentially be connected to many real-world applications.

While the encoder models we employed are based on existing work, this is primarily for demonstration purposes and does not constitute a significant portion of the context. We would like to emphasize that this level of prior-work reuse has been common in many prior works, including our baselines from [1].

Nevertheless, our intention was not to combine other people’s work but to provide a clearer demonstration. We truly appreciate your advice in pointing out this issue and will ensure to clarify this in the revised version.

Q: The experimental results are not extensive. The author should clearly demonstrate the data modality of the chosen benchmarks. It seems that other modalities, such as text, are not included. The author should explain this.

A: We have created a table summarizing the modality components and sample size of each dataset in the attached PDF file. These details can also be found in the dataset details section in Appendix B. The datasets we tested include multiple modalities such as time series, text, image, ECG, and video frames.

Q: Please explain more concisely how the gating functions can stabilize the imbalance and sparse multi-modal data. The paper should point this out more concisely and better with experimental results.

A: We start with the computation of gating functions to demonstrate the advantage of Laplace gating. For each token $\mathbf{h}$, and each expert $i$ with embedding $\mathbf{e}_i$, the similarity score $s_i$ is computed as:

$s_i = \mathbf{h} \cdot \mathbf{e}_i,$

which represents the affinity between the token and the expert. These scores are then passed through a Softmax function to obtain the gating probabilities $g_i$:

$g_i = \frac{\exp(s_i)}{\sum_{j} \exp(s_j)},$

where the gating probabilities determine the weight or importance of each expert in contributing to the final decision. Representation collapse occurs when the Softmax gating mechanism consistently assigns high probabilities to a small subset of experts, causing these experts to dominate the decision-making process while the others become redundant. This issue arises due to:

• If a few experts have significantly higher similarity scores $s_i$ than others for most tokens, the Softmax function will amplify these differences, leading to very high gating probabilities for these experts.
• The exponential nature of the Softmax function makes it sensitive to differences in similarity scores, causing the highest scores to dominate.

As for the Laplace gating function, instead of using the inner product, it computes the similarity score $s_i$ as the negative L2-distance between a token’s hidden representation $\mathbf{h}$ and an expert embedding $\mathbf{e}_i$:

$s_i = -\| \mathbf{h} - \mathbf{e}_i \|_2,$

which is a distance-based similarity measure. The $L_2$-distance measures how far apart the token representation and expert embedding are in the feature space. This approach does not inherently favor any expert based on magnitude, unlike inner product which can be biased towards experts with larger norms. By considering the distance, the Laplace gating function ensures that all experts have a more balanced opportunity to be selected based on how close they are to the token representation, rather than being dominated by a few experts with higher dot products.

When dealing with heterogeneous inputs, such as multimodal data (e.g., text, images, time series), the feature distributions can be very different across modalities. The $L_2$-distance is a more robust measure that can handle these differences without being overly sensitive to the scale and variance of the input features. In addition, it can gracefully degrade in the presence of missing data, rather than causing abrupt changes in gating probabilities that might occur with inner product-based measures.

For experimental results on the Laplace gating function, we have added results on ImageNet using the Vision-MoE framework, which can be found in the attached PDF file. In our paper, we have already tested the Laplace gating function on MIMIC-III, MIMIC-IV, CIFAR-10, and PAM. Additionally, the results presented on the CMU-MOSI and CMU-MOSEI datasets also utilize the Laplace gating function.

Sincerely,

Authors

Reference

[1] Zhang et al., (2023) Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling, ICML 2023.

W1: Yes, we mentioned this as one of our limitations in line 316 of Section 5. This issue was observed during our empirical evaluation, where the Time2Vec method we employed transforms a univariate time series into a high-dimensional vector. This transformation helps capture trend/seasonality components and long-term dependency. However, when the number of samples is small and lacks sufficient information, transforming to a high-dimensional vector can be redundant and lead to over-parameterization.

This problem only occurred in a few scenarios among the numerous experiments we conducted. As mentioned in Section 5, we plan to address this issue in future work.

W2: We have included a discussion comparing the computational efficiency of various methods in Figure 12 of Section H.3. Additionally, we have described the computational resources used in Section G.1.

We have also tested FuseMoE on large-scale multimodal datasets, including MIMIC-III, MIMIC-IV, and CMU-MOSEI. As shown in Table 1 of the MultiBench paper [1], these datasets are all featured as large-scale multimodal datasets, with MIMIC-III containing 36,212 samples, MIMIC-IV containing 73,173 samples, and CMU-MOSEI containing 22,777 samples. Even though CMU-MOSI is relatively small-scale with 2,199 samples, our experiments have overall tested a sufficient number of large-scale multimodal datasets compared to most prior works (e.g. [2]). These datasets span various data types, including text, images, time series, ECG, video frames, and audio signals, demonstrating strong generalization capacity across domains.

Please find tables summarizing the information of these datasets in the attached PDF file.

W3: We respectfully disagree that we are merely combining existing literature. In our work, the MoE fusion layer and missing modalities sections are newly proposed and form important parts of FuseMoE. The theoretical insights of Laplace gating are another integral aspect of this paper, leading to extensive empirical evaluations under multiple circumstances. Most importantly, the new setting and motivation can potentially be connected to many real-world applications.

While the encoder models we employed are based on existing work, this is primarily for demonstration purposes and does not constitute a significant portion of the context. We want to emphasize that this level of prior-work reuse has been common in many prior works, including our baselines from [2]. We truly appreciate your advice in pointing out this issue and will ensure to clarify this in the revised version.

Q1: We wish to first emphasize that “unlimited” does not mean infinite. According to [3], most real-world multimodal problems do not exceed four modalities. FuseMoE can scale to recently introduced high-modality problems, which combine modalities from different tasks. The scalability advantage is particularly prominent when compared with the pair-wise cross-attention-based approach.

FuseMoE is designed to address real-life multimodal datasets with missing modality or irregularity issues, which can be caused by malfunctioning equipment, different patient conditions, human-level decisions, etc. In principle, the current method can be applied to arbitrary sparsity or modality combinations, as we have only tested on available real-world multimodal datasets. However, we did not manually create sparse data to meet the potential “extremely sparse” criteria. If such situations occur, we can employ additional strategies to enhance the robustness of the current framework, such as leveraging hierarchical MoE where a specific subgroup of experts is assigned to handle missing modalities, and applying regularization or dropout during training to make the model robust to missing modalities.

Given that the model architecture remains the same, as the number of modalities N increases, the FuseMoE-based method scales linearly in $\mathcal{O}(N)$, whereas the cross-attention method scales in $\mathcal{O}(N^2)$.

Q2: We have added additional results on ImageNet using the Vision-MoE framework for the Laplace gating, this can be found in the attached PDF file. In our paper, we have also tested the Laplace gating on MIMIC-III, MIMIC-IV, and PAM. Additionally, the results presented on the CMU-MOSI and CMU-MOSEI datasets utilize the Laplace gating function.

Q3: Yes, please see Figure 12 (c) in Appendix H, which visualizes the modality contribution of the MIMIC dataset. Additionally, we have included the modality contributions of the CMU-MOSI and CMU-MOSEI datasets in the attached PDF file.

Q4: There are three main differences between our FuseMoE and the Laplace MoE considered in the referenced papers:

1. Conditional Distributions: Under the FuseMoE model, the conditional distribution of $Y | X$, with the density given in Equation (4), where $X$ is an input and $Y$ is an output, is a mixture of Gaussian distributions. In contrast, the Laplace MoE model utilizes a mixture of Laplace distributions.

2. Sparse MoE versus Dense MoE: In FuseMoE, only a subset of Gaussian distributions is activated for each input $X$. Specifically, the conditional distribution of $Y | X$ is a mixture of K Gaussian distributions, where K is often set to one or two in practice. This mechanism is enabled by the Top K operator described below line 154. Meanwhile, in the Laplace MoE model, all Laplace distributions are activated for each input.

3. Gating Kernels: In FuseMoE, the mixture weights in Equation (4) use the Laplace kernel with the scale parameter set to one. In contrast, the gating kernel in the Laplace MoE model is an exponential value of a linear kernel.

References

[1] MULTIBENCH: Multiscale Benchmarks for Multimodal Representation Learning

[2] Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling

[3] High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

Dear Reviewer P6x6,

We deeply appreciate your insightful comments and positive feedback. We are heartened by your recognition that our paper is well-written, with well-elaborated motivations. We are also pleased that you acknowledge the theory and extensive experiments we provided to justify the effectiveness of our method. Below, we address your questions and concerns in detail.

Q: The proposed method achieves promising results on the tested benchmark, which are still comparably small scale datasets, e.g, CMU-MOSI is with 2000+ samples. I am wondering if this work can be applicable to training/fine-tuning with larger scale data.

A: Yes, we have tested FuseMoE on large-scale multimodal datasets including MIMIC-III, MIMIC-IV, and CMU-MOSEI. As shown in Table 1 of the MultiBench paper [1], these datasets are all featured as large-scale multimodal datasets, with MIMIC-III containing 36,212 samples, MIMIC-IV containing 73,173 samples, and CMU-MOSEI containing 22,777 samples. Even though CMU-MOSI is relatively small-scale with 2,199 samples, our experiments overall have tested a sufficient number of large-scale multimodal datasets. These datasets span various data types including text, images, time series, ECG, video frames, and audio signals, demonstrating strong generalization capacity across different domains.

Please find tables summarizing the information of these datasets in the attached PDF file.

Q: Again, for vision task, only CIFAR-10 is used. This cannot fully justify its effectiveness for more general vision or multi-modal tasks

A: As explained above, several of the multimodal datasets we tested include vision components, such as chest X-rays in MIMIC and video frames in CMU-MOSI and MOSEI. Additionally, we have conducted further evaluation of the ImageNet dataset. Detailed information can be found in the attached PDF document.

Q: The capability to handle missing modalities is promising. The author is encouraged to explain why, in some scenarios, missing modalities + the proposed method surpasses the variant with full modalities (Fig 4 c). I am still wondering about its generalizability under more scenarios.

A: Full modalities mean that patient records with incomplete modalities will be discarded. In contrast, incorporating data with missing modalities allows us to access a broader array of samples, resulting in the utilization of all available patient data. As mentioned in Table 6 of Appendix B.1, for instance, when examining the 48-IHM or LOS tasks in the MIMIC dataset, the total number of patient samples is 35,129. However, the number of patients with clinical notes is only 32,038, and with chest X-ray (CXR) only 8,731. If we strictly enforce the inclusion of complete modalities involving CXR, the total number of samples available is constrained by this bottleneck.

Therefore, being able to utilize incomplete modalities is a critical step in making use of the vast amount of patient data without full modalities. This approach will greatly improve generalization due to the increased number of available samples.

Q: The visualization and qualitative analysis of the learned gating weight would help the reader better understand the method.

A: We have included the visualization and qualitative analysis of the learned gating weights in Figure 12 of Appendix I. To better address your request, we have also added additional results on the CMU-MOSI and MOSEI datasets in the attached PDF file.

Q: In table 5, Laplace gating shows limited superiority against other variants, any insight for it?

A: In the results of Table 5, Laplace gating is still close to the best-performing method. We wish to emphasize that the advantage shown in Table 4 comes from the MoE architecture improvement. Most fusion methods listed in Table 4 are only suited to two-modality problems and cannot be extended as we scale to more modalities in Table 5. The comparison between Softmax, Gaussian, and Laplace gating functions in Table 5 reflects improvements within the same MoE architecture.

These two types of improvements are distinct: sometimes one factor (e.g., the MoE architecture) is more important than the other (e.g., the gating function). Additionally, the performance advantage of each method also depends on the modality combination. Adding or removing modalities can influence the effectiveness of the Laplace gating mechanism.

Q: In table 4c, why the method achieves better results with less modalities? Any further insight or ablations?

A: Based on the context, we believe you mean Figure 4c instead of Table 4c. We would like to refer you to our answer to your first question for more details on this matter.

Please let us know if there are specific aspects or sections you would like us to elaborate on further, or if you have any additional questions. Thank you!

Sincerely,

Authors

Reference

[1] Liang et al., (2021) MULTIBENCH: Multiscale Benchmarks for Multimodal Representation Learning, NeurIPS 2021.