Sparse MoE as a New Treatment: Addressing Forgetting, Fitting, Learning Issues in Multi-Modal Multi-Task Learning

[Cons 1. The Novelty of The Work is Limited?]

We respectfully disagree. We point out its novelty richness from three distinct aspects:

1. [SMoA Designs] Our approach addresses the modality forgetting problem through the novel SM$^4$ structure, comprising SMoE and SMoA modules governed by modality-specific routing policies. Note that the load and importance balancing loss is not designed for and can not solve the modality forgetting problem.
2. [Automatic Expert Allocation] We emphasize that AEA is NOT a standard hyperparameter tuning. While hyperparameters rely on human intervention, our approach automatically decides the expert number based on the model training dynamics. Therefore, in our case, the expert number is not a hyperparameter.
3. [Automatic Learning Pace for Different Modality] The ALP automates the learning pace of each modality in a dynamic fashion, which is also not the standard hyperparameter tuning technique.

[Cons 2. The Connection of The Three Problems is Not Organic?]

We respectfully argue that these facets are intrinsically linked. Our proposal seeks to tackle the optimization problem in multi-modal multi-task learning, wherein the issues of gradient direction, optimization step size, and model capacity represent a comprehensive view of this overarching challenge. Our proposed SM$^4$ structure, featuring SMoA and SMoE modules alongside tailored training techniques, serves as a unified solution to alleviate these intertwined optimization challenges of multi-modal multi-task learning.

[Cons 3. Confused About The Results in 5. What Does 32N Mean?]

For the clarification of “32N”, we've revised the notation to N=4, N=8, N=16, and N=32. The optimal number of experts in Mixture-of-Experts (MoEs) remains a subject of debate, with conflicting views on the ideal quantity. While several studies advocate that increased experts enhance performance, an excessive number may introduce redundancy or noise. Table A3 of [1] demonstrates that increasing the expert number brings extra performance will gradually diminish as the expert number goes up, and Table 11 in [2] shows the large expert number may decrease performance. Therefore, we investigate this further for multi-modal multi-task learning, and our findings suggest that larger expert pools consistently yield performance improvements.

[1] Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners

[2] TUTEL: Adaptive Mixture-of-Experts at Scaler

Summary

Thank you for acknowledging the comprehensive of our experiments. Below are our responses addressing your concerns:

[Cons 1. AEA Introduces an Additional Hyper-Parameter, Additional Pre-training Phase?]

The AEA is not working on a pre-train phase; it is executed during our multi-modal multi-task learning, which takes into account multi-task learning, and the AEA stopped tuning expert numbers by itself, not a hyper-parameter. The “n” in Algorithm 2 is actually the number of iterations within a single multi-modal multi-task training epoch. Therefore, we do not include additional hyper-parameters. As we mentioned in Section 3.3, the load and importance balancing loss we used in routing networks make sure most experts will be utilized. Moreover, as shown in T.1 our method doesn't increase training time significantly. The above concerns have been included in our revision.

T.1 SM$^4$ and HighMMT multi-modal multi-task training time.

[Cons 2. Why Larger Training-Validation Loss Gap Can Lead to Better Generalization?]

As mentioned in [1], the ``generalization gap’’ is defined as the difference between a model’s performance on training data and its performance on unseen data drawn from the same distribution (e.g., the performance difference between training set and validation set). We adopt loss value to measure model performance and define the gap as the difference between training and validation loss. A larger generalization gap indicates that the model is not overfitted on the training set (i.e., higher training loss) but predicts well on the unseen data (i.e., lower validation loss), which indicates better generalization. We've revised the definition of the "generalization gap" to align with formal definitions, clarifying the significance of a higher training-valid loss gap indicating better generalization performance.

[Cons 3. What does "learning pace" Mean in This Context in ALP? ]

The learning pace in ALP refers to the learning rate (or the optimization step size) of different modalities.

[Cons 3. What is The "setting" in Table 1, 2 Refers to?]

The “setting” here refers to the number of tasks that follow the HighMMT.

[Cons 4. Model Parameter Number of Medium Setting is More Than The Large Setting.]

Although the large setting contains more tasks, more task number does not mean more parameters. As shown in Table 14 and Table 15 of our paper, the total number of experts in the medium setting is $32$, and we use $16$ experts in the large setting. Therefore, the number of model parameters in the medium setting is larger than in the large setting.

[Cons 5. The Results of HighMMT Seem to be Quite Different From The Origine.]

Our implementation of HighMMT adheres to their official repo [HighMMT] (https://github.com/pliang279/HighMMT) and the default settings outlined in the HighMMT paper. For instance, our utilization of learning rates is as follows: 0.0005 for the small setting, 0.001 for the medium setting, and 0.0008 for the large setting. Furthermore, our supplementary materials contain the HighMMT official code and reproduction scripts, ensuring the exact replication of all hyper-parameters specified in the HighMMT paper. If Reviewer cbhp could kindly point out another better re-implementation of HighMMT, we would like to follow it and update the configurations.

[Cons 6. Expression Suggestion.]

We've made adjustments according to your invaluable suggestions, including improvements to Figure 2 for clarity.

[1] Predicting the Generalization Gap in Deep Networks with Margin Distributions.

Dear authors,

In Algorithm 2, there is a step that state $loss\\_val\\_i = train(model)$ with a comment training model for 1 epoch. So does this step reuse the losses in the pseudo-code that you outlined or it requires re-training the model for 1 epoch?

If we directly plug Algorithm 2 in the overall_training pseudo-code, I guess there might be a lot of nested training loops in the early stage because each AEA step also requires measuring the validation loss of each modality.

Algorithm 2 also instructs to continue to train the model for the remaining epochs (2nd last line). So I think it needs to be revised to integrate to the overall_training pseudo-code.

Lastly, this is a conceptual questions. Since the authors stated that the expert allocation to a modality is kept until end of training when the flag improved is set to true. This suggests that early stages of training will mostly perform a search to allocate experts to modalities and this configuration will remain fixed. Is it an optimal strategy? Because in early epochs, the experts may not learn well enough and maybe it is beneficial to "reallocate" the experts once they have learned good representations. One naive strategy could be perform a reallocation (re-run AEA) after some fixed epochs.

Does this step reuse the losses "loss_val_i" in the pseudo-code?

Yes, this step reuses the losses in the pseudo-code.

Revise the Algorithm 2 Thanks to the suggestion, we revised the Algorithm for integrating the overall_training pseudo-code and modified overall_training pseudo-code below:

def adaptive_expert_allocation(model, loss):
    for modality in modality_set:
	    # If the modality is signed improved, skip this modality
	    if check(modality):
		    continue
		n_experts = modality_topk[modality]
        loss_val = loss[valid][modality]
		# if the expert number of this modality is increased last time
		if increase_expert(modality):
			if loss_decrease(loss_val):
				improved = True
		else:
			# if the valid loss does not decrease
			if loss_decrease(loss_val): 
				if not improved:
					Sign this modality as improved
					modality_topk[modality] = n_experts - 1
				else:
					modality_topk[modality] = n_experts + 1
					improved = False
    return modality_topk

def overall_training(modality_topk, modality_weights):
	for i in range(max_epochs):
		# training 1 epoch
		# losses include the average valid loss of all modalities in this epoch
		# routing_entropy includes the average entropy of all modalities in this epoch
		val_losses, routing_entropy = train(model, modality_topk, modality_weights)
		
		# setting topk of all modalities
		# if the monitoring of specific modality is ended, AEA will skip to tuning the topk of specific modality
		modality_topk = AEA(model, val_losses)
		
		modality_weights = ALP(model, routing_entropy)

Is it an optimal strategy?

This problem needs further investigation. Reallocation is a potential solution to further improve the performance of SM$^4$. The problem is this will involve additional hyper-parameters to decide when to reallocate experts which makes the training more complex. However, we think this is a remarkably interesting and important problem, how to decide the optimal model capacity of each modality during training, and expect to investigate this further in our future work.

Thanks for your insightful comments!

T6. Performance, parameter usage, and FLOP of our model, and HighMMT in the large setting.

T3. Detailed performance, parameter usage, and FLOPS of multi-modal learning method on MultiBench benchmark. For the ''FLOPS(G)'', ''-'' indicates the MultiBench does not provide official implementation. Notably, the empty FLOPS of the MultiBench Model (MFAS)'' is due to the FLOPS of ''MFAS'' being dynamic during training. For each dataset, we choose multi-modal models with the best/worst performance and multi-modal models with the largest/smallest parameter numbers, respectively.

T4. Performance, parameter usage, and FLOP of our model, and HighMMT in the small setting.

T5. Performance, parameter usage, and FLOP of our model, and HighMMT in the medium setting.

Summary

We are glad that reviewer mdyv appreciates the comments on our writing as “impressive”, “informative”, and “easy to follow”, our proposal as “well-motivated and addresses important issues”, and our experiments as “detailed” and “solid”. To address reviewer mdyv’s questions, we provide pointwise responses below.

[Cons 1. Without Checking Robustness.]

We conduct extra experiments to examine the robustness of SM$^4$ to missing modalities. Due to the limited time in the rebuttal period, we will add the robustness evaluation of noisy modalities in our final version. As shown in Table T1, we assess SM$^4$ and HighMMT with the model UR-FUNNY under three missing scenarios of missing text, video, and audio, respectively. Our results imply that SM$^4$ has a relatively better robustness towards missing modalities.

T.1 The robustness comparison between SM$^4$ and highMMT. We show the $\delta$ value, which is defined as the value of performance drop when missing one modality. A smaller $\delta$ value indicates better robustness against the modality missing.

[Cons 2. The Trade-off Between Training Cost and Model Performance of SM^4 is Unclear.]

Thanks for the great point.

• The total number of training epochs for SM$^4$ and other baselines are the same. The procedure of deciding the number of experts for each modality does NOT require extra training epochs. Specifically, both HighMMT and SM$^4$ use $100, 100, 100$ training epochs for the small, medium, and large settings, respectively.
• To further convince review mdyv, we measure the training time of SM$^4$ and HighMMT in the same device (i.e., single NVIDIA A30 GPU). Our results demonstrate that SM$^4$ uses similar training costs to reach consistently enhanced performance, as evidenced in Table T.2. It indicates an improved trade-off between training cost and model performance.

T.2 SM$^4$ and HighMMT multi-modal multi-task training time.

[Cons 3. The Reported Performance of MultiBench Models in Table 2 Maybe Overly Simplified.]

To address the concern regarding the detailed performance of MultiBench models, we included a more comprehensive performance report below (Table T.3, T.4, T.5, and T.6 below) and in our revision (Table 10 (page 22) in our revision).

Also, the typo in Table 1 has been corrected in our revision.

[Summary]

Acknowledgments to Reviewer m8oX for recognizing our work as "impressive" and appreciating the clarity of our Figure 2, outlining its directness. We value the constructive feedback provided, which is instrumental in refining our paper. To address reviewer m8oX's queries, we present specific responses below:

[Cons 1. What are Exactly the Experts, and Where do They Come From? Are They Grouped? .]

The experts are duplicated multi-layer perceptrons (MLP). Specifically, in SMoE, the experts are duplicated from the original feedforward networks. The experts in SMoA are duplicated from the query, key, and value MLPs, respectfully.

In order to distinguish experts from different sources, we name them as different expert groups. Experts are grouped by the nature of where they are duplicated from.

The above clarifications have been included in our revisions.

[Cons 2. Motivation of ALP. How do the Policy Changes Link to the Routing Distribution Entropy?]

1. What does an unstable policy mean?

An unstable routing policy means the changes in the policy across iterations.

2. How do such changes link to routing distribution entropy?

They are linked. We measure the routing distribution of each expert across the iterations and then calculate an averaged entropy across all the experts. Therefore, the entropy here directly measures the stability of the routing policy over the training iterations. For example, a high entropy indicates that expert routing is quite converged and stable.

3. Motivation of ALP.

The motivation of adaptive learning pace (ALP) is to align different learning paces between modalities, aiming to synchronize the optimization of multiple objectives. To be specific, from Figure 2(d), the stability of modality-specific routing seems to reveal the convergence of modality-specific training, serving as a good guidance to decay its learning rate. The above clarifications have been included in our revisions.

[Cons 3. Performance Difference of HighMMT.]

Thanks for pointing it out.

• We highlight that our performance of HighMMT is produced with the official implementation from HighMMT’s repository (https://github.com/pliang279/HighMMT). We strictly follow the default configurations reported in their paper, as shown in Tables 8, 9, and 10. For example, we use learning rates of 0.0005, 0.001, and 0.0008 for the small, medium, and large settings, respectively.
• Additionally, our supplementary materials have included the HighMMT code and reproduction scripts, maintaining exact replication of their hyper-parameter settings.
• If Reviewer m8oX could kindly point out another better re-implementation of HighMMT, we would like to follow it and update the configurations.

[Cons 4. Datasets/Tasks Selection.]

The selection of our three multi-modal task groups follows HighMMT’s standards.

• The small setting encompasses similar research areas with varying modality inputs;
• The medium setting spans three domains featuring different modalities;
• The large setting involves three domains incorporating diverse modalities.

Yes, our proposed method is a general multi-modal, multi-task pipeline, which can be leveraged for various combinations of tasks and modalities. In our submission, we follow the standard multi-modal multi-task settings from the HighMMT paper. That is, HighMMT/Multibench currently only provides the default hyper-parameter configurations for the small, medium, and large settings, which are adopted in our paper as the representative baseline. In our future works, we will examine our methods for extra combinations of tasks and modalities.