CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

Q1: Why MoE in the vision encoder and connector enhances the model's capabilities

A1: MoE has been widely used in LLMs to improve the capacity [1] of text generation as it improves the model size during training while keeping inference costs lower at inference. In our work, we apply MoE in the vision encoder and connectors of multimodal LLMs, which has the potential to generate versatile visual tokens and further improve the model capabilities on many vision-language instruction-following tasks. We verify our assumptions with detailed ablation studies on the effectiveness of the proposed MLP-MoE and CLIP-MoE.

[1] Shard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Q2: Whether merely increasing the parameters of the vision encoder alone will also enhance the model's capabilities for comparisons

A2: It depends. We compare the CLIP-ViT-L encoder with the other two larger vision encoders: CLIP-ViT-H and SigLIP-SO400M. CLIP-ViT-H has 8x more parameters than CLIP-ViT-L while performing much worse due to low-resolution inputs. SigLIP-SO400M has 0.13B more parameters and performs consistently better than CLIP-ViT-L. These findings have been also explored in recent multimodal LLMs [2,3] that the model size is not the top factor that affects the overall performance of multimodal LLMs.

[2] MM1: Methods, Analysis, and Insights from Multimodal LLM Pre-training

[3] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

We further apply CLIP-MoE and MLP-MoE to the stronger SigLIP-SO400M vision encoder and they can still make improvements upon a larger and more powerful vision encoder.

Q3: CuMo's LLM is mainly limited to Mixtral-8x7B, and the author should include comparisons with other LLM backbones.

A3: CuMo is mainly focusing on Mistral-7B and Mixtral-8x7B. Our ablations are based on Mistral-7B and verified the effectiveness of CLIP-MoE and MLP-MoE separately. We further evaluate CuMo on Vicuna-7B-v1.5 as shown in the table above with added CLIP-MoE and MLP-MoE to show their effectiveness.

Q4: Since CuMo is three-stage training, the increased amount of training data may lead to an unfair comparison with other baselines, such as LLaVA1.5.

A4: In Table 2, we maintain a fair comparison to LLaVA1.5 under the same training data and CuMo-7B consistently performs better than LLaVA1.5, either under Vicuna7B or Mistral-7B. In Table 1, we compare CuMo with models that use various datasets even with private datasets like LLaVA-NeXT and MM1, while CuMo is trained under fully open-sourced datasets. We may remove LLaVA1.5 from Table 1 and keep LLaVA-NeXT for comparison in the updated version to avoid confusion.

Q1: Computational efficiency and training time of MoE.

A1: We provided the breakdown of additional parameters during training in Table 6 and here we further include the training time of CuMo-Mistral-7B with MLP-MoE and CLIP-MoE. We used a single 8xA100 machine and LLaVA-665K data for reference here. Note that we only used the model parallelism in deepspeed for implementation while the training can be faster if expert parallelism is further added.

Q2: Lack of discussion on the interpretability of MoE decisions within CuMo.

A2: Following Section 5 of Mixtral-8x7B[1], we did the expert distribution analysis in Figure 4, which shows that the experts are loaded pretty balanced overall across the layers. We further used the subsets of images in MME by topic and found that they show a preference towards certain experts in some layers, which may imply some hidden patterns of the assignment of experts based on the applications or topics. We may update these results in Section 4.4 as parts of the analysis.

[1] Mixtral of Experts, https://arxiv.org/abs/2401.04088

Q3: The paper shows that Mixtral-8×7B's CuMo does not consistently outperform Mini-Gemini on several datasets, particularly on TextVQA and MME.

A3: CuMo-Mixtral-8x7B is better than Mini-Gemini-Mixtral-8x7B on MME (+0.5), MMMU(+3.2), and MM-Vet (+2.9), while worse on TextVQA (-3.2) and MMBench (-0.3), as shown in Table 1. One main reason behind this is that Mini-Gemini takes high-res inputs and benchmarks like TextVQA are sensitive to input resolution as shown in the Table 2 & 3 in Mini-Gemini.

Q4: The experiments primarily focus on Mistral or Mixtral-8×7B architectures with MoE integration. The author should conduct a comparative study with non-MoE architectures like Vicuna or Llama3, focusing solely on CuMo's projection MoE and Vision Encoder MoE.

A4: Our ablation studies are mainly on the Mistral-7B, which is a non-MoE LLM to verify the effectiveness of MLP-MoE and CLIP-MoE. We further evaluate CuMo on Vicuna-7B-v1.5 to make comparisons as shown in the table above under the same LLaVA-665K training data. The CLIP-MoE and MLP-MoE also make improvements over the Vicuna-7B-v1.5.

Q5: Table 1 highlights are misleading.

A5: Thanks for the suggestions. We'll revise that and highlight the best performance across models in each session of Table 1.

Q1: Limit on the gains from upcycling and when training MoE-based ViTs or MoE connectors from scratch will be much better / potentially start outperforming the dense-only case?

A1: The conclusion of using 20% additional capacity to catch up the upcycled model in original sparse upcycling and V-MoE is based on a much more intensive computation and data budget for pre-training LLM. However, in the CuMo setup, our motivation for using upcycling is to stabilize the training of the MoE module under a small data and compute budget, because training from scratch of connectors or ViTs is not comparable to the dense models due to the training instabilities as shown in Table 3(a). To estimate the gains of upcycling compared to training from scratch, we may need to train a CLIP-MoE from scratch, which is out of our training budget and beyond the scope of our work.

Q2: Diminishing gains as the base model size increases and how this will apply to the CuMo setup? If we scale the CLIP-ViT model or use a much stronger model like SigLIP-ViT, will the gains from CuMo still hold?

A2: We think the diminishing gains also exist in the CuMo setup if we use larger and stronger pre-trained CLIP with MoE while keeping training data unchanged. Here we use the pre-trained SigLIP-SO400M as the vision encoder and add MoE to it as shown in the table above. SigLIP-SO400M has a much better performance on ImageNet zero-shot classification than CLIP-ViT-L (83.2 vs 76.6). The added MoE can still make improvements to this stronger vision encoder but the average improvement shrinks compared to CLIP-ViT-L. However, the training data here is limited to LLaVA-665K for quick verification, which may not show the full potential of the model if training with more data.

Q3: Suggestions regarding Table 1, 2, captions, Figure 6 and Appendix.

A3: Thanks for the suggestions. We'll update Table 1 by highlighting the best performance numbers in each section and Table 2 by highlighting QwenVL's TextVQA number, as well as the table captions to make them clear to the audience. For Figure 6, the responses are generated by CuMo-Mistral-7B. For the Appendix, the 'M3 model' refers to the CuMo-Mistral-7B, we'll revise it in the updated version as well.

Q4: Non-society license, release, and usage guidelines.

A4: We plan to release the code under Apache 2.0 and the weights of all CuMo checkpoints under CC BY-NC 4.0 for non-commercial use. All the datasets and pre-trained weights we used for training and evaluation are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses.

Q5: Limitations.

A5: Thanks for the suggestions. We'll add discussions of RAG and the potential for biased outputs in the limitation section.