MoVA: Adapting Mixture of Vision Experts to Multimodal Context

Dear Reviewer FVeU,

Thanks for appreciating our work and your advice. We will address your concerns below.

Q1: The number of components in line 102 does not match the serial numbers.

There are four components in MoVA. We will fix this typo in the revised version.

Q2: The process involves two rounds of LLM inference and two Base Encoders, which results in significant inference time and GPU memory usage.

1. As described in the general response, MoVA only runs the base encoder once during inference since its base vision feature can be reused.
2. Please refer to the Q3 of our global response.

Therefore, we think the inference cost is not the bottleneck for MoVA.

Q3: The method seems similar to the adapter+agent approach described by Wang et al [1].

Both MoVA and ModaVerse are MLLM frameworks but they are different:

Advantages:

1. ModaVerse uses a single encoder to encode inputs of various modalities. However, MoVA can dynamically select the most appropriate vision experts that are relevant to current inputs and tasks.
2. ModaVerse adopts a set of linear projection layers to align multimodal inputs and the LLM. In our paper, we propose the MoV-Adapter as the projector and demonstrate it can attain better performance than conventional linear layers.

[1] ModaVerse: Efficiently Transforming Modalities with LLMs. CVPR 2024.

Q4: Why is the last row of MoVA results in Table 1 different from those in Table 3? Do all models in Table 1 use LLaVA-1.5 data? If so, I would like to ask if you can release the checkpoints of these models.

1. The MoVA model in Table 3 uses more training data than the models in Table 1. The detailed data settings of Table 3 MoVA model are presented in the line 210.
2. As described in the line 229, we additionally incorporate several datasets (DocVQA, ChartQA, RefCOCO, LLaVA-Med, VQA-RAD, and SLAKE) to the LLaVA-1.5 665k SFT data to train the models in Table 1.
3. We will definitely release our codes and models to facilitate the community if the paper will be accepted.

Dear reviewer gqRq,

Thanks for your comments. We will address your concerns below.

Q1: Is there an adapter structure in MoVA? Why do we call it MoV-Adapter?

Similar to conventional MLLMs, MoVA comprises a pretrained LLM, a base vision encoder, multiple vision experts, and a projector that connects the LLM and vision encoders. We call this projector "MoV-Adapter" since it also can adapt output features of various activated vision experts to the base vision encoder feature space by elaborate expert knowledge extractor.

Q2: The comparison is not fair with different SFT data.

Please refer to the Q2 of our global response.

Q3: More vision encoder enhancement methods should be compared using the same SFT data.

We compare our method with VCoder [1], MoF [2], and MouSi [3] under the same SFT data and LLM (Vicuna-v1.5-7B) settings. We directly follow their corresponding evaluation benchmarks.

Note that the authors of BRAVE [4] did not publicly release the codes, models, and data. Due to the limited time of the rebuttal period, it is hard to reproduce its performance during rebuttal. We will add this comparison in the revised manuscript.

[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.

[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)

[3] MouSi: Poly-Visual-Expert Vision-Language Models

[4] BRAVE: Broadening the visual encoding of vision-language models

Q4: Using LLM as a router will decrease the inference speed. How to solve this problem?

As highlighted by the general response, the expert routing only brings marginal inference costs thanks to our elaborate designs:

1. We use 144 compressed image tokens for expert routing. Note that the current state-of-the-art baseline LLaVA-NeXT employs the LLM to process 2880 image tokens.
2. Only the vision feature of the base vision encoder is used for expert routing.
3. During expert routing, we prompt the LLM to directly generate the option's letter of selected vision experts from the given model pool (e.g., "A" for DINOv2). As presented in the general response, the routing answer is short and the average length of the routing output token sequence is only 3.24.

Q5: In line 102, I just found four components, but the paper claims five.

This is a typo and it should be "comprises four key components". We will fix this typo in the revised version.

Thanks for your response. I found the results you provided in the rebuttal period, including VCoder-7B, MoF-7B, MouSi-7B were the same as the results in their paper. I strongly doubt that the authors did not use the same SFT data for a fair fine-tuning. In our paper lines 230-232, the authors mention that they use additional SFT data. Please explain this, thanks!

For Q4, I think comparing with LLava-Next is not appropriate, it is better to show the inference speed of MoE structure methods in [1-5].

[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.

[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)

[3] MouSi: Poly-Visual-Expert Vision-Language Models

[4] BRAVE: Broadening the visual encoding of vision-language models

[5] MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

Furthermore, after reading the reviews from Lntx, FVeU. I have the same question about the paper's novelty. About the general response Q1(1,2), the typical MoE methods have the expert selection mechanism, i.e. choosing K from N experts (e.g. Switch Transformer). For Q1(3), it has been studied in Mousi [3] and is not the main contribution of this paper.

Thanks for your follow-up questions. I'm delighted to address your concerns.

Q1: SFT data

We use the same SFT data as the original VCoder [1], MoF [2], and MouSi [3] implementations. Therefore, we directly report their performance scores in the paper.

1. VCoder. We employ the same COST dataset as the VCoder-7B to train the MoVA-7B model.

2. MoF and MouSi. Both of them use LLaVA-665k as the SFT data. To be specific, we institute the SFT data described in the line 210 for LLaVA-665k in our experiments.

3. The SFT data mentioned in the line 230 is only used for the model training in Table 1.

[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.

[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)

[3] MouSi: Poly-Visual-Expert Vision-Language Models

Q2: Inference comparison

We perform inference comparisons among MoVA, VCoder, MoF, and MoE-LLaVA [5] since the codes of both MouSi and BRAVE [4] are still publicly unavailable.

For a fair comparison, we adopt the same number of output tokens (400 tokens). The benchmark setting is identical to the aforementioned benchmarking in the rebuttal.

The results reveal that the latency values of VCoder, MoF, and MoVA exhibit negligible differences. Moreover, MoE-LLaVA runs much slower than other frameworks even though it is equipped with a smaller LLM Phi2-2.7B.

[4] BRAVE: Broadening the visual encoding of vision-language models

[5] MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

Q3: The typical MoE methods have the expert selection mechanism

Selecting $K$ experts from $N$ experts is a common practice of the mixture of experts (MoE) and it is not our contribution. Our paper focuses on how to select $K$ vision experts based on the current task in multimodal scenarios. Specifically, we explore how to leverage the reasoning ability of LLM and the prior knowledge of vision encoders to flexibly select the most $K$ appropriate vision experts. This idea has not been explored in the current mulitmodal community.

Q4: Comparison with MouSi

We compare our method with MouSi in the table:

We will discuss and cite these works [1-4] in our revised paper.

[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.

[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)

[3] MouSi: Poly-Visual-Expert Vision-Language Models

[4] BRAVE: Broadening the visual encoding of vision-language models

Thanks for your response. I wonder why expert selection (0.14s) in MoVA is so fast. I think the inference time for LLM should be longer.

Thanks for your question. We will address your concern below.

We show the inference latency of each step in Q3 of our global response.

The final response generation of LLM (Step 5) takes $\frac{10.24}{405.06}=0.0253$ seconds to generate a token, and the output speed of the LLM routing (Step 3) is $\frac{0.14}{3.24}=0.0432$ s/token. Considering we adopt different prompt templates and image features for routing and final response generation, such speed value difference is reasonable. Specifically, we use the prompt template in Table 2 of our paper for expert routing and keep the same Vicuna template for response generation.

Therefore, the expert routing is so fast since MoVA only generates a few tokens (the average length is 3.24 tokens in our experiments) for routing. The routing designs described in the Q4 of the rebuttal for reviewer gqRq also decrease the inference latency.

Dear Reviewer dPiw,

Thanks for giving so many constructive suggestions for our paper, I will clarify the settings.

Q1: The rationality of using task-specific vision models.

1. We have conducted extensive experiments to benchmark different vision encoders in Table 1. We find a single CLIP encoder cannot generalize to various image domains, such as text recognition tasks.
2. Besides, a task-specific model can achieve optimal performance in its proficient task. It is reasonable to incorporate task-specific vision experts to improve performances on their relevant tasks. We also find the "plain fusion" of multiple vision experts can bring performance gains to the baseline with a single CLIP encoder in Table 1.
3. Many previous works [1-5] have demonstrated that the CLIP encoder fails to generalize to some domains and the visual capacity of MLLM can be enhanced by introducing additional vision experts.

[1] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CVPR 2024.

[2] VCoder: Versatile Vision Encoders for Multimodal Large Language Models. CVPR 2024.

[3] SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models. ICML 2024.

[4] SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models. ECCV 2024.

[5] Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models. ECCV 2024.

Q2: We cannot include all models that cover different tasks exhaustively.

1. We agree with this idea but we do not aim to exhaustively incorporate extensive vision encoders to cover all the tasks in this paper.
2. Instead, we aim to reveal the common limitations of previous mixture of vision experts methods and propose a new paradigm. This paradigm provides a new perspective for the community that we can dynamically incorporate relevant vision experts for specific tasks with poor performances in real-life scenarios.

Q3: Adding resolution is more reasonable.

1. We agree that enlarging the image resolution can bring performance gains on OCR tasks. We also find both adding resolution and adding vision experts (MoVA) are orthogonal approaches to enhance visual capacity in our experiments. For example, we train a MoVA-7B model with the image split technique of LLaVA-NeXT (MoVA-HD) and compare it with the baseline on OCR tasks. Specifically, we divide an input image into 4 patches. The MoVA-HD can improve the score on the DocVQA benchmark from 81.3 to 84.0. This indicates such a high resolution setting also benefits MoVA.
2. The experiments in Table 1 have demonstrated the text recognition expert, such as Pix2Struct, can surpass the CLIP encoder on text-oriented VQA benchmarks by clear margins with aligned image token settings.
3. As described in Q2 of our global response, MoVA can achieve superior performance to LLaVA-NeXT with high resolution technique under the same settings.

Q4: It is hard to validate the gains are from the MoV experts.

Please refer to the Q2 of our global response.

Q5: Does the method introduce error propagation?

The incorrect routing results may harm the performance of the MLLM. In some cases, we find the multimodal gating in MoV-Adapter can assign low scores to these experts that are not relevant but incorrectly activated. We release some routing cases in the pdf file of global response.

Q6: Routing strategy design.

1. In the Table 14 of paper appendix, we discuss the effect of LLM for expert routing. Specifically, we compare the LLM router with BERT router and MLP router. For smaller router variants, they require more routing training data and fail to attain comparable performance.
2. We only use 1 LLM for expert routing and response generation. Using a tiny LLM in the routing stage can bring addtional model parameters.
3. To ablate the routing performance of tiny LLM router (InternLM2-1.8B) and standard LLM router (Vicuna-7B), we mutually label a subset of our selected 500 routing evaluation samples as "easy" or "hard". The routing performance of MoVA on these 500 samples have been discussed in Appendix A.3. Finally, We obtain 50 easy samples and 20 hard samples. The routing accuracy is obtained by human evaluation. As shown in the table, the tiny LLM router achieves comparable accuracy on simple evaluation samples while failing to catch up to the standard LLM router on hard evaluation samples.

Dear Reviewer Lntx,

Thank you for appreciating our approach. We will address your concerns below.

Q1: Leveraging multiple visual encoders in MLLM design does not introduce novelty. Additionally, the design of MoV-Adapter also lacks innovation.

Leveraging multiple visual encoders is not our novelty. Please refer to the Q1 of our global response.

Q2: This paper lacks a comprehensive evaluation regarding the routing selection of multiple visual encoders.

1. We have ablated the design of routing selection in Table 9, Table 10, and Table 14. We verify the effectiveness of our routing data in the line 598.
2. We present some routing cases in the pdf file of the global response.
3. We calculate the activation probability for each expert on several benchmarks:

After checking the peer review comments and the author's responses, I decided to maintain the given score for this work.