Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

1. The main contribution of this paper is to extend pre-trained LLMs to monolithic MLLMs via multimodal MoE and multi-stage pre-training. However, the novelty of this work is limited, as it mainly relies on previous works, especially VLMo, which proposed multimodal MoE and stagewise pre-training in the VLMs era. With full respect to the author's contribution as wrote in the strengths, but Mono-InternVL is very much a VLMo that combines the experience of data and training engineering tricks widely used in existing MLLMs.
2. This paper lacks fair comparison and in-depth analysis on modular and monolithic MLLMs. As a counterfact to Mono-InternVL, the authors should provide a modular version (Modular-InternVL) to fairly compare the performance (Table 2,3,4). Take Table 2 as an example, compared with other modular MLLMs, too many factors that have a significant impact on performance make such comparisons unfair and unconvincing (LLM backbone, training data, evaluation setting like image resolution).
3. This paper seems to avoid comparing with strong modular and monolithic MLLMs, such as InternVL2, Qwen2-VL, Phi-3-Vision, and Reka (a very strong monolithic MLLM). These comparisons are meaningful and important. It is very strange that the authors ignore these comparisons, especially when InternVL2 shares the same LLM backbone with Mono-InternVL, and the LLM backbone has a significant impact on performance.

• Confusion in Structural Design: In L314-315 of the manuscript, it is mentioned that unfreezing the parameters of the MHA component during the Alignment Learning stage helps achieve better vision-language alignment. However, this improvement does not seem intuitive. Furthermore, will unfreezing the MHA while freezing the T-Ext lead to potential losses in language capability?

• Scalability of the Method:

  • Compared to mainstream paradigms based on ViTs, the monolithic architecture requires a substantial amount of image-text data to facilitate endogenous visual pre-training. My concern is that, unlike ViTs which can adapt to various LLMs through lightweight image-text alignment (e.g., LLaVA 558K), the visual expert trained within Mono-InternVL is specific to a particular LLM. Specifically, The semantic space and model structure of the visual expert are both specialized for InternLM, making it difficult to reuse them with other LLMs (e.g., Qwen, LLaMA3). It significantly limits the scalability of the approach, especially considering the high training costs involved.
  • Additionally, performance validation was only conducted on a 1.8B parameter scale. While the performance is notable compared to other ViT-based models of similar size, scaling up to 7B or even 72B while continuing to utilize a visual expert reveals a clear disadvantage in training costs compared to ViT-based models.

• Fairness in Performance Comparison: Another concern arises from Table 6. In the comparison of computational overhead, is it truly necessary to only compare with InternVL-1.5-2B? For instance, as the authors demonstrate in Table 3, several ViT-based models perform comparably to Mono-InternVL under similar activation parameters (e.g., MiniCPM-V-2). To my knowledge, these models utilize only SigLiP or CLIP-L as their image encoder, resulting in a minimal number of image tokens. In contrast, Mono-InternVL employs a significantly larger number of patches. Therefore, I believe a comprehensive comparison of performance and training costs between these models and the proposed Mono-InternVL is necessary to accurately reflect the advantages of the monolithic model.

I hope the author can provide a more detailed explanation and proof of the proposed method. I will consider changing my perspective if the author's response is valuable.

1. While the paper introduces several key contributions, such as the multimodal MoE architecture and the progressive training strategy, it does not clearly delineate the performance gains attributable to each component.
2. The concept of a multimodal MoE structure has been explored in prior works such as MoE-LLaVA, MARS, and Uni-MoE. However, the paper lacks a detailed discussion on how this model differs from these existing approaches.
3. In Section 3.3, the authors mention the inclusion of additional data sets for video understanding and handwritten text recognition. However, the paper does not clarify the impact of these data sources on the model's performance.

[1] MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

[2] MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis

[3] Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

• A key weakness of the paper lies in the lack of clear motivation for adopting monolithic MLLMs over modular MLLMs. Understanding the advantages and rationale behind choosing monolithic structures is crucial for assessing the significance and justification of the work. The authors need to provide a detailed explanation of why this approach is preferable and what specific benefits it offers compared to modular MLLMs.
• The method proposed in the paper lacks sufficient novelty. To address catastrophic forgetting, the paper introduces independent visual parameters, an approach that is already widely used in modular MLLMs such as QwenVL and InternVL-1.5. The authors need to clearly explain how their approach differs from these existing methods. Simply transferring these techniques from modular to monolithic MLLMs does not constitute a significant innovation.
• The paper's approach to processing visual input in monolithic MLLMs employs PatchEmbed and MLP, while modular MLLMs typically use a visual encoder and MLP. PatchEmbed, as a learnable module, effectively encodes visual input into tokens that can be fed to the LLM, similar to the role of a visual encoder. This raises the question of how the proposed method truly differentiates itself from modular approaches, as both share the fundamental idea of transforming visual input into LLM-readable tokens. The authors need to clarify how this design embodies the monolithic concept and what the essential differences are compared to modular MLLM structures.
• The paper introduces a progressive learning approach that divides the pre-training process into three distinct stages. To substantiate the effectiveness of this method, the authors should provide ablation studies demonstrating its impact. Additionally, it would be beneficial to explain why the pre-training data is segmented into these three categories, potentially from the perspective of data distribution. Moreover, it raises the question of whether instruction-tuning data would also benefit from a similar progressive approach.
• The benchmarks used for comparison in the paper are incomplete, as they do not include prominent MLLMs such as LLaVA-1.5 and Qwen2-VL. The absence of these mainstream models weakens the claim of superiority for the proposed method.
• The authors need to compare MLLMs of different sizes to demonstrate the scalability of their approach. Currently, the models evaluated in the paper are all under 2B parameters. To validate the proposed method's applicability and performance at scale, it is essential to include experiments with larger models, such as 7B and 13B.
• The paper lacks detailed implementation specifics for FFN_v and FFN_t. It states that FFN_v is initialized from FFN_t to leverage pre-trained knowledge. The author should provide a comprehensive explanation of the implementation process. This detail is crucial for understanding the decoupling and interaction between visual and textual knowledge.