NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints

Thank you for your valuable and detailed comments. We address your questions as follows.

W1: Clarity / Self-Containedness: There is a lack of clarity regarding the model's architecture. The paper is not well self-contained in this regard, and requires the reader to reference external work (e.g., Mono-InternVL) to understand crucial architectural details. A clear architectural diagram that clearly displays the exact model setup (and possibly contrasts it with the “compositional” counterpart) would be a significant improvement.

A1: The model architecture is introduced in text form in the paper. L95-103 first defines the meta architecture of MLLM, consisting of a visual encoder (with an MLP connector), an LLM, and a MoE architecture injected into the LLM. L262-270 further describes the detailed architecture of NaViL-2B, including model parameter size, attention mechanism, positional embeddings, and more. Appendix C explains the detailed implementation of the modality-specific MoE adopted by our approach.

Due to the policy of NeurIPS 2025, we were unable to provide the architectural diagram during the rebuttal process. However, to further clarify the model configuration, the following table lists the key hyperparameters of NaViL-2B. Both the table and the figure will be updated in the final version. The code for the model structure and inference will also be open-sourced.

W2: Vague Terminology: The terms “native” and “compositional” MLLMs are not clearly defined or well-contrasted with existing community terms like "early-fusion" (most standard) or "monolithic".

A diagram could help clear this up additionally (see W1)

A2: Please refer to A7. In addition, a figure for comparing “native” and “compositional” MLLMs will be included in the final version.

W3.1: The paper has several minor but frequent English writing errors (e.g., "with Data Constraint," "Besides,") that reduce its overall polish. Sending it through a grammar/spell checker would help.

A3: Thanks for the suggestion. The paper will be polished to eliminate such errors in the final version.

W3.2: Some mathematical notation is overloaded (e.g., the symbol is used for two different concepts in §3.3.2 and §4.1), which could be easily rectified.

A4: Thanks for the suggestion. Different notations will be used in the final version.

W3.3: Citation [24] referencing pretraining vision encoders on L20 seems like a mis-citation? This citation appears to be about weight interpolation

A5: Thanks for the suggestion. However, citation [24] is not a mis-citation, as it refers to the repository page for the OpenCLIP software project on Zenodo (one of the pre-trained visual encoders widely adopted by MLLMs), not to weight interpolation.

Q1: Architectural Diagram: As noted in W1, the paper is difficult to understand without an architectural diagram. To address this, could you please add a figure that clearly illustrates the NaViL architecture? It would be very helpful to see the data flow from the trainable vision encoder through the connector to the LLM, and to visualize the differences between the MoE and non-MoE variants you tested.

A6: Please refer to A1. An additional figure will be included in the final version, illustrating both the data flow and the differences between MoE and non-MoE variants.

Q2: Clarification of Terminology: To resolve the ambiguity mentioned in W2, could you more precisely define "native" vs “compositional” MLLMs?

1. “Early-fusion” a la Chameleon had a more clear-cut definition. What is the defining characteristic of a “native” model? That the vision model is trained from scratch?

2. What if the vision model were initialized with pretrained vision-model weights (just as the paper advocates for initializing the LLM component with pretrained LLM weights)—would this still be a native MLLM?

A7: (1) The distinction between native MLLMs and compositional MLLMs primarily lies in whether the training process and training objectives can be unified.

As discussed in L69-71, compositional MLLMs have different components initialized by separate unimodal pre-training, where different training objectives and strategies are employed to train the LLM and visual encoder. For example, the visual encoder can be trained using an image-text contrastive learning objective (e.g., CLIP, SigLIP) or a self-supervised learning objective (e.g., DINOv2). The complexity of such training process increases the difficulty of scalability.

As shown in L79-83, the generally accepted definition of a native MLLM [1] is that it optimizes both image and text modalities end-to-end using a unified training objective (i.e., next-token prediction (NTP)). This avoids introducing additional bias and significantly simplify the scaling effort.

In contrast, early-fusion (and monolithic) models define MLLMs from a model architecture perspective, meaning that visual and text modality inputs are fused in the early stage and no modality-specific parameters are used. For example, Chameleon is both an early-fusion model and a native MLLM because it integrates visual and textual inputs from the outset and trains the entire model end-to-end using a unified training objective.

(2) Initializing the vision encoder with a contrastive learnining pre-trained model introduces different training objectives and thus violates the definition of native MLLM in our paper.

Q3: These scaling laws are validated only at a small scale, though there are “final results” in the appendix up to 9B. It is unrealistic to run these experiments, so perhaps a limitation should be listed that it is unknown if the observed scaling trends would hold at drastically larger scale (~30B, ~70B, 100+B, etc).... ie, caveat the observed scaling laws by emphasizing they are observed at a small scale.

A8: Thanks for the suggestion. We will mention this as our limitation and potential future work in the final version.

[1] Shukor, Mustafa, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby. "Scaling laws for native multimodal models." arXiv preprint arXiv:2504.07951. 2025.

thanks for the detailed rebuttal. most of my concerns have been addressed.

A1: I indeed read the lines you reference in your submission but found them confusing and insufficient to understand. this was the point of writing this weakness.

please be sure to include a good architecture diagram (A1, A6) in the final version.

A7: this explanation was very helpful, thank you. a high-level conceptual diagram or much better explanation in the paper would be a big improvement here.

Thank you for your valuable and detailed comments. We address your questions as follows.

W1: The final result of the article only provides a 2B-sized model. It is better to conduct experiments on a larger model to verify the effectiveness of the scaling law.

A1: We also report the results of NaViL-9B in Appendix A. With a similar parameter size, our NaViL-9B outperforms all existing native MLLMs by a large margin on almost all benchmarks. Besides that, compared to the compositional baseline model InternVL-2.5-8B with a similar parameter size, NaViL-9B also achieves competitive performance. Such results demonstrate that NaViL can be scaled up to larger parameter sizes and achieve consistent performance gains.

W2: The paper lacks detailed discussion of computational overhead and training efficiency comparisons between native and compositional approaches, which is crucial for practical adoption.

A2: As described in L143-146 and L262-271, NaViL adopts a modality-specific MoE architecture, resulting in the same number of parameters and inference FLOPs as the compositional counterpart. The implementation details of the modality-specific MoE are provided in Appendix C. A comparison of total training tokens is shown in Table 1 in the Appendix. Our approach achieves comparable performance while using significantly fewer training tokens, resulting in improved training efficiency.

Q1: The related work section should include more comprehensive discussion of visual scaling paradigms, particularly foundational work such as Zhai et al. (2022) "Scaling Vision Transformers" (CVPR), which established important principles for visual model scaling that are relevant to this work.

A3: Thank you for your suggestion. More discussion on the visual scaling paradigm will be included in the related work section of the final version. “Scaling Vision Transformers” (Zhai et al., 2022) is an important and valuable work related to our research, which will also be discussed in the final version.

Thank you for your valuable and detailed comments. We address your questions as follows.

W1.1: Section 3.2 presents limited novel insights across its three subsections. The LLM initialization exploration (Section 3.2.1) yields unsurprising conclusions that initialization of pre-trained weights help.

A1: We understand your concerns about LLM initialization benefits seeming intuitive. However, this intuition doesn't always hold even in computer vision; for example, [1] shows ImageNet pretraining doesn't improve object detection and instance segmentation performance.

In the MLLM community, while LLM initialization has become the default practice, this fundamental question remains unanswered. Therefore, we quantitatively demonstrate LLM initialization's impact in Fig. 2, breaking this black box and addressing this fundamental question for the entire community. From this perspective, our conclusions and results provide valuable insights to the community.

W1.2: (1) Section 3.2.3 only examines width-depth trade-offs and reaches conventional conclusions that shallow models converge faster while deeper models achieve better performance. (2) Have the authors considered more sophisticated architectural explorations such as attention mechanism variations and different normalization strategies?

A2: (1) Similar to W1.1, visual encoder width-depth design for MLLMs remains controversial. Previous studies [2,3] focus on single visual encoders for pure vision tasks, but these principles may not transfer to MLLMs with large-scale LLMs. Most works use fixed-size visual encoders (e.g., SigLIP) or discard them entirely (Mono-InternVL), making it difficult to draw clear conclusions about width-depth design for native MLLMs.

We systematically ablated visual encoder width-depth design and found a different conclusion that visual encoders achieve near-optimal performance across wide depth/width ranges. Except for extremely shallow/deep configurations, relatively shallow encoders converge faster while deeper ones perform better in later training stages. These findings provide valuable guidance for future native MLLM research.

(2) We focus on encoder size scaling properties, keeping meta-architecture (attention, normalization, activation functions) the same while varying only depth and width, following common practices for scaling law research [4,7]. Better architectural components will be explored in future work.

W2: Section 3.2.2 (lines 137-151) provides inadequate description of the MoE architecture implementation and fails to ensure fair experimental comparison. The paper does not specify the exact parameter counts for vanilla versus MoE configurations, the number of experts used, or the expert selection mechanism.

A3: The number of activated experts is set to one (L145-L146) and static modality routing is used (L143-145). Therefore, the activation parameter count and training FLOPs are the same for both vanilla and MoE configurations, which ensures fair comparison and is consistent with the experimental configurations in many prior works [4]. Appendix C has introduced the details of MoE implementation. We thank the suggestion and will revise by discussing more implementation details of our MoE in the main paper.

W3: The downsampling rate mentioned in line 232 is represented by the symbol \tau, which is also used to represent another quantity used in line 211, potentially causing ambiguity.

A4: Thanks for the suggestion. Different symbols will be used in L232 and L211 for the camera-ready version to avoid ambiguity.

W4: Tables 1 and 2 contain substantial missing experimental results across multiple baseline models and benchmarks and without further explanation. This incomplete evaluation results undermines the paper's claims about competitive performance against existing methods and makes it difficult to assess the true relative standing of NaViL.

A5: In Tab. 1 and 2, we report only the official experimental results of other baseline models, sourced from their respective papers. Since different papers evaluate their models on different benchmarks, some results are missing. To further support our claims about NaViL's performance, we report the full results of Qwen2VL-2B, SAIL, and EVEv2, as shown in the following tables (* denotes our reproduced results).

The conclusion remains consistent: NaViL's average performance is still ahead of previous native MLLMs and is competitive with the compositional baselines.

Q1: Section 3.2.2 lacks crucial implementation details about the MoE architecture. What is the exact MoE configuration used in the comparison? How many experts are employed, and what is the expert selection mechanism? Most critically, what are the precise parameter counts for both vanilla transformer and MoE configurations to ensure fair comparison? Without parameter equivalence, the observed performance improvements in Figure 3 may simply reflect increased model capacity rather than architectural advantages.

A6: Please refer to A3.

Q2: Section 3.2.3 explores visual encoder architecture only through width-depth variations. Have the authors considered more sophisticated architectural explorations such as attention mechanism variations, different normalization strategies, or alternative activation functions that might yield more novel insights for native MLLM design?

A7: Please refer to A2.

Q3: While the log-linear relationship in Figure 7 is empirically observed, what theoretical principles could explain this phenomenon? Besides, the only three pairs (0.5B, 2B, 7B) of datapoints are quite limited for such a claim of scaling law. Can the authors make sure this trend can generalize to larger scales?

A8: Several theories explain the log-linear relationship in Fig. 7, including encoder-decoder scaling laws in neural machine translation [5] and multimodal scaling laws for text, speech, and images [6]. Similar theories could apply to native MLLMs, but our paper focuses on design space and scaling properties under data constraints, leaving theoretical analysis for future work.

Scaling laws predict larger-scale model performance from small-scale experiments. Previous works [7] don't show all datapoints for every scale. Since native MLLM scaling research typically uses models below 8B parameters [4], we follow this convention while planning verification on larger MLLMs. We will further verify the results of the scaling law on a larger MLLM.

Q4: Baseline Evaluation. Tables 1 and 2 show incomplete results for baselines evaluation across multiple benchmarks. Would complete baseline evaluation change the relative performance rankings?

A9: Please refer to A5.

Q5: As mentioned in Section 5.1, NaViL-2B is built upon InternLM2-1.8B, initializing its text part parameters with the weights of InternLM2-1.8B. I assume this means that NaViL-2B is, to some extent, further trained based on InternLM2-1.8B. While this approach may facilitate training convergence, is it really fair for the performance comparisons? If NaViL-2B were trained from scratch, would it still achieve similar performance results?

A10: NaViL-2B's textual parameters are initialized from InternLM2-1.8B. Since this paper focuses on native MLLM scaling properties under data constraints, convergence speed is crucial. All MLLMs in Tab. 1-2 use pre-trained LLM initialization for practical high performance, ensuring fair comparison. As shown in L113-115 and Fig. 2, while non-LLM initialization can achieve similar final performance, it requires substantial high-quality data, violating our data-constrained setting.

[1] He, Kaiming, Ross Girshick, and Piotr Dollár. "Rethinking imagenet pre-training." In ICCV. 2019.

[2] Alabdulmohsin, Ibrahim M., Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. "Getting vit in shape: Scaling laws for compute-optimal model design." In NeurIPS. 2023.

[3] Zhai, Xiaohua, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. "Scaling vision transformers." In CVPR. 2022.

[4] Shukor, Mustafa, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby. "Scaling laws for native multimodal models." arXiv preprint arXiv:2504.07951. 2025.

[5] Ghorbani, Behrooz, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. "Scaling Laws for Neural Machine Translation." In ICLR. 2022.

[6] Aghajanyan, Armen, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. "Scaling laws for generative mixed-modal language models." In ICML. 2023.

[7] Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361. 2020.