Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

Detailed Benchmark Setting

We thank Reviewer aaRq for highlighting the benchmark setting. For Tables 2–5, we use the validation sets of VizWiz, SEED, DocVQA, and ChartQA for our investigations. For the final results, the models are evaluated on the test sets to ensure alignment with the evaluation protocols of other compared models. To minimize the risk of overfitting to specific benchmarks, we also reduce the overlap between the benchmarks used for our investigations and those used for final comparisons with other models.

Questions on Fine-tuning Data

Thank you for highlighting this potential confusion! On lines 141–142, our intention is to describe the general type and format of the fine-tuning data, which is multimodal conversations. For this purpose, we utilize Eagle 1.8M, a dataset we collected that includes a diverse mixture of multi-modal conversation data spanning various tasks.

All the models from Table 2-5 and Fig. 4 are trained in two stages. On the first stage the model is pretrained on LLaVA-595k and the second stage on Eagle 1.8M. For experiments involving pre-alignment, as shown in Tables 6 and 8, the models undergo a three-stage training process. During the first pre-alignment stage and the second pretraining stage, we use a combination of LLaVA-595k and Eagle 1.8M as the training data.

Additional computation cost of unfreezing vision encoders

Thank you for the question! The throughputs and FLOPs are all tested under inference mode, which can not reflect the difference in back-propagation.

To clarify the additional computational costs incurred during training, we have provided a detailed comparison of training time and memory usage in the table below. All experiments were conducted on 32 A100 GPUs across four nodes using DeepSpeed Zero-2 optimization. Each GPU processes a batch size of 4, resulting in a total batch size of 128.

As shown in the table below, unfreezing the vision encoder during training increases both training time and memory usage. However, we find this step to be crucial for enhancing performance, particularly when the input resolution changes or when the vision encoder has not been pretrained on vision-language alignment tasks. For example, unfreezing the vision encoder during the second stage improves the average performance of EVA02 (pretrained on detection) from 543.7 to 639.1. Please refer to Table 3 for more detailed comparisons.

Considering the performance gain and the fact that unfreezing the vision encoder during training does not increase inference time, we believe the performance gains fully justify the additional training time.

The comparison between tiling and interpolation

Thank you for the question! There is indeed a trade-off in FLOPs between the tiling and interpolation approaches. Tiling divides the input image into patches, which are processed by the ViT independently, whereas interpolation treats the image as a whole, allowing all patches to participate in the attention calculation. This is analogous to window attention versus global attention: tiling resembles window attention, while interpolation aligns with global attention.

Hence, as shown in Table 2, under the same number of visual tokens and input resolutions, the tiling approach has fewer FLOPs. However, its throughput is lower, likely because throughput can be significantly influenced by hardware capabilities and computation parallelism. Advanced hardware and software optimizations, such as flash-attention, have greatly improved the efficiency of attention computation and helps boost parallel computation. Consequently, even when processing an input resolution of 672 directly, the system does not encounter a performance bottleneck. In contrast, tiling requires multiple forward passes (e.g., four for a single image), which reduces throughput due to limited parallelization opportunities.

Theoretically, as with window and global attention, window attention has linear complexity with respect to the number of input tokens, whereas global attention has quadratic complexity. Hence, as the input resolution further scales up, it may hit the bandwidth of the platform and cause lower throughputs or even cause out-of-memory issues.

Question about Clip encoder L

The size of the CLIP-L encoder, as shown in table 2 has 304M parameters in total. It’s pretrained on 336 inputs by OpenAI. The pretrained checkpoint can be accessed at https://huggingface.co/openai/clip-vit-large-patch14-336.

We thank Reviewer GjPo for kindly reminding us and apologize for missing this key paper. We agree that Prismatic VLM should be cited given its pioneering study in the design space of VLMs, including the use of a mixture of vision encoders and channel concatenation. We have included detailed comparisons and discussions in the introduction, methodology, and related work sections in the revised manuscript (marked in orange). That being said, we would like to emphasize a few important points:

Strange not citing/discussing with PrismaticVLM

It is our mistake missing this important work. However, we did not try to avoid the discussion with related works, as the reviewer might have hinted here. Instead, we stated clearly in paper (Line 86) that "Our work is not the first one to leverage multiple vision encoders in MLLM." Many of the cited works are concurrent or even earlier than Prismatic VLM with MoVE (Mixture of Vision Encoders) designs. For instance, one of the cited works, SPHINX (Nov 13 2023), also uses channel concatenation to mix multiple encoders, similar to Prismatic VLM (Feb 12 2024). This work does not try to paint itself as the first one to propose MoVE or MoVE with channel concatenation. But we agree with the reviewer that it's important to make this part clearer.

Findings/contributions are trivial

We emphasize that this work is not a mere extension of Prismatic VLM to more vision encoders. The differences in contributions are significant between the two works:

1. Goals and exploration spaces are quite different. The purpose of Prismatic VLM aims to study the VLM design in general where MoVE is part of the explored designs bringing benefits, whereas this work delves much deeper into the design of MoVE by studying/comparing a spectrum of MoVE strategies carefully, in response to the lack of systematic study in MoVE. Our study covers critical aspects such as vision encoder selection, fusion paradigms, and pre-training strategies tailored for combining multiple vision encoders. These contributions introduce insights complementary to Prismatic VLM and provide novel, valuable knowledge to the community.

2. Key conclusions are different and more comprehensive. Prismatic VLM, like many other cited works in the paper, belongs to the family of works with frozen vision encoders. In fact, Prismatic VLM found that unlocking vision encoder during fine-tuning hurts the performance, whereas this work leads to a different conclusion that unlocking vision encoder during fine-tuning is the key to help MoVE work. This difference is probably caused by the improved data and training recipes adopted in this work, and better echoes the more recent findings in other works, such as Cambrian-1 and LLaVA-OneVision. In fact, our frozen vision encoder baselines can be seen as a proxy of the proposed strategy in Prismatic VLM under our apples-to-apples comparison. And the improvements from unlocking and pre-alignment are significant. As such, it may be incorrect to conclude that the contributions are limited.

3. Comparing the performance gain is "comparing apples with oranges". Compared to Prismatic VLM, the baseline of this work started with larger data (Eagle 1.8M), stronger baseline (e.g., higher TextVQA and AI2D) and covers a wider range/domain of benchmarks. Achieving the same delta in performance gain is thus significantly more difficult. On the other hand, a significant portion of gain in Prismatic VLM (e.g., Table 7 SigLIP + DINOv2) came from object-centric datasets such as RefCOCO/RefCOCOg/RefCOCO+/OCID-Ref, which is not that surprising considering the weak object localization of SigLIP and the strong object localization of DINOv2. Thus, directly comparing the two works in performance gain is not fair and does not help to draw meaningful conclusions. More importantly, we aim to demonstrate MoVE with more encoders as an effective way to scale-up the vision foundation of VLMs and benefit their performance. This is clear in comparison to InternVL which has 6B parameters but shows significantly lower results.

4. The additionally introduced costs in vision encoder is relatively marginal compared to the LLM auto-regressive generation as a whole. To provide quantitative evidence, we evaluate image captioning as a real-world use case to measure the inference speed of different vision encoders. The experiment uses Vicuna-7B as the base model, employing float16 precision and KV cache optimization for efficient LLM generation. The generation process is halted after producing 128 tokens (approximately 100 words). The input consists of a 1024 x 1024 resolution image and a prompt—“Describe this image in detail.”—which results in 50 input text tokens plus a chat template.

   We calculate four distinct computational metrics:

   • Generation time: The time taken to generate 128 tokens, measured from the input of the model to the output of the final token.

   • Vision encoder latency: The time required for the vision encoder to process the input image and produce feature maps.

   • FLOPs of the vision encoder: The total floating-point operations performed by the vision encoder.

   • FLOPs of the first forward pass: The total operations required to compute the hidden states for both the input image and text embeddings.

   These metrics allow us to provide a comprehensive analysis of computational costs and efficiency for the vision encoders within the context of a full MLLM pipeline.

Avg. means the average score across a number of VLM benchmarks.

The results show that the latency of the vision encoder constitutes only 5.7% of the total generation time for the heaviest model, X5. This proportion decreases further with larger language models, such as 3% for a 13B model, and becomes even smaller in scenarios involving longer token generation, such as multi-round conversations or detailed explanations.

In terms of floating-point operations (FLOPs), the vision encoder accounts for less than 50% of the operations during the initial forward pass. Meanwhile, the auto-regressive generation process requires repeated computation for each subsequent token, significantly outweighing the vision encoder's contribution to overall computation.

We would also like to mention that our current implementation is based on a naive PyTorch setup, where each vision encoder requires stand-alone processing code and is computed sequentially. However, in practice, each vision encoder can be assigned to a separate GPU since they operate independently. In such a scenario, the inference time for the entire vision encoder ensemble would be largely determined by the slowest vision encoder. For Eagle-X5, this would reduce the total inference time to ~0.10s instead of 0.24s.

Thanks for the constructive feedback! We hope to clarify the confusion and answer the questions below. Feel free to let us know if you have any further questions!

Details of the projector

In stage-2, we initialize only the weights of the vision encoder from stage-1, i.e., the pre-alignment training. The projectors, however, are newly initialized in the second stage. Inspired by your feedback, we believe it is worth exploring an alternative approach: concatenating the projectors from stage-1 to form a new projector. If this modification proves effective, we will update the manuscript accordingly.

Incorrect Cross-reference

Thank you for pointing this out. We have addressed this concern and made the necessary adjustments in the revised manuscript.

Throughput of the vision tower

For the second concern, we want to first clarify that, for a MLLM, the additionally introduced costs in vision encoder is relatively marginal compared to the LLM auto-regressive generation as a whole. Hence, all of the methods share similar generation time with the same language model and generation length. To provide quantitative evidence, we evaluate image captioning to measure the inference speed of different vision encoders. The experiment uses Vicuna-7B as the base model, employing float16 precision and KV cache optimization for efficient LLM generation. The generation process is stopped after producing 128 tokens (approximately 100 words). The input consists of a 1024 x 1024 resolution image and a prompt—“Describe this image in detail.”—which results in 50 input text tokens plus a chat template.

We calculate four distinct computational metrics:

• Generation time: The time taken to generate 128 tokens, measured from the input of the model to the output of the final token.
• Vision encoder latency: The time required for the vision encoder to process the input image and produce feature maps.
• FLOPs of the vision encoder: The total floating-point operations performed by the vision encoder.
• FLOPs of the first forward pass: The total operations required to compute the hidden states for both the input image and text embeddings.

These metrics allow us to provide a comprehensive analysis of computational costs and efficiency for the vision encoders within the context of a full MLLM pipeline.

Avg. means the average score across a number of VLM benchmarks.

The results show that the latency of the vision encoder constitutes only 5.7% of the total generation time for the heaviest model, X5. This proportion decreases further with larger language models, such as 3% for a 13B model, and becomes even smaller in scenarios involving longer token generation, such as multi-round conversations or detailed explanations. In terms of floating-point operations (FLOPs), the vision encoder accounts for less than 50% of the operations during the initial forward pass. Meanwhile, the auto-regressive generation process requires repeated computation for each subsequent token, significantly outweighing the vision encoder's contribution to overall computation.

The reviewer mentioned that methods with similar inference cost should be compared for a fair comparison, which we think is very reasonable. We choose the current largest vision encoder, InternVL 6B, as the vision encoder with the same training data and recipe. With similar FLOPs and over 2x parameters, InternVL-6B still falls behind our method.

In addition, we would also like to mention that our current implementation is based on a naive PyTorch setup, where each vision encoder requires stand-alone processing code and is computed sequentially. However, in practice, each vision encoder can be assigned to a separate GPU since they operate in parallel independently. In such a scenario, the inference time for the entire vision encoder ensemble would be largely determined by the slowest vision encoder. For Eagle-X5, this would reduce the total inference time to ~0.10s instead of 0.24s.

We thank reviewer La1k for the valuable feedback. Here, we aim to expand the discussion on the comparison between sequence append and channel concatenation. The most critical concern with sequence append approach is its significant increase in sequence length, which is highly undesirable for MLLM due to the following issues:

• Increased training and inference cost: As shown in the table below, longer sequences lead to a substantial rise in both the first forward time and overall training time.
• Occupying context length and position IDs in the LLM: This may limit the model's ability to handle longer input contexts effectively.

To quantitatively illustrate this, we interpolated the visual tokens to different sequence lengths and measured the first forward time and total training time during the supervised fine-tuning stage. As shown in the table below, increasing the sequence length from 1024 to 3072 results in a 61% increase in first forward time and a 50% increase in training time.

More importantly, as extensively studied in LLM research, increasing the sequence length causes challenges in identifying relevant content—commonly referred to as the "needle in a haystack" problem [1]. As shown in the last two rows of Table 4 in the manuscript, sequence append lags behind channel concatenation in performance.

The increased number of visual tokens also consumes more position IDs, making the generalization of position embeddings difficult [2]. For instance, in our experiments with Vicuna-7B/13B and Yi-34B, the maximum context length is 4096 as defined by the max_position_embeddings in their config [3]. This constraint limits the model's ability to accommodate more than three vision encoders with sequence appending, assuming each encoder outputs a sequence length of 1024 in our cases.

1. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack, NeurIPS 2024
2. YaRN: Efficient Context Window Extension of Large Language Models, ICLR 2024.
3. https://huggingface.co/lmsys/vicuna-7b-v1.5/blob/main/config.json https://huggingface.co/lmsys/vicuna-13b-v1.5/blob/main/config.json https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B/blob/main/config.json

We thank Reviewer La1lk for the thoughtful feedback! We will include additional discussion on the motivation and implementation of the deformable attention (DA)-based fusion approach. We incorporated DA as a fusion method for two main reasons:

• DA is widely adopted in various vision tasks, achieving state-of-the-art performance while maintaining high efficiency.
• DA shares conceptual similarities with the MiniGemini architecture. It can also be seamlessly integrated.

First, DA has become a critical component in many cutting-edge approaches across a range of vision tasks, such as DINO for object detection, Grounding-DINO for visual grounding, and BEVFormer for autonomous driving. Given its proven effectiveness in these applications, we were motivated to explore its potential for enhancing the visual representation within MLLMs.

Second, DA closely resembles the patch information mining architecture proposed by the important compared method, Mini-Gemini, where each pixel attends to neighboring pixels to facilitate information fusion. However, DA can dynamically predict the neighborhood, enabling larger and more flexible receptive fields while maintaining high efficiency.

Initially, we did not anticipate that simple direct concatenation would already yield strong performance. As a result, we tested several architectures in parallel, with DA considered an enhancement of MiniGemini’s fusion method. Therefore, we included DA in our comparisons to provide a comprehensive evaluation.

Here we list the training time comparison between unfreezing/freezing the vision encoders during supervised fine-tuning. The training is conducted on 32 A100 GPUs across four nodes using DeepSpeed Zero-2 optimization. Each GPU processes a batch size of 4, resulting in a total batch size of 128.

As shown in the table below, unfreezing the vision encoder during training increases both training time and memory usage. However, we find this step to be crucial for enhancing performance, particularly when the input resolution changes or when the vision encoder has not been pretrained on vision-language alignment tasks. For example, unfreezing the vision encoder during the second stage improves the average performance of EVA02 (pretrained on detection) from 543.7 to 639.1. Please refer to Table 3 for more detailed comparisons. Considering the performance gain and the fact that unfreezing the vision encoder during training does not increase inference time, we believe the performance gains fully justify the additional training time.

Here we address all the concerns on the efficiency trade-off of adding more vision experts.

We want to first clarify that, for a MLLM, the additionally introduced costs in vision encoder is relatively marginal compared to the LLM auto-regressive generation as a whole. Hence, all of the methods share similar generation time with the same language model and generation length. To provide quantitative evidence, we evaluate image captioning to measure the inference speed of different vision encoders. The experiment uses Vicuna-7B as the base model, employing float16 precision and KV cache optimization for efficient LLM generation. The generation process is stopped after producing 128 tokens (approximately 100 words). The input consists of a 1024 x 1024 resolution image and a prompt—“Describe this image in detail.”—which results in 50 input text tokens plus a chat template. We calculate four distinct computational metrics:

• Generation time: The time taken to generate 128 tokens, measured from the input of the model to the output of the final token.
• Vision encoder latency: The time required for the vision encoder to process the input image and produce feature maps.
• FLOPs of the vision encoder: The total floating-point operations performed by the vision encoder.
• FLOPs of the first forward pass: The total operations required to compute the hidden states for both the input image and text embeddings.

The results show that the latency of the vision encoder constitutes only 5.7% of the total generation time for the heaviest model, X5. This proportion decreases further with larger language models, such as 3% for a 13B model, and becomes even smaller in scenarios involving longer token generation, such as multi-round conversations or detailed explanations. In terms of floating-point operations (FLOPs), the vision encoder accounts for less than 50% of the operations during the initial forward pass. Meanwhile, the auto-regressive generation process requires repeated computation for each subsequent token, significantly outweighing the vision encoder's contribution to overall computation.

The reviewer mentioned that methods with similar inference cost should be compared for a fair comparison, which we think is very reasonable. We choose the current largest vision encoder, InternVL 6B, as the vision encoder with the same training data and recipe. With similar FLOPs and over 2x parameters, InternVL-6B still falls behind our method.

In addition, we would also like to mention that our current implementation is based on a naive PyTorch setup, where each vision encoder is computed sequentially. However, in practice, each vision encoder can be assigned to a separate GPU since they operate in parallel independently. In such a scenario, the inference time for the entire vision encoder ensemble would be largely determined by the slowest vision encoder.

Thank you for your valuable feedback on performance gain of including more vision encoders!

The performance gain after X2, though less significant on numbers, remains consistent. More importantly, it demonstrates a greater advantage compared to simply scaling a single encoder (e.g., InternVL-6B). This is attributed to the heterogeneity among expert architectures, the diversity of pre-trained knowledge, and improved coverage of various domain-specific expertise.

Here’s how we interpret the relatively reduced improvement:

1. ConvNeXt is indeed a powerful vision encoder with comprehensive capabilities, pre-trained on vision-language alignment tasks. Its convolutional nature complements the CLIP encoder effectively and supports high-resolution. Additionally, ConvNeXt’s large size (846M parameters) provides greater capacity, resulting in a boost to overall performance. In comparison, the other added vision encoders are more task-specific, primarily enhancing performance on benchmarks closely related to their pretraining tasks.

2. The benefit in the expansion of domain-specific expertise and capabilities should not be overlooked. This is particularly true for a generalist model, and the benefits here are sometimes implicit and not fully reflected by the scores on paper. In particular, many additional vision encoders are task-specific. For instance, Pix2Struct helps document and chart understanding, while EVA02 and SAM help object-centric, localization, and open-world recognition tasks. As shown in Table 12 of the appendix, from X3 to X5 where more specialized encoders like Pix2Struct and SAM are added, the improvements are more task-specific—most pronounced on datasets such as VizWiz, DocVQA, and ChartVQA. Although these experts can’t boost the average performance significantly as a strong vision language experts, they can help solving complex real-world tasks with special representations inherited from its own pretraining task. First part of Fig. 5 has listed some of the examples.

I thank the authors for their detailed response, and additional results, I am pleased to raise my score to 6.

Long Sequence

We thank reviewer La1k for the valuable feedback. Here, we aim to expand the discussion on the comparison between sequence append and channel concatenation. The most critical concern with sequence append approach is its significant increase in sequence length, which is highly undesirable for MLLM due to the following issues:

1. Increased training and inference cost: As shown in the table below, longer sequences lead to a substantial rise in both the first forward time and overall training time.

2. Occupying context length and position IDs in the LLM: This may limit the model's ability to handle longer input contexts effectively.

To quantitatively illustrate this, we interpolated the visual tokens to different sequence lengths and measured the first forward time and total training time during the supervised fine-tuning stage. As shown in the table below, increasing the sequence length from 1024 to 3072 results in a 61% increase in first forward time and a 50% increase in training time.

More importantly, as extensively studied in LLM research, increasing the sequence length causes challenges in identifying relevant content—commonly referred to as the "needle in a haystack" problem [1]. As shown in the last two rows of Table 4 in the manuscript, sequence append lags behind channel concatenation in performance.

The increased number of visual tokens also consumes more position IDs, making the generalization of position embeddings difficult [2]. For instance, in our experiments with Vicuna-7B/13B and Yi-34B, the maximum context length is 4096 as defined by the "max_position_embeddings" in their config [3]. This constraint limits the model's ability to accommodate more than three vision encoders with sequence appending, assuming each encoder outputs a sequence length of 1024 in our cases.

Reference

[1] BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack, NeurIPS 2024

[2] YaRN: Efficient Context Window Extension of Large Language Models, ICLR 2024.

[3] https://huggingface.co/lmsys/vicuna-7b-v1.5/blob/main/config.json https://huggingface.co/lmsys/vicuna-13b-v1.5/blob/main/config.json https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B/blob/main/config.json

Thanks for the constructive feedback! We hope to clarify the confusion and answer the questions below. Feel free to let us know if you have any further questions! Here we address all the concerns:

1. the performance gain of including more vision encoders;
2. the throughputs of the vision tower;
3. the additional cost of unfreezing the vision tower during training.

The performance gain from including more vision encoders

The performance gain after X2, although less significant number-wise, is still very consistent. What’s most important, is the more significant benefit compared to simply scaling a single encoder (for example, InternVL-6B), due to the heterogeneity among expert architectures, the diversity in pre-trained knowledge, and the better coverage of different domain-expertise.

Here’s how we interpret the relatively reduced improvement:

1. ConvNeXt itself is indeed a very strong vision encoder. It is convolutional, which well complements the CLIP encoder. It is large (846M in size). And it is pretrained on vision-language alignment tasks. Note that, the relative improvement is influenced by the capability of each encoder. And our round-robin strategy would systematically bias towards earlier rounds of improvement. It is also very natural that everything has a diminishing gain. Considering these factors, the improvements from MoVE is actually very promising.
2. The benefit in the expansion of domain-specific expertise and capabilities should not be overlooked. This is particularly true for a generalist model, and the benefits here are sometimes implicit and not fully reflected by the scores on paper. In particular, many additional vision encoders are task-specific. For instance, Pix2Struct helps document and chart understanding, while EVA02 and SAM help object-centric, localization, and open-world recognition tasks. As shown in Table 12 of the appendix, from X3 to X5 where more specialized encoders like Pix2Struct and SAM are added, the improvements are more task-specific—most pronounced on datasets such as VizWiz, DocVQA, and ChartVQA. Although these experts can’t boost the average performance significantly as a strong vision language experts, they can help solving complex real-world tasks with special representations inherited from its own pretraining task. First part of Fig. 5 has listed some of the examples.

We would like to thank Reviewer Y3dK for the valuable feedback. Here, we provide additional analysis on how more vision encoders improve the performance and discuss the associated efficiency trade-offs.

First, regarding the less significant performance gains after incorporating three vision encoders: initially, we add strong vision encoders pre-trained on visual-language alignment, i.e., the ConvNeXt vision encoder, which leads to significant overall performance improvements. Subsequently, we incorporate vision encoders pre-trained on more specialized tasks—for example, Pix2Struct for document and chart understanding, and EVA02 for object perception. However, compared to general-purpose vision encoders, these additional task-specific encoders primarily enhance performance in areas directly related to their pretraining tasks, resulting in less noticeable gains across general benchmarks. This pattern is evident in Table 12 of the appendix. The addition of the general-purpose vision alignment expert, ConvNeXt, yields a substantial performance boost. By contrast, moving from X3 to X5, where specialized encoders like Pix2Struct and SAM are added, the improvements are more task-specific, with the most pronounced gains observed on datasets such as VizWiz, DocVQA, and ChartVQA.

Although these task-specific capabilities is less helpful for the overall performance, they are also important for many real-world tasks. Fig. 5 has given some examples on how these task-specific encoders improve the CLIP + ConvNeXt model. At the same time, we want to emphasize that the primary contribution of our work lies in the systematic investigation process and proposed training strategies rather than the X5 model itself. Users can adapt our methodology to create their own encoder combinations.

We also want to clarify that, incorporating more vision encoders will affect the throughput of the vision encoder alone. However, considering the MLLM as a whole, we only increase the total generation time a little, because the language model and the auto-regressive generation will take up most of the time. The language model will have more parameters and the vision tower only participates in the first forward pass of the auto-regressive iterations. To provide quantitative evidence, we evaluate image captioning to measure the inference speed of different vision encoders. The experiment uses Vicuna-7B as the base model, employing float16 precision and KV cache optimization for efficient LLM generation. The generation process is halted after producing 128 tokens (approximately 100 words). The input consists of an image and a prompt—“Describe this image in detail.”—which results in 50 input text tokens with 1024 visual tokens. We calculate four metrics for efficiency analysis:

• Generation time: The time taken to generate 128 tokens, measured from the input of the model to the output of the final token.
• Vision encoder latency: The time required for the vision encoder to process the input image and produce feature maps.
• FLOPs of the vision encoder: The total floating-point operations performed by the vision encoder.
• FLOPs of the first forward pass: The total operations required to compute the hidden states for both the input image and text embeddings.

The results show that the latency of the vision encoder constitutes only 5.7% of the total generation time for the heaviest model, X5. This proportion decreases further with larger language models, such as 3% for a 13B model, and becomes even smaller in scenarios involving longer token generation, such as multi-round conversations or detailed explanations. In terms of floating-point operations (FLOPs), the vision encoder accounts for less than 50% of the operations during the initial forward pass. Meanwhile, the auto-regressive generation process requires repeated computation for each subsequent token, significantly outweighing the vision encoder's contribution to overall computation.

Thank Reviewer Y3dK for the feedback! Here we discuss the efficiency comparison with other models.

We have provided a detailed model size comparison with other models in the Table below. Our model achieves the best performance across all benchmarks while maintaining a model size smaller than competitors such as InternVL and Monkey. While our model size is slightly larger than some alternatives, it remains within a similar range, balancing performance and efficiency. The full results can be found in Table 6 of our paper.

Thank the reviewer Y3dK for the questions and the suggestions for more comprehensive experiments! Here we address all the questions and suggestions on the experiment setup.

1. Does pre-alignment work by simply extending SFT training with more iterations?

we would like to highlight that in Table 7, row 2 of our paper, we present the experimental results of a configuration where stage 1 is removed and stage 3 is instead trained for an additional epoch to balance the total training effort. As shown, even under this modified setup, our model with pre-alignment (see row 5) demonstrates superior performance, underscoring the benefits of incorporating the pre-alignment stage. This result further supports our approach, indicating that the observed performance improvements are not solely due to an imbalance in training rounds and data points but are indeed a result of the pre-alignment's effectiveness.

2. Why is it necessary to fine-tune the vision encoders in stage 3?

In response to your question regarding the necessity of further fine-tuning the vision encoders in stage 3 after their pre-alignment in stage 1, we have conducted an experiment where the vision encoders are frozen in stage 3. The results, shown in the table below, comparing the last two rows of each group, freezing the vision encoders during supervised fine-tuning case significant drop, yet it’s better than freezing the vision encoders without pre-alignment. These results indicate that keeping the vision encoders unfrozen in stage 3 is indeed essential for optimal performance. The plausible reason is that the language models are updated during the SFT, making the vision encoders and language model mis-aligned again. Hence, unfreezing the vision encoders helps to mitigate this misalignment, ensuring better integration and performance.

We appreciate the reviewer’s feedback on the contributions of our paper. While we acknowledge that our project does not introduce groundbreaking new architectures, we wish to emphasize that it offers several important conclusions and methods that contribute novel insights to the community that are untouched by prior works. These findings provide a practical blueprint for building high-resolution multimodal LLMs with strong performance across diverse tasks using straightforward combinations of multiple vision encoders, all without requiring large-scale pretraining. Based on the extensive comparisons and ablations, we want to highlight three parts of contributions.

• Systematic Exploration of Vision Encoder Fusion
  • We systematically study the design space of fusing multiple vision encoders, which is missed by prior works. We demonstrate that even simple architectures can effectively combine multiple vision experts. This avoids the need for designing or fine-tuning complex fusion architectures or scaling up vision encoder parameters and pretraining data.
• Pre-Alignment to Address Misalignment Issues
  • We are the first to identify and address the critical misalignment issue between different vision encoders by proposing a novel pre-alignment strategy.

As acknowledged by Reviewer La1k, our work addresses a critical issue in the development of multimodal LLMs, demonstrating that our approach represents meaningful progress by integrating effective modifications to enhance model coherence and performance.

Additionally, as noted by Reviewer 1gkn, our pre-alignment strategy—where non-text-aligned vision experts are individually fine-tuned with a frozen LLM before joint training—constitutes a valuable contribution to the field. Notably, no prior studies have specifically addressed the pretraining challenges associated with multi-vision encoder setups. Our pre-alignment method effectively addresses this issue, offering a design that is both efficient and effective.

Together, we believe these contributions underscore the impact and utility of our work, providing practical insights and methodological advancements that benefit the broader research community.

We thank Reviewer Y3dK for the valuable feedback! In our experiments, we use the CLIP encoder as the baseline LLaVA 1.5 to ensure a fair comparison. Additionally, we note that the compared methods, such as LLaVA-NeXt, LLaVA-UHD, LLaVA-HR, and Mini-Gemini, also utilize the CLIP encoder. However, as you highlighted, SigLip has become a popular choice for MLLMs. In response, we have re-conducted the experiments, substituting the CLIP in our models with the SigLip encoder.

First, using the same experiment setting as Fig. 4 in the main paper, we replaced the CLIP with SigLip and conducted a round-robin vision encoder combination search again. As shown in the table below, when using SigLip as the base model, the model's performance consistently improves as the number of vision experts increases. This finding aligns with our observations when CLIP encoder serves as the base model.

We also substituted the CLIP model in Eagle-X5 with SigLip-L encoder, and the results are presented in the table below. While replacing the CLIP encoder with SigLip encoder leads to some fluctuations across different benchmarks, the overall performance remains very similar. Notably, the pre-alignment strategy yields a more significant performance improvement, further emphasizing the effectiveness and necessity of pre-aligning different vision encoders.

On the other hand, we also observed that SigLip encoder is not an obvious win over CLIP encoder in our experiments. We believe the main reason is that instead of using 336x336 resolution input, we modify the CLIP encoder to take 448x448 resolution input and unfrozen it during fine-tuning, which leads to a substantial improvment over the original one.