Unveiling Encoder-Free Vision-Language Models

We are grateful for your meticulous and insightful review. We have carefully considered your questions and polished our paper.

Q1: [Encoder-Free?] How do you justify classifying EVE as an "encoder-free" model?

(1) A more essential difference between PEL and image encoders is whether they involve strong inductive bias and semantic priors in abstracting visual representation by pre-trained models via pretext tasks. We introduce CA1 and CA2 mainly due to efficiency consideration and holistic information for raster-order token inputs respectively. Note that Table 4 shows that CA1 and CA2 are optional and have a less significant impact on performance.

(2) As discussed in ALL-Q1, we found that while PAL aids in early convergence and training efficiency, it becomes less crucial as the data scale significantly increases. Besides, we can completely remove it during inference, allowing EVE to function as a pure encoder-free architecture like Fuyu-8B. Hence, we attribute our work to the ‘encoder-free’ track.

Q2: [Almost Lossless?] The use of average pooling, which is inherently a lossy operation?

Good question. CA1 may not deviate from the "almost" lossless concept due to its dense feature aggregation strategy in a restricted receptive field, only using pooling features as the query. We adopt this design, mainly considering the trade-off between large token numbers and training efficiency, especially with limited resources.

Q3: [Performance Gap?] [Complexity Analysis?] [Flexibility Benefits?]

(1) We highlight that EVE shows potentially more promising scaling with pretraining duration, which is the key motivation behind our efforts to build encoder-free VLMs. EVE attempts to remove strong vision inductive bias and transmit visual signals almost losslessly for better scaling properties. In Figure 2 of the rebuttal PDF, we observe that encoder-based models often suffer from collapse. Only the (VE)-(ALL) training strategy avoids this issue by freezing LLM weights during pre-training and unfreezing them during the SFT stage. In contrast, EVE shows better scaling properties and gradually approaches the performance of well-developed encoder-based VLMs with only 33M data scale.

(2) Encoder-free VLMs are promising for scaling but require enormous training data to develop vision perception from scratch. Here, with only 33M pre-training data, our pioneering exploration currently lags behind but performs comparably to popular encoder-based methods. Interestingly, subsequent PaliGemma [a] also explores an encoder-free version via 1B pre-training image-text pairs, showing promising early results alongside its encoder-based counterpart across 37 validation datasets (see Figure 3 of the rebuttal PDF). They particularly mention that only the separate vision encoder i.e. SigLIP, has been trained with 40B image-text pairs, far greater than 1B data of encoder-free version. They also indicate that decoder-only VLMs may be a promising direction although currently suffering in training efficiency due to building vision perception from scratch.

(3) The 'Any' image ratio, simple architecture, and efficient deployment are bonuses of encoder-free VLMs. Recent studies on encoder-based VLMs reveal that (i) Due to the limitations of pre-trained encoders, existing VLMs exhibit vulnerabilities in basic capabilities rooted in visual encoding trade-off [b, c]. (ii) Various vision encoders show uneven levels of capability due to pretext pretraining tasks, relying heavily on the corrective capabilities of LLMs for multimodal understanding [d, e]. In contrast, encoder-free VLMs remove semantic priors in abstracting visual representation, theoretically allowing VLMs to autonomously acquire all available information. While 'any image ratio' and 'FLOPS gains' are natural benefits of the encoder-free approach, the primary reason for exploring an encoder-free model is its scaling efficiency with less inductive bias. In this premise, removing the vision encoder provides only a modest bonus in terms of flexible image input and deployment efficiency. Notably, the encoder-free track is still in early development and has a long way to explore its limits.

[a] PaliGemma: A versatile 3B VLM for transfer. Google DeepMind. arXiv 2407.07726.

[b] LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images. Xu et al. arXiv 2403.11703.

[c] HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models. arXiv 2407.08706.

[d] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. Tong et al. CVPR2024.

[e] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. Tong et al. arXiv 2406.16860.

Q4: The performance gap with some encoder-based VLMs? How to address it?

(1) The key issue with EVE is the massive data needed to build vision perception from scratch. Encoder-based models first train a separate visual encoder with billions of data points, while EVE constructs vision perception from scratch using only 33 million data points. Figures 2-3 show performance is far from saturated. Larger data scales and more computing resources are required for further exploration.

(2) Besides, several important factors deserve to be explored. We found image resolution, reducing modality perturbations, the base LLM’s size and capability, data size and quality, etc. also impact the scaling efficiency of encoder-free VLMs. For instance, pre-training EVE with 4M high-resolution data (1344 longest edge) without vision supervision, can match the performance of the original version, using 12M data and vision supervision. We would polish these factors and develop a fully and stronger encoder-free VLM in the follow-up study.

Q5: Figure 1 is never referenced in the main text. Figure 5 is mentioned after Figure 6 in the text, disrupting the logical flow.

Thanks for your kind reminder and we will polish them in the revised manuscript.

Your constructive comments are much appreciated. We have addressed all your points and revised the paper accordingly to ensure its improvement.

Q1: IIUC, Fuyu learns visual features from scratch, while EVE actually distillates from an existing CLIP-ViT-L-336px vision encoder, so it’s not fair to claim that EVE is better than Fuyu under the same “encoder free” setting. It’s more like “encoder distilled” vs. “encoder free”. Therefore, it would be more insightful if we could ablate encoder free vs. distilled in 4.3 (if the training stability issue can be solved), i.e., to investigate why EVE is much better than Fuyu.

(1) As discussed in ALL-Q1, while the vision encoder aids in early convergence, it becomes less crucial as the data scale increases. We introduce vision encoder distillation to improve training efficiency, especially with a limited data scale, which is less significant after 24M pre-training data in our experiments. This indicates that eliminating the vision encoder for a fully encoder-free VLM is practical during both training and inference.

(2) The LLM-guided pre-aligning stage is one crucial factor for training stability and scaling efficiency. Without this stage, even with enormous training data and vision distillation, training decoder-only VLMs remains highly challenging in Table 5 and Figure 6, suffering from model collapse and large performance gaps.

(3) Compared with Fuyu-8B, EVE adopts image resolution with 672 image longest edge during pre-training due to device constraints and efficiency consideration. For fairness, we attempt to pre-train EVE with 4M high-resolution data (1344 longest edge) without vision supervision, which can match the performance of our original version, using 12M data and vision supervision. This indicates that EVE has room for further improvement and could potentially widen the gap beyond Fuyu-8B.

Q2: As mentioned in the paper, the language ability is largely affected in stage 2. It’s very common to add an additional LM task and there have been many open-sourced corpuses already. It’s worth explaining why this common approach was not tried (though the paper suggests solving this in future work).

While maintaining language ability in stage 2 is a concern, it is not the primary challenge in developing encoder-free VLMs effectively. The main issue is addressing optimization problems and bridging the gap in multi-modality capabilities compared to encoder-based VLMs. We spent a long time exploring and finally discovered that an LLM-guided pre-aligning stage and vision encoder supervision are essential for training stability and early convergence, respectively. Considering text-only data is less critical at this stage, we decided to solve it in the follow-up study.

Q3: Even though EVE-7B is comparable with the selected encoder-based baseline models of the same size, it is still largely behind some recent models with even half size and lower resolution, such as PaliGemma-3B [1].

(1) Actually, such a comparison is unfair due to many notable differences.

• [Data Magnitude]: As mentioned before, EVE learns vision perception from scratch, where the vision encoder could be removed using the entire 33M pretraining data. However, PaliGemma’s SigLIP encoder has seen 40B image-text pairs during Stage 0, with Stage 1 and Stage 2 further processing about 350B and 90B tokens, respectively.
• [LLM Capability]: Despite being smaller, Gemma-2B exhibits similar capabilities to our Vicuna-7B.
• [Finetuning Stategy]: Table 1 in PaliGemma shows that they train a 'specialist' model for each task with specific hyperparameters, while EVE serves as a 'generalist' model for downstream tasks.

(2) We highlight that EVE shows potentially more promising scaling with pretraining duration.

In Figure 2 of the rebuttal PDF, we observe that EVE shows better scaling properties and gradually approaches the performance of well-developed encoder-based VLMs with only 33M data scale. Interestingly, PaliGemma also explores an encoder-free version via 1B pre-training image-text pairs, showing promising early results alongside its encoder-based counterpart across 37 validation datasets (see Figure 3 of the rebuttal PDF). They particularly mention that their vision encoder i.e. SigLIP, was trained with 40B image-text pairs, and indicate that decoder-only VLMs may be a promising direction although currently suffering in training efficiency due to building vision perception from scratch.

Q4: It’s unclear whether 35m data is enough (likely not?). Have you tried data scaling ablations?

35M data points provide valuable insights for constructing encoder-free VLMs but fall short of peak performance. Figure 1-2 in the rebuttal PDF show that performance is not yet saturated, and EVE offers better training stability and scaling efficiency. Due to budget and time constraints, we will scale the pre-training corpus to a billion scale in future work.

Q5: In line 234, it should be Table 3?

Thanks for the kind reminder. We will revise it.

Q6: It’s worth mentioning EVE inherits the same limitations from the resued LLM, i.e., Vicuna-7B (likely also including CLIP-ViT-L-336px).

This is a common problem for both encoder-based and encoder-free VLMs. We believe that as existing LLMs continue to advance impressively, these limitations, we think, would be properly addressed. Besides, encoder-free VLMs escape the limitations of pre-trained vision encoders, unlike their encoder-based counterparts.

Thank you for your thoughtful remarks. We have responded to all your queries and made the necessary changes to enhance the paper.

Q1: (((1))) EVE outperforms Fuyu-8B, but lags behind many encoder-based VLMs. why we need it? (((2))) Any explanations for the benefits of 'Any image ratio'? (((3))) No enough benefits of FLOPs(G) and Time(s) when considering the vision part and LLM part together.

(1) We highlight that EVE shows potentially more promising scaling with pretraining duration, which is the key motivation behind our efforts to build encoder-free VLMs. EVE attempts to remove strong vision inductive bias and transmit visual signals almost losslessly for better scaling properties. In Figure 2 of the rebuttal PDF, we observe that encoder-based models often suffer from collapse. Only the (VE)-(ALL) training strategy avoids this issue by freezing LLM weights during pre-training and unfreezing them during the SFT stage. In contrast, EVE shows better scaling properties and gradually approaches the performance of well-developed encoder-based VLMs with only 33M data scale.

(2) Encoder-free VLMs are promising for scaling but require enormous training data to develop vision perception from scratch. Here, with only 33M pre-training data, our pioneering exploration currently lags behind but performs comparably to popular encoder-based methods. Interestingly, subsequent PaliGemma [a] also explores an encoder-free version via 1B pre-training image-text pairs, showing promising early results alongside its encoder-based counterpart across 37 validation datasets (see Figure 3 of the rebuttal PDF). They particularly mention that only the separate vision encoder i.e. SigLIP, has been trained with 40B image-text pairs, far greater than 1B data of encoder-free version. They also indicate that decoder-only VLMs may be a promising direction although currently suffering in training efficiency due to building vision perception from scratch.

(3) The 'Any' image ratio, simple architecture, and efficient deployment are bonuses of encoder-free VLMs. Recent studies on encoder-based VLMs reveal that (i) Due to the limitations of pre-trained encoders, existing VLMs exhibit vulnerabilities in basic capabilities rooted in visual encoding trade-off [b, c]. (ii) Various vision encoders show uneven levels of capability due to pretext pretraining tasks, relying heavily on the corrective capabilities of LLMs for multimodal understanding [d, e]. In contrast, encoder-free VLMs remove semantic priors in abstracting visual representation, theoretically allowing VLMs to autonomously acquire all available information. While 'any image ratio' and 'FLOPS gains' are natural benefits of the encoder-free approach, the primary reason for exploring an encoder-free model is its scaling efficiency with less inductive bias. In this premise, removing the vision encoder provides only a modest bonus in terms of flexible image input and deployment efficiency. Notably, the encoder-free track is still in early development and has a long way to explore its limits.

[a] PaliGemma: A versatile 3B VLM for transfer. Google DeepMind. arXiv 2407.07726.

[b] LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images. Xu et al. arXiv 2403.11703.

[c] HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models. arXiv 2407.08706.

[d] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. Tong et al. CVPR2024.

[e] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. Tong et al. arXiv 2406.16860.

Q2: One minor question (not urgent, authors could simply ignore it if time does not allow): the benchmarks presented are normally simple VQA benchmarks, how would the model perform on detailed description tasks?

From the LMMs-Eval leaderboard [a], we observe that LLaVA-1.6 surprisingly achieves lower CIDEr scores than LLaVA-1.5 on COCO-Cap and Flickr-cap under the same model capacity. These description tasks do not well reflect the capability of multimodal models due to out-of-distribution issues and evaluation metrics, which are seldom evaluated by most existing VLMs.

[a] LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models. Zhang et al. arXiv 2407.12772

Thank you for your valuable comments. We respond to all questions you raise to address your concerns and make the necessary revisions to improve the quality of the paper.

Q1: The training is using an existing visual encoder as teacher, which kind of defeats the title of "encoder-free". Is this necessary? The contribution would be more valuable if it can be completely encoder-free (during both training and inference)

As discussed in ALL-Q1, we found that while the vision encoder aids in early convergence, it becomes less crucial as the data scale increases. Inspired by this, we empirically found that pre-training EVE with 4M high-resolution images (1344 longest edge) without vision supervision can match the performance of our original version, which used 12M data points and vision supervision. This indicates that eliminating the vision encoder for a fully encoder-free VLM is practical during both training and inference.

Q2: Related to 1., if there is a need to train the model on some out-of-domain datasets, does it need to first train the separate visual encoder, and then redo the 3 training stages described in this work? This makes the training even more complex than encoder-based VLMs.

Good question. Transferring the visual encoder initially may be beneficial due to its strong inductive bias and susceptibility to out-of-domain issues. Fortunately in our experiments, we discovered that after 24M pre-training data, vision encoder supervision becomes less essential for EVE to develop visual perception from scratch. This enables EVE to circumvent these issues by abandoning the vision encoder supervision and preserving the original three stages during the training process.

Q3: In Fig.6, what does “from scratch” mean exactly? Is the language model also randomly initialized?

Yes. "From scratch" denotes that the entire model is initialized randomly. We empirically discovered that training EVE from scratch faces extreme optimization challenges and usually suffers from model collapse.