HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding

We appreciate your suggestions and feedback. In the following, we respond to the major concerns.

• Q1: This early-fusion design is conceptually similar to related work such as EVE, offering only minor innovations over EVE when combined with a customized training recipe.

Response: We distinguish our HaploVL from EVE through the following differences:

These differences highlight our architectural innovations and efficiency in training, which contribute to the superior performance of HaploVL compared to EVE.

• Q2: The claim about "end-to-end large multi-modal model in a single transformer"

Response: We acknowledge that the model employs a dual-decoder architecture during training. However, this dual-decoder setup is used exclusively for the training phase. During inference, both the image and text inputs pass through the same single transformer model. This aligns our approach more closely with end-to-end processing. Other models like EVE also utilize multi-stage training strategies.

Furthermore, we have experimented with jointly training all components from the start using a unified objective. The results indicate that "training all components from the start using a unified objective" yields lower performance compared to our two-stage training approach. Specifically, the average score across 5 benchmarks was 60.5 for the two-stage method, versus 55.5 for the one-stage joint training.

• Q3: The subsequent explanation in lines 210–215 does not sufficiently justify the claim that the architecture is encoder-free.

Response: First, we do not claim that HaploVL is encoder-free. Second, we outline the differences between our pre-decoder and Vision Transformer (ViT) architecture:

• Q4: There should be Llama-3-8B based VLMs, e.g. MiniCPM-V2.5 and Llama-3.2.

Response: We will include the results of MiniCPM-V2.5 and Llama-3.2 in our main table for comparison. However, it should be noted that MiniCPM-V2.5 utilizes 778M training data, whereas our best model is trained on less than 6M samples. Additionally, we fully leverage open-source data, which ensures our model's ease of reproducibility.

• Q5: The teacher model of the text embedding.

Response: It is the text embeddings of the language model. We will clarify this in the final version.

• Q6: Ablation study for using the same pre-trained encoders and LLM.

Response: First, we report the total parameters of the two decoders. The parameters of our two decoders are almost equivalent to those of EVE (7.3B vs 7B). Second, we conducted comparisons using the same LLM (Vicuna-7B) and equivalent data with LLaVA-1.5-7B. Our findings demonstrate that HaploVL-Vicuna-7B exhibits superior performance compared to EVE-7B and surpasses LLaVA-1.5-7B on fine-grained perception tasks.

These ablation results validate our architectural innovations and demonstrate significant performance enhancements.

Thank you for your suggestions and feedback. We respond to the major concerns in the following.

• Q1: The comparison method is not detailed enough.

Response: Due to page limitations, we have compared our model against several widely-recognized methods, which include very mainstream approaches. For single-transformer LMMs, we have also included comparisons with the latest models. To provide a more comprehensive comparison, we will add the results of more state-of-the-art models to our main table.

However, regarding recent state-of-the-art separate LMMs, such as QwenVL-2 and InternVL-2.5, it's important to note that these models used closed datasets, although their model weights were released. According to their technical reports, QwenVL-2 utilized 1.4 trillion tokens during pre-training, and InternVL-2.5-7B used 142 billion tokens. In contrast, our HaploVL-7B relies solely on open-sourced data, using only 1.2 million samples (~7 billion tokens). This ensures that our model can be easily reproduced by the research community to build their single transformer models.

• Q2: The necessity of this design.

Response: Our design aims to explore a novel architecture that utilizes a single transformer during inference. Building on the well-established vision knowledge acquired from web-scale image data by Vision Transformer, we leverage this by initializing our pre-decoder with ViT and employ it as the teacher model. This allows us to inherit its vision knowledge and significantly reduce the required data and training costs compared to other early-fusion and single-transformer models. For example, while single-transformer LMMs like EVE and Emu3 have notable performance gaps when compared to separated LMMs such as LLaVA, our approach strives to narrow this performance gap. By integrating the strengths of ViT in the initial stages, we can achieve enhanced efficiency and performance, making our model a compelling candidate despite the apparent complexity.

• Q3: The strong language model may narrow the performance gap.

Response: We conducted comparisons using the same LLM (Vicuna-7B) and equivalent data with LLaVA-1.5-7B. This can eliminate the effect of the strong language model. Our findings demonstrate that HaploVL-Vicuna-7B exhibits superior performance compared to EVE-7B and surpasses LLaVA-1.5-7B on fine-grained perception tasks.

These results validate our architectural innovations rather than relying solely on the superior capabilities of a strong language model. This demonstrates that our design contributes significantly to performance enhancement.

Thank you for your insightful suggestions and feedback. We have responded to the key concerns in the details below.

• Q1: Limited Benchmark Comparisons.

Response: While recent state-of-the-art LMMs such as QwenVL2-7B and InternVL2.5-7B achieve higher MMBench scores, it is important to note that these models use closed data despite releasing their model weights. According to their technical reports, QwenVL2-7B utilized 1.4 trillion tokens during the pre-training phase, and InternVL2.5-7B used 142 billion tokens. In contrast, our HaploVL-7B model relies solely on open-sourced data, comprising only 1.2 million samples (~7 billion tokens). This significantly lower data usage underscores the efficiency of our approach. Moreover, our reliance on open-sourced data ensures that our model can be easily reproduced by the research community to build their single transformer models. Nevertheless, we acknowledge the importance of more recent benchmarks and will incorporate the results of these state-of-the-art models into our main table for comprehensive comparison.

• Q2: Context-Length Limitations

Response: The context length of HaploVL is indeed scalable. Specifically, with an input resolution of 336, the context length is 2048 tokens, and it extends to 6144 tokens when the maximum image size is 672x672. Even at a context length of 2048 tokens, HaploVL-Vicuna-7B (HaploVL-7B) demonstrates superior performance compared to LLaVA-1.5-7B on the MMVP, MMStar, and SEED benchmarks.

• Q3: Impact of Retaining Visual Supervision in Stage 2

Response: We retain a vision loss by adding an image decoder during the second stage of our method. This approach is similar to the Masked Autoencoder (MAE), where image embeddings are decoded into RGB images. To assess the impact, we conducted experiments using the LLaVA-665K instruction data.

Models trained with vision loss perform worse, with GQA and MMStar scores dropping from 62.5 to 60.8 and from 34.5 to 34.0, respectively. This is because additional vision loss conflicts with the textual loss used in multimodal understanding. We will include this result in the latest version.

• Q4: Baseline Fairness

Response: Due to the high workload involved in reproducing the EVE model (35 million data samples and 2 A100-80G GPUs running for 9 days), we compared the performance of HaploVL utilizing the same Vicuna-7B language model. The results demonstrate that HaploVL-Vicuna-7B outperforms EVE-7B. Specifically, on the SEED benchmark, HaploVL-Vicuna-7B achieves 67.5%, while EVE-7B (based on Vicuna-7B) only scores 54.3%.

Furthermore, when utilizing the same Qwen2.5-7B language model, our HaploVL-7B-Pro also surpasses the performance of the improved EVE-2.0 (Qwen2.5-7B).

These results confirm that the performance improvements are attributable to HaploVL’s architectural innovations rather than just the superiority of the LLM backbone.

Thank you for your valuable suggestions and feedback. We address the primary concerns as follows.

• Q1: Clarify more beyond stating that early fusion leads to efficient data usage.

Response: We propose utilizing a pre-decoder to fuse image and text data in the early stages of processing. This pre-decoder is designed to inherit prior vision knowledge from a visual model (like ViT), thereby requiring minimal additional data. The inherited prior vision knowledge enables the pre-decoder to effectively integrate visual information with textual data using fewer samples. Moreover, empirical results demonstrate that this method achieves superior performance with the same LLM and vision teacher, as evidenced by the MMStar benchmark scores (HaploVL-Vicuna-7B: 34.5 vs EVE-Vicuna-7B: 28.2).

• Q2: Used for image generation.

Response: Our current work verifies that this method is feasible for image understanding. In order to explore image generation, we introduce a vision loss by adding an image decoder during the second stage of our method. This approach is similar to the Masked Autoencoder (MAE), where image embeddings are decoded into RGB images. To assess the impact, we conducted experiments using the LLaVA-665K instruction data.

Models trained with vision loss perform worse, with GQA and MMStar scores dropping from 62.5 to 60.8 and from 34.5 to 34.0, respectively. While the symmetrical usage of visual and text modalities seems plausible, the introduction of vision loss for image generation currently degrades the performance of our model in image understanding tasks. This is because additional vision loss conflicts with the textual loss used in multimodal understanding. We plan future experiments to further explore and possibly mitigate these conflicts to successfully integrate image generation capabilities.