Dense Connector for MLLMs

Q1.Can DC be compatible with the visual resampler architecture?

This is a very worthwhile research question. Dense Connector (DC) is compatible with models similar to BLIP-2, which have visual resampler or Qformer structures.

Notably, models like BLIP-2 use visual resampler to obtain learnable queries as final visual representations, which still need MLPs for converting visual queries into the input space of the language model in structures like BLIP-2. We can replace these MLPs with DC to enhance visual perception. Thus, the DC can be used wherever a vision encoder and a connector are present.

Specifically, the visual resampler compresses high-level features into learnable queries. We then use an interpolation function to downsample tokens of different layer features to align the number of queries. DC is then used to transform these features from different layers into the input space of the language model. The table below shows our experimental results: using the visual resampler architecture, DC improved the average performance by 2%.

It is important to note, as referenced in [1], that the larger number of parameters in the visual resampler leads to poorer convergence. As a result, when trained on the same data, the performance of the visual resampler is suboptimal. Nonetheless, the table above demonstrates that DC is compatible with the visual resampler and enhance its performance.

[1] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q2.Summarize the model architectures compatible with Dense Connector

Thank you for pointing this out! We agree that summarizing the compatible scope of model architectures for DC would enhance the clarity and comprehensiveness of this paper.

DC can be widely applied to current MLLM architectures. In current MLLMs, any model utilizing a vision encoder can integrate the DC. The reason is that, regardless of whether the models are based on MLPs (LLaVA) or Qformers (BLIP-2), they require linear layers to convert visual features into the language space. Specifically, LLaVA employs two linear layers for high-level feature transformation, while BLIP-2 initially uses Qformer to obtain learnable queries, followed by a single linear layer for feature conversion. This linear layer is where the DC comes into play. Therefore, any model using a vision encoder can be compatible with the DC.

In summary, in this paper, we applied DC to a wide range of different architectures, including LLaVA-1.5, Mini-Gemini, LLaVA-NeXT (please refer to reviewer sj7M's Q1), and the Visual Resampler architecture. We will include these in the revision.

Thank you for the reviewer's comments.

We used a visual resampler based on Flamingo [1], with 64 query tokens, 6 layers, a hidden size of 1024, and 16 heads. In the first stage, we fine-tune the visual resampler and the linear layer that follows it, while freezing the ViT and LLM. In the second stage, we fine-tune the visual resampler, the linear layer, and the LLM, keeping only the ViT frozen.

Thank you very much for your time and comments! Please let us know if there are any further questions that need clarification.

[1] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in neural information processing systems 35 (2022): 23716-23736.

Q1. More comparisons with COMM [1]

[1] combines visual features from all layers by simply adding them together, which can lead to information loss. Additionally, ViT's low, middle, and high layers contain different information. Therefore, COMM's approach of adding them all together lacks prior knowledge.

In contrast, DC groups multi-layer visual features, providing prior knowledge during integration. Additionally, DC concatenates multi-level features along the channel dimension, effectively utilizing the dimensional transformation characteristics of MLPs connectors. It achieves feature fusion and transformation without additional modules. The tables below show the performance comparison between [1] and DC.

Would authors provides any possible fair comparisons with the work [1] (Only CLIP vision encoder)?

To ensure a fair comparison, we fine-tuned the model using LLaVA-1.5 data and a single ViT CLIP-L/336px. For COMM, we used LLN-Layerscale(all) [1] for integrating visual features.

The results above indicate that DC is a more effective method, achieving an average performance improvement of 2.1% compared with [1].

[1] Jiang et al., From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models, Arxiv 2023.

Q2.Statement about DC

While this work outlines three architectural configurations for the DC, it primarily employs a single variant – the Dense Channel Integration (DCI).

Current MLLMs research primarily focuses on high resolution and data, with less attention to multi-layer features. Previous work [1] has simply added visual features together, lacking further discussion. Our paper fills this gap by being the first to extensively discuss the fusion of multi-layer visual signals along the token (STI) and channel (SCI, DCI) dimensions.

We explored three instantiation methods, all enhancing MLLM performance across various settings. Among them, DCI achieved the best performance, so we primarily adopted it for scaling-up experiments.

It is important to note that the DC module serves more as an empirical trick rather than a theoretical methodology.

DC was designed to enhance MLLM performance from a visual perspective. Inspired by DenseNet and FPN, we explored the use of multi-layer visual features in MLLM and sought effective model designs. Thus, we developed DC, which utilizes multi-layer features in a way distinct from previous works. This module is tailored to MLLM, utilizing the dimension transformation of connectors to fuse multi-layer visual features.

Importantly, we believe this work benefits the community. First, DC outperforms previous connectors, demonstrating applicability across architectures (LLaVA-1.5, MGM, LLaVA-NeXT and Qformer). Second, we address the gap in integrating multi-layer visual features from the token and channel dimensions. Finally, we hope this paper will attract more attention to the visual modality in MLLM.

Q3.Performance of DC on MGM with Vicuna-13B

I think the performance gain is limited, especially on high-resolution MGM.

We believe the improvement brought by DC is significant. DC improved performance on GQA, SQA and MMB by 1.8, 2.7 and 2.5, respectively. In comparison, a more powerful ViT (CLIP-L->SigLIP) only improved GQA, SQA, MMB by 0.4, 1.0 and 1.6, and a stronger LLM (7B->13B) only improved by 1.2, 2.5 and 3.0.

Based on reviewer sj7M's suggestion, we extended DC to other high-resolution methods LLaVA-NeXT, achieving performance improvements. Please refer to reviewer sj7M's Q1.

I encourage the authors to extend their comparisons utilizing a Vicuna-13B model based on the MGM framework.

We extended the MGM experiments to Vicuna-13B with additional benchmark results. The table shows that DC can enhance MGM-13B's performance across various benchmarks.

Q4. DC's performance on larger models

It is speculated that the incremental performance gain might become even less pronounced when employing larger language models (LLMs)

Thank you for highlighting this concern. We will address it from two aspects and hope our response alleviates the reviewers' concerns.

• Due to limited resources, we previously used LoRA for fine-tuning, which affected the performance. Here, we provide a comparison of the 34B model w/ and w/o the DC under the LoRA fine-tuning. The results show that DC consistently enhances the model's performance in this scenario.

• We utilized more resources (32 A100 GPUs) to fully fine-tune 34B model, demonstrating that DC can achieve excellent results with larger models.

Q5.Bar charts for clearer comparisons.

Thanks. We provided bar charts in the PDF for clearer comparisons.

Q6. Combine multiple vision encoder

DC can be further to combine multiple vision encoder, like DINOv2.

We extended DC to multiple vision encoder, structures. Based on the results in [2] and our findings, we discovered that DINOv2 may not perform well. Therefore, in addition to the combination of CLIP and DINOv2, we also combined CLIP with ConvNeXT. The results show that the CLIP and ConvNeXT performs better.

[2] Tong et al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, CVPR 2024.

Thank you for the reviewer's comments. Due to the word limit, we only provided a portion of the benchmark results in our previous response. We now offer a more comprehensive performance comparison.

Q1.More comparisons with MGM

Thank you for your feedback. We agree that adding more benchmark comparisons will be helpful for this paper.

We provide further comparisons between MGM-7B and MGM-13B here. According to the table below, DC improved performance across these benchmarks, with an average increase of 1.1 on MGM-7B and 1.9 on MGM-13B.

In current MLLMs, no matter which model we consider, there are inevitably instances where performance gains on certain benchmarks are minimal. Our experiments trained various models across different settings to validate the generalization of DC, including different vision encoders, LLMs, training data, modalities, and architectures. Given the diversity of settings, it's understandable that some results may appear less pronounced.

We provided official results to show that even representative models (LLaVA, MGM) are unable to achieve significant improvements across all benchmarks. For example, in LLaVA-1.5, both the 7B and 13B models achieved the same score of 85.9 on POPE. In LLaVA-Next, the 7B to 13B models only improved by +0.7 on POPE, +0.7 on MathVista, and +0.6 on MMMU. And expanding LLaVA-1.5 13B to the high-resolution LLaVA-NeXT yielded only a 0.3 improvement on POPE. Similarly, in MGM, when extended to higher resolution, MGM-13B showed improvements of +0.1 on MMBench, -0.8 on MMMU, and +0 on Math.

We want to convey that, like representative models in MLLM such as LLaVA and MGM, even they cannot achieve consistent improvements across so many benchmarks.

We analyzed that, on one hand, the POPE is relatively simple, with results already being high, making further improvements more difficult. Additionally, POPE evaluates the model's ability to handle hallucinations, and since neither LLaVA nor DC is specifically designed to address this, the gains on POPE are both limited. On the other hand, benchmarks like SQA and GQA often rely more on language capabilities than visual signals. These benchmarks don't require strong visual perception to perform well on VQA tasks. For instance, the test question in SQA below can be answered without any visual input. Since DC is designed to enhance visual perception, its impact on SQA and GQA may be limited in these cases.

SQA: "question_id": "2828", "prompt": "<image>\nWhat is the capital of Iowa?\nA. Davenport\nB. Helena\nC. Lansing\nD. Des Moines\nAnswer with the option's letter from the given choices directly."

DC has demonstrated noticeable performance gains across most benchmarks, achieved with almost no additional overhead. For benchmarks that rely more on visual perception, such as VQAv2, MM-Vet, and MMBench, DC improved MGM-13B by 1.1, 3.8, and 2.2, respectively.

Q2.More comparisons with COMM

Thank you for your suggestion! Adding more benchmark comparisons will further strengthen this paper.

We offer additional benchmark comparisons with COMM, showing that DC's average performance improved by +1.4% over LLaVA and by +1.7% over COMM.

We hope this result convinces the reviewer that DC is robust across various benchmarks and outperforms other similar methods.

Thank you very much for your time and comments! Considering that the deadline for discussion is approaching, please let us know if there are any further questions that need clarification.

Q1. Combining Dense Connector with LLaVA-NEXT's AnyRes technology

Thanks! This is a very worthwhile question to explore. We extended the Dense Connector (DC) to dynamic resolution scenarios. Using dynamic resolution, DC remains effective. Compared to the baseline, DC improved performance on TextVQA and MMMU by 1.1% and 2.2%, respectively. Additionally, using only LLaVA-1.5 data, DC outperformed LLaVA-NeXT on several benchmarks.

Q2. Explanation regarding the performance of larger language models

Thank you for raising this concern. We will address it from two aspects and hope our response alleviates the reviewers' worries.

• In previous experiments, constrained by limited computational resources, we used LoRA to reduce memory consumption for fine-tuning the 34B model. Here, we present a comparison of the 34B model w/ and w/o the DC under LoRA fine-tuning. The results indicate that the inclusion of DC consistently enhances the model's performance in this context.

• Now, with more computational resources, we fully fine-tuned the 34B model, addressing concerns about DC's performance on larger language models. The results are shown in the table below.

Q3.Will fine-tuning ViT improve performance?

Thanks! That's a great question. We fine-tuned ViT using the settings from LLaVA-NeXT, specifically a learning rate of 2e-6 in the second stage. The results indicate that fine-tuning ViT can further improve the performance of DC.

Q4. About training time and inference time

For the Sparse Token Integration method, the addition of visual tokens results in a 20% increase in training and inference time. However, for Sparse Channel Integration and Dense Channel Integration, the training and inference times remain unchanged. Specifically, using 8 A100 GPUs (40G) to train the Vicuna-7B model, DC takes approximately 4 hours in the first stage and 12 hours in the second stage, similar to LLaVA-1.5.