Improving Multi-modal Large Language Model through Boosting Vision Capabilities

Thanks a lot for your time and feedback. Below are address all raised concerns of the paper.

Q1. I think the proposed MM-LoRA is greatly inspired by some previous works like P-LoRA [1] in InternLM-XComposer2, visual-only modules in mPLUG-Owl2 [2] and CogVLM [3], which somehow reduces the novelty of MM-LoRA. The authors should tell the differences between MM-LoRA and these methods, along with some experiments on effectiveness and efficiency to prove the necessity of MM-LoRA.

A1. Thank you for the reviewer’s valuable question! Indeed, the design of MM-LoRA was inspired by works such as P-LoRA, mPLUG-Owl2, and CogVLM. However, these methods still have limitations in multi-modal tasks, while MM-LoRA introduces a novel approach to address these challenges. We will explain in detail the differences between MM-LoRA and these methods in the following sections, and provide corresponding experimental results to demonstrate its effectiveness and necessity.

Specifically, mPLUG-Owl2 only decouples the Key-Value (KV) in the Self-Attention mechanism but does not deeply explore the varying importance of visual and language modalities in multimodal tasks. InternLM-XComposer2 introduces Visual LoRA to enhance visual representation. However, during training, the LLM decoder remains involved, keeping the visual and language modalities coupled, which may lead to feature competition. In contrast, CogVLM adopts a simplified version of MM-LoRA, implementing a fully decoupled strategy where the same number of parameters are allocated separately to train the visual and language modalities.

When beta and gamma are set to 0.5, MM-LoRA's design is similar to CogVLM's full decoupling strategy. However, experiments show that this is not the optimal configuration. The importance of the visual and language modalities is not balanced in multimodal tasks. By adjusting the values of beta and gamma, the learning space can be more effectively allocated, thus improving performance. Comparative experimental results indicate that the model performs significantly better with configurations such as beta = 0.25 and gamma = 0.75 compared to beta = 0.5 and gamma = 0.5. The specific experimental results are shown in the table below.

Thanks a lot for your time and feedback. Below are address all raised concerns of the paper.

Q1. There is a lack of comparison with the latest open-source VLMs: LLaVA-OneVision, Qwen2-VL, InternVL2, etc. While these methods may use higher-quality training data and achieve stronger results, it is essential for readers to be aware of the current SoTAs. You may also explain why direct comparisons may not be feasible. It is acceptable for a research paper to fall short of SoTA results due to data quality differences, but these results should still be presented for context.

A1. Thank you for the reviewer’s valuable feedback! We fully understand your concern regarding comparisons with the latest open-source VLM models (such as LLaVA-OneVision, Qwen2-VL, InternVL2, etc.). Below is our response: These latest open-source VLM models have indeed achieved stronger results, supported by high-quality training data. However, it is important to note that the large amounts of high-quality training data used by these methods (such as proprietary or domain-specific data) have not been made publicly available, making it unfeasible to directly replicate these methods and perform a fair comparison.

Nevertheless, to provide a more comprehensive context, we will include performance reports of these latest methods in the revised version, clearly outlining their data advantages and how they differ from our approach in the discussion. Additionally, our research focuses on exploring how structural improvements (such as Q-Ladder and MM-LoRA) can enhance visual perception abilities under the same data conditions, rather than solely relying on the expansion of data scale.

Thank you again for this important suggestion! We will include the relevant information in the updated version to help readers better understand the positioning and contributions of both current state-of-the-art methods and our approach.

Q2. MMVP is crucial for demonstrating visual capability, but only QLadder is ablated on the MMVP benchmark. Why not conduct an ablation of MM-LoRA on MMVP as well? This would provide stronger support for the claims.

A2. Thank you for the reviewer’s suggestion! Your proposal to perform ablation studies on MM-LoRA using the MMVP benchmark is very important, and we fully understand that this would provide a more comprehensive validation of the effectiveness of our method.

In the initial version, we primarily focused on ablation experiments for Q-Ladder on the MMVP benchmark to highlight its direct contribution to visual perception abilities. However, to further support our research conclusions, we have also conducted additional experiments to evaluate the performance of MM-LoRA on the MMVP benchmark.

The experimental results show that MM-LoRA not only significantly improves performance in multi-modal tasks (such as language generation and dialogue) but also demonstrates a strong supporting role in visual tasks.

In the revised version, we will include these experimental results and integrate them into the ablation study section to further showcase the synergistic effect of MM-LoRA and Q-Ladder in enhancing visual perception capabilities. Thank you again for your thoughtful suggestion. We will ensure that the analysis in this aspect is more comprehensive in the updated paper.

[1]Tong, Shengbang, et al. Eyes wide shut? exploring the visual shortcomings of multimodal llms. CVPR, 2024.

A3. Was the visual encoder tuning in Table 7 conducted at the pre-training or instruction fine-tuning stage?

Q3. Thank you for the reviewer’s question! In the experiments presented in Table 7, we ensured that all visual encoder adjustments (tuning) were validated during both the pre-training and instruction fine-tuning stages. This means that, whether in the model's pre-training phase or the instruction fine-tuning phase, we applied the same adjustment strategy to ensure consistency and fairness in the experimental results.

This setup is designed to comprehensively evaluate the impact of visual encoder fine-tuning on model performance across different training stages and further validate the effectiveness of our proposed method. Thank you again for your attention, and we will clearly outline this experimental detail in the revised version!

Thank you for your valuable feedback! Below are address all raised concerns of the paper.

Q1. The proposed method leverages additional learning parameters to enhance the visual capabilities of MLLMs. Recent studies (e.g., LLaVA-Next, VILA-1.5, Qwen-VL-2) have shown that simply improving image resolution using various (any resolution) techniques is a straightforward and effective way to address this issue. I am skeptical that the proposed method will achieve performance comparable to these AnyRes approaches, particularly on tasks requiring high resolution. The proposed method appears limited by the visual encoder, despite the incorporation of additional LoRA modules.

A1. Thank you for the reviewer’s comments! We understand your concerns and greatly appreciate your suggestions regarding the AnyRes method. Below is our detailed response to this issue:

First, it is indeed true that increasing image resolution is a direct and effective approach to enhancing visual capabilities. However, this method typically requires training the visual encoder at high resolutions, which can result in significant computational overhead. Additionally, simply increasing resolution does not address specific task-related challenges in visual representations (such as fine-grained perception or improved modality alignment), which are the key problems that Q-Ladder and MM-LoRA aim to tackle.

Regarding the models you mentioned, VILA-1.5 and Qwen-VL-2, we note that their data has not been open-sourced, making it difficult to directly replicate their results. To more clearly demonstrate the applicability of Q-Ladder within the AnyRes method, we conducted experiments within the publicly available LLaVA-Next framework and combined it with the AnyRes technique to validate the effectiveness of Q-Ladder.

The experimental results show that the introduction of Q-Ladder significantly enhances model performance in multi-modal tasks. Even in the high-resolution AnyRes setting, it demonstrates its unique advantage in performance improvement. This indicates that the design of Q-Ladder not only makes full use of the capabilities of existing visual encoders but also further enhances model performance on top of the resolution enhancement method.

Q2. The focus of this study is on the visual capability of MLLMs. However, only one ViT is examined, and there are no ablations on different ViTs. This raises doubts about the generalizability of the proposed approach.

A2. Thank you for the reviewer’s comments! We fully understand your concern regarding the generalization ability of the method across different visual encoders. To address this, we conducted further experiments using a variety of visual encoders, including CLIP-ViT-L, CLIP-ViT-H, and SigLIP encoders, to assess the applicability and robustness of our proposed method. The experimental results are as follows:

The experimental results demonstrate that both Q-Ladder and MM-LoRA significantly improve model performance across various benchmarks, regardless of whether the visual encoder is CLIP-ViT-L, the higher-capacity CLIP-ViT-H, or the SigLIP encoder with different training strategies. This strongly indicates that our method exhibits good adaptability and generality across different types and scales of visual encoders.

Q3. Are the benefits of Qladder/MM-LORA consistent across scales? If we increase the scale of LLM and Visual Encoder, will Qladder/MM-LORA still show benefits?

A3. Thank you for the reviewer’s question! To validate the performance of Q-Ladder and MM-LoRA across different model scales, we conducted a series of experiments to explore whether these modules continue to deliver significant performance improvements with larger LLMs and visual encoders.

First, we scaled the LLM from 7B to 13B to verify whether MM-LoRA remains effective as the model size increases. The experimental results are as follows:

It can be observed that, despite the increase in LLM size, MM-LoRA still improves the model's performance across multiple benchmarks. This indicates that the design of MM-LoRA provides consistent benefits across LLMs of different scales.

Next, we upgraded the visual encoder from ViT-L to ViT-H to further explore the benefits of Q-Ladder and MM-LoRA with larger-scale visual encoders. The experimental results are as follows:

The experimental results also demonstrate that these modules continue to enhance model performance with larger-scale visual encoders. It is important to note that due to the resolution limitation of the open-source ViT-H in the CLIP model, which is restricted to 224x224, we could only conduct experiments at this resolution. Nevertheless, even under these conditions, Q-Ladder and MM-LoRA still exhibited significant advantages.

In conclusion, the experiments show that Q-Ladder and MM-LoRA consistently deliver substantial performance improvements across different scales of LLMs and visual encoders, validating the effectiveness of their structural design in large-scale models.

Q4. MiscellaneousIs the beta gamma ration study consistent across a range of LORA ranks (say 64 - 1024)? here it was set to 256.

A4. Thank you for the reviewer’s question! Regarding the beta-gamma ratio, we set the rank of LoRA to 256 in our paper based on preliminary experimentation, aiming to balance model performance and computational efficiency. However, we recognize that this ratio may vary with different LoRA ranks.

To address your query, we further investigated how the beta-gamma ratio changes with LoRA ranks of 256 and 512. The results show that the ratio remains stable across these ranks, with only slight variations. Below are the experimental results:

The experiments show that the beta-gamma ratio demonstrates strong stability, remaining largely unaffected by changes in rank size.

We are currently conducting additional experiments to further validate the stability of the beta-gamma ratio, especially under a broader range of rank settings, and to explore its potential impact on model performance. Relevant results will be included in the revised version. We greatly appreciate your patience and suggestions!

Thank you for your valuable feedback! Below are address all raised concerns of the paper.

Q1. The summary section (line 519-520) mentions "achieving notable performance improvements even with limited data resources". However, the problem of limited data sources is not convincing.

A1. Thank you for your comment. We agree that the challenge of limited data might not be immediately clear. In the context of Multimodal Language Models (MLLMs), the key challenge lies in the alignment of different modalities (vision and language)[1][2]. While pretrained models benefit from large-scale data, aligning these modalities for specific tasks requires significant amounts of Supervised Fine-Tuning (SFT) data. Previous work has demonstrated that fine-tuning on high-quality, task-specific data is essential for optimizing the interaction between vision and language, especially for tasks like localization, color recognition, and detection.

What we mean by "limited data resources" is specifically the scarcity of SFT data needed to align these modalities effectively for multimodal tasks. Annotating such data is often time-consuming and expensive, particularly in specialized domains. Our approach focuses on optimizing model structure—through components like Q-Ladder and MM-LoRA—to improve performance even when such fine-tuning data is limited.

In summary, while large-scale datasets are used for pretraining, the bottleneck for multimodal tasks is the availability of sufficient SFT data for alignment, which our method helps address.

[1] Ye et al. mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration. CVPR, 2024.

[2] Liu, Haotian, et al. Visual instruction tuning. Neurips, 2024.

Q2. The contributions of the proposed components are unclear. For instance, the benefit of LORA for limited-data-adaptation has been well studied in the past (e.g. [1]). The importance of introducing additional visual tokens to visual encoders has also been shown in [2]. In the light of the prior works, the paper should more clearly distinguish it’s technical contributions.

A2. Thank you for the reviewer’s feedback! We understand your concerns regarding the similarities between our method and existing studies. Below, we will clarify our technical contributions to distinguish our approach from prior work.

First, regarding the application of LoRA in limited-data adaptation, although LoRA has been extensively studied, our MM-LoRA module introduces a fundamental distinction. MM-LoRA employs a multi-modal parameter decoupling design, which introduces separate LoRA modules for vision and language modalities, enabling each modality to learn dedicated parameters. This approach promotes more efficient multi-modal information fusion.

This design not only enhances the independent learning capabilities of each modality but also effectively strengthens their interaction. In contrast, previous LoRA implementations primarily focus on single-modality adaptation and do not achieve such decoupling and fusion. To validate this, we compared the performance of LoRA and MM-LoRA within MLLM models. The experimental results are as follows:

Second, regarding the introduction of visual tokens, while studies such as "Vision Transformers Need Registers" suggest that adding visual tokens can enhance the performance of visual encoders, the innovation of the Q-Ladder module goes beyond this. Q-Ladder introduces a learnable "ladder" structure that deeply aggregates intermediate-layer features from frozen pre-trained visual encoders (e.g., CLIP).

This structure preserves the generalization capabilities of the pre-trained encoder while further improving task-specific performance. Unlike simply adding visual tokens, Q-Ladder emphasizes the effective integration of intermediate-layer features from the visual encoder, thereby enhancing the model's capacity for fine-grained visual tasks. Additionally, we compared the performance of Q-Ladder with the prompt-based approach from [2]. The experimental results are as follows:

In summary, the key distinctions of our work compared to existing studies lie in the multi-modal parameter decoupling design and the introduction of the ladder structure. The combination of these two innovations enables our method to achieve significant performance improvements in multi-modal tasks under limited data conditions.

Thank you for your valuable feedback! Below are address all raised concerns of the paper.

Q1. In terms of motivation, the paper aims to resolve MLLM visual perception issues such as color recognition, object counting, small object understanding, and spatial location. However, the structural designs of QLadder and MM-LoRA do not seem specifically tailored to address these problems.

A1.Thank you for the reviewers' feedback! Our research is motivated by the limitations of MLLMs in multimodal feature alignment, visual information representation, and cross-modal interaction, which hinder their visual perception capabilities. To highlight these issues, we introduce specific examples such as color recognition, object counting, small object understanding, and spatial position awareness, emphasizing their prevalence in real-world scenarios and the necessity of addressing them.

To overcome these limitations, we designed two modules, Q-Ladder and MM-LoRA, to enhance the model’s adaptability in complex visual tasks through structural improvements:

1. MM-LoRA introduces two parallel LoRA modules tailored to the visual and language modalities, respectively, enabling modality-specific parameter learning. This design adapts to the unique characteristics of each modality and facilitates efficient integration of multimodal information.

2. Q-Ladder incorporates a learnable "ladder" structure to deeply aggregate intermediate representations of frozen pretrained visual encoders (e.g., CLIP image encoders). This approach preserves the general visual representation capabilities while addressing the shortcomings of pretrained models in specific tasks, thereby improving the visual representation ability of MLLMs.

To demonstrate the effectiveness of Q-Ladder and MM-LoRA in tasks such as color recognition, object counting, small object understanding, and spatial position perception, while minimizing the influence of data quality differences, we conducted experiments using a unified dataset. These experiments were evaluated based on subclass metrics from MMBench and MME Benchmark. The results indicate the following:

FP-C represents Fine-grained Perception (Cross Instance), and FP-S represents Fine-grained Perception (Single Instance). The introduction of Q-Ladder and MM-LoRA significantly improves MLLM performance across tasks including color recognition, counting, position estimation, localization, and fine-grained perception. Combined with the ablation studies presented in the paper, the results intuitively demonstrate the effectiveness of these modules in addressing the limitations of MLLMs’ visual perception abilities, while also validating the rationality and generality of their structural design.

Q2. leading to the impression that performance improvements may stem from data rather than a well-targeted structural design, which appears somewhat forced into explaining the results.

A2. We understand the concern about whether the performance improvements stem from the dataset rather than the structural design. To validate the genuine effectiveness of Q-Ladder and MM-LoRA, we ensured that all ablation studies in the paper were conducted on a unified dataset to guarantee experimental fairness.

Specifically, we verified the independent contributions of the modules through the following approaches:

1. Ablation Study: We incrementally added Q-Ladder and MM-LoRA to the baseline and observed performance changes across multiple benchmarks, clearly identifying the independent contributions of each module.

2. Unified Dataset: All experiments used the same dataset to avoid inconsistencies related to data quality or quantity.

3. Multi-Dimensional Evaluation: We evaluated performance across diverse benchmarks (e.g., MMBench, MME, TextVQA) using metrics such as color recognition, object counting, and localization. The results consistently showed that the structural designs of Q-Ladder and MM-LoRA led to significant improvements under identical data conditions.

The results demonstrate that the performance gains are due to the structural innovations of Q-Ladder and MM-LoRA, not merely the dataset, highlighting their effectiveness in enhancing the visual capabilities of MLLMs.