Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering

WK3: 3B Model Outperforms 7B and 13B

A3: We appreciate the reviewer’s attention to the performance discrepancies noted in Tables 3. We are quite confident that this is not an anomaly and would like to provide a detailed explanation to clarify this observation:

• LLM Differences: The 3B parameter model utilizes Phi-2 as its base LLM, which has demonstrated superior performance on several benchmarks due to cleaner training data and higher-quality pretraining compared to Vicuna v1.5, which is used in the 7B and 13B models [1].

• Visual Encoder Pairing: The 3B model is paired with the SigLIP visual encoder, while the 7B and 13B models use CLIP. This design choice aligns with the methodology of TinyLLaVA, which shows that combining Phi-2 and SigLIP outperforms the 7B and 13B configurations of LLaVA 1.5 across various tasks [2].

• Fitting Ability: In Table 3, the evaluation focuses on fitting capability on a single dataset. Smaller models like the 3B tend to converge faster and fit better with the same number of training epochs, which aligns with established neural network training dynamics.

This performance gap highlights the importance of our LLaVA Steering Factory framework, which enables researchers to freely explore different combinations of components. This flexibility facilitates deeper insights into how design choices affect performance, making it a valuable tool for future research.

We hope this explanation resolves the reviewer’s concerns and highlights the motivation and flexibility offered by our proposed framework.

[1] Li, Yuan-Fang et al. “Textbooks Are All You Need II: phi-1.5 technical report.” Arxiv

[2] Zhou, Baichuan et al. “TinyLLaVA: A Framework of Small-scale Large Multimodal Models.” Arxiv

Thanks for the rebuttal. After reviewing your response, I have a few questions:

1. While the improvements reported for MMMU, POPE, and SQA are evident, they appear relatively modest and could potentially be attributed to test-time fluctuations, such as adjusting hyperparameters like the temperature. Could you provide further evidence or analysis to demonstrate the superiority of your approach beyond these minor fluctuations? For instance, are there any scenarios or metrics where your method offers clear advantages?

2. I noticed that you use different backbones for the 3B, 7B, and 14B models. This could introduce confounding factors and potentially interfere with the robustness and scalability conclusions of your method. Moreover, it may obscure the ability to demonstrate its scalability with respect to scaling laws. Would it be possible to include experiments in the future using the same backbone for all model sizes to ensure a consistent basis for comparison and to better evaluate the scalability of your approach? Additionally, why are the baseline experiments based on the 3B LLM? I believe llava1.5-7B is more general and representative.

3. If your proposed method is indeed effective, have you considered incorporating it into the finetuning stage? Would this not further improve the model's performance? Exploring this direction could provide additional insights and demonstrate the flexibility and utility of your approach.

WK4: Is Increasing Attention Scores in Deep Layers Necessary?

A4: We appreciate the reviewer’s reference to FastV and its observation that the role of visual attention diminishes in deeper layers. We share a similar observation and would like to clarify how increasing attention scores in deep layers is still necessary, without conflicting with the findings of FastV.

• Experimental Validation on Layer-wise Attention: To further validate our approach, we conducted experiments to assess the performance when applying MoReS to shallow, middle, and deep layers of MLLMs. The results show that MoReS applied to deep layers (16-31) consistently achieves the best performance. In MLLMs, the deep layers of the LLM primarily encode high-level semantic features, enabling the model to focus on abstract and complex reasoning tasks. Retaining high attention scores for a few critical tokens in these layers ensures that key visual information remains influential during the inference process. This reinforces the idea that increasing attention scores in the deep layers is both effective and necessary for optimal performance.

• Visual Attention Distribution and Consistency with FastV: As shown in Figure 1 in our current manuscript, the sum of visual attention scores across layers reveals a significant degradation in deep layers (25-31), which aligns with the findings of FastV. Notably, the distribution of visual attention remains sparse in these layers, with only a few tokens carrying the majority of the attention. This sparsity can further benefit token pruning strategies like FastV. Additional visualization results can be found in the Appendix A.3 for further reference.

We view methods such as FastV as complementary approaches for efficiency optimization, which can work synergistically with MoReS. While MoReS focuses on rebalancing modality representations, FastV’s optimization techniques can be seamlessly integrated with MoReS to enhance computational efficiency, and MoReS can help FastV perform more effective visual token pruning. We are actively exploring the incorporation of various efficiency optimization methods, including FastV, into our LLaVA Steering Factory. In the final version of our paper, we will update the relevant content to provide a more detailed breakdown of these experiments and the associated performance improvements.

WK2: Text-Only Tasks

A2: We sincerely appreciate Reviewer NCDD’s emphasis on evaluating MLLMs' capabilities on text-only tasks. We fully agree that this is a crucial aspect of assessing their generalizability across different modalities.

We quickly tested the text-only tasks given your comment, the results show that MoReS achieves superior performance on two widely recognized LLM benchmarks for text-only tasks. One of the key advantages of MoReS is that it preserves 100% of the world knowledge in LLMs, unlike other tuning methods that may result in performance degradation on text-only tasks [1].

The results clearly demonstrate that MoReS excels in text-only tasks, further emphasizing its ability to retain and effectively leverage the inherent world knowledge stored in LLMs. This capability showcases MoReS’ generalizability not only for multimodal tasks but also for text-dominant tasks. In the revised version of our paper, we will provide a more detailed analysis of these comparative experiments to further emphasize the advantages of MoReS in both text-only and multimodal contexts.

[1] Zhang, Yi-Kai et al. “Wings: Learning Multimodal LLMs without Text-only Forgetting.” NeurIPS 2024.

We would like to begin by expressing our appreciation to Reviewer NCDD for recognizing the innovative nature of our representation steering approach aimed at tackling modality imbalance. We also value the acknowledgment of our open-source initiative, the LLaVA Steering Factory, which has made a significant contribution to the research community.

WK1: Is a higher visual attention score necessarily better?

A1: We understand the concerns regarding the necessity of increasing attention scores for performance. We agree that increasing visual attention scores does not always guarantee improved performance across all tasks. However, we believe optimizing visual attention scores strikes a better balance between performance and trainable parameters in MLLMs.

MoReS addresses this by steering representations rather than adjusting parameters. Many existing PEFT and full fine-tuning methods focus on parameter adjustments, often neglecting the importance of visual information. Our analysis shows that this can lead to parameter redundancy and disruption of the model’s inherent knowledge. MoReS retains 100% of the LLM's language capabilities while rebalancing the attention distribution, achieving superior performance compared to IA3, which also increases visual attention but fails to fully preserve the LLM's world knowledge.

By increasing visual attention scores with minimal trainable parameters, MoReS substantially improves the performances on visual understanding tasks such as MMMU, POPE, and SQA.

MoReS achieves state-of-the-art performance on multi-disciplinary tasks requiring advanced reasoning, while reducing hallucination rates and maintaining parameter efficiency. In the revised version, we will provide more detailed comparative results to emphasize the strengths of our approach.

Q2: Considered as an engineering Contribution of LLaVA Steering Factory:

A2: We thank Reviewer jb5P for recognizing the value of our LLaVA Steering Factory as a significant contribution to the research community. As one of the key engineering and open-source aspects of our paper, LLaVA Steering Factory provides a highly versatile and comprehensive platform to support multimodal research and experimentation.

During our initial exploration of modality imbalance, we found it particularly challenging to verify results across multi-scale and multi-combination MLLMs on various benchmarks, especially when using all mainstream PEFT methods. This has significantly hindered researchers' academic progress. To better advance the research in this area, we developed the LLaVA Steering Factory, adhering to the software principles of modularity, simplicity, and extensibility. The framework enables all configurations to be implemented in a straightforward and unified scripting manner. The main features of the LLaVA Steering Factory are as follows:

• Broad Support for Mainstream Models and Benchmarks: LLaVA Steering Factory is designed to integrate most mainstream LLM backbones and visual encoders. It supports various training strategies and evaluation across a wide range of benchmarks, including both visual and text-only tasks, offering researchers unparalleled flexibility.

• Support for Diverse Tuning Methods: The platform includes implementations of multiple tuning methods, including our advanced MoReS framework. These options cater to varying computational resource constraints, making it accessible to researchers with different hardware setups.

• Motivation and Community Impact: During our development of various MLLMs and tuning strategies, we often encountered inefficiencies and engineering challenges, especially when dealing with different fine-tuning methods and benchmarking requirements. Inspired by LLaMA Factory [1], a highly influential open-source project with over 34k stars on GitHub that fine-tunes over 100 LLMs and provides tools for fine-tuning and benchmarking, we developed LLaVA Steering Factory to address similar needs in the multimodal domain. Our goal is to further facilitate the research community by streamlining the process of building and evaluating MLLMs.

• Comparison with Previous Contributions: While earlier efforts, such as Tiny LLaVA Factory [2] and Prismatic VLMs [3], have made valuable contributions, LLaVA Steering Factory significantly extends their capabilities. Below, we provide a detailed comparison: | Factory | Multi-scale LLMs | Diverse Vision Encoders | Mainstream PEFTs | Text-only Tasks | Multimodal Benchmarks |Computational Optimization | Multiple Training Strategies | |:------|:-----:|:------:|:------:|:------:|:-----:|:------:|:------:| | TinyLLaVA | ✗ | ✓|✗ | ✗| ✓| ✗ |✓ | | Prismatic | ✓ | ✓| ✗ | ✗| ✓ | ✗|✗ | | LLaVA Steering (Ours) | ✓✓ | ✓ | ✓ | ✓ | ✓✓| ✓ | ✓|

• Commitment to Open Science: We are committed to releasing all code and scripts for LLaVA Steering Factory as soon as the double-blind review period ends, ensuring that our contributions can benefit the community immediately.

[1] Zheng, Yaowei et al. “LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models.” ACL 2024

[2] Jia, Junlong et al. “TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models.” ArXiv

[3] Karamcheti, Siddharth et al. “Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models.” ICML 2024

First, we would like to extend our gratitude to Reviewer jb5P for acknowledging the importance of our findings regarding modality imbalance in MLLMs. To the best of our knowledge, we are the first to thoroughly investigate this phenomenon when applying different tuning methods. We also deeply appreciate Reviewer jb5P's kind recognition of the LLaVA Steering Factory as a valuable contribution.

WK1&Q1: Trade-off between visual and textual understanding when rebalancing the modalities

A1: We understand the reviewer jb5P's concern that if we could still achieve good performance on images with both rich visual and textual information after increasing visual attention score. The answer is yes where MoReS will still perform good on tasks with both rich visual and textual information and the reasons are as follows:

MoReS maintains the model's ability to leverage its LLM-derived 100% world knowledge while incorporating visual context more effectively. This enables richer multimodal reasoning especially for images with both rich visual and textual information. Compared to other PEFT methods, MoReS achieves the highest performance on the MMMU benchmark while utilizing the fewest trainable parameters. MMMU is specifically designed to evaluate three essential skills in MLLMs: perception, knowledge, and reasoning. Notably, the predominant image types in this benchmark—diagrams, charts, and tables—contain a blend of rich visual and textual information. Additionally, we evaluated MoReS on the MMLU benchmark, which primarily focuses on textual information understanding. MoReS also outperformed other methods in this evaluation, leveraging its LLM-derived 100% world knowledge without compromising the pre-trained model's inherent capabilities.

We will update the relevant content in the final version of our paper to provide a more detailed breakdown of these experiments.

Thank you again for your insightful suggestions!

Thank you for your insightful comments and suggestions. We hope the updated manuscript resolves the points you raised and are glad to discuss any additional questions you might have!

WK3: Potentially High Runtime Overhead

A3: Thank you for your valuable feedback! We understand that inference overhead is an important consideration when evaluating the efficiency of MLLMs. We are aware of the overhead incurred in MoReS, however, there are several measures to decrease the overheads effectively.

Unlike LoRA, where the learned weights can be merged into the model’s original parameters to achieve zero computational overhead during inference, MoReS requires the linear transformation layers to remain in the computation graph of the MLLM. While this introduces a small overhead, we have worked to minimize it effectively.

To mitigate runtime overhead, we performed several experiments focusing on key factors: Subspace Rank, Steered Visual Token Rate and Steering Layer Configuration. These experiments helped us reduce the additional computational burden. Specifically, by choosing a 1% Steered Visual Token Rate, a Subspace Rank of 1, and employing a sparse Steering Layer Configuration, we achieved the minimum runtime overhead of about 0.08 seconds each sample. This is significantly lower compared to other PEFT methods, such as Adapter (0.3 seconds) and OFT (2.8 seconds).

We will update the relevant content in the final version of our paper to provide a more detailed breakdown of these experiments and the resulting performance improvements. Thank you again for your insightful suggestions!

WK2: LoRA Configuration Setting

A2: We understand that LoRA can be configured with a smaller rank to reduce trainable parameters. However, to ensure a fair and robust comparison, we selected a rank of 128 for LoRA in our work based on the following considerations:

• Empirical Evaluation of LoRA Rank Settings: We conducted a series of experiments to identify the optimal LoRA rank configuration for our comparisons. The results of these ablation studies indicate that a rank of 128 consistently delivers the best performance, establishing a strong baseline. Even with smaller ranks, such as 4, LoRA still requires significantly more trainable parameters (5.90M) than MoReS (0.16M), highlighting the parameter efficiency of our approach.

• Alignment with Influential Prior Work: Several influential works in this domain, such as LLaVA 1.5 [1] and TinyLLaVA [2], also adopt a LoRA rank of 128. Using the same configuration ensures consistency and comparability with widely accepted benchmarks and methodologies in the field.

Based on these two considerations, we selected a LoRA rank of 128 for our experiments. In the revised version of our paper, we will include these supplementary results and further clarifications to provide a more comprehensive justification for this choice.

[1] Liu, Haotian et al. “Improved Baselines with Visual Instruction Tuning.” CVPR 2024

[2] Zhou, Baichuan et al. “TinyLLaVA: A Framework of Small-scale Large Multimodal Models.” Arxiv

First, we would like to extend our gratitude to Reviewer dR1a for acknowledging the novelty of our representation steering method and the valuable contribution of LLaVA Steering Factory to the research community. Through comprehensive evaluation conducted via this framework, MoReS has demonstrated exceptional effectiveness.

WK1&Q1: Simply increasing visual attention scores may not directly enhance performance across all tasks.

A1: Thank you for the reviewer dR1a’s feedback! We understand the concerns regarding the Correlation Between Attention Scores and Performance. We believe that optimizing visual attention scores can help strike a better trade-off between performance and trainable parameters in MLLMs.

MoReS addresses this trade-off by focusing on representation steering rather than parameter tuning. Most existing PEFT methods and full fine-tuning approaches primarily adjust parameters, often overlooking the critical role of visual information. Through our analysis, this oversight can lead to parameter redundancy and unintended disruptions to the LLM’s inherent world knowledge.

By rebalancing visual attention scores with the fewest trainable parameters, we do observe a direct improvement on several visual understanding benchmarks, including MMMU, POPE, and SQA.

Through the enhanced visual attention scores, MoReS achieves state-of-the-art performance on diverse, multi-disciplinary tasks that require college-level subject knowledge and deliberate reasoning. Furthermore, it significantly reduces hallucination rates while maintaining parameter efficiency. In the revised version, we will provide a more detailed description of these comparative experiments to further highlight the advantages of our approach.

Thank you for your detailed review and valuable feedback. We hope our revisions have sufficiently addressed your concerns, and we’re happy to provide further clarifications if needed!

WK3: Insufficient explanation of the mechanism behind MoReS

A3: Thank reviewer vetu for acknowledging the effectiveness of our steering approach. We understand your concerns regarding the mechanism behind MoReS, particularly the rationale behind the chosen dimensionality, as this could provide stronger support for understanding how MoReS functions.

MoReS is a sparse, token-level representation tuning method, steering fewer than 1% of visual tokens while preserving textual token inference and LLM parameters. For clarity, we have included an UML diagram and a MoReS module flowchart in the Appendix A.2 to detail the implementation. Additionally, we will release the code alongside the LLaVA Steering Factory, ensuring transparency and enabling reproducibility of our method.

Regarding the rationale behind the chosen dimensionality, we had provided both theoretical and empirical analysis in the current manuscript to demonstrate the effectiveness of our chosen dimension.

• Theoretical perspective: We posit that MLLMs represent concepts as subspaces in high-dimensional vector spaces, consistent with previous work [2][3][4]. MoReS capitalizes on this by employing a downsampling function that projects selected high-dimensional visual representations into a lower-dimensional subspace using a low-rank projection matrix with orthonormal rows. This approach ensures that the subspace retains critical multimodal task features. Further theoretical details can be found in Appendix A.1.

• Empirical perspective: We conducted an ablation study on the chosen dimensionality, as presented in Table 5, to empirically validate the rationality of our choice. The evaluation results can confirm the efficacy of the selected dimensionality in maintaining performance while ensuring efficiency.

Thank you again for your valuable feedback! We will update the relevant content accordingly. If you have any further questions/concerns/suggestions, please do let us know.

[2] Maiorca, Valentino et al. “Latent Space Translation via Semantic Alignment.” NeurIPS 2023.

[3] Park, Kiho et al. “The Linear Representation Hypothesis and the Geometry of Large Language Models.” ICML 2024.

[4] Mikolov, Tomas et al. “Linguistic Regularities in Continuous Space Word Representations.” North American Chapter of the Association for Computational Linguistics (2013).

WK2: Insufficient ablation studies on the impact of removing linear transformations or varying the downsampling rate

A2: Thank you for highlighting the importance of additional ablation studies. We agree that these experiments will strengthen the validation of our method.

In the current manuscript, in fact, we had provided an ablation study on the downsampling rate (Subspace Rank) with Table 5. In addition to that, we now further study this problem with additional experiments to investigate the effect of removing linear transformations in specific layers.

In Table 5 of the current submission, we conducted experiments using different subspace ranks to modify the downsampling rate. The results indicate that MoReS exhibits relatively robust performance across various subspace ranks. Additionally, by selecting a smaller subspace rank, we can further improve computational efficiency.

Furthermore, as shown in the tables here, we conducted experiments applying MoReS with different fixed intervals and also evaluated its performance when applied exclusively to the shallow, middle, and deep layers. These experiments highlight that the choice of steering layers can effectively balance computational efficiency and performance. We suggest that, when using MoReS, it is optimal to apply it to all layers initially to achieve the best performance. Then, by skipping fixed intervals, we can further reduce inference overhead while maintaining performance. Regarding the choice of shallow, middle, and deep layers, we found that applying MoReS to the deep layers yields better performance. We believe that deep layers encode more abstract concepts and are more suitable for steering in the subspace. We will provide these supplementary results to further clarify this aspect in the final version of our paper.

First, we would like to extend our gratitude to reviewer vetu for acknowledging the efficacy of our MoReS method, a subspace-based representation steering approach. Below, we first summarize the key concerns of every weakness point and then provide our reply correspondingly.

WK1: The "steering" module might face challenges in achieving consistent performance across tasks that demand varying levels of visual or textual information.

A1: Reviewer vetu concerns that relying more on visual information may not perform well for certain tasks which require both world knowledge and visual information. We share Reviewer vetu's perspective in considering the roles that both modalities play in MLLM multitask scenarios and we also considered this when we design MoReS.

Specifically, MoReS preserves 100% of the pre-trained world knowledge in the LLM by neither modifying its parameters nor interfering with textual token inference. This design allows MoReS to excel in understanding both visual and textual information. Unlike many existing methods, which often alter model weights and risk degrading pre-trained knowledge [1], MoReS employs a representation-steering approach to selectively enhance the performance of the visual modality.

To demonstrate that MoReS performs consistently well across tasks with varying modality dependencies, we selected three representative benchmarks:

• POPE: Visual-dependent.
• HellaSwag: Textual-dependent.
• MMMU: Dependent on both modalities.

The results demonstrate that MoReS performs effectively across tasks requiring varying levels of visual or textual information. In the revised version, we plan to include the new comparison results to clarify MoReS's capability to handle tasks with varying modality dependencies.

[1] Zhang, Yi-Kai et al. “Wings: Learning Multimodal LLMs without Text-only Forgetting.” NeurIPS 2024.

Thank you for your time and thoughtful review. We hope the revisions have addressed your concerns. If there are any remaining questions or updates, we’d be happy to clarify further!