From Bulk to Budget: Best Practices To Compress Multimodal Large Language Models

The experiments are not comprehensive with just limited model selection. Please include newer MLLM architectures like InternVL, CogVLM, MiniCPM-v;

We agree it benefits our study to evaluate our methods on a broader range of MLLM architectures. In response, we have extended our experiments to include the InternVL [1] model to test the generalizability of our best practices for pruning and recovery. The results are in the revision Appendix I. Our findings with InternVL confirm that the proposed methods are effective across architectures, demonstrating the applicability of our techniques beyond the models initially presented.

We provide an analysis of the results of the new model below. We observe that widthwise pruning offers better performance without recovery training. With InternVL, widthwise pruning retains 97.4% of the model’s original performance at a 15% compression ratio, compared to 96.7% for layerwise pruning, reinforcing its suitability as a default strategy in low-resource scenarios.

Additionally, we find that finetuning only the multimodal projector is sufficient at small compression ratios, where pruning minimally impacts the language model but disrupts multimodal alignment. These results reinforce our observations from earlier models and validate the transferability of our proposed practices.

Moreover, our initial findings show that combining supervised finetuning with intermediate representation distillation consistently yields the highest performance across compression ratios. With InternVL, this combined approach achieves 98.2% recovery at a 15% compression ratio and 87.2% at a 30% compression ratio, confirming its effectiveness.

Notably, All of the pruned models are recovery trained on only 3% of the original dataset, which highlights the data efficiently. Overall, the results from InternVL indicate that our methods generalize well to newer MLLM architectures. We will incorporate these findings in the final version of the paper to strengthen our contributions.

We agree with the reviewer on the scope of the paper and would be happy to change the title to “From Bulk to Budget: Best Practices To Compress Multimodal Large Language Models through Structural Pruning and Recovery.” and refactor our paper accordingly to clarify the contribution. Nevertheless we would like to emphasize that the contributions of our paper —offering best practices for MLLM compression via pruning and recovery—are substantial. We believe these insights will be valuable to a broader audience and practitioners.

The authors' claim of being the first to investigate general-purpose MLLM compression overlooks existing work, particularly the survey paper on efficient MLLMs [1], which contradicts the authors’ claim. The literature review could be more comprehensive, especially in acknowledging related work in MLLM efficiency.

We thank the reviewer for pointing out this related work in MLLM efficiency. We recognize the value of the survey; however, our focus is different. The survey investigates building efficient small MLLMs using pretrained efficient components and techniques rather than compressing existing MLLMs to meet specific size requirements. The survey offers an excellent overview of the existing literature on efficient MLLMs while we conducted extensive experiments, i.e. combination of two pruning methods, two ways of finetuning, three different knowledge distillation losses across two models, to find the best practices for compressing MLLMs. This experimental rigor distinguishes our contribution from the broader scope of the survey.

We recognize the value of efficient MLLMs mentioned in the survey. While building MLLMs based on the pretrained efficient components such as SLMs and optimized vision models does help reduce the model size and improve the latency, the fixed size of the underlying components constrains their flexibility. Furthermore, training an efficient component such as SLM [2] from scratch to meet desired specifications is computationally expensive.

Our work explicitly targets methods for customizing the size of existing MLLMs through structured pruning and recovery strategies. Notably, recovery training only uses a tiny fraction of the training data, making it more cost-efficient than training a smaller MLLM or SLM from scratch.

We have included [1] and other relevant studies on efficient MLLMs in the related work (Sec. 2) in our revision.

The technical contributions largely adapt existing LLM compression techniques to MLLMs without introducing significant novel methods.

Indeed, we are not presenting a new compression method. However, we emphasize the main contribution of this paper is best practices for MLLM compression. To our knowledge, this is the first systematic investigation of pruning and recovery training techniques on MLLMs, as pointed out by reviewer T1Q7.

In the majority of the existing MLLMs, the parameter count is dominated by the LLM. Hence, our main contribution lies in systematically adapting, comparing, and analyzing existing LLM compression techniques to MLLMs. We identify and address the unique challenges posed by their multimodal nature.

We evaluate various structured pruning strategies and recovery methods, quantifying the trade-offs between compression ratio, performance retention, and data efficiency. In the revision, we also added 8-bit quantization experiments on the LLaVA model. The results are in the updated Appendix H. There, we demonstrate that pruning and recovery training can be effectively combined with quantization to achieve minimal performance loss.

Furthermore, we extend our best practices to the new model Mini-InternVL-Chat-4B-V1-5 [1]. The results are in the updated Appendix I. Our additional experiments with InternVL confirm that our findings generalize and validate our best practices transfer to other MLLM architectures. In summary, by establishing these best practices, we provide practitioners with insights and practical guidance for selecting appropriate techniques for MLLM compression based on specific deployment needs, which we believe is a strong technical contribution.

The findings mostly confirm expected behaviors from LLM compression research. The paper primarily combines existing techniques rather than introducing new methodological advances. The exploration could better highlight MLLM-specific challenges and solutions that differentiate it from general LLM compression.

We agree that certain observations from LLM compression carry over to the MLLM setting, such as the model retaining more performance when the compression ratio is smaller[3][4]. However, our work also addresses unique challenges specific to MLLMs. For example, in Section 4.2, we investigate how compression affects the alignment of textual and visual features, which is essential for MLLM performance. Our findings indicate that, at compression ratios up to 15%, fine-tuning only the projector network effectively recovers more than 95% of the model’s performance. This means that most degradation is due to misalignment between visual and textual feature space, thus suggesting a cost-efficient strategy for maintaining performance at lower compression levels. For larger compression ratios, we observe additional benefits from fine-tuning the language model as well.

We appreciate the reviewer’s feedback and have emphasized the MLLM-specific aspects of our work more clearly in the revision section 4.2.

Reference

[1] Jin, Yizhang, et al. "Efficient multimodal large language models: A survey." arXiv 2024.

[2]Chu, Xiangxiang, et al. "Mobilevlm: A fast, strong and open vision language assistant for mobile devices." arXiv 2023.

[3] Ma et al. "Llm-pruner: On the structural pruning of large language models." NeurIPS 2023.

[4] Men, Xin, et al. "Shortgpt: Layers in large language models are more redundant than you expect." arXiv 2024.

The second concern I have about technical contribution is not addressed. The author even sold the concept of ''best practices for MLLM compression''; however, I do not accept it. Therefore, I keep my score as 3.

While the framework demonstrates potential at lower compression ratios, its performance gains are limited at higher compression ratios. This limitation may reduce the practical appeal of the method for applications that demand more aggressive compression while still preserving task performance.

We acknowledge that performance degradation at high compression ratios is a significant challenge. However, our study focuses on establishing best practices for compressing MLLMs through structured pruning, with a particular emphasis on the trade-off between model size reduction and performance retention. To this end, we systematically investigate a wide range of compression ratios, a scope often overlooked in prior work [2]. Our primary goal is to provide practitioners with a comprehensive framework to make informed decisions about compression strategies, rather than solely aiming to optimize performance at extreme compression levels.

Although the proposed combination of multiple pruning and recovery strategies is thorough, it increases implementation complexity. We acknowledge the reviewer’s concern about the implementation complexity. While our paper presents a broad comparison of pruning and recovery strategies, our aim is to provide an in-depth evaluation of different methods, highlighting their strengths and weaknesses so that practitioners can select the most suitable approach for their specific needs without conducting extensive experiments themselves.

The reliance on complex data structures and recovery steps may hinder deployment. An author should clarify this.

Our analysis does not rely on complex data structures. To simplify deployment in data-constrained scenarios, we use only a small fraction of the original training dataset for recovery training (see Figure 4). Specifically, our study shows that practitioners can achieve 95% performance recovery using just 5% of the original dataset, significantly reducing the reliance on large datasets and making the process more practical for real-world applications.

What are the limitations of the chosen pruning strategies for different types of tasks within MLLMs?

The weakness of both pruning strategies is the limited performance in the high compression ratio scenarios. As shown in Table 6, both pruning methods lead to significant performance degradation across all benchmarks at high pruning ratios. Notably, when only the multimodal projector is finetuned, the layerwise pruned model recovers substantial performance, suggesting that layerwise pruning primarily harms the alignment between visual and textual features.

In practice, the implementation requirements for the two pruning methods differ. Widthwise pruning requires gradient information, leading to higher memory demands, while layerwise pruning has lower memory requirements as it relies only on activations.

Can the proposed data-efficient recovery techniques be generalized to other model architectures or require specific adjustments?

We appreciate the reviewer’s question regarding the generalizability of our best practices. The techniques, indeed, generalize to other architectures as shown by our new experiments with the Mini-InternVL-Chat-4B-V1-5 [3] model. We added it to the revision in Appendix I. The results confirm that our observations hold for this model, demonstrating the applicability of our best practices to other MLLMs. Notably, all pruned models are recovery trained on only 3% of the original dataset, which highlights the data efficiently.

In the table below, we observe that widthwise pruning continues to offer better performance without recovery training. With InternVL, widthwise pruning retains 97.4% of the model’s original performance at a 15% compression ratio, compared to 96.7% for layerwise pruning, reinforcing its suitability as a default strategy in low-resource scenarios.

Additionally, we find that finetuning only the multimodal projector is sufficient at small compression ratios, where pruning minimally impacts the language model but disrupts multimodal alignment. With InternVL, finetuning only the projector recovers 96.9% of the performance at a 15% compression ratio, compared to 97.8% when both the projector and language model are finetuned. At a 30% compression ratio, projector-only finetuning recovers 75.1% while finetuning both components recovers 86.6%. These results reinforce our observations from earlier models and validate the transferability of our proposed practices.

Moreover, our initial findings show that the combination of supervised finetuning with intermediate representation distillation consistently yields the highest performance across compression ratios. With InternVL, this combined approach achieves 98.2% recovery at a 15% compression ratio and 87.2% at a 30% compression ratio, confirming its effectiveness.

Overall, the results from InternVL indicate that our methods generalize well to newer MLLM architectures. We have incorporated these findings in the updated version of our paper to strengthen our contributions.

Reference

[1] Isaac–Chassande et al. "Dedicated hardware accelerators for processing of sparse matrices and vectors: a survey." ACM 2024: 1-26.

[2] Ma et al. "Llm-pruner: On the structural pruning of large language models." NeurIPS 2023.

[3] Chen et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." CVPR 2024.

The proposed techniques are evaluated within the context of MLLMs; however, they lack comparisons with other prevalent compression methods. This absence makes it challenging to assess their effectiveness against existing solutions, especially as structured pruning and knowledge distillation are already well-established in the field.

We appreciate the reviewer’s insight and understand the importance of comparing the performance of pruning and recovery strategies to other compression techniques. In particular, our findings highlight that quantization is a complementary compression approach that can be effectively combined with structured pruning, achieving significant memory savings with minimal performance loss.

We included 8-bit quantization experiments on the LLaVA model and further analysis in the revision Appendix H to compare prevalent compression methods.

These results demonstrate that quantization reduces the memory footprint of the base model by 44.5%, with only a 0.43pp decrease in average performance. However, quantization introduces a fourfold increase in model latency, which poses challenges for deployment scenarios requiring real-time inference. Additionally, we evaluated the combination of quantization with structured pruning and recovery training. For compression ratios of 15% and 30%, this combination reduced the memory footprint by 40% and 44%, respectively, while incurring only 0.4pp and 1.3pp drops in average performance. These experiments showcase the complementary nature of structured pruning and quantization as effective and practical compression strategies.

Unstructured pruning is not included in our evaluation as it provides limited memory saving, especially for edge devices, and requires specialized hardware and software for speed-up [1]. In contrast, structured pruning removes entire groups of parameters (e.g., neurons or layers), enabling significant memory reductions and making it better suited for resource-constrained environments like edge deployment.

Overall, we want to highlight our study's aim to identify best practices for compressing MLLMs using structured pruning and efficient recovery methods. We believe that our findings can offer valuable guidelines for deploying MLLMs in resource-limited environments.