CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models

Thank you for your valuable feedback and the time you have spent reviewing our work. We address the concerns raised and provide answers to your questions accordingly.

Responses to Weakness 1

The setting in our paper addresses a prevailing and significant challenge. Instruction tuning has emerged as a critical technique and a primary method for adapting MLLMs to downstream tasks in recent works [1-8], especially given the computational constraints of pre-training MLLMs from scratch.

The major challenge in MLLM instruction tuning lies in effectively aligning the feature encoder for downstream tasks with the text features, ensuring more relevant modality-specific information. Our paper proposes a new method that addresses this challenge by dynamically balancing multimodal learning progress by dynamically coordinating learning rates on the feature encoder and LLM.

Responses to Weakness 2 and Question 2

As discussed in Section 4, our paper focuses on achieving a balance between the feature encoder and the backbone LLM during MLLM instruction tuning. Our method is designed to be easily implemented and compatible with a wide range of MLLMs during fine-tuning. To demonstrate the generalizability of our approach, we use popular and widely adopted backbone model architectures in state-of-the-art MLLMs. For instance, the backbone models BLIP-2 [1] and LLaVA [2] used in our paper are both highly cited.

Moreover, based on popular leaderboards [9-11], LLaVA [2] achieves the best performance among open-sourced models on MLLM-Bench [9]. Additionally, InternVL outperforms Qwen on benchmarks such as MLVU [10] and MMMU [11]. To further support the flexibility of our method, we include SALMONN[8] for audio tasks, demonstrating its adaptability across different modalities. Given the popularity and diversity of the backbone models we selected, we believe they are sufficient to prove the generalizability and effectiveness of our method.

Responses to Weakness 3

We have already explained all the baselines in detail in lines 389–397 of the paper. To summarize:

1. Constant LR: This is the standard supervised fine-tuning (SFT) approach. Both the feature encoder and backbone LLM are fine-tuned using LoRAs with a fixed learning rate of $1e^{-4}$.
2. Feature CD: The feature encoder is updated first until its weights stabilize, followed by training the backbone LLM with the same learning rate.
3. Language CD: The reverse of Feature CD, where the backbone LLM is trained first, and then the feature encoder is updated.
4. CoMMIT Variants: We also evaluate CoMMIT (our proposed method) and CoMMIT-CLR (an ablation of CoMMIT without the regularization term), both of which use an initial learning rate of $1e^{-4}$. \end{itemize}

Responses to Question 1

We have already included the SFT method as a baseline in our experiments, which is Constant LR. As previously mentioned, our method is designed to address the challenge of achieving synergy through cooperative learning across different modalities. This enables LLMs to adapt their reasoning abilities to downstream tasks while feature encoders adjust to provide more relevant modality-specific information. By leveraging this unique architecture of MLLMs, our approach can be seen as a more effective solution compared to traditional SFT methods. DPO falls outside the scope of this work. However, as highlighted earlier, our setting is generalized enough to be broadly applicable to multimodal instruction tuning.

[1] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." International conference on machine learning. PMLR, 2023.

[2] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

[3] Chen, Zhe, et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." CVPR. 2024.

[4] Zhang, Renrui et al. “LLaMA-Adapter: Efficient Fine-tuning of Large Language Models with Zero-initialized Attention.” International Conference on Learning Representations (2024).

[5] Gao, Peng, et al. "Llama-adapter v2: Parameter-efficient visual instruction model." arXiv:2304.15010.

[6] Lee, Byung-Kwan, et al. "Collavo: Crayon large language and vision model." arXiv:2402.11248.

[7] Han, Jiaming, et al. "Imagebind-llm: Multi-modality instruction tuning." arXiv:2309.03905.

[8] Tang, Changli, et al. "SALMONN: Towards Generic Hearing Abilities for Large Language Models." ICLR. 2024.

[9] Ge, Wentao, Shunian Chen, and G. Hardy Chen. "MLLM-Bench: evaluating multimodal LLMs with per-sample criteria." arXiv:2311.13951.

[10] Zhou, Junjie, et al. "MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding." arXiv:2406.04264.

[11] Yue, Xiang, et al. "Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi." CVPR. 2024.

Thank you for your valuable feedback and the time you have spent reviewing our work. We address the concerns raised and provide answers to your questions accordingly.

Response to Weakness 1 and Question 1

In Figure 4 and Figure 5, we empirically studied the learning curves of ConstantLR and CoMMIT on multiple datasets, A-OKVQA, IconQA, TextVQA, ClothoAQA, MACS, and SDD, with multiple backbone models, including LLaVA-1.5, BLIP-2, SALMONN, and InternVL2 in both the vision and audio domains. In Line 399 - 405, We demonstrated the learning inefficiency problem in ConstantLR and showed that CoMMIT can accelerate the learning process and converge to lower estimation errors. In addition, we also included the comparative study in Table 1 of four different backbone MLLMs in multiple datasets, showing consistent improvement of CoMMIT.

To provide more empirical results on the study of the learning oscillation problem, we include an analysis of the standard deviations $\kappa$ for LLaVA-1.5 and InternVL on multiple datasets. Based on the empirical results, we can observe consistent improvement of CoMMIT on the learning oscillation problem by reducing the variance of learning curves.

Response to Weakness 2 and Question 2

We would like to emphasize that one major contribution of our work (as claimed in two contributions at the end of the introduction section) contributes a novel theoretical framework that addresses the learning imbalance problem in MLLM instruction tuning, offering insights into improving optimization through a dynamic learning rate adjustment and a loss regularization term, which fundamentally enhances instruction tuning theory. As discussed in Section 4, our paper focuses on achieving a balance between the feature encoder and the backbone LLM during MLLM instruction tuning. Our method is designed to be easily implemented and compatible with a wide range of MLLMs during fine-tuning.

We validate our approach across multiple datasets, A-OKVQA, IconQA, TextVQA, ClothoAQA, MACS, and SDD, which include Visual Question-answering, Optical Character Recognition, Audio Question-answering, and Audio Captioning from diverse domains, including vision and audio tasks. We also evaluate four MLLMs, LLaVA-1.5, BLIP-2, SALMONN, and InternVL2, demonstrating CoMMIT's generalization ability and outperforming or matching baselines in widely adopted evaluation protocols. The evaluation in Table 2 is only to justify that no fixed value of learning rate consistently yields the best performance for Constant LR, while our proposed CoMMIT can dynamically adjust its learning rate (as discussed in Line 520 -523).

Our use of datasets and evaluation protocols aligns with recent literature [a,b,c,d,e,f,g,h], and we contribute to the growing body of work by addressing optimization challenges rather than proposing another model architecture.

[a] Wang, Sheng, et al. "PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA." arXiv preprint arXiv:2402.16902 (2024).

[b] Jie, Shibo, et al. "Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning." Forty-first International Conference on Machine Learning.

[c] Panos, Aristeidis, et al. "Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models." arXiv preprint arXiv:2407.16526 (2024).

[d] Zhu, Didi, et al. "Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models." Forty-first International Conference on Machine Learning.

[e] He, Jinghan, et al. "Continual instruction tuning for large multimodal models." arXiv preprint arXiv:2311.16206 (2023).

[f] Zhang, Renrui et al. “LLaMA-Adapter: Efficient Fine-tuning of Large Language Models with Zero-initialized Attention.” International Conference on Learning Representations (2024).

[g] Gao, Peng, et al. "Llama-adapter v2: Parameter-efficient visual instruction model." arXiv preprint arXiv:2304.15010 (2023).

[h] Li, Yifan, et al. "Facial Affective Behavior Analysis with Instruction Tuning." arXiv preprint arXiv:2404.05052 (2024).

Thank you for your valuable feedback and the time you have spent reviewing our work. We address the concerns raised and provide answers to your questions accordingly.

Response to Weakness 1

Yes, the multimodal projector is treated as part of the encoder $S$.
Specifically, when training with BLIP2, we freeze the image encoder and finetune the q-former $S$ and LLM $X$. This is reasonable since there should still exist the insufficient and oscillation problem when learning with these two modules. Our results on BLIP2 show that our CoMMIT is generic to cooperative learning between different modules in MLLM instruction tuning.

Response to Weakness 2

Since our major contribution is the multimodal learning theory in MLLM instruction fine-tuning, we follow previous works in model optimization [a,b,c] and report the learning curve comparisons between different learning methods in Table 4 and Table 5. Empirically, since CoMMIT does not change the fine-tuning structure of MLLMs, we are expecting a similar computation complexity. In addition, we analyze the computational cost for calculating $k_t$ and the associated regularization terms as $T = 2N_S + N_t + 2N$, where $N_S$ and $N_t$ are significantly smaller than $N$ due to parameter-efficient tuning, making the additional cost marginal relative to the overall training.

Response to Weakness 3

As explained in our previous response, since our major contribution is the multimodal learning theory in MLLM instruction fine-tuning, we follow the instruction tuning evaluation protocol [d,e,f,g] to evaluate on fine-tuning performance.

Response to Weakness 4

Thanks for the suggestion. We added the suggested comparison in Appendix B (Figure 7), which compares the normalized learning gradients between ConstantLR and CoMMIT. We can observe that CoMMIT could significantly improve the gradient diminishing issue.

Response to Question 1

The two methods evaluated in Figure 3, Encoder LR ↑ and Language LR ↑, indicate a setting of imbalanced learning rates during training. As stated in Observation 5.2 (Line 265 - 267), we observe that imbalanced learning that inclines toward X or S can result in gradient diminishing and inferior training performance.

Response to Question 2

Yes, as mentioned in line 370, our method does not modify the optimization algorithm itself but focuses on updating the learning rate. Therefore, it can be extended to any gradient-based optimization method. We have tested it with stochastic gradient descent (SGD), and our method consistently outperforms the baseline, demonstrating its generalizability.

Response to Question 3

We apologize for the typo. In L271, "Such In such cases" should be "In such cases".

[a]. Iiduka, Hideaki. "Appropriate learning rates of adaptive learning rate optimization algorithms for training deep neural networks." IEEE Transactions on Cybernetics 52.12 (2021): 13250-13261.

[b]. Liu, Liyuan, et al. "On the variance of the adaptive learning rate and beyond." arXiv preprint arXiv:1908.03265 (2019).

[c]. Na, Gyoung S. "Efficient learning rate adaptation based on hierarchical optimization approach." Neural Networks 150 (2022): 326-335.

[d] Yin, Zhenfei, et al. "Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark." Advances in Neural Information Processing Systems 36 (2024).

[e] Chen, Chi, et al. "Position-enhanced visual instruction tuning for multimodal large language models." arXiv preprint arXiv:2308.13437 (2023).

[f] Li, Zou, Ning Pang, and Xiang Zhao. "Instruction Tuning Large Language Models for Multimodal Relation Extraction Using LoRA." International Conference on Web Information Systems and Applications. Singapore: Springer Nature Singapore, 2024.

[g] Panagopoulou, Artemis, et al. "X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning." arXiv preprint arXiv:2311.18799 (2023).

[e] Jin, Yizhang, et al. "Efficient multimodal large language models: A survey." arXiv preprint arXiv:2405.10739 (2024).

[f] Chen, Shaoxiang, Zequn Jie, and Lin Ma. "Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms." arXiv preprint arXiv:2401.16160 (2024).

[g] Xue, Le, et al. "xgen-mm (blip-3): A family of open large multimodal models." arXiv preprint arXiv:2408.08872 (2024).

[h] Gao, Peng, et al. "Llama-adapter v2: Parameter-efficient visual instruction model." arXiv preprint arXiv:2304.15010 (2023).

[i] Tang, Changli, et al. "Salmonn: Towards generic hearing abilities for large language models." arXiv preprint arXiv:2310.13289 (2023).

[j] Ye, Qilang, et al. "Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios." European Conference on Computer Vision. Springer, Cham, 2025.

[k] Sagare, Shivprasad, et al. "Audio-visual training for improved grounding in video-text LLMs." arXiv preprint arXiv:2407.15046 (2024).

Thank you for your valuable feedback and the time you have spent reviewing our work. We address the concerns raised and provide answers to your questions accordingly.

More empirical results of the standard deviations of $\kappa$ for LLaVA-1.5 and InternVL on multiple datasets. Our proposed CoMMIT induces $\kappa$ with smaller variances, improving training stability.

Response to Weakness 1

Parameter-efficient fine-tuning with LoRA in MLLM instruction tuning is a technique adopted in a wide range of works [a,b,c,d,e,f,g,h,i,j,k]. How to balance the learning between these two components can be a general challenge that concerns different model backbones [f,g,h,i], and modalities [i,j,k]. We will highlight our contribution on such points to avoid misunderstanding.

Response to Weakness 2

Quantitatively, the insufficient learning corresponds to $\kappa>>1$ or $\kappa\rightarrow 0$ (Hypothesis 4.2). When these happen, the learning would be primarily attributed to the updates on only one of the encoders or LLM, resulting in the other module (LLM or encoder) being learned insufficiently. On the contrary, sufficient learning refers to $\kappa$ close to 1 with both the encoder and LLM being involved in the learning dynamics.

Response to Weakness 3

From the Table above (along with Figure 6), our proposed CoMMIT resolves the oscillation problem via inducing less variant $\kappa$, i.e., the training is more stable without oscillating between the encoder and LLM.

Additionally, it can be observed from Figure 6 that the $\kappa$ from CoMMIT is closer to 1, suggesting both the encoder and LLM are involved in the training dynamics. This improves the learning sufficiency as compared to biasing on either the encoder ($\kappa\rightarrow 0$) or LLM ($\kappa>>1$), e.g., Encode LR $\uparrow$ and Language LR $\uparrow$ in Figure 2.

From a theoretical perspective, as shown in Equation 13, applying COMMIT results in a better bound on the norm of the gradient. Specifically, $\lambda$ is always greater than 1, leading to an upper bound on the gradient norm that is smaller than the bound achieved by the original Adam algorithm. This indicates more efficient and sufficient learning.

Response to Weakness 4

In Section 8, we provide the information (on Line 378 - 380) of the instruction tuning datasets for vision tasks, TextVQA (34K training, 5K test), A-OKVQA (17K training, 6K test), and IconQA (18K training, 6K test). For audio tasks, we explain the datasets (on Line 385 - 387) ClothoAQA (21K training, 8K test), MACS (3K training, 393 test), and SDD (1K training, 746 test). We follow the original data split provided by the individual dataset. We follow the original prompt template for InternVL2 and LLaVA-1.5. We enable the instruction tuning by only calculating the loss on the response tokens. We will add such information in our paper to enable better implementation details.

Response to Question 1

$\kappa$ of smaller variance suggests the learning is not oscillating between the encoder and LLM, thus is an indicator of training stability. In the table above, our proposed CoMMIT improves the training stability with a much lower variance in $\kappa$ compared to ConstantLR. As a result, CoMMIT constantly yields better performance than ConstantLR (in Table 1).

Response to Question 2

Thanks for the suggestion. We added the suggested comparison in Appendix B (Figure 7) corresponding to our findings in Section 5.2 (Figure 3), which compares the normalized learning gradients between ConstantLR and CoMMIT. We can observe that CoMMIT could significantly improve the gradient diminishing issue.

Response to Question 3

We apologize for the typo. in lines 24-25 the sentence should be "which potentially enables a more accurate estimation ...".

[a] Wang, Luping, et al. "Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies." arXiv preprint arXiv:2410.19878 (2024).

[b] Zhou, Xiongtao, et al. "An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models." arXiv preprint arXiv:2406.05130 (2024).

[c] He, Jinlong, et al. "Pefomed: Parameter efficient fine-tuning on multimodal large language models for medical visual question answering." arXiv preprint arXiv:2401.02797 (2024).

[d] Li, Zou, Ning Pang, and Xiang Zhao. "Instruction Tuning Large Language Models for Multimodal Relation Extraction Using LoRA." WWW, 2024.

Thank you for your valuable feedback and the time you have spent reviewing our work. We address the concerns raised and provide answers to your questions accordingly.

Response to Weakness 1

Thanks for pointing them out. We have corrected these errors in our current draft.

Response to Weakness 2 and Question 4

In Figure 2 (c), the $\kappa_t$ "Encode LR $\uparrow$" is actually close to 0 at the earlier steps as compared to later when it converges to near 1. This demonstrates the imbalanced learning that inclines toward the encoder when $\kappa_t\rightarrow0$ in Hypothesis 4.2.

We contend that it is not increasing the lr of the encoder that causes $\kappa_t$ going to 1 in later training steps, since all three different lr setups in Figure 2 (c) end up $\kappa_t$ converging to close to 1. In explaining this, we cautiously speculate that the numerator and denominator $\kappa$ are both more and more resulting from the randomness of similar scales when close to converge. This causes $\kappa_t$ close to 1 in the later stage of training irrespective of the learning rate setup.

Response to Weakness 3

We conducted a comprehensive ablation study of the proposed regularization in Table 1. As discussed in Line 484 - 487, the proposed regularization in CoMMIT can promote larger step sizes in gradient descent, which enlarges differences in the generated output distributions between different time steps.

Response to Weakness 4

We compute $k_t$ every $K = 10$ training step, which balances the need for accurate estimation with computational efficiency, as this interval ensures that the values remain relevant without incurring excessive overhead. The computational cost for calculating $k_t$ and the associated regularization terms is $T = 2N_S + N_t + 2N$, where $N_S$ and $N_t$ are significantly smaller than $N$ due to parameter-efficient tuning, making the additional cost marginal relative to the overall training. Given the reduced parameter sizes and the periodic computation of $k_t$, the latency caused by this operation is negligible compared to the dominant costs of training the entire MLLM. Therefore, the choice of the updating interval $K$ is to optimize the trade-off between maintaining accurate updates to $k_t$ and minimizing computational impact.

Response to Question 1

Logits refer to the unnormalized outputs of the model's final layer, consistent with their definition in deep learning. These are the raw scores produced before applying any activation function.

Response to Question 2

We apologize for the confusion. The term $T_t$ should actually refer to $X_t$, the pre-trained language model $X$ at the t-th step of training.

Response to Question 3

As defined on Line 137, it represents the prediction distribution of the generated response when the multimodal components are jointly updated.

Response to Question 5

On L 311, we refer to the learning process only relying on instruction without access to the responses. Such regularization can be applicable to unsupervised learning in representation learning [a], domain generalization [b,c], and domain adaptation [d,e].

Response to Question 6

Sorry for the confusion. On L 311, the notation should be $\tilde{X}_{\theta^x_t}=T(X;\theta^x_t)$. We will add the definition to improve readability.

Response to Question 7

We apologize for the typo. $N$ should actually be $K$, representing the number of component functions.

Response to Question 8

$\|\nabla F(x_k)^2_2\|$ represents the squared norm of the gradient. Ideally, at a global minimum, this value should be 0. In optimization, proving that this term is bounded by a finite value is a key step to demonstrate convergence.

[a] Xie, Baao, et al. "Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models." arXiv preprint arXiv:2407.18999 (2024).

[b] Cheng, De, et al. "Disentangled Prompt Representation for Domain Generalization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[c] Choi, Juhwan, et al. "VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Tasks." arXiv preprint arXiv:2407.19795 (2024).

[d] Zhang, Huanyu, et al. "LogoRA: Local-Global Representation Alignment for Robust Time Series Classification." IEEE Transactions on Knowledge and Data Engineering (2024).

[e] Chen, Dongjie, et al. "Empowering Source-Free Domain Adaptation with MLLM-driven Curriculum Learning." arXiv preprint arXiv:2405.18376 (2024).