LLaCA: Multimodal Large Language Continual Assistant

For Question 4 about ''The "checkpoints" mentioned in Figure 3 and Algorithm 1 are unclear'':

In fact, we do not load the checkpoints during the training process. Here we just temporarily store the parameter checkpoints in the form of a Parameter Dictionary in Pytorch for calculating the Eq. (13). It should be clarified that we only load the checkpoint at the beginning of the current dataset training process.

For Question 8 about ''The term "but results for unseen tasks are missing'':

Thanks for your valuable suggestion. For the convenience of comparison, we have placed the unseen (zero-shot) task performance (marked with bold) and seen task performance in one table (as shown below). The horizontally arranged dataset represents the test dataset, while the vertically arranged dataset represents the training dataset.

For example, the value in the third row and the fourth column means the accuracy of the model trained on TextVQA and tested on VizWiz. Another example is that the value in the fourth row and the fifth column means the accuracy of the model trained on GQA and tested on VQAv2. Notice that baseline equals the LoRA Fine-Tuning.

LLaCA Instruction Type 1

LLaCA Instruction Type 2

LoRA Instruction Type 1

LoRA Instruction Type 2

Zero-shot performance of original LLaVA, please kindly refer to Table 1 and Table 2 in the paper.

Our model shows a significant improvement in zero-shot performance compared to LoRA-Tuning and original LLaVA, which we believe is due to the following reasons:

(1). In terms of anti-forgetting during continual instruction tuning, considering that our model can effectively protect old knowledge (including pre-trained knowledge), our model owns better zero-shot capability compared to LoRA-Tuning.

(2). In terms of learning how to generalize existing knowledge to new instructions, considering that the instruction templates in each fine-tuning task are similar, such as:

TextVQA: Use a single word or a short phrase to respond to the question.

VQAv2: Answer the question with a single word or a brief phrase.

Therefore, compared to the original LLaVA, our model owns better zero-shot capability after fine-tuning.

Dear reviewer RGJd, thanks for your valuable suggestions. Here are our responses:

For Weakness 1 about ''how it enhances multimodal learning'':

Thanks for your valuable suggestion. To show how it addresses the interaction between multiple modalities and further enhances multimodal learning. We conduct the following ablation experiments (conducted under Instruction Type 1): Normal. Use the normal gradient update method to update the projection layer from the visual model to the language model and LoRA in LLM. 2): Dy(Proj). Use dynamic EMA update (our method) to update the projection layer from the visual model to the language model, while updating the LoRA in LLM with the normal gradient update method. 3): Dy(LLM). Use dynamic EMA update (our method) to update the LoRA in LLM, while updating the projection layer from the visual model to the language model with the normal gradient update method. 4): Dy(Proj+LLM). Use dynamic EMA update (our method) to update the LoRA in LLM and the projection layer from the visual model to the language model. 5): Dy(Vis). Extra insert LoRA into the vision encoder and use dynamic EMA update (our method) to update the LoRA in the vision encoder, while updating the projection layer from the visual model to the language model, and LoRA in LLM with the normal gradient update method. 6): Dy(Proj+LLM+Vis). Extra insert LoRA into the vision encoder and use dynamic EMA update (our method) to update the LoRA in the vision encoder, LoRA in LLM, and the projection layer from the visual model to the language model. The results are shown as:

From the above experimental results, we can explore how our method enhances multimodal learning. The analysis is as follows:

(1). When training the projection layer, it is equivalent to mapping the encoded visual features to the language-text space. Both the first-order and second-order gradients of the projection layer collect the information from two modalities. Therefore, when calculating the EMA update weight of the projection layer, it can be seen as a result obtained on the basis of addressing the interaction between multiple modalities.

(2). Similarly, after concatenating the encoded visual features and the textual features into union features, we input them into LLM. Therefore, the features processed by LoRA are a mixture of visual and textual features. Naturally, both the first-order and second-order gradients of LoRA contain information from two modalities. Therefore, when calculating the EMA update weights of the LoRA layer by utilizing the first-order and second-order gradients, it can also be seen as a result obtained on the basis of addressing the interaction between multiple modalities.

(3). It is worth mentioning that although LoRA is also inserted into the vision encoder in Dy(Proj+LLM+Vis) and dynamically updated according to the proposed method, its performance is slightly better than that in Dy(Proj+LLM). Similarly, in Dy(Vis), we also insert additional LoRA into the Vision Encoder and update it according to the proposed method, while keeping other trainable parameters updated with the normal gradient update method. It is still inferior to the results in Dy(Proj) and Dy(LLM). The above experiments all explain that in the case of a single modality (image), our method has little effect and can only achieve maximum benefits in multimodal scenarios. This also reflects that our method enhances multimodal learning.

For Question 10 about ''Using a single equation number for each derivation would be clearer'':

Thanks for your valuable suggestion, we have utilized a single equation number for each derivation.

For Question 7 about ''The term "baseline" is used throughout the paper'':

1. Thanks for your advice. The term "baseline" consistently refers to LoRA fine-tuning, which has been clarified in Line 357-359. In Table 1, we also illustrated that the baseline equals the LoRA fine-tuning, masked as LoRA (Baseline) in the “Method” column. Besides that, we have added the further explanation that in the whole paper, baseline refers to the LoRA fine-tuning.

2. We compare the zero-shot performance (Table 4) of the stronger PGP method (in your suggestion) with ours under instruction type 1. Here are the results. The first column represents the continual training tasks. The second column represents the average accuracy of zero-shot inference datasets with PGP. The third column represents the average accuracy of zero-shot inference datasets with ours (here, we have the same zero-shot inference setting as Table 4).

It can be seen that our method owns better zero-shot performance than PGP.

For Question 6 about ''There are inconsistencies in the implementation details'':

Thanks for your advice. There is indeed a bit of confusion, we have made a note of it. All experiments except Table 3 are conducted on 8-A100. Considering that Table 3 does not involve specific performance metrics (e.g. ACC/Forgetting), and the 8-A100 machine was occupied during the production of Table 3, thus we tested the training time and training parameters on the 8-A6000.

For Question 5 about ''The framework caption in Figure 3 could benefit from more detailed explanations'':

Thanks for your valuable suggestion, we have added more detailed explanations for Figure 3 and further improved the comprehensibility of the image.

For Question 9 about ''The paper cites many arXiv preprints'':

Thanks for your valuable suggestion, we have rectified the reference form which has been published.

For Question 3 about ''Moving it to a more suitable section'':

Thanks for your valuable suggestion, we have moved the subsection "Multimodal Continual Instruction Tuning Definition" in Section 2 to a more suitable location and enhanced the clarity of the paper.

For Question 2 about ''ten kinds of instruction'':

(1). In our setting, each tuning task includes all ten kinds of instruction templates. In other words, they are task-shared, not task-specific. Thus, they cannot serve as implicit task identifiers.

(2). We obtain the conclusion from an experimental perspective that all performances of almost all methods fell short on Instruction Type 2 compared with results on Instruction Type 1.

For Question 1 about ''The authors should reproduce the experiment themselves'':

Thanks for your valuable suggestion, we reproduce the experiment according to our training datasets and orders (ScienceQA->TextQA->GQA->VizWiz->VQAv2->OCRVQA). The following are experimental results.

Instruction Type 1

Instruction Type 2

For Weakness 2 about ''Including them would provide a more complete evaluation'':

Thanks for your valuable suggestion, we add the Image Classification Task (ImageNet-1K) and Referring Expression Comprehension (RefCoCo) in the MCIT task, consistently with CoIN [1]. The new tuning sequence order is as follows: ScienceQA->TextVQA->ImageNet->GQA->VizWiz->Grounding ->VQAv2->OCRVQA, with a total of 8 tasks. The following are the experimental results:

Instruction Type 1

Instruction Type 2

It can be seen that under 8 instruction tuning tasks, our method still has better performances compared to Baseline (Inserted with LoRA) and MOELoRA [1]. It is worth noting that our judgment of whether the prediction results are correct or not is strictly based on the direct comparison between outputs of MLLMs and ground truths, which is defined as Instruction Following Ability in [1]. Therefore, our judgment criteria would be more stringent. It can be seen that under such strict standards, the baseline model (LLaVA+LoRA) experiences extremely severe catastrophic forgetting, but our model still has excellent performance.

[1] COIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Model

For Weakness 4 about ''Discussion on Optimization Analysis'':

Thank you for your valuable suggestion. It is very meaningful, and innovative, which indeed provides insight points. We haven't done any experiments related to Optimization Analysis before, and this is our first attempt to try it.

Based on Landscape Analysis [7], we study the loss landscape maps of LLaCA and LoRA Fine-tuning on Task 1 (ScienceQA) and Task 2 (TextVQA). Our conclusion is that in our method, the model can be continuously updated (Loss reduced to the minimum value area of Task 2) while ensuring that the Loss on Task 1 is near the minimum value area.

For Plasticity: we find that the loss landscape map of LLaCA is more convex than LoRA Fine-tuning both in Task 2 (TextVQA). It can be explained that LLaCA promotes flat minimizers and prevents the transition to chaotic behavior [7]. The corresponding phenomenon is that the New Task Accuracy of LLaCA on Task 2 is 59.69. However, the New Task Accuracy of LoRA Fine-tuning on Task 2 is 50.63. It can be seen that due to the better convex of loss landscape map, LLaCA has better New ACC performance than LoRA Fine-tuning.

For Stability: In LoRA Fine-Tuning, it has been found that during the training of Task 1, the loss decrease to the minimum value region of the current task. When training Task 2, the loss also decreases to the minimum value area of the current task. However, on Task 1, the loss of model at this training task is in a larger value area. It means that the training of Task 2 disrupts the loss value of Task 1, causing it to move from a minimum value area to a larger value area, and resulting in poor performance of Task 2 trained model on Task 1 (catastrophic forgetting). However, in the LLaCA method, during the training of Task 1, the loss decreases to the minimum value region of the current task. When training Task 2, constantly reviewing the model parameters on Task 1 is like a rope pulling the model parameters, so that the range of loss variation on Task 1 does not exceed the minimum value area too far.

[7] Visualizing the Loss Landscape of Neural Nets

For Weakness 3 about ''more backbone models'':

Thanks for your suggestion. Considering that both Mini-GPT4 [4] and OpenFlamingo [5] have significant differences in the input forms of data compared to LLaVA codebase, it is not possible to deploy our method on these two frameworks in time. We will gradually implement this in the future. But, here, we choose InternVL [6] as another MLLM and deploy our method and continual instruction setting on it. The reason why we choose InternVL is that it has a more powerful visual encoder (6B InternViT) and a three-stage model pre-training phrase, which provides it with powerful multimodal information processing capabilities. It is crucial to verify whether our method is effective on stronger models. Here, we still choose to insert LoRA on the LLM side (7B) of InternVL in each layer as the baseline. Meanwhile, we add our dynamic EMA weight update method to the baseline as our method for comparison. The following table shows the experimental results.

It can be seen that our method is also effective for InternVL, significantly improving its continual instruction tuning ability, which proves the generalization and effectiveness of our method for more MLLMs.

[4] MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

[5] OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

[6] InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

For Weakness 2 about ''The inherent differences between the different PEFT methods'':

Thanks for your advice. We transfer our method from the LoRA Tuning paradigm to the Adapter Tuning paradigm [2], leaving other settings unchanged.

Instruction Type 1 (With Adapter Tuning)

Avg.ACC = 60.98, Forgetting = 2.27, New ACC = 62.81

Instruction Type 2 (With Adapter Tuning)

Avg.ACC = 60.30, Forgetting = 2.58, New ACC = 62.45

Overall, when compared with the performances between LoRA and Adapter, our method achieves equivalent Avg.ACC performance. However, there are some differences in the Forgetting performance. The reasons we analyze are as follows:

(1). [3] deems that although Adapter and LoRA belong to two completely different fine-tuning paradigms, their mathematical forms remain consistent. Therefore, they should have similar comprehensive performance (Avg.ACC) on the same anti-forgetting method.

(2). Specifically, the Adapter process inputs by adding a parallel bypass module, and finally merges the obtained results with the output of the original module, without any interaction with the original model parameters in this process. However, LoRA constructs a new matrix with the same dimensions as the original module by multiplying two low-rank matrices, and merges the new matrix with the original module parameters based on element-wise sum. It interacts with the original model parameters during this process. The essential mechanism of catastrophic forgetting is that the changing of model parameters disrupts the original parameter distribution. Therefore, LoRA which generates interaction performs worse in anti-forgetting than Adapter which does not generate interaction.

[2] Introducing language guidance in prompt-based continual learning

[3] Gradient Projection For Continual Parameter-Efficient Tuning

Dear reviewer ubn8, thanks for your valuable suggestions. Here are our responses:

For Weakness 1 about ''further validate the effectiveness of LLaCA'':

Thanks for your valuable suggestion, we add one image classification task (ImageNet-1K) and one Grounding task (based on the COCO dataset) in the MCIT task. The new tuning sequence order is as follows: ScienceQA->TextVQA->ImageNet->GQA->VizWiz->Grounding ->VQAv2->OCRVQA, with a total of 8 tasks. The following are the experimental results:

Instruction Type 1

Instruction Type 2

It can be seen that under 8 instruction tuning tasks, our method still has better performances compared to Baseline (Inserted with LoRA) and MOELoRA [1]. It is worth noting that our judgment of whether the prediction results are correct or not is strictly based on the direct comparison between outputs of MLLMs and ground truths, which is defined as Instruction Following Ability in [1]. Therefore, our judgment criteria would be more stringent. It can be seen that under such strict standards, the baseline model (LLaVA+LoRA) experienced extremely severe catastrophic forgetting, but our model still has excellent performance.

[1] COIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Model

For Question1 about ''more comprehensive evaluation'':

Thanks for your advice. Based on Instruction Type 1, we measure four fixed EMA weights for comparison, e.g. 0.500 0.990 0.991 0.992. In addition, we also measure the experimental results of a naive EMA weight decay method ($\beta_t=0.9991+0.0002 (t-2), t > 1$, while $t=1$ without EMA update). Finally, to further supplement our conclusion, we also measure the experimental results based on the parameter weight decay (EWC) method. To sum up, our method is still superior to the above methods.

For Question2 about ''The writing could be polished for clarity'':

Thanks for your valuable suggestion, we have moved some detailed explanations from the main text to the Appendix. Additionally, we also simplify the Section 4.4.

For Weakness 5 about "The paper does not compare its approach with SOTA methods for mitigating catastrophic forgetting in MLLMs'':

We transfer the method in [6] to the MLLM continual tuning framework used in this paper, and the experimental results are compared as follows:

It can be seen that our method outperforms the method in [6] with two instruction templates and three metrics (Avg. ACC, Forgetting, and New. ACC). We hold the view that the method proposed in [6] is only suitable for one or two downstream fine-tuning tasks, as its starting point is to protect pre-trained knowledge and alleviate its forgetting during downstream task generalization. Therefore, it is not suitable for multiple continual tuning tasks (such as six tasks in our experimental setting).

As emphasized in Appendix A. More Details about Related Work [6], the author claims that “while (He et al., 2023) focuses on sequential tasks and measuring the forgetting of older tasks when learning new tasks, our focus is on the forgetting of the pre-trained MLLM after fine-tuning specific tasks”. Different from their goals, we focus on preserving knowledge of historical tasks (including both pre-training tasks and continual fine-tuning tasks) and mitigating catastrophic forgetting during the fine-tuning process.

It is worth noting that our method can also be directly applied to the setting in [6], which can be seen as a sub-task of multimodal continual instruction tuning (one-step continual learning). A similar task can be seen in Table 4 in our paper (The difference is that [6] test on the pre-training tasks, while we test on other unseen tasks). Besides that, considering the time constraints, we will try the setting in [6] with our method in future work.

[6] Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models

For Weakness 4 about ''This method is applicable to LLMs without any modifications'':

1. Please kindly note that [2] actually synthesizes the historical text data based on LLMs for rehearsing, but this process requires additional training time and computing power (synthesizing pseudo rehearsal data). In addition, the method in [2] is not applicable for the cross-modality continual instruction tuning task, as the image encoder of the multimodal large language modal can only be used to compress features and cannot be used to generate/synthesize images. On the other hand, this is a very different solution for anti-forgetting, which implies [2] could cooperate with our approach in some data-synthesized scenes.

2. By comparison, our method belongs to the non-rehearsal method, which has the advantages of 1) avoiding the need to save or generate rehearsal samples, reducing computational costs and memory space; 2) No need to train additional rehearsal examples, shortening the overall training time. 3) Strong generality, not limited by any tasks, scenarios, or models.

3. To further compare with [3], we transfer our method to the continual fine-tuning framework in [3] (Based on Mistral 7B (v0.3) [4]) and conduct continual tuning experiments on the Continual Classification Benchmark (MRCC->SST-2->Sentiment140). The following is the comparison of the experimental results (CURLORA results are reported from the original paper). It can be seen that our method has equal stability and better plasticity in LLM continual fine-tuning tasks compared to the method in [3]. The first column in the two tables represents the accurate values obtained at the current task, while the second column represents the accurate values obtained at the final task.

Ours

CURLORA

4. Finally, we transfer the method in [3] to the MCIT framework used in our paper and conduct continual instruction tuning experiments on six VQA datasets (under Instruction Type 1). The following is a comparison of the experimental results.

Experiments in 3 and 4 demonstrate that our method outperforms than method in [3].

5. By the way, the method in [3] only focuses on the LoRA fine-tuning paradigm and cannot be well transferred to other fine-tuning paradigms such as Adapter [5]. By comparison, our method can seamlessly transfer to other fine-tuning paradigms (the experimental results of replacing LoRA Tuning with Adapter Tuning are shown in the table below). It can be seen that our method has considerable performance under the LoRA and Adapter fine-tuning paradigms, and even more other Parameter Efficient Fine-Tuning (PEFT) Paradigms.

Instruction Type 1 (With Adapter Tuning)

Avg.ACC = 60.98, Forgetting = 2.27, New ACC = 62.81

Instruction Type 2 (With Adapter Tuning)

Avg.ACC = 60.30, Forgetting = 2.58, New ACC = 62.45

[2] Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal

[3] CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation

[4] Mistral 7b

[5] Introducing language guidance in prompt-based continual learning

For Weakness 3 about ''whether $\beta_t$ should be calculated to balance across different modalities'':

Thanks for your valuable suggestion. To analyze the $\beta_t$ implicitly leverages the modality-specific information, we conduct the following ablation experiments (conducted under Instruction Type 1). 1): Normal. Use the normal gradient update method to update the projection layer from the visual model to the language model and LoRA in LLM. 2): Dy(Proj). Use dynamic EMA update (our method) to update the projection layer from the visual model to the language model, while updating the LoRA in LLM with the normal gradient update method. 3): Dy(LLM). Use dynamic EMA update (our method) to update the LoRA in LLM, while updating the projection layer from the visual model to the language model with the normal gradient update method. 4): Dy(Proj+LLM). Use dynamic EMA update (our method) to update the LoRA in LLM and the projection layer from the visual model to the language model. 5): Dy(Vis). Extra insert LoRA into the vision encoder and use dynamic EMA update (our method) to update the LoRA in the vision encoder, while updating the projection layer from the visual model to the language model, and LoRA in LLM with the normal gradient update method. 6): Dy(Proj+LLM+Vis). Extra insert LoRA into the vision encoder and use dynamic EMA update (our method) to update the LoRA in the vision encoder, LoRA in LLM, and the projection layer from the visual model to the language model. The results are shown as:

It can be seen that adding our proposed method at any position in the LLaVA model can help further improve the performance of MCIT. The next is the key observations: :

(1) When training the projection layer, it is equivalent to mapping the encoded visual features to the language-text space. Both the first-order and second-order gradients of the projection layer collect the information from two modalities. Therefore, when calculating the EMA update weight of the projection layer, it can be seen as a result obtained on the basis of balancing the two modalities.

(2) Similarly, after concatenating the encoded visual features and the textual features into union features, we input them into LLM. Therefore, the features processed by LoRA are a mixture of visual and textual features. it can also be seen as a result obtained on the basis of balancing the two modalities.

(3) In addition, we have also tried using other methods to obtain modality-specific capabilities, such as directly sampling the feature space of the vision encoder and LLM, but the computational cost is expensive. Considering that we have inserted LoRA parameters in each layer of LLM, we need to sample the feature space at every layer, which is also an unbearable computational cost.

(4) It is worth mentioning that although LoRA is also inserted into the vision encoder in Dy(Proj+LLM+Vis) and dynamically updated according to the proposed method, its performance is slightly better than that in Dy(Proj+LLM). Similarly, in Dy(Vis), we also insert additional LoRA into the vision encoder and update it according to the proposed method, while keeping other trainable parameters updated with the normal gradient update method. It is still inferior to the results in Dy(Proj) and Dy(LLM). The above experiments all explain that in the case of a single modality (image), our method has little effect and can achieve maximum benefits in multimodal scenarios. This also reflects indirectly that $\beta_t$ is calculated to balance across different modalities.

For Weakness 2 about ''L1-norm demonstrates exceptional performance'':

Considering that the L2 regularization is the most widely used method, we replace the L1 regularization in our method with L2 regularization and conduct multimodal continual instruction tuning experiments under two different instructions. The results are shown as:

It can be seen that there is a significant difference in performance between L2 regularization and L1 regularization, and the L1 regularization results we chose have better performance than L2 regularization. Analyzing the reasons, we believe that it is because L1 penalizes the absolute values of the parameters, and the sign changes of the parameters have no effect on the regularization term. Compared to other regularization methods such as L2, L1 regularization is more robust to outliers [1] because it does not excessively penalize large weights, thus maintaining the stability of EMA self-adaption weights.

[1] Robust L1 Norm Factorization in the Presence of Outliers and Missing Data by Alternative Convex Programming

Dear reviewer huvJ, thanks for your valuable suggestions. Here are our responses:

For Weakness 1 about ''maintain consistent loss values rather than preserving the parameters themselves'':

Thanks for your suggestion. Here is a brief reason why we don’t maintain consistent loss values.

(1) LLaCA is a non-rehearsal method (considering some scenarios, such as online learning or privacy protection, where we cannot obtain historical data. Additionally, saving historical data also faces severe storage space costs), therefore we are unable to calculate the loss of the current model on historical tasks, which we think is a more practical setting.

(2) The second equation in Eq.(3) is only an ideal condition where unchanged parameters can best protect the model performance on historical tasks. In fact, model updates are necessary in our method. While we hope to minimize the impact of model updates on model performance on historical tasks through our method, thereby avoiding the influence on model stability. To this end, we introduce the first equation in Eq.(3), which can ensure the plasticity of the model. In conclusion, the two equations in Eq.(3) are actually a multi-task learning objective, enabling our model to have both good stability and plasticity simultaneously.

Thanks for your response to my question. While I still have concerns about the scope, I appreciate the authors' hard work during the rebuttal process and have raised my score to 5 accordingly.