C-CLIP: Multimodal Continual Learning for Vision-Language Model

(2) Some important hyperparameters are not tested properly, e.g., for LoRA integration, the ratio of CKC loss and CLIP loss, etc.

In our work, for all experiments with C-CLIP, we simply set $\alpha=0.5$ , and it performs well. Our method introduces no additional hyperparameters for balancing the CKC loss and CLIP loss. Specifically, the CKC loss is formulated in a contrastive manner, naturally aligning with the CLIP loss. We have re-emphasized these points in the revised version. We aim for C-CLIP to be a simple yet solid method, with the important advantage of introducing no new hyperparameters to the vanilla CLIP model.

(3) Only a subset of tasks are reported in major figures, e.g., Figure 3, 4, 6, 7 and Table 4, 5. It would be beneficial to see if a similar tendency holds for the rest of the tasks during training end-to-end (at least for some crucial setups like ablation studies).

Therefore are eight tasks during continual learning, and we conducted ablation studies in two datasets that belong to the first two tasks to demonstrate the effectiveness of each component of our method. A key characteristic of continual learning is that the model learned at stage $t$ serves as the initialization for stage $t+1$. In previous CL methods, the performance differences between methods are often small on the initial tasks but grow significantly as more incremental tasks are introduced. Therefore, we present ablation experiments on the first two tasks. During subsequent learning sessions, the initial performance is already strong, making the impact of each component more pronounced as learning progresses.

Q3. Distillation in continual learning was addressed in Contrastive Continual Learning (ICCV 2021). It would be necessary to provide a comparison between CKC and previous lines of work leveraging similar concepts.

Thanks for bringing this work to our attention. We have added the citation in the revised version. The Co2L method regulates the sample similarity matrix, a regularization strategy also adopted in recent continual VLM works, such as Mod-X (ICML 2023) and DKR (AAAI 2024). Experiments show that our method outperforms this strategy. We believe that constraining the sample similarity matrix is not ideal because new datasets are often not well-distributed in the old feature space, leading to a trade-off between learning new knowledge and preserving the old feature space. In contrast, the proposed CKC loss, which has a formulation similar to the CLIP loss and is well-aligned with it (as demonstrated in Figure 3), enables simultaneous learning of new and old knowledge.

Q4. Eq. 23 is recursive with respect to K, could this bound serve as a meaningful bound? (could be considerably large for some nontrivial L and M?)

Thank you for pointing this out. Because Eq. (23) is recursive, the Lipschitz constant $K$ can grow exponentially with the number of layers if $L_\sigma M_w > 1$, making the bound potentially very large and less meaningful. To ensure the bound remains practical, we need to keep $L_\sigma M_w \leq 1$, for example, selecting activation functions with $L_\sigma \leq 1$ and controlling the weight norms $M_w \leq 1$.

Q5. L96 - cite the correct paper (pioneering works, not recent ones)

Thank you for this suggestion. These two recent ones are survey papers on continual learning. Following your suggestion, we cite pioneering works like EWC in the revised version.

Q6. $h_{\psi}$ in L277 used elsewhere after training? Why is an identical projector used for both vision and text embeddings in equation 5?

Thank you for your comments. Below is our explanation regarding the CKC projector:

• The projection layer $h_{\psi}$ in Line 277 is only used to compute the CKC loss. After training, C-CLIP and CLIP retain the same structure, with LoRA modules integrated into the model via reparameterization, and the projection layer $h_{\psi}$ is discarded.
• The vision encoder and text encoder share a single projector for the CKC loss because the text and vision embeddings in CLIP are already aligned by the two built-in projection layers in CLIP itself. These existing projection layers in CLIP align the outputs of the vision and text encoders. CKC projector works in the text-vision aligned space, a single shared projector is sufficient.

We sincerely thank Reviewer L3rn for your review and are grateful for the time you spent on our submission. We are glad for the acknowledgment that the problem is practically important, the proposed method is effective and reasonable, and the experiment is strong. We appreciate the detailed suggestions for improving the presentation of the paper. Below we would like to give detailed responses to each of your comments.

Q1. Delivery should be significantly improved.

Thank you for your careful reading. We revised the submission following your comments and tried our best to check and correct other confusing descriptions and typos.

(1) What CIL and MTIL stand for? (I don't find Table 1 to be really necessary, though.)

CIL denotes class-incremental learning, and MTIL denotes multi-domain task incremental learning (MTIL) benchmark, which is proposed by Zheng et al., (2023) and followed by Yu et al. (2024). Following your suggestion, we provide a clear explanation of CIL and MTIL in section 2.1. We have added a new table, Table 2, below Table 1 to provide a clearer comparison between the VLCL benchmark and previous CL benchmarks. These tables aim to highlight the distinctions of the VLCL benchmark.

(2) Figure 2 does not effectively convey what CKC is about.

Thanks for your comment. We provide a new Figure 4 to illustrate the proposed CKC in the revised version. Compared with other methods, CKC differs primarily in two aspects:

• CKC introduces a projector $h_{{\psi}}: \mathcal{Z} \rightarrow \mathcal{Z}$ after the text encoder and optimize the model in the projected space.

• CKC optimizes CLIP to learn a better feature space from the old model, rather than aligning with it. This is achieved by increasing positive and negative pairs for contrastive learning: the relevant old feature instances of each image-text pair are treated as positives, while those of other pairs are treated as negatives. This optimization learns new knowledge and prevents forgetting simultaneously.

We hope the newly added Fig. 4 illustrates the differences between CKC and previous regularization methods, and explain why the CKC loss can be optimized alongside the CLIP loss.

(3) The range covered by the y-axis in Figure 3c seems too wide,..,if is universally observed in other tasks.

Thanks for this suggestion. We have improved Fig. 3c to illustrate more details, showing that the CKC loss and CLIP loss exhibit the same downward trend during training. Of course, this phenomenon is consistent across all tasks.

(4) What does Figure 3 in L284 refer to?

In our previous submission, Figure 3 in L284 refers to Figure 3c, which compares the training loss curves among our method and other regularization losses. In the revised version, we modify it to the newly provided figure 4.

(5) Please assign proper variables for equations 5 and 6, instead of using z everywhere.

Good suggestion. We have modified the $\widetilde z^{t}_i$ as $\widetilde h^{t}_i$, which denotes the prejected features.

(6) Many typos, e.g., space is view as (L280), loss is use in the (L294), etc.

Thank you again for your careful review. We have corrected those typos and thoroughly checked the revised paper for any other errors.

Q2. While performance improvements in all major experiments are impressive, it is hard to tell whether these results are reproducible and rigorously analyzed.

We promise to release the source code and model checkpoints. Additionally, our method is simple and modular, the LoRA integration and CKC loss are easy to implement. In the following, we respond to each of your concerns in detail.

(1) The codes are not provided, and the details in the paper may not be sufficient to fully reproduce the experiments reported.

The code of this work (the proposed benchmark and our method) is promised to be publicly available once the work is published. Implementation details are provided in Section 5, and we are glad to share any additional details about our method.

We sincerely thank Reviewer 45wV for your review and are grateful for the time you spent on our submission. We are glad for the acknowledgment that the proposed benchmark is rich and our method is novel. Below we would like to give detailed responses to each of your comments.

Q1. The introduction needs a significant extension.

Thank you for bringing those valuable studies to our attention. We have added them to the introduction section of the revised version, for a comprehensive review of relevant research papers. In the revised version of our paper, we have reorganized the existing challenges into three main sections:

• Compared to standard unimodal continual learning scenarios, multimodal settings present greater complexity.
• The evaluation of vision-language models (VLMs) remains insufficient.
• Traditional CL methods apply regularization to reduce forgetting, which hinders the learning of new tasks.

We hope the extended introduction aligns with your suggestions. It reorganizes challenges and provides a clearer structure to emphasize the unique contributions of our work.

Q2. I think the overall benchmark definition is confused and badly explained.

Thanks for this comment. Following your suggestion, we have thoroughly reorganized the description of the benchmark, added a new Table 2, and refined the content in section 5.2. Here the is updated description of the benchmark (excluding the citations of datasets for the Markdown presentation):

VLCL benchmark. To evaluate the continual learning performance of vision-language models, we establish a novel benchmark that includes three evaluation tracks, summarized in Table 2. (1) Multimodal continual learning. Eight image-caption datasets are used in this track. Among them, Flickr30K and COCO-Caption are general real-world datasets. Other datasets, including Pets, Lexica, Simpsons, WikiArt, Kream, and Sketch, represent specific domains such as AI-generated images, art, clothing, illustrations, and sketches. (2) Zero-shot retrieval. One held image-caption dataset, HAVG, is used to assess retrieval performance on unseen datasets. (3) Zero-shot classification. Previous CL work on CLIP has overlooked an important aspect: the forgetting of general representations (i.e., zero-shot generalization). To evaluate this, we tested zero-shot performance on six image classification datasets: ImageNet, CIFAR-100, StanfordCars, Flowers, DTD, and Food101.

In the revised paper, we clarified an important point: previous CL work often divides a single dataset into multiple incremental tasks, requiring a detailed explanation of the dataset partitioning. However, such a setting does not exist in our method, as we treat each dataset as a separate task.

Q3. It's worth mentioning the whole corpus of rehearsal-free continual learning methods based on parameter-efficient tuning.

Following your suggestions, we added rehearsal-free continual learning methods based on parameter-efficient tuning in section 2.1 on Page 2. Besides, to further demonstrate the strength of our method, we conduct experiments to compare our method with prompt-based CL methods such as L2P and CPE-CLIP, and added a new Table 8 in the revised paper. For L2P, we use a pool of 40 prompts (length 5) and append 5 prompts at the input. For CPE-CLIP, we add 2 prompts per layer, following the original configuration. Results in Table 8 reveal the following observations:

• Prompt-tuning maintains general zero-shot performance in ImageNet-1K accuracy. While only using LoRA leads to forgetting general zero-shot knowledge as tasks increase.
• Prompt-tuning does not eliminate forgetting on downstream tasks. Updating prompts for new tasks causes obvious forgetting of prior tasks, e.g., the performance on the first task (I2T-T1) and all tasks (I2T@R1) is much worse than LoRA after learning all eight tasks.

In summary, although prompt-tuning exhibits less forgetting on the original model, it learns little from new tasks and forgets newly acquired knowledge easily when the model underperforms on downstream tasks. The combination of LoRA and CKC proves to be more effective in this scenario.

Q4. Is the CLIP (first line) in Table 2 fine-tuned or just frozen?

The CLIP (first line) in Table 2 is just frozen, and the results are zero-shot performance. We have made this clear in the revised version.

Q5. Please state from the beginning that we are in a class-incremental learning scenario and thus that the benchmark is for CIL, as usual in any CL paper.

Thanks for this comment. Compared to the well-defined category-based continual learning in image classification, we focus on multimodal continual learning for VLM like CLIP, and view each dataset as one task, similar to the domain-incremental learning in classical CL or capability continual learning. The task ID information is not needed at inference time. We have made this clear on Page 4 (Evaluation metric part) in the revised version.

Q6. Line 255. "leran" -> "learn".

Thanks. We have corrected those typos and tried our best to check and correct other typos.

Q7. "single-modal scenario" is an unpopular term. I think the literature agrees with using the term "unimodal" instead.

Good suggestion. We have modified this point in the revised version following your suggestion.

We sincerely appreciate Reviewer 8g5w for the positive recommendation as well as the valuable suggestions. We really appreciate your kind words that our work is innovative with key strengths. Below, we would like to give detailed responses to each of your comments.

Q1. Analysis of the computational costs involved in using CKC during the fine-tuning process.

Thanks for this suggestion. We have added the results in Appendix A.3 of the revised version. As shown in Table 9, the additional losses introduce some overhead. For instance, on COCO, LWF and Mod-X increased training time by 19.3% and 28.2%, respectively. The computational overhead of CKC is similar to these regularization losses, leading to a 25.2% increase in training time.

Q2. Comparison against other well-known parameter-efficient CL methods that could strengthen the evaluation.

Good suggestion. In Table 8 of the revised paper, we compare our method with prompt-based CL methods such as L2P and CPE-CLIP. For L2P, we use a pool of 40 prompts (length 5) and append 5 prompts at the input. For CPE-CLIP, we add 2 prompts per layer, following the original configuration. Results in Table 8 reveal the following observations:

• Prompt-tuning maintains general zero-shot performance in ImageNet-1K accuracy. While only using LoRA leads to forgetting general zero-shot knowledge as tasks increase.
• Prompt-tuning does not eliminate forgetting on downstream tasks. Updating prompts for new tasks causes obvious forgetting of prior tasks, e.g., the performance on the first task (I2T-T1) and all tasks (I2T@R1) is much worse than LoRA after learning all eight tasks.

In summary, although prompt-tuning exhibits less forgetting on the original model, it learns little from new tasks and forgets newly acquired knowledge easily when the model underperforms on downstream tasks. The combination of LoRA and CKC proves to be more effective in this scenario.

Q3. Could the C-CLIP approach be generalized to VLMs beyond CLIP, or are specific adjustments required?

C-CLIP mainly consists of LoRA integration and constrastive knowledge consolidation (CKC). (1) The LoRA integration technique can be applied to any VLM such as BLIP, BLIP2, or InstructBLIP, as well as more advanced multimodal large language models (MLLM) such as LLaVA and miniGPT4. (2) The CKC can be directly applied to any contrastive training-based model. For multimodal language models that use language modeling loss, it is difficult to adapt CKC to them. However, since almost all VLM and MLLM leverage CLIP encoder, therefore the proposed C-CLIP can be used to enhance the version-language alignment ability of MLLMs. Inspired by your valuable suggestion, we added a discussion of this point in the revised paper, and will consider the continual learning for LLaVA in future work.

Q4. Could the authors further elaborate on why CKC avoids the trade-offs associated with other regularization methods?

Thank you for this comment. Specifically, other regularization methods typically align the new model with the old feature space. However, for new tasks, the old feature space could perform poorly, forcing the model to trade off between learning new knowledge and preventing forgetting. In our method:

• We introduce a projector $h_{{\psi}}: \mathcal{Z} \rightarrow \mathcal{Z}$ after the vision and text encoders, optimizing the model in the projected space. In this way, the new and old feature spaces keep connected but are not identical, which improves the plasticity for learning new tasks while also maintaining the previous knowledge.

• CKC optimizes CLIP to learn a better feature space from the old model, rather than aligning with it. This is achieved by increasing positive and negative pairs for contrastive learning: the relevant old feature instances of each image-text pair are treated as positives, while those of other pairs are treated as negatives. This optimization learns new knowledge and prevents forgetting simultaneously, thereby avoiding trade-offs.

We have added a new Figure 4 in the revised paper to illustrate CKC. We hope that Figure 4 helps to clearly demonstrate the differences between CKC and previous regularization methods, as well as explain why the CKC loss can be effectively optimized alongside the CLIP loss.

We sincerely appreciate Reviewer v9wt for the review and are grateful for the time you spent with our submission. We are glad for the acknowledgment that our approach is promising, the proposed benchmarks are valuable and the experiments are solid. We wish to address your concerns by giving detailed responses to each of your comments as follows:

Q1. Results with more multi-modal models, such as other CLIP variants with different backbones (say ViT-L/14).

Following your suggestions, we have implemented our method with different backbones, such as ViT-B/32, ViT-B/16, ViT-L/14 and ViT-L/14@336px, and the results are shown in Table 6 in the revised paper. Specifically, due to memory constraints, we reduce the batch size to 256 for the larger ViT-L models. As shown in Table 6, ViT-L significantly outperforms ViT-B, with substantial improvements in zero-shot performance on downstream datasets. Our method performs well across various ViT architectures, maintaining strong zero-shot capabilities and excelling in image-text retrieval on downstream tasks.

Q2. The authors could also explore more vision-language tasks, e.g., generative VLMs such as LLaVA.

Thanks for this suggestion. This paper focuses on classical version-language models like CLIP, which uses separate vision and text encoders aligned via contrastive loss to maximize the similarity between paired image-text embeddings. It is designed for tasks like retrieval and zero-shot classification.

While generative VLMs like LLaVA integrate a vision module with large language models (LLM), in which the version module is frozen and only the LLM is fine-tuned on downstream datasets with language modeling autoregressive loss. Therefore, it is difficult to directly adapt the proposed constrastive knowledge consolidation to LLaVA. Nevertheless, considering the version encoder in LLaVA is from CLIP and plays an important role in LLaVA, we believe the proposed C-CLIP can also enhance the version ability of LLaVA.

Inspired by your valuable suggestion, we added a discussion of this point in the revised paper, and will consider the continual learning for LLaVA in future work.

Q3. (Minor) The authors have missed a close work [Ref1]. That said, the main contributions on modeling side are quite different.

Thanks for bringing this interesting work to our attention, and we have added it to the related work (section 2.1) of the revised version.

Q4. If the model is provided, is the task id in the data-stream? Does it know that the task is say pet-classification or is that information hidden?

The task id is not provided, and we do not need the task id information at inference. We focus on multimodal continual learning of VLMs and view each dataset as one task, similar to the domain-incremental learning in classical continual learning. From the perspective of methodology, on the one hand, we integrate the LoRA during the continual learning process, and we only have one set of LoRA after training. Therefore, the task ID is not needed at inference time; on the other hand, the proposed CKC is based on constrastive loss, which also does not need the task ID information at both the training and inference stages. We have made this clear on Page 4 (Evaluation metric part) in the revised version.