CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models

Dear Reviewer ey1B,

Thank you for your comments.

• We assure you that we will enlarge Figure 1 in the next version of the paper to improve its readability and clarity.

• We will correct the reference order accordingly and discuss the relevant reference in the related works. We apologize for any confusion this may have created.

Dear Reviewer KrxH,

Thank you for your comments and suggestions. In what follows, we have tried our best to address your concerns.

• Our method stands out without replay, replay further boosts its performance: Thank you for your comment. We would first like to mention that our additional rebuttal experiments without memory replay (Table 2 in the rebuttal pdf) clearly show that our method outperforms the compared SOTA even without replay.

The use of memory replay in our proposed continual learning method is thus not redundant, but rather complementary to CLIP's existing strengths. CLIP's power primarily comes from its strong alignment between visual and textual modalities. As we finetune the model to incremental tasks, memory replay helps maintain this crucial alignment for previously learned tasks. Without it, the model might drift, compromising its image-text retrieval performance and in turn, its zero-shot capabilities on previously seen concepts while learning new ones. More importantly, like other CL setups, our use of replay is confined to the training process. Once the model is fine-tuned, there's no additional computational overhead during inference brought about by memory replay.

In addition, from the perspective of prior regularization, memory replay has broadly been used for adapting/finetuning a range of foundation models, e.g. the adaptation of vision and language models [1], latent diffusion models [2], and large language models [3]. These practical scenarios often demand that the deployed foundation models maintain their zero-shot transfer ability as well as their inference-time efficiency. Replay thus helps boost the former without affecting the latter.

Lastly, in Section 5 of the main paper, we discuss that our proposed method is agnostic of the strategy used to select exemplars for memory replay. To support this, in Table 18, Appendix D.2, we show that our method works well even with an entropy-based exemplar selection strategy – a setup where existing deterministic methods lag due to their poor predictive confidences.

• Working of VGA module: Thank you for your comment and apologies for the working of the VGA module being unclear from the main paper – we mention it explicitly in Appendix A.2 “Training for memory consolidation”. The VGA module is indeed shared across different tasks and its parameter is updated normally during the training phase when our training data comprises the current task data as well as the replay memory data. This is followed by the training phase for memory consolidation where we finetune our model on the class-balanced dataset of new data and rehearsal data. Following other well-established parameter-isolation CL algorithms [4-5], here we freeze the task-shared VGA parameters to avoid interference with the knowledge acquired during the normal training phase (given that during the consolidation phase, our training data is vastly reduced and comprises only the class-balanced data maintained in the small replay buffer).

• Computational requirements of our method: We provide a detailed comparison of the parameter and time analyses for different methods in Fig. 4 and Appendix Table 16, respectively. A major computational overhead for our proposed probabilistic method is the number of Monte Carlo samples. In App C.1, we thoroughly report the accuracy-runtime tradeoff for our proposed method with the varying numbers of MC samples. Namely, the accuracy remains poorer in the range [1,10], grows in the range [10, 20], and in general, saturates thereafter. On the other hand, the runtime grows roughly by 103% as we increase the number of MC samples from 1 to 50.

From the perspective of probabilistic finetuning, we also analyze the performance vs efficiency trade-off for the choice of prior type in Appendix C.4. As shown in Table 15, compared to the static standard normal prior, the data-driven and language-aware priors offer us minor performance gains in terms of last and avg. accuracy and backward transfer scores. However, these performance gains are neutralized by the higher cost of runtime per inference iteration. As a result, we stick to using the standard normal prior throughout our work.

References:

[1] Smith, James Seale et al. “Adaptive Memory Replay for Continual Learning.” CVPR 2024 Workshops.

[2] Kumari, Nupur et al. “Multi-Concept Customization of Text-to-Image Diffusion.” CVPR 2022.

[3] Wang, Yifan et al. “InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions.” ACL 2024.

[4] Castro, Francisco Manuel et al. “End-to-End Incremental Learning.” ECCV 2018.

[5] Douillard, Arthur et al. “DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion.” CVPR 2022.

Dear Reviewer bibK,

Thank you for your comments and suggestions. In what follows, we have tried our best to address your concerns.

• Why do we do probabilistic modeling of text feature space? We opt for probabilistic modeling of task-specific text feature space rather than image feature space mainly in light of the practical constraints imposed by the class-incremental learning (CIL) setting. In CIL, at test time, we are not given the task labels for images. As such, if we were to use task-specific adapters to model task-specific visual features distribution (rather than task-specific text features distribution as we do now), then we must know which images should be routed to what adapter – something not plausible at test time. A naive getaway would be to route the text-guided visual features to all available adapters and then infer the correct prediction based on the adapter outputs. Such an exhaustive routing mechanism would greatly increase our computational burden at test time. Modeling the distribution of visual-guided text features helps us overcome this because now our visual features serve as a shared context to which all task-specific text features (which we can distinguish simply by their labels) can attend. By sampling from the distributions over such visual-guided task-specific text features, we can compute their cosine similarities with the visual features to obtain our predictive logits.

• Comparison with zero replay memory size: In Table 2 of the rebuttal pdf, we have provided a comparison of our proposed method with the SOTA models for continual learning without replay with ViT (i.e., CODA-Prompt) and with CLIP (i.e., AttriCLIP). Here, leveraging the instance-conditioned and semantically diverse prompts of AttriCLIP provides an edge. Our variant leveraging AttriCLIP further improves its performance surpassing the SOTA. In the bottom two rows of the table, we ablate the role of our proposed language-aware distribution regularization and weight init. components and find that the former is crucial in avoiding forgetting in this setting.

• Inference time overhead: The inference time of our method is highly dependent on the number of MC samples as well as the prompt type being used. For instance, in Table 16 Appendix C.5, we show that using multi-modal prompts (MaPLe) with Ours leads to an average inference time of 0.064s which is comparable to and even lower than other existing methods like AttriCLIP (0.257s). Also, in our rebuttal to Reviewer giBN, we state that the inference time of several variants of our method remains lower than the compared SOTA (PROOF) -- 0.163s (Ours) vs 0.177s (PROOF).

We nonetheless agree that MC sampling for probabilistic modeling endows our method with higher inference time overhead compared to other deterministic methods (something we have thoroughly ablated in Fig. 8, Appendix C.1). However, such a caveat is known to be general for probabilistic models, and we believe that rejecting our method based on this (while downweighing the wide range of performance advantages) would be unfair. Put together, our proposed method outperforms several existing SOTA with better accuracy, backward and forward (zero-shot) transfer ability, and calibration, and also performs better on settings without memory replay and with restricted computational budget, as shown in Table 2 and Table 4 in the rebuttal pdf, respectively.

• Comparison with every single recent work: We would like to state that continual learning with pre-trained foundation models is a rapidly evolving field, as is the evolution of new pre-trained foundation models itself. Given this fast-moving environment, it is challenging to provide an exhaustive comparison with every single recent work. While we strive for comprehensive analysis, we hope the reviewer understands that an all-encompassing comparison may not be practically achievable within the scope of this work. That said, we have thoroughly covered the most relevant SOTA models for vision-language models (PROOF and AttriCLIP) as well as for vision-only models (DualPrompt and CODA-Prompt) across a range of datasets and settings.

Dear Reviewer giBN,

Thank you for your comments and suggestions. We have tried our best to address your concerns below.

1-1) Fair comparison with L2P and DualPrompt: The scores we report for L2P and DualPrompt are fair and use a CLIP-based backbone as well as memory replay with the same number of exemplars as our method. As also mentioned in our global rebuttal, both L2P and DualPrompt originally use the ViT-B/16 backbone pre-trained on the ImageNet-21K dataset [1] which leads to higher performance scores on the compared datasets. However, replacing this pretrained ViT backbone with that of OpenAI CLIP’s pretrained ViT backbone leads to a drop in the performance across all datasets and clearly necessitates the need for memory replay to catchup with our proposed method. The aforesaid drop in performance has also been observed by other studies adapting L2P and DualPrompt for continual learning with CLIP [2].

1-2) CODA-Prompt as baseline: Based on your comment, we have included CODA-Prompt (reimplemented using the pre-trained OpenAI CLIP’s ViT backbone) into our baselines. Like L2P and DualPrompt, the original CODA-Propmt paper uses ImageNet-21K pre-trained ViT. Replacing this with CLIP’s ViT backbone necessitates the need for memory replay (see Table 1 vs Table 2 in the rebuttal pdf) to catch up with the performance of our method. Hence, our comparison with CODA-P, L2P, and DualPrompt remains fair and consistent.

1-3) Thank you for your concern. We have included the comparison with PROOF on the Cross-Dataset Continual Learning setup in Table 3 of the rebuttal pdf.

Also, adding to Table 16, the avg. inference time for PROOF remains 0.177s which is slower than the base (Ours) and the MaPLe variant of ours but faster than our other variants utilizing task-conditioned (CoOp) and instance-conditioned (AttriCLIP) prompts.

2. Computationally-budged CL setup: Regarding computationally-budgeted CL setup, we would like to state that not all CL training setups require to be computationally-budgeted. In fact, upon finetuning of large pre-trained models like CLIP, it is often the case that their performance post-deployment, both in terms of accuracy/zero-shot transfer ability and inference time, counts the most. Our thorough evaluations make a clear statement about the upper hand of our method in terms of performance based on a number of metrics – accuracy, backward and forward (zero-shot) transfer ability, and calibration. Our inference time is also mainly dependent on the type of prompts used with our method.

In Table 4 of the rebuttal pdf, we have nevertheless provided an ablation on the “normal” budgeted CL setup of [3] where, on each incremental training task, we allocate the number of training iterations equivalent to 1 epoch on the first (base) task of each dataset. Here, our variant utilizing instance-conditioned prompts of AttriCLIP outperforms other compared methods. A further ablation shows that our proposed weight distribution regularization technique indeed remains a crucial component at tackling forgetting on the budgeted setup (see the two bottom-most rows in Table 4).

3. “The proposed algorithm does not achieve overwhelmingly superior performance .. “: In light of the consistent superior performance scores of our method without using additional exemplar memory replay and on computationally budgeted CL setup, we hope that the reviewer reconsiders their remark.

4. Justification on exemplar memory: We would like to state that the use of memory replay in our proposed continual learning method is not redundant, but rather complementary to CLIP's existing strengths. To justify this, in Table 2 of the attached pdf, we show that our method is indeed compatible with setups without memory replay, and performs either on par or better than other compared methods, including ViT-only methods.

Moreover, we would like to highlight two such perspectives to clarify that the usage of a small additional exemplar memory is not to be seen as an overhead in continual learning. First, for pre-trained foundation models, a critical performance measure for deploying their (incrementally) finetuned variants is often their inference time overhead and their downstream/zero-shot transfer abilities. The use of exemplar memory does not affect the inference time overhead of these methods. In fact, comparing Tables 1 and 2 from the rebuttal pdf, it is clear that using replay memory boosts their downstream task performance. Lastly, we clearly state the superior zero-shot (forward) and backward transfer abilities of our proposed method in Sec. 4.1 in the main paper. Second, if we were to look at exemplar memory from a prior regularization perspective, then it becomes immediately clear how predominant the usage of these is across different domains, e.g. the adaptation of Vision-Language Models [4], latent diffusion models [5], and LLMs [6].

References:

[1] Dosovitskiy, Alexey et al. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.” ICLR 2021.

[2] Zhou, Da-Wei et al. “Learning without Forgetting for Vision-Language Models.”

[3] Prabhu, Ameya et al. “From Categories to Classifier: Name-Only Continual Learning by Exploring the Web.” ArXiv abs/2311.11293 (2023).

[4] Smith, James Seale et al. “Adaptive Memory Replay for Continual Learning.” CVPR 2024 Workshops.

[5] Kumari, Nupur et al. “Multi-Concept Customization of Text-to-Image Diffusion.” CVPR 2022.

[6] Wang, Yifan et al. “InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions.” ACL 2024.