LADA: Scalable Label-Specific CLIP Adapter for Continual Learning

Dear Reviewer oKy3,

Thank you for your detailed review. We address your concerns one by one in the followings.

W1 Claims And Evidence: the evidence provided in Table 4 focuses on ablation without comparisons to other methods. Experimental Designs Or Analyses: computational efficiency would further support the paper’s claims.

A1 We conducte a computational efficiency experiments. We compare our method with other methods using a consistent input batch size of 64 in the full-shot setting. Time comparisons for MoE-Adapters and ours were conducted on a single NVIDIA 4090 GPU. The results are summarized in the table below:

W2 Essential References Not Discussed:

1. The related work should discuss CL methods based on CLIP that utilize PEFT techniques.
2. The authors should discuss related works such as CODA-P, APG, and EvoPrompt.

A2 We appreciate this suggestion and will add a dedicated subsection titled "Continual Learning Methods Based on PEFT" to the related work section of the revised manuscript:

Parameter-efficient fine-tuning (PEFT) has been widely adopted in continual learning to enhance representation learning for specific tasks. Prompt-based methods, such as L2P [1], expand the prompt pool as new tasks are learned, relying on auxiliary losses. CODA-Prompt [2] propose a end to end prompt selection methods to increase plasticity. APG [3] introduces a prompt generator to reduce the gap between pretraining and future tasks, while EvoPrompt [4] proposes an adaptive and continuous prompting approach to alleviate issues of selection mismatches and limited prompt shareability. MoE-Adapters [5] insert a mixture of adapters into the image encoder, activating a few for each task. However, these methods typically involve an auxiliary parameter selection step during inference to extract image features using the expected prompts or adapters, which can lead to misassignments and degrade classification performance.

W3 Essential References Not Discussed: The paper lacks discussion and comparison with prior CL methods that leverage prototypes for pseudo-feature replay, such as HiDe-Prompt and CPP.

A3 We are grateful for the reviewer’s suggestion. We will incorporate a discussion on these approaches in the related work of the revised manuscript:

Prototype-based methods leverage pseudo-feature replay to mitigate forgetting in continual learning. HiDe-Prompt [6] optimizes hierarchical components with task-specific prompts using a single prototype per class, while Contrastive Prototypical Prompt [7] employs task-specific prompt-tuning with contrastive learning and multiple prototypes per class to address semantic drift and prototype interference. However, as highlighted in our introduction, while backward forgetting has been effectively addressed, these approaches often overlook the impact of pseudo-feature replay on the model’s original generalization ability, leading to forward forgetting.

W4 what does the i represent in Eq. (5)?

A4 We apologize for the oversight. The notation i represents the image embedding, as defined in line 115 of the original manuscript. We will explicitly clarify this directly at Eq. (5) in the revised version.

W5 More explanation is needed for the statement “a linear weighting between the logits produced by LADA and the corresponding text features”. Clarifying this process is crucial for understanding how final predictions for seen classes are made.

A5 We regret the confusion. The linear weighting is a fixed hyperparameter (set to 1.0 in experiments), not learned, balancing LADA logits and text features equally. This will be clarified in the revision.

W6 Questions For Authors: My primary concern is the lack of discussion on related works.

A6 We have rewritten the related work in A2 and A3 . We commit to integrating these revisions into the manuscript.

[1] Wang, Zifeng, et al. Learning to prompt for continual learning. CVPR. 2022.

[2] Smith, James Seale, et al. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. CVPR 2023.

[3] Tang, Yu-Ming, et al. When prompt-based incremental learning does not meet strong pretraining. ICCV 2023.

[4] Kurniawan, Muhammad Rifki, et al. Evolving parameterized prompt memory for continual learning. AAAI 2024.

[5] Yu, Jiazuo, et al. Boosting continual learning of vision-language models via mixture-of-experts adapters. CVPR 2024.

[6] Wang, Liyuan, et al. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. NeurIPS 2023.

[7] Li, Zhuowei, et al. Steering prototypes with prompt-tuning for rehearsal-free continual learning. WACV 2024.

Dear Reviewer EaVq,

Thank you for your detailed review. We address your concerns one by one in the followings.

W1: Essential references not discussed of [1] and the second point in Other Strengths And Weaknesses of comparisons with RanPAC [1]

A1: We acknowledge the reviewer’s comments regarding the article and address the concerns about the omission of RanPAC [1] as an essential reference, as well as the need for a comparison between RanPAC and our proposed method.

1. Core Idea of RanPAC

   RanPAC [1] projects features into a higher-dimensional space, which has been shown to benefit continual learning. RanPAC [1] tackles multicollinearity in CLIP’s original feature dimensions by employing random Gaussian projection matrices and nonlinear activation functions (e.g., ReLU). Specifically, it extracts CLIP features using a frozen encoder, projects them via a Gaussian matrix $W$, applies nonlinear transformations, and uses ridge regression on 0-1 encoded labels to train classifiers. This approach leverages second-order statistics to reduce feature redundancy and enhance classifier accuracy.

2. Key Differences and Advantages of LADA

   • Feature Optimization

     RanPAC optimizes features using static random Gaussian projection matrices to reduce multicollinearity in CLIP features, but it lacks adaptability to task-specific nuances, often failing to capture fine-grained or diverse class characteristics effectively. In contrast, LADA introduces a label specific memory dynamically amplifying CLIP feature most relevant to each class. This process condenses features into sparse, high-dimensional representations tailored to individual tasks, enhancing discriminability without modifying the frozen CLIP encoder. By leveraging this class-specific adaptability, LADA achieves superior plasticity and stability.

   • Cross-Task Adaptation

     RanPAC’s dependence on fine-tuning the CLIP encoder for the first task incurs substantial computational overhead and risks eroding CLIP’s pre-trained general knowledge, making it impractical for continual learning across diverse tasks. In contrast, LADA takes a fundamentally different approach by freezing the CLIP encoder entirely and training lightweight a label-specific adapter.

3. Empirical Validation

   To compare LADA with RanPAC, we integrate RanPAC into our framework (denoted as BF+RanPAC+DPT, where BF refers to baseline fine-tuning and DPT to Distribution-Preserved Training) and evaluate it against our method (BF+LADA+DPT). In the X-TAIL 16-shot setting, the results demonstrate that BF+LADA+DPT outperforms BF+RanPAC+DPT by 1.4% in Transfer, 1.4% in Average, and 1.0% in Last metrics, averaged across the 10 tasks.

[1] McDonnell, Mark D., et al. "Ranpac: Random projections and pre-trained models for continual learning." NeurIPS,2023.

W2: It is recommended to add a flowchart of the LADA method or end-to-end loss function in the Overall Framework section

A2: We appreciate the reviewer’s suggestion. The overall framework and end-to-end loss function of LADA are available in an anonymized link. We will include them in the revised manuscript.

W3: Experiments on some important datasets are necessary, such as CIFAR100.

A3: Thank you for your suggestion. CIFAR-100 is indeed an important dataset. However, due to overlapping class names with other datasets, it was not included in the original X-TAIL dataset to ensure accurate evaluation. Nevertheless, we have conducted additional experiments on Inclusion of CIFAR-100 in the X-TAIL to further demonstrate the robustness of our approach. Under the 16-shot setting, our method still achieves state-of-the-art results, with improvements of 4.3%, 2.0%, and 1.0% on the Transfer, Average, and Last metrics, respectively. Please refer to the anonymized link for detailed experimental results.

We hope this rebuttal addresses your concerns. Please let us know if further refinements are needed!

Dear Reviewer vJJv,

Thank you for your detailed review. We address your concerns one by one in the followings.

W1 Other Strengths And Weaknesses: Label-Specific CLIP Adapter bears a strong resemblance to the CLIP-based few-shot learning method Tip-Adapter [5]. Questions For Authors: Similarity to Tip-Adapter.

A1 We agree with the reviewer that the adapter module in LADA shares similarities with Tip-Adapter. However, our key contribution lies in generalizing this module for continual learning tasks—a scenario where Tip-Adapter fails to perform effectively.

We demonstrate that the adapter (which we term label-specific memory) can naturally capture task-specific data distributions and mitigate forgetting. To maintain scalability and simplicity, we initialize the memory cache using k-means clustering and adopt Tip-Adapter’s output formulation. In our view, adapting Tip-Adapter to continual learning is nontrivial: LADA introduces significant improvements to address both backward and forward forgetting.

Moreover, LADA seamlessly extends to few-shot learning, matching Tip-Adapter’s original setting. Under the 16-shot benchmark (using CLIP ViT-B/16, following [6]), LADA outperforms Tip-Adapter-F by 1.6% (see table below). Notably, it achieves this with only 1/4 of the feature embeddings required by Tip-Adapter, underscoring its efficiency.

The performance results for Tip-Adapter-F are sourced from recent work [6].

Summary of contributions

Contribution 1: We design a text encoder fine-tuning framework which enhances the classification ability and naturally obtains the transfer ability of CLIP model.

Contribution 2: We propose LADA, a lightweight CLIP adapter that condenses task-agnostic knowledge to transform CLIP representation into label-specific features, eliminating the need for parameter selection as seen in previous mainstream methods.

Contribution 3: We propose Distribution-Preserved Training which controls the influence of prototypes by incorporating their contribution weights into the loss function. We not only addresses backward forgetting but also tackles forward forgetting, a challenge overlooked in prior methods.

W2 Essential References Not Discussed: There are several essential references related to Distribution-Preserved Training that are either briefly mentioned or not cited in the paper.

A2 Thanks for the reviewer highlighting these relevant papers and we will include citations and discussions. Our approach aligns with a standard Gaussian distribution when $\lambda_2=1$, similar to the prototype augmentation method in [1]. When $\lambda_2 > 1$, our loss function (Equation 10) leverages the mixture weights $\pi$ from the GMM to perform more sophisticated augmentation. As shown in Table 4, when $\lambda_2=1$, the Transfer performance is lower compared to when $\lambda_2 > 1$. This indicates that the effectiveness of DPT while prior methods in [1-4] do not address forward forgetting in pre-trained models, our approach explicitly tackles this issue.

W3 Other Comments Or Suggestions: The improvement of BF achieved by the presented modules appears to be limited.

A3 In Tab.2 and Tab.3, our basic fine-tuning framework (BF) already outperforms several baseline methods, demonstrating the challenges prior approaches face in task-agnostic settings. In the full-shot setting, these modules improve Transfer by an average of 2.3%, Average by 2.0%, and Last by 1.3% compared to the BF framework alone.

W4 Other Comments Or Suggestions: It is unclear whether this linear weighting is a hyperparameter.

A4 We regret the confusion. The linear weighting is a fixed hyperparameter (set to 1.0 in experiments), not learned, balancing LADA logits and text features equally. This will be clarified in the revision.

We hope this rebuttal addresses your concerns. Please let us know if further refinements are needed!

[1] Zhu, Fei, et al. Prototype augmentation and self-supervision for incremental learning. CVPR 2021.

[2] Tang, Yu-Ming, et al. When prompt-based incremental learning does not meet strong pretraining. ICCV 2023

[3] Zhang, Gengwei, et al. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. ICCV 2023.

[4] Huang, Linlan, et al. Class-incremental learning with clip: Adaptive representation adjustment and parameter fusion. ECCV 2024.

[5] Zhang, Renrui, et al. Tip-adapter: Training-free adaption of clip for few-shot classification. ECCV 2022.

[6] Zanella, Maxime, et al. Low-rank few-shot adaptation of vision-language models. CVPR 2024.

Dear Reviewer 4dD8,

Thank you for your detailed review. We address your concerns one by one in the followings.

W1 It should have been clearly mentioned that finetuning text encoder is part of the proposed approach.

A1 We sincerely apologize for the lack of clarity and fully agree with your assessment that the role of text encoder finetuning should have been explicitly stated early in the paper. We will make the following modifications in the next version.

1. Introduction (Section 1):

   • We will add a paragraph in the introduction explicitly stating that our proposed approach jointly optimizes both the image features and text encoders. Specifically, we will emphasize that we first establish a text encoder fine-tuning approach for continual learning by combining parameter freezing and distillation techniques.

2. Methodology (Section 3):

   • We will explicitly outline the full pipeline, including text encoder fine-tuning for continual learning, before introducing implementation details. Also, an overall framework will be added, illustrating both text encoder fine-tuning module and Label Specific CLIP Adapter module.
   • The section titled "A Simple Baseline via Text Encoder Fine-tuning" will be renamed to "Text Encoder Fine-tuning Framework for Continual Learning" to clarify its role as an ablated variant of our approach. We will add a transition sentence: "To address continual learning in X-TAIL senario, we first introduce a baseline framework with text encoder finetuning only."

W2 The finetuning of text encoder does not clearly mention the full form of the loss function used. If I understood correctly, Eq. 3 and 4 are used for current and past tasks respectively and the total loss is a summation of the two.

A2 We apologize for the confusion. In the revised manuscript, we will add the following sentence in Section 3.2 after Equation 4: "The total training loss is the sum of the current task loss (Eq. 3) and the distillation loss (Eq. 4), enabling joint optimization of both current and past task objectives."

W3 Is Eq. 8 or 10 used for previous task? Here too, the writeup does not clearly state the full loss function.

A3 We will clarify that total loss combines Eq. 7 (current task) and Eq. 10 (Distribution-Preserved Training loss for previous task) in next version.

W4 The introduction is unnecessarily long and not well written.

A4 Thank you for the constructive feedback. We will revise the introduction by:

1. Merge paragraphs 2-3 to concisely present the dual challenges of memory stability (preserving past knowledge) and learning plasticity (adapting to new tasks) in continual learning.
2. Remove detailed discussions of prior methods (e.g., regularization-based approaches) and shift them to the Related Work section.

W5 An analysis or ablation on why GMM modeling was needed would be good to justify this choice.

A5 We acknowledge that the the choice of GMM should be explicitly stated. We will add the following discussions in the next version of paper.

Choice of GMM: Distilling only cluster centers loses fine-grained distribution information. GMM models the full feature distribution of past tasks, which controls the influence of prototypes by incorporating their contribution weights $\pi$ into the loss function Eq. 10.

Impact of GMM: Table 3 shows DPT based on GMM boosts Transfer and Average performance. Table 4 shows increasing the number of image prototypes $\lambda_2$ further improves Transfer performance.

W6 Is "CLIP Adapter" the right term for the technique proposed? Would it not give the impression that the method is using adapters for image encoder before one reads the intro?

A6 Thank you for raising this concern. The term "CLIP-Adapter" has been recognized in vision-language research [1] to denote lightweight feature adaptation after the frozen CLIP backbone extracts representations, not input-level modifications. This naming convention emphasizes post-encoder refinement for downstream tasks, aligning with its technical definition and community usage. We will clarify this distinction explicitly in the revised manuscript to avoid potential misinterpretation.

[1] Gao, Peng, et al. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision 132.2 (2024): 581-595.

W7 Will rely on other reviewers for novelty assessment.

A7 Please refer to A1 and Summary of contributions of reviewer vJJv.

We hope this rebuttal addresses your concerns. Please let us know if further refinements are needed!