Fine-tuning CLIP’s Last Visual Projector: A Few-Shot Cornucopia

Q1. How does ProLIP compare to LP++ with the proposed regularization?

• ProLIP fine-tunes the last visual projection matrix of the vision encoder and the regularization keeps these trainable weights close to the pretrained ones. LP++ keeps the entire vision encoder frozen and adds a linear classifier that is initialized from text embeddings. Adding our regularization to LP++ would not help as the encoder is frozen. However, adding regularization to the newly initialized linear classifier could be possible but unclear in impact as LP++ already has specific regularization from its optimizer. We will explore this in future work.

W1. In-depth analysis and discussion

• Please refer to answer W1 of Reviewer NwVL on a similar question.

W2. Significant performance improvements only in specific datasets

• We would like to highlight that ProLIP achieves top-performance on few-shot image classification on CLIP on 11 datasets on different few-shot regimes outperforming runner-up methods by up to 2% (Tab. 1), few-shot domain generalization with highest average performance on both in-distribution and out-of-distribution data (Tab. 3), robustness to distribution shift in test-time adaptation without training data (Tab. 5).
• On cross-dataset generalization (Tab. 4) zero-shot CLIP outperforms on average all fine-tuning strategies, while ProLIP is on average second to it, outperforming all fine-tuning methods.
• In addition, ProLIP is extremely simple and does not change the original CLIP architecture nor impacts runtime.

W3. The paper conducted a grid search for training parameters

• We do conduct a grid-search for the tuning of the hyper-parameters and for getting insights in the inner-functioning of ProLIP. However we also show that using the regularizer ($\lambda>0$) is better to avoid severe overfitting, and that state-of-the-art results can be still achieved with a fixed learning rate over all the datasets, and a fixed $\lambda$ which can be chosen as a decreasing function of the number N of shots.
• For the other methods we report fairly the scores obtained in their original settings with the tunings of their authors or as reported in prior works.

W4. Missing performance comparison

• Thank you for the suggested references. Both PromptSRC and CoPrompt leverage learnable visual prompts and are ViT-specific. Additionally, CoPrompt leverages the combination of learnable adapters and prompts. Their experiments focus on base-to-novel classification and domain/dataset generalization of few-shot learning.
• In the evaluation of ProLIP we have considered multiple different few-shot protocols and methods that are compatible with both ResNet and ViT encoders in line with recent published works in this area.
• We will consider these works in the update.

S1. Potentially mistaken review summary

This fine-tuning uses a masked reconstruction loss to learn semantic contributions for each image patch, enhancing the model's ability to capture local semantics without the need for additional annotated data.

• This summary does not seem to describe our method for few-shot fine-tuning of CLIP for image classification. We do not use masking, patch-level semantics or reconstruction loss.

W1. Novelty - trivial trick

• Our approach is indeed very simple and highly practical, but we do not consider it as a trivial trick. To the best of our knowledge such an approach has not been proposed before in the CLIP adaptation literature.
• Most of the related approaches add new layers or adapters or learnable prompts on the vision encoder or text encoder side. However they change the original architecture and/or require full back-propagation steps, impacting both training and inference time.
• Linear probing variants add a new layer on top of CLIP and train with a regular classification loss.
• ProLIP does not change the architecture of CLIP and fine-tunes only the last vision projection layer. The supervision signal is given via text embeddings, thus remaining faithful to the original pretraining regime of CLIP. In addition we propose a regularization strategy to prevent overfitting during the few-shot training.
• Moreover, the architecture-agnostic nature of ProLIP makes it generic and suitable to different multi-modal pretrained networks (ResNet, ViT). Due to intrinsic differences across architectures, finding a unified method to efficiently fine-tune vision-language models based on the pretrained model weights is not trivial.

W2. Experimental performance

• Thank you for the suggested references. In our evaluation we have considered different few-shot protocols and methods that are compatible with both ResNet and ViT encoders in line with recent published works in this area.
• MaPLe is a ViT-specific method as it uses learnable visual prompts. It focuses on base-to-novel classification and domain/dataset generalization of 16-shot learning. Preliminary experiments on base-to-novel generalization with ViT-B/16 show that ProLIP is better than MaPLe on base classes and worse on novel, with similar harmonic mean scores.
• We will update the references and experiments.

W3. Sensitivity to choice of regularization strength $\lambda$

• We respectfully disagree. We show that the use of different regularization strengths reduces overfitting and it is compatible with a wide range of learning rate values (See Fig.2 and Tab. 8). When no regularization is applied, accuracy drops dramatically for learning rates higher than 1e-5.
• Besides we propose in Sec. 3.2 a strategy to reliably set $\lambda$ as a function of the number of shots $N$ available for training (see Tab. 2)

W4. The paper assumes that pre-trained models contain sufficient knowledge for local semantics

• ProLIP does not make any assumptions on local semantics. ProLIP deals with image classification and fine-tunes the last projection layer of CLIP that processes global features.

W1. Trivial work

• Our approach is simple and effective. Despite its extreme simplicity, adapting CLIP using our approach has not been proposed in the literature. Please refer to answer W1 of Reviewer biKw for additional responses.

W2. Few insights into the design

• Please refer to answer W1 of Reviewer NwVL on a similar question.

W3. Limited scope

• Indeed we do not make other claims than few-shot learning for vision-language models, as also illustrated in the title. We believe that in the era of large foundation models trained with minimal or no supervision on large amounts of data, devising strategies to adapt such models on few labeled samples with good performance and without overfitting is a challenging and important research topic and application. This is also confirmed by the numerous publications in this area in the past 2 years.
• Besides, in the evaluation we consider multiple types of experiments and settings: {1,2,4,8,16}-shot learning on 11 datasets, domain generalization, cross-dataset generalization, test-time adaptation.

W4. Misaligned results in Tab. 1 and 2

• We believe there is a misunderstanding from the reviewer here. In Tab. 1 (Sec. 3.1) we report ProLIP results with hyper-parameters identified by grid-search and compare with SoTA methods. In Tab. 2 (Sec. 3.2) we report results for our strategy to set the regularization $\lambda$ as a function of N with different learning rate values.
• The aim of Tab. 2 is to show that we can obtain good (SoTA) scores for ProLIP without much effort spent on hyper-parameter tuning.

W1. In-depth analysis on effectiveness

• By fine-tuning only the last visual projection matrix with supervision from text embeddings via cross-entropy loss, we argue that this strategy is closer in principle to CLIP's original pretraining regime. This is different from other methods that insert additional parameters and/or add a linear layer or MLP with regular classification supervision.
• We also argue that the regularization that we impose to keep the trainable weights close to the pretrained ones helps in preserving the rich knowledge encapsulated in CLIP weights and in preventing overfitting and forgetting in the few-shot learning regime.
• Towards a better understanding of ProLIP, we study and evaluate quantitatively ProLIP under different tasks (few-shot classification, domain generalization, cross-dataset generalization, test-time adaptation) and hyper-parameter configurations (grid search, with or without validation set, etc.)

W2. Experiments with larger backbones, like ViT-Large

• For fair comparison with prior works, in this paper we evaluated on multiple CLIP encoders (ResNet50, ResNet101, ViT-B/16, ViT-B/32) also considered in previous works.
• We agree that it would be interesting to extend to larger backbones and we will consider it for future versions of this work.

W3. Ablation on different shot numbers from 1-shot to N-shots

• We consider the few-shot evaluation protocol from previous works with {1,2,4,8,16}-shot settings.
• For all experiments and ablations on few-shot learning we consider all shot numbers, average over 11 datasets with 10 different support sets for each (see Tab. 1-2, Tab. 6-9, Fig. 2-3)
• If the reviewer refers to the 4-shot protocol for domain generalization and cross-dataset generalization, we follow the protocol proposed in ProGrad for fair comparison.
• Did the reviewer had other few-shot setting in mind?

W4. Limitation on tasks with longer texts

• We believe that in general CLIP is limited to short text sequences, as addressed by Zhang et al. (2024), and that all methods fine-tuning CLIP are likely to face such limitations whether using real text tokens or learnable ones.

Q1. Could ProLIP be combined with prompt tuning method?

• Thank you for the nice idea. Yes they can be combined. We will add this experiment in future work.

Q2. How does ProLIP perform on datasets not represented in CLIP’s pretraining (e.g., medical or aerial imagery)?

• Thank you for the question. Indeed we are interested in such datasets where fine-tuning would be necessary to steer CLIP features towards the downstream task.
• Among the 11 datasets we study for few-shot classification there are the specific datasets EuroSAT (aerial imagery) and FGVCAircraft (fine-grained aircraft models) that are likely less represented in CLIP's pretraining and that we highlight in the text.
• In the main paper we report scores aggregated over the 11 datasets, but we provide detailed results in Table 6-7. For example, ProLIP is top-performer on EuroSAT.
• Furthermore, in cross-dataset generalization for EuroSAT (Table 4), ProLIP is again top-performer.

Q3. Comparison of training and computation time

• Thanks for the suggestion. We will include this in future work.

References B. Zhang et al., Long-CLIP: Unlocking the Long-Text Capability of CLIP, ECCV 2024