Cross-Model Semi-Supervised Prompt Learning for Vision-Language Models

We thank Reviewer xki8 for finding the paper well-written and easy to follow. Below are our responses to the specific concerns.

If Figure 1(a,b) are averaged over several random seeds: Thanks for the query. All the results given in the paper including the ones shown in Figure 1(a,b) are taken as average over $3$ different random seeds.

Comparison with zero-shot CLIP and other prompt engineering methods: . Being one of the first works in multi-modal semi-supervised prompt learning, we carefully designed the baselines for a comprehensive assessment. The Text Prompt Learning (TPL) and Visual Prompt Learning (VPL) baselines exactly follow the CoOp [1] approach, one with only text prompts as in the CoOp paper and the other with only visual prompts. As we found that XPL surpasses these baselines (refer in Figures 3 and 4 of the main paper) by a huge margin, we withheld from adding the zero-shot CLIP values which has been shown to be greatly inferior to CoOp method itself [1]. However, in view of the reviewers comment, we compared the lowest value for XPL obtained in 1-shot (refer Figure 4b in main paper) with the 0-shot CLIP results. As observed in the table below, XPL even for 1-shot surpasses the 0-shot CLIP values by a huge margin.

[1] Conditional prompt learning for vision-language models, CVPR 2022

We thank Reviewer 18u3 for finding our framework simple but effective and the training policy interesting. Below are our responses to the specific concerns.

Applicability of our methods on recent works on prompt learning: As ours is the first work in semisupervised prompt learning in VLMs, we designed many possible SSL baselines for learning prompts by leveraging on unlabeled data. We put forth TPL$^u$, VPL$^u$ and MPL$^u$ baselines by carefully culling SSL literature from related fields and showcase the efficacy of learning rich prompts using our framework by comparing with these over 15 datasets [refer Figures $3$ and $4$ of main paper]. For XPL, the main strength lies on efficient transfer learning in VLMs by exploiting the hugely available free knowledge (unlabeled data) available in the wild. However, for the sake of completeness and to address the reviewers concerns regarding the applicability of XPL, we evaluate the generalizability of our approach from base to new classes (Table 2 of main paper and Table 4 in supplementary section) and compare the results with more recent approach CoCoOP [1]. The following tables showcase the supremacy of XPL on the individual datasets.

Analysis for why the different prompt lengths is better. While prompt learning enables efficient and faster adaptation paradigm, their low capacity may not allow a single prompt learning model to achieve best performances in all cases. As shown in the Figures 1(a) and (b) of the main paper, two models varying in the number of learnable prompts (8 and 16 prompts) exhibit diverse category-wise performance. Some classes in these datasets are better suited to 16 prompts while some show improvement with 8 prompts. Our co-teaching approach exploits these multiple prompt learners to learn complimentary knowledge and thus can complement in providing better semi-supervision to each other. Further, we identify that directly using the same adaptation model to produce confident pseudo-labels for the unlabeled data may miss crucial information for certain categories. As shown in notable works of [2, 3], the discriminative power of a single model is weaker to assign large number of high-quality pseudo labels to the unlabeled data. Similar conjuncture can be observed by using a fix length of prompts as shown in Figure 7 of the paper. Instead, the idea is to construct multiple pseudo-labels of different versions of the same unlabeled data and allow them to complement each other. In our work, we extend this idea further in a multiple prompt learning paradigm. In addition to using different views of the unlabeled samples, the two pathways have different lengths of learnable prompts. Such a co-teaching framework enables better representation learning by not only forcing invariance to different views of the unlabeled data but also enforcing invariance towards prompts of different lengths. To the best of our knowledge, such dual invariance applied to semi-supervised VLMs has not been explored earlier. We have also showcased the effectiveness of our cross-model co-teaching design over several semi-supervised baselines in the main paper and performed another additional baseline in accordance with the different queries of the reviewers.

Clarification for the utilization of unlabeled data in TPL$^u$, MPL$^u$: All the $3$ baselines of TPL$^u$, VPL$^u$ and MPL$^u$ are based on naive Fixmatch [4] method as the underlying semi-supervised approach. The training on the unlabeled samples is carried out by generating pseudo-labels from the weakly augmented version and using it to reduce the loss across its strongly-augmented counterpart following the Fixmatch philosophy.

We thank Reviewer MsJj for finding our work an interesting and timely topic. Below are our responses to the specific concerns.

Novelty and Advantage of Co-teaching using prompts of different lengths: Thanks for the query. To the best of our knowledge, there is no existing Semi-Supervised Learning (SSL) works either in prompt learning or its applications in the multi-modal setting of VLMs. The primary challenges faced are of two folds. 1) The low quality of prompts learned in presence of only a few labeled data with a vast set of unlabeled samples and 2) Exploiting both the text and visual modalities using prompts only, to extract rich representations from unlabeled samples. As mentioned in Section 1 of the main paper, although approaches like [1,2] employ a cross-model representation learning, such paradigms have neither been exploring to tuning prompts nor in the multi-modal context of VLMs. These incur different challenges for learning efficient prompts in a multi-modal setting using frozen VLMs. The main motivation of using a co-training policy is to harness the complementary knowledge across two models having prompt inputs with different lengths. While prompt learning enables efficient and faster adaptation paradigm, their low capacity may not allow a single prompt learning model to achieve best performances in all cases. As shown in the Figures 1(a) and (b) of the main paper, two models varying in the number of learnable prompts (8 and 16 prompts) exhibit diverse category-wise performance. Some classes in these datasets are better suited to 16 prompts while some show improvement with 8 prompts. Our co-teaching approach exploits these multiple prompt learners to learn complimentary knowledge and thus can complement in providing better semi-supervision to each other. Further, we identify that directly using the same adaptation model to produce confident pseudo-labels for the unlabeled data may miss crucial information for certain categories. To the best of our knowledge, such dual invariance applied to semi-supervised VLMs has not been explored earlier. We have also showcased the effectiveness of our cross-model co-teaching design over several semi-supervised baselines in the main paper and performed another additional baseline in accordance with the reviewer's subsequent question.

We thank Reviewer f2tm for acknowledging that our approach could substantially reduce the need for large annotated datasets. Below are our responses to the specific concerns.

Novelty and difference from existing SSL applications in prompt learning: Thanks for the query. To the best of our knowledge there is no existing Semi-Supervised Learning (SSL) work either in prompt learning or its applications in the multi-modal setting of VLMs. The primary challenges faced are of two folds. 1) The low quality of prompts learned in presence of only a few labeled data with a vast set of unlabeled samples and 2) Exploiting both the text and visual modalities using prompts only, to extract rich representations from unlabeled samples. As shown in Figures 1a and b of the main paper, two models leveraging unlabeled data but with different number of learnable prompts exhibit markedly different category-wise performance. Typically (as shown by [$1$,$2$]) in low-labeled data regime, the learned representations tend to lack enough discriminative power for downstream tasks thereby also failing to generate high-quality pseudo-labels. In such a constrained scenario, a combination of multiple models can be of rescue. However, such multiple model approach has not been applied for learning prompts in a semi-supervised setting for large VLMs. Our XPL uniquely combines a cross-model approach to employ a co-training policy for learning quality prompts. This novel prompt learning setup for a large frozen VLM harnesses the complementary knowledge across two models having prompt inputs with different lengths. In addition to using different views of the unlabeled images (vision), the two pathways have different lengths of learnable prompts (language). Such a co-teaching framework enables better representation learning by not only forcing invariance to different views of the unlabeled data but also enforcing invariance towards prompts of different lengths. In view of the different existing conventional SSL approaches, our carefully designed baselines do employ the various standard SSL techniques for prompt learning in VLM. These are illustrated in the following paragraph.

As ours is the first work in semisupervised prompt learning in VLMs, we designed many possible SSL baselines (namely TPL$^u$, VPL$^u$ and MPL$^u$) by carefully culling SSL literature from related fields and showcase the efficacy of our framework by comparing with these over 15 datasets [refer Figures $3$ and $4$ of main paper]. These $3$ baselines are based on Fixmatch [3] as the underlying semi-supervised approach that was used for semisupervised classification of images. As mentioned in Section 4.1, TPL$^u$, VPL$^u$ and MPL$^u$ make use of text prompts, visual prompts and both text and visual prompts respectively. The training on the unlabeled samples is carried out by generating pseudo-labels from the weakly augmented version and using it to reduce the loss across its strongly-augmented counterpart following the Fixmatch philosophy and not using multiple models. As can be observed from all the plots in Figures $3$ and $4$ of the main paper, XPL (using multiple models in addition) outperforms all these baselines showing the effectiveness of the cross-model design. In Figure $4(b)$, XPL provides $2.1$% improvement on average even in 1-shot scenario over MPL$^u$(the strongest baseline). Moreover, XPL offers a significant jump of $5$% for the fine-grained DeepWeeds dataset in 1-shot setup. Further, we have also demonstrated the effectiveness of our cross-model XPL over other traditional semi and self supervised approaches - Pseudo-labeling [4], which does not employ co-teaching and also over MoCo [5] which employs momentum encoder(refer Figure $6$ of the main paper). As observed, our XPL outperforms both PL and the self-supervised MoCo baselines for all considered datasets across al the scenarios.