Consistency-guided Prompt Learning for Vision-Language Models

Part 1

The idea of adapters and prompt-tuning already exists in the literature. Merely combining the two ideas seems an incremental work and not novel.

While adapters and prompt-tuning already exist in the literature (which we also acknowledged in the paper), combining them poses a considerable challenge that has not yet been solved by any prior works. This is due to the fact that the model can easily overfit the additional learnable parameters, thus losing generalizability. This phenomenon is has been reported by prior works. For example, MaPLe reported the optimal performance by adding prompts to 9 out of 12 layers of the encoder and not all 12. We highlighted the impact of the consistency constraint in our ablation study in Table 4. In the table below, we provide additional experiments emphasizing the difficulty of combining adapters with prompts for existing methods without our proposed consistency constraint.

As observed in this table, CoPrompt's performance of 80.48% drops to 78.45% when prompts and adapters are trained without the proposed consistency constraint. Similarly, the optimal performance of MaPLe is 78.55% , which drops to 77.61% when we increase trainable parameters in the form of prompts and adapters. This study signifies the fact that the adapters and prompts can not be combined to improve the overall accuracy without the consistency constraint proposed in our method. We have now added this discussion to Appendix A.4 (Page 14).

The idea of retaining the generalizability of the CLIP using consistency loss has already been explored in the paper [1]. Hence, the consistency loss doesn't contribute towards the novelty. The authors have not compared their approach to the above paper and there is also no reference to the paper.

Kindly note that we were not aware of this work at the time of submission since this paper was accepted in ICCV 2023, which was around the same time that we were preparing for our submission to ICLR. It is worth noting that our paper had been publicly available on arXiv before PromptSRC. Our paper was posted on arXiv on Thu, 1 Jun 2023, while PromptSRC was posted on Thu, 13 Jul 2023. However, while both PromptSRC and our proposed CoPrompt share a common goal of ensuring consistency, the proposed solutions in the two methods differ significantly. These distinctions are as follows:

• While prompts are the only training parameters in PromptSRC, we introduced the concept of integrating adapters with prompts. It provides additional tunable parameters for adopting CLIP to a new domain, and training under the influence of consistency helps the model to improve generalization without over-fitting the few labelled samples.

• On the language branch, PromptSRC uses a pool of text templates to increase the number of text prompts, which are all hand-designed prompts. In contrast, our proposed solution uses an LLM to generate a more descriptive sentence. From the ablation study on PromptSRC and CoPrompt, we see a larger improvement with our approach.

• For the consistency constraint, PromptSRC used L1 loss, whereas our CoPrompt utilizes a cosine loss. As opposed to L1 loss, cosine loss captures the angular similarity between vectors rather than relying solely on their magnitudes. Nonetheless, we already explored L1, and also MSE as the consistency loss, which was presented in Table 5(b). The results showed comparatively better performance for cosine loss over L1 or MSE.

• PromptSRC utilized independent prompts on language and vision branches, while CoPrompt employs a multi-modal prompt with a coupling function between language and vision branches.

• We now directly compare our results to PromptSRC in the following tables. On the base-to-novel generalization task, our method performs better than PromptSRC in the evaluation of 11 datasets, with an average HM of 80.48 vs. PromptSRC's 79.97. Similarly, on the cross-dataset evaluation, CoPrompt outperforms PromptSRC by 1.19%.

Part 2

We now discuss the similarity of our method with PromptSRC in the Method section (Page 4, Paragraph 3), and also expand the results in Tables 1, 2 and 3 (Pages 6 and 7) to compare with PromptSRC.

The improvements in the Domain generalization is marginal given that authors have fine-tuned the model. Same is true for cross-dataset evaluation.

While the average improvement in cross-dataset evaluation and domain adaptation is relatively lower compared to the improvements in few-shot and zero-shot performance, certain datasets exhibit notable progress. For example, EuroSAT demonstrated a notable improvement of 3.84% in cross-dataset evaluation. Kindly note that the improvements of existing methods in these two evaluations were on a comparable scale. To illustrate, MaPLe demonstrated enhancements of 0.56% and 0.15% for cross-dataset evaluation and domain generalization, respectively. Similarly, PromptSRC (accepted in ICCV'23) exhibited a 0.38% improvement in domain generalization, but a 0.49% drop in cross-dataset evaluation.

Are the vision side prompts conditioned on text side? Do authors follow MaPLe settings or Independent VL prompting?

For the implementation of the prompts, we followed the same approach as used in MaPLe.

Dear authors, thank you for addressing my comments

• I am satisfied with the explanation of using prompts and adapters jointly. Authors have done a nice job.
• Also, the differences between PromptSRC and author's work seem acceptable.

However, I still feel the work lacks content in terms of novelty. Based on the responses, I have increased my rating accordingly.

Thank you!

Regarding the method as prompt learning is a bit ambiguous in my opinion.

Prompts are the main learnable parameters in our method, and the goal of our solution is to train the learnable prompt while ensuring effective adaptation to the target domain with few labelled samples without losing generalization. This is why we refer to it as a prompt learning method. Kindly note that existing methods in the literature (e.g. KgCoOp and ProGrad) that share similar conceptual frameworks are also considered prompt learning methods.

The comparison between CoPrompt and Zero-shot CLIP is not fair enough. The diverse text prompts generated by LLM can improve the zero-shot classification ability of CLIP on downstream tasks.

To fully investigate the answer to this question, we show the results for the following two scenarios: (1) removing the LLM-generated prompts from CoPrompt and (2) adding LLM-generated prompts to CLIP for its zero-shot evaluation.

The result of the first experiment is presented in Table 5(c) (first row), where CoPormpt obtains an accuracy of 80.09%. While this is a 0.39% drop from the optimal performance of CoPrompt, it is 8.99% better than the zero-shot performance of CLIP (80.09% compared to 71.1%). We show the result of the second experiment in the Table below. The result from this experiment shows that LLM-generated text prompts do not improve the zero-shot classification ability of CLIP on downstream tasks, but rather drop the performance from 71.70% to 69.14%. This can be explained by the fact that using a more complex sentence instead of the simple template decreases the likelihood of the sentence embedding being aligned with the corresponding image, potentially leading to inaccuracies in CLIP's predictions. We have added this discussion to Appendix A.5 (Page 14).

Can CoPrompt reaches better result on Eurosat if λ=0?

Using λ=0 will effectively turn off the consistency constraint, which, according to the ablation study, leads to sub-optimal performance. In the table below (which is an expansion of Table 6), we show the results for all datasets with λ=0 and another smaller value of λ=0.01, which does not improve the performance of any dataset further. Accordingly, we have expanded Table 6 (Page 8) and added these results.

What if combined consistency constrain learning with other existing methods.

Although we did not originally design CoPrompt as an add-on to other prompt tuning methods, the consistency constraint can be included in any method with a similar tuning mechanism. While CoCoOp constitutes a distinct prompt-tuning approach featuring a text branch conditioned on tokens generated by the image branch, the current form of the consistency constraint cannot be applied to it. However, it is feasible to integrate the consistency constraint into the CoOp method. We show the result of this experiment in the following table. Adding the constancy constraint to CoOp results in a considerable improvement in accuracy for novel classes (from 63.22 to 64.87), contributing to an overall increase in the harmonic mean (from 71.66 to 72.75).

We have added this discussion to Appendix A.6 (page 14).

What is the training overhead in terms of time? What is training and test-time inference speed compared with prior methods?

CoPrompt requires about 25% more training time than the previous SOTA MaPLe. For example, one epoch of training on the Flower102 dataset takes 2 minutes and 9 seconds compared to 1 minute and 43 seconds for MaPLe on a single GPU. There is no overhead in the inference phase since the consistency constraint, the main component that adds the overhead, only affects the training process. We have updated the discussion on the parameters and computational complexity in Section 4.6 (Page 9) with the abovementioned information.

Dear authors,

Thank you for your reply. The majority of my concerns have been addressed. Taking into consideration the feedback from other reviewers and the author's rebuttal, I decide to raise my rating.

The paper's self-consistency terms seem similar to those in self-supervised learning (SSL) methods.

Our model does indeed use the principles of SSL for applying consistency constraint on the vision-language framework. While in general SSL frameworks have two learnable encoders which are both trained, in our method, one encoder is frozen while the other is learnable. This method effectively enables knowledge distillation from the frozen encoders to the learnable ones, thus maintaining the generalization strength of the pre-trained base model while tackling a new task in a few-shot scenario. In some sense, our approach is somewhat between SSL and distillation. We have included this discussion in the Introduction (Page 2, Paragraph 2).

Missing recent relevant studies.

Thank you for pointing out two related works which we were not aware of before. We now provide a discussion on these two methods and compare the results as follows:

Bayesian Prompt Learning [1] is a prompt learning method that approaches the task from a Bayesian perspective, formulating it as a variational inference problem. ProDA [2] proposed a data-driven approach, that learns the soft prompts from a few downstream samples, discovering the task-related content with less bias than manual design. Our approach is quite different from these methods since we build over a multi-modal prompt tuning technique and propose three components aiming to improve both few-shot and zero-shot performance.

Following, we show the comparison of our method with Bayesian prompt learning. We can't compare the results with ProDA since it adopted a different training protocol (only evaluated on few-shot learning) than most existing methods, including ours.

As we find from the comparison of base-to-novel generalization, CoPropmt outperforms Bayesian Prompt by 3.05% in harmonic mean over 11 datasets. It performs better than Bayesian Prompt on both base and novel categories. On the cross-dataset evaluation, CoPrompt outperforms Bayesian Prompt by 1.05% on average over all datasets. Finally, on domain generalization, CoPrompt performs only 0.02% lower than Bayesian Prompt on the average over all datasets, but looking into individual datasets, we find that CoPrompt outperforms Bayesian Prompt on all datasets except ImageNetA. We now discuss Bayesian Prompt and ProDA in our Related Work section (Page 3, Paragraph 2), and also expand the results in Tables 2 and 3 (Pages 6 and 7) to compare with Bayesian Prompt. We could not include Bayesian Prompt in Table 1 since it did not report the base, novel, and HM results like other methods.

Part 2

The diagrams in the paper are of very poor quality.

As per your suggestion, we improved the figures to make them visually more appealing. Also, we have now visualized the adapters in Figure 1 (Page 1). However, we removed Figure 2 due to its overall quality and redundancy in presenting information already available in Table 1, and to make room for the revisions based on the reviews. We would appreciate it if the reviewer could kindly point out any further changes that they deem necessary, and we will happily make those additional changes.

I think there are some writing logical errors in the paper. For example, in the Adapters heading in section 3, adapter based method are being mentioned but prompts have been written instead of the adapter blocks.

We checked the writing of the mentioned section, and are glad to inform you that there are no writing errors. However, we have revised the writing to resolve this confusion. The sentence mentioning 'prompt' referred to an existing work (Khattak et al., 2022) that discussed the fact the additional parameters (whether prompt or adapter) introduces the risk of overfitting. We have now revised the description on Page 5, Paragraph 1 to say: 'For instance, Maple achieved the best performance when adding learnable parameters to only nine out of twelve layers of the CLIP backbone.'

Part 1

In Figure 3, no coupling functions are visible. Visual prompts (orange color) are not shown in intermediate layers of CLIP visual encoder.

Thank you for pointing this out. We have now visualized the coupling function and fixed the colour in the visual branch, in Figure 2 (Page 5).

It will be good to see the proposed method generalization for a newer V-L model. For example, on EVA-CLIP [1] model.

In the original submission, we only focused on CLIP since all existing methods we compared had reported their results on the same encoder. As per your suggestion, we have now investigated the performance of the proposed method, as well as the previous SOTA MaPLe, with the EVA-CLIP backbone. The results are presented below:

This table demonstrates that CoPrompt also works well on the EVA-CLIP backbone and shows similar improvements over MaPLe. Overall, the EVA-CLIP encoder performs slightly better than the CLIP encoder. We have added this discussion to Appendix A.3 (page 14).

There is a recent prompt learning method PromptSRC [2]. How is the proposed method different from this work?

Kindly note that we were not aware of this work at the time of submission since this paper was accepted in ICCV 2023, which was around the same time that we were preparing for our submission to ICLR. It is worth noting that our paper had been publicly available on arXiv before PromptSRC. Our paper was posted on arXiv on Thu, 1 Jun 2023, while PromptSRC was posted on Thu, 13 Jul 2023. However, while both PromptSRC and our proposed CoPrompt share a common goal of ensuring consistency, the proposed solutions in the two methods differ significantly. These distinctions are as follows:

• While prompts are the only training parameters in PromptSRC, we introduced the concept of integrating adapters with prompts. It provides additional tunable parameters for adopting CLIP to a new domain, and training under the influence of consistency helps the model to improve generalization without over-fitting the few labelled samples.

• On the language branch, PromptSRC uses a pool of text templates to increase the number of text prompts, which are all hand-designed prompts. In contrast, our proposed solution uses an LLM to generate a more descriptive sentence. From the ablation study on PromptSRC and CoPrompt, we see a larger improvement with our approach.

• For the consistency constraint, PromptSRC used L1 loss, whereas our CoPrompt utilizes a cosine loss. As opposed to L1 loss, cosine loss captures the angular similarity between vectors rather than relying solely on their magnitudes. Nonetheless, we already explored L1, and also MSE as the consistency loss, which was presented in Table 5(b). The results showed comparatively better performance for cosine loss over L1 or MSE.

• PromptSRC utilized independent prompts on language and vision branches, while CoPrompt employs a multi-modal prompt with a coupling function between language and vision branches.

• We now directly compare our results to PromptSRC in the following tables. On the base-to-novel generalization task, our method performs better than PromptSRC in the evaluation of 11 datasets, with an average HM of 80.48 vs. PromptSRC's 79.97. Similarly, on the cross-dataset evaluation, CoPrompt outperforms PromptSRC by 1.19%.

We now discuss the similarity of our method with PromptSRC in the Method section (Page 4, Paragraph 3), and also expand the results in Tables 1, 2 and 3 (Pages 6 and 7) to compare with PromptSRC.