Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models

We thank reviewer zJm8 for the positive comments. Below, we address the concerns raised in your review.

Q1: ... but I am not sure if the results shown in Table 1 and Figure 7 (a) ...

Specifically, from Table 1 and Figure 7(a), we find that applying CT in isolation (P-Prompt) achieves a better score than the baseline CoCoOp in most cases, which shows the efficiency of the CT module. Besides, our PBPrompt outperforms other variants when combined with CT and Bayesian frameworks.

Q2: comaprison with ProDA and VPT (Derakshhani et al. 2022)

H score of CoCoOp, ProDA, PBPrompt, and CoOp+VPT on Base-to-New task

Thank you for your valuable suggestion. We do not report the results of ProDA and VPT because of their unreleased codes. Fortunately, we find some results from previous works [1,2], and we report the comparison above (detailed results can be found in Table C.7 and Table C.9 in the Appendix). From the results, we find that our proposed PBPrompt outperforms ProDA and VPT by achieving significant improvements in most cases, which demonstrates the superiority of our approach.

Q3: For the new class

We model the posterior as $q(\mathbf{r_c}|c) = \mathcal{N}(u(\mathbf{e_c}), \sigma(\mathbf{e_c}))$, where $\mathbf{e_c}$ denotes the class embedding of c-th class (the bert embedding is used in our experiment). Thus, we can infer the distribution based on the embedding of the new class, which is easy to obtain in practice. We will add this in our revision.

We thank reviewer EsJ6 for the positive comments. Below, we address the concerns raised in your review.

Q1: Is it possible that "stochastic" method works just because it provides more input variance in the training, therefore reduces the overfitting

For a given class, previous deterministic methods represent the class as a point in the latent embedding space (the output of a powerful single prompt remains a point). This may fail to cover diverse visual attributes of the given class. We in this paper propose stochastic prompt generation under the Bayesian framework, where a class is modeled as a distribution in the embedding space. The stochastic methods can be viewed as a Bayesian extension of the deterministic methods by introduce the variance (uncertainty), which showing greater potential in modeling complex and structural data. Due to the introduced distributed representation, the stochastic methods capture diverse vision concepts of the given class, resulting in comprehensive label representation.

Q2: ...Are they any randomness in inference? How about the variances?

Thank you for your careful reading! One of the main ideas of Fig.7(b) is to show the diverse prompts, and it is reasonable that not all the randomly generated prompt correlates to the class label well. After trained on a training dataset, the learned posterior of the given class captures diverse visual semantics. Some sampled prompts may focus on the surrounding environment features and thus have probability to attend to the background.

Like previous Bayesian models, in the test stage, we use the mean of the posterior to generate the sampled prompt, which reduces the variance and results in robust performance.

Q3: Ablations on ImageNet

Table 1 The results of various variants at the few-shot task on ImageNet

Table 2 The ablation results of PBPrompt on ImageNet with more 50 training epochs.

Note that the values in brackets denote the difference from the original results in Table C.7 in the manuscript.

Following your advice, we have reported the ablation results on ImageNet above (Due to the limited time, we only report the ablation results that are newly conducted in the rebuttal period and more details about these ablation study can be found in Appendix C.10 . We will add the results of Monte Carlo sampling numbers and coefficient $\eta$ if necessary). We find that our approach shows robustness and effectiveness on the ImageNet dataset.

We thank the reviewer again for valuable suggestions, which helped us improve the quality of the submission.

We thank reviewer H1KY for the comments and suggestions. Below, we address the concerns raised in your review. We hope our efforts can help you improve your rating of the paper.

Q1: Novelty of this paper.

We first thank the reviewer for the suggested paper. It is nice to see that several studies are proposed to introduce uncertainty into prompt tuning, which is a critical challenge for robust prompt searching. We summarize the main difference between SHIP and ours below:

(1) modeling of $\mathbf{z}$. Both SHIP and PBPrompt introduce uncertainty into the prompt generation process. However, the latent variable $\mathbf{z}$ ($\mathbf{r}$ in PBPrompt) models different levels of uncertainty and comes from different assumptions. SHIP introduces the stochastic prompts into each image, and infers a sample-dependent posterior:

$

q(\mathbf{z_i}) = \mathcal{N}(u(\mathbf{x_i}), \Sigma(\mathbf{x_i})),

$

where $\mathbf{x_i}$ denotes the feature of $i$-th image. While PBPrompt views each category as an underlying distribution and infers a label-specific posterior:

$

q(\mathbf{z_c}) = \mathcal{N}(u(\mathbf{e_c}), \Sigma(\mathbf{e_c})),

$

where $\mathbf{e_c}$ denote the embedding of $c$-th category.

(2) Prior on $p(\mathbf{z})$. SHIP simply adopts the standard Gaussian as the prior of $\mathbf{z}$, e.g., $p(\mathbf{z})=\mathcal{N}(0, \mathbf{I})$, while PBPrompt utilizes the contextual prior to capture label-specific features: $p(\mathbf{z_c})=\mathcal{N}(\mathbf{e_c}, \mathbf{I})$. This difference enables PBPrompt to access additional label semantics, achieving better prior guidance.

(3) Training pipelines. SHIP introduces an additional feature reconstruction loss to pre-train the VAE, and then finetunes the prompt via the task-specific loss. Our PBPrompt naturally integrates the stochastic prompts into the CLIP framework and directly optimizes the prompt via the combined ELBO.

As discussed above, we want to note that SHIP and our PBPrompt are quite different in terms of generation of $\mathbf{z}$, priors on $\mathbf{z}$, and the training pipelines. We will add these discussions in our revision.

Please note that it is not trivial to apply the CT directly to the prompt tuning. Unlike previous prompt tuning methods that often represent the image and label with the global features (e.g., the <CLS> and <EOS> embeddings), we here view the image patches and prompt embeddings as two discrete distributions $P$ and $Q$. This makes it natural to apply the CT to regularize the alignments between vision-language domains.

Q2: Missed baselines and weak performance

Table.1 Additional New-to-New results. The results of CoOP +VPT and CoOp + SHIP are copied from SHIP paper

Table 2 Additional Cross-dataset results. The results of CoOP +VPT and CoOp + SHIP are copied from SHIP paper

Following your advice, we have reported the additional results of SHIP above. To have a fair comparison, we report the results of CoOp + SHIP on base-to-new task and cross-dataset transfer learning task in Table 1 and Table 2 respectively. We find that our approach outperforms SHIP in most cases, which indicates the effectiveness of our label-specific Bayesian generation.

As for the suggested PromptSGC and MaPLe, we find that they belong to multimodal prompt tuning and aim to optimize both the vision and text prompts, which is beyond the scope of this paper. It is unfair to compare the proposed textual prompt tuning PBPrompt with such multimodal methods.

Q8: Comparison with Multimodal prompt tuning ...

We thank the reviewer for the valuable suggestion. theoretically, our proposed model can be modified into the multimodal prompt tuning task, and we will leave it as a future work.

Q9: Results of Table.1 and Figure.7(a)

Thank you for your careful reading again.

We apologize for the confusion caused by my mistakenly switching these two lines of results (B-Prompt and P-Prompt) in Tabel 1. Now this mistake has been corrected and you can check it in reversion. The results do indicate that the proposed Bayesian approach is more capable of introducing uncertainty thereby enhancing generalization performance and reducing overfitting which is consistent with the results reported in the Table. 1 and the Figure 7(a).

Q10: when we increase the number of training epochs to 50 or even 200 in base-to-new experiments, does it affect the generalization performance

Effect of PBPrompt with more 50 training epochs on base-to-new generalization

$(\cdot)$ denoted the difference from the original results in Table C. 7. $\Delta$: The improvements of harmonic mean compared to CoCoOp (without additional training epochs).

I appreciate your constructive suggestions. Indeed, there is a trade-off between performance on base and new classes according to the number of training epochs. Specifically, more training epochs lead to better accuracy on base classes and lower it on new classes. Our proposed model shows better generalizability by achieving higher performance (The H score) on the base-to-new task and more details can be found in Sec C.9. Like previous works, it will affect the results when the training epochs increase.

We thank reviewer 19zzfor the comments and suggestions. Below, we address the concerns raised in your review. Please let us know if you have any further concerns or whether this adequately addresses all the issues that you raised with the paper.

Q1 & Q2: Comparison with PLOT

First, we want to note that it is unfair to compare our model with PLOT-pp (https://github.com/CHENGY12/PLOT/tree/main/plot-pp). PLOT-pp is a multimodal prompt tuning algorithm, where both the vision and text prompts are tuned to improve the performance. However, this paper aims to address the uncertainty issue in text prompt tuning, where the vision embedding is fixed according to previous works.

Based on the above facts, and to be consistent and fair, we modify the official PLOT by only replacing the RN50 with ViT-B/16 and optimizing the textual prompts under the same training pipeline (this ensures that the PLOT results reproduced in this paper are credible).

From all results compared with PLOT using ViT-B/16, we find that PBPrompt has a significant improvement over PLOT on almost all datasets.

Moreover, we find that PLOT is sensitive to the backbones, and it shows inconsistent performance on RN50 and ViT-B/16 (The authors of PLOT explain that the training of OT distance needs discriminative local visual features, which may not be the case in ViT-based encoders. Please refer to https://github.com/CHENGY12/PLOT/issues/1 for more details). In contrast, our model introduces the uncertainty under the Bayesian framework, which shows strong robustness with different backbones (as discussed in Sec4.3 in the manuscript).

Q3: ..., the domain generalization experiments using RN50 are missing

Thank you for your constructive suggestions. We have added the domain generalization experiments using RN50 above. We find that PBPrompt using RN50 outperforms PLOT by achieving the 3/4 best results. This demonstrates the superiority of the proposed method.

We have added the comparison and detailed discussion in Sec. C.7 in the new revision.

Abalation studies on OT ( $\text{PBPrompt}\_\text{OT}$) and SPG ($\text{PBPrompt}\_\text{w/o-S}$)

Q4 & Q5: ..., what is the main contribution of using CT instead of OT

First, we thank the reviewer for the appreciation of our proposed stochastic prompts generation (SPG), which is one of the main contributions of this paper. Both OT and CT have the ability to measure the semantic distance between two sets, and we here choose the CT as the regularization mainly because:

1. Mathematically, CT measures the distance bidirectionally, which shows a more holistic alignment:

$

\mathcal{L} = \frac{1}{M}\sum_{m=1}^M \sum_{c=1}^C C(\mathbf{u_m}, \mathbf{g_c}) \frac{p_c exp(\mathbf{u_m}^T \mathbf{g_c})}{\sum_{c'=1}^C p_{c'}exp(\mathbf{u_m}^T \mathbf{g_{c'}})} + 
\sum_{c=1}^C p_c \sum_{m=1}^M C(\mathbf{g_c}, \mathbf{u_m})\frac{exp(\mathbf{g_c}^T\mathbf{u_m})}{\sum_{m'=1}^M exp(\mathbf{g_c}^T \mathbf{u_{m'}})},

$

where the first patch-to-prompt term calculates the transport cost from the image patches to prompt embeddings. This forces the expected prompt to have the closest semantics with patch embeddings so that it can receive a higher transport probability than other labels. The second prompt-to-patch term calculates the cost from an opposite view, which makes the prompt act as a weighted cluster that shares similar semantics with $M$ visual patches. These two terms guide the learning of informative and distinguishing prompts, resulting in better representation.

2. Unlike OT which typically requires an inner iteration to search the estimated transport plan, CT can be directly calculated according to the cost matrix and transport probabilities. This makes CT more flexible to jointly optimize with the task-specific loss in an end-to-end manner, often showing higher performance and stronger generalizability.

Following your suggestion. We have added the ablation study on the proposed conditional transport (CT) and optimal transport (OT) with five datasets above. We find that CT outperforms OT in most cases. These results demonstrate the efficiency of the introduced CT module.

Q6: The number of Moter Carlo sampling

It is worth noting that the Moter Carlo sampling number is different from the number of PLOT prompts. More specifically, C, the number of PLOT prompts, denotes C sets of learnable prompt vectors. And N, the Moter Carlo sampling number in our method, denotes that we sample N prompts from one set of learnable prompts. Thanks to the SPG module, our model infers the posterior of the prompt based on one set of learnable prompts. This makes it possible to sample 20 prompts without introducing 20 learnable prompt vectors. Therefore, it is not reasonable to set the two parameters to be the same.

Q7: ... the proposed approach uses much more parmas ...

The additional parameters mainly come from the SPG, which is the main component of the proposed model and aims to generate stochastic prompts given the class name. To evaluate the efficiency of the SPG, we here build a variant (named as $\text{PBPrompt}\_{\text{w/o-S}}$) which uses the inferred mean of the Gaussian $\mathbb{\mu}$ as the sampled $r_c$ in Eq.(2) in the manuscript, and report the results above. We find that PBPrompt outperforms its $\text{PBPrompt}\_{\text{w/o-S}}$ with a significant gap at all datasets, which shows that the improvement comes from the introduced model rather than the additional parameters.