Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models

Q1: Lack deeper theoretical justification on why surrogate features retain meaningful semantic information.

A1: Thanks for the comments. We provide a theoretical analysis based on the universal approximation theorem. Due to character limits, please refer to the Q&A 1 of Reviewer jTad for detailed analysis.

Q2: No qualitative comparison between surrogate and explicitly learned prompts.

A2: Thanks for the insightful comments. We provide direct visualization map comparisons between surrogate and explicit FG prompts (due to the computation resource limit, we can’t afford to reproduce explicit ID prompt learning). As shown in explicit-versus-surrogate.png, heatmaps generated from explicit and surrogate prompts focus on the same regions of the image, particularly at large FG scales ($Z=3,4$). For small FG scales ($Z=1,2$), while the heatmaps show minor differences, they still consistently focus on the target object. These comparison results indicate the effectiveness of SFG.

Q3: No discussion about the failure cases of SurPL.

A3: We first provide the direct comparison between explicit FG prompt learning and surrogate FG prompt learning on 9 datasets (except ImageNet and SUN397 due to the computational resource limit). The averaged results indicate surrogate features don’t exhibit significant failures compared to explicit features.

We further compare the generalization ability between explicit and surrogate prompts on 10 datasets (excluding ImageNet). The results indicate that surrogate prompts initially exhibit weaker generalization ability. We attribute it to the lightweight architecture of SFG. Generating features from SFG may overfit more easily. However, through our proposed SurPL-G, we effectively address this limitation and achieve remarkable performance.

Finally, we compare SurPL with the state-of-the-art work GalLop. While GalLoP achieves marginally superior results on certain datasets (e.g., ImageNet), we attribute this advantage to its multiple global prompt learning and dropout strategy. This strategy induces diversity through randomization, enhancing the performance. We will further explore how to incorporate such strategy into our SurPL.

Q4: Hyperparameter sensitivity analysis is missing.

A4: We provide hyperparameter analysis at the Q&A 2 of Reviewer jTad. The results indicate that the hyperparameters applied are reasonable and relatively stable.

Q5: Analyze how reducing SFG complexity affects performance.

A5: We appreciate this insightful suggestion. Due to the time limit, we only explore this trade-off by reducing the dimension of $d_{mid}$ in $\theta _ {fc1}$ and $\theta _ {fc2}$ in SFG. The results indicate that accuracy remains relatively stable despite parameter reduction. This inspires us to further reduce the parameter of projection layers $\theta _ {in1}$, $\theta _ {in2}$, $\theta _ {in3}$ and $\theta _ {out}$.

Q6: How does SurPL handle domain shifts compared to explicit diverse prompt learning?

A6: Thanks for the question. Domain shift tightly associates with model’s generalization ability. Due to the computation resource limit, we can’t reproduce explicit prompt learning on cross-domain experiment (based on ImageNet). We alternatively conduct the comparison of explicit and surrogate prompt learning on base-to-novel setting. Please refer to Q&A 3 for detailed analysis.

Q7: How sensitive is SurPL to the choice of conditional signals?

A7: Thanks for the question. The conditional signals are pre-defined according to the existing diverse prompt learning methods. Please refer to Q&A 4 of Reviewer Zno2 for details.

Q8: Can SFG be trained separately from the main model? Consider knowledge distillation techniques to refine surrogate feature generation.

A8: Thanks for the valuable suggestion. At this stage, SFG cannot be trained separately since it is intrinsically a fine-tuning approach and builds upon $w$. We will keep on research to explore the idea of applying pre-trained models and knowledge distillation to replace the cross-attention module for SFG.

Q9: Does SurPL introduce additional latency during inference?

A9: No significant additions compared with baseline DVLP. Please refer to Table.2 for details.

Q10: Effect of $\rho$.

A10: We explore the effect of $\rho$ on cross-dataset setting. The results indicate a seen-unseen trade-off.

Q1: The paper doesn't provide much theoretical justification for why surrogate features effectively replace the original prompted features, relying instead on empirical results.

A1: Thanks for the valuable comments. We provide a theoretical analysis to demonstrate the effectiveness of the surrogate features.

Notation.

Visual instance $x$, basic text prompt $s$, explicit diverse text prompt $s_E$, conditional signal $\alpha \in \mathbb{R}^d$, basic prompted text feature $w=T([s,c])\in \mathbb{R}^d$, surrogate prompted text feature: $h=\theta _ {SFG}(w,\alpha) \in \mathbb{R}^d$, the original explicit prompted text feature $w_E=T([s_E,c])\in \mathbb{R}^d$.

Proof.

To validate the effectiveness of replacing $w_E$ with $h$ in diverse prompt learning, we analyze their feature similarity through $||h-w_E||=||\theta _ {SFG}(w,\alpha)-w_E||$. Here, we mainly focus on providing the proof of surrogate instance-dependent prompt learning, which imposes stricter constraints (in surrogate fine-grained prompt learning, $\alpha$ is also learnable parameters, which directly simplified the condition to $h=\theta _ {SFG,\alpha} (w)$).

$w$, $\alpha$, and $w_E$ are bounded-dimensional representations within the vector space $\mathbb{R}^d$ (outputs of continuous neural network mappings), and their corresponding inputs $s$, $x$ and $s_E$ are related via $s_E=s+\alpha=s+V(x)$ in instance dependent prompt learning (CoCoOp). Hence, there exists a continuous function $g$, such that: $w_E = g(w, \alpha)$. According to the universal approximation theorem, for any continuous function $g(\cdot)$ and any $\epsilon>0$, there exists a neural network with sufficient capacity (here, we utilize the cross-attention module $\theta_{SFG}$, which has been proved as a proper universal approximator$^\text{a}$) such that for all $(w,\alpha)$,

$$|| g(w,\alpha)-\theta_{SFG}(w,\alpha)||<\epsilon.$$

By taking the approximation error $\epsilon \to 0$, we obtain $||h-w_E||\to 0$ at the optimal parameters $\theta_{SFG}^{\star}$, which confirms that $h$ effectively approximates and replaces $w_E$. This analysis validates the theoretical existence of $\theta_{SFG}^{\star}$, while extensive empirical studies in the manuscript further elaborate on its implementation and optimization, and substantiate its practical effectiveness.

a. Are Transformers universal approximators of sequence-to-sequence functions? ICLR 2020.

Q2: Analysis of performance sensitivity to hyperparameter choices, particularly for different loss components, the number of fine-grained text features and multi-scale constant factor.

A2: Thanks for the valuable suggestion. Here we carefully analyze the effect of different hyperparameters. Due to the character limit, we only report the averaged results on 11 datasets.

Different loss components ($\lambda_1$ and $\lambda_2$): The results are quite stable under different loss coefficient hyperparameter settings. Since $\lambda_1=25$ and $\lambda_2=10$ achieve the best averaged performance, we choose them as the default setting.

The number of fine-grained text features Z: Comparing the results between $Z=0$ (No FG prompts applied) and other settings validates involving FG information can significantly boost the performance. We observe consistent performance gains when increasing $Z$ from 1 to 4, which indicates applying sufficient FG prompt is necessary to capture comprehensive FG information. However, further increasing Z leads to performance degradation, which we attribute to the inclusion of ineffective information (e.g., background noise). Therefore, we select $Z=4$ as the default setting.

Multi-scale constant factor $\eta$: In this paper, we utilize multi-scale strategy to obtain FG visual information under different scales. The results show that applying a relatively small $\eta$ (5 or 10) can achieve better performance, since applying over-large scale may involve ineffective information (e.g., background noise), thus leading to a negative effect.

Q3: Limited analysis of whether the surrogate features might be less effective for certain types of tasks or datasets.

A3: Thanks for the comments. Due to the character limits, please refer to Q&A 3 of Reviewer 4GML for details analysis.

Q1: How does the FG loss work with images that contain multiple classes? Would other instances in the figure disturb the classification since we only need to predict the most significant object? For instance, the author could provide heatmap visualization on such images.

A1: Thanks for the insightful comments. Other instances will not disturb the classification process. In both diverse prompt learning and surrogate prompt learning, each input text description is comprised by the text prompts and the classname. Classnames are pre-defined and remain fixed during optimization, thereby providing consistent class-level semantic guidance. To this end, every FG prompted text feature embeds information that strongly associated with its corresponding class, which ensures the classification procedure not being disturbed by other instances in the image.

To further validate this problem, we provide additional visualization results. We select images that contain at least two distinct object categories, and visualize the attention heat maps of different surrogate fine-grained text features on such images. The corresponding heat maps are shown in “multiclass-heatmap.png”, which demonstrate the effectiveness of FG prompted text features in our proposed method.

Q2: The description in Figure 2 is not clear enough, such as how the ID conditional signals interact inside the Surrogate Feature Generator. Moreover, the structure on the right does not match Eq(4), where $w \in M\times d$, $\alpha \in Z\times d$; then how does the output $h^{FG}$ become $Z\times M\times d$? If the conditional signals interact individually, you should improve the figure to show it, or it could be confusing to the reader.

A2: Thanks for the valuable suggestion. We are sorry for the unclear description of Eq(4). The conditional signals interact individually with each basic prompted text feature $w_m$. We rewrite Eq(4) as:

$$h_m = \theta _ {SFG}(w_m,\alpha).$$

This equation clearly states that for the basic prompted text feature corresponds to $m$-th class $w_m$, we will generate $Z$ fine-grained surrogate prompted text features $h^{FG} _ {m} \in \mathbb{R} ^ {Z \times d}$. To this end, we have $h^{FG} \in \mathbb{R} ^ {Z\times M\times d}$ for total $M$ classes.

We will correct the corresponding description in the next version of the manuscript. We also add the input and output notations of SFG in Fig.2 for clear description, which is shown in “Revised-Fig2.png”.

Q3: Since the fine-grained module improves performance according to the ablation study, has the author tried to adopt this idea to existing prompt learning techniques? I wonder why there is no such usage in the current community.

A3: Thanks for your comments. First, the idea of fine-grained prompt learning has been utilized in several VLM-based prompt learning methods, which we have already cited in the manuscript (e.g., Chen et al., 2023, Lafon et al., 2024). However, these methods suffer from the huge computation complexity problem, hindering their practical application in real-world scenarios.

Second, we claim that SurPL is adaptable and integrable to implement different diverse prompt learning ideas based on any single prompt learning approach. To this end, we have tried to implement SurPL on different existing single prompt learning methods, including CoOp, MaPLe and PSRC. Specifically, we consider these methods as baseline and additionally introduce both surrogate instance-dependent and fine-grained text features on each method. The comparison results are actually reported in Appendix Sec.E Table.12, and demonstrate that utilizing both instance-dependent and fine-grained can significantly improve the performance of existing single prompt learning methods.

To explicitly explore the effectiveness of the fine-grained idea, here we further conduct the following experiment. We consider CoOp, MaPLe and PSRC as baseline approaches, and exploit our method to only generate surrogate fine-grained features based on each approach. We provide the comparison of average performances on 11 datasets below.

Q4: Why is the ID signal derived from the visual feature while the FG signal is randomly initialized?

A4: Thanks for the question. For existing ID prompt learning methods (such as CoCoOp), text prompts are not randomly initialized for optimization, but require visual information as prior knowledge. So, the ID signals are derived from the visual feature, thus providing visual guidance. For existing FG prompt learning methods (such as PLOT and GalLoP), text prompts are randomly initialized and supposed to capture fine-grained information during optimization. Therefore, we also keep FG signals randomly initialized.

Q1: The O(M) computational complexity claim of SurPL lacks a formal derivation.

A1: Thanks for the valuable comments. We provide a theoretical analysis of computational complexity here.

Notations.

Loss $L$, classnames ${c}=(c_m) _ {m=1}^M$, text encoder parameter: ${T}=({T}^{k}) _ {k=1}^K$, SFG parameter: $\theta_{SFG}$, text prompts parameter: $s=(s^k) _{k=1}^K$, encoder layer depth: $K$, the parameter size of SFG is much smaller than that of text encoder: $\theta _ {SFG}<<T$.

Derivation.

Optimizing the text prompt $s^1$ at the text encoder input position requires computing the gradient through the entire model. For each single surrogate output text feature, the back-propagation computation of SurPL can be written as:

$$\frac{\partial L}{\partial s^1}=\underbrace{\frac{\partial L}{\partial \theta _ {SFG}(T^K([s^{K},c _ m^k]);\alpha_j)}\cdot \frac{\partial \theta _ {SFG}(T^K([s^{K},c _ m^k]);\alpha _ j)}{\partial T^K([s^K,c _ m^K])}} _ {\text{SFG}}\cdot \underbrace{\frac{\partial T^K([s^K,c _ m^K])}{\partial T^{K-1}([{s}^{K-1},c _ m^{K-1}])}\cdots \frac{\partial {T}^{1}([s^1,c _ m^1])}{\partial s^1}} _ {\text{Text Encoder}}.$$

This gradient computation can be conceptually divided into two parts, which correspond to the text encoder and SFG. We denote the computational complexity for each output text feature as $O_T$ and $O_{SFG}$, respectively. As shown in Fig.1 (c), SurPL simultaneously involves $M$ text features back-propagation w.r.t text encoder, which equals to the complexity in single prompt learning: $O(M) = M \cdot O_T$. For the SFG stage, $(B+Z)M$ text features are involved in computation. The overall complexity of SurPL can thus be expressed as:

$$O_{SurPL} = M \cdot O _ T + (B+Z)M \cdot O _ {SFG}$$

Since $\theta_{SFG}<<T$, it follows that $O_{SFG}<<O_T$. Therefore, the second term is negligible, and we can approximate: $O _ {SurPL} \approx M \cdot O_T = O(M)$.

Q2: Comparison with non-prompt-based PEFT methods.

A2: Thanks for the suggestion. We further compare SurPL with Adapters (CLIP-Adapter(Gao et al., 2024), Tip-Adapter(Zhang et al., 2022), TaskRes(Yu et al., 2023)), BitFit (CLIPFit$^\text{a}$) and LoRA (CLIP-LoRA$^\text{b}$) based methods. The averaged performances of 11 datasets are shown below, which demonstrate the superiority of SurPL compared to other PEFT techniques. We will add more detailed results and analysis in the next version of the manuscript.

a. Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification. EMNLP 2024.

b. Low-Rank Few-Shot Adaptation of Vision-Language Models. CVPRW 2024.

Q3: Experiments focus only on classification. Testing on VQA, object detection, or video understanding would provide a broader evaluation.

A3: Thanks for your valuable suggestion. First, to our best knowledge, almost all studies of VLM-based PEFT techniques (e.g. prompts, adapters, LoRA, BitFit) mainly evaluate on classification tasks, and we simply follow these established protocols for fair comparison.

Second, although some researches have applied PEFT techniques on other visual tasks, they tend to leverage existing methods rather than explore new approaches. Diverse prompt learning has not been well-explored on other tasks, making it challenging to identify a suitable baseline for SurPL on such tasks. Additionally, exploring the effectiveness of diverse prompt learning on VQA and detection from scratch and then implementing SurPL require significant time, making it difficult to include such results within the rebuttal period.

However, we sincerely appreciate this insightful suggestion and will actively explore this direction in the near future.

Q4: Hyperparameter sensitivity analysis is missing.

A4: Thanks for the comments. Due to the character limit, we provide comprehensive hyperparameter analysis at the Q&A 2 of Reviewer jTad. The results indicate that the hyperparameters applied in this work are reasonable and relatively stable. Please refer to that response for details.

Q5: Why was cross-attention chosen as SFG over other feature generation techniques (e.g., Transformer-based mechanisms)?

A5: Thanks for the comments. SFG aims to take basic prompted text feature and conditional signal as input, and generate the surrogate prompted feature according to the given signal. Cross-attention module is the most intuitive choice for this requirement. While Transformer-based mechanisms can also achieve it, stacking multiple attention blocks would significantly decrease the efficiency.

Q6: Minor suggestions of writing.

A6: Thanks for the valuable suggestions. Due to the character limit, we are not able to provide detailed modifications in the rebuttal. We will revise them in the next version of the manuscript.