Token Coordinated Prompt Attention is Needed for Visual Prompting

Thank you for your appreciation of our novelty, effectiveness and comprehensive experiments.

(The images mentioned below are available at the anonymous link: https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E.)

Q1: More Visualizations

1. Thank you for your valuable feedback. We have included additional 2D and 3D attention map visualizations with more samples in the appendix.
2. As shown in Fig.4.1 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/4.1.png), the same trend is observed in the additional visualizations. In existing visual prompt methods, the attention regions of prompts are highly similar, leading to nearly identical features being extracted from the CLS and image tokens. In contrast, our proposed TCPA module enhances more diverse attention across prompts, CLS tokens, and image tokens.
3. This is because our method selects different prompts for different tokens and performs attention-based interactions, encouraging the model to extract more diverse and comprehensive discriminative information.

Q2: Hyperparameter

1. As shown in Fig.4.2 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/4.2.png), we have added hyperparameter experiments on the Dog and GTSRB datasets, which exhibit the same trend observed on the CUB dataset.
2. When the prompt pool is too small, prompt diversity is limited, causing high overlap in selected prompts and making the extracted features indistinguishable. On the other hand, an excessively large prompt pool increases learnable parameters, which can lead to overfitting and decreased performance. Optimal performance is achieved with a moderate pool size.
3. Additionally, we have included hyperparameter experiments for the weight parameters $\lambda_i$ and $\lambda_c$. As shown in Fig.3.2 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/3.2.png), the best performance is achieved when $\lambda_i=0.03$ and $\lambda_c=0.02$. When $\lambda_i$ and $\lambda_c$ are too large, they interfere with the learning of prompts and the classifier, leading to performance degradation. When $\lambda_i$ and $\lambda_c$ are too small, the indicators corresponding to the prompts cannot be effectively learned, making it difficult to accurately match prompts to different tokens, which also results in performance degradation.

Q3: Pseudocode of the Method

1. Thank you for your valuable suggestions.
2. We have added the pseudocode of the method in the appendix to clearly illustrate the proposed approach.

Q4: Applicability of the Method

1. Existing visual prompt learning methods can be categorized into two types based on the prompt placement: image-based and token-based. Our approach is compatible with both.
2. For token-based visual prompt learning methods, our TCPA can enhance the attention interaction process, thereby improving performance.
3. For image-based methods, token prompts can be incorporated alongside the original approach, integrating our TCPA to provide continuous prompting during feature extraction.

Thank you for your appreciation of our novelty, effectiveness and comprehensive experiments.

(The images mentioned below are available at the anonymous link: https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E.)

Q1: Cosine Distance-based Matching Function

1. We conduct a visualization experiment on the image prompts selected by different image tokens. As shown in Fig.3.1 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/3.1.png), tokens corresponding to different parts of an object select different prompts. This indicates that our matching mechanism can recognize the different semantic information contained in tokens to some extent and assign the corresponding prompts accordingly.
2. During the method design process, we explored various alternative matching strategies, including Euclidean distance, Kullback-Leibler (KL) divergence, and weight prediction through a learnable MLP. As shown in the table below, among these matching strategies, cosine distance achieved the best performance.
3. In future research, we will further optimize the prompt matching mechanism by incorporating neighborhood token information to achieve more accurate prompt assignment.

Q2: Hyperparameter

1. Thank you for your valuable suggestions. We conduct an ablation study on the weight hyperparameters $\lambda_i$ and $\lambda_c$ on the CUB dataset. As shown in Fig.3.2 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/3.2.png), the best performance is achieved when $\lambda_i=0.03$ and $\lambda_c=0.02$.
2. When $\lambda_i$ and $\lambda_c$ are too large, they interfere with the learning of prompts and the classifier, leading to performance degradation.
3. When $\lambda_i$ and $\lambda_c$ are too small, the indicators corresponding to the prompts cannot be effectively learned, making it difficult to accurately match prompts to different tokens, which also results in performance degradation.

Q3: Base-to-novel Task

1. First, for a fair comparison with existing visual prompt learning methods, we conducted experiments on the HTA and VTAB benchmarks to validate the effectiveness of our approach.
2. Existing visual prompt learning methods typically use a ViT backbone. When adapting a pretrained ViT model to downstream tasks, a task-specific classifier must be learned based on the number of categories in the target task. Consequently, due to the constraints of the classifier, ViT models cannot conveniently train on base classes and generalize to novel classes as CLIP models do.
3. Additionally, we adapt several visual prompt learning methods, including VP, VPT, and VFPT, and our TCPA, to the CLIP model to evaluate their performance on the base-to-novel task. As shown in the table below, using the VFPT method as an example, our TCPA improves performance by 0.5%–0.9% on base classes and by 0.5%–1.3% on novel classes. Similarly, our TCPA also achieves consistent performance improvements on the VP and VPT methods. This improvement stems from our approach’s ability to assign different prompts to different image tokens, enabling the model to capture more comprehensive and fine-grained discriminative information for each category, thereby enhancing generalization to novel classes.
4. In future research, we plan to incorporate part-level modeling of learned categories to further improve the applicability and generalizability of our approach.

Thank you for your appreciation of our motivation, effectiveness and comprehensive experiments.

Q1: Theorem

1. Theorem 4.1 and Theorem 4.2 mentioned in our paper are established theories from existing work, which we have appropriately cited.
2. In their original paper, these theorems were used to analyze the rank of the attention matrix in vision models. In our paper, we reference these theorems alongside Fig.3 to illustrate that existing prompt learning methods tend to focus on overlapping information, whereas our TCPA assigns different prompts to different tokens, enabling a more comprehensive extraction of discriminative features.
3. We have now included the proof of Theorem 4.1 and Theorem 4.2 in the appendix for completeness.

Q2: Hyperparameter

Thank you for your valuable suggestions. We have included hyperparameter experiments for the weight parameters $\lambda_i$ and $\lambda_c$, as detailed in Reviewer LMp2 Q2:Hyperparameter.

Q3: Section 5.5.4

1. In Section 5.5.4, we primarily present the t-SNE visualizations of the features extracted by both the baseline methods and our approach, further demonstrating that our method captures more comprehensive discriminative information.
2. We have revised and refined the last two sentences in this section to eliminate redundancy.

Q4: Notations in Section 3

1. We have thoroughly checked the formulas in the paper. For example, we have standardized the use of superscripts to indicate different attributes of tokens, prompts, and indicators (i.e., whether they belong to the CLS token or image token), while indices representing layer numbers and patch indices are consistently placed in subscripts. For example, the notation $\boldsymbol{h}^1_i$, which originally represented the first token in the $i$-th layer, has been revised to $\boldsymbol{h}_{i,1}$. Additionally, to improve clarity, we have changed $M$, which originally represented the number of patches, to $N$, and $N$, which originally represented the number of network layers, to $L$.
2. Furthermore, we have carefully reviewed other parts of the paper and corrected grammatical issues and typos. For example, in Eq.9, we have corrected $\boldsymbol{k}$ to $\boldsymbol{\kappa}$.

Q5: Optimization Objective

1. The objective of Eq.14 is to minimize the distance between image tokens and their corresponding prompt indicators, ensuring that both $\sum{\mathcal{S}(\boldsymbol{h}_m, \boldsymbol{\kappa}^i_m)}$ and $\sum{\mathcal{S}(\boldsymbol{c}_j, \boldsymbol{\kappa}^c_j)}$ are as small as possible.
2. To enhance clarity, we have added an explanation of the optimization objective before Eq.14.

Q6: Eq.11

1. The binarized matrix $\mathrm{\mathbf{\hat{A}}} \in \{0,1\}^{M \times N_i}$ has dimensions matching the number of image tokens, but in the actual model, the CLS token needs to be considered. In other words, the dimensions of $\mathrm{\mathbf{\hat{A}}}$ need to match the number of image tokens plus one. Eq.11 is designed to achieve this dimensional alignment.
2. To help readers better understand, we have added an explanatory note before Eq.11 in the paper.

Q7: "Row 9, Column 8" in Figure 1

1. The first two are objects, and a background token is selected as a reference for comparison to make the experiment more comprehensive.
2. Existing methods focus on the same regions regardless of whether the tokens correspond to objects or the background, whereas in our approach, different object tokens attend to different object regions, and the background token focuses more on the background.
3. This is because our TCPA matches different prompts to tokens with different semantics for attention interactions, enabling more comprehensive extraction of discriminative information from the image.

Q8: Notation $p_{j+1}^{i,d}$ in Section 3.2

1. This is a typo on our part. The output $p_{j+1}^{i,d}$ for each layer should be discarded and not used for the next layer's output.

Q9: Eq.9

1. Yes, $k_{k}^{i}$ and $\kappa_{k}^{i}$ in Eq.9 represent the same notation.
2. We have standardized it to $\kappa_{k}^{i}$.

Q10: Notation $\hat{A}_{m,k}$

1. Yes, $\hat{A}_{m,k}$ represents an element of the matrix $\tilde{A}$.
2. We have standardized it to $\hat{A}$ and no longer use $\tilde{A}$ to represent it.

Q11: Typo

1. We have revised $\mathrm{\mathbf{M}}^i$ to $\mathrm{\mathbf{M}}^c$ in the sentence below Eq.11 in Section 3.3.
2. We have also carefully checked and corrected other sections of the paper for grammatical issues and typos, such as changing "SPT" to "VPT" in Section 5.5.

Thank you for your appreciation of our clearity, motivation and sound experiments.

(The images mentioned below are available at the anonymous link: https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E.)

Q1: Load Balance of Different Prompts

1. Since the number of CLS and image tokens varies across different semantics, some tokens share similar meanings, while others are more distinct. As a result, prompts associated with different semantics should be selected at different frequencies. Therefore, we did not impose a strict load-balancing design for different prompts.
2. To ensure that every prompt is optimized rather than some prompts never being selected, we randomly choose prompts during the first 10 epochs of model training. In the subsequent 90 epochs, prompt selection is guided by a prompt indicator, which adjusts selection based on semantic information. This design ensures that all prompts and their corresponding indicators are optimized while maintaining differing selection frequencies for prompts associated with different semantics.
3. Furthermore, we visualize the selection frequency of prompts, as shown in Fig.1.1 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/1.1.png). The results indicate that different CLS prompts are selected at frequencies ranging from 0.13 to 0.45, while different image prompts are selected at frequencies ranging from 0.05 to 0.24. This verifies that all prompts in our method are effectively utilized, with none remaining unused.

Q2: Eq.14

1. We have carefully reviewed Eq.14 and the related Eq.9 and confirmed that our formulations are correct.
2. Eq.9, $\mathcal{S}(\boldsymbol{h}_m^{j},\boldsymbol{\kappa}^i_k)=1-\mathrm{cos}(\boldsymbol{h}_m^{j}, \boldsymbol{\kappa}^i_k)$, computes the cosine distance between $\boldsymbol{h}_m^{j}$ and $\boldsymbol{\kappa}^i_k$. A higher similarity between $\boldsymbol{h}_m^{j}$ and $\boldsymbol{\kappa}^i_k$ results in a larger $\mathrm{cos}(\boldsymbol{h}_m^{j},\boldsymbol{\kappa}^i_k)$, leading to a smaller cosine distance $\mathcal{S}(\boldsymbol{h}_m^{j},\boldsymbol{\kappa}^i_k)$.
3. The objective of Eq.14 is to minimize the distance between image tokens and their corresponding prompt indicators, ensuring that both $\sum{\mathcal{S}(\boldsymbol{h}_m,\boldsymbol{\kappa}^i_m)}$ and $\sum{\mathcal{S}(\boldsymbol{c}_j,\boldsymbol{\kappa}^c_j)}$ are as small as possible.
4. To enhance clarity, we have added an explanation of the optimization objective before Eq.14.

Q3: Fig.3

1. Figure 3 in our paper visualizes the attention map in ViT, defined as $A=\text{Softmax} \left(\frac{Q K^T}{\sqrt{d_k}}\right)$.
2. In (b) and (d), both the x-axis and y-axis are token indices, with the y-axis corresponding to queries and the x-axis to keys. The color variations indicate the attention weights.
3. In (a) and (c), the z-axis is the attention weights, while both the x-axis and y-axis correspond to token indices. The difference is that the x-axis represents queries, and the y-axis represents keys. Notably, for clarity, we did not visualize all queries but instead focused on CLS tokens and a subset of prompt and image tokens.
4. We have revised Fig.3, explicitly annotating the meaning of each axis. The updated figure is provided in Fig.1.2 (https://anonymous.4open.science/r/ICML-2025-Paper35-Rebuttal-7E9E/1.2.png).

Q4: Notation

1. In our original paper, $M$ represents the number of patches. It is italic, uppercase, and not bold, used to denote a scalar. In contrast, $\mathrm{\mathbf{M}}$ represents the final mask. It is upright, uppercase, and bold, used to denote a tensor.
2. To improve clarity, we have changed $M$, which originally represented the number of patches, to $N$, and $N$, which originally represented the number of network layers, to $L$.
3. The paper does not use subscripts $c$ and $h$. Instead, superscripts $c$ and $i$ are used to differentiate between CLS token-related and image token-related prompts and indicators. These are lowercase, italic, and not bold. Meanwhile, $\boldsymbol{h}$ represents tokens in the network, which are vectors and are written in lowercase, italic, and bold.
4. To enhance readability, we have standardized the use of superscripts to indicate different attributes of tokens, prompts, and indicators (belong to CLS token or image token). Meanwhile, indices indicating layer numbers and patch indices are consistently placed in subscripts. For example, the notation $\boldsymbol{h}^1_i$, which originally represented the first token in the $i$-th layer, has been revised to $\boldsymbol{h}_{i,1}$.

Q5: Typo

1. We have corrected "SPT" to "VPT" in Sec.5.5.
2. Additionally, we have thoroughly checked and revised other parts of the paper for grammatical issues and typos. For example, in Eq.9, we have corrected $\boldsymbol{k}$ to $\boldsymbol{\kappa}$.