CVPT: Cross-Attention help Visual Prompt Tuning adapt visual task

Firstly, we would like to show our gratitude. The paper you cited (Ref1) has greatly inspired us, supporting many of our ideas and significantly helping our subsequent work.

W1&W2:

(1)Lack of representative information. The lack of representative information we refer to is relative to NLP. In NLP, a token represents a word, whereas in ViT, a token represents an image patch. Comparatively, tokens extracted from high-dimensional images contain richer information. When using the same approach to concatenating prompts with image patches, the prompts lack representative information.

(2)Benefit from over-emphasis & relationship between the distribution of the weights and performance. From Table 1, we can see how over-emphasis and weights affect the results. Specifically, over-emphasis on prompts leads to the neglect of the weight of embedded tokens in self-attention. Significantly affecting VPT's performance on the VTAB benchmark (Table 1), especially on VTAB-Natural.

(3)Prompts learned a more suitable and efficient representation than the embedded tokens. We agree with your points. In some situations (especially on VTAB-Structured), giving a large number of prompts can achieve better performance. In Ref1, the authors mentioned that when the distribution of downstream data significantly differs from pre-training data (e.g., VTAB-Structured), the feature representations captured by the pre-training parameters may not be suitable for downstream tasks. A small prompt set can be inserted for better adaptation when the data is similar (e.g., VTAB-Natural). This aligns with our observed results. Although prompts can learn better representations compared to embedded tokens in some cases, we believe this should not come at the expense of disrupting the self-attention among embedded tokens. In Table 2, we demonstrate that CVPT outperforms VPT in both the Natural and Structured groups, highlighting the benefits of preserving complete self-attention. Furthermore, CVPT's performance on ADE20K shows that prompts in CVPT adapt to downstream tasks much better than those in VPT. We will include the above analysis and discussion in the revised version.)

W3 (Cross-attention should not be assigned in Sec3.2.): This is a great suggestion, but we do not claim cross-attention as our contribution; it is included in Section 3.2 because we utilized cross-attention. In fact, a similar writing was used in E2VPT (Sec 3.2 Visual Prompts, in Ref2). Of course, we will cite the original paper of cross-attention in our revised version.

W4 (More discussions with E2VPT): Actually, E2VPT's approach involves concatenating prompts in the K-set and V-set (Fig. 2a and 2e in Ref2). This method is similar to VPT but differs in that E2VPT adds prompts only at the end of the K-set and V-set, rather than concatenating with embedded tokens in VPT (where prompts are added in Q, K, and V). This represents a fundamental difference from our approach. Additionally, considering that CVPT has already been extensively compared with VPT, the length of the paper does not permit further discussion on E2VPT, which is similar to VPT.

W5 (Inconsistent experiment setup): Fig. 2 shows the attention of the cls_token to prompts and embedded tokens when they participate together in self-attention in VPT. However, in our CVPT, prompts do not participate in self-attention with embedded tokens (Fig. 3). Consequently, CVPT does not affect the self-attention of embedded tokens and cls_token will not be affected by prompts. Therefore, we couldn't conduct a similar experiment on CVPT.

W6 (Performance on other transformer architecture): Our method can be adapted to Swin Transformers. Specifically, after computing W-MSA and SW-MSA in the Swin block, we calculate the cross-attention between prompts and embedded tokens. The result on VTAB is 76.0, which is higher than VPT (70.7) and E2VPT (75.2). It needs to be emphasized that due to the constraints of time and devices, we did not conduct many ablation experiments on Swin and tune the hyperparameters to their optimal values. This indicates that there is significant potential for further improvement. Therefore, we believe that CVPT can be adapted to other transformer-based architecture.

[Ref1] Han, Cheng, et al. "Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning?." The Twelfth International Conference on Learning Representations.

[Ref2] Han, Cheng, et al. "E^ 2VPT: An Effective and Efficient Approach for Visual Prompt Tuning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

We appreciate the reviewer's responses. Below, we address the reviewer's concerns:

(1) The explanation for the lack of representative information can be substantiated by the number of parameters. Specifically, in the case of a ViT-base model, each prompt contains 768 parameters. In contrast, an adapter typically consists of both upsampling and downsampling layers, often with a factor of 8 or higher, and usually includes two adapters per block. As a result, the parameter count in prompts is generally smaller compared to an adapter. To achieve a parameter count equivalent to that of an adapter, it typically requires 36 or more prompts.

In Ref1 (Sec 4.4), the authors demonstrated that fine-tuning (FT) surpasses VPT in performance as the dataset size increases. This suggests that when a certain data scale is reached, the number of trainable parameters has a significant impact on performance, a trend also observed in other prompt- and adapter-based approaches (Table 5 in Ref2, Table 3 in Ref3). Consequently, we mentioned in L48 that "it is necessary for VPT to use an abundant amount of prompts to fine-tune models." Therefore, from the perspective of parameter count, the claim that "lack of representative information" is valid.

(2) In Ref1 (Sec4.3), the authors demonstrate that for prompts to learn a more suitable representation, two conditions must be met: first, the features learned by the pre-trained model are not suitable for the downstream task, and second, the dataset is small in scale. In our paper, we show that when the pre-trained features align with the downstream task, the emphasis on prompts in VPT can distort self-attention, which does not contradict the findings in other papers. Below, we illustrate the performance variations on certain datasets within the VTAB-Natural as the number of prompts increases. It is clear that representations learned by embedded tokens are more suitable.

Based on this, we believe our experiments and the experiments in Ref1 adequately demonstrate in which situations prompts will learn suitable representations. Besides, our experiments also indicate that prompts in CVPT learn a more suitable representation compared to those in VPT.

Furthermore, our work mainly focuses on proposing an improved prompt-based method, and we believe our experiments sufficiently demonstrate why CVPT outperforms VPT. Therefore, considering that the theoretical analysis of whether prompts learned better representations in VPT does not occur in other papers of derived prompt methods, it may be beyond the scope of our paper.

(3) Finally, we respectfully disagree with the reviewer's comment that 'I can see a new structural design in this paper, but no further contribution is introduced for the PEFT community.' Specifically, our contribution lies in analyzing the weakness of the method used in previous prompt-based approaches for associating prompts with embedded tokens and proposing a new method for improvement. This significantly enhances the performance of prompt methods, making them competitive with adapter methods. Actually, many researchers have abandoned prompts due to their weak performance, our work will re-inspire the community's research on prompts. Therefore, we think it is not merely a 'new structural design.'

[Ref1] Han, Cheng, et al. "Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning?." The Twelfth International Conference on Learning Representations.

[Ref2] Bandara, Wele Gedara Chaminda, and Vishal M. Patel. "Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Action Recognition." 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 2024.

[Ref3] Chen, Shoufa, et al. "Adaptformer: Adapting vision transformers for scalable visual recognition." Advances in Neural Information Processing Systems 35 (2022): 16664-16678.

W1 (An unnecessarily large number of prompts): For VTAB (comprising 19 datasets), 10 datasets achieved the best performance using 50 or more prompts. For FGVC (comprising 5 datasets), 3 datasets performed best with 50 or more prompts. Additionally, complex downstream tasks such as semantic segmentation or video classification, often require more prompts even over 200(Table 4 in CVPT, Table 6 in Ref 1). Therefore, it is essential to consider scenarios with a larger number of prompts, and our settings are reasonable. Considering that the difference in memory usage and FLOPs between CVPT with prompt=1 and prompt=10 is minimal (usually less than 200M), we believe that when using a small number of prompts (less than 20), CVPT has an advantage in accuracy. Moreover, when using a large number of prompts (more than 50), CVPT demonstrates significant advantages in both efficiency and performance.

W2 (How does CrossViT relate to addressing the three issues of VPT): We need to emphasize that our contribution lies in optimizing the insertion of prompt used in previous prompt-based methods by decoupling it from self-attention and introducing cross-attention to establish a connection between prompts and embedded tokens. In contrast, CrossViT combines self-attention and cross-attention to capture information at different scales, which is fundamentally different from our approach.

Realizing that this approach might lead to quadratic complexity and destruction of self-attention between embedded tokens, we aimed to decouple prompts from self-attention to preserve the complete pre-trained features. However, this means we need to consider how to establish the connection between prompts and embedded tokens so that prompts can guide the model's fine-tuning. Naturally, we thought of using cross-attention to compute the relationship between two sequences, introducing linear complexity while preserving the complete self-attention in ViT. Additionally, unlike VPT, which treats prompts and embedded tokens equally by combining them into a single sequence for self-attention, using cross-attention compensates for the lack of semantic information in prompts. Experiments in Sec 3.3 also demonstrate that cross-attention can process the fine-tuning information contained in prompts more effectively and efficiently. Meanwhile, other derived methods do not recognize the drawbacks of combining prompts with embedded tokens and continue using the same method as VPT. Therefore, it is unsurprising that our CVPT outperforms them.

W3 (provide code): We have organized our code and released it (following the rules, we sent it to AC separately).

Q: Yes, we will clarify it in our revised version.

[Ref1] Bandara, Wele Gedara Chaminda, and Vishal M. Patel. "Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Action Recognition." 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 2024.

Thank you for the rebuttal. I will maintain my score.

Hi,

Could you take a look at the authors rebuttal and finalize your rating?

Thanks, AC

W1 (Limited novelty): In fact, our contribution lies in optimizing the prompt insertion of VPT by decoupling the prompt from self-attention and linking the prompt with embedded tokens using cross-attention. CVPT doesn't modify the self-attention mechanism in ViT, nor does it involve the combination of self and cross-attention. Additionally, regarding the two papers you mentioned as similar to our work: the contribution of Paper 1 is the introduction of a drone dataset and a convolutional network-based action recognition method, which is unrelated to our contribution. The contribution of Paper 2 is the proposal of V-PEFT, which combines adapters and prompts for network fine-tuning and it is not similar to CVPT.

W2 (Limited impact and efficiency): Actually, the number of prompts (fewer than 20) we employed in VTAB was a strategy to avoid extensive hyperparameter searches. Using more prompts can still lead to performance improvements (Table 1). Additionally, the authors of VPT listed the optimal number of prompts for various downstream tasks in the appendix. For VTAB (comprising 19 datasets), 10 datasets achieved the best performance using 50 or more prompts. For FGVC (comprising 5 datasets), 3 datasets performed best with 50 or more prompts. Furthermore, for more complex downstream tasks such as semantic segmentation or video classification, an increased number of prompts can significantly enhance performance (Table 4 in CVPT, Table 6 in Ref 1). As mentioned above, a large number of prompts is common to prompt-based methods. Therefore, we believe that the efficiency improvements of CVPT are significant.

W3 (Limited performance): We have read these two papers you mentioned which show better performance and found that, although their reported performance on FGVC is higher than ours, their performance on VTAB is lower. Additionally, during our training process, we observed that different code frameworks used in various papers resulted in performance discrepancies. For instance, our code is based on the RepAdapter, and the results we obtained for VPT are more than 1% higher than those reported in VPT. Furthermore, training on FGVC and VTAB is highly sensitive to hyperparameters; changing the seed alone can sometimes lead to a more than 5% difference in accuracy. This is why PEFT methods typically require extensive hyperparameter searches(Ref2, Ref3). This implies that to some extent, the results on VTAB and FGVC depend significantly on the extent of the hyperparameter search conducted. To mitigate this effect, we reran VPT within our code framework and maintained consistent hyperparameters for comparison (Table 1). In contrast, training the ADE20K dataset with MMSegmentation yields more stable results, with random seed variations affecting the results by only about 0.3%. Based on this, we consider the results of ADE20K to be more persuasive. Therefore, we consider that "CVPT achieving SOTA" is valid.

[Ref1] Bandara, Wele Gedara Chaminda, and Vishal M. Patel. "Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Action Recognition." 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 2024.

[Ref2] Jia, Menglin, et al. "Visual prompt tuning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[Ref3] Zhang, Yuanhan, Kaiyang Zhou, and Ziwei Liu. "Neural prompt search." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

Dear Authors,

Thanks for your response. However, I was totally not convinced by them on most of the issues I proposed for the weakness. The response is inconvincible and sophistic, which is not acceptable. I would maintain a clear rejection score on this paper.

Best, Reviewer

W1(Conclusion seems to be more of an abstract): Below is our modified conclusion, and this will be introduced in our revised version.

In the current field of visual fine-tuning, many researchers overlook prompts in favor of adapters due to their strong performance. The few prompt-based derived works do not realize the drawbacks of combining prompts with embedded tokens, continuing to use the method from VPT. In light of this, we thoroughly analyzed the shortcomings of such deployment and proposed CVPT. Its advantages are as follows: 1) It uses cross-attention to establish a connection with embedded tokens, decoupling prompts from self-attention. 2) It employs weight-sharing to avoid the large number of learnable parameters introduced by cross-attention. Additionally, we conducted extensive experiments on CVPT, demonstrating its efficiency and performance improvements over VPT and the effectiveness of cross-attention and weight-sharing. Therefore, we prove that prompt-based methods can perform comparably to advanced adapter methods in the visual fine-tuning domain.

W2(More implementation details): We have released our code (following the rules, we sent it to AC separately), we believe this will help readers understand our method.

W3(The ablation for self-attention): In Fig. 3, we present the implementation of CVPT. As shown, we decouple prompts from embedded tokens and use cross-attention to establish the connection between prompts and embedded tokens. This prevents prompts from participating in the self-attention calculations among the original tokens in ViT. We use cross-attention because it computes the attention between two sequences, whereas self-attention can only process a single sequence. Therefore, in our method, the ablation for self-attention is not feasible.

W4(The understanding of the effectiveness of our method): As we understand, 'the base cases with null text' is the approach of not using prompts and merely setting the last classifier layer as learnable. In fact, this method is Linear Probing, which we have discussed in our paper (the caption of Fig. 4) and introduced its performance for comparison (Table 2, 3, 4).

Q(how does it affect the generation): Actually, PEFT is widely used in diffusion models. For example, the popularity of LoRA (Low-Rank Adaptation) stems from its application in the AI art community. For text-to-image (T2I) diffusion models represented by Stable Diffusion, PEFT can modify the generation style by introducing a small number of additional parameters to adjust the weights of various parameters in the model without altering the base model. Regarding diffusion models represented by DDPM, we think that inserting adapters or prompts into the UNet or Transformer components could alter the style of the generated noise, thereby influencing the model's generation.

Hi,

Could you take a look at the authors rebuttal and finalize your rating?

Thanks, AC