Attention Prompt Tuning

1. Larger-scale experiments are needed to validate the proposed parameter-efficient tuning as an alternative to full fine-tuning. For models like ViT-S and ViT-B, the results on SSV2 are significantly better, and the cost of fine-tuning is still affordable, so the advantage of the proposed method is not clear. (Cheaper training but sacrificing performance)
2. It would be better to compare the proposed method with other VPT variants, such as [1].

[1] Han C, Wang Q, Cui Y, et al. E^ 2VPT: An Effective and Efficient Approach for Visual Prompt Tuning[J]. ICCV 2023.

1. The paper lacks an in-depth analysis of the proposed method. The authors should conduct a more comprehensive explanation of the re-parameterization technique they proposed. For example,the authors should clearly explain why re-parameterization can enhance robustness and convergence speed while reducing loss and what is the fundamental distinction between the proposed re-parameterization and the standard re-parameterization method [1].

2, Some of the experimental results are confusing. The authors claim that the proposed re-parameterization technique can improve hyper-parameter robustness. However, there is a lack of supporting experiments. APT still exhibits sensitivity to prompt length (Table 1) and dropout rate (Table 2). The only comparison between with and without re-parameterization is about the loss curve, which may not adequately demonstrate robustness.

3. Regarding generalizability, the paper only conducts experiments on the standard ViT architecture and lacks experiments on more advanced ViT architectures. Furthermore, since APT is both effective and efficient compared to VPT, the authors could also evaluate it across various modalities to demonstrate its generalizability.

[1] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks. In ACL'22.

1. My main concern lies in that from my perspective, APT is highly similar to prefixing tuning[1-2], which has already been verified effective for visual tasks in previous works [3-4]. Additionally, a similar reparameterization strategy for prompt tuning has already been explored in [5] (authors also DO NOT cite this method in the paper). In light of previous works, novelty and contribution of APT are substantially restricted.

2. To compare with other baselines, it would be appreciated to include views (temporal and spatial ones) in the table for video models.

3. As for efficient tuning on videos, it is important to not only ensure tunable parameters as few as possible but also consider keeping to low GFLOPS, which may also have great impacts on efficiency, training costs and application value. On that basis, APT may improve performance at the cost of high GFLOPS, e.g., APT-400 outperforms other PEFT methods in Table 5 while bringing about the most increase in GLOPS and in Table 6, AIM hits 66.4 with GFLOPS of 624 while APT-400 operates with the GFLOPS of 192.0533 = 1728.45.

4. The experiments involved in the paper are based mainly on ViT-S and ViT-B, which may be due to some practical limitations. Whereas the performance of delta tuning on large models is more valuable to the community.

[1] AIM: Adapting Image Models for Efficient Video Action Recognition, Li et al. ACL 21.

[2] Towards a Unified View of Parameter-Efficient Transfer Learning, He et al. ICLR 22.

[3] Towards a Unified View on Visual Parameter-Efficient Transfer Learning, Yu et al. Arxiv 22.

[4] FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learning, Sun et al. ICCV 23.

[5] MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators, Tan et al. ACL 22.

1. The proposed method lacks innovation. Firstly, the prompt in existing methods can be inserted not only after the tokenizer, so Figure 1 is problematic. Secondly, there have been numerous studies discussing the position of prompts in the vision transformer, with basic choices including Prompt Tuning (Pro-T) [1] and Prefix Tuning (Pre-T) [2]. The Pre-T technique has been widely used in existing computer vision methods, e.g., [3]. Is there any difference between the first technique of this paper and Prefix Tuning?

[1] The Power of Scale for Parameter-Efficient Prompt Tuning, EMNLP 2021.

[2] Prefix-Tuning: Optimizing Continuous Prompts for Generation, ACL 2021.

[3] DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning, ECCV 2022.

2. The authors do not connect the techniques proposed in this paper and video. Why can these proposed techniques solve the special problem of prompt tuning in video applications? The performance of Pre-T is inherently better than that of Pro-T. Is the performance improvement derived from Pre-T itself or from solving the problem?

3. There is no detailed analysis on how much parameter reduction is achieved compared to MLP-based reparameterization, nor any changes in performance.

4. There should be a fine-grained control of variables for each model (GFLOPS and performance under fine-tuning different parametric quantities should be compared in a more granular way).

In conclusion, this paper is more similar to a technical report, rather than a research paper. It applies a widely used trick Prefix Tuning to prompt tuning in video-based applications, and treats it as the core contribution of this paper. More importantly, this paper lacks the analysis for the special problem in video-based applications (i.e., need many prompts) and lacks the insights to design their techniques for solving this special problem. Pre-T has inherently better performance than Pro-T, so the performance improvement comes from Pre-T technology itself or from solving the problem? In addition, regarding the second contribution, why do the authors not provide the comparison between their prompt reparameterization technique and existing prompt reparameterization techniques? I believe that this paper falls considerably short of the standard required for ICLR and, as such, I recommend rejection.