ADAPT: Adaptive Prompt Tuning for Pre-Trained Vision-Language Models

The authors thank the reviewer for the feedback. Here is one-to-one response:

• Weakness 1: The ICLR submission rule suggests the total number of pages to be 9. Following the reviewer’s suggestion, we add the analysis of the total context lengths in Appendix A.8.

We believe the comment “There is more than one page to write.” is a misunderstanding. In the page length section of ICLR call for papers, it is stated that “We encourage authors to be crisp in their writing by submitting papers with 9 pages of main text. We recommend that authors only use the longer page limit in order to include larger and more detailed figures. However, authors are free to use the pages as they wish, as long as they obey the page limits.” (please refer to the submission guidance https://iclr.cc/Conferences/2025/CallForPapers).

In our original submission, we show the result of pruned binary masks in Appendix Fig. 8. The results indicate that the context lengths are highly heterogeneous. We added a section (Appendix A.8) to analyze the total context lengths in text and image branches. The total context lengths are highly related to datasets as shown in Appendix Figure 6.

• Weakness 2: We believe this is an oversight. We did upload the appendix with our original submission. Please refer to the link for supplementary material.

• Weakness 3: The number of tokens to be pruned depends on the importance of the tokens. We use a scoring function to evaluate the importance of tokens. We do not constrain the number of tokens per layer. Please refer to Algorithm 1 for more details. If the context length for a certain layer is 0, it means there is no prompt for this layer.

• Weakness 4: Prune rate $r_p$ is a hyperparameter. We show the result of using different pruning rates in Appendix A.3. The result suggests that different pruning rates do not significantly change the final mask, which indicates the robustness of the pruning strategy.

• Weakness 5: We added more text explanations for the algorithm at the bottom of page 5 and text descriptions in the Algorithm 1 (line 7, 8, 9 and 12).

• Weakness 6: We split the paragraphs in Introduction into multiple paragraphs. The first paragraph gives an overview of the PEFT methods and introduces prompting methods. The second paragraph elaborates on the two categories of prompting methods. The third paragraph talks about works related to continuous prompts. The fourth paragraph talks about the motivation why we propose our method. The fifth paragraph gives the whole picture of the Adapt method.

• Weakness 7: We are not sure which dynamic neural networks the reviewer is referring to. Can the reviewer give some related references?

We checked classic dynamic neural network references. One line of work focuses on dynamic neural architecture. The depth [1-2], width [3-4] and structure [5] can change. Adapt uses the pre-trained model and the model weights and structure are fixed during the training process. Another line of work focuses on dynamic parameters. The model parameters such as kernel shape [6], channels [7] and features [8] can change. Adapt changes the context lengths during the training process, which do not affect the model parameters or structures. Dynamic neural networks generally use differentiable parameters to adaptively change parameters. The parameters in Adapt are not determined in a differentiable way. It changes in a discrete way.

We thank the reviewer’s suggestion. Here is what we did to improve the presentation:

1. We changed “Adapt” to “Adapt (ours)” in Figure 1
2. We split the paragraphs in the Introduction into multiple paragraphs
3. We added one paragraph to explain more details in Algorithm 1 and text descriptions within Algorithm 1
4. We added the analysis of total context lengths in Appendix A.8

• Issue: We thank the reviewer’s suggestion. We changed “Adapt” to “Adapt (ours)” in Figure 1.

We thank the reviewer for the thoughtful feedback and for recognizing the strengths of our work. We are especially grateful for the positive assessment and the raised scores

The authors thank the reviewer for the comments. Here is one-to-one response:

• Weakness 1: We would like to start by clarifying that reference [1] only focuses on applying differentiable masks to model weights, which is different from our soft prompt learning setup. The differences between reference [1] and Adapt are:

1. [1] uses differentiable masks while Adapt does not use differentiable masks. The variation of masks in [1] depends on the loss function and threshold function. Our method changes masks by pruning process and scoring functions. Hence, using differentiable masks introduces additional trainable parameters.
2. [1] applies masks to the model parameters (e.g. MSHA layer parameters) while Adapt applies masks to soft prompts.
3. [1] uses the mask to zero out some model parameters (element-wise multiplication) while Adapt uses the mask to select valid context tokens and there will be no zero-out tokens. When applying [1] to deep prompts, the context length does not change. When applying Adapt, the context length does change.

Following the reviewer's suggestion, we did experiments using differentiable masks with the closest possible setup. Since deep prompting methods for vision-language models such as MaPLe [2] and VPT [3] do not insert the deep prompts in the same way as we do (comparison is shown in Figure 2 (a) and (c)), we tested the differentiable mask + traditional deep prompting denoted as “VPT + Differentiable Mask” (prompts are inserted for query, key and value and removed after self-attention) and the differentiable mask + our deep prompting method denoted as “Adapt + Differentiable Mask” (prompts are inserted for only key and value computation, no inserted prompts are removed after self-attention).

Differentiable masks do not apply a constraint on the total context length, so the prompt complexity can be different on different datasets. For a fair comparison, we include the performance of Adapt (adaptive $T_{target}$) that uses optimal $T_{target}$ determined by the validation dataset (details are described in Appendix A.9). In differentiable mask methods, the threshold alpha for the binary function I(M > alpha) is set to be 0.5. We do find a performance degradation using differentiable masks, especially using the traditional deep prompting method (a comparison between traditional deep prompting and our deep prompting methods is shown in Figure 2 (a) and (c)). The performance comparison is shown below:

We have added this new result and cited [1] in Appendix A.13.

• Question 1: We examine the performance of using $T_{target} = 256$ and $T_{target} = 512$. The new results are included and marked in blue color in Table 2 of the updated manuscript. The results empirically indicate that the upper bound for the average accuracy is 81.70%.

• Question 2: In Adapt, prompt depth and context length are automatically determined. Hence, they are not hyperparameters of our model. The hyperparameters used by Adapt is $T_{target}$, $r_p$ (pruning rate), and accumulation steps (number of steps to compute the accumulated scores). The effect of $T_{target}$ is reported in Table 2. The effect of $r_p$ is reported in Appendix A.3.

Please refer to line 13 in Algorithm 1 and Figure 8 in Appendix. The prompt depth and context length are automatically determined.

• Question 3: We add results using 1/2/4/8-shot training setting in Appendix A.10 using Adaptive $T_{target}$. The Adaptive $T_{target}$ is reported in Appendix A.9 and The few-shot learning result is shown in Appendix A.10.

[1] Zheng, Kecheng, et al. "Regularized mask tuning: Uncovering hidden knowledge in pre-trained vision-language models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Khattak, Muhammad Uzair, et al. "Maple: Multi-modal prompt learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Jia, Menglin, et al. "Visual prompt tuning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

Dear Reviewer qkws,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission8374

I thank the authors for their response.

Some concerns have been addressed. I appreciate this work, but I still believe that the core idea is similar to [1], these two methods both add Binary Mask to some parameters (e.g., model weights or prompt weights).

Additionally, the ADAPT does not show the advantages in few-shot learning tasks (1/2/4/8). As the number of available images decreases (from 16 shot to 1 shot), the performance advantage of this method becomes inferior to existing methods (e.g., MaPLe\MaPLe\LAMM). This indicates that the method is highly dependent on the amount of data (more than 8 shots) and is not very robust.

Regarding method innovation and robustness, I think there is still some room for improvement in this work. I will keep my original score of 3.

Thanks for your response.

I greatly appreciate the author's design in the subfield (i.e. using binary masks to solve how CLIP can be used for downstream task training, such as few shot) that utilizes binary masks effectively. We would like to clarify my viewpoint again. I am not claiming that ADAPT has low novelty due to the similar notion of "binary mask". But rather, in this subfield, the use of binary masks for trasferring CLIP to few-shot tasks has already been explored, and I think the core idea 'use binary mask for transferring CLIP to few-shot' is similar. The authors claim that 'this is the first work to prune prompts'. If the core idea is to 'use binary mask in prompts', prompt learning has a wide rang e of applications and should be validated for its effectiveness in various tasks, such as prompt learning in LLM.

Additionally, the ADAPT is highly dependent on the amount of data (more than 8 shots). As shown in Figure 1, the authors only show the results in 16 shot, but in 1/2/4/8 shot setting the performance has significantly decreased. In few-shot setting of CLIP, 1/2/4/8 shot settings are also important.

The initial version did not provide a detailed comparison with the binary mask scheme used in the CLIP field in related work and experiments. I find that the updated version provided these and can be improved by more detailed comparision and related works. After carefully reading the updated version, I will raise my score to 5 based on the Sparse Training of related work and comparision with differential mask from the authors. But I am still concerned about performance degradation with smaller shot.

The authors thank the reviewer for the feedback. Here is one-to-one response:

• Weakness 1: Originally, we set a fixed $T_{target}$ to ensure that on different datasets, Adapt has a similar model complexity and is fair to the baselines. The reviewer is correct that by making $T_{target}$ adaptive to each dataset, the performance can be further improved. To demonstrate this point, we consider the setting allowing the dataset-dependent $T_{target}$. The details are reported in Appendix A.9, the performance comparison is:

• Weakness 2: What the reviewer points out is the uniqueness of the Adapt method that introduces the heterogeneous context lengths. We analyzed the context lengths on different datasets. The results are added in Appendix A.8. We noticed that for datasets that have a larger degree of out-of-distribution, there will be more context tokens added to the image branch.

• Weakness 3: We added results using 1/2/4/8-shot. We use the validation dataset to determine the optimal $T_{target}$ named Adapt (Adaptive $T_{target}$). Details regarding Adapt (Adaptive $T_{target}$) are shown in Appendix A.9 and the result of 1/2/4/8/16-shot is shown in Appendix A.10.

• Weakness 4: We wanted to include UPT work for the comparison, but the code is not included in the repo [1]. We implemented UPT [2] on our own. Below is the performance we got:

We have added the comparison in Appendix A.12 and cited UPT work in the updated version.

[1] https://github.com/yuhangzang/UPT

[2] Zang, Yuhang, et al. "Unified vision and language prompt learning." arXiv preprint arXiv:2210.07225 (2022).

Dear Reviewer bqgw,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission8374

Thanks for the responses. The rebuttal shows that $T_{target}$ should be set to different values on different datasets. This suggests that automating the selection of $T_{target}$ would be a valuable improvement. Additionally, the performance of ADAPT in other training setting is lower than that of the state-of-the-art methods. Therefore, I maintain my initial score.

We thank the reviewer for the time and effort spent in evaluating our work. As the author-reviewer discussion phase is ending soon, we would like to kindly ask if the concern has been addressed with our latest response (https://openreview.net/forum?id=1L9vdc7BB5&noteId=PVCF08vTPX)?

The authors thank the reviewer for the feedback. Here is one-to-one response:

• Weakness 1: Some datasets do not have the best performance

We understand the reviewer’s concern and agree that this is an important issue to elaborate on. Despite the performance variances across datasets, our method achieves the best performance on 7 out of 11 datasets and ranks in the top 3 on 8 out of 11 of them. The second best method LAMM achieves the best performance on 0 out of 11 datasets and the top 3 performance on 7 out of 11 datasets. The third best method ProGrad achieves the best performance on 2 out of 11 datasets and the top 3 performance on 6 out of 11 datasets.

We force the same $T_{target}$ to ensure a similar number of trainable parameters across datasets. When allowing different $T_{ target}$ for different datasets, we can have a significant performance gain on some datasets such as Food101 as shown in Table 1 of the updated manuscript. The details regarding Adapt (adaptive $T_{target}$) are reported in Appendix A.9.

We also want to note that the “no-winner-takes-all” finding has been consistently observed in the study of continuous prompting methods for VLMs. When PLOT (ICLR 2023) [1] is published, there are two prompting method baselines. It achieves the best performance on 6 out of 11 datasets. When LAMM (AAAI 2024) [2] is published, there are two prompting method baselines. It achieves the best performance on 7 out of 11 datasets. Thus, none of the methods consistently rank within the top 3 on all the datasets

On the Pets dataset, different methods have similar performance. Even the zeroshot CLIP can achieve reasonably good performance. We would like to draw attention to the fact that our method has pronouncedly better performance on challenging datasets such as DTD, EuroSAT and Aircraft where the zeroshot CLIP has poor performance.

• Weakness 2: Heterogeneous prompt lengths could make the model harder to implement in practical scenarios

Our method uses $T_{target}$ as a hyperparameter to automatically determine the context lengths at different depths by pruning (please refer to Algorithm 1 line 13). The manually designed deep prompting method has a hyperparameter of prompting depth. Hence, different depths can have different context lengths (when the depth is equal to or smaller than d, the context length is t per layer; when the depth is larger than d, the context length is 0 per layer). Our method intentionally introduces heterogeneous context lengths and finds that it achieves better performance than manually designed homogeneous context lengths.

When we report the performance, it is averaged over 3 runs. We included the standard deviation in Appendix A.6.

Finally, to fully address the reviewer’s comment, could the reviewer elaborate on the difficulty in practical scenarios in terms of consistency and predictability?

• Weakness 3: There is no explicit mechanism to ensure the two branches are aligned

This is a great suggestion for an ablation study. In Adapt, we intentionally enable the different total context lengths in the text and image branches. Therefore, the setting of the same total context length in text and image branches is included in our search space of the current setting. Following the reviewer’s suggestion, we conducted experiments that ensured the same context lengths in the image and text branches. The modification we made was that instead of combining scores in text and image branches, we ranked scores in the text branch and scores in the image branch. In each pruning step, we pruned r_p tokens in the text branch and r_p tokens in the image branch. We noticed a pronounced performance degradation by applying constraints on the total context length of two branches to be the same (Adapt Constraint). The results comparison is:

The experiment indicates that two branches do not need to be aligned. Less constraint is beneficial to fully exploit the power of prompt tuning. Please refer to Appendix A.11 for more details.

• Question 1: more details on the scoring function

In our initial submission, we tested Snip, Gradient norm and l2-norm as scoring functions (shown in Eq. 6 and the ablation study “Score computation” in Sec. 4.3). We empirically found that Snip works better. Snip considers gradients and magnitudes of model parameters. Gradient norm only considers gradients. l2-norm only considers magnitudes of model parameters.

Dear Reviewer Tr3E,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission8374