ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models

We sincerely thank the reviewer for finding our work practical and effective, and giving us constructive feedback to improve further. While we respond to the reviewer’s specific comments as below, we would be keen to engage in any further discussion.

Ablation for different combinations of modules

Although the paper performs ablation studies on individual modules such as low-rank approximation with a diagonal matrix and feature sharing, it lacks ablation experiments on different combinations of these modules. Without evaluating different combinations, it is challenging to fully understand the synergistic effects and the relative contributions of each module to the overall performance.

Thank you for your suggestion. We have evaluated all possible combinations of {diagonal matrix, feature sharing (FS), intrinsic-dimensional clipping}. The results are provided below.

First, we observe that using all the proposed modules together results in significantly better performance compared to using individual modules or pairs of modules. This demonstrates that each component works harmoniously to contribute to the generation of effective results.

Additionally, from the transitions 1→6, 4→7 and 5→7, we find that combining the low-rank approximation with diagonal matrix with intrinsic dimensional clipping yields more pronounced performance improvements (+4%, +4.5%, +3.6%) compared to other combinations.

These findings suggest that while each component is effective on its own, their combination creates a complementary synergy that maximizes overall performance. In future work, we plan to conduct an in-depth analysis to uncover the underlying mechanisms behind this synergy. This will provide deeper insights into its practical utility, paving the way for its application to a broader range of tasks.

Effectiveness of diagonal matrix?

The paper lacks an ablation study to isolate the effect of low-rank approximation alone, making it unclear if improvements are mainly due to the diagonal matrix. This analysis would clarify the diagonal matrix's contribution.

We kindly remark that the ablation analysis the reviewer requests is already provided in Table 6 in Appendix. This results clearly shows that adding a simple diagonal matrix (# of parameters < 10) enhances the performance of low-rank approximation, improving the average performance by +1.8%.

Citation in Figure 1

The caption for Figure 1 should include citations for the baseline methods (BAR, BlackVIP, BPT-VLM) to provide appropriate references and context for these comparisons. This would enhance clarity for readers unfamiliar with these specific methods.

Thank you for your suggestion. We will reflect it to our final version.

We sincerely appreciate the reviewer’s recognition of our work as “well-suited” and “user-friendly” for practical applications. We are also deeply grateful for the reviewer’s suggestion on additional ablations, which we believe has significantly improved our manuscript. We hope our response has adequately addressed your concerns, and yet, please let us know if there are any remaining issues. We would be eager to address them further.

We sincerely appreciate the reviewer’s recognition and the constructive feedback. We have addressed the reviewer’s specific comments below, while we remain open to any additional suggestions.

Other ZO Optimization methods?

Comparisons with additional baseline methods in zeroth-order optimization could strengthen claims of superiority.

We kindly request the reviewer to clarify whether it is other BBPT methods (that leverage ZOO) or literally other zeroth-order methods (e.g., ZO-SGD, SPSA-GC [1]) that are being questioned. If it is the former, to the best of our knowledge, there are no additional BBPT methods leveraging ZOO beyond those already considered in our work. If it is the latter which involves comparisons between other ZOO algorithms and their clipped versions, such as SPSA-GC and SPSA-GC with clipping, please let us know. We will address this through additional experiments.

Complex datasets?

Its results on ImageNet in Table 1 are comparatively modest, suggesting limitations in scalability to complex datasets. An in-depth analysis of ZIP's performance on larger, diverse datasets would clarify its robustness and potential for broader application.

Thank you for raising this concern. We would like to point out that a 5,000 API query setting may not be sufficient to fully train a complex dataset such as ImageNet. For comparison, prior work [1] used significantly more queries, with 625,000 API queries employed for ImageNet training. To provide a more thorough evaluation of the performance of ZIP, we conduct additional experiments with 20,000 API queries. The results are summarized in the table below.

These findings reveal several key points. With a 5,000 API query budget, ZIP achieves an ImageNet accuracy of 66.0%, comparable to strong baselines such as Manual Prompt (66.7%) and BlackVIP (65.5%). Increasing the query budget to 20,000 further improves the accuracy of ZIP to 67.2%, outperforming those baselines by margins of 0.5% and 0.9%, respectively.

Moreover, compared to prior work that leveraged significantly more API queries (e.g., BlackVIP: 67.1% on ImageNet), ZIP delivers outstanding performance using only 20,000 API queries. This demonstrates strong scalability and effectiveness, even on challenging datasets like ImageNet.

If you still have any remaining concerns regarding this issue, we would sincerely appreciate it if you could recommend a complex dataset to evaluate ZIP. We will gladly consider it as part of our future work and look forward to sharing the results based on your suggestion.

[1] Oh, Changdae, et al. “BlackVIP: Black-box visual prompting for robust transfer learning.”, CVPR, 2023.

We sincerely thank the reviewer for acknowledging the contributions of our work and offering constructive feedback. We address the reviewer’s specific comments below and we would be keen to engage in any further discussion.

Soft prompts not directly applicable?

Authors motivate fine-tuning black-box models with the use case of improving proprietary LLMs (e.g. GPT-4, Gemini) which are only accessible through API. However, this interface only accepts text and images as input, not soft prompts or embeddings, so the proposed method would not be directly applicable to API-based models.

Thank you for this insightful comment. We acknowledge that our soft prompt-based approach cannot be directly applied to the scenario described by the reviewer. Our study builds on the Language-model-as-a-Service (LMaaS) scenario [1], widely recognized in prior works as a plausible framework for black-box prompt tuning [2, 3]. While GPT-4 or Gemini do not currently support this specific scenario, we believe that exploring the LMaaS approach is essential for enabling more flexible and efficient fine-tuning of black-box models via APIs.

Apply to other model families?

It should be evaluated on other model families such as multimodal LLMs for robustness and generality.

Thank you for the suggestion. It would be interesting to see how ZIP evaluates on multimodal foundation models such as LLaVA to demonstrate its robustness and generality. Unfortunately, due to resource and time limitations during the rebuttal period, we were unable to conduct such experiments.

Instead, we provide additional results on SigLIP [4], another vision-language model distinct from CLIP, in the table below. While ZIP does not perform the best for all datasets, it achieves significantly higher accuracy on several datasets such as DTD, SVHN, EuroSAT, Resisc45, CLEVR, and UCF101, and records the highest on average, demonstrating its robustness and generality of our method compared to existing other BBPT methods.

Need validation accuracy

Figures 2, 4, 6 and 7a should report validation accuracy since there could be overfitting.

Thank you for pointing this out. To address concerns about overfitting, we now include validation accuracy for all cases in Figure 9, 10, 11, and 12 in Appendix A. The results show that training accuracy and validation accuracy generally exhibit similar trends, with no significant signs of overfitting observed.

Clarify in Introduction

It is not until the background section that I understood what zeroth-order intrinsic-dimensional prompt-tuning means. I suggest to improve the introduction to make it clearer from early on.

We appreciate this feedback. We will make sure to revise the paper and clarify what “zeroth-order intrinsic-dimensional prompt-tuning” means earlier in Introduction.

[1] Sun, Tianxiang, et al. “Black-box tuning for language model as-a-service.” ICML, 2022.
[2] Yu, Lang, et al. “Black-box prompt tuning for vision-language model as-a-service.” IJCAI, 2023.
[3] Song, Jiang-Long, et al. “Competition solution for prompt tuning using pretrained language model.” arXiv preprint arXiv:2212.06369 (2022).
[4] Zhai, Xiaohua, et al. “Sigmoid loss for language image pre-training.” ICCV, 2023.

We sincerely thank the reviewer for acknowledging the clear motivation of our method and providing constructive feedback. We address the reviewer’s specific comments below and welcome any further discussion.

Novelty?

I'm not familiar with this research field, i.e. black box prompt tuning. Therefore, it's hard for me to accurately judge the novelty of the proposed method compared with existing works.

Thank you for the comment. We will first outline the problem context and then explain our proposed solutions.

Black-box models that are accessible only through APIs can be trained using black-box prompt tuning (BBPT). However, users are typically constrained by limited API budgets, making query efficiency a critical factor for the practicality of BBPT. Despite this, prior works have not adequately addressed this issue. For instance, BlackVIP and BPTVLM often require tens of thousands of queries, which severely limits their practicality. To address this, we are the first to explicitly highlight this problem and propose several techniques to significantly improve query efficiency.

Specifically, we introduce two key technical contributions:

• To address the challenge of dimensionality dependency (i.e., the number of queries scales with the dimensionality of the problem), we propose a novel low-rank representation. This approach reduces the dimensionality while effectively mitigating the loss of expressive power through feature sharing.
• High variance in zeroth-order information can significantly degrade query efficiency. To tackle this, we propose a threshold-free gradient clipping method, termed “intrinsic dimensional clipping”. Inspired by prior studies on clipping thresholds [1, 2, 3], we set the clipping threshold to $\sqrt{d}$, which corresponds to the standard deviation of the zeroth-order gradient, where $d$ is the dimensionality of the problem. This approach not only reduces the variance of zeroth-order information but also achieves near-optimal performance without requiring manual tuning (See Figure 7b and 20 in revisioned paper).

[1] Zhang, Bohang, et al. “Improved analysis of clipping algorithms for non-convex optimization.” NeurIPS, 2020.
[2] Zhang, Jingzhao, et al. “Why are adaptive methods good for attention models?” NeurIPS, 2020.
[3] Zhang, Jingzhao, et al. “Why gradient clipping accelerates training: A theoretical justification for adaptivity.” arXiv preprint arXiv:1905.11881 (2019).

Dear reviewer BGp2

We sincerely appreciate your valuable feedback and constructive comments. Given that some time has passed since we shared our response, we kindly ask whether it has adequately addressed your concerns. Please let us know if there is anything else we need to address. We would be happy to discuss further. Otherwise, if you find our response reasonably satisfactory, we would greatly appreciate it if you could consider re-evaluating your initial rating.

Best wishes,
The authors.

Dear reviewer BGp2

As the discussion period is closing soon, we would like to see if our response has sufficiently addressed your concerns. Should there be any remaining issues, please do not hesitate to let us know. If you find our responses satisfactory, we would be sincerely grateful if you could consider revisiting your initial rating.

Best wishes,
The authors.