IPO: Interpretable Prompt Optimization for Vision-Language Models

(1) Performance on Base vs. Novel Classes:

Previous prompt learning methods like CoOP and CoCoOP, which rely on gradient-based optimization, tend to overfit to base classes, resulting in performance loss on novel classes. Our IPO, however, uses LLMs to optimize prompts with a focus on learning more generalizable prompts across datasets. This results in better performance on novel classes compared to gradient-based methods. In response to Reviewers U899 and 4MWY, we have conducted additional experiments demonstrating that our method can achieve a balance in performance between novel and base classes by increasing the capacity of the LLM and LMM, which allows for better handling of longer context information.

(2) Comparison with Recent Methods:
We appreciate the reviewer’s suggestion. We have provided comparisons with LFA and PLOT in the tables below, using the same experimental setting. Our IPO method consistently outperforms both LFA and PLOT. Regarding DFA, we were unable to locate the corresponding paper. If possible, could the reviewer please provide the title or more details on DFA? We will include these comparisons with recent prompt-tuning methods (LFA, PLOT, DFA, etc.) in the revised manuscript.

Comparison with LFA across 11 datasets in 16-shot scenarios:

Comparison with PLOT on average accuracy across 11 datasets in 1-shot and 16-shot scenarios:

(3) Scalability for Large Datasets:

Our IPO method is primarily designed for few-shot scenarios. However, when dealing with large domain-specific datasets, the need to generate extensive image descriptions, which can lead to substantial computational costs due to the large text inputs required for LLMs. Currently, our model uses an input length of approximately 5,000 tokens. When scaled to larger datasets, the input length may increase to around 50,000 tokens. Using GPT-4 with an 8k context length, the cost for our current input size (5,000 tokens) is approximately 0.15 dollars per input (0.03 dollars per 1,000 tokens). For the expanded input size of 50,000 tokens, the cost would rise to approximately 3.00 dollars per input. If we were to use GPT-4 with a 32k context length, the cost for the 50,000-token input would be approximately 3.00 dollars for the first 32,000 tokens and an additional 1.08 dollars for the remaining 18,000 tokens, totaling approximately 4.08 dollars per input. Since our IPO method requires 100 iterations during training, the costs would multiply accordingly when scaled to large inputs.

A potential solution is to fine-tune an LMM to input training images directly, thus eliminating the need for additional description generation. We will clarify this limitation and potential solution in the revised manuscript. We believe the cost is justified, especially for domains like health, legal, and finance, where human interpretation of prompts is key.

(1) Line 150:
Fixed. Thank you.

(2) Table 6:
We will add citations for each method in Table 6 in the revised manuscript.

(3) Cross-Dataset Experimental Evaluation:
We conducted a cross-dataset experimental evaluation, following the traditional setting, and found that our IPO outperforms previous gradient-based prompt learning methods. The results are shown in the table below, and we will include this experiment in the revised manuscript.

(4) Computational Cost and Efficiency:
Since our model uses LLMs to optimize prompts, it doesn't involve gradient calculations and only requires API calls to the LLM and LMM for prompt optimization. Therefore, compared to directly using CLIP, our method does not incur additional memory overhead. Regarding training time, we use GPT-3.5 Turbo as our default optimizer, iterating 100 steps for each dataset to derive the final prompt (as noted in Line 199). The training speed is primarily dependent on the response time of GPT-3.5 Turbo in generating prompts. During testing, the generated text prompts are directly fed into the text encoder, resulting in no additional computational cost, and inference efficiency remains consistent with CLIP.

(5) In-Depth Analysis of IPO Components:
From a regularization perspective, our IPO leverages episodic memory to prevent overfitting by considering the performance of past prompts, ensuring the generation of more generalizable prompts. In (7) Impact of Prompt History Length, we found that the length of Prompt History affects the prompts generated by IPO, further proving that episodic memory effectively addresses overfitting. Additionally, the incorporation of image descriptions ties the prompts closely to the data, reducing variance in predictions, as demonstrated in Table 4 of our paper. Furthermore, using LLMs for non-gradient-based prompt generation introduces variability, further mitigating the risk of overfitting. We will include this analysis in the revised manuscript.

(6) Shorter Training Time:
Yes, compared to gradient descent approaches, IPO’s parameterless training process is shorter because it does not require gradient calculations or parameter updates.

(7) Impact of Prompt History Length:
We have evaluated the impact of different prompt history lengths on the final performance, as shown in the table below.

We found that without prompt history, model performance decreases due to the lack of contextual information for the LLM, making it difficult to converge. As the history length increases, performance gradually improves, reaching convergence at n=20. Although n=100 yields the best average performance, a longer history increases the length of LLM input, leading to higher API costs. Therefore, we selected n=20 for our IPO. We will include this experiment in the revised manuscript.

(1) Missing Important References:
We thank Reviewer 4MWY for bringing the CVPR 2024 paper by Liu et al. and the forthcoming ECCV 2024 paper by Mirza et al. to our attention. Both works are indeed relevant and will be discussed in the related work and experimental sections of our revised manuscript. While these papers contribute valuable insights to the field, they do not compromise our novelty claim.

Liu et al. propose a method that utilizes LLMs as black-box optimizers for vision-language models, iteratively refining prompts based on in-context examples. Their approach focuses on leveraging ChatGPT to improve prompt templates for visual classification tasks. In contrast, our IPO method differs in that we incorporate past prompt history as episodic memory within the design process. This allows our model to generate better prompts by considering the contextual information of previous successes and failures. Additionally, we use LMMs to generate image descriptions during the prompt design process, enabling the creation of more accurate, dataset-specific prompts. Our experimental results demonstrate that IPO outperforms Liu et al.'s method, with a 13.19% improvement in 1-shot base-to-novel generalization (74.29% vs. 61.1%).

Mirza et al. explore a different aspect of prompt optimization by focusing on zero-shot vision-language models. Their method does not utilize a specific scoring mechanism for prompt optimization, whereas our IPO method employs the training sample's loss and accuracy as a scoring mechanism, optimizing prompts for better performance in few-shot scenarios. While Mirza et al. make important contributions, our approach is distinct in its focus on few-shot learning and the use of LMMs to enhance prompt accuracy.

(2) Performance on Base Classes and One-Shot Classification (Table 5):

In Table 5, the performance of our proposed IPO method on base classes is indeed lower compared to some baselines. This is because IPO is designed to optimize prompts with a focus on generalization across both base and novel classes. While this approach enhances performance on novel classes, it can result in slightly reduced performance on base classes where the model might not fully exploit specific base class features to avoid overfitting.

However, this difference in performance tends to diminish when relying on higher capacity models, which are better equipped to handle the complexities of both base and novel classes. To further clarify, we evaluated IPO in a usual one-shot classification setting, where both base and novel classes were treated as base classes during training. The results, presented in the table below, show that IPO still outperforms previous methods in this setting, demonstrating its ability to generate more generalized and effective prompts across diverse datasets.

(3) Clarification of "H":
Following the common convention in the prompt learning literature CoOp [50], CoCoOp [51], and MaPLe [18], "H" refers to the harmonic mean. We will clarify this in the revised manuscript.

(4) 1-shot in Training:
Yes, the reviewer's understanding is correct. In the 1-shot setting, one sample per category in the base classes is provided during training, with no samples from novel classes used during training. This approach is consistent with the traditional base-to-novel setting [50, 51, 18].

(5) Typo on Line 165:
This is a typo; it should refer to six models instead of four. We will correct this in the revised manuscript.

(1) Impact of LMM:

We thank Reviewer U899 for the insightful suggestion. Indeed, when we replaced the 2.8B parameter MiniCPM-V-2 LMM with the higher capacity GPT-4o (estimated 500B~1T parameters), we observed a performance improvement for 10 out of 11 datasets. On average, the performance across 11 datasets improved: Base: +1.66%, Novel: +1.89%, and H: +1.77%.

(2) Impact of LLM:

We observed a similar positive impact when upgrading the LLM capacity. To demonstrate that using a stronger LLM, such as GPT-4, can generate more effective prompts for our model, we conducted further experiments utilizing both GPT-4 and GPT-4o. Specifically, when upgrading the LLM to GPT-4o and pairing it with the GPT-4o LMM, the overall H-score increased by 1.77% compared to the original results with GPT-3.5-turbo and MiniCPM-V-2. This improvement highlights the benefit of using larger models for enhancing task generalization.

These improvements are reflected in the quality of prompts generated by the upgraded models. For example, on the DTD dataset, the prompt generated by GPT-3.5 turbo changes from "Classify the intricate <CLASS> texture" to "Analyze and classify the detailed <CLASS> texture in this image, considering its unique patterns and variations." Similarly, on the ImageNet dataset, the prompt generated by GPT-3.5 turbo changes from "Take a high-quality photo of a <CLASS>" to "Capture a sharp, high-resolution photo of a <CLASS> with clear details and vibrant colors."

(3) Other Related Tasks:

In other prompt-based vision tasks, such as segmentation and detection, the design of the text prompt is crucial. Our method, being task-agnostic, can be easily embedded into any vision task to optimize the text prompt. For instance, in our experiments, we incorporated IPO into pre-trained semantic segmentation models [a] [b], where the original text prompt was "a photo of a [CLASS]." Using GPT-4o as the LLM and LMM, we crafted more effective text prompts specifically suited to the open-vocabulary semantic segmentation task, leading to enhanced performance and demonstrating the value of IPO in optimizing text prompts for this application. We intend to further investigate the use of IPO in other vision tasks in future work.

[a] Xu, et al. "A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model." ECCV 22

[b] Zhou, et al. "Zegclip: Towards adapting clip for zero-shot semantic segmentation." CVPR 23

(1) Comparison with Knowledge Bank-Based Prompt Learning Methods:

We sincerely thank the reviewer for pointing out these three interesting works. First, we would like to clarify the differences between our IPO and these methods. The works mentioned (L2P, AttriCLIP, DualPrompt) are based on visual prompt learning for continual learning, where they train a prompt bank during the training phase, and during testing, the appropriate prompt is retrieved from this prompt bank using the test sample. In contrast, our IPO primarily focuses on few-shot VLMs, where during training, LLMs are utilized as our prompt bank to generate dataset-specific prompts based on contextual information. During testing, predictions are made directly using the learned prompts without the need for additional retrieval.

Since our IPO optimizes text prompts using LLMs and cannot be directly applied to these three methods, we conducted a fair comparison by applying L2P within the Visual Prompt Tuning (VPT) [c] framework, which also learns prompts in the visual space and applies them to few-shot VLM tasks. VPT + L2P trains a prompt bank during training, and during testing, the test samples query the prompt bank to find the appropriate prompt. The table below shows the comparison between VPT + L2P and our method across 11 datasets in the 16-shot setting. We found that while L2P does improve VPT's performance, demonstrating L2P's effectiveness in VLM, our IPO still outperforms VPT + L2P. We will include this comparison in the revised manuscript to demonstrate that the effectiveness of IPO is not solely due to a memory retrieval mechanism.

[c] Jia, et al. "Visual prompt tuning." European Conference on Computer Vision 2022.

(2) Comparison with Other Prompt Learning Methods:

In Table 6, we compare our IPO with other methods in the 16-shot setting. To clarify, prompt learning includes both textual prompt tuning and prefix tuning. Prompt tuning methods like CoOP and CoCoOp treat the textual prompt as learnable parameters optimized using few-shot samples, while prefix tuning involves adding learnable tokens to the text encoder, vision encoder, or both. Examples include MaPLe, PromptSRC, and CoPrompt. Our method falls under prompt tuning, so we mainly compare against other prompt tuning methods. From Table 6, our IPO shows a 4.38% improvement in average performance compared to CoCoOp, while providing an interpretable prompt as well. We will provide a complete comparison table of the 16-shot performance across 11 datasets in the appendix of the revised manuscript.

(3) Performance on Large-Scale Generic Datasets:

IPO with GPT-3.5 Turbo, indeed, does not show an improvement on the large-scale ImageNet. This is because ImageNet has a large number of classes and samples, which results in longer LLM input when generating descriptions for each sample. GPT-3.5 Turbo has limited performance in handling long-text inputs. The table below shows the results on ImageNet when IPO uses GPT-4o, which has superior long-text understanding compared to GPT-3.5 Turbo. We found that IPO using GPT-4o leads to better performance improvements over other methods as well as a considerable improvement over IPO with GPT-3.5 Turbo.

Additionally, using the harmonic mean as the average metric is a common strategy in prompt learning for VLMs, as seen in works like CoOp, CoCoOp, MaPLe, and CoPrompt. We will clarify this in the revised manuscript.

(4) Clarification of Table 6:

Table 6 uses the same benchmarks as Table 5, with the 16-shot setting across 11 datasets. We will reference Table 6 in Line 283 of the revised manuscript and clarify the benchmarks used. Thank you.

I am happy with your response. However, some concerns are still not solved.

1. Please add these clarifications to the revised manuscript to make the manuscript clearer: In contrast, our IPO primarily focuses on few-shot VLMs, where during training, LLMs are utilized as our prompt bank to generate dataset-specific prompts based on contextual information. During testing, predictions are made directly using the learned prompts without the need for additional retrieval.

2. Please extend Table 6 to be as detailed as Table 5 because this setting is very important. (If you have space limits, please provide the result of large-scale datasets such as ImageNet)

3. I can not understand why ImageNet results in a longer LLM input when generating image descriptions for its large samples and categories. Is this because of a larger batch size? In my view, it is not necessary to input all the dataset categories when you processing one training sample. Therefore, I am not satisfied with this answer. Please make more clarification.

You are almost close to your 6 rates. Please hold on.