GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models

Below we report these results for LlaVA-OV on ImageNet and Imagenet-R:

• ImageNet:

• ImageNet-R:

These results show that our GLOV can show strong performance gains over the base model (LlaVA-OV) in this new sub-shot setting. For example, (for ImageNet) by only compromising 1.8% accuracy, our GLOV can find the optimal prompt (by using only 50 classes as compared to the full set of 1000) in less than 3 hours. Further, hinting that the discovered prompts can be generalized by having access to only a few classes. We believe that these experiments further enhance the practicability of our approach by reducing the time required for optimization by many folds.

We also point out that in LN 364-365, we provide the GPU memory required to run our experiments. Specifically, our prompt search for CLIP can run on a single NVIDIA 3090 (24 GB) GPU and for LlaVA a single A40 (48 GB) suffices. The total number of GPUs used for our experiments is reported in the appendix (LN 820-822). During test time, there is no extra overhead, as previously mentioned. All our evaluations and of all other methods, fit on the same GPUs mentioned above.

W4 - Comparisons with PEFT: In the main manuscript we report 1-shot results obtained with a parameter efficient fine-tuning (PEFT) based method -- CoOp [6]. These results are consistent with the 1-shot results reported in the literature [7] for the same CLIP backbone (ViT B/32). We ran their (CoOp [6]) official code-base with the settings listed in the paper. One reason for the low results of CoOp can be severe over-fitting due to the extremely low data regime.

As suggested by the reviewer, we also evaluate LORA by fine-tuning it on 1-shot data and find that when LORA is applied to all the matrices of the encoders, it results in severe overfitting and the results even degrade lower than the baseline model. We further evaluate by applying LORA fine-tuning by only applying it to the attention blocks and find that the results are better but still our method outperforms this fine-tuning variant as well.

These results suggest that in extremely low data regimes, few-shot methods might not fare well due to overfitting. On the other hand, our GLOV is parameter-update-free and only relies on suitable prompts for performance gains, thus, making it more effective for low data.

The results are listed as follows:

Q - Selection of few-shot data and variance of results: Consistent with the few-shot literature [6] we do not control the few-shot samples chosen and in our experiments sample the 1-shot training data randomly. As suggested by the reviewer, in the following table we report the mean and the variance over $5$ independent runs. The results indicate that the performance improvements are indeed statistically significant.

[1] Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs

[2] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

[3] Tree of Thoughts: Deliberate Problem Solving with Large Language Models

[4] Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations

[5] Complexity-Based Prompting for Multi-Step Reasoning

[6] Learning to Prompt for Vision-Language Models

[7] LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections

Thank you for the time and effort spent in reviewing our paper. In the following, we reply to all the concerns raised in the review.

W1 - Novelty: We would like to stress the fact that our GLOV only builds upon [1] but introduces many novel aspects as listed below:

• Iterative Refinement. Only the conceptual idea for meta-prompting is inspired by [1]. We restructure it to fit the needs of our task of iterative optimization by providing $top_k$ and $bottom_k$ (in-context) prompts with their effectiveness for generating new (more tailored) solutions for the downstream task. The original meta-prompt [1] does not iteratively refine the prompts. Our GLOV is centered around iterative refinement. The results show that this iterative refinement helps to improve the results over [1] by 2.5% (compared with GLOV - w/o guidance), when averaged over the 16 datasets. These results highlight the effectiveness of our modifications to the original meta prompt and the iterative refinement scheme.

• Novel Guidance Scheme. To further bias the output language structure strictly towards the task of interest we have proposed a novel embedding space guidance scheme. The results show that the embedding space guidance can improve over the vanilla GLOV and can further boost the performance w.r.t [1]. For example, by applying our guidance scheme we outperform GLOV (w/o guidance) by 2.6% and [1] by 5.1% when the results are averaged over the 16 datasets. These results further signify the importance of the novel embedding space guidance scheme.

• Downstream Tasks. The applicability of [1] is restricted to only the task of object classification since it was designed for that particular task. On the other hand, our GLOV has widespread utility. Along with evaluating the classical task of object recognition, we have also evaluated the VQA task in Table 3 of our submitted manuscript. Further, in this rebuttal, we also extend the downstream task to general Visual Question Answering and enhancing the safety of the models. The results for these tasks are provided as the response to the W2 of reviewer $qqDY$.

W2 - Impracticality: We would like to point out that finding effective prompts for downstream tasks is an active area of research [2, 3, 4, 5]. In our work, we take this burden off the humans to find the effective prompts for diverse downstream visual tasks and instead propose a generic optimization method that can be applied to find effective prompts for many downstream tasks, such as image classification, visual question answering, and enhancing the safety of models. We believe that the diverse nature of tasks that can be addressed by our method makes it extremely practical for real-world use cases.

Furthermore, we would also like to point out that our method is designed to be applied offline -- where the optimal prompts can be found once and can be used without incurring extra overhead during test time. Even the training is very cheap since it only requires 1-shot training data for refining the prompts w.r.t a fitness function.
We further request the reviewer to consider the newly evaluated (sub-shot) setting (that has fixed cost, independent of target dataset size) listed in response to the (next) W3, which further enhances the practical aspects of our approach.

W3 - Efficacy of the method and GPU memory: Thank you for the comment. Here, we would like to point out that our GLOV only discovers optimal prompts during an offline training phase. Once the prompt is discovered, there is no added overhead during the evaluation (at test time).

To further expand, for CLIP, at test time there is no added cost for GLOV other than ensembling of prompts -- which are only 3 -- discovered during the optimization procedure. Whereas, the original CLIP paper proposed, and the recommended way to use CLIP is, to ensemble $80$ prompts (for ImageNet), which is much more expensive. Our ensemble of only $3$ prompts also outperforms CLIP’s costly ensembling of $80$ prompts by $1.2\%$ for ImageNet and $1.7\%$ on average, over the 16 datasets. For LlaVa-type models, we only use a single best prompt because ensembling is not feasible for those models.

We would also like to thank the reviewer for raising this concern because it indirectly led us to decrease the training time by many folds as well, with negligible loss in accuracy, and further helped us to discover a unique property that is specific to only GLOV, in contrast to other few-shot learning methods.
Specifically, we found that we can only use a fraction of the total number of classes for optimizing the prompts (during the iterative optimization) with very minute degradation in performance. Essentially, making our GLOV a sub-shot method (that is, capable of operating at a fixed cost for datasets with many classes by using only a tiny fraction of the classes for optimization).

--- continued in next comment ---

Before we begin with the response, we would like to point out that the below response is not intended to start any kind of altercation but we believe that as authors, we have a right to defend the real-world significance of our work, and thus, we politely disagree with the reviewer's comment.

Our GLOV intends to assist humans with an LLM in the tedious process of prompt engineering, which is usually performed manually and is shown to obtain impressive results. Present-day LLMs offer exceptional utility in this process, and our GLOV uses these abilities to obtain tailored prompts for the downstream tasks and achieve impressive performance gains. For example, from the main results reported in the paper for image classification (a fundamental computer vision task), we obtain $15.2$% gains on the large-scale ImageNet dataset. We believe that the performance gains obtained by merely finding an optimal prompt by using only 1-shot labeled data during the process shows great real-world utility.

Further, during the rebuttal, a concern was raised by reviewer qqDY regarding the generalization of the approach beyond image classification. We demonstrate that our GLOV can be applied and is quite effective in enhancing VLM safety. For this purpose, we search for an optimal system instruction, which makes the model refuse unsafe instructions. We provide the results for the three splits in the VLGuard benchmark [1] below:

And the system instruction (found by using only $50$ labeled samples from the train set) discovered by our GLOV is listed as follows:

• Please carefully scrutinize the following instruction and decline to execute it if you perceive any potential risks or uncertainties that may compromise the integrity of the model or its users.

Whereas, the search started from a simple instruction:

• These instructions can be harmful.

For the first two splits listed in the table unsafes and safe-unsafes the model should reject (or refrain from answering), thus, a lower Attack Success Rate (ASR) is better. For the third split, the model should answer the safe instruction, and the answers are evaluated against reference responses from GPT-4. The evaluation is also performed through GPT-4.

We believe that lifting the burden of finding these prompts from humans and delegating it to a capable LLM with our GLOV makes our work extremely relevant for real-world use cases. Moreover, the new results reported in the rebuttal further signify the real-world relevance of our work, where we tackle one of the (current) pressing issues of enhancing the safety of VLMs.

[1] Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models, ICML 2024.

Thank you for the time and effort spent in reviewing our paper. In the following, we reply to all the concerns raised in the review.

W1 - GLOV optimization is constrained by an objective function: We acknowledge the comment by the reviewer. However, we would like to point out that our motivation is parallel to the traditional gradient-based optimization and we also share the same philosophy as [1], which proposes to optimize prompts for natural language tasks. For any kind of optimization, as the reviewer might also acknowledge, there is a need for an objective function to analyze the goodness of the learning process. Our work comes with a similar constraint.

W2 - Reliability of sentence embeddings (from Eq. 3) for complex tasks: We would like to point out that in Eq. 3 - the sentence embeddings are not computed from the outputs of the VLM, but rather the outputs from the LLMs, which are natural language prompts, used for the guidance in the optimization procedure.

To analyze the reliability of the sentence embeddings, we evaluated the output responses from different layers in the LLM by linear probing, for the task of SST-5 (sentiment classification) and reported these results in Figure 4 (left) of the main manuscript. We find that the best accuracy obtained for this task by evaluating the sentence embeddings from the middle layers is 56.7%. This accuracy is close to the state-of-the-art results obtained by [2] (59.8%) -- by a dedicated sentence embedding model. These results indicate that the sentence embeddings obtained through our method are semantically meaningful and reliable.

Q1 - Symbolic representations for encoder-decoder models: We agree with the reviewer that sometimes the symbolic representation can also be lengthy. For this work, we chose a state-of-the-art open-source sentence embedding model from HuggingFace [3] which has been widely used by the community and is known to be reliable for extracting meaningful sentence embeddings even from long text. Furthermore, this type of evaluation (for encoder-decoder models) has also been extensively used in prior works [4, 5, 6], that also use an embedding model. The strong capabilities of these models can help them to extract meaningful semantics even from long text.

Q2 - Choice of top-k and bottom-k: The motivation behind the current choice of the $top_k$ and $bottom_k$ in-context examples is that we intend to provide contrasting examples to the LLM from the opposite end of the spectrum (of goodness and badness) so that the LLM can make sense of what are the type of responses preferred by the downstream VLM. We have added this motivation in the method section of the updated manuscript.

We also want to thank the reviewer for suggesting dynamic thresholding for the choice of $top_k$ and $bottom_k$, which can essentially make the learning algorithm more robust if we mix it with the current strategy. However, currently, we leave it as an exciting future work direction.

Q3 - Dynamic modification of rank interval for P+ and P-: Thank you again for the suggestion and this can definitely be one of the future directions to improve the current algorithm.

For the current optimization algorithm: In our initial experiments we tried different ways to choose $P_+$ and $P_-$, such as selecting the best and worst prompts as negative and positive, however, that resulted in unstable optimization. That could be because of the larger values in the guidance vector. This is the reason, we chose to select points closer to each other (best and second best). The motivation (also present in LN 299-304) is that we compute a form of a gradient-like differential between averages of token hidden states, intuitively trying to identify a characteristic of task-specific improvement. Thus, the intuition behind computing the differential between the best and the second best (in terms of fitness) is to make it between points closest to the maximal value of the objective -- which is a common mathematical intuition.

[1] Large Language Models as Optimizers

[2] An Algorithm for Routing Vectors in Sequences

[3] https://huggingface.co/sentence-transformers/all-mpnet-base-v2

[4] Vocabulary-free Image Classification

[5] Democratizing Fine-grained Visual Recognition with Large Language Models

[6] RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition

Thank you for the time and effort spent in reviewing our paper. In the following, we reply to all the concerns raised in the review.

W1 - Lack of comparisons: Thank you for the suggestion. In the following, we provide comparisons with three methods on the ImageNet dataset while using the CLIP ViT-B/32 backbone.

As we can see GLOV outperforms TIP, and performs on par with AWT and Sus-X. However, there are important distinctions between AWT, Sus-X and GLOV in terms of the task definition (operating assumptions) that make them (AWT and Sus-X) not directly comparable to GLOV. In particular, these methods synthesize category-specific prompts (as opposed to GLOV which optimizes for shared prompts for all categories of the same task). Category-specific prompts are naturally more expensive both in terms of optimization as well as inference time. Additionally, as we discuss more extensively in a new experiment performed due to the reviewer's suggestion below, GLOV can successfully operate (almost without losing performance) even under significantly more challenging (and more practical) conditions of knowledge of only a small fraction of the downstream task classes. This is impossible for the other (TIP, Sus-X, AWT) methods that always require the a-priori knowledge of all categories for their operation. Moreover, Sus-X also strongly relies on an external diffusion model making it unfair to compare to GLOV. Finally, TIP, Sus-X, and AWT are only designed to work with encoder-only models (like CLIP) and cannot be directly applied to encoder-decoder models (like LlaVA-OV) as opposed to GLOV.

More Specifically:

• Sus-X and AWT require per-category (multiple) descriptions that are generated from an LLM (i.e., GPT). For large-scale datasets like ImageNet, containing 1000 categories, this can become prohibitively expensive. In contrast, our method does not require generating category-level descriptions but only finds optimal prompts (i.e., 3) that are generalizable for the entire dataset.

• Sus-X also distills knowledge from an external model – requiring generating a support set (of images for all categories in the dataset) through a stable diffusion model, which is an external model and further enhances the cost of using their method.

• All these methods are only suitable for object classification through dual-encoder models (e.g., CLIP) because they all require an output probability distribution over the entire class space to work, which cannot be obtained by the generative (encoder-decoder models like LlaVA). In contrast, our method extends beyond dual-encoder models and is also applicable to other open-ended visual question-answering tasks (in addition to image classification reported in the submitted manuscript), as well as also applicable for enhancing safety of encoder-decoder models. These results are discussed in the response to the weakness 2 below.

W2 - Generalization of GLOV to other tasks: In the submitted manuscript, we evaluated our GLOV for the task of VQA in the context of image classification on the FOCI benchmark. Here, in response to the reviewer's comment, we further extend the evaluation of GLOV to $3$ additional tasks.

The first two tasks are: ChartQA (VQA for charts) and GQA (compositional questions created from image scene graphs). Our GLOV can provide an improvement of 1.0% (on average) for these tasks, where it improves the LlaVA model from 79.44% --> 80.21 for ChartQA and 61.13% --> 62.21% for the GQA dataset. These results highlight the applicability of GLOV to general VQA tasks, different from the image classification. For both these datasets, GLOV is tasked with finding an instruction, which is prepended to the actual question. These prompts are added to the appendix of the updated manuscript.

Furthermore, we demonstrate that our GLOV can be applied and is quite effective in enhancing VLM safety. For this purpose, we search for an optimal system instruction, which makes the model refuse unsafe instructions. We provide the results for the three splits in the VLGuard benchmark [4] below:

---- continued in next comment ----

Dear Reviewer qqDY,

Thank you for the acknowledgment of our rebuttal. Here we provide the answer and discussion to the two questions:

Q1. For TIP and AWT - we used 1-shot data from the train set of ImageNet, which is consistent with the setting we follow in our work. We use the ViT-B/32 architecture for the main results in our paper (Table 1). TIP tests ViT-B/32 architecture in their paper but does not provide individual results for the datasets, whereas, AWT does not test with ViT-B/32 architecture. To obtain the results with the same backbone we run their official codebase, using the setting mentioned in their paper. For few-shot results with AWT, we used the optimal multi-modal adapter as listed in their appendix (B3), while for TIP we followed all settings from their code base. On the other hand, for SuS-X, the authors provide the results with different backbones in Table 20 (page 27) of the appendix of their Arxiv submission (https://arxiv.org/pdf/2211.16198) and we report the ViT-B/32 results from there.

Q2. During the rebuttal, we have provided results for $3$ additional tasks: ChartQA, GQA, and (MMLM Safety) VLGuard. The results show that the proposed method can generalize beyond image classification to other VQA tasks and also a very relevant task of making the VLM’s responses safe.

While we acknowledge the observation of the reviewer regarding ‘tailored’ prompts for ChartQA and GQA - we would like to point out that the motivation of our GLOV is to optimize for task-specific prompts to be applied to instances of a particular task. On the other hand, general benchmarks like SEED and MME might require optimization for instance specific prompts, which is left as an exciting future work direction not touched upon in our current work.

We would further like to emphasize, that the optimization of task-specific prompts can be extremely helpful in achieving specific goals, e.g., improving VLM safety (as we show in the rebuttal).

We will add these discussion points to the updated manuscript. Furthermore, we would also be happy to discuss if there are more open points from your end. Thank you again for your time!

Thank you for the time and effort spent in reviewing our paper. In the following, we reply to all the concerns raised in the review.

W1 - Feature level guidance poses white-box LLM constraint: We acknowledge the concern of the reviewer and indeed the guidance scheme can only be applied when the activations of the LLM are accessible. However, we would like to point out that in our paper we proposed two variants of our approach, i.e., with and without guidance. Our experiments show that our white-box approach also fares well compared to baselines. For example, our GLOV (without guidance) achieves 2.5% and 19.0% average improvements (over 16 datasets) for the base CLIP and LLaVA One Vision models. Furthermore, our GLOV (without guidance) also outperforms the other white-box approach LLM-OPT by 1.2% (averaged over 16 datasets) for the CLIP models. These results highlight that our prompt optimization method can indeed be helpful in obtaining strong performance gains even when the activations of the LLM are not accessible.

W2 - Sensitivity to LLM choice is not clear: Thank you for pointing it out. In our paper, we employed the Llama-3.0 (7B) model. At the time of performing the experiments, Llama-3.0 was one of the state-of-the-art open LLMs. Further, we chose the smallest variant (7B) to keep the cost of the optimization low. To respond to the reviewer's concern, we further evaluated our GLOV with the Llama-3.1 (70B) variant, which is considered one of the more capable LLMs in terms of instruction-following abilities. We find that the accuracy improves by employing a stronger (and larger) LLM. Below, we list the comparison of our GLOV with Llama-3.0 (7B) and Llama-3.1 (70B) for the CLIP ViT-B/32 model. For ensuring reproducibility, the prompts produced by Llama-3.1 (70B) are added to the appendix for inspection, in the updated version of our manuscript.

We would also like to thank the reviewer for this suggestion since it helps showing that our method scales well with LLM size increase.

Q1 - Clarity of Algorithm 1 and Choice of Prompts: Thank you for pointing this out. We made a minor mistake when listing the algorithm in the submitted manuscript. $K$ is indeed an integer, representing the number of prompts generated at each iteration. In lines 9 and 12 - the list of prompts should be referenced to find the $positive$ and $negative$ prompts. Similarly, in lines 28 and 31, $K$ should be replaced with the list containing the $NewPrompts$ generated at a certain iteration. We have corrected this minor mistake in the updated algorithm in the appendix.

We select the best prompts found (w.r.t the 1-shot train set accuracy) at a single iteration (step) during the optimization and not through all steps. We also list this in the LN 360-361 in the main manuscript, where we list the implementation details.

Q2 - Sensitivity to hyper-parameters: For choosing the optimal layer $l$ for guidance we ran a sweep over different layers in the Llama model, only for the ImageNet dataset, and found that guidance on layer $17$ performs the best for the downstream recognition task. We also evaluated the sentence embeddings for the sentiment classification task (SST-5) through linear probing of different layers and again found that the middle layers in the Llama model performed best. These results were included in the original submitted manuscript: plotted in Figure 4 and discussed in the ablation section.

To clarify, $alpha$ is not actually a hyperparameter but a parameter optimized by GLOV automatically on the (1-shot) respective training set for each task. The optimization is done by an alpha sweep on the 1-shot training set used for GLOV optimization. Thanks to the reviewer’s suggestion, we have done some analysis on the $alpha$ automatically found by GLOV. For encoder-only models, GLOV found: $alpha = 1.0$ to be beneficial for the fine-grained classification datasets (e.g., Stanford Cars, Oxford Flowers); $alpha = 0.75$ to benefit datasets consisting of natural images (e.g., ImageNet); and $alpha = 0.25$ to be optimal for the out-of-distribution ImageNet variants like ImageNetA. For the encoder-decoder models (i.e., LlaVA-OV), GLOV found $alpha=1$ to be optimal for all datasets.

Dear Reviewer h9hv,

Thank you again for the time and effort spent on your thorough review of our paper. Since the author-reviewer discussion deadline is fast approaching, we kindly ask for feedback on our responses. We would be happy to discuss more if there are still some open questions.

Best Regards,

Authors

Thanks for the responses from the authors as well as the other reviewers. The authors' comments resolved most of my concerns. I find the results on Llama-3.1-70b especially inspiring as it demonstrates the scalability of the proposed method. I would encourage that results on all datasets with Llama-3.1-70b be included in the final draft to further show the robustness of the performance improvement, if time permits.

Although I agree with other reviewers on some practical limitations of the work (e.g., time consuming), I would still acknowledge some value of this work given that the relevant research on LLM for optimization are at a fairly preliminary state.

With these considerations, I will raised my score to be positive.