Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

We thank the reviewer for their valuable comments. In the following, we provide a response to the questions raised in the review:

Additional MANTIS results. As suggested by the reviewer, we present some additional results of our method using the MANTIS-LLama3 model on OKVQA, Vizwiz, Flower, and CUB:

We will include these results in the final draft of the paper.

Additional Dataset results. As suggested by the reviewer, we present additional results of our method on the following benchmarks using VILA-1.5:

Leveraging MTV for Single-Image LMMs. We thank the author for the suggestion to leverage MTV for single-image LMMs. This question is especially interesting as single-image pretraining causes the underlying model to struggle with handling multi-image inputs [1].

We apply MTV to LLaVA-1.5-13B on VizWiz. The model has a zero-shot accuracy of 54.5% and an accuracy of 62.1% with MTV. This result is encouraging as it indicates MTV’s ability to enable multi-image, few-shot, and many-shot multimodal reasoning for models pretrained on exclusively single-image data.

We hope the above points have clarified and addressed your concerns. We would be happy to provide any further clarifications as requested.

References:

[1] Doveh, S., Perek, S., Mirza, M.J., Alfassy, A., Arbelle, A., Ullman, S., & Karlinsky, L. (2024). Towards Multimodal In-Context Learning for Vision & Language Models. ArXiv, abs/2403.12736.

Please note that our checklist is provided in our Supplementary pdf, and thus, the PCs have decided not to desk-reject this paper. As this has been universally applied to all submissions, our submission is thus in accordance with guidelines, and fairness should not be adversely affected.

We thank the reviewer for the insightful comments and respond to all points in the following, incorporating all clarifications and additional results into the final paper:

MTVs for Text-only domain. Using LLaMA3, we evaluate MTV on text-only tasks. The two tasks are English-Spanish translation and Antonym generation, as done in [1]:

We find that MTV is also effective in the text-only domain.

MTV’s unique fit for multimodal tasks. As suggested by the reviewer, we provide the token count per image for each model to highlight the cost of image inputs:

A maximum of 8192/64 = 128 images can be encoded assuming no language inputs, already lower than the number in base MTV (400 shots).

Considering language tokens, we present the percentage of the input that is text and image on VILA 1.5 for datasets we evaluated on:

This shows that images quickly consume many of the input tokens for multimodal tasks. The data presented thus demonstrates the value of MTV, especially in the multimodal domain.

Motivation for encoding shots in the activation space. We highlight our paper’s motivation in addressing the context length token limitation of LMMs by encoding ICL shots in the activation space. An additional limiting factor in the token space is the physical constraints of memory and runtime, which we ablated in Section 5.6 of the paper. For example, 25-shot ICL is actually the maximum number of vanilla ICL shots that can be run on a single 48GB A6000 GPU for Qwen-VL. Regardless, following the reviewer’s suggestion, we demonstrate the degradation of increasing numbers of multimodal token-space ICL shots (VizWiz-QwenVL):

Compared to the results shown in Figure 2 of our paper, vanilla ICL improvement degrades at lower accuracy than MTV.

REINFORCE-based attention-head extraction. We perform an experiment replacing mean activations of ALL heads for VizWiz. Interestingly, QwenVL achieves 0% accuracy on Vizwiz in this setting. We assume this is due to the query contexts being completely overwritten by the mean activations.

Past work has shown that some subset of the heads include important ICL information and showed several ways to extract this information [1,2,3]. In our work, we select this subset of attention heads using a REINFORCE-based method to optimize a distribution over a set of attention heads as opposed to one-at-a-time like CMA [1], leading to superior performance (Section 5.5).

MTV Shot Quality. We assess the connection between textual and activation-space shot quality by comparing MTV using random selection with MTV using high-quality shots selected with the Facility Location algorithm [5]. We apply MTV to QwenVL and use the Qwen GTE embedding model to obtain embeddings for the Facility Location algorithm:

We find that choosing higher-quality examples drastically improves the performance of MTV, indicating that shot quality in the textual and activation spaces is closely connected.

MTV interpretability. We appreciate the reviewer's note on interpretability. We posit an alternative perspective that MTV shots are no less interpretable but are simply the same set of examples represented as a set of activations. Furthermore, task vector methods like MTV give additional insight into which attention heads are relevant for an ICL task [1,4].

Cost of prompt iteration. Please note that we provide an analysis of the cost of prompt iteration in Section 5.6 of the paper. We show that MTV is more runtime efficient than 4-shot ICL and more memory efficient than 16-shot ICL.

MTV with noisy exemplars. We compare the robustness of MTV with vanilla ICL as suggested. For QwenVL on VizWiz and OKVQA, we replace 1 of the 4 examples in each iteration of 4-shot ICL and 4-shot-100-iteration MTV with an example from the opposite dataset. We report both accuracy and degradation.

The results suggest that MTV and vanilla ICL have similar robustness to noisy examples.

Document datasets. We evaluated MTV on document datasets ChartQA and TextVQA with VILA-1.5:

These results indicate MTV’s capabilities on document-based tasks.

Thank you. We hope this addresses all concerns, and we are happy to clarify anything further.

References:

[1] Todd et. al. FV

[2] Hendel et. al. "In-Context Learning Creates Task Vectors."

[3] Olsson et. al. "In-context Learning and Induction Heads."

[4] Hojel. et. al. VTV.

[5] Schreiber et. al. Apricot.

We thank the reviewer for the insightful comments and respond to all points in the following, incorporating all suggested changes into the paper:

Comparison with “Many-Shot In-Context Learning in Multimodal Foundation Models”. We thank the reviewer for sharing this concurrent work with us and will add the reference to our final version. While the topic of many-shot in-context learning is a shared feature with our work, one should note that the suggested paper focuses exclusively on closed, proprietary models such as GPT-4o and Gemini-1.5 for image classification. Our work focuses on enabling many-shot, in-context learning capabilities for open-source, interleaved models with limited context lengths. We also evaluate on VQA benchmarks, not just image classification tasks.

Tables 1 and 2 format clarification. We clarify the exact format and number of shots for each experiment. In Table 1, the prompt for zero-shot is formatted as simply <Query> (the question to be answered). For 4-shot, the prompt consists of four randomly-selected concatenated examples of multimodal question-answer pairs followed by the query: <Q1> <A1> <Q2> <A2> <Q3> <A3> <Q4> <A4> <Query>. The 8-shot case is identical except with 8 exemplars instead of 4.

Finally, we use a setting of MTV with 400 examples (4 shots per 100 iterations) for calculating mean activations and 100 examples (zero-shot per 100 iterations) for extracting the attention-head locations.

For Table 2, the 2-way-1-shot formulation is used for all experiments. The prompt structure includes one positive example, one negative example, and the query image: <image_1> <class_label_1> <image_2> <class_label_2> <query_image>, where the task is to classify the <query_image> correctly as one of the given classes. The baseline consists of just a single one of these prompts, while the MTV case calculates mean activations from 100 iterations of one 2-way-1-shot example. The exact prompt we use is provided in Section C.3 of the supplementary material.

Function Vectors (FV) and Visual Task Vectors (VTV) methods explained. We include a more detailed description of Function Vectors (FV) [1] and Visual Task Vectors (VTV) [2]:

FV are a type of text-only task vector that has the following key differences from our method: Function Vectors uses a fixed number of shots without exploring the variation of both the number of shots or iterations for many-shot ICL. FV utilizes Causal Mediation Analysis [3] to extract the attention head locations. At a high level, this method selects attention heads by calculating the causal impact of each attention head (i.e. how much the LLM’s probability of outputting a correct answer increases given a set of corrupted, incorrect, ICL examples). The layer to replace activations is chosen via a simple linear search. This is in contrast to MTV which selects attention heads by learning a distribution over the attention heads across all layers.

VTV is a type of image-only task vector that has the following key difference from our method: VTV is a visual prompting method for vision-transformer models [2] VTV also uses a fixed number of single-shot examples without exploring the variation of shots and iterations to enable many-shot ICL. In contrast to VTV, MTV uses a token-level loss to align with the usage of an LMM for a multimodal task. VTV uses a single set of one-shot examples for both mean activation calculation and attention head extraction. In contrast, MTV decouples this process by first calculating the mean activations and then using a separate set of examples that are specifically aligned with the format of the downstream task.

We find that our method’s distinct properties—increase in shots and iterations, token-level loss, and the decoupling of the activation aggregation and attention-head extraction—are key factors that enable MTV to improve over FV and VTV.

MTV with more shots and iterations. Following the reviewer’s suggestion, we present results on MTV with higher shots (20, 25) and iteration counts (200, 400) for Qwen on Vizwiz. 25 shots is the highest number allowable by the memory constraints (48 GB) of a single A6000 GPU. We present the results here:

The purpose of our experiment shown in Figure 2 is to show the ability of MTV to scale, not necessarily to find the optimal shot-iteration setting. This exact value will differ based on the dataset and model. Nevertheless, the above results demonstrate the ability of MTV to scale even beyond 16 shots per iteration.

Measuring the impact of permutation. As the reviewer suggested, we explore the effect of permutation in more depth by evaluating Qwen-VL on VizWiz using the following (shot, iteration) counts for calculating mean activations: (4,100); (4,200); (8,100); (8,200). We evaluate on 5 seeds for each setting as well as 4-shot and 8-shot ICL as a baseline. In the following table, we present the mean and standard deviation for each run:

MTV:

Few-shot

MTV for 4 and 8 shots for different iterations is always more stable with respect to example permutation compared to vanilla ICL in our experiments.

Thank you. We hope these points address all of your concerns, and we are happy to clarify anything further. We also commit to fixing all paper writing errors pointed out in the final paper draft.

References:

[1] Todd et. al. FV

[2] Hojel et. al. VTV

[3] Imai et. al. "A General Approach to Causal Mediation Analysis."