VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning

Thank you very much for your review and questions. Here are our responses to your comments.

W1. Experiments on In-context demonstration selection strategy

Thank you for your suggestion, we agree this is a useful additional experiment. We employ a similarity-based selection strategy mentioned in all suggested papers for experiments. Specifically, for image-to-text tasks (CLEVR and TextOCR), we use CLIP image embeddings to retrieve the top-k most similar images and their corresponding text. Conversely, for text-to-image tasks (CobSAT), we use CLIP text embeddings to retrieve the top-k most similar texts and their associated images. Additionally, we experiment with three different arrangements of the selected top-k demonstrations: (1) from the most similar one to the least similar one, followed by the query; (2) from the least similar one to the most similar one, followed by the query; and (3) a random order of the top-k demonstrations. These are compared to the original setup in the paper, which used random selection. We select several top-performing models for our experiments, with the results presented below. In each cell, the first three numbers correspond to the three arrangements of the selected top-k demonstrations (1), (2), and (3), while the final value represents the baseline performance with random selection. For both text-to-image and image-to-text tasks, similarity-based retrieval consistently enhances performance. Specifically, similarity-based selection methods have a ~3% improvement in peak accuracy and a 3% boost in ICL efficiency compared to baseline (details in appendix Table 10). These findings demonstrate that selection strategies can enhance ICL tasks, highlighting a promising direction for future work aimed at improving performance across various tasks.

Table 1. Results on CLEVR dataset

Table 2. Results on TextOCR dataset

Table 3. Results on CobSAT dataset

We have included the new experiments and relevant references in the revised paper.

Re: format learning in VQA and captioning, to our knowledge, it has not been widely discussed, besides brief mention in the concurrent work [1]. And as far as we know, we are the first to quantitatively evaluate and demonstrate this effect. If you know more references related to this issue, we are happy to include them.

[1] Towards Multimodal In-Context Learning for Vision & Language Models. arXiv 2024.

Thank you very much for your review and questions. Here are our responses to your comments.

W1. Clarification on image generation evaluation

Thank you for your comments. We would like to argue that while generating images under complex instructions is an important task, it is orthogonal to our learning of image generation with ICL. The goal of our task is to assess whether VLMs have the capabilities to learn the concepts, attributes, rules etc., from the in-context demonstrations and then generate the desired image. To our knowledge, complex instruction based generation such as [1] are zero-shot rather than ICL based. Therefore we have a different problem setting and use-case.

W2. Clarification on possible usage of the benchmark

Thank you for your comments. We do evaluate the generation and understanding capabilities separately for different models as you mentioned in the first situation. For example, as shown in Table 2 and 3, we evaluate different sets of models for image-to-text tasks and text-to-image tasks. With our evaluation, we find that vision-language ICL is challenging in both types of tasks and thus require a comprehensive benchmark to rigorously measure the performance and encourage future development. Therefore, we are providing such a comprehensive ICL suite for both types of models and we believe it will contribute to the research community.

Thank you very much for your review and questions. Here are our responses to your comments.

W1. Different trends between Figure 3(a) and 4

Thanks for asking about this part. We agree Figure 3a shows a certain amount of learning, and we have revised the associated text accordingly, in particular the claims in the caption that may have been previously too strong. Note that as we explain further in Figure 3, the learning is primarily about how to format the answer, as shown by our analysis there. When we compare Figure 4 with Figure 3b that aims to remove the impact of formatting or style, there is a very clear difference in behavior. Moreover, while both Figures 3a and 4 show curves trending upwards, the trend is stronger in Figure 4, highlighting the scale of ICL present in text-only settings. To further support this, we have computed the ICL efficiency for the different cases and these show the following:

• Average ICL efficiency across the datasets and models tested in Figure 3a (VLLMs): 7.48
• Average ICL efficiency across the datasets and models tested in Figure 4 (LLMs): 14.75 As a result the benefit of ICL is indeed significantly stronger in text-only settings than in VQA/captioning.

W2. Design choices of using different models in Figure 3 and 4

Thank you for your question. Figure 3 highlights a general issue: popular VLLMs do not benefit significantly from in-context learning (ICL) on tasks like image captioning and VQA. In contrast, we select a subset of models in Figure 4 to demonstrate that VLLMs do inherit the ICL capabilities of their base LLMs. The selected models are stronger and more representative, excluding relatively weaker models like OpenFlamingo and Otter. (OpenFlamingo and Otter VLLMs also do not accept text-only input).

W3. Clarification on data contribution

Thank you for your comments. The primary contributions of our benchmark are to construct a comprehensive evaluation suite that allows researchers to assess their models’ multimodal ICL capabilities, and a series of insights from evaluating existing models on that suite.

In constructing the evaluation suite, we collated 10 diverse tasks in one place, which required re-designing the structure and annotations of source datasets in common format for ICL. There is plenty of precedent for this kind of contribution (eg: Triantafillou’s highly cited meta-dataset in ICLR20). Creating the task suite was itself non-trivial. The operator induction datasets and tasks are fully created by us. While the other tasks use existing images, we had to repurpose them in novel ways, e.g. by stacking the images on top of each other to create composed images. While the matching tasks use existing images, the learning task is newly created by us and required deriving new annotation, etc. In summary, we do not simply re-use existing images and annotations directly - they all had to be repurposed in novel ways to design tasks that facilitate thorough testing of ICL.

Re dataset statistics: we do have the data statistics and tested capabilities of each subset in Table 1.

Q1. Clarification on VQA and captioning

We discuss ICL in VQA and captioning in the initial part of our paper as these are the main applications where multimodal ICL is commonly assessed (Bai et al., 2023; Awadalla et al., 2023; Sun et al., 2023a; Laurencon et al., 2023). However, as we discuss in the paper, VQA and captioning see limited benefits from ICL and so we design a benchmark that studies various different tasks, e.g. concept binding, induction or reasoning, to assess multimodal ICL more rigorously.

Q2. Why peak accuracy is used instead of average accuracy

We use peak accuracy as one of the metrics in addition to zero-shot accuracy and ICL efficiency in order to study the models from diverse perspectives. ICL efficiency already captures behavior across various numbers of shots and so is very similar to average accuracy over all shots, except that it discounts the zero-shot baseline. Peak accuracy is an interesting metric to report as it tells us about how well a model can perform when given an “ideal” number of shots. It has also been used in literature, in particular (Jiang et al., 2024b).

Q3. Evidence on ICL difficulty of dealing with larger number of shots

Thank you for your question. Some models obtain negative impact from more shots because the total input sequence length exceeds the models’ context length. E.g., in LLaVA, one image translates to 576 tokens, causing an 8-shot setting to exceed its 4k context window. We provide a summary of the context window of the evaluated VLLMs in Table 6.

Thank you very much for your review and questions. Here are our responses to your comments.

W1. Dataset sampling and filtering strategy

How many samples from the original sources we use is dataset-dependent. In cases such as operator induction we use all possible options. In others such as CLEVR or TextOCR we use a subset as using all samples from the set would make our benchmark unnecessarily large. When we use only a subset, we do filtering in some cases depending on if there are data-quality issues. In particular in TextOCR we did filtering to ensure we use only valid texts that are not marked as rotated. After the filtering we use random subsampling to ensure compactness of our benchmark. We now provide the details of sampling and filtering (if applicable) for each dataset in Appendix A.

W2. Explanation on ICL efficiency metric

We define ICL efficiency as the performance improvement as more examples are seen. Specifically, given an experimental result of accuracy vs number of shots (Fig 3, 4, 5, etc), we calculate a scalar efficiency value as the proportion of the area between the few-shot curve and the zero-shot baseline. Thus the maximum efficiency of 100% area between curves occurs if zero-shot baseline achieves 0% accuracy, and 1-shot and higher performance performance is 100% accuracy. Efficiency is 0% if performance doesn’t improve with more shots and for any number of shots is always equal to the zero-shot baseline (ie: a model that completely ignores its context set). Efficiency is negative if the few-shot performance becomes worse than the zero-shot performance. This can happen if a model fails to properly exploit the context set and simply becomes confused by the larger number of tokens – for example VILA-7B suffers from this. The goal of ICL is to exploit the context set to improve performance, and this efficiency metric quantifies exactly this effect.

W3-1. Reason of saturated ICL performance

Several multi-models are restricted by context length or perception capabilities:

Context-length limits: the models fail to benefit from more tokens after a certain point (typically related to the length of contexts models have been trained with). This point can be reached in fewer shots for a multi-modal than text model since the images provide more tokens.

Perception limits: the model may be able to induce the rule from in-context examples, but imperfect perception still limits the ability to solve the test examples if the digits/symbols are not always recognised correctly. Perception bottlenecks impact multi-modal models more than text models.

Nevertheless, please note that we do have tasks where there are some models that do not saturate in 1-shot. Figure 8 in the appendix shows that some strong models, e.g. GPT4V and LLaVA-OneVision-72B, can really learn from the in-context demonstrations and show a sharply growing trend w.r.t. shots in some tasks such as Fast Open-Ended MiniImageNet and operator induction.

W3-2 Why our benchmark offers a better evaluation of ICL capability

Some of them indeed have visually similar trends, however we argue that the main trends are different. In Fig 3b, we show that captioning and VQA start with high zero-shot scores, indicating that VLLMs can do these tasks well without using the context set, and performance does not increase with more shots. Compare this to Fig 6 (and Fig 8 in Appendix for an even stronger contrast) where performance at zero-shot is low, indicating a context set is necessary for solving these tasks, and for at least some models performance increases with shots. This difference between zero-shot and ICL performance is summarized by our ICL efficiency metric discussed above, and reported in Tab 2. Several models and tasks show a good ICL efficiency in Tab 2, but ICL efficiency for Fig 3b is approximately zero (more specifically 0.65% on average, compared to average of 14.53% across all models and datasets in our benchmark).

In summary, with captioning and VQA, performance generally does not improve above the zero-shot level, thus not demonstrating any learning from in-context examples; while in our VL-ICL performance starts at chance level and only increases above chance-level after seeing the support set. This makes it a better benchmark to assess ICL capabilities.

Q1. Bolden best-performing numbers in Tables

Thank you for your suggestion. We have bolden the best performance numbers in Table 5 in the revised paper.