Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models

1.1.Why training auxiliary model.

Just as you commented, ICL does not need to update the model parameters. However, this refers to LLM or LVLM, not to the auxiliary models used for selecting good ICDs. In fact, NLP researchers have developed various auxiliary methods to retrieve and order ICDs [41,42,44,45], while few studies have done so for VL ICL as of the submission deadline. Our work is a pioneering effort in this area. Furthermore, we are the first to apply the concept of treating ICD generation as a Language Modeling task, a strategy yet unexplored in NLP, to the more challenging VL field.

1.2. More benchmarks and models. We test Lever-LM on two tasks of VL-bench [A] due to computation limitation and show the results in Table I. It can be found Lever-LM achieves higher performance than other retrieval-based methods, validating the generalizability of Lever-LM. We also follow your suggestion to validate Lever-LM's generalization ability using IDEFICSv2 from the diverse LVLMs you mentioned, due to computational constraints. Our choice is based on two factors. First, IDEFICSv2 is an open-source model specifically designed for ICL. Second, modern LVLMs with robust ICL ability belong to two mainstream architectures : (1) Flamingo-based (using cross-attention to fuse vision and language, like Open-Flamingo or IDEFICSv1 that are tested in our paper) and (2) LLaVA-based (directly concatenating image and language tokens like IDEFICSv2). Thus, testing with IDEFICSv2 assesses Lever-LM's generalization across both architectures. Since IDEFICSv2 directly concatenate vision and text tokens, its input sequence will be longer than IDEFICSv1 and 4-shot inference reaches our GPU limitation, thus we report the results of 1-4 shots. The results in Table J show that Lever-LM achieves better results than RS and SIIR. Note that RS outperforms SIIR here, one possible reason is that IDEFICSv2 is more likely be damaged by short-cut inference due to similar ICDs are used, while Lever-LM is more robust.

To validate our hypothesis, we review examples of text generated by SIIR, such as the following:

SIIR Model Output [The first is the most similar ICD, and the last is the query]:
   Caption:Many children are posing together outside of the building window. [ICD1]
   Caption:A group of children are sitting together wearing dresses and suits and ties. [ICD2]
   Caption:Many children are posing together outside of the building window. [Query Prediction]
Lever-LM Model Ouput:
   Caption:School children cheer a tennis match with a pirate and giant tennis racket. [ICD1]
   Caption:A boy in a baseball cap holding baseball mitt.  [ICD2]
   Caption:A group of school children posing for a photo. [Query Prediction]

Ground Truth
1. Many small children are posing together in the black and white photo. 
2. A vintage school picture of grade school aged children.
3. A black and white photo of a group of kids.
4. A group of children standing next to each other.
5. A group of children standing and sitting beside each other.

It is evident that the model directly copied the caption from the first ICD, and such errors are not uncommon. We speculate that the LVLM's coarse visual models cannot distinguish excessively similar images, leading the LVLM to treat the ICD and the query as the same image, resulting in the model copying the ICD's caption.

Additional, we test Lever-LM in NLP ( A.2.1 of R.RjFC) and show more fine-grained analyses based on IC and VQA (A.3.2 of R.sise). A.2.2 of R.RjFC also discusses why Lever-LM is a generalizable ICD configuration strategy in VL.

[A] VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning

2.Zero-shot Performance.

We evaluate the zero-shot performance in Table H, showing that when compared with zero-shot performance, all the ICL methods achieve significantly improvements. We will incorporate zero-shot results in all Tables in revision.

3.Constant performance of more shots.

Many-shot ICL performance is influenced by two factors: LVLM's capacity for addressing long input sequences and the in-context sequence quality. Studies on Open-Flamingo v2 and IDEFICSv1, Which are two LVLMs used in our experiments, indicate that these LVLMs exhibit limited ability with many-shot ICDs [12][13][29]. Specifically, these LVLMs are trained by 5-shot ICEs, meaning limited capacity for more shot inputs. Our findings, such as Table 1, also show limited performance gains from various retrieval-based methods using more-shot ICDs. Yet, Lever-LM outperforms these methods on average. This does not imply Lever-LM's generalization to more-shot cases is compromised. We appreciate your feedback and plan to test Lever-LM with LVLMs better suited for longer inputs.

4.ClipScore.

We appreciate your suggestion about calculating the CLIPScore. High-quality captions generated from a model should hinge on two aspects: linguistic correctness and visual congruity. CIDEr and CLIPScore respectively values the former and the latter. The CLIPScores of diverse methods are given in Table K. We can find that RS has higher CLIPScore than SIIR, while SIIR has higher CIDEr than RS (Table 1). One possible reason is that when using SIIR, LVLM might copy the captions of ICDs with similar images as the query and overlook the vision content of query image. However, since RS returns less similar in-context images as the query, the short-cut inference is alleviated and thus has higher CLIPScore. Furthermore, we find Lever-LM has the highest CLIPScore, suggesting Lever-LM generates dissimilar ICDs to the query while meantime help LVLM generate the captions which are more grounded to the images. These comparisons validate that Lever-LM can capture useful statistic patterns between high-quality ICDs instead of simply generating similar ICDs.

We appreciate your response. Due to the time constraints during the rebuttal phase, we only generated 150 high-quality ICD sequence data, which may have led to a decrease in extrapolation ability. After Rebuttal, we continued to generate more data (500 high-quality ICD sequences) and in the VL-ICL benchmark and trained a Lever-LM with it. The results are as follows:

The results show that the Lever-LM still has the extrapolation abilities.

1. The role of CLIP

The role of the CLIP model is to encode the data in the supporting set. Therefore, in practical applications, the dataset can be pre-encoded and stored locally.

2. Lever-LM Inference Time.

During inference, only two layers of Transformer decoders are needed, which results in low computational overhead. We show the inference time of diverse methods in Table G. It can be found that Lever-LM does not have a significant gap in inference time compared to the SIIR method.

2. Compared with other Ranking Methods.

In Line 102, we mention that the ordering of ICD sequences has been primarily studied within NLP [17,24]. Their methods are mostly tested on datasets with limited labels (e.g., binary sentiment classification) or require calculating the probability of each input token. Transferring these methods to VL tasks presents two challenges: 1. VL tasks often include continuous image features, and modeling the probability distribution of continuous features is difficult. 2. VL tasks like IC or VQA are often open-ended word generation tasks, which do not have a limited label space. Facing these challenges, there is limited study explores how to order VL ICDs for improving ICL performance. In fact, we are the first to model the ordering of VL ICD sequences up until the submission deadline, thus it is hard to find suitable comparable methods. Table 4 demonstrates that when the quality of the selected ICDs is the same, the order produced by Lever-LM is optimal, indicating that Lever-LM can generate the proper order.

3. Typo in Figure 3. We apologize for the typo, and we will revise this part in revision.

1. Lever-LM with Different Sizes.

We follow your suggestion to obtain different sizes of Lever-LM by controlling the number of Transformer layers and test these Lever-LMs on the IC. As shown in Table E, we evaluate 1-layer/4-layer Transformer decoder layers. We find that the size of Lever-LM has minimal impact on performance. We believe that capturing the ICD sequence distribution is a simple task that can be learned by only a few Transformer Decoder layers. Moreover, our motivation is to use a small model to enhance the ICL performance of a large model, so it is not appropriate to design Lever-LM to be too large.

2.1 Lever-LM in NLP.

We follow your suggestion to train a Lever-LM in NLP. Specifically, we use Qwen1.5 [A] 1.8B and generate 2-shot ICD datasets for SST-2 [B], which is a sentiment classification task. The accuracy results are displayed in the Table F. It can be observed that our Lever-LM outperforms the Random method and STTR in Avg:1~2 and Avg: 4~8, demonstrating the potential of Lever-LM in NLP.

[A] Qwen Technical Report

[B] Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

2.2. Generality in VL.

Our approach is actually a generalizable method in VL. Different VL tasks require diverse configuration strategies. For example, in IC, [20] proposes that the image and caption need to be matched. In VQA, [21] proposes the best ICD selection strategy involves sorting based on image similarity first and then question similarity. In other words, LVLM requires different heuristic ICD selection strategies for different tasks, and it is difficult to find a universal and effective configuration strategy. Our method, on the other hand, provides a generalizable solution that can directly adapt to different models and tasks.

3. The format of IC and VQA.

We have detailed the prompt template in A.3.Prompt Template of Appendix. Our prompt template is entirely based on the settings described in the Open-Flamingo and IDEFICS.

4. The Inference Time of Lever-LM.

Please refer to A.2 of R.Djwv where we show the inference time of Lever-LM and SIIR.

1. Random Sampling.

The goal of randomly selecting samples to form the sub-supporting set $\mathcal{D_{\mathcal{M}}}$ is to enhance training data diversity, promoting Lever-LM to capture complementary knowledge among ICDs. We initially deemed this to be a strategy sub-optimal, exploring alternatives like selecting similar texts and images (Lines 265-269). Surprisingly, Table 2 (9)-(11) shows random sampling as most effective. This finding (Lines 288-291) suggests that using similar ICDs to form $\mathcal{D_{\mathcal{M}}}$ may make Lever-LM generated ICDs contain redundant knowledge, hindering ICL performance. This also aligns with multiple studies underscoring the benefits of diversity for ICL performance [43,54,A,B].

[A] Diversity of Thought Improves Reasoning Abilities of LLMs

[B] In-Context Learning with Iterative Demonstration Selection

2.Fixed ICD Configuration.

Actually, the existence of a good fixed set for IC proves that ICL works on IC. This is because ICL's main goal is to facilitate efficient learning of new tasks with minimal examples. To achieve this, one ongoing research direction in ICL is to find a minimal supporting set for ICD selection [C,D], and our discovery of a "golden fixed set" requiring only 8 ICDs is an extreme instance of this direction. Hence, our findings robustly suggest ICL's potential for IC.

We also follow your suggestion to compare the fixed set learned by Lever-LM with the one constructed by random selection. Specifically, we use OpenFlamingo to randomly select 3 sets of k-shot ICD sequences with different seeds for the image captioning task and then conduct ICL tests. As shown in Table A in the rebuttal PDF, it can be observed that randomly selecting a fixed set of ICD sequences results in relatively poor performance. The Golden-Set outperforms the best Fix Random set (seed 1) in Avg:1~8 by 16.14 points. This also demonstrates the importance of high-quality ICD sequences.

[C] Finding Support Examples for In-Context Learning

[D] Compositional Exemplars for In-context Learning

3.1.Why VQA and IC.

It should be stressed that our experimental settings are different from M-ICL [E] that only needs to forward LVLM for ICL inference, while we need to train Lever-LM by modifying various settings (Lines 262-292) to identify which factors influence the learning. Given limited resources, we focus on essential vision-language tasks, IC and VQA, to evaluate Lever-LM. IC and VQA also play important roles in LVLM ICL. First, they are commonly used for evaluating LVLM's ICL performance [12,13], even are specifically studied to help understand the LVLM ICL abilities [20,21]. Also, IC and VQA incorporate diverse vision-language skills: IC tests object, attribute, and relation recognition, while VQA covers diverse question types, integrating various capabilities like classification ('what is the object?') or counting ('how many of one object?'). Modern benchmarks for evaluating LVLM often derive from the IC (COCO dataset) and VQA (VQAv2 dataset). For example, SEED-Bench [F], MME [G] and MMBench [H] incorporate core characteristics from IC and VQA tasks, assessing various aspects such as scene understanding, instance identity, spatial relations, and diverse question types.

[E] What Makes Multimodal In-Context Learning Work?

[F] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

[G] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

[H] MMBench: Is Your Multi-modal Model an All-around Player?

3.2.Fine-grained Analyses.

We value your fine-grained analysis suggestions but find MM-Vet [I] and SEED-Bench not fitting our study. MM-Vet's limited scale (200 images, 218 questions) is not sufficient for effective Lever-LM training. SEED-Bench, designed for multiple-choice questions, does not match well with LVLMs that focus on word generation in ICL, requiring specific adaptations that are challenging to us given the rebuttal's time constraints. However, as previously mentioned, VQA and IC are appropriate for fine-grained analysis. For IC, we assess object hallucination using CHAIR [J], providing object-level results, where larger CH$_s$ and CH$_i$ values indicate more object hallucinations. As Table B&C shows, Lever-LM ICDs, compared to random sampling-based and similarity-based retrieval methods, achieve smaller hallucination scores because they contain fewer similar images to the query, preventing LVLM from short-cut learning. For VQA, we calculate the accuracy of three question types for fine-grained analyses: Yes/No, Counting, Others (Table D), which shows that Lever-LM significantly outperforms other methods in Counting, suggesting Lever-LM ICDs helps LVLM get stronger fine-grained object recognition ability.

[I] MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

[J] Object Hallucination in Image Captioning

3.3.More Models and Benchmarks.

We follow your suggestion to test Lever-LM on more models and benchmarks. Specifically, we examine the generalizability in VL-ICL-Bench (A.1.2 of R.BDyQ), Qwen1.5 model in NLP (A.2.1 of R.RjFC), and by another LVLM architecture, IDEFICSv2 (A.1.2 of R.BDyQ).

4.Order strategies.

The comparative results in Table 4 primarily stem from Lever-LM unifying the generation and ordering of ICDs. Lever-LM generates ICDs with complementary knowledge, enhancing the overall performance and diminishing the importance of ICD ordering. Thus, it is hard to mirror the significant improvements as the studies focusing solely on ICD ordering. Yet, Table 4 shows that Lever-LM's generated order outperforms random order given high-quality generated ICDs, demonstrating its effectiveness in ordering ICDs.