LIVE: Learnable In-Context Vector for Visual Question Answering

W1: Generalizability of L-ICV

We also follow your suggestion to test L-ICV on the VL-ICL Benchmark[A] and the results are given in Table C. Due to the rebuttal's time constraints, we only tested two subtask performances in the VL-ICL Benchmark. We also tested L-ICV on IDEFICS v2 and some new tasks to validate the generalizability of L-ICV in Table C&E and please refer to "W2&W3" of R.fRTc for more details.

[A] VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning.

W2: Performance Improvements

It should be stressed that the major motivation of ICV is to accelerate the inference speed instead of improving the ICL performance. To train L-ICV, it costs us 11 $\times$ 3090 GPU hours on 8000 VQAv2 samples, occupying 1.8% of the VQAv2 dataset, which means that the total cost is not intolerable. Notably, all ICDs are randomly sampled from 8000 samples. After training, the inference speed can be largely improved and please refer to "Q1" of R.8z8x to see the trade-off between reduced inference time and additional training time. Furthermore, if L-ICV is trained by 1-shot ICD, the cost will be much less, 1$\times$ 3090 GPU hours on 8000 samples, while 1-shot L-ICV achieves 5.45%/6.76% higher accuracy than 1-shot ICL on VQAv2/OKVQA, which is a significant improvement. In this case, even 1-shot L-ICV achieves better results than 32-shot ICL on VQAv2 and 16-shot ICL on OKVQA. In the 32-shot case, the improvement of L-ICV is less significant, while one reason is that modern LMM with ICL ability can not deal with long input sequences. Specifically, these LMMs are trained by 5-shot ICEs, meaning limited capacity for more shot inputs [6,7]. We also compared L-ICV with the LoRA method requiring training, and L-ICV consistently outperformed LoRA across various training set sizes, indicating that not all training-required methods achieve such good performance.

W3.1: Comparison with RICES and MMICES

Retrieving ICDs based on query sample and learning an ICV represent two major but orthogonal directions in ICL. The former exploits more individual characteristics of the query sample to improve the ICL performance. While the latter exploits more common knowledge to get a single intervene vector to improve the ICL inference speed. Considering their different motivations, we do not directly compare them in the submitted manuscript. Here we follow your suggestion to discuss their relative advantages and disadvantages.

1. Table H compares L-ICV with RICE/MMICES, showing that on average, L-ICV is better than RICE/MMICES on VQAv2/OKVQA, while be worse than RICE/MMICES on OKVQA when using more-shot ICDs since this tasks asks for more external knowledge and RICE/MMICES can provide such knowledge when returning similar ICDs.
2. RICE and MMICES require additional image and text encoders to retrieve ICDs from a database, occupying computational resources and causing significant latency during inference. Moreover, RICE and MMICE require LMMs to deal with many-shot ICDs, thus requiring longer inference time. To sum up, it is less favorable for real-time deployment than our L-ICV.
3. RICE and MMICES retrieve ICDs based on feature similarity, which makes LMM produce more short-cut learning as demonstrated in [16,17]. This is because that LMM tends to mimic the text description of the most similar samples and then neglect the vision contents. However, since L-ICV tries to capture common knowledge from diverse combinations of ICDs, it is less affected by short-cut learning.

W3.2: Using RICE samples to learn L-ICV

Actually, at the beginning, we also tried to use RICE samples to learn L-ICV. However, the results were unsatisfactory, with only 57.68% accuracy on VQAv2, while using random samples to train L-ICV got 58.54% accuracy. This may be because using RICE samples will make each training input sequence contain abundant individual characteristics, increasing the difficulty for L-ICV to learn common task knowledge, and even causing L-ICV to learn spurious correlations between the similar ICDs and the query. We will add this experiment and discussion in the revision.

W4: Why update parameters in a parameter-efficient way.

Indeed, the advantage of ICL is that it does not require parameter updates, but its sensitivity to ICDs is a drawback. Therefore, many papers attempt to train auxiliary models to select or order ICDs for improving ICL performance [10, 11]. Our L-ICV follows this research direction but uses less data and updates fewer parameters than other parameter-efficient methods like LoRA. Moreover, due to the sequence length inference issue, ICL faces significant latency during deployment when applying some retrieval-based methods. However, L-ICV achieves more stable and superior performance with minimal training, striking a trade-off between PEFT and ICL, combining their strengths to enhance inference performance and reduce latency.

We appreciate your response. We conducted additional experiments to show how L-ICV performs with a few demonstrations and to explore its transferability. We also include more discussion of these questions.

Q1: A few demonstrations to train L-ICV.

L-ICV only uses 800 samples (the total number of samples in the training split) in the VL-ICL Clevr and Textocr tasks.

To further explore how much data L-ICV requires to be effective for those tasks, we conducted additional experiments. Theoretically, even with a small number of ICD samples, they can generate a large number of ICD sequences. For example, there are $A_{300}^{32}$ ICD sequences for 300 data samples. Therefore, by extending the training steps of L-ICV, the model can learn from more randomly arranged samples. We trained for 5 additional epochs on the original basis. In the TextOcr task, only 50 samples were needed to achieve an accuracy of 27.5% (3.5% higher than 32-shot ICL). In the Clevr task, 200 samples were required to achieve an accuracy of 34.5% (1% higher than 32-shot ICL)．Additionally, we conducted further tests on VQAv2. Due to time constraints, we used 4-shot training on 300 samples for L-ICV. After 15 epochs, L-ICV achieved an accuracy of 53.75% (0.23% higher than 4-shot ICL). These results indicate that L-ICV can be effective even with a relatively small amount of data. Given time, as we continue to increase the number of ICD sequences and training epochs, we may get satisfactory L-ICVs with fewer samples.

Q2: The transferability for L-ICV.

We followed your good suggestion to evaluate the transfer performance of L-ICV and ICL. Specifically, for ICL, we used VQAv2/OKVQA samples as ICDs during inference when testing on OKVQA/VQAv2 (VQAv2->OKVQA/OKVQA->VQAv2). Similarly, for L-ICV, we used L-ICV trained on VQAv2/OKVQA for inference on OKVQA/VQAv2 (VQAv2->OKVQA/OKVQA->VQAv2). The results are shown in the table below:

It can be seen that L-ICV achieves better transferability compared to ICL. For example, in the OKVQA->VQAv2 experiment, L-ICV outperformed ICL by 1.28%, and in the VQAv2->OKVQA experiment, L-ICV outperformed ICL by 0.96%. One possible reason is that when using ICL, the individual characteristics of the ICDs from another task may have more damage than the general task summary of another task, which is captured by L-ICV. This demonstrates that the L-ICV we trained can be directly transferred to similar tasks while maintaining relatively good performance. It can also be found that the transferred L-ICV is worse than the original L-ICV, e.g., OKVQA->VQAv2 L-ICV is worse than VQAv2->VQAv2 L-ICV. This aligns with intuition, because when using samples from a task to train L-ICV, L-ICV will learn the Task Learning ability (refer Q1&W1 of R.VKDo) and learn the common knowledge of the task, which may have a distribution shift with anohter task and lead to the performance decline in other task.

To compensate this effect, we further propose a general ICV, it enables us to combine knowledge from multiple datasets. As shown in Appendix C, Table 16, we simply evaluated that by using the mean vector of L-ICV trained on VQAv2 and OKVQA, and it can achieve results on both datasets that are comparable to 32-shot ICL. We believe this could be a promising direction for future research, as a general ICV learned from similar tasks could effectively address scenarios where a new task has only a few demonstrations and no large annotated dataset.

However, it is out of scope of the discussion of this paper and we hope to leave it as future work.

W1: Inference Time Comparison of TV and FV

TV, FV and L-ICV have similar inference time. Specifically, TV takes 57.14 ms, FV takes 57.08 ms, L-ICV takes 56.81 ms for one sample, and Zero-Shot takes 56.69 ms. This similarity in inference times is because TV and FV replace the ICDs with a single vector to intervene in the model’s forward process, just like our L-ICV. However, L-ICV significantly surpasses TV and FV in terms of performance, achieving higher accuracy while also reducing inference costs. We will include these in the final.

W2: Generalizability of the Method

To test the generalizability, we extended L-ICV to IDEFICS v2-8B [A]. Notably, IDEFICS v2 differs in architecture from IDEFICS v1 (used in the paper). Currently, there are two main LMM architectures for ICL: (1) Models like IDEFICS v1 that use cross-attention to fuse image and text tokens, and (2) Models like IDEFICS v2 that employ additional projection layers to concatenate image and text tokens. Testing L-ICV on IDEFICS v2 can further demonstrate the generalizability across architectural designs. Since IDEFICS v2 concatenates image and text tokens, it has a longer input sequence than IDEFICS v1 when the same image-text pair is used. For instance, an image is represented by 64 tokens, resulting in 512 vision tokens in an 8-shot scenario, which pushed our GPU to its computational limits. Consequently, we trained L-ICV with an 8-shot ICD sequence and compared it with 1-8 shot ICL. The results in Table E show that L-ICV outperforms 8-shot ICL, achieving 4.1%/0.84% higher accuracy on VQAv2/OKVQA, respectively. Additionally, L-ICV is 8.95 times faster in inference speed compared to 8-shot ICL on IDEFICS v2.

[A] What matters when building vision-language models?

W3: L-ICV on Other Tasks

Note that L-ICV is inherently task-agnostic as Equations (1-4) in Sec.3 can be used among various tasks. We initially focused on VQA because it covers a wide range of vision-language challenges (Lines 56-59), which is a more challenging task for FV/TV. We follow your suggestion to test L-ICV on image captioning, VL-ICL Benchmark and OcrVQA (Table C). It can be found that L-ICV achieves 11.07 higher CIDEr than 32-shot ICL on IC, 3.5 higher ACC than 32-shot ICL on VL-ICL-clevr, 2.5 higher ACC than 32-shot ICL on VL-ICL-textocr, 1.2 higher ACC than 32-shot ICL on OcrVQA, validating the generalizability of L-ICV on various tasks.

W4: Ablation with Untrained L-ICV

We evaluated the impact of untrained L-ICV by initializing them randomly with a normal distribution and tested on VQAv2 (Table F), revealing that untrained L-ICV has similar results as zero-shot, which are largely worse than the trained L-ICV. We also visualized the embeddings using t-SNE to see the shift effect (Figure A), showing that the shift effect of untrained L-ICV is similar to zero-shot ICL, while learned L-ICVs align more closely with 32-shot ICL, proving that the training process is important.

---

Q1: Why do we choose 32-shot demonstrations and the performance gap with more shots?

The choice of 32-shot demonstrations is influenced by the LMMs’ capacity for understanding long texts, rather than by limitations of our method. Many related studies [6,7] typically use up to 32 shots due to these practical constraints. To investigate the impact of more-shot ICDs, we evaluated L-ICV with 48/64-shot ICDs (Table G), showing that while adding more shots improves performance for both methods, the relative gain for L-ICV remains higher than ICL. For example, with 64 shots, L-ICV achieves a 2.99%/1.85% improvement over ICL on VQAv2/OKVQA. Thus, although using more ICDs improves ICL performance, L-ICV has a larger improvement.

Q2: Why unique L-ICV for each layer and the comparable design for LoRA?

Since [22] finds that each layer plays distinct roles during ICL inference (Lines70-72), we use unique shift vectors for each layer where Equations (3-4) are applied to each single layer, making L-ICV layer-specific. Using layer-specific L-ICV requires 32 $\times$ 4096 parameters. We integrated LoRA into the token classification head of the last layer to minimize the trainable parameters (still 8 times more parameters than L-ICV). To ensure a comprehensive comparison, we also evaluated LoRA when applied to the O_proj of all layers, as shown in Table B. This configuration results in an 80-fold increase in parameters, allowing LoRA to achieve decent performance but still falling short of L-ICV by 0.97% in VQAv2 and 5.05% in OKVQA, suggesting that L-ICV is more effective than LoRA.

Q3: How to address the performance sensitivity to ICD selection with L-ICV?

As discussed in Lines 67-70, it is precisely because we aimed to reduce the sensitivity of ICD that we randomly select 32-shot examples during training. Generally, each ICD has individual characteristics and some common knowledge about the task, while the ICL sensitivity stems from the individual characteristics. To encourage the ICV to learn common knowledge, during training, each in-context sequence is set to a random combination of ICDs, which forces ICV to capture the most common knowledge while disregarding these individual characteristics. Please refer to "Q1 & W1" of R.VKDo for more details.

Q1: trade-off between reduced inference time and additional training time

We believe that performing additional lightweight training to reduce inference time is worthwhile. Firstly, the initial one-time training investment is minimal and yields substantial long-term benefits. For instance, in our local machine (3090 GPUs) tests on VQAv2 experiments, training L-ICV takes approximately 11 GPU hours. In comparison, inferring 20,000 data samples using 32-shot ICL takes about 13.4 GPU hours, whereas L-ICV achieves the same inference in just 0.5 GPU hours. The time saved for 20,000 samples inference is comparable to the time spent on training. Consequently, the cumulative savings in inference costs can easily exceed this initial investment. With the increasing number of tokens required by scaled LMMs (e.g., IDEFICS v2), L-ICV's efficiency in saving memory and reducing inference costs becomes even more pronounced (Table E). This advantage makes L-ICV better suited for batch inference and enhances the throughput of inference services at scale. Secondly, unlike L-ICV, ICL's performance is highly unstable. While various methods exist to stabilize ICL performance—such as selecting ICDs similar to the query—these approaches inevitably incur additional computational and memory costs. In contrast, L-ICV offers stable performance while substantially reducing inference time In the upcoming revision, we will further discuss this issue, emphasizing the feasibility and necessity of lightweight training in exchange for faster inference.

---

Q2: How much would performance drop if each layer shared the same trained bias vector?

Thank you for your suggestion. We conducted experiments to investigate the impact of using a shared bias vector across all layers (as shown in Table D), and we observed a significant performance drop compared to ours: 20.6% on VQAv2 and 9.01% on OKVQA. This performance drop aligns with the findings in [22] that different Transformer layers play diverse roles in ICL (Lines 70-72). Also, Equations (1-4) in Sec.3 show that L-ICVs are layer-specific. Then using a single shared vector fails to capture the distinct information processed at each layer, leading to the observed decline in accuracy. Additionally, since the number of tuned parameters is limited to 32 × 4096 for layer-specific L-ICVs, using a shared 1 × 4096 ICV across layers does not substantially impact training speed. During inference, the computational cost of using a shared ICV versus non-shared ICV remains comparable.

Q1 & W1: Task Recognition (TR) and Task Learning (TL)

Thanks for your recommendations of these studies. In our final discussion, we are happy to include these recent works on the interpretation of ICL abilities of LMMs. After thoroughly reviewing [A,B], we think that theoretically, L-ICV is more akin to TL. This is because during the training of L-ICV, we use randomly sampled 32-shot ICDs to encourage L-ICV to learn the common knowledge of the task (Lines 68-71), while overlooking the individual characteristics of the ICDs. In this way, L-ICV does not exploit the individual characteristics of the ICDs to narrow down the label space for predicting as TR mentioned in Sec.3.1 and Figure 1 of [B]. Moreover, Equations (1-4) in Sec.3 of our paper might provide a better mathematical explanation for TL. Specifically, in the latent space, the contextual information of ICDs helps shift the query representation toward the direction of the correct answer. The role of L-ICV is to learn the most stable shift vector from various ICD combinations. In this way, L-ICV makes LMM less affected by short-cut inference since this often over-exploits the individual characteristics of the ICDs. The analyses in Line 302 "Hallucinations and Invalid Responses" show that L-ICV is less affected by yes/no hallucination, suggesting L-ICV is more robust that it does not simply copy yes/no from the ICDs as the answer. Additionally, we follow [B] to deploy the "new-mapping" experiment for testing TL ability, where the results are given in Table A. Specifically, when training L-ICV and testing, all "yes/no" labels are changed to "tiger/lion". When applying ICL, we also change "yes/no" to "tiger/lion". In this way, if L-ICV has good TL ability, it will learn the mapping between the correct answer to one new label instead of recognizing "yes/no". Results in Table A show that L-ICV achieves better results than ICL, suggesting it has good TL ability.

[A] What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning

[B] How to Configure Good In-Context Sequence for Visual Question Answering

Q2 & W2: Fair LoRA Comparisons

1. First, we would like to clarify that in Section 4.3, the term "external knowledge" specifically refers to the OKVQA dataset, not to imply that "since both VQAv2 and OKVQA require external knowledge, LoRA has worse performance." For VQAv2, which does not require external knowledge, LoRA performs reasonably well (as shown in Figure 4, main paper). However, our L-ICV still achieves superior results with a notable improvement of 2.36%.
2. We followed your suggestion to train LoRA in a fairer setting by using 32-shot ICDs and applying the same KL loss as our L-ICV. As shown in Table B, even with this modified design, LoRA's performance on VQAv2 and OKVQA remains significantly lower than that of L-ICV (49.48% vs. 58.54% in VQAv2 and 37.02% vs. 50.08% in OKVQA). Additionally, we evaluated LoRA with more parameters, as suggested by Reviewer fRTc, and the results are presented in Table B. Similarly, we find that increasing the parameter count does not bridge the performance gap between LoRA and our L-ICV, demonstrating the effectiveness of L-ICV's unique design.

W2: More Extension Tasks

We followed your advice to evaluate L-ICV on OcrVQA, which also does not require external knowledge. The results are shown in Table C, and we find that L-ICV outperforms LoRA and 32-shot ICL by 1.6% and 1.2% respectively. Additionally, we evaluated our performance on Image Captioning (Table C) and another multimodal model architecture, IDEFICS2-8B (Table E). These consistent improvements over 32-shot ICL and LoRA demonstrate that our method is highly effective and efficient across various tasks and models.