What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration

We sincerely appreciate your thorough and insightful comments on our work. In the following, we will clarify your concerns and we would greatly appreciate it if you can reconsider the work in light of our clarification.

Q1: The difference between our work with previous studies [1,2].

R1: Thanks for your insightful feedback. We sincerely clarify the main differences from two aspects:

1. Unified Perspective. Our work makes the first attempt to unify the current multi-modal In-context learning (MM-ICL) process, which takes a meaningful step to offer a unified view and guideline for researchers to build better MM-ICL. Previous works [1,2] can also be integrated in our perspective.
2. Comprehensive Investigation. According to the unified perspective, we conduct a comprehensive exploration and provide insightful observations for each process within the unified perspective. However, previous studies [1,2] mainly focus on exploring the sample representation subpart of the retrieval process, which is only a subset of our exploration.

More intuitive comparisons can be found in Figure 1 of the supplementary material. We will follow your suggestions to add a detailed discussion in the next version.

[1] Baldassini et al. What Makes Multimodal In-Context Learning Work? CVPR 2024 Workshop.

[2] Chen et al. Understanding and Improving In-Context Learning on Vision-language Models. ICLR 2024 Workshop.

Q2: The paper does not consider standard benchmarks and metrics.

R2: Thanks for your constructive comments and we will answer your concerns point by point.

• Benchmark Concern: We use the standard benchmark M3IT [1], which consists of a series of common datasets divided into four categories. Detailed information is shown in Table 1. We will add more data descriptions in the next version.
• Metric Concern: In our experiment, we use the standard CIDEr metric for image caption task. However, since M3IT includes various VQA tasks with many free-form answers, the Acc. metric is not suitable. Therefore, inspired by the success of free-form and precise answer hybrid evaluation in machine reading comprehension, we use the Token F1 and BERTScore as metrics for evaluating semantics and exact keyword accuracy. We will add more discussion in the next version.

Table 1: Dataset in M3IT, where IC: Image Captioning, CLS: Classification, VQA: Visual Question Answering, R: Reasoning (with NL rationale).

[1] Li et al. M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning. Arxiv 2024.

Q3: Table 1 shows that IDEFICS 8B is comparable to GPT4V! However, IDEFICS 8B is worse on almost all benchmarks. Why？

R3: Thanks for your insightful comment. Sorry for confusing you. We totally agree that IDEFICS 8B is worse on almost all benchmarks. However, in our work, Table 1 shows that the results of IDEFICS2 8B [1], not IDEFICS 8B. In fact, IDEFICS2 is comparable with GPT4V, as shown in the following Table 2. We will add more discussion in the next version to make it clearer.

Table 2: Performance comparison.

[1] Laurençon et al. What matters when building vision-language models? Arxiv 2024

[2] Lu et al. MathVista: Evaluating Math Reasoning in Visual Contexts. ICLR 2024

[3] Liu et al. MMBench: Is Your Multi-modal Model an All-around Player? Arxiv 2023

Q4: Some claims should be refined or supported with more evidence:

1. Other works usually show the more shots the better the performance.
2. Other works show that increasing model size increases ICL performance.

R4: Thanks for your valuable suggestions. We will answer your questions point by point.

• Response to (1): Yes, your understanding is correct. In fact, previous works [1-3] concluded that 'more shots lead to better performance' mainly for the VQA task. We also observed this trend in VQA task. But in other tasks like multi-modal chain-of-thought reasoning, it can even worsen results with more demonstrations [4]. In the future, we will follow your suggestion and provide more fine-grained conclusions, such as offering fine-grained observations for different tasks.
• Response to (2): We totally agree with you. With the same model, a larger model generally performs ICL better. In our work, we were surprised to find that IDEFICS2 (8B) outperformed GPT-4V (>100B) on some tasks, indicating that beyond parameter size, multi-modal context understanding and alignment are also crucial for MM-ICL. We will refine our conclusion in the next version according to your suggestion.

[1] Alayrac et al. Flamingo: a visual language model for few-shot learning. NeurIPS 2022.

[2] Laurençon et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NeurIPS 2024.

[3] Shukor et al. Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning. ICLR 2023.

[4] Chen et al. M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. ACL 2024.

Q5: Other writing typos and tips (e.g., wrong citations; paper figures are not properly explained)

R5: Thanks for your valuable feedback. We will follow your suggestions to thoroughly fix wrong citations and add detailed figure captions. Specifically, different colors denote different models in Fig 10. The scores in Fig 8 and 9 represent the Average score (AVG).

Q6: What is BELU score?

R6: Sorry for the confusion. It is BLEU. We will polish our work in the next version.

Thanks for your timely feedback. We would like to further clarify your concerns.

Q1: Seeing the evaluation and conclusions, it seems that using the M3IT benchmark is not a good choice.

R1: Thanks for your suggestion. In our detailed response R2 in previous rebuttal, M3IT includes four classic types of tasks that contain almost all of your mentioned datasets (e.g., VQAv2, COCO captioning, TextCaps, OK-VQA, TextVQA). We sincerely believe that these cover a broad range of categories for evaluation. We will add more details in the next version.

Q2: Showing 2 scores of IDEFICS2 that are worse but close to GPT4V is not enough to conclude that these 2 models are even comparable.

R2: Thanks for your comment. We sincerely believe that there are some misunderstandings. In fact, in some previous findings [1,2,3], IDEFICS2 even outperforms GPT4V by 2 scores (see the Table below). As shown in Table 1 in the submitted paper, the performance of these two models are also comparable. The core difference lies in the image captioning task. Nevertheless, the metrics you mentioned, such as CIDEr, are inappropriate for evaluating open-ended image caption. Human judgments of alignment often differ significantly because GPT-4V tends to provide more detailed responses, whereas IDEFICS2 prefers shorter ones. However, in most image captioning datasets, the golden captions are brief, which presents challenges for traditional metrics [4]. Therefore, such observations motivate us to explore semantic-level BERTScore metric in addition to CIDER for open-ended image caption task.

[1] Laurençon et al. What matters when building vision-language models? Arxiv 2024

[2] Lu et al. MathVista: Evaluating Math Reasoning in Visual Contexts. ICLR 2024

[3] Liu et al. MMBench: Is Your Multi-modal Model an All-around Player? Arxiv 2023

[4] What you see is what you read? improving text-image alignment evaluation. NeurIPS2023

Q3: I highly encourage the authors to work on the evaluation and avoid any claim that goes against previous serious works.

R3: Thanks for your constructive comment. Actually, our findings are not in conflict with previous work. For example, our findings in multi-modal chain-of-thought reasoning that simply adding more shots does not always lead to improved performance, which is consistent with the previous work [1]. We think that the following might be reasons why more demonstrations may not improve performance in these tasks, which is acknowledged by Reviewer #Wrbn.

1. Cognitive Overload: For complex tasks, understanding complex demonstrations is difficult. More demonstrations can overwhelm the model, making it harder to process and integrate information effectively.
2. Complexity of Reasoning Tasks: We observe that for reasoning tasks, the performance improvement brought by the number of demonstrations is not even as good as using different retrievers. It shows that reasoning tasks require sophisticated integration of information, where quality trumps quantity.

In addition, in our experiments, we also observe that more shots lead to better performance in VQA task, which also align with [2][3][4]. We will add more discussion in the next version.

[1] Chen et al. M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. ACL 2024.

[2] Alayrac et al. Flamingo: a visual language model for few-shot learning. NeurIPS 2022.

[3] Laurençon et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NeurIPS 2024.

[4] Shukor et al. Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning. ICLR 2023.

Q4: Regarding the originality

R4: Thanks for your kind mention. We sincerely think that our work is the first to unify the current multi-modal In-context learning (MM-ICL) process and conduct a comprehensive exploration for each process within the unified perspective. The provided findings in this work are meaningful and interesting, which have been recognized by Reviewer #Pm2y, Reviewer #Wrbn, Reviewer #jbhA.

• Multi-modal alignment is the bottleneck for MM-ICL.
• Intra-demonstration ordering holds greater importance than inter-demonstration ordering.
• Introductory instruction guides better task understanding for MM-ICL.

In addition, when the mentioned work [1] is presented at CVPR Workshop, the NeurIPS deadline has already passed.

[1] Baldassini et al. What Makes Multimodal In-Context Learning Work? CVPR 2024 Workshop.

We greatly appreciate the time you've invested in reviewing our response. We hope the further clarification can address your concerns and we sincerely hope that you can reconsider your score in light of our clarifications. We promise to incorporate your all suggestions to improve our work.

We would like to extend our sincere gratitude for your thoughtful and detailed feedback. In addition, we are very grateful for the opportunity to further clarify your worry. We notice that your main concern is “Does F1/BertScore correlate with the accuracy?” and we sincerely agree with your concern. In the following, we will do our best to address your concern.

Q: Does F1/BertScore correlate with the accuracy?

A: Thanks for your insightful feedback. The answer is “Yes”. According to your suggestion, we have adopted accuracy for evaluating several VQA datasets including OK-VQA and VQAv2. The results are shown in Table 1 and Table 2 below:

From the results, we observe that accuracy is positively correlated with BERTScore and F1. We attribute it to the fact that higher semantic relevance and exact keyword performance can lead to higher accuracy. Such observation also demonstrates the main paper messages cannot be changed. We sincerely believe that the findings in our work can directly contribute to the MM-ICL community. In the next version, we promise to add more experiments and discussions to enrich our work.

Your insights and suggestions can significantly contribute to enhancing the quality and clarity of the work. We are very encouraged that the further clarifications can address your concern. Thank you once again for your selfless contributions to the community.

Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:

Q1: The 'Exploration of MM-ICL Prompt Construction' section can be explained more clearly.

R1: Thanks for your constructive suggestion. We will follow your suggestion to explain more details about the 'Exploration of MM-ICL Prompt Construction' section, including providing more description in Figure 4 and adding more details in three instruction categories explanation.

Q2: Some parts of the appendix, such as the implementation details of the baseline, can be moved to the main text.

R2: Thanks for your insightful feedback. We totally agree with your point. We will add the implementation details of the baseline to the main text in the next version.

Q3: The 'Limitations' section can include some specific limitations related to this work.

R3: Thanks for your insightful suggestion. We will discuss more limitations in the next version. For example, one of limitations in our work lies in the mis-consideration of some image instructions, such as grounding and adding extra arrows. These may require more sophisticated human design and are not supported by most current models.

Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:

Q1: The captions in the paper can be enriched to help readers gain a better understanding.

R1: Thanks for your constructive suggestion. We will follow your suggestion to add more detailed captions in the next version.

Q2: In Section 5.3, can you provide more analysis on why the number of demonstrations does not significantly impact MM-ICL?

R2: Thanks for your insightful feedback. We find that in complex reasoning tasks such as multi-step multi-modal chain-of-thought reasoning, more demonstrations will not lead to better performance, which is consistent with the observation [1]. We think that the following might be reasons why more demonstrations may not improve performance in these tasks:

1. Cognitive Overload: For complex tasks, understanding complex demonstrations is difficult. More demonstrations can overwhelm the model, making it harder to process and integrate information effectively.
2. Complexity of Reasoning Tasks: We observe that for reasoning tasks, the performance improvement brought by the number of demonstrations is not even as good as using different retrievers. It shows that reasoning tasks require sophisticated integration of information, where quality trumps quantity.

We will add more discussion in the next version.

[1] Chen et al. M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. ACL 2024.

Q3: The experimental section needs to be supplemented with a description of the LLM backbone.

R3: Thanks for your constructive suggestion. We will add a detailed description of each LLM backbone used in the next version according to your suggestion.

Q4: In your experiments, how did you handle multiple image inputs?

R4: Thanks for your insightful comment. There are two main categories for handling multiple image inputs:

• Tokenizer-based LVLM: It supports directly tokenizing the images and then directly concating them with textual tokens in an interlaced sequence.
• Encoding-based LVLM: The soft encoding of images is concated at the embedding layer to form an interlaced sequence of images and text.

Q5: Will different prompts affect the exploration experimental results?

R5: Thank you for your insightful feedback. In our preliminary experiments, we found that different prompts do not affect the overall conclusions. For example, we used different prompts (including instructions and delimiters) with the same semantics but different expressions. The results are shown in Table 2 below, and it can be seen that the impact of different prompts is not very large. We will add more discussion in the next version.

Table 2: Performance across different prompts.

Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:

Q1: The results of the experiments seem apparent and intuitive.

R1: Thanks for your constructive feedback. We sincerely believe that our work is a meaningful step in the systematic exploration of the MM-ICL and provides some insightful observations.

1. Previous studies mainly considered the inter-demonstration ordering. To our knowledge, we are the first to explore the intra-demonstration ordering and find that this ordering is significantly more important than inter-demonstration ordering, hoping to provide insightful guidance in the future.
2. Additionally, we thoroughly explored how to effectively construct MM-ICL prompts by investigating Introductory Instruction, Summative Instruction, and Intra-demonstration Instruction, which is less explored in the previous research.
3. In addition, we provide more insights on how to insert separators, optimize domain selection, and refine distance metrics, offering guidance on best practices for constructing MM-ICL.

We will follow your suggestion to add more discussion in the next version.

Q2: Some qualitative analysis is missing. It would be beneficial to compare how different in-context examples affect the answers.

R2: Thanks for your insightful suggestion. Actually, the Exploration of MM-ICL Demonstration Ordering and Exploration of MM-ICL Prompt Construction can reflect how different in-context examples affect the answers. We will add more discussion and add more qualitative analysis for better understanding in the next version.

Q3: As shown in Table 1, in most scenarios, a single textual retriever is sufficient. Therefore, using a multi-modal retriever, which may consume more computational resources, is unnecessary.

R3: Thank you for your insightful feedback. This is an interesting research question. Actually, multi-modal retrieval attains better performance in many scenarios like Image Caption and VQA. However, our experiments show that textual retrieval works well for classification and reasoning tasks.

Based on the qualitative analysis, we observe that due to the semantic richness of the labels and rationales, textual retrieval can obtain more similar samples. However, the current multi-modal retrieval struggles with complex text semantics, often favoring image similarity. This aligns with recent work [1,2], which is valuable for future exploration.

In the future, we can consider how to integrate the strengths of text and visual modalities for better performance.

[1] Tong et al. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CVPR 2024.

[2] Tong et al. Massproducing failures of multimodal systems with language. NeurIPS 2023.

Q4: Is it appropriate to combine all tasks to provide a universal paradigm for constructing MM-ICL strategies?

R4: Thanks for your constructive feedback. We sincerely think that providing a universal paradigm can help researchers conduct unified and fairer comparisons and studies within a unified framework. For example, in each task of this unified paradigm, researchers can explore how to improve a type of task targetedly to achieve better MM-ICL performance. We will add more discussion in the next version.

Q5: Some minor mistakes in the paper.

R5: Thanks for your kind mention. We will follow your suggestions to correct these issues one by one (e.g., modifying terminology, highlighting results, and fixing incorrect citation).

Q6: For multi-modal demonstration retrieval, why do you use a multi-modal encoder to calculate the similarity instead of using the product of text and vision single modality similarity?

R6: Thanks for your insightful feedback. In our experiment, as shown in Table 1 below, we found that the performance is far inferior to cosine similarity, which is also consistent with our conclusion. We attribute it to the fact that the model is more expected to obtain a semantic direction similarity rather than a distance similarity.

We will add more discussion in the next version.

Table 1: Performance of different similarity metrics on GPT4o.

Q7: For the OpenAI GPT-4V model, how do you reconstruct the order of the text and the image?

R7: Thanks for your valuable feedback. GPT4V API provides a content list that allows for controlling the order of text and images. An example is as follows:

{
  "model": "gpt-4-vision-preview",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What’s in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": f"data:image/jpeg;base64,{base64_image}"
          },
        }
        {
          "type": "text",
          "text": "What’s in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": f"data:image/jpeg;base64,{base64_image}"
          },

        }
      ]
    }
  ],
  "max_tokens": 300
}

We will add more details in the next version.

We sincerely appreciate the time and effort you have dedicated to reviewing our response. We are very pleased that all your concerns have been addressed. We sincerely hope that you can consider raising the score after we have addressed all the concerns. We greatly appreciate your selfless contributions to the community.