True Multimodal In-Context Learning Needs Attention to the Visual Context

Thank you for recognizing the novelty, efficiency, and improved performance of our method, as well as acknowledging the clarity of our presentation. We address each of your questions in detail below.

1. Performance Gains of DARA.

Thanks for raising this point! While DARA’s absolute gains may seem modest in some cases, DARA is efficient, flexible, and brings consistent improvement, making it a practical contribution. We highlight the following key points:

1. Consistent Gains across Models and Tasks with a few Parameters. DARA achieves consistent improvements across different models and tasks, using only ~100 trainable parameters and requiring no structural changes. This makes it especially appealing in MICL, where rapid adaptation and parameter efficiency are essential.

2. High Flexibility and Extensibility. DARA is light-weight and can be flexibly integrated with methods like LoRA. It also improves performance when applied on top of full-parameter LoRA (Table 4 in main paper).

3. No Trade-of on Standard VL Benchmarks. DARA maintains strong performance on benchmarks like VQAv2 and COCO (Table 3 in main paper), confirming it does not harm general vision-language capabilities.

In summary, DARA offers a novel and efficient solution to visual ignorance in MICL.

---

2. More Exploration Ablation on DARA, e.g., adding DARA to more than the 1st layer.

Thanks for the suggestion! We conducted ablation studies by extending DARA to multiple layers, as shown in Figure 5 in appendix. These experiments demonstrated that applying DARA to additional layers yields only comparable performance. One key reason is that the first layer serves a crucial role in establishing early cross-modal fusion and alignment. By the second layer, tokens have already accumulated entangled information from other tokens through self-attention, making it harder to differentiate between modalities. Given this trade-off between parameter count and computational cost, we opted to focus on the first layer with several attention heads as our final configuration.

---

3. Convincingness of PureMICL.

Thanks for raising this important point. We address it from both empirical and design perspectives as follows:

1. Human study validates task design. We conducted an additional user study with 10 participants to assess whether PureMICL tasks require multimodal context (table below). Without demonstrations (0-shot), participants struggled on most tasks except basic patterns like Sudoku and Palindrome. Performance improved significantly with demonstrations, confirming that prior knowledge alone is insufficient and multimodal context is essential.

2. PureMICL is unsolvable through other means such as prior knowledge or text hints. We designed PureMICL to minimize reliance on priors or textual cues. For example, in the operator induction task, even with arithmetic knowledge and text indicating a math task, the model cannot infer the correct operator without visual cues from the demo images.

3. PureMICL focuses on reasoning over multimodal context rather than low-level visual perception. When designing PureMICL avoids tasks requiring fine-grained visual cues (e.g., counting lines or detecting small changes). Instead, it emphasizes global, semantic patterns such as OCR and object shapes.

4. Performance is not limited by visual perception. Current MLLMs can generally recognize objects, numbers, and layouts when prompted, which suffices for PureMICL. If visual perception were the main bottleneck, increasing demonstrations wouldn’t help—yet performance improves significantly from 0-shot to 4-shot. This suggests the core challenge of PureMICL lies in learning and reasoning over multimodal context, rather than the visual perception.

We thank the reviewer for the comments and will include the discussion in the revision.

---

Thank you for recognizing the importance of our motivation and for acknowledging the quality of our writing and the value of our proposed benchmark. We provide detailed responses to each of your concerns below.

1. Clarifying the relationship between DARA and LoRA.

Thanks for this thoughtful observation! We address your concern as follows:

1. DARA can be seen as a constrained variant or LoRA rather than a technical equivalent. The misunderstanding likely arises from viewing DARA as equivalent to LoRA in practice. In fact, as stated in Appendix A, DARA can be theoretically seen as a special constrained variant—not a practical implementation-of LoRA. The Eq. (6) is a theoretical construct to show that $\mathbf{F}$ can be expressed via low-rank updates. However, as you rightly point out, this requires input-dependent updates and token-position-specific factors, which deviate from standard LoRA assumptions.

2. Such constraints provide practical benefit to MICL. While LoRA is more general, it can be indirect and suboptimal to rebalance the attention allocation for better MICL. In contrast, DARA's constrained formulation enables targeted visual attention modulation with far fewer parameters, leading to more efficient MICL improvement.

Thanks again for the pointing this out and we will revise the paper to clarify the theoretical nature of the proof and emphasize DARA’s practical benefits.

---

2. Using a representative LoRA setting and comparing DARA with full fine-tuning.

Thanks for raising this point! We address the concern as follows:

1. Parameter-matched LoRA reflects low-resource MICL scenarios. We agree that our LoRA setting deviates from the standard configuration. However, our goal is to simulate low-resource MICL use cases, where models should adapt quickly without heavy fine-tuning. Since DARA is very lightweight (~ 100 parameters), we matched LoRA’s parameter scale to ensure a fair efficiency-focused comparison.

2. Full LoRA underperforms on unseen tasks. While more trainable parameters normally improve the task performance, this often reduce generalization to unseen tasks. We observe that full LoRA exhibits limited transfer ability to untrained tasks, whereas DARA shows relatively better generalization as shown in Table 9 in Appendix

3. DARA complements LoRA. DARA can be integrated to full-parameter LoRA and consistently improves MICL performance across tasks and models, as shown in the table below. This suggests that DARA's hundreds of parameters significantly influence attention distribution, even when combined with LoRA's millions of parameters.

4. Full fine-tuning is costly and less practical. While full fine-tuning may achieve better performance on specific tasks, it brings significantly computational cost, lower parameter efficiency, and a hier risk of overfitting, making it less suitable option for fast-adapting MICL use cases.

We appreciate this helpful comments and we will update the paper to include more clarification and discussion.

Thank you for recognizing our motivation, DARA’s performance, the value of PureMICL, and the scope of our experiments! Below, we address each of your concerns in detail.

1. Visualization and Empirical Analysis of DARA.

Thank you for the suggestion. We have added two additional analyses as shown in this figure:

1. Attention distribution of text vs. image tokens (left). Without DARA, models focus primarily on text tokens, giving only 28% of the attention to image content. After applying DARA, we observe that attention to image tokens significantly increases to 48.7%, indicating a substantial shift toward visual content during response generation.

2. Attention heatmaps over demo and query images (right). Attention heatmap of input images further show that, without DARA, both demonstration and query images do not receive much attention, as indicated by predominantly blue regions. After tuning with DARA, attention intensifies over relevant image areas (more orange/red), confirming improved visual focus.

Thanks again and we will include these analyses in the next revision.

---

2. Hard-coded Attention Adjustment as another baseline.

Thank you for the suggestion! We address this point as follows.

1. Hard-coded attention adjustment disrupts learned priors. As each attention head is with certain learned priors from the pre-training, hard-coded modification can lead to degraded performance. We followed your suggestion and implemented a hard-coded variant where we forced specific attention heads to focus exclusively on visual tokens. Concretely, we selected the first half of the attention heads in a layer and masked out text-token positions in two settings: (1) only at the first layer, and (2) across the first 8 layers. In both cases, performance dropped to near-random, suggesting that rigid constraints disrupt the pre-trained roles of individual heads.

2. DARA enables adaptive attention modulation. Unlike manual masking, DARA introduces learnable visual amplification factors. This allows the model to adaptively reweight attention to image tokens without discarding text, thus lead to better MICL performance.

3. Future work: training-free attention adjustments. We agree that how to better understand the learned priors in each head and how to design a training-free attention adjustment is a very interesting direction for future work.

Thanks again for your thoughtful suggestion and we will include these discussions in the next revision.

---

3. Performance on SOTA VLMS and the necessity of DARA.

Thank you for this important question. We evaluated GPT-4o and observed stronger performance (see table above). However, this does not diminish the value of DARA, for the following reasons.

1. It is difficult to attribute improvements solely to model size or data scale. Closed models lack transparency, making it difficult to attribute their gains solely to scale. Their undisclosed architecture, data, and fusion strategies prevent us from isolating the source of performance improvements.

2. Scaling alone still does not solve the problem completely. When we increased the difficulty of PureMICL tasks (e.g., we designed a hard version of Sudoku as shown here, GPT-4o’s accuracy dropped significantly, suggesting that the core challenge of MICL persists even at scale.

3. DARA is not in conflict with scaling. Its lightweight design allows integration during or after training. Our results (Table 4 in the main paper) show that DARA further improves performance when combined with LoRA, and could similarly benefit models like GPT-4o.

4. DARA offers a practical and effective way to enhance MICL ability for resource-constrained open models. Though these models cannot match GPT-4o’s scale, they are also widely adopted in practice and the MICL improvement from DARA is still critical for their downstream applications.

5. DARA’s contribution is conceptually novel. To our knowledge, DARA is the first to improve MICL by targeting attention imbalance. This perspective has proven effective through improved performance and is complementary to existing strategies.

We believe these reasons make DARA still a valuable and complementary contribution to the community. Thanks again and we will add the results and discussions to the main paper in the next revision.

---

We thank the reviewer for recognizing the proposed benchmark as “convincing and meaningful” and the proposed method as “simple and effective”. We provide detailed responses for each concern below.

1. Specify the prompt design and add more information to the prompt for comparison.

Thank you for raising this point! Our text prompts are designed to be simple and minimal to avoid leakage of visual content and to examine the visual understanding ability. For instance, the operator induction task adopts the following original prompt:

System: Learn from the demos and give the answer based on the query.
User: <demo1> Question: What is the result of the following mathematical expression? Answer: 12
… (repeated for 4 demos)
<query> Question: What is the result of the following mathematical expression? Answer: 

where <demo1> and <query> specify the positions of the corresponding images. To see whether models benefit from more explicit prompting, we followed your suggestions and conducted additional experiments using stronger instruction cues, stating the concrete tasks, announcing the upcoming examples, and reminding the model to follow the answer style. This revised prompt with detailed instructions is:

System: The following image-text pairs demonstrate a consistent mathematical operation applied to numbers displayed in the images. Your task is to carefully observe each example and identify the underlying rule. Then, apply this rule to the query image to compute the correct answer. Pay close attention to how the input numbers are presented and how the answers are derived in each case.
User: Here is the demo image 1: <demo1>
The question is: What is the result of the following mathematical expression?
The answer is: 12
(repeated for 4 demos)
Here is the query where you need to answer, following the previous answers:
<query> The question is: What is the result of the following mathematical expression?
The answer is: 

Our experiments show that different prompts can produce slightly different outputs and performance, as shown in the table below. However, these prompt variations do not significantly improve overall MICL performance. While carefully crafted text prompts can help guide response formatting, they offer limited improvement in visual understanding capabilities and therefore cannot significantly enhance performance on PureMICL. Besides, as DARA is orthogonal to manual prompt design, they can be applied together in applications. Thanks for this feedback and we will add all the detailed prompts for each task in the next revision.

---

2. Definition of the “No-image” baseline.

Thanks for your question. The “No-image” baseline is designed to highlight the indispensable role of visual information in MICL, ensuring that the query cannot be correctly answered without referring to the demo images. Specifically, we remove all demonstration images without replacing them with textual descriptions. Each demonstration thus contains only the textual question and answer, followed by the query image and its question.

Replacing images with textual descriptions is certainly a possible alternative and the results largely depend on the level of detail in the descriptions. When the descriptions are general or vague (e.g., “an image with numbers”), they provide little to no benefit. In contrast, highly specific descriptions can lead to improved performance, such as those explicitly listing OCR results (e.g., “an image with a number of 2, 3, and a question mark in the middle”). Because we aim to show the task is unsolvable without looking at demo images, the “No-image” baseline does not include the textual descriptions.

We appreciate the reviewer’s insightful comment and will clarify this point more explicitly in the revised version.

---

3. Add an “Overall” Column.

Thank you very much for this valuable suggestion! We put the average performance across tasks above. The overall performance also highlights the performance boost from DARA compared with baselines. We will update the main experimental table in the revision.

---