M²IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering

Thank you for the nice comments and valuable suggestions. Below we offer clarifications to the concerns you raised.

Q1: Could you provide more intuition behind the synergistic loss?

R1:

• Intuition: Our goal is to distil multimodal in-context sequences into a compact vector set that preserves, and in practice often surpasses, the performance obtained when the model receives demonstrations explicitly. This objective imposes two simultaneous demands on the vectors: (i) they must compress the sequences to reduce redundancy, and (ii) they must still expose the fine-grained semantic cues that guide LVLMs to recognise the task implicit in the demonstrations. These requirements appear contradictory because aggressive distillation usually erases precisely the subtle relations and attributes on which task recognition depends, and relying solely on a mimicry loss based on output distributions is unlikely to address this issue thoroughly. Prior works on multimodal ICL suggests that in certain complex scenarios the context is more inclined to provide task recognition signals about what task should be executed, while exhaustive scene details for task learning may contribute little to downstream reasoning. Inspired by these insights, we envisioned a straightforward solution: first select information from the sequences and discard most unnecessary parts, then perform fine-grained distillation on the remaining information. To implement this in the model’s latent space, we take note of the complementary inductive biases of MHA and MLP. We aim to partially decouple their functions through explicit intervention during processing, thereby realizing a filtering-followed-by-distillation pipeline. Accordingly, we introduce the synergistic loss that enables the attention mechanism to select salient relational information while guiding the MLP to refine those cues into a denser, task-aware encoding, thus performing selection and fine-grained distillation in concert.
• Mechanisms: To enforce coordination we build, at each layer, the normalised cross-view correlation matrix $M^l$. The synergistic loss presses the diagonal of this matrix toward one, forcing each attention axis to remain colinear with its MLP counterpart. Attention therefore acts as a focused gating mechanism that picks out the relational features most indicative of the latent task, and the MLP is obliged to refine primarily those same features instead of drifting to unrelated content. Because the two branches now speak in a common semantic basis, their residual contributions add constructively, preserving the evidence needed for accurate task recognition during inference. The loss simultaneously drives every off-diagonal entry toward zero. By penalising cross-axis correlation we prevent either branch from encoding information unrelated to core semantics. This decorrelation enlarges the effective subspace spanned by the combined outputs, so attention can continue to represent where and with whom tokens interact while the MLP can devote its full capacity to how their attributes support the hypothesised task. The joint pressure of diagonal alignment and off-diagonal suppression therefore realises our intended pipeline inside the model’s latent space: selection occurs when attention gates the task-relevant relations, and fine-grained distillation occurs when the MLP sharpens those relations into a compact yet semantically rich representation.

Q2: How well do these linear decompositions capture the actual behavior within complex, non-linear LVLMs?

R2: Theorems 1 and 2 aim to demonstrate how MHA and MLP process demonstrations differently, enabling their unique roles in ICL. The decomposition is not intended to fundamentally prove why MHA handles broader semantics and MLP handles fine-grained details, but rather to show how these components can be utilized, each exploiting its unique properties, to embed in-context sequences as vectors within the LVLMs.

Our approach assumes a linear MLP for simplicity, focusing on the linear part $xW_{1}$ while initially disregarding the non-linear activation and subsequent linear transformation. However, this simplification does not compromise the generality of our findings. In reality, standard MLP blocks with non-linear activations can be represented as $\text{MLP}(x)=\sigma (xW_{1})W_{2}$, where $\sigma$ is the activation function. Even when incorporating the activation function $\sigma(\cdot)$ and the $\text{MLP}(\cdot)$ operation on both sides of the equation, our conclusions remains valid. Applying it to Theorem 2, we obtain: $\text{MLP}(a^i_{\mathbf{C},\mathbf{Q}}) = \zeta_i \cdot \sigma(a^i_{\mathbf{C}} W_1) W_2 + \eta_i \cdot \sigma(a^i_{\mathbf{Q}} W_1) W_2$. This is because these elements can be seamlessly integrated into a more comprehensive model framework, thereby preserving the essential characteristics of the model's behavior.

Thanks for your constructive suggestions. Your endorsement of our method and experiments gives us significant encouragement. Here are our clarifications.

Q1: M2IV requires task-specific training, which partially undercuts the appeal of ICL as a parameter-free adaptation method.

R1: Your question goes to the heart of our contribution. ICL is valued not just because it avoids parameter updates, but because its efficiency. Moreover, our study is grounded in ICL, but it focuses on representation engineering. This line of work seeks to convert many explicit prompts into a compact internal vector that guides the model without being shown at inference time. Our goal is to devise a representation-engineering method that boosts accuracy in the challenging multimodal ICL setting and surpasses existing static techniques. In this context, a small amount of task-specific additional training is necessary, yet it does not dilute the efficiency that make ICL attractive to users. In the paper we articulate this point from two angles:

• The Necessity of Training: Existing training-free methods have been shown to be insufficiently effective in complex multimodal ICL tasks, as they are typically confined to purely textual tasks with small label spaces and relatively homogeneous task types within a sequence. Equally limiting, a fixed projection is tied to the single sequence from which it is computed and thus cannot encode knowledge dispersed across a larger knowledge base, a capability that real applications routinely require. Untrained representations cannot accurately or fully capture the content of a dataset. Only through specialized training can information from numerous examples be effectively compressed into a single vector that meets usability standards.
• Training and Inference Efficiency of M²IV: This is the issue we highlight in the first paragraph of Section 4.3. We demonstrate M²IV’s efficiency using two visualizations, one for inference cost and one for total runtime. These plots show that altering certain parameters or introducing training does not reduce efficiency compared to vanilla ICL. On the contrary, in scenarios requiring multiple ICL, M²IV achieves remarkable efficiency improvements over vanilla ICL. Moreover, as the dataset grows, M²IV’s additional training time increases far more slowly than the inference time penalty incurred by appending more examples to the model input. These results demonstrate that on moderate-sized benchmarks and in large-scale real-world tasks, M²IV improves LVLM performance and achieves significant efficiency improvements. As modern users increasingly prioritize efficiency, M²IV doesn't undercut the appeal of ICL but also delivers greater practical value.

Q2: The approach, though efficient at inference, introduces a non-negligible training overhead and relies on curated training data and clustering pipelines.

R2: As we noted earlier, M²IV is not only efficient in inference but its training overhead is not 'non-negligible' and does not affect its overall efficiency. Moreover, because M²IV requires training and our goal is to minimize the data needed, high-quality training data and a dedicated preprocessing pipeline are essential.

• Dilutable training overhead: Figure 5 compares, for each benchmark, the total time needed for M²IV (training plus inference) with the inference time of vanilla ICL and shows that, after adding the training phase M²IV is still faster overall. In real-world applications this advantage becomes more pronounced, because the inference cost of vanilla ICL rises sharply as the knowledge base grows and larger context windows include more demonstrations, whereas M²IV incurs its training expense only once. Consequently, the training of M²IV does not diminish its overall efficiency.
• Reduced data requirement: Experiments in Appendix K demonstrate that, for M²IV training, data quality is more important than sheer quantity. With carefully curated samples, M²IV already delivers strong performance, and adding too many examples can even cause a slight drop in accuracy. This outcome confirms that our layer-wise steering vectors for the MHA and MLP branches capture the distribution of the full knowledge base more effectively than simply enlarging the dataset. The result is crucial for representation engineering, where many existing approaches overlook the search for optimal training data. Notably, compared with LIVE our method achieves better accuracy while using 76 percent fewer training examples. The saved training budget partially offsets the cost of data processing. Furthermore, our pipeline is straightforward to implement because it relies only on pretrained CLIP encoders and unsupervised k-means clustering.

Thank you for the time, thorough comments, and nice suggestions. We are pleased to clarify your questions in detail.

Q1: Model-specific datastore overhead Does the proposed method require constructing a separate datastore module for every backbone, simply to maintain a consistent representation space?

R1: Thank you for this important question. Below, we address the concern in detail:

• It is not necessary to construct a separate vector datastore module for each backbone. Our VLibrary is designed solely for storing and indexing the vectors learned during training. It performs no inter-vector operations, so maintaining a consistent representation space across different backbones does not require restructuring the datastore. As a result, the structural requirements of VLibrary remain minimal and unchanged across models.

• The system is lightweight, modular, and does not introduce deployment overhead. In practice, VLibrary is implemented using an off-the-shelf object store such as AWS S3, which integrates easily into existing pipelines. During retrieval, the system queries using the M²IV’s designated parameter set, $\alpha^{a}$ (line 199). This architecture avoids the overhead of rebuilding datastores and instead supports straightforward, scalable deployment.

To further clarify this, we provide the engineering details of our VLibrary Workflow implementation below.

• Storage Backend: The VLibrary is hosted on AWS S3, where each M²IV asset is a structured binary object containing the learned vectors ($V^a$, $V^m$) and associated scalars ($\alpha^a$, $\alpha^m$) for every LVLM layer. We serialize each asset using Protocol Buffers to preserve floating-point precision, then compress it with Zstandard before uploading.

• Addressing and Deduplication: To guarantee uniqueness and support lifecycle management, we use content-based addressing. Specifically, we normalize the index parameters $\alpha^a$ by enforcing a fixed precision and deterministic layer ordering. We then compute a SHA-256 hash, referred to as the M2IV_Content_Hash, which serves as the S3 object key.

• Lookup and Retrieval: For efficient access, a Redis-based Mapping Service maintains associations between human-readable versioned identifiers (e.g., model_name@v1.2:task_name@v1.0) and their corresponding M2IV_Content_Hash. When an application requests a specific M²IV, it provides the versioned keys, retrieves the hash from Redis, and uses it to fetch the binary object from S3. The object is then decompressed and deserialized back into a usable in-memory M²IV structure. We further accelerate frequent queries through application-level caching.

In summary, this architecture ensures low-latency lookups, scalable and cost-effective storage, and avoids model-specific datastore duplication, facilitating easy integration and robust real-world deployment.

Q2: Handling of very long text in CLIP The paper does not explain how CLIP (or any vision-language encoder) is adapted to cope with overly long textual prompts.

R2: Within the M²IV pipeline we use CLIP only during the clustering step that constructs the training sequences. In our main experiments the textual parts of every VQA benchmark remain within 77 token limit, so we rely on the vanilla CLIP. In the section of VLibrary, we may encounter inputs, such as extended rationales, that exceed this limit. To avoid extra training cost and potential semantic loss we directly switch to LongCLIP[1], a well-trained variant that adapts CLIP to longer text. In addition, we experimented with three alternative methods: (i) truncating the sequence and feeding the shortened version to vanilla CLIP; (ii) applying a sliding-window segmentation with max_len = 77 tokens and stride = max_len//2; and (iii) first using an advanced closed-source LLM to summarize the text within the length limit and then embedding the summary with CLIP.

Table 1*: Comparison of different long-text processing methods in explainability, instruction-following, and jailbreak. We experimented with multiple prompts for GPT and Gemini and this table reports the best results.

Table 1* shows LongCLIP is best. Since embeddings are only used for clustering, long-text methods have limited impact, and only some inputs need them, so overall performance stays stable. To reduce training costs and avoid additional LLM API calls for summarization, LongCLIP is an excellent choice.

We sincerely appreciate the time and effort you devoted to reviewing our manuscript. Below, we provide detailed responses to each of your concerns.

Q1:

Provide more fine-grained analysis or visualizations to demonstrate how M2IV captures interdependencies among demonstrations and supports semantic distillation. Notably, as our methods distills in-context sequences into vectors and injects them directly into the zero-shot generation process, we cannot obtain visualizations of how it adjusts inter-demonstration attention in few-shot generation.

R1:

We add a fine-grained ablation study that isolates the distinct contributions of the MHA and MLP branches within the M²IV architecture.

• Branch-Isolation Ablation: Motivated by[1], which categorizes multimodal ICL tasks based on task mappings into specific-mapping and generalized-mapping tasks, we conduct branch-isolation ablations on each type. In specific-mapping tasks, all demonstrations and the query sample share highly similar task mappings, which is the focus of existing representation engineering methods, which emphasize MHA. By contrast, generalized-mapping tasks exhibit fine-grained variation between demonstrations and between each demonstration and the query; this variation becomes more pronounced across the entire knowledge base and demands fine-grained multimodal features. We select VL ICL and a new task HatefulMemes[2] as representative specific-apping tasks, while VizWiz and A-OKVQA serve as examples of generalized-mapping tasks. After training M²IV, we consider three scenarios: full injection, injection of only the MHA branch, and injection of only the MLP branch.

  Injection	VL-ICL	Hatefulmemes	VizWiz	A-OKVQA
  Full	33.36	84.62	43.40	53.59
  MHA	30.39	81.47	36.71	48.90
  MLP	22.67	73.58	34.07	46.18

  Table 1*: Comparison of M²IV performance under different injection methods.

  Table 1* shows that MHA remains fundamental for multimodal ICL because it identifies the core task and processes the relationships between demonstrations and the query sample. Removing MHA causes a significant performance drop across all tasks. In contrast, omitting MLP leads to a larger accuracy decline on generalized‐mapping tasks than on specific‐mapping tasks, demonstrating MLP’s essential role in fine‐grained semantic distillation.

• Fine-grained Semantic Fidelity: To further validate the role of the MLP branch for fine-grained semantic distillation, we enhance the experiments in Section 5.2 which use the LLaVA benchmark to assess instruction following abilities. We augment every demonstration in the original support set with subset-specific markers: Conversation answers begin with [Sure, here’s what I found.] and end with [Feel free to ask more!]; Detail descriptions weave connective adverbs such as [In particular,] [Additionally,] [Moreover,]; Complex-reasoning demos follow a step-by-step format [To address the question,] … [First] … [Second] …… [To conclude] …. (During evaluation, we also accept synonyms of those words, provided they are enclosed in []). At this stage, the LVLM’s task shifts from relatively open-ended instruction following to generating fixed-format responses with fine-grained constraints on how it conveys image content. Meanwhile, evaluation extends the original GPT-4 scoring by also measuring each genrated answer’s keyword hit rate and the correctness of the markers’ prescribed order. Using the untouched query split, we train M²IV under the same training protocol as the original study. We examine the performance differences between the original and this fine-grained settings across two injection methods.

  Setting	Injection	Conversation	Detail	Complex
  Original	Full	72.53	56.41	75.32
  	MHA	68.86	52.98	70.43
  Fine-grained	Full	67.71	50.36	64.89
  	MHA	59.29	41.77	55.91

  Table 2*: Comparison of M²IV performance under full injection versus injection without the MLP branch.

  Results in Table 2* show that in the fine grained setting, performance drops more severely without the MLP branch. Therefore, to overcome coarse grained distillation and achieve fine grained distillation of task specific semantic information (such as adapting style to a particular response format) the MLP is essential. Moreover, the synergistic loss unlocks the MLP’s capacity. This is one of the key factors behind M²IV’s success in outperforming existing methods.