Efficient Multi-modal Long Context Learning for Training-free Adaptation

Q1: Disparity between the pre-trained MLLM dataset and adaptation tasks. In [1], the paper demonstrates that task disparity might have a huge impact on vision task adaptation; would this be the case for in-context learning of MLLM?

A1: We appreciate the reviewer’s insightful question on task disparity in MLLM adaptation. The great work [1] highlights the impact of task disparity on vision adaptation, which inspires us to explore influence of task disparity on in-context learning of MLLMs. To evaluate this, we conducted experiments on ImageNet100, OK-VQA, and MedXpertQA.

Table R4.1: Impact of task disparity on EMLoC

For ImageNet100 and OK-VQA, where the MLLM has seen similar data and tasks during pre-training, EMLoC effectively adapts with limited data (200 and 20 examples, respectively). Full fine-tuning struggles on OK-VQA, probably due to the overfitting with only 20 examples. In contrast, all methods fail on MedXpertQA, a complex multi-choice medical dataset, where the baseline model shows poor performance, indicating no prior knowledge in this domain. With only 20 examples, adaptation remains ineffective. These results suggest that EMLoC could efficiently adapt pre-trained knowledge with minimal data, but struggles when entirely new capabilities are required. This finding aligns with the conclusion in [1]. When there is a significant discrepancy between the pre-trained and downstream tasks, adaptation can be considerably more difficult.

Q2: More PEFT fine-tuning methods should be included for completeness.

A2: We appreciate the suggestion. In response, we compare EMLoC with VPT [1], E²PT [2], and M²PT [5] across three multi-modal tasks.

On ImageNet100 with 200 training examples, M²PT achieves the best performance due to its strong adaptation capacity, leveraging three different adapters(textual/visual prompts and multi-modal projector). However, when the number of training samples is reduced to 20 (as in MME-RW and OK-VQA), M²PT tends to overfit due to its increased number of optimized parameters. VPT, which optimizes only the visual prompts of the visual encoder, has fewer trainable parameters and a lower optimization capacity, leading to a weaker performance across all three tasks. However, its limited parameter tuning helps to mitigate overfitting risks to some extent. With visual prompt tokens and the powerful shared KV prompt tokens, E$^2$PT achieves better fine-tuning performance than VPT. Following M²PT, the number of visual prompts and textual prompts are 20 and 10, respectively. The learning rate is set to 7e-4. The number of KV prompt token is 5. For Imagenet100, we optimize 5 epochs with 125 steps. For MME-RW and OK-VQA, we just fine-tune 25 steps.

As a training-free method, EMLoC achieves competitive performance across various tasks and lowers the risk of overfitting. This makes it effective in low-data regimes, where traditional fine-tuning methods may struggle. These methods will be compared in our revised manuscript. We will also release the code of these PEFT methods in LLaMAFactory for the research community.

Table R4.2: Comparison with other PEFT methods on multi-modal benchmarks

Q3: The advantages of our EMLoC.

A3: EMLoC is a training-free adaptation method designed for multi-modal long-context learning, effectively leveraging the strong capabilities of pre-trained MLLMs and offering a reduced risk of overfitting. This aligns with the growing trend in the test-time scaling era.

To the best of our knowledge, EMLoC is the first multi-modal KV-cache pruning method to achieve performance comparable to that of fine-tuning methods, while maintaining a high compression ratio. As shown in Table R2, EMLoC surpass the second best method PyramidInfer by 5.3% in accuracy with a less retention ratio. In addition, EMLoC can also facilitate the online long-video understanding. In Table R1.2, EMLoC reduces context length from 27.9k to just 2.3k, LLM FLOPs from 554.8T to 272.0T, and inference time from 7 hours to 5 hours, while preserving a consistent accuracy (60.1 vs. 60.3).

Our adaptive pruning strategy provides valuable insights into multi-modal pruning and the importance of different layers in MLLMs in Fig.4 and Fig.5, which may inspire further exploration in multi-modal pruning techniques.

Q1: There are too many hyperparameters (retention ratio, JS threshold) that need to be found heuristically.

A1: Thanks for the comments. Those parameters have clear meanings and are easy to adjust. For a high compression ratio, we can set a smaller retention ratio and a higher JS threshold $\delta$, and the optimal pruning strategy will be identified heuristically. Our method avoids manually adjusting numerous parameters like FastGen or PyramidInfer. Our experiments also show that the default hyperparameters are stable across different tasks, as seen in following two tables.

Table R3.1: $\delta$ across tasks

Table R3.2: retention ratios across tasks

Q2: The iterative algorithm can save memory usage, but I do not think this is computationally efficient.

A2: While the iterative algorithm in EMLoC introduces a modest adaptation time(~144 s), this cost is significantly smaller compared with its gains in inference efficiency and performance. As depicted in L326-328 and Table R2, on ImageNet100, EMLoC reduces inference time from 31 minutes of MLoC to 18 minutes, and memory usage from 19G to 17G without accuracy degradation. Compared to the in-context learning method RICES in Table R1.1, EMLoC reduces inference time from 5 hours to 18 minutes and memory from 43G to 17G. Besides, PyramidKV sacrifice 16.1% accuracy to achieve faster adaptation(55s vs 144s).

To further address the computational concerns, EMLoC offers flexibility:

1. Group-wise pruning: By grouping adjacent layers (e.g., 2 layers per group), adaptation time drops to 85 seconds (a 40% reduction), with a slight accuracy drop.
2. Decoupled adaptation/inference: Adaptation needs a one-time cost per task, while the pruned KV cache may be repetitively used for thousands of times during inference.

Meanwhile, as shown in Table R1.2, EMLoC is also an efficient method in online long-video understanding, reducing the total LLM time from 7 hours to 5 hours, and peak GPU memory from 38G to 24G.

Q3: Why compare EMLoC with LoRA and full fine-tuning in Table 3. The comparison with LoRA and full fine-tuning for adaptation time seems less convincing. Comparison with other context compression methods and MLoC for time and memory consumption.

A3: Results in Table 3 and Table R1.3 demonstrate that EMLoC is comparable to fine-tuning methods on multiple multi-modal tasks. Those comparison demonstrate that our training-free EMLoC can achieve comparable performance with fine-tuning which needs extra training iterations. Therefore, EMLoC is a promising adaptation method with better efficiency.

Our EMLoC is an efficient adaptation method that makes long-context learning practical even on consumer GPUs, with minimal inference cost. We further compare the time and memory consumption of various compression methods in Table R2 (Reviewer 6UND), where our method shows clear advantages in both inference efficiency and accuracy. For additional analysis, please refer to A2 of Reviewer 6UND and A2 of Reviewer Niy5.

Q4: How did you train the model parameters for the context compression? Is it just fine-tuning the model for ImageNet?

A4: EMLoC is a training-free adaptation method that adaptively searches for the optimal pruning strategy under a JS divergence constraint, without fine-tuning the pretrained MLLM. For each task, a distinct pruned KV cache is generated and loaded into memory during inference, eliminating the need for model retraining or redeployment.

Q5: Figure 1 seems less intuitive. Rather than using demonstration examples on the X-axis, why not use context length?

A5: Thanks for the suggestion! We have revised Figure 1 by replacing the X-axis with context length, making it more intuitive in illustrating our advantages in both performance and efficiency.

Q6: The details of LoRA and full fine-tuning should be more elaborated in the Appendix than in the current manuscript.

A6: In LoRA adaptation, we apply LoRA adapters to all linear modules of the LLM, including qkv_proj, out_proj, up_proj, and down_proj, while keeping the vision encoder and multi-modal projector frozen. The rank and alpha are set to 16 and 32, respectively. In full fine-tuning, only the LLM is fine-tuned with DeepSpeed ZeRO-3, leaving other parameters frozen. Other unspecified settings follow the default configurations in LLaMAFactory.

Q1: Each chunk contains several examples as shown in experiment details. How does the author ensure each example has the same length?

A1: Thank you for your comment. In ImageNet100, 200 multi-modal examples are evenly divided into 10 chunks. Each image (224×224) is encoded into approximately 64 tokens (may vary slightly due to dynamic aspect ratios), and the corresponding question-answer pair adds around 20 tokens, resulting in about 80 tokens per example. Each chunk contains 20 examples (roughly 1.6k tokens). The system prompt appears only at the start of the first chunk. Below is an example structure:

# Start of 1st chunk
<|im_start|> system\n You are a helpful assistant.<|im_end|>
## sample 1
<|im_start|> user\n <|vision_start|> <Image1.jpg> <|vision_end|> What category does the image belong to? <|im_end|>
<|im_start|> assistant\n <class 1>. <|im_end|>   ...
# Start of 2nd chunk
## sample 21
<|im_start|> user\n <|vision_start|> <Image21.jpg> <|vision_end|> What category does the image belong to? <|im_end|>
<|im_start|> assistant\n <class 11>. <|im_end|> ...

For other image benchmarks, each image is encoded into 256 tokens(448×448 resolution). Each chunk has 4 examples, resulting in a chunk size of 1.1k–1.6k. For the YouCook2 video benchmark , each video with 8 frames is encoded into 1024 tokens, with 4 videos per chunk, yielding a 4.7k chunk size. If sample lengths vary significantly, we use a greedy algorithm to progressively fill each chunk up to a maximum size.

Q2: The key contribution is the context compression technique while the author didn't discuss its relation/advantage over KV-cache algorithms such as PyramidInfer and FastGen. How about the performance/efficiency improvements with other KV cache methods?

Table R2: Comparison with other context compression methods on ImageNet100

A2: Thanks for the comments. In Section 4.3 and Table 4 of the original paper, we compared our adaptive EMLoC with two static KV-cache algorithms. Table R2 extends this comparison (Table 4) by including PyramidInfer and FastGen. Most KV-cache methods focus on uni-modal text compression, but fail to maintain original performance with a high compression ratio. EMLoC retains only 22.4% of tokens while achieving 63.7% accuracy, outperforming FastGen (49.3% accuracy with 36% tokens) and PyramidInfer (55.6% accuracy with 24.6% tokens). Unlike existing KV-cache methods, EMLoC effectively maintains the full-context performance while significantly reducing the context length, thus improving efficiency.

To optimize the trade-off between adaptation cost and inference performance, we explore increasing the chunk count (10 → 20) and a group-wise strategy (every two layers share the same retention ratio). This variant, EMLoC*, reduces adaptation time from 144s to 85s and memory from 38G to 24G, at the cost of a slight accuracy degradation(63.7 → 60.9) and a higher retention ratio (22.4% → 27.6%). This allows for a flexibility implementation on computation constrained scenarios. The adapation cost is significantly smaller compared with its gains in inference efficiency. More discussion can be seen in the response to Q2 from Reviewer Niy5.

Q3: Can this method be applied alongside model parallelization?

A3: Yes, the proposed method is compatible with model parallelization. EMLoC only compresses the key-value cache of the context at each layer, and it does not alter the model weights or architecture. Modern model parallel frameworks, using pipeline or tensor parallelism, can distribute the KV cache across devices, allowing for parallelization techniques during EMLoC's adaptation or inference. Varying KV-cache lengths across layers may cause the computational or communication imbalance. To mitigate this, we can manually split the model based on the pruned KV-cache lengths, or automatically share free GPU resources with other models using Multi-Process Service (MPS)[1] or Transparent GPU Sharing (TGS)[2].

[1] NVIDIA MPS: https://docs.nvidia.com/deploy/mps/index.html [2] Transparent GPU sharing in container clouds for deep learning workloads. (NSDI 23)

Q1: Comparison with other multi-modal in-context learning methods in Table 1.

Table R1.1: Comparison with multi-modal in-context learning methods

A1: Thanks for the suggestion! We have compared EMLoC with two other open-sourced multi-modal in-context learning methods, RICES[1] and MTV[2], on ImageNet100, MME-RW, and OK-VQA in Table R1.1. RICES retrieves the top 1/4 most relevant in-context samples from all samples. MTV extracts the mean activation of in-context examples as task vectors and finds the optimal replacement position of these task vectors. During inference, MTV replaces these task vectors at the optimal position of the test sample, which fails to facilitate these tasks. Our EMLoC achieves better average performance across the three benchmarks. It's worth noting that RICES is an online retrieval-augmented method, so it needs to forward the retrieved long context during each inference step. RICES takes 5 hours inference time and 43G memory cost on ImageNet100, while our EMLoC requires only 18 minutes with 17G memory, showing clear advantages in efficiency.

[1] Flamingo: a visual language model for few-shot learning (NeurIPS 22)

[2] Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning (NeurIPS 24)

Q2: Comparison with LongVA and experiment on long-video benchmarks.

Table R1.2: Comparison with LongVA and MLoC on VideoMME w/o subtitles with 384 frames

A2: Thank you for the insightful suggestion. As required by the reviewer, we have conducted experiments on the long-video benchmark VideoMME without subtitles, using 384 frames per video. In LongVA, each frame consists of 144 tokens, whereas in Qwen2-VL, 144 tokens represent every two frames through temporal pooling. Compared to our baseline MLoC, EMLoC significantly reduces computational overhead while maintaining nearly the same accuracy. Specifically, EMLoC reduces the average context length from 27.9k to just 2.3k tokens, LLM FLOPs from 554.8T to 272.0T, inference time from 7 hours to 5 hours, and peak GPU memory from 38G to 24G, while preserving a consistent accuracy (60.1 vs. 60.3).

To achieve this efficiency, we set $\delta$ =0.04 and configured the retention ratio to [0.02, 0.1, 0.5, 1.0]. Instead of optimizing the retention ratio for each layer individually (layer-wise), we adopt a group-wise strategy, where every 14 layers are treated as a single group and share the same retention ratio. This allows for a more stable and efficient selection process during online inference. Under identical setup(384 frames at the same resolution), both MLoC and EMLoC outperform LongVA while requiring significantly fewer computations. EMLoC also enables the real-time long-video understanding on consumer-grade GPUs such as the NVIDIA 3090, making it a more practical solution for real-world applications.

Q3: Expand evaluation to more diverse multi-modal benchmarks.

Table R1.3: Comparison with fine-tuning methods on more multi-modal benchmarks

A3: We appreciate the reviewer’s suggestion for further benchmarking. To this end, we have compared EMLoC with LoRA and full fine-tuning on additional multi-modal benchmarks(MME-RW and OK-VQA) as seen in Table R1.3. Our EMLoC shows comparable performance to LoRA and full fine-tuning in a training-free manner. This flexibility is essential in real-world applications, where fine-tuning may not always be feasible. The optimization steps and hyperparameters are the same as depicted in the Appendix C.1.