6. Line 288 Reference to "stable $F_v$"

In Finding 7, we noted that the image reference was unclear. Specifically, "Figure 1b" refers to the left image in Figure 2 in the same RQ2 section, while "Figure 1d" refers to the right image in Figure 2 of the same section. We will make sure to update this to improve clarity of our paper.

The entire section on Finding 7 discusses the influence of "Disruptive Prompts." We describe the experimental setup regarding "Disruptive Prompts" in the first paragraph of RQ2, around line 270.

7. Ambiguous Experiment Setting in Line 295

In our paper, we refer to the setting of "original POPE queries with visual prompted images" as "Only visual prompt" to enhance the conciseness of the text and image legends. However, we recognize that this may lead to confusion. To improve clarity, we will update the reference to "Visual prompt without text reference."

8. Training Data Influence in Finding 9

Thank you for the suggestion. To verify that this effect is not specific to LLaVA’s training format, we conducted the same experiment on the Qwen model. Through the experiment, we observe that the information flow trends holds with those traced on LLaVA. Settings refer visual prompts through texts presents higher $F_v$. These results support the generality of Finding 9 across models, suggesting that the enhancement is not merely due to LLaVA’s formatting, but reflects a broader multimodal behavior.

Qwen 2.5 VL 3B result for RQ3 on POPE （external PDF Figure4）

[1] Li, Bohao, et al. "Seed-bench: Benchmarking multimodal large language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Zhu, King, et al. "LIME: Less Is More for MLLM Evaluation." arXiv preprint arXiv:2409.06851 (2024).

Dear Reviewer fJZx,

Thank you for your valuable comments. We have carefully reviewed your comments and have provided responses to each of the points you raised.

1. Proofs of Lemmas 3.3 and 3.4

We will add detailed proof of Lemmas 3.3 and 3.4 in the appendix to improve the clarity.

2. Model Diversity in Experiments

We have expanded our experiments to include Qwen 2.5 VL, a more advanced and powerful MLLM. The results are presented in the table below, with key points presenting curve trends.

Our findings indicate that the Qwen model also exhibits a four-stage information processing flow pattern, consistent with our observations from LLaVA. This alignment between Qwen VL and LLaVA confirms that the findings reported in our paper are applicable across different MLLMs, further strengthen the generality of our findings.

Qwen 2.5 VL 3B Result $F_t$（external PDF Figure2(a), Figure2(b)）

Qwen 2.5 VL 3B Result $F_v$（external PDF Figure2(c), Figure2(d)）

3. Generality of the Four-Stage Finding

As noted in response to question 2, we observe a consistent four-stage information flow pattern in Qwen, despite it being a different MLLM with distinct ViT architecture, language decoder architecture, and parameter counts. This observation helps to validate the generalizability of the findings presented in our paper.

4. Experiment Setting in Finding 3 (Caption vs. QA)

In the context of our study, we consider Caption and QA as two distinct tasks and Caption is not regarded as QA, as we did not mention Caption can also be regarded as one type of QA in our paper. In studies related with MLLM evaluation, it is a common practice to contain both Captioning and QA tasks. [1, 2] The primary objective of Finding 3 is to conduct an analysis of information flow patterns in MLLMs across these two different multimodal tasks. We select Caption and QA as they represent two critical and classical tasks in vision-language research, making their comparison particularly valuable for understanding model behaviors.

5. Potential Distribution Shift in Findings 4 & 5 (COCO-Cap vs. Concept Encoder)

Our current four benchmarks cover both in-domain and out-of-domain data, which are carefully selected to compare the information flow patterns of MLLMs across different tasks (captioning and question-answering) and different data distributions (including in-domain and out-of-domain, such as COCO with Hal-Eval and VQA with AOKVQA).

Based on such experimental design and the comparisons made across these settings, including the in-domain and out-of-domain settings, we derive several insights, including Findings 4 and 5. Specifically, we find that: (1) QA tasks rely more heavily on injected contextual information compared to captioning tasks; and (2) when considering different data sources (in-domain vs. out-of-domain, and with or without inconsistent textual prompts), information injection is less effective, particularly for out-of-domain data with inconsistent prompts.

Dear Reviewer j1jo,

Thank you for your valuable comments. We have carefully reviewed your comments and have provided responses to each of the points you raised.

1. Generalization Across Architectures

To further strengthen the generalizability of our findings, we have expanded our experiments to include Qwen 2.5 VL, a more advanced and powerful MLLM with a different architecture from LLaVA. The results are presented in the table below, with key points presenting curve trends.

We observe that the Qwen model also exhibits a four-stage information processing flow pattern, consistent with our findings from LLaVA. This alignment between Qwen VL and LLaVA confirms that the findings reported in our paper are applicable across different MLLMs.

Qwen 2.5 VL 3B Result $F_v$（external PDF Figure2(a), Figure2(b)）

Qwen 2.5 VL 3B Result $F_t$（external PDF Figure2(c), Figure2(d)）

2. Generalization Across Vision Embeddings

As mentioned previously, we further evaluated the information flow pattern in Qwen 2.5 VL, which employs a more advanced Vision Transformer (ViT) than CLIP, incorporating convolution, attention, normalization, and other techniques. [1]

The four-stage information flow pattern identified by our methods in LLaVA is also evident in Qwen, despite its different ViT and language decoder architecture. This consistency further strengthens the generalizability of the findings presented in our paper.

3. Design Implications for Future MLLMs

We have combined and analyzed the information flow patterns we traced, and we present some prospective guidance for the design of MLLM architectures and training mechanisms.

One prospective guidance on training is adding Layer-wise auxiliary losses during traning traced by our methods, such as an (1) auxilary loss term to minimize $F_v$ after the filtering stage to encourage maintaining more visual information, or (2) an auxiliary term that maximize $F_t$ - $F_v$ near the output layer in knowledge-intensive tasks which can hopefully enhance task-relevant alignment and LLM knowledge injection of MLLMs. [2]

Regarding model architecture, our findings related to RQ(3) indicate that textual tokens significantly influence visual information during MLLM inference, which may lead to visual information loss. Therefore, it may be beneficial to design new architectures that allow visual information to bypass certain layers, thereby preserving more specific visual details and potentially reducing biases in MLLMs. [3, 4]

[1] Bai, Shuai, et al. "Qwen2. 5-vl technical report." arXiv preprint arXiv:2502.13923 (2025).

[2] Wu, Junda, et al. "Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent." arXiv preprint arXiv:2502.11740 (2025).

[3] Ghatkesar, Aarti, Uddeshya Upadhyay, and Ganesh Venkatesh. "Looking beyond language priors: Enhancing visual comprehension and attention in multimodal models." arXiv preprint arXiv:2505.05626 (2025).

[4] Wang, Chenxi, et al. "Mllm can see? dynamic correction decoding for hallucination mitigation." arXiv preprint arXiv:2410.11779 (2024).

4. Comparisons to simpler diagnostic tools

We conducted experiments using several existing simple metrics to measure the information differences in hidden states between image tokens and instruction tokens. These metrics include Euclidean distance [7], cosine similarity [8], Pearson correlation [9] and JS Divergence [10].

Our observations reveal that the current simple evaluation metrics fail to (1) effectively trace changes in information flow across layers. The results of some metrics exhibites mostly monotonic increases or decreases with subtle fluctuations. For example, Cosine similarity and Pearson correlation increase gradually from layer 5 to 35 (e.g., from 0.20 → 0.59), showing almost identical trends with no distinct inflection points, thereby failing to reveal any interpretable processing stages. Other metrics such as JS Divergence degrades to a constant near output layers (near layer 35), which suggests that the metric loses discriminative power near the output. Such limitations prevent them from capturing the distinct stages identified by our methods.

(2) Additionally, these simple metrics do not provide a theoretically explainable perspective on the evaluated results. They measure the similarity of hidden states in a straightforward manner (e.g., through direct calculations of cosine or Euclidean distance), which lacks a meaningful theoretical framework when dealing with the complex semantic vectors in MLLMs. These findings further underscore the contribution of our information evaluation methods, which successfully trace more nuanced information flow patterns than simpler metrics while also offering theoretically grounded explanations.

Result of simper diagnostic tools（external PDF Figure1）

[1] Quantmeyer, Vincent, Pablo Mosteiro, and Albert Gatt. "How and where does CLIP process negation?." arXiv preprint arXiv:2407.10488 (2024).

[2] Kim, Hazel, et al. "Detecting LLM Hallucination Through Layer-wise Information Deficiency: Analysis of Unanswerable Questions and Ambiguous Prompts." arXiv preprint arXiv:2412.10246 (2024).

[3] Khanam, Rahima, and Muhammad Hussain. "Yolov11: An overview of the key architectural enhancements." arXiv preprint arXiv:2410.17725 (2024).

[4] Zhang, Shizhao, et al. "Domain adaptive yolo for one-stage cross-domain detection." Asian conference on machine learning. PMLR, 2021.

[5] Wei, Jian, Qinzhao Wang, and Zixu Zhao. "YOLO-G: Improved YOLO for cross-domain object detection." Plos one 18.9 (2023): e0291241.

[6] Tao, Mingxu, et al. "Probing multimodal large language models for global and local semantic representations." arXiv preprint arXiv:2402.17304 (2024).

[7] Alshamrani, Sultan. "Distance Matters: Euclidean Embedding Distances for Improved Language Model Generalization and Adaptability." IEEE Access (2024).

[8] Bashier, Housam Khalifa, Mi-Young Kim, and Randy Goebel. "Disk-CSV: distilling interpretable semantic knowledge with a class semantic vector." Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

[9] Nasir, Inzamam Mashood, et al. "Pearson correlation-based feature selection for document classification using balanced training." Sensors 20.23 (2020): 6793.

[10] Sutter, Thomas, Imant Daunhawer, and Julia Vogt. "Multimodal generative learning utilizing jensen-shannon-divergence." Advances in neural information processing systems 33 (2020): 6100-6110.

Dear Reviewer JoXW,

Thank you for your valuable comments. We have carefully reviewed your comments and have provided responses to each of the points you raised.

1. Mathematical Notation Ambiguity

Thanks for your feedbacks, we will update the notations to imporve the clarity.

2. Empirical Support for the Four-Stage Interpretation

We would like to mention that the four-stage view is an optional descriptive lens rather than the core contribution of our paper. It illustrates the structured dynamics observed in our empirical results, helping interpret mutual information trends across layers. The key contributions of our work lie in introducing a principled information-theoretic framework to quantify and trace multimodal information flow in MLLMs, which remain valid regardless of how one segments the observed dynamics. Besides, there are relevant studies that also offer empirical analyses with rigorous theoretical methodologies. [1, 2] Such studies highlight the importance of experimentation in validating theoretical models.

Additionally, we further perform a selective ablation experiment which corroborates our findings. Specifically, we imply intervention on image tokens by masking these tokens. The results shows that when image tokens are disabled:

(1) Information Compression (Stage 4) remains virtually unaffected, , consistent with our claim that this stage primarily involves text-response generation rather than visual processing. (2) Stages 1–3 (Pre-processing, Filtering, and Contextual Injection) exhibit significant disruption when image tokens are masked. The characteristic bimodal trend vanishes, collapsing into a monotonic decrease. This aligns with our proposal that these stages actively process visual information whose dynamics are dependent on image tokens.

Qwen 2.5 VL 3B intervention on VQA（external PDF Figure3）

3. Evaluation Scope and Baselines

We have expanded our experiments to include Qwen 2.5 VL, a more advanced and powerful MLLM. The results are presented in the table below, with key points presenting curve trends.

We observe that the Qwen model also exhibits a four-stage information processing flow pattern, consistent with our findings from LLaVA. This alignment between Qwen VL and LLaVA enhances the generalizability of the findings reported in our paper.

Qwen 2.5 VL 3B Result $F_v$（external PDF Figure2(a), Figure2(b)）

Qwen 2.5 VL 3B Result $F_t$（external PDF Figure2(c), Figure2(d)）

These 80 number of classes in our evaluation methods are not tied to COCO, which are actually extracted leveraging YOLO's anchor mechanism. YOLO's anchor mechanism has reliable cross-domain adaptability [3,4,5], which can be extended to accommodate various downstream evaluation tasks.

Additionally, our choice of 80 classes from standard YOLO aligns with common practices in vision-language research, where such categories serve as a standardized semantic bottleneck for fair comparison across different datasets and settings. [6]

Dear Reviewer WuoZ,

Thank you for your valuable comments. We have carefully reviewed your comments and have provided responses to each of the points you raised.

1. Technical Contribution and Novelty

Our motivation generates from the triger of the CBM as you mentioned in [1], however our theoretical and technical details are novel.

Teoretically, our main contribution is the proposal of two information evaluation perstective for MLLM information injection combining the Upper Bound for Mutual Information Upper Bound.

2. Research Question Scope

We have further specified our research questions to enhance their alignment with our findings and ensure they can be more effectively addressed by the findings. For example, in RQ2, our finding mainly focus on the influence of "disruptive prompts", we modify the RQ2 as "How do disruptive textual instructions influence multimodal information flow in MLLMs?" to specify a clearer question scope. For RQ4, we can modify the RQ as "How does key words in different instructions influence MLLM knowledge injection?" to specify a clearer question scope.

3. Model Choice and Benchmark Sufficiency

We have expanded our experiments to include Qwen 2.5 VL, a more advanced and powerful MLLM. The results are presented in the table below, with key points presenting curve trends.

Our findings indicate that the Qwen model also exhibits a four-stage information processing flow pattern, which is consistent with our observations using LLaVA. This consistency between Qwen VL and LLaVA reinforces the validity of our findings across different MLLMs.

Qwen 2.5 VL 3B Result $F_v$ （external PDF Figure2(a), Figure2(b)）

Qwen 2.5 VL 3B Result $F_t$（external PDF Figure2(c), Figure2(d)）

Regarding the benchmarks, we have carefully selected them to encompass various scenarios and facilitate meaningful comparisons. And many related works also use these benchmarks. [1, 2, 3] This carefully designed benchmark selection has yielded valuable insights. Specifically, the benchmarks cover different tasks (captioning and other assignments) and diverse data sources (both in-domain and out-of-domain). Through our comparisons across these benchmarks, which feature varying settings with comparable complexity, we have identified several key findings. Notably, findings 3 and 4 highlight different information patterns across various visual tasks, while finding 5 reveals distinct information patterns influenced by in-domain and out-of-domain data, as well as inconsistent textual inputs.

[1] Tao, Mingxu, et al. "Probing multimodal large language models for global and local semantic representations." arXiv preprint arXiv:2402.17304 (2024).

[2] Nguyen, Duy-Kien, and Takayuki Okatani. "Multi-task learning of hierarchical vision-language representation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[3] Huo, Jiahao, et al. "Mmneuron: Discovering neuron-level domain-specific interpretation in multimodal large language model." arXiv preprint arXiv:2406.11193 (2024).