Thanks for the constructive comments and suggestions. Below please find our clarification regarding your concerns.

---

[Q1] Definition of parametric and contextual knowledge.

[A1] Your understanding of parametric and contextual knowledge is correct, and this distinction aligns with our definition and is widely adopted in the LLM literature [1] [2].

Regarding Figure 4, we would like to clarify a possible misunderstanding: in the upper part of the figure, the logits represent parametric knowledge, which does not have access to the context (depicted by the absence of blue squares). In contrast, the lower part illustrates contextual knowledge, where the context (blue squares) is visible. This setup is consistent with the definitions of the two types of knowledge as used throughout our paper.

We will revise the figure caption and description in the paper to make this distinction clearer.

---

[Q2] Explanation about Equation 11.

[A2] Thank you for your question. We will add more details in the revised manuscript to clarify the intent and components of Equation 11. Specifically:

1. $logit_c$ and $logit_p$ represent the logits derived from contextual knowledge and parametric knowledge, respectively.

2. $\lambda_{v}^{t}$ and $\lambda_{c}^{t}$ denote the summed attention weights over image tokens and context tokens at decoding step $t$. These values reflect the relative importance of the image and context at that time step. We will revise the explanation in the revised paper to improve the clarity and interpretability of this equation.

---

[Q3] Question about Figure 6.

[A3] Your statement is correct and aligns well with the key contribution of our paper: accurately identifying which context tokens to attend to is essential, and this is precisely what our model is designed to achieve.

As shown in Equations (6) and (7), we compute the relevance between the question and each context token and use it as a guiding signal for attention reallocation. Such a design allows the model to assign higher attention weights to context tokens that are more semantically aligned with the question.

In the case illustrated in Figure 6, our model correctly focuses on the token "Japan", as it appears near the word "Park", which is also a keyword in the question. This localized relevance enables the model to ground the answer more accurately. In contrast, LLaVA with MRAG or CAD, which lacks such fine-grained context control, tends to over-rely on visual cues (e.g., the European-style building in the image), leading to incorrect predictions like "Holland" or "Netherlands".

In summary, our approach enables fine-grained control over context-level attention with question-context similarity, which dynamically reweights attention and reduces interference from less relevant image features. This leads to better knowledge utilization and more accurate predictions, which is aligned with your observations. We will revise the text to better highlight this point and clarify this contribution.

---

[Q4] Y-axis of Figure 2.

[A4] Thank you for the insightful question. In this figure, we first sum the attention weights assigned to image tokens and context tokens separately, and then normalize these two values so that their sum equals one. The normalization allows us to reflect the relative proportion of attention allocated to each type of token, rather than their absolute magnitudes. We referred to the Y-axis of Figure 2 as a “distribution” because it represents a normalized split of attention across the two modalities. To avoid confusion, we will revise the y-axis label to a more precise term like 'Normalized Attention Weights', and clarify the computation process in the main text.

---

[Q5] Missing Limitation section.

[A5] We apologize for the confusion. Due to space constraints, the Limitation section was placed in the Appendix. In the revised version, we will move it into the main paper to ensure better visibility and completeness of the discussion.

---

Reference

[1] Knowledge Conflicts for LLMs: A Survey.

[2] Mitigating Knowledge Conflicts in Language Model-Driven Question Answering.

Thank you for the constructive comments and suggestions. Below please find our clarification regarding your concerns.

---

[W1] Applicability across a broader spectrum of vision-language tasks.

[R1] We appreciate the reviewer’s valuable suggestion. As suggested, we conducted additional experiments on two vision-language tasks using LLaVA-1.5 and InstructBLIP: image captioning and multimodal hallucination mitigation.

For image captioning, we selected 500 images from the COCO dataset and evaluated caption reliability using the CHAIR metric (lower is better) and caption detailedness using Recall (higher is better). For multimodal hallucination mitigation, we conducted experiments on the POPE dataset, with accuracy and F1 score measuring the model’s ability to reduce hallucinations.

Evaluation on image captioning

Evaluation on multimodal hallucination mitigation

As shown in the tables, our approach consistently improves performance across all metrics and tasks by effectively retrieving similar images and relevant descriptions. These new experiments further highlight the versatility and robustness of our method in enhancing vision-language models to generate more reliable and detailed outputs.

---

[W2] Missing references.

[R2] We appreciate the reviewer’s valuable suggestion. We will include the recommended paper along with other recent relevant works to ensure a comprehensive and up-to-date review of the MLLM research landscape in the revised manuscript.

Thank you for the constructive comments and suggestions. Below please find our clarification regarding your concerns.

---

[W1] Validation on newer LVLMs.

[R1] We appreciate the reviewer’s suggestion. In response, we have conducted **additional experiments on two newer LVLMs, LLaVA-Next and Qwen2.5-VL, using the Infoseek dataset. The tabulated results below show that our method improves with both models consistently, validating its effectiveness and robustness on more recent architectures.

---

[W2] Framework’s robustness.

[R2] We appreciate the reviewer’s suggestion. We would share that the robustness of our framework has been validated from four perspectives:

1. Robustness to knowledge quality: As shown in Figure 1 of the submitted paper, our model achieves consistent improvements when the quality of retrieved knowledge varies. This indicates that the model can adaptively determine which knowledge to rely on, reflecting its robustness.

2. Robustness across MLLMs: As shown in Tables 2, 3, and 4, our model integrates effectively with four different MLLMs, demonstrating its compatibility and generalizability across diverse architectures.

3. Robustness to retrievers: As shown in Table 6, our model consistently improves MRAG performance across different retrievers, highlighting its tolerance to retrieval noise.

4. Robustness to model scale: As shown in Table 7, our method yields consistent gains on 13B-scale MLLMs, confirming its scalability with respect to model size.

---

[W3] Details of the compared methods.

[R3] We appreciate the reviewer’s suggestion. For all compared methods, we follow the implementation and model settings as described in their original papers to ensure a fair comparison. We will include more detailed descriptions of these methods and their configurations in the revised version for better clarity and reproducibility.

---

[Q1] Analysis of different intervention layers might be better placed in the main paper.

[A1] Thank you for the helpful suggestion. We agree that the analysis of different intervention layers is an important component of our study. Accordingly, we will revise the paper to include this experiment in the main paper as suggested, ensuring that readers can better appreciate its contribution.

---

[Q2] Definitions of certain variables.

[A2] We apologize for the confusion. The definitions of the variables are as follows:

1. As described in Line 176, $\beta$ is a factor that controls the strength of adjustment for image tokens.

2. As described in Line 190, $\gamma$ controls the strength of adjustment for each context token.

3. As defined in Equation 10, $\lambda$ represents the summed attention weights over image and context tokens, used for weighted fusion.

We ensure that all these definitions will be stated clearly and prominently in the revised paper to improve readability and understanding.

Thanks for the constructive comments and suggestions. Below please find our clarification regarding your concerns.

---

[W1] Attention weighting and logits fusion are borrowed from previous works.

[R1] We acknowledge that attention weighting and logits fusion have been explored in prior studies, represented by [1] and [2]. However, our approach differs significantly in motivation, method design, and empirical performance, as outlined below:

1. Motivation: Prior works [1][2] focus on the multimodal hallucination task, [1] by enhancing visual attention, and [2] by subtracting visual logits to counteract language priors. In contrast, our motivation is orthogonal: we aim to reduce image interference via attention reallocation and fuse parametric and retrieved knowledge via logits addition to enhance knowledge utilization. This distinct objective is supported by thorough analysis and empirical validation, indicating that our method is not a simple reuse of existing techniques.

2. Method Design: Unlike [1][2], which apply fixed factors for attention reweighting or logits subtraction, our method employs retrieval similarity and question-context similarity to dynamically adjust attention scores for image and context tokens. Additionally, we leverage these attention scores for adaptive knowledge fusion, enabling the model to selectively utilize different knowledge sources. This design is particularly beneficial when retrieved knowledge is noisy or partially incorrect.

3. Performance: To further demonstrate the uniqueness and effectiveness of our approach, we conducted ablation studies replacing our components with those from [1][2]. As shown in the table below, these substitutions consistently led to performance drops, validating the advantage of our design over prior methods.

We appreciate the reviewer’s comment and will include a more detailed discussion and comparison with related work in the revised version.

---

[W2] Inference cost.

[R2] We appreciate the reviewer’s concern regarding the inference cost. Compared with the baseline MRAG, our model introduces a modest overhead for parametric knowledge modelling. However, the extra cost remains acceptable as the input for modeling parametric knowledge without context is relatively short. Besides, all compared methods except the baseline need to model both contextual and parametric knowledge. The inference cost of our method is comparable to that of the compared methods in a broader context. We evaluate all compared methods over one sample on both discriminative and generative tasks. The tabulated results below confirm that our method takes comparable inference time. In fact, our method can achieve competitive inference efficiency as the baseline MRAG by processing the two knowledge sources in parallel. We will elaborate on this issue with the new experiments in the revised manuscript.

---

References

[1] Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs.

[2] Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding.