Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models

Thank you for your constructive comments and evaluation of our work. We address each of your specific concerns as follows: 

Q1: The paper presents a method for hallucination mitigation in MLLMs by re-focusing on visual details relevant to the textual prompt. While the concept of improving model performance through prompt-related visual cues is not entirely new, prior works like API[1], AGLA[2] exploring similar ideas. Despite some innovation in the design, the contributions do not seem to offer substantial new value or advancements in the field.

We acknowledge the pioneering efforts in the field, including the works API[1] and AGLA[2]. Below, we highlight the unique innovations of our method across motivation, framework, efficiency, and capability. 

1) Motivation difference. API[1] focused on visual prompting without addressing textual queries, its motivation is not for hallucination mitigation. AGLA[2] argues that LVLMs predominantly attend to prompt-independent global image features while failing to capture prompt-relevant local features. However, we argue that hallucination may be caused by modality imbalance, i.e., LLMs struggle to perceive and memorize visual information than text, thus eliciting our idea of ''look twice before you answer'', which is grounded in a human cognitive process: when the initial memory of certain critical visual details fades, it is intuitive to look at them for the second time to search for the accurate and hallucination-freon e answer. 

2) Framework difference. API [1] uses an auxiliary model like CLIP to generate an attention heatmap for the input image dependent on the text query, then the heatmap simply multiplies the pixel values of the original image to obtain the actual input image for the LVLM. AGLA[2] uses an image-prompt matching scheme to generate an augmented view that captures local features from images, and then assembles the logits derived from both the original and augmented images. Unlike these approaches, MemVR is an architecture-agnostic, plug-and-play solution that re-injects visual features into an intermediate layer suffering from high uncertainty with the dynamic injection strategy. Moreover, MemVR also can be extended to other modalities (e.g., video, audio, 3D data), but API and AGLA cannot. 

3) Efficiency difference. More importantly, AP and AGLA methods introduce additional time overhead (e.g., AGLA has to reason twice to get an answer), but MemVR only with one regular inference without incurring added storage requirements and time overhead, which allows MemVR to be potentially applicable to real-world scenarios. 

4) Capability difference. We conduct comprehensive experiments on diverse benchmarks, including MME, POPE, CHAIR, VizWiz-VQA, MMBench, MM-Vet, and LLaVA-Bench (in-the-wild), results demonstrate that MemVR not only significantly mitigates hallucination issues compared to SOTA methods (i.e. VCD, OPERA) across various MLLMs, but also excels in general benchmarks (notably, other approaches tend to degrade performance in general benchmarks), thus emphasizing its potential for widespread applicability. 

Here are the characteristics of different methods. We further compared HALC[3], MVP[4], VACoDe[5], SID[6]. 

We have elucidated these distinctions in the revised manuscript to underscore the novelty and practical implications of our contributions.

 [1] Attention Prompting on Image for Large Vision-Language Models.

[2] AGLA: Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention.

[3] HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding, ICML, 2024.

[4] Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning.

[5] Vacode: Visual augmented contrastive decoding.

[6] Self-introspective decoding: Alleviating hallucinations for large vision-language models, 2024.

Q2: The paper's writing style can be overly embellished, making it less intuitive in certain parts, which may not clearly convey the intended message. 

We appreciate your observation regarding the writing style. We agree that clarity is important. We have revised the manuscript to adopt a more straightforward and concise writing style, encompassing the introduction, methodology, experiments, and conclusions, ensuring that the technical details and innovative aspects of our work are conveyed with precision and accessibility. To ensure these modifications are easily identifiable, we have distinctly marked them in blue in the updated version of our paper. 

Q3: The phrase "signifies the other side of the coin" at line 114 is unclear and requires further explanation to understand its intended meaning. 

This expression was intended to highlight the complementary nature of our approach to existing methods. The representative methods like VCD and OPERA target to eliminate language priors, but MemVR seeks to replenish visual information. More intuitively, whether eliminating linguistic priors or supplementing visual information, the common objective is to balance the model's understanding of different modalities, which signifies the two sides of the coin. We have provided a clearer explanation in the revised manuscript to ensure that readers grasp the intended meaning and the rationale behind our methodological choices.  

Q4: While the authors claim at line 499 that MemVR is flexible and compatible with various architectures, they also mention at line 514 that adapting the method to other structures is challenging. It raises the question of whether MemVR can be generalized to models like LLaVA-next and InstructBlip. 

First, sorry for the confusing words above. In line 499, we regard MemVR as a flexible approach as it only takes a few lines of code without modifying transformers lib, and therefore can be easily applied to other MLLMs. In line 514, where we discussed the limitations of our work, we aim to express that MemVR needs some adjustments to achieve optimal performance when applied to a brand-new model due to the difference in suitable hyperparameters across different models. We have demonstrated its effectiveness across various architectures, including LLaVA (MLP), Qwen-VL (Q-Former), and GLM-4V (MLP), as follows: 

We are confident to say that MemVR can be easily generalized to other models like LLaVA-Next, LLaVA1.6-mistral, or others. But we'd like to discuss with you a certain case: IntructBLIP, which is the only model that does not work well with MemVR. InstructBlip uses a very special Q-Former that transforms visual information into a few tokens (often no more than 20 tokens). This makes the MemVR calculation very difficult to keep the retraced information and conduct the attention-like operation, as the core calculation of MemVR can be shortened as $\text{SiLU}(\mathbf{h}\cdot\mathbf{z}_v^{\top})\cdot\mathbf{z}_v$, where $\mathbf{z}_v \in \mathbb{R}^{n_v \times d}$ denotes visual features. Such short visual token brought by InstructBlip will lead to the fact that the retraced visual information be compressed into a $$ n_v, n_v $$ matrix where $n_v$ is usually less than 20, while the original visual information is represented as $$ n_v, d $$ where $d$ is much higher than $n_v$, leading to information loss. Therefore conducting MemVR calculation with IntructBLIP does not seem to have any effect on improving the quality of generated answers. We provide some benchmark evaluation results of MemVR+IntructBlip as follows, 

In all, MemVR can be easily applied to almost all open-source MLLMs and achieve substantial improvements across various benchmarks.

Q5: The analysis of uncertainty in section 3.2 is based on the entropy of the decoder's output distribution, but the impact of uncertainty on the method, as shown in Figure 7, is analyzed by adding noise to the images. The claim that the method is unrelated to the biases of language priors may be questionable, as the entropy-based uncertainty measure could be connected to these biases. Additionally, it is unclear how the analysis of uncertainty in section 3.2 is directly related to the proposed method, beyond its application in the Dynamic Injection Strategy. 

Thanks for your valuable feedback. We would like to clarify a few points regarding the concerns you've raised: 

• Regarding the relationship between the uncertainty analysis in Section 3.2 and Figure 7: 1) In Section 3.2, we quantify model uncertainty based on the entropy of the decoder’s output distribution. This semantic uncertainty is used to quantify the state of confusion at the different layers. 2) However, in Figure 7, we introduce noise to the visual feature when visual retracing, not the image, i.e., only affects the visual features of 'second look'. This visual uncertainty implies the cleanliness of the feature being reinjected. This experiment is intended to demonstrate that MemVR is sensitive to the quality of visual features that need to be reinjected when visual retracing, and is efficient at understanding shallow information. We have revised Figure 7 and the text to avoid confusion. 
• On the relationship between our method and language prior biases: We apologize for possibly misinterpreting you. 1) we do not claim that our approach is independent of linguistic a priori bias. 2) We claim that in contrast to previous methods, which primarily focus on eliminating biases of language priors, MemVR seeks to replenish visual information as in line 112. 3) The entropy-based uncertainty measure is used to guide our dynamic injection strategy to optimize the visual retracing process. Our experimental results also indicate that this strategy effectively enhances model performance across diverse scenarios. 
• The direct connection between the uncertainty analysis in Section 3.2 and the proposed method: The uncertainty analysis in Section 3.2 serves as a theoretical foundation for our dynamic injection strategy. By quantifying the uncertainty in the decoder’s output, we gain insight into the states of different layers across varying inputs, which guides the design of our strategy to optimize the information injection process. This analysis directly supports the core concept of our method and contributes to the improvement of overall model performance. As Reviewer eZB5 said: ''The paper provides a clear introduction to the hallucination problem in MLLMs, analyzes its causes, presents solutions, and offers a theoretical understanding of the methods used''

Q6: There is a typo error in line 390 where "Preception" should likely be "Perception." 

Thank you! We apologize for the typographical error and have corrected "Preception" to "Perception" in the revised manuscript. 

Q7: I recommend that the authors restructure the paper, including both figures and text, to express their ideas more clearly and logically, rather than attempting to embellish the research to make it appear more sophisticated. 

We concur with your recommendation to restructure the paper for improved clarity and logical flow. We have revised the figures and text to better express our ideas in the updated version, focusing on clarity and conciseness. We present our research in a manner that is both rigorous and accessible to the community.

We are committed to addressing these concerns thoroughly and believe that these revisions will significantly enhance the quality and impact of our manuscript. Thank you once again for your valuable feedback.

Dear Reviewer YSEL,

Thank you for your valuable time and the constructive feedback you have provided. We genuinely hope reviewer YSEL could kindly check our response.

As the deadline for the discussion period is nearing, we would greatly appreciate it if you could kindly let us know whether there are any further questions. Thank you!

Best wishes,

Authors

After reviewing the updated version, I noticed that many key concepts still lack the necessary context, such as “visual prompts” mentioned in line 19, which was not sufficiently clarified as per my previous comments. Additionally, the authors have avoided providing results for their method on the LLAVA-Next model, which I had requested. Given these concerns, I will maintain my original score.

Concern1: I noticed that many key concepts still lack the necessary context, such as “visual prompts” mentioned in line 19. 

We believe that anyone working on MLLMs would be familiar with the phrase 'visual prompt', as it's one of the most basic concepts within this field. It's quite a pity that you have no idea what it stands for, we would kindly provide you with a simple explanation: visual prompts often refer to the visual (image) information that will be taken as the input of the Language Model. For the models mentioned in this paper, visual prompts are visual (image) tokens with the same dimension as embedded text tokens.   

Concern2: Additionally, the authors have avoided providing results for their method on the LLAVA-Next model, which I had requested. 

To the end, we've already provided 4 MLLMs with 7 kinds of benchmark evaluations. These MLLMs used different Language models, text-image aligning methods, and training strategies to ensure effectiveness and generalization across the various testing environments. And, we have demonstrated that MemVR can be easily applied to almost all open-source MLLMs. Compared with SOTA methods, we have done the most complete experiments, as follows,  

We have provided 4 MLLMs across 7 kinds of benchmark evaluations, including hallucination and general benchmarks, which is sufficient to demonstrate the efficacy and generalization of our method. But you're asking for more experiments on LLaVA-next, which I would doubt for your so-called 'concerns'. 

Even though this is an extremely unreasonable request, we have decided to fulfill it. Here's the MME evaluation of LLaVA-Next (Llama3-8B) model and +MemVR: 

Further, we also evaluated LLaVA-Next (Mistral-7B) and LLaVA-Next (Vicuna-1.6-7B),

Thank you for your insightful comments. We address your concerns as follows.

Q1: The paper’s explanation for the occurrence of hallucinations in MLLMs is not sufficiently persuasive. While the authors identify uncertainty in the decoding process, they fail to provide adequate evidence that this uncertainty stems from the forgetting of visual features, subsequently leading to hallucinations. To strengthen this claim, I recommend that the authors conduct additional experiments, such as directly masking input image tokens to simulate the effect of forgetting. They could then test whether MEMVR is able to recover visual features through the "look twice" mechanism while utilizing the normal image tokens during the recovery process.

Thank you for this valuable suggestion. Following your recommendation, we conducted additional experiments by masking input images with randomly generated noise. These masked images were then used as input to the VLM, allowing us to test the ability of MemVR to retrace features from the unmasked image through the "look twice" mechanism. We experimented with varying levels of noise, where higher levels simulate increased forgetting of the original image. At 900 noise steps, the human eye can no longer distinguish the content of the picture.

Mask experiment results of llava-v1.5-7b on MME, with Perception, Cognition, and Total (Per+Cog) scores at different noise steps as follows,

Perception:

Cognition:

Total (P+C):

As shown in the above tables, MemVR is able to recover some of visual memories through the 'look twice' mechanism under different noise steps, enhancing performance. The experiments demonstrate that retracing ('look twice') using the unmasked image indeed enhances the MLLM's understanding of the masked inputs. We believe this result offers strong evidence for the effectiveness of the "look twice" mechanism, adding further rigor to the logical framework of our paper. Thank you for this valuable suggestion, which has made a meaningful contribution to our work.

Q2: The presentation of the proposed method is convoluted and lacks clarity. The authors unnecessarily complicate straightforward concepts by introducing numerous "unusual" symbols and formulas. [...]

Sorry for the unclear presentation of MemVR. To address your concern, we have provided a clearer explanation in the revised manuscript to ensure that readers understand our methodology.

Here's a detailed explanation of how MemVR is implemented. In brief, as the title "Look twice before you answer" suggests, this is the main process of our MemVR. Specifically, this involves two issues: 1) at which layer to re-look, and 2) how to re-look at it.

For the first issue - at which layer to re-look, we design a dynamic strategy to select a specific injection layer for 're-look', i.e., replenish visual information in the selected layer. In particular, Algorithm 1 in line 277 shows the code flow of our dynamic injection strategy. We first compute the uncertainty of the next token probability on each early layer and set the threshold $\gamma$. When the uncertainty of a specific layer is over $\gamma$, 're-look' is triggered in this layer.

Then, how to 're-look' at it, i.e., how visual retracing is implemented. We take the visual feature after the connector as visual evidence to be re-injected at the trigger layer selected in the previous step. Taking the hidden state $x \in \mathbb{R}^{d}$ of the trigger layer as query, and the visual evidence $z_v\in \mathbb{R}^{n_v \times d}$ as both key and value, then we perform a similar cross-attention operation, i.e, $\operatorname{Retrace}({z}_v \mid {x})=\phi({x} \times {z}_v^\top) \times {z}_v$. Further, in FFN, we re-inject the retrieved visual memory to the hidden-state with ${\operatorname{FFN}}({x} \propto z_v)=(1-\alpha)\operatorname{FFN}(x) + \alpha\operatorname{Retrace}({z}_v \mid {x})$, where $\alpha$ denotes the injection ratio of visual memory.

That's the whole process of our method, simple, but efficient and effective. We have revised the specific process of MemVR in the revised manuscript to make our method more clear and accessible. Please do let us know if you have any questions or concerns on it. 

Here are some suggested modifications: 1.Provide a clear definition and explanation for each symbol when it is first introduced, especially for non-standard usage like in Equation (2). 2.Reorganizing the text to keep related equations and their explanations closer together, particularly for Equations (2) and (3). 3.Clearly define the terms $k$ and $v$ and explain the rationale behind treating them as keys and values if they are indeed FFN parameters. 4.Recommending the inclusion of a notation table or glossary to help readers keep track of symbols and their meanings throughout the paper. 5.Try to review the entire manuscript to identify and simplify overly complex formulations without losing essential information.

Thanks for your valuable suggestions. We have provided a clearer explanation and definition for each symbol, simplified overly complex formulations, and reorganized the text in the revised manuscript to ensure that readers understand our methodology. These modifications are highlighted in blue in the revised version of our paper to facilitate easy identification. We appreciate your guidance in improving the clarity of our discussion.

Dear Reviewer tPgc,

Thank you for your valuable time and the constructive feedback you have provided. We genuinely hope reviewer tPgc could kindly check our response.

As the deadline for the discussion period is nearing, we would greatly appreciate it if you could kindly let us know whether there are any further questions. Thank you!

Best wishes,

Authors

Thank you for your insightful comments. We address your concerns as follows.

Concern 1: Dependence on Threshold-based Mechanism: MemVR's use of an uncertainty threshold for reinjection may lead to sensitivity to specific datasets or model architectures, potentially affecting its generalizability. The adaptability of this mechanism across different models without manual tuning remains uncertain.  

Thank you for pointing out that. MemVR does need a few rounds of threshold adjustments to achieve optimal performance when applied to a brand-new model, this process is typically brief and does not present significant overhead (usually no longer than 2 hours). More importantly, in most cases, even a randomly chosen parameter could improve the performance compared to the baseline, indicating a degree of robustness. Once the optimal threshold (usually set at 0.75) is determined, MemVR achieves significant performance gains across a wide range of benchmarks, showcasing its adaptability and effectiveness in diverse settings. On Part 2/3 Q1, we provide performance results for LLaVA under the different thresholds across three benchmarks, and the results demonstrate that MemVR is not sensitive to the threshold. 

Concern 2: Section C.3 Analysis: Section C.3 highlights only the successes of MemVR. Including instances of failure would provide a more balanced analysis. Without external data as used by other methods, MemVR might risk reinforcing biases in the original model by reinjecting similar visual features. Addressing these issues and examining cases where MemVR underperforms would offer a more comprehensive understanding of its limitations and inform future improvements. 

Thank you for your valuable suggestions! We appreciate the importance of providing a balanced analysis and found your feedback very helpful. Specifically, we are keen to explore how reinjecting similar visual features without external data might affect model biases. To address this, we have collected failure cases from the MME benchmark, particularly in the 'Celebrity,' 'Scene,' and 'Landmark' sub-tasks, where MemVR underperforms compared to the default model.

We categorize MemVR's failures into two types:

(a) Cases where the default model provides the correct answer, but MemVR outputs an incorrect one.

(b) Cases where both the default model and MemVR produce incorrect answers.

For (a), we attribute the failure to over-disturbance of the default model's reasoning process. In these instances, the original visual features are sufficient for reasoning, and the reinjected tokens inadvertently disrupt this process, leading to errors. We are actively investigating methods to mitigate such disturbances.

For (b), the failures arise from either the excessive complexity of the image or gaps in the LLM's knowledge base, which prevents correct reasoning even after retracing.

These failure cases have been added to Section C.3 in the revised version of the paper. We hope this analysis addresses your concerns and provides a more comprehensive understanding of MemVR's limitations, paving the way for future improvements. 

Concern 3: Presentation Style: The Q&A format in Section 4.2 on quantitative results lacks logical structure and clarity. It would be more effective to present this section in a structured format that clearly outlines the key findings, for example: “Q2: How well does MemVR perform on general-purpose benchmarks?” Can be simply phrased as “MemVR performance on general-purpose benchmarks.” 

Thank you! We have re-paraphrased this section to make it easier to read and more logical. These modifications are highlighted in blue in the revised version of our paper to facilitate easy identification.

Concern 4: The paper provides valuable insights but needs more attention to detail in presenting results. Specific issues include: In Table 2, Qwen-VL-10B + OPERA's best F1 score on MSCOCO is not bolded. Table 7 shows LLaVA1.5-7B + OPERA and LLaVA1.5-7B + VCD have top scores on gnkwrec and ocrgnsp, respectively, but these are not bolded. Additionally, a value for LLaVA1.5-7B + OPERA is listed as +1.4 instead of +20. Table 6 reports MemVR’s improvement on Convs as +0.5 instead of +5.0. 

Thank you for your detailed review! We have checked all tables and in-line data to ensure these mistakes are all fixed.

Q1: Could the authors clarify how the uncertainty threshold was selected, and how sensitive MemVR is to this parameter across different datasets?

Sure, when running MemVR calculations, we first calculate the early layer entropy by analyzing its early-exit logits, and we make MemVR only to be triggered when a specific layer entropy exceeds the threshold, where the threshold is determined by a set of test experiments on the MME benchmark that reveal the average uncertainty (which can also be regarded as semantic entropy) of early layers is over 0.75 when hallucination occurs. That's the reason the threshold is set to 0.75. We set the threshold to be fixed, which effectively improves performance in all benchmark tests. For example, we set the threshold to 0.75 in LLaVA-1.5 on 9 benchmarks across different domains, MemVR achieves dynamic injection and stable performance improvement in all of them.

To further address your concern, we have now provided performance results for LLaVA under the different threshold across three benchmarks,

Threshold sensitivity experiments of LLaVA-1.5+MemVR on MME:

Threshold sensitivity experiments of LLaVA-1.5+MemVR on POPE (COCO):

Threshold sensitivity experiments of LLaVA-1.5+MemVR on MMBench_DEV_EN:

From the results of the sensitivity experiments, it can be seen that MemVR is not sensitive to this parameter across different datasets. For further analysis, it can be seen that if the threshold is set too low, the dynamic injection strategy becomes static fixed-layer injection, while setting the threshold too high makes MemVR difficult to trigger and degrades into the vanilla model.

Q2: Are there specific scenarios where MemVR fails to reduce hallucinations, could you provide a few concrete examples of failure cases and limitation analysis on them?

Sure. Please refer to the reply for Concer2 above, and we've also updated some failure cases to C.3.

Additionally, we’d like to mention another failed case with MemVR. We've tested several models and almost all of them received significant improvements, except InstructBLIP, which is the only model that does not work well with MemVR. InstructBlip uses a very special Q-Former that transforms visual information into a few tokens (often no more than 20 tokens). This makes the MemVR calculation very difficult to keep the retraced information and conduct attention-like operation, as the core calculation of MemVR can be shortened as, $\operatorname{Retrace}({z}_v \mid {x})=\phi({x} \times {z}_v^\top) \times {z}_v$. Such short visual token brought by InstructBlip will lead to the fact that the retraced information will be compressed into a [$n_v, n_v$] matrix where $n_v$ is usually less than 20, while the original visual information is represented as [$n_v, d$] where $d$ is much higher than $n_v$. Therefore conducting MemVR calculation with InstructBLIP does not seem to have any effect on improving the quality of generated answers. We consider its poor performance with MemVR due to using blip2-like Q-Former architecture, which could lead to serious information loss.

We provide some benchmark evaluation results of MemVR+IntructBlip as follows,

Note: Please refer to C.2, Implementation details for experimental settings.

Please do let us know if you have any questions or concerns on it.

Q3: Would MemVR function effectively if extended to other modalities (e.g., audio, 3D data)? Or would it require significant adjustments?

The core contribution of MemVR is 'retracing visual information at the second time'. Following this idea, there are two questions when extending it to more modalities

1. How's the alignment work going? Can the large language model fully understand the transformed token from, e.g., audio or 3D mesh or something else? If yes, it means that you now have a powerful alignment tool, like efficient projection MLPs. In this way, we can ensure that the model can, somehow, understand the tokens that are about to be retraced.

2. Will there be information loss during MemVR calculation? As introduced in Q2, highly compressed external information (like the Q-Former of InstructBlip outputs few visual tokens) may lead to poor improvement of performance after retracing. Please ensure that the information from other modalities you'd like to retrace won't be over-compressed.

Once the 2 critical issues listed above can be resolved, we believe that MemVR can be directly and effectively extended to other modalities.   Please feel free to write your comment if you still have any questions or concerns. Thank you!

Dear Reviewer iT3w,

Thank you for your valuable time and the constructive feedback you have provided. We genuinely hope reviewer iT3w could kindly check our response.

As the deadline for the discussion period is nearing, we would greatly appreciate it if you could kindly let us know whether there are any further questions. Thank you!

Best wishes,

Authors

Q1: In formula (2), the feedforward neural network comprises two fully connected layers. As MemVR is a training-free method, how are the weight matrices W1 and W2 obtained? If they utilize weights from a pre-trained projector within the MLLM, how can their effectiveness be ensured?

The W1 and W2 mentioned here are not for visual retracing procedures but for regular FFN calculation. You may refer to our explanation for Concern 1, which may help you fully understand how MemVR is applied.

Q2: In formula (3), K and V are regarded as visual projectors. What is their relationship with the keys and values in the attention mechanism? This seems to differ from the kv cache approach, leading to some ambiguity in the expression.

Sorry for the ambiguity in the paper. Here, we first introduce how visual retracing is implemented. We take the visual feature after the visual projector as visual evidence to be re-injected at the trigger layer. Taking the hidden state $x \in \mathbb{R}^{d}$ of the trigger layer as query, and the visual evidence $z_v\in \mathbb{R}^{n_v \times d}$ as both key and value, then we perform a similar cross-attention operation, i.e, $\operatorname{Retrace}({z}_v \mid {x})=\phi({x} \times {z}_v^\top) \times {z}_v$. Further, in FFN, we re-inject the retrieved visual memory to the hidden-state with ${\operatorname{FFN}}({x} \propto z_v)=(1-\alpha)\operatorname{FFN}(x) + \alpha\operatorname{Retrace}({z}_v \mid {x})$, where $\alpha$ denotes the injection ratio of visual memory. MemVR has nothing to do with kv cache approach. We have provided a clearer explanation in the revised manuscript to ensure that readers understand the details.

Q3: The injection ratio α is a manually tuned hyperparameter. How should the validation set be selected during the tuning process? Does the parameter range differ across scenarios such as healthcare and transportation?

According to the massive experiments we conducted, a proper set of hyperparameters that performs well in one specific benchmark mostly means it also works on most evaluation sets. Say, the best retracing ratio for LLaVA1.5 here, mostly stands for 0.12. Under the retracing ratio set to 0.12, MemVR can improve the performance compared to the baseline on 9 benchmarks across different domains steadily (including code, math, commonsense, lives, etc.). For Qwen-VL, it would be 0.28.

When facing different scenarios such as healthcare and transportation, it needs some changes to the injection ratio, but it is easy to find the optimal parameter. We will continue to study how to adjust the injection rate automatically. You may refer to our explanation for Concern 2.

Q4: In line 279, it is mentioned that details about the dynamic injection strategy will be presented in Appendix C.1; however, C.1 only discusses benchmarks. 

Sorry, we missed the details in Appendix C.1. The dynamic injection strategy has been explained in Concern 1 and supplemented in the revised version of the paper. These modifications are highlighted in blue in the revised version of our paper to facilitate easy identification.

Please feel free to write your comment if you still have any questions or concerns. Thank you!

Thank you for your insightful comments and positive evaluation of our work. We address your concerns as follows. 

Concern 1: The explanation of the specific process of MemVR is not sufficiently clear; the discussion of the dynamic injection strategy is brief and potentially confusing. 

Thanks for your feedback. To address your concern, we have provided a clearer explanation in the revised manuscript to ensure that readers understand our methodology.

Here's a detailed explanation of how MemVR is implemented. In brief, as the title "Look twice before you answer" suggests, this is the main process of our MemVR. Specifically, this involves two issues: 1) at which layer to re-look, and 2) how to re-look at it.

For the first issue - at which layer to re-look, we design a dynamic strategy to select a specific injection layer for 're-look', i.e., replenish visual information in the selected layer. In particular, Algorithm 1 in line 277 shows the code flow of our dynamic injection strategy. We first compute the uncertainty of the next token probability on each early layer and set the threshold $\gamma$. When the uncertainty of a specific layer is over $\gamma$, 're-look' is triggered in this layer.

Then, how to 're-look' at it, i.e., how visual retracing is implemented. We take the visual feature after the connector as visual evidence to be re-injected at the trigger layer selected in the previous step. Taking the hidden state $x \in \mathbb{R}^{d}$ of the trigger layer as query, and the visual evidence $z_v\in \mathbb{R}^{n_v \times d}$ as both key and value, then we perform a similar cross-attention operation, i.e, $\operatorname{Retrace}({z}_v \mid {x})=\phi({x} \times {z}_v^\top) \times {z}_v$. Further, in FFN, we re-inject the retrieved visual memory to the hidden-state with ${\operatorname{FFN}}({x} \propto z_v)=(1-\alpha)\operatorname{FFN}(x) + \alpha\operatorname{Retrace}({z}_v \mid {x})$, where $\alpha$ denotes the injection ratio of visual memory.

That's the whole process of our method, simple, but efficient and effective. We have revised the specific process of MemVR in the revised manuscript to make our method more clear and accessible. 

Concern 2: Concerns about usability: The method requires manual tuning of hyperparameters, which may necessitate multiple rounds of adjustments to achieve optimal performance in different scenarios.

Thank you for the insightful comment. Yes, MemVR does need a few rounds of adjustments to achieve optimal performance when applying to a brand-new model, this process is typically brief and does not present significant overhead. Take Qwen-VL as an example, we test it with MME and it takes only 6 rounds to find the optimal performance, which is no longer than 2 hours. And we'd like to mention that this does not mean MemVR won't work without manual tuning, as in most cases even a randomly chosen parameter could improve the performance compared to the baseline, indicating a degree of robustness. Besides, MemVR has been tested with 9 different benchmarks, just to ensure that our approach is effective and general, and it turns out that MemVR outperforms all other methods in all evaluation results. More importantly, MemVR is quite easy to extend to other models, such as llava-next series, qwen2 series, etc. with few lines of code, which makes the usability of MemVR pretty impressive.

Here, we supplement the experiments on MME under different injection rates, it can be seen that MemVR is not sensitive to the injection rate setting. Under all injection rates, LLaVA+MemVR can improve the performance compared to the baseline.

Thus, hyperparameters do not limit our approach, and we will continue to study how to adjust the injection rate automatically.

Dear Reviewer eZB5,

Thank you for your valuable time and the constructive feedback you have provided. We genuinely hope reviewer eZB5 could kindly check our response.

As the deadline for the discussion period is nearing, we would greatly appreciate it if you could kindly let us know whether there are any further questions. Thank you!

Best wishes,

Authors