Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models

We sincerely thank Reviewer 8UAJ for the insightful comments.

Q1: Given that most layers are not aligned with the vocabulary head, what such experiments indicate is doubtful and the conclusions do not make sense...

The idea of applying language heads directly to the hidden states of the middle layers, known as early exit [1][2][3], has proven to be effective even without the special training process [4][5] for alignment to the vocabulary head.

We also presented it in the paper (Line 263, right).

[1] Teerapittayanon et al. Branchynet: Fast inference via early exiting from deep neural networks. ICPR, 2016.

[2] Maha Elbayad et al. Depth-adaptive transformer. ICLR, 2020.

[3] Tal Schuster et al. Confident adaptive language modeling. NeurIPS, 2022.

[4] Wei-Tsung Kao et al. Bert's output layer recognizes all hidden layers? some intriguing phenomena and a simple way to boost bert. arXiv, 2020.

[5] Chuang Y S et al. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR 2024.

Q2: About 'amnesia' and not seeing memory-involved design

"Amnesia" is similar to the concept of the information flow of image tokens converges at shallow layers and diverges at deeper layers in [6], where truncating image tokens no longer affects the answers in the middle or deep layers.

VR is implemented in FFN, and FFN stores knowledge from training data in the form of KV memories [7], thus we call VR in memory space.

[6] From Redundancy to Relevance: Enhancing Explainability in Multimodal Large Language Models, 2024.

[7] Geva M, et al. Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP, 2021.

Q3: Figure 5 case is not a clear hallucination problem, which is not a language prior or statistical bias

The ground-truth of Figure 5 case is "four mangosteen", but LLaVA outputs "a pomegranate", the count (four->one) and object (mangosteen->pomegranate) are wrong. We don't know why it is not a clear hallucination case.

1. About this case, Reviewer 8UAJ thinks it is not a language prior or statistical bias. We get the reviewer's view that only examples with a language prior or statistical bias, like the misidentification of a black banana as a yellow banana, can be called hallucinations. However, the language prior or statistical bias is merely a possible source of the hallucinations proposed in the Contrast Decoding series of papers, e.g., VCD, but not the only source.

2. We're not imitating VCD or other papers to present a language prior or statistical bias as the cause of hallucination, so we did not purposely choose like “black bananas“ with a strong language prior for this example.

3. Further, the other hallucination cases with a language prior or statistical bias in our experiments also show the same regularity, i.e., hallucination tokens exhibit higher entropy than correct ones.

Q4: No theoretical claims

We have provided a theoretical framework based on information theory in the paper, Reviewers 2zfj,bEEH,3FtN mentioned.

Q5: The experimental designs or analyses has weak relation to experimental designs

Firstly, we actually don't know much about what you mean by weak ties.

We conducted comprehensive experiments and analysis on 8 benchmarks and ablation studies examining the impact of different injection ratios, uncertainty thresholds, and static vs. dynamic triggering strategies.

We've already provided 5 MLLMs with 8 kinds of benchmark evaluations. These MLLMs used different Language models, text-image aligning methods, and training strategies to ensure effectiveness and generalization across the various testing environments. And, we have demonstrated that MemVR can be easily applied to almost all open-source MLLMs. Compared with SOTA methods, we have done the most complete experiments, as follows,  

If you still think our experimental design has problems, could you be more specific? We are willing to discuss with you.

Q6: VR appears to be similar to V*, about the differences

We actually don't understand why Reviewer 8UAJ suddenly gave us a paper that is irrelevant to hallucinations, nor in a similar field. Nonetheless, we follow your comment for comparison.

In short, the difference between V* and ours is as follows,

1. The goal is different. V* is an LLM-guided visual search algorithm, not a method to mitigate hallucination.

2. The framework of V* is the normal structure of MLLMs, which is completely different from our "look-twice" where an extra bypass is created.

3. V* needs training by LoRA, but MemVR is a plug-and-play method.

[1] V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs.

We sincerely thank Reviewer 3FtN for the constructive comments on our work. We promise to revise the paper.

Q1: About textual uncertainty

Yes, textual uncertainty cannot be a sufficiently necessary condition for the existence of hallucinations, but when hallucinations are occurring, textual uncertainty tends to be above the threshold, and MemVR triggers override the cases in which hallucinations occur.

Besides, regarding the ablation in Fig. 7 (left), the curve may not be obvious due to the value being over 1800 and the change being within 100. Specificaly, we set $\gamma$ from 0.5 to 1.0, when $\gamma$ is set to less 0.6 MemVR is triggered early without performance improvement, while between 0.6 and 0.95 can improve performance, with the optimal threshold around 0.75 (can increase 32 scores), which means MemVR works well under higher uncertainty.

Q2: The analysis in Section 3.2 not beyond

There are many works that have discussed why MLLM generates hallucinations. Thus, our discussion in Section 3.2 firstly introduces previous insights, including inherent biases in the training data, visual uncertainty resulting from the model’s statistical bias and priors, and the limitations of current models in accurately discerning context and fact throughout the output generation process.

Then, we presented our argument that the imbalance of modalities leads to a substantial deviation from the accurate representation of visual input, eventually giving rise to hallucinations, which prior work did not raise. We will continue our efforts to delve into the causes of hallucinations.

Q3: Regarding "text feature"

The 'text feature' here refers to the embedded text tokens, which are concatenated with visual tokens to form the input of LLMs.

Q4: Does VR introduce image-related bias?

[1] constructs the image-biased model by adjusting the attention weight matrices (i.e., an amplification to the image tokens) within the vanilla model.

In Equation (6), VR only modifies the hidden state in the FFN layer (i.e., memory about visual information) that uses the original hidden state as query to search supplemental visual feature, rather than a direct amplification to the image tokens in attention weight matrices, thus VR does not introduce image-related bias as [1].

[1] IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding

Q5: Does MemVR amplify image redundancy errors?

In conclusion of [2], image tokens are highly redundant after the cliff layer, where truncating image tokens no longer affects the answers in the middle or deep layers. This is because LLMs are more text-informed with "attention sinks", and token embedding of image tokens may not align well with the model’s text-based training. This does not mean that image redundancy leads to errors.

In MemVR, VR supplements visual information to the middle or deep layers of LLMs by taking the original hidden state as queries to search for visual features that need to be supplemented, without introducing image-related bias, nor amplifying image redundancy errors. Experimental results on 8 benchmarks show that VR effectively improves performance.

[2] From Redundancy to Relevance: Enhancing Explainability in Multimodal Large Language Models

Q6: Why not use the uncertain score as $\alpha$?

Thank your nice suggestion. The uncertain score is usually between 0.5 and 0.99, which is large, and the original knowledge would be confused if we use the uncertain score as $\alpha$. To achieve a dynamic injecting ratio, we calculate $\alpha$ by 2*(uncertain_score – threshold), which this named MemVR++, and the results are as follows,

LLaVA-Bench:

POPE (MSCOCO):

The results show that, compared with MemVR, MemVR++ can also achieve the same improvement, and even better.

Q7: Tested benchmarks and contribution

Follow your suggestion, we test MemVR on the challenging HallusionBench, the results are as follows,

where the evaluation is conducted on the GPT-4o-mini.

The results demonstrate that MemVR and MemVR++ achieve superior performance on HallusionBench.

Q8:Minor typos

We have revised our paper accordingly, the typo "preception" is revised to "perception".

We sincerely appreciate your valuable suggestions again.

We sincerely thank Reviewer bEEH for the constructive comments on our work. We are very grateful to the reviewer for recognising the novelty of our idea and the richness and rationality of our experiments.

Q1: Missing comparisons with cross-attention based retrieval.

Sorry for the confusion, we wish to clarify that the ‘cross-attention’ mentioned in our paper refers specifically to a text–image alignment strategy used during the training, rather than the specific method for mitigating VLM hallucinations. We are simply stating that the VR strategy has less overhead than injecting information through cross-attention. Besides, the cross-attention mechanism needs to be trained.

We make complexity analyses between cross-attention and FFN, and VR operations. Let $d$ be the dimension of the hidden state, $D$ denote the dimension of FFN, and $N_v$ and $N_t$ denote the number of vision/text tokens. We have computational complexity (CO) analysis, CO_cross-attn=$\mathcal{O}((N_v+N_v)N_v d+N_vd^2 )$; CO_FFN=$\mathcal{O}((N_v+N_t) d D)$; CO_VR=$\mathcal{O}((N_v+N_t) d N_v)$. Where CO_VR< CO_FFN < CO_cross-attn.

Q2: The layout can be further improved and Typo.

Thanks for your helpful suggestion. We have revised our paper accordingly, including layout and typos, so that it will be easier to read.

Q3: Hyperparameter Tuning.

We value your concerns. We develop MemVR++, which alters the injection rate of visual information from fixed to self-adaptive, thus discarding the hyperparameter $\alpha$. Specifically, the retracing ratio $\alpha$ is determined by 2*(layer_entropy – entropy_threshold) when MemVR is triggered. This ensures that the higher layer_entropy is, which indicates that the model is more confused, the higher $\alpha$ would be. This is a global strategy for MemVR. We've tested it on several benchmarks, and the evaluation shows that this dynamic retracing strategy is also significantly outperforming the default one. The results are as follows,

LLaVA-Bench:

POPE (MSCOCO):

POPE (A-OKVQA):

POPE (GQA):

MME Benchmark:

And we supplement the testing of MemVR on HallusionBench.

HallusionBench:

where the evaluation is conducted on the GPT-4o-mini.

The results demonstrate that MemVR and MemVR++ achieve superior performance. We'll include this part in our revised version.

We sincerely thank you for your kind suggestions.

Thank you for your insightful comments and kind suggestions. 

Q1: The contribution of revisiting visual tokens.

MemVR is fundamentally different from the current approaches, you may refer to Table 2 in the paper, where we show comprehensive comparisons with existing methods. 

1. Retrieval-based approach incorporates knowledge from the external database, which brings about a large memory footprint. MemVR achieves self-improvement without external knowledge.

2. CD-based and attention intervention strategies both bring about high inference latency, due to multiple rounds of inference or the rollback operation. MemVR mitigates hallucinations and excels beyond SOTA methods without incurring additional time overhead.

3. Retrieval-based, CD-based, and attention intervention methods generally act on textual/visual input (i.e., instruction/image), output logits, or attention matrix. MemVR reinjects visual information at hidden states.

4. Compared with other methods, which succeed in hallucination benchmarks but fail on general benchmarks, MemVR enables constant performance boosting in both hallucination and general benchmarks.

Q2: Add analysis of failure cases.

Thank you for your valuable and helpful suggestions. We are keen to explore how reinjecting similar visual features without external data might affect model biases. To address this, we have collected failure cases from the MME benchmark, in the 'Celebrity,' 'Scene,' and 'Landmark' sub-tasks, where MemVR underperforms compared to the default model, as follows. 

We categorize MemVR's failures into two types:

1. Cases where the default model provides the correct answer, but MemVR outputs an incorrect one.

2. Cases where both the default model and MemVR produce incorrect answers.

For failure type 1, we attribute the failure to the over-disturbance of the default model's reasoning process. In these instances, the original visual features are sufficient for reasoning, and the reinjected tokens inadvertently disrupt this process, leading to errors. We are actively investigating methods to mitigate such disturbances. For failure type 2, the failures arise from either the excessive complexity of the image or gaps in the LLM's knowledge base, which prevent correct reasoning even after VR.

We will be including the bad cases as well as the analysis in our paper.

Q3: Why are specific layers selected for VR

The layer selection for the dynamic trigger is theoretically grounded in information entropy analysis. Specifically, we employ layer-wise entropy $H(x)=-\sum p(x) \log p(x)$ as an information bottleneck metric to identify critical transition points where feature uncertainty reaches local maxima, which guides information entropy of the trigger layer from high to low state under the Data Processing Inequality[1]. While our implementation adopts an efficient thresholding mechanism, the core selection criterion stems from established analysis of hierarchical feature stabilization patterns in [2], rather than arbitrary heuristics.

[1] Cover, T. M. et al. Entropy, relative entropy and mutual information. Elements of Information Theory, 2(1):12–13, 1991.

[2] Teerapittayanon et al., BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks, ICPR 2016.

Q4: detailed breakdown of computational overhead.

LLaVA components include ViT+MLP+LLM (self-attention and FFN). Suppose $L_v$ and $L_l$ are the number of layers of ViT and LLM, $d_v$,$d$, and $D$ are dimensions of ViT, self-attention, and FFN, $N_v$ and $N_t$ denote the number of vision/text tokens.

1. Computational overhead of VCD: FLOPs$_{\text{VCD}}$=$2*(L_v(N_v^2 d_v+N_v d_v^2)+N_v d_v d+L_l[(N_v+N_t)^2 d+(N_v+N_t) dD])$

2. Computational overhead of MemVR: FLOPs$_{\text{MemVR}}$=$L_v(N_v^2 d_v+N_v d_v^2)+N_v d_v d+L_l[(N_v+N_t)^2 d+(N_v+N_t) dD] + \underline{(N_v+N_t) d{N_v}+L_o (N_v+N_t) d}$. where $L_o \lt L_l$.

Clearly, MemVR adds $\underline{\text{underline}}$ overhead terms, but it is negligible as $(N_v+N_t) d$ is low computation, and $N_v \ll D$, for instance $D = 11008$ and $N_v = 256$ for LLaVA, thus $(N_v+N_t) d{N_v}\ll (N_v+N_t) d D$, which makes VR operation efficient in total.

We hope the content above can address your concerns. Please let us know if you have further questions.

We sincerely thank you for your kind suggestions again.