Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention

We sincerely thank you for taking the time to review our paper and for offering constructive feedback. We truly value your effort and have carefully addressed your concerns and questions below.

Comment 1: The training-free method is plug and play, but the methods applied in this paper are too few, only LLaVA and Minigpt4 are included, and the results of other models should be supplemented.

Response 1: Thank you for your valuable feedback. To address your concerns, we have extended our experiments to include additional models, such as LLaVA-13B and Llama-3.2-11B-Vision, which are both modern and representative, with Llama-3.2-11B-Vision released just two months ago. Using the same settings on the COCO dataset, our method demonstrated significant improvements, reducing hallucination rates by approximately 6 points for Llama-3.2-11B-Vision and achieving a 10-point improvement in CHAIR_S for LLaVA-v1.5-13B. Here are the detailed results:

We are actively working on expanding our experiments to include additional models and plan to incorporate these results in a future revision of the paper.

Comment 2: The paper discusses the attention-map relationship of text tokens between LVLMs/LLM. Although the setting is a hallucination problem, this mechanism should be attributed to the cause of LLM. So can datasets be extended to other datasets like ScienceQA, GQA, textVQA, POPE, mmbench, etc.?

Response 2: Thank you for your insightful question. The datasets you mentioned primarily involve short-form tasks with a focus on comprehension and discriminative features. For instance, POPE includes questions such as determining whether a chair or car exists in an image with a "yes" or "no" answer, where the object in question is explicitly mentioned in the prompt. These tasks differ fundamentally from open-ended generation tasks, where models must sequentially generate a series of tokens to complete the task without such explicit guidance in the prompt. We applied our method, AD-HH, to these discriminative tasks and observed some improvement, though the gains were relatively modest:

The limited improvement may be attributed to the inherent differences between discriminative and generation tasks. In open-ended generation tasks, language bias becomes increasingly pronounced during the later stages of token generation [1][2], as the generated tokens deviate further from the input image tokens, leading to hallucinations. In contrast, in discriminative tasks, the generated tokens' position is closer to the image tokens and are less affected by such biases. This suggests that interventions targeting attention heads, like AD-HH, may not yield significant benefits for these tasks.

[1] Yiyang Zhou et al. Analyzing and mitigating object hallucination in large vision-language models. ICLR, 2024

[2] Qidong Huang et al. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. CVPR, 2024.

Question 3: If it is a task like VQA, the generated text token is only one, and only ABCD is answered, whether this hallucination pattern does not exist. If it is the SFT method, then this result should also be applicable to VQA tasks?

Response 3: Thank you for your question. We have included experimental results and a discussion on VQA tasks in Response 2.

Question 4: It is written in the paper that the author has hallucinatory attention-head in the middle layer or deep layer. Why is this? The complete distribution of 32 heads was not seen in the supplementary materials.

Response 4: Thank you for your question. We are happy to clarify. Our conclusion that hallucination heads predominantly occur in the middle or deep layers is based on the complete distribution of all attention heads, as shown in Figure 2 (LLaVA-7B) and Figure 10 (MiniGPT-7B). These figures depict the contributions of all 1024 attention heads (from 32 layers with 32 heads per layer). The color coding in these figures represents each attention head's contribution to hallucination behavior. From the color distribution, it is evident that hallucination heads are concentrated in the middle and deep layers.

Question 5: Has it analyzed the difference between the attention patterns of hallucination heads and regular attention heads across system tokens, image tokens, prompt tokens, and output tokens? Is there any distinction, or can differences only be observed through output tokens?

Response 5: Thank you for your question. We believe you are referring to Figure 4, which presents the attention patterns of hallucination and non-hallucination heads, albeit limited to output tokens. To address your concern, we have included an additional analysis in Figure 13 in the Appendix, which examines attention patterns across the entire sequence, encompassing system tokens, image tokens, prompt tokens, and output tokens. Please let us know if we have misunderstood your question.

Specifically, Figure 13 in the Appendix highlights that hallucination heads allocate significantly more attention weight to text tokens, often neglecting visual tokens. In contrast, non-hallucination heads distribute attention more evenly, with a greater focus on visual tokens. These findings are consistent with the observations in Figure 4 and further support the conclusion that hallucination and non-hallucination heads exhibit distinct attention patterns when processing input sequences.

Question 6: What about the distribution of attention when judging LLMs hallucinations? LLMs unable to generate a caption to an image, right? Therefore, I don't think this kind of attention head observation at the decoder terminal is very reasonable.

Response 6: Thank you for your question. If we understand correctly, you are drawing a connection between the hallucination heads identified in LVLMs and their behaviors in LLMs, and asking whether our conclusions extend to LLMs. Please let us know if we’ve misunderstood your intent.

To clarify, hallucinations in LVLMs are fundamentally different from those in LLMs, and our conclusions do not directly apply to hallucinations in LLMs. Specifically, LVLMs generate responses by integrating visual and textual information. The hallucinations we observed are caused by an imbalance in attention between image and text tokens in certain hallucination heads. This imbalance likely arises because LVLMs are fine-tuned from LLMs, where language bias tends to dominate during question answering, causing the model to undervalue visual input. Thus, this issue is inherently tied to modality competition between vision and language in LVLMs.

In contrast, hallucinations in LLMs occur in a single-modal setting and are primarily related to factors within in the language model itself, which requires separate investigation. Our findings are specific to multimodal models and do not generalize to LLMs in isolation.

---

We sincerely thank you for your thoughtful review. We hope the revised manuscript and the clarifications provided above have addressed your concerns effectively. If our responses have satisfactorily resolved your concerns, we would greatly appreciate it if you could consider updating your review score. However, if you have any remaining concerns or require further clarification, please do not hesitate to let us know. We are more than willing to provide additional explanations or updates as needed.

Questions 7: Do you think "hallucination heads" exist in the same way in larger scale (eg. 70b) VLMs? Would the same training-free method work similarly on them? Would be interesting to see.

Response 7: Thank you for raising this thought-provoking question. We have extended our method on larger models including Llama-3.2-11B-Vision and LLaVA-v1.5-13B, which still works on them. It is indeed exciting to consider whether hallucination heads behave differently or even vanish in larger-scale models like 70B. Unfortunately, we currently lack the computational resources and time to study this topic. However, we can share some preliminary thoughts that future work may explore further.

First, we note that large models differ from small models in both representation power and optimization dynamics. Intuitively, larger models are expected to possess greater representation capacity. Additionally, neural network learning theory, such as the Neural Tangent Kernel framework, suggests that larger models tend to change their parameters slowly at each training step. Building on this foundation, we propose two potential scenarios based on the extent of training:

• Short-training regime: In this scenario, larger models may strongly inherit pre-existing language biases, and hallucination heads are likely to exist. This outcome is expected when the model is fine-tuned on small-scale dataset to adapt to specific downstream tasks, especially if the language model backbone has been extensively trained on text data alone.
• Long-training regime: With sufficient training, larger models can utilize their superior representation power to learn accurate patterns from extensive datasets—something smaller models often struggle to achieve. Hallucination heads are expected to disappear in this scenario.

In conclusion, we cannot provide a definitive answer at this time. The persistence or mitigation of hallucination heads likely depends on the scale of training efforts. This relationship is challenging to predict in advance but is an interesting area for future research. We hope our study can provide some insights and plan to study this topic in the future.

---

We sincerely thank you for your valuable review comments. We hope the revised manuscript and the clarifications provided have effectively addressed your concerns. If our responses meet your expectations, we would greatly appreciate your consideration in updating your review score. Should you have any remaining questions or require further clarification, please do not hesitate to reach out. We are more than happy to provide additional explanations or updates as needed.

Thank you for reading our paper and providing valuable feedback. We greatly appreciate your comments and have addressed your concerns and questions below.

Comment 1: There is limited discussion on how the proposed interventions affect the model's overall language generation capabilities. Potential trade-offs between reducing hallucination and maintaining fluency or coherence are not thoroughly examined.

Response 1: Thank you for raising this concern. We acknowledge the potential trade-offs between reducing hallucination and maintaining the overall generation quality. This trade-off was observed when applying the initial naive method in our experiments, as discussed in Section 4.3.1 (refer to Figure 6).

To address this issue, we developed approaches that achieve a more balanced trade-off, such as adaptive deactivation of hallucination heads and targeted fine-tuning. Supplementary results in Table 4, Figures 14 and 15 provide a detailed analysis of these trade-offs, illustrating how varying hyperparameters in our methods impacts both hallucination reduction and generation quality.

Comment 2: The paper is almost entirely focused on object hallucination.

Response 2: Thank you for your comment. Object hallucination is a significant challenge for LVLMs in open-ended generation tasks, which is why we chose to focus on it in this study. Nevertheless, our method is not limited to object hallucination and can be extended to address other types of hallucination, such as counting and positional errors. For instance, on the MME benchmark, we observed a 5-point improvement in counting performance (from 148 to 153), a 10-point improvement in positional accuracy (from 128 to 138). Detailed results are reported below:

Comment 3: The experiments are conducted on 7B parameter models (LLaVA-7B and MiniGPT-4). Given the trend towards larger models in the field, it would be valuable to assess whether the identified hallucination heads and mitigation strategies are applicable to larger models (e.g., 70B parameters) and whether similar patterns emerge at different scales.

Response 3: Thank you for this insightful comment. Due to limited computational resources, our initial experiments focused on 7B parameter models. We completely agree that investigating hallucination behaviors in larger models and studying scaling and emergence effects is interesting and important. To address your concern, we extended our experiments to larger models, including Llama-3.2-11B-Vision and LLaVA-v1.5-13B. Unfortunately, resource and time constraints prevented us from exploring 70B parameter models. Nevertheless, our findings indicate that the proposed method is effective on 13B-sized models. Comparative results for hallucination on the COCO dataset are provided below.

Comment 4: There are minor issues with the writing and presentation that could be improved for clarity and professionalism. For example, phrases like "Our Method Run Fast in Generation" could be rephrased for better readability.

Response 4: Thank you for your suggestion. We have revised the paper to enhance clarity.

Question 5: Can you provide more details on why certain hallucination heads exhibit slow changes during instruction tuning? What factors contribute to this "laziness," and how might future work address this issue?

Response 5: Thank you for the thoughtful questions. We address them separately below.

1. Why do certain hallucination heads exhibit slow changes during instruction tuning?

Our analysis of gradient norms for these hallucination heads revealed that their gradients are consistently smaller compared to non-hallucination heads throughout training. For instance, at iteration 0, the gradient norm for hallucination heads was 1.3 times smaller than that of non-hallucination heads. This trend persisted across multiple iterations (e.g., 500, 1000, 2000, 3000, 4000, and 5000), explaining their slower updates during training.

We hypothesize that this "laziness" arises from over-optimization during pre-training and instruction tuning of the base language model. Over time, certain attention heads may become less responsive to new inputs, such as multi-modal data in LVLMs. Ideally, landscape analysis could provide insight into the sharpness and plasticity of these attention heads. However, existing visualization techniques are not yet scalable to 7B models, limiting this approach.

2. How might future work address this issue?

Addressing this challenge requires careful fine-tuning, particularly in low-data scenarios where models tend to inherit shortcut patterns from their base language models. These inherited biases can amplify hallucinations in open-ended tasks, making it crucial to adapt LVLMs to specific tasks and datasets effectively.

We highlight a few future directions:

• From the perspective of training algorithms, future research could explore targeted mitigation strategies inspired by continual learning literature (see e.g., [1]).
• From the representation perspective, techniques such as model expansion (see e.g., [2,3]), which introduce more flexible components, could enhance adaptability.
• From the data perspective, additional efforts to curate and augment datasets could further improve performance and robustness.

[1] Dohare, Shibhansh, et al. "Loss of plasticity in deep continual learning." Nature 632.8026 (2024): 768-774.

[2] Yoon, Jaehong, et al. "Lifelong learning with dynamically expandable networks." ICLR 2018.

[3] Anagnostidis, Sotiris, et al. "Navigating Scaling Laws: Compute Optimality in Adaptive Model Training." ICML 2024.

Question 6: Have you evaluated how the proposed interventions affect the overall language generation quality of the models? Specifically, does reducing reliance on text tokens in hallucination heads impact the fluency, coherence, or descriptiveness of the generated captions? It would be helpful to see metrics (human eval) or analyses addressing potential trade-offs.

Response 6: Thank you for the suggestion. To evaluate the impact of our interventions, we conducted a manual assessment involving two Ph.D. students and one undergraduate student. They evaluated the responses using a 1 to 5 scoring system based on two criteria: (1) Non-hallucination performance, where higher scores reflect fewer hallucinations, and (2) Generation quality, where higher scores indicate more fluent and coherent outputs.

For both the baseline and our proposed methods, the evaluators assessed a total of 500 generated responses per method, resulting in 1500 responses overall. These results demonstrate that our methods effectively mitigate hallucination while maintaining high generation quality.

Thank you very much for the additional experiments and commentary. I feel more confident now about the results presented in the paper. I feel like the rating for this paper now may be updated to 7 which is unfortunately not an option here.

1- However, I still believe that showing the scalability of this method (at least to the 30B~) scale would be very substantial. I understand that it may require a lot of VRAM to analyze a 70B, but it would be interesting to analyze a more intermediate model like LLaVa[1] 34B

2- Have you also explored "early fusion" approaches such as Chameleon[2] (https://huggingface.co/facebook/chameleon-7b https://huggingface.co/facebook/chameleon-30b) and whether hallucination heads exist there? A negative result would be fine and could make sense.

3- Are you willing, upon paper acceptance, to release open-source code for AD-HH, FT-HH, as well as for the analysis shown in Sec. 4 of your paper so that future works may further investigate the very interesting phenomenon you present in this paper? Open source weights for FT-HH models would also be interesting.

If any one of those points are addressed I would be more than happy to update my score.

[1] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024). [2] Team, Chameleon. "Chameleon: Mixed-modal early-fusion foundation models." arXiv preprint arXiv:2405.09818 (2024).

Thank you for your feedback. We are pleased to hear that our previous responses addressed your concerns. We apologize for the typo of "FT-HH" in the previous response, and we have corrected it. Below, we have provided responses to your new questions:

Question 1: However, I still believe that showing the scalability of this method (at least to the 30B~) scale would be very substantial. I understand that it may require a lot of VRAM to analyze a 70B, but it would be interesting to analyze a more intermediate model like LLaVa[1] 34B.

Response 1: We understand your concern and have tried our best to study the LLaVA-v1.6-34B (LLaVA-34B for short) model. We have applied our method to attribute and intervention on this model and found reduced hallucination rate. Detailed results are presented below:

First, we evaluate the hallucination performance of LLaVA-34B on the COCO dataset. For LLaVA-34B, the hallucination rate is 23.2% for CHAIR_S and 6.4% for CHAIR_I, which are smaller than that of LLaVA-v1.5-7B (LLaVA-7B for short) and LLaVA-v1.5-13B (LLaVA-13B for short). According to the technical report of LLaVA-34B, these improvements can be attributed to a better LLM backbone and an increased input image resolution. We observe a general trend of decreasing hallucination rates as model size increases. However, as noted, this improvement is likely to be influenced by multiple factors.

Second, we apply our modular attribution to analyze hallucination heads across different scales of LVLMs. To fairly compare the number of hallucination heads across models of varying scales, we identify salient hallucination heads —those with contrastive influence values exceeding 25% of the maximum contrastive influence value for each model. The 25% threshold is temporarily selected for comparison purpose. We evaluate both their absolute numbers and their ratio relative to all attention heads.

Our findings above reveal that hallucination heads tend to diminish as model size increases and sufficient post-training is applied. Although we cannot perfectly isolate the contributions of individual factors (e.g., the LLM backbone, data size and sources, image tokenizer), our observations tend to align with our hypothesis: larger models possess stronger representational power to learn correct behaviors from data, whereas smaller models are more susceptible to language bias.

Finally, we apply our targeted modular intervention method to LLaVA-34B and find that the hallucination rate can be reduced by 2.8% on CHAIR_S and 0.8% for CHAIR_I. Note that this improvement is not easy given the superior performance of LLaVA-34B. We also perform preliminary generation quality evaluation and found that the BLEU score is comparable with the one before intervetion.

Thank you for initiating the discussion on model behaviors, particularly regarding model size scaling. We have updated our paper to include these results in Appendix A.3. We believe these findings could help us better understand hallucinations in LVLMs and hope they could be valuable to the community.

Question 2: Have you also explored "early fusion" approaches such as Chameleon [2] (https://huggingface.co/facebook/chameleon-7b https://huggingface.co/facebook/chameleon-30b) and whether hallucination heads exist there? A negative result would be fine and could make sense.

Response 2: Thank you for highlighting the "early fusion" model. We appreciate the work on Chameleon, which offers a comprehensive study of early fusion-based multi-modal training, enabling reasoning and generation across modalities using shared representations. Chameleon-30B is easy to use and achieve state-of-the-art performance on many tasks, but we have observed that it still exhibits hallucination behaviors.

First, we observed that Chameleon-30B has a hallucination rate of 38.0% on CHAIR_S and 12.6% on CHAIR_I on the COCO dataset. Next, we identified the top 10 most salient hallucination heads in Chameleon-30B, as shown in Figure 16(c) in the Appendix. By deactivating these identified hallucination heads, we successfully reduced the hallucination rate on CHAIR_S from 38.0% to 34.8%, with a comparable BLEU score.

Question 3: Are you willing, upon paper acceptance, to release open-source code for AD-HH, FT-HH, as well as for the analysis shown in Sec. 4 of your paper so that future works may further investigate the very interesting phenomenon you present in this paper? Open source weights for FT-HH models would also be interesting.

Response 3: Absolutely! Upon paper acceptance, we will release the code and model weights necessary to reproduce our analysis and experiment results. We hope this will benefit the research community and inspire further investigation.

---

We sincerely thank you again for your valuable review comments. We hope our responses have adequately addressed your questions, and we are happy to provide further clarification if needed.

Comment 3: Some findings look obvious.

a) It is not so surprising that the multi-head-attention is more critical than feed-forward network, since the former takes a majority of parameters and does much more things than the latter in the transformer.

b) It is a well-known fact that hallucination is more related to language bias (rather than) image bias and multimodal models often ignore the information from images compared to that from text.

Response 3: Thank you for the discussion. We address parts (a) and (b) separately:

• For part (a): We would like to respectually point out a factual inaccuracy in your claim. In transformers, the MLP modules usually contain more parameters than the multi-head attention (MHA) modules. For instance, in LLaMA-2-7B, MLP modules account for approximately 65% of the total parameters, whereas MHA accounts for only 33% (roughly half of the MLP parameters). Therefore, the assertion that MHAs are more critical solely because they contain more parameters is incorrect.

  • Our findings demonstrate that despite having fewer parameters, MHA plays a disproportionately significant role in affecting hallucination. This insight is new and important, leading deeper investigations into the behavior of MHAs, as detailed in Section 4.2 of our paper.

• For part (b): We assume you are referencing prior works (e.g., Leng et al., 2024; Huang et al., 2024), which have also observed that LVLMs tend to underutilize visual information during text generation. We acknowledge and appreciate the contributions of these studies. However, our work focuses on identifying specific network components responsible for hallucination, moving beyond the broad notion of language bias.

  • Our findings highlight that not all MHAs are equally implicated in hallucination; rather, a small subset (fewer than 3%) is primarily responsible. Furthermore, we demonstrate that targeted interventions on these components are effective in mitigating hallucination. In summary, our work offers new and non-obvious insights into the mechanisms of hallucination in LVLMs and provides actionable strategies to address it. We believe this makes a meaningful contribution to the field.

Question 4: A more in-depth analysis is required to explain why the fine-tuning approach is not as effective as the training-free approach.

Response 4: We would like to clarify that in our experiments, the fine-tuning approach (TF-HH) performs comparably to the training-free approach (AD-HH). This is evident in Tables 1 and 2. While AD-HH achieves a slightly better average score in Table 1 (24.06 vs. 24.45 for TF-HH), TF-HH slightly outperforms AD-HH in Table 2 (29.9 vs. 29.05).

Theoretically, the training-free approach operates directly in the function space of the Transformer by manipulating the outputs of attention heads (e.g., setting the outputs to zero). In contrast, the fine-tuning approach operates in the parameter space, where the final result is determined by the training method and the model's parameter capacity. This distinction makes it more challenging for the fine-tuning approach to achieve precise manipulations, such as setting outputs to zero, as effectively as the training-free method. However, the fine-tuning approach offers greater flexibility in preserving generation quality by incorporating richer, data-driven adjustments.

We believe both approaches have unique advantages and limitations. In practice, we observe that neither dominates the other, and both achieve comparable performance under different scenarios.

---

We sincerely thank you for your thoughtful review. We hope the revised manuscript and the clarifications provided above have addressed your concerns effectively. If our responses have satisfactorily resolved your concerns, we would greatly appreciate it if you could consider updating your review score. However, if you have any remaining concerns or require further clarification, please do not hesitate to let us know. We are more than willing to provide additional explanations or updates as needed.

Thank you for taking the time to review our paper and provide valuable feedback. We greatly appreciate your efforts and have addressed your concerns and questions below.

Comment 1: This work fails to cite several related recent works. They may need to be cited and compared in experiments.

Response 1: Thank you for bringing these works to our attention. We have added a discussion of these studies to the related work section and appreciate their contributions:

• [Deng et al., 2024] proposed a CLIP-guided decoding approach that utilizes CLIP as an external tool to alleviate hallucination during decoding.
• [Liu et al., 2024] addressed the issue by constructing the Large-scale Robust Visual (LRV)-Instruction dataset, which includes both positive and negative instructions to enhance the robustness of visual instruction tuning.
• [Zhang et al., 2024] introduced a large-scale instruction-tuning dataset name REVERIE with reflective rational annotations, to enable the model to justify whether the reponses are correct or incorrect.

Different from these works, our work takes a different direction by specifically identifying, analyzing, and adapting the components within the model responsible for hallucinations. Additionally, we have conducted experiments to empirically compare our method with these baseline approaches on the LLaVA model (see Table 8 in the Appendix ). The results below demonstrate that, although these baselines also help reduce hallucination errors, our method, which employs targeted interventions, is more effective in mitigating object hallucinations in open-ended generation tasks.

Our current evaluation focuses on the LLaVA model, as the results of these methods are not directly applicable to our settings due to differences in model versions and evaluation protocols (details provided in the Appendix A). Consequently, re-implementing these baselines is necessary, and some methods, such as LRV and REVRIE, require extensive training that demands significant computational resources. We are actively conducting further assessments with additional methods and settings and plan to share updated findings later. Nevertheless, the current evidence strongly supports the effectiveness of our proposed methods in mitigating hallucinations.

Comment 2: The evaluation only depends on two CHAIR metrics.

Response 2: Thank you for raising this concern. To address your concern, in addition to the two CHAIR metrics used to evaluate the effectiveness of our method in mitigating hallucination, we have also conducted a human evaluation as suggested by Reviewer Ex6k.

Specifically, we asked two Ph.D. students and one undergraduate student to manually evaluate the responses. They were instructed to score each response on a scale of 1 to 5 based on two criteria: (1) non-hallucination performance, with higher scores reflecting fewer hallucinations, and (2) generation quality, with higher scores indicating more fluent responses. For both the baseline and our proposed methods, the evaluators assessed a total of 500 generated responses per method, resulting in 1500 responses overall. These results demonstrate that our methods effectively mitigate hallucination while maintaining high generation quality.

Furthermore, we also evaluated our approach using MM-Vet, as presented in Table 2. The results confirm that intervening on hallucination heads leads to improved performance in general tasks, further validating the effectiveness of our approach.

Question 6: Why the Algorithm 1 directly deactivated the whole text attention weights for the hallucination heads? Isn't it too aggressive as we can see the BLEU scores drop by a lot in Figure 6 (c)? Moreover, why the accuracy on general tasks somehow improves when you apply Algorithm 1? I thought the performance should drop based on the observation in Figure 6 (c).

Response 6: Thank you for highlighting these important points. We would like to clarify a potential misunderstanding: Algorithm 1 employs an input-dependent adaptive strategy to selectively deactivate text attention weights for hallucination heads, unlike the naive strategy used in Figure 6(c), which lacks such adaptability. Therefore, the performance of Algorithm 1 cannot be directly inferred from the observations in Figure 6(c).

To address your concerns about aggressiveness, it is important to note that hallucination heads account for only a small subset (less than 3%) of the Transformer's overall attention mechanism. This ensures that other attention heads remain unaffected, preserving overall generation quality. Thus, it is not overly aggressive to fully deactivate the entire text attention weights for hallucination heads. Evidence for this is provided in Table 4 of the Appendix, which shows that the BLEU score of Algorithm 1 (17.8) is comparable to that of normal generation (17.9).

Finally, the improvement of accuracy in general tasks, as outlined in Table 2, can be attributed to the mitigation of spurious dependencies introduced by hallucination heads. This ultimately enhances model robustness and improves task-specific performance.

Question 7: What is the difference between downscaling weight on text attention and upscaling the weight on image attention (Figure 6 (a) and (b))? I thought they meant the same thing as the softmax in attention would keep the sum of text attention and image attention to be one.

Response 7: In our experiments, scaling is applied after computing the softmax attention scores by multiplying a scaling factor to either downscale or upscale specific components. As a result, the sums of text and image attention may not necessarily equal one. This approach allows us to adjust the text attention weights independently without directly affecting the visual information component. Consequently, we can isolate and analyze the separate contributions of text and visual information more effectively.

---

We sincerely thank you for your thoughtful review. We hope the revised manuscript and the clarifications provided above have addressed your concerns effectively. If our responses have satisfactorily resolved your concerns, we would greatly appreciate it if you could consider updating your review score. However, if you have any remaining concerns or require further clarification, please do not hesitate to let us know. We are more than willing to provide additional explanations or updates as needed.

Thank you for reviewing our paper and for your positive feedback. We truly appreciate your thoughtful comments and have addressed your concerns and questions below.

Comment1: In Section 4.3, the metrics (CHAIR) and the dataset that is evaluated should be explained.

Response1: Thanks for this suggestion. We have added more explanation about the CHAIR metrics and te dataset in Section 4.3 to improve readability.

Comment 2: In Tables 1 and 2, it would be better if there is an additional column indicating which method is training-free and which is not.

Response 2: Thanks for this suggestion. We have added symbol † to denote training-free and symbol * to denote training-based in our paper to make this clear.

Question 3: How this method can extend beyond object-level hallucination? For example, do equation (1) and (2) only tell if the existence of the objects but cannot capture if the model incorrectly counts the objects? Then the proposed method can not detect the heads which make hallucination on counting.

Response 3: Thanks for raising this discussion. We would like to clarify that our framework is flexible and can be extended to attribute errors beyond object-level hallucinations, including errors related to counting, positional reasoning, and other aspects.

Take the hallucination by counting the example you mentioned, we have conducted the experiment on the MME dataset. We take the wrong answer as hallucination token and correct answer as non-hallucination, and identify heads that are mostly associated with counting hallucination applying equation (1) and (2). The counting performance is improved by 5 points from 148 to 153. Besides, many other aspects such as position, posters, OCR are also improved by adaptively deactivating hallucination heads. Detailed results are reported below:

Question 4: How many samples is required to compute equation (1)? And what data split do they come from?

Response 4: We randomly select 1500 samples from the COCO training split to compute equation (1). This detail is provided in the main text (Second paragragh in Section 4.1).

Question 5: What is the time required to compute Equation (2) to obtain the scores for all heads?

Response 5: Computing Equation (2) for all heads takes about about 1 minutes per sample. We understand your concerns regarding the computational complexity associated with the number of attention heads. We would like to highlight that there are faster methods for calculating the influence as in Equation (2). These methods leverage techniques like Taylor expansion to approximate the influence of all attention heads in a single backpropagation step (e.g., as described in [1]). Such approaches are highly efficient, requiring only a single forward and backward pass, and the computation time is independent of the number of heads. These approaches can be explored in future work.

[1] Achtibat, Reduan, et al. "Attnlrp: attention-aware layer-wise relevance propagation for transformers." arXiv preprint arXiv:2402.05602 (2024).