From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models

W3: About experiments.

• Baselines: HA-DPO and HACL are both methods requiring training. Do you train them with the same training set with PATCH?
• We care about hallucination, but we do not want to hurt the utility of LVLMs. Therefore, besides experiments on hallucination benchmarks like POPE and PhD, results on utility benchmarks (e.g., MME, MMbench, and SEED) are also important to demonstrate that hallucination mitigation is not at the cost of utility.
• Overall, I think this is an interesting paper, which, however, needs further work on 1) method clarification, 2) effectiveness analysis (i.e., what it can and cannot do), and 3) utility experiments.

Response： 1）Baselines: Thank you for your detailed review. This is indeed an important issue that requires careful consideration, as differences in training set can impact the fairness of performance comparisons which is a common challenge in multi-modal hallucination tasks. In our experiments, we utilized the POPE (adversarial version) dataset constructed from the A-OKVQA dataset provided by [1] as our training set. In contrast, HA-DPO and HACL were trained on data generated by GPT-4, leading to differences in training datasets. To ensure fairness in comparisons, for hallucination mitigation methods requiring training, we directly reported the best results from their original papers. Furthermore, we plan to retrain HA-DPO and HACL on the same training set to provide a more direct comparison and will update the corresponding results in the revised version.

2）Influence on utility benchmarks: Thank you for raising this concern. In practice, for tasks where hallucination mitigation is not significantly impacted, users can opt to bypass PATCH and rely on the original model directly, thereby avoiding any potential negative effects introduced by the additional tokens. For tasks prone to hallucinations, such as VQA, PATCH can slightly enhance the performance, especially for tasks highly related to object attributes. We conducted relevant experiments on VQAv2, and the results are presented as follows. From the experimental results, we observe that our method improves performance on fundamental VLM tasks. Detailed experimental results will be further enhanced and provided in the revised version.

(1) origin: zero-shot inference on MiniGPT-v2, prompt: <Img>[Image]</Img>[vqa][Question]. (2) PATCH: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}".

Reference [1] Li Y, Du Y, Zhou K, et al. Evaluating object hallucination in large vision-language models[J]. arXiv preprint arXiv:2305.10355, 2023.

Thank you for your valuable feedback and constructive comments. We would like to address your concerns in weaknesses and questions one by one as follows.

W1: About the Cascade Mask-RCNN head.

Response: We sincerely apologize if we did not explain this clearly enough. In our preliminary experiments, we combine the visual module of MiniGPT-v2 with a pre-trained Cascade Mask R-CNN head (i.e., the latter case you mentioned). In fact, [1] has provided the pre-trained parameters based on the joint training of the MiniGPT's visual encoder and the Cascade Mask R-CNN head, so we do not need to pre-train it by ourselves.

In practical applications, users have the flexibility to leverage existing high-performance object detection models to detect object categories and their corresponding bounding boxes in images, thus mitigating the additional computational cost associated with training a dedicated object detector.

W2: About PATCH.

Response: Thank you for your valuable feedback.

1）Redundancy in detection information: The experimental results mentioned in the paper have demonstrated that PATCH can effectively integrate the embedding of objects into the prefill stage, improving errors related to attribute hallucination. However, when the query is unrelated to the objects in PATCH, redundancy may arise. We mitigate this redundancy through the following strategies: When the task is related to object attributes, we incorporate the PATCH module to enhance the model's focus on object-related details. Conversely, when the task is unrelated to object attributes, we exclude the PATCH module and utilize task-specific labels (i.g., [identify], [vqa], [refer]) mentioned by MiniGPT-V2 for model guiding.

2）We visualize the attention scores of the last token in the input sequence with respect to the next token across each layer of the LLM. The heatmap reveals that when querying the presence of objects in the detection information, specific embeddings of the objects exhibit higher scores, which indicates that our method effectively enhances semantic alignment through virtual tokens. The corresponding attention map is provided in Appendix D in the revised manuscript.

3）Objects beyond the recognizable classes and beyond experiment numbers: We have conducted ablation experiments by replacing the detected object categories with an generic placeholder "object{idx}". The experimental results have demonstrated a 5.36% accurancy improvement compared to the original baseline, which indicates that PATCH can also play a significant role in mitigating hallucinations, even in scenarios where the visual detection categories are uncertain. The detailed experimental results are below.

(1) origin: zero-shot inference on MiniGPT-v2, prompt: <Img>[Image]</Img>[vqa][Question]. (2) PATCH: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}". (3) PATCH + placeholder: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{idx}{<x1><x2><x3><x4>}".

Reference [1] Fang Y, Wang W, Xie B, et al. Eva: Exploring the limits of masked visual representation learning at scale[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 19358-19369.

Q1: About title.

Response：Since this paper aims to explore the fundamental causes of multimodal hallucination, we investigate the visual encoder (feature extraction) and modality alignment module (feature disentanglement) from an architectural perspective, specifically the process from image pixels to LLM tokens. Therefore, we added "From Pixels to Tokens" to the title. Based on your suggestion, we will continue to consider whether a more suitable title might better reflect the scope of our work.

Q2: About the potential solution.

Response：

1）We obtain the object-related information by utilizing the prediction results of Cascade Mask R-CNN.

2）To address cases where the object names are uncertain, we conducted ablation experiments by replacing the detected object categories with an generic placeholder "object{idx}". The zero-shot experimental results have demonstrated a 2.66% accurancy drop compared to the original baseline, indicating that relying solely on generic placeholders is insufficient for optimal performance. While further combining it with PATCH has resulted in a 5.36% accuracy improvement, demonstrating the effectiveness of PATCH in enhancing the model's understanding ability, particularly in scenarios where the object names are uncertain in the image. For objects that are not present in the image, the visual detection head does not detect any corresponding objects, which makes it more likely for the LVLM to generate a "no" response.

(1) origin: zero-shot inference on MiniGPT-v2, prompt: <Img>[Image]</Img>[vqa][Question]. (2) origin + placeholder: zero-shot inference on MiniGPT-v2, promp)t: <Img>[Image]</Img>Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{idx}{<x1><x2><x3><x4>}". (3) PATCH: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}". (4) PATCH + placeholder: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{idx}{<x1><x2><x3><x4>}".

Dear Reviewer,

We sincerely appreciate your time and effort in reviewing our paper and offering valuable suggestions. Since there will be no second stage of author-reviewer discussions, and the discussion phase is drawing to a close, we would to confirm whether our responses have effectively addressed your concerns a few days ago. And we hope they have adequately addressed your issues. If you require further clarification or have additional concerns, please don’t hesitate to contact us. We are more than willing to continue our communication with you.

Best regards.

Thank you for your valuable feedback and constructive comments. We would like to address your concerns in weaknesses and respond to your questions one by one as below.

W1: The encoder is an additional significant source of hallucinations. Need results after the replacement of prompt$_1$ with prompt$_2$.

Response: Thank you for your valuable feedback. The results in Table 1 suggest that 74.58% hallucinations occur when object detection is accurate while the model’s inference is incorrect, indicating that in the current experiment, the modal alignment module is the primary cause of hallucinations, with a frequency ratio of 3:1 compared to the visual encoder (we believe this ratio would be even higher if a more advanced detector head is used).

Additionally, we conducted experiments with the replacement of prompt$_1$ with prompt$_2$. A comparison of Prompt2's inference results on MiniGPT-v2 with the detection results revealed that incorporating object information into the LLM effectively mitigates multi-modal hallucinations. Furthermore, the tabular results demonstrate consistent behavior between inference errors and detection errors after adding object-related information. We attribute these errors to the limitations of the detector's performance, suggesting that using a more advanced detector could further enhance the performance of our method.

W2: The model relies on an additional object detector that should also be trained.

Response: Thank you for raising this concern. In our experiments, we employed a pre-trained object detection head provided by [1] for object recognition. Consequently, the training cost of this component was not included in the computational cost reported for our method. Notably, MiniGPT-v2 has a total of 7,748,467,072 parameters, while the detection head contains only 48,527,951 trainable parameters. Given the significantly smaller parameter count of the detection head compared to the overall model, training it would not impose a substantial computational burden. Moreover, in practical applications, users have the flexibility to leverage existing high-performance object detection models to detect object categories and their corresponding bounding boxes in images, thus mitigating the additional computational cost associated with training a dedicated object detector.

We appreciate your insightful suggestion about reallocating the computational cost of training an object detector towards enlarging the encoder, adapter, or LLM to potentially achieve a more streamlined solution for reducing hallucinations. This is indeed an intriguing direction for future research, and we plan to explore this aspect further in our future work.

W3: Are the detections necessary, or can they be achieved solely from additional soft-prompting with more learned tokens?

Response：Thanks for posting this valuable question. We conducted a token number ablation experiment where only virtual tokens were added without incorporating the detection results. The experimental results are shown below.

(1) origin: zero-shot inference on MiniGPT-v2, prompt: <Img>[Image]</Img>[vqa][Question]. (2) + k token: fine-tuning k virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens][vqa][Question].

The results clearly demonstrate that the inclusion of detection results is crucial, which indicates that merely increasing the number of trainable prompting tokens is insufficient for mitigating hallucinations in LVLMs. Additionally, increasing the number of virtual tokens raises the fine-tuning costs for the model, which in turn results in a gradual decline in performance when the learning rate and number of training epochs remain fixed.

Reference [1] Fang Y, Wang W, Xie B, et al. Eva: Exploring the limits of masked visual representation learning at scale[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 19358-19369.

Q1 Why localization information is important for VQA? Can you compare the accuracy of the bboxes to their hallucination removal rate?

Response：We conducted ablation experiments by replacing the detected object categories with an generic placeholder "object{idx}". The zero-shot experimental results have demonstrated a 2.66% accurancy drop compared to the original MiniGPT-v2, indicating that relying solely on generic placeholders is insufficient for optimal performance. By observing the third and fourth rows, we can find that adding either only object category information or only object location information allows PATCH to partially mitigate hallucinations in LVLMs. Notably, providing object location information proves to be more effective. This may be attributed to the fact that detecting box information can help LVLMs achieve precise visual localization, thereby ensuring the faithful responses to the input image.

(1) origin: zero-shot inference on MiniGPT-v2, prompt: <Img>[Image]</Img>[vqa][Question]. (2) origin + placeholder: zero-shot inference on MiniGPT-v2, prompt: <Img>[Image]</Img>Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{idx}{<x1><x2><x3><x4>}". (3) PATCH: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}". (4) PATCH + placeholder: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{idx}{<x1><x2><x3><x4>}". (5) PATCH - bboxes: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category".

Q2: Is the VLM even needed, given the soft prompt and the detection results? What is the baseline of soft-prompting a text-only LLM on the object detection results, without the use of images?

Response：We conducted an ablation experiments on MiniGPT-v2's language model (LLaMA2) to evaluate the baseline performance of prompting a text-only LLM solely based on object detection results without incorporating images. The experimental results indicate that relying solely on object detection prompts makes it challenging for the text-only model to infer the existence of objects in the image. Furthermore, as the amount of introduced information increases, the model's ability to effectively utilize relevant information diminishes. This finding further validates that our proposed PATCH method effectively enhances the LLM in LVLMs, enabling it to better leverage the valuable information contained in the input detection results.

(1) prompt1: zero-shot inference on Llama2, prompt: Based on the detection results: [Object], [Question]. The object-related information is included at the [Object] position, formatted as "category". (2) prompt2: zero-shot inference on Llama2, prompt: Based on the detection results: [Object], [Question]. The object-related information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}".

Dear Reviewer,

We sincerely appreciate your time and effort in reviewing our paper and offering valuable suggestions. Since there will be no second stage of author-reviewer discussions, and the discussion phase is drawing to a close, we would to confirm whether our responses have effectively addressed your concerns a few days ago. And we hope they have adequately addressed your issues. If you require further clarification or have additional concerns, please don’t hesitate to contact us. We are more than willing to continue our communication with you.

Best regards.

Thank you for your valuable feedback and thoughtful comments. We would like to address your concerns in weaknesses one by one as follows.

W1: Lack of Clarity in Preliminary Experiments.

Response: We are sorry for the unclear description. In our preliminary experiments, we combine the visual module of MiniGPT-v2 with a pre-trained Cascade Mask R-CNN head, which serves as the visual detection model for object recognition. When the detection model recognizes the object present the positive sample (i.e., the answer of the sample is "yes") or does not detect the object in the negative examples (i.e., the answer of the sample is "no"), we define the result as a "correct detection". In other cases, we define the result as a "wrong detection". We will add more explanation on this in our revised paper.

W2: Insufficient Support in Analysis.

Response: Thank you for your thoughtful feeaback. As mentioned in the paper, we hypothesize that multi-modal object hallucinations in LVLMs arise from two main sources:

1. Insufficient extraction of visual features: The visual encoder may fail to capture critical details or misinterpret objects in the image, resulting in inaccurate or incomplete visual representations.
2. Inadequate decoupling of visual features: Even when the visual encoder generates accurate features, the visual projection layer may struggle to align these features correctly with the corresponding textual embeddings in the LLM's semantic space.

By observing the first row and first column in Table 1 mentioned in our article, we can see that the number of samples with correct detection results exceeds the number of samples with correct inference results. This suggests that the potential of the visual encoder for image understanding has not been fully leveraged by the language model of LVLMs. Consequently, we attribute the primary cause of hallucination issues in LVLMs to the inadequate decoupling of visual features.

If you have any further suggestions, please feel free to share them with us. We would be glad to address them by conducting additional experiments to provide a thorough explanation.

W3: Contradiction in Motivation.

Response: Thank you for raising this concern. In our article, redundancy refers to instances where the detected object has weak relevance to the given question, rendering the object embeddings redundant. In such cases, PATCH assists the language model of the LVLM in filtering and optimizing the use of detection information. By employing trainable and pluggable virtual tokens as a bridge, PATCH enables LVLMs to focus on task-relevant image features, thereby improving the alignment between visual and textual representations within the semantic space. We will provide further clarification on this in the camera-ready version to ensure better understanding for the readers.

W4: Limited Novelty.

Response: Thanks for your feedback. The primary contribution of our research lies in identifying the specific module responsible for hallucination generation in LVLMs, rather than proposing a new lightweight fine-tuning method for enhancing the LVLM performance.

Our preliminary experiments reveal that hallucinations primarily arise in the modal alignment module. To address this issue, we propose the PATCH strategy to enhance the alignment between visual and textual features. PATCH is a plug-and-play method that can be dynamically adjusted based on the requirements of specific applications. Furthermore, we have compared our method with several influential prior approaches, and our experimental results consistently demonstrate superior performance.

W5: Unfair Experimental Setup.

Response: We are sorry for the unclear description. We use the POPE (adversarial version) dataset constructed from the A-OKVQA dataset provided by [1] as our training set. This dataset is identical to the POPE test set only in terms of data format. Since the training set is separate from the test set, the fairness of the baseline comparison can be ensured.

W6: Incomplete Baseline Comparisons.

Response: We understand your concern. We use the POPE (adversarial version) dataset constructed from the MSCOCO dataset provided by [1] as our test set, which is also the test set for all the mentioned baseline methods. To ensure fairness in the comparison, for hallucination mitigation methods that require training, we directly used the experimental results from the original papers.

Reference [1] Li Y, Du Y, Zhou K, et al. Evaluating object hallucination in large vision-language models[J]. arXiv preprint arXiv:2305.10355, 2023.

W7: Omission of Experimental Details.

Response: We apologize if our explanation was not very clear. In our experiments, we did not modify the decoding strategy, and all methods consistently use the greedy decoding strategy. These details will be provided in the revised version of our paper.

W8: Absence of Ablation Studies.

Response: We sincerely apologize for the unclear description. In our experiments, we did not fine-tune the language modeling head or the visual projection layer. Instead, only the virtual tokens on the encoding side were trained during the training phase. We conducted a thorough ablation study in the paper, and for improved clarity, we provide the results of all ablation experiments below. All ablation experiments were conducted based on MiniGPT-v2.

(1) origin: zero-shot inference on MiniGPT-v2, prompt: <Img>[Image]</Img>[vqa][Question]. (2) origin + objs: zero-shot inference on MiniGPT-v2, prompt: <Img>[Image]</Img>Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category". (3) origin + objs & bboxes: zero-shot inference on MiniGPT-v2, prompt: <Img>[Image]</Img>Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}". (4) origin(sft) + objs & bboxes: LoRA fine-tuning on MiniGPT-v2, prompt: <Img>[Image]</Img>Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}". (5) PATCH: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}". (6) PATCH - bboxes: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens]Objects:[Object][vqa][Question]. The object-related information is included at the [Object] position, formatted as "category". (7) PATCH - bboxes & objs: fine-tuning on 20 virtual tokens on MiniGPT-v2, prompt: <Img>[Image]</Img>[Virtual_tokens][vqa][Question].

Among these expeirments, (1)-(3) are proposed to evaluate the impact of different formats of detection results on MiniGPT-v2's inference. (4) is proposed to compare the effects between two different fine-tuning methods: virtual tokens and LoRA. (5)-(7) are proposed to evaluate the impact of different formats of detection results on PATCH.

W9: Overclaimed Results Some claims.

Response: Thanks for raising this concern. First, the results of Table 3 mentioned in our article demonstrate that our method outperforms other approaches on the POPE dataset, achieving superior performance. Second, we visualize the attention scores of the last token in the input sequence with respect to the next token across each layer of the LLM. The heatmap reveals that when querying the presence of objects in the detection information, specific embeddings of the objects exhibit higher scores, which indicates that our method effectively enhances semantic alignment through virtual tokens. The corresponding attention map is provided in Appendix D in the revised manuscript.

W10: Incomplete Related Work.

Response：Thanks for your advice. In the revised version, we will update the related work section to include the latest developments in LVLMs. Additionally, we will add more comparisons with related work in the experimental section (as shown below).

Reference [1] Yang, Jianwei, et al. "Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v." arXiv preprint arXiv:2310.11441 (2023). [2] Zhang Y, Ma Z, Gao X, et al. Groundhog: Grounding large language models to holistic segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024: 14227-14238. [3] Wang, Peng, et al. "Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution." arXiv preprint arXiv:2409.12191 (2024). [4] Li, Bo, et al. "Llava-onevision: Easy visual task transfer." arXiv preprint arXiv:2408.03326 (2024). [5] Liu, Fuxiao, et al. "Mitigating hallucination in large multi-modal models via robust instruction tuning." The Twelfth International Conference on Learning Representations. 2023. [6] Leng, Sicong, et al. "Mitigating object hallucinations in large vision-language models through visual contrastive decoding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [7] Jin, Peng, et al. "Chat-univi: Unified visual representation empowers large language models with image and video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [8] Li, Zhaowei, et al. "Groundinggpt: Language enhanced multi-modal grounding model." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. [9] Yuan, Yuqian, et al. "Osprey: Pixel understanding with visual instruction tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [10] Rasheed, Hanoona, et al. "Glamm: Pixel grounding large multimodal model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Dear Reviewer,

We sincerely appreciate your time and effort in reviewing our paper and offering valuable suggestions. Since there will be no second stage of author-reviewer discussions, and the discussion phase is drawing to a close, we would to confirm whether our responses have effectively addressed your concerns a few days ago. And we hope they have adequately addressed your issues. If you require further clarification or have additional concerns, please don’t hesitate to contact us. We are more than willing to continue our communication with you.

Best regards.

W3: Limited Plug-and-Play Functionality.

Response: We understand your concern. Due to its lightweight design, PATCH can be used in a plug-and-play manner. For tasks prone to hallucinations, such as VQA, virtual tokens can be added to significantly enhance the performance. Especially for tasks highly related to object attributes, we conducted relevant experiments on VQAv2, and the results are presented as follows. In contrast, in scenarios such as content understanding or relational reasoning, virtual tokens can be removed to maintain the original capabilities of the LVLM. This is what we call "plug-and-play", and such flexibility ensures that PATCH remains practical and adaptable for LVLMs already deployed in real-world applications.

W4: Lack of Baseline Comparisons.

Response: Thank you for your valuable feedback. According to your suggestions, we add new comparisons with similar methods, including baseline plug-and-play strategies, such as visual prompt-based methods (i.g., Set-of-Mark, GAVIE, Osprey) and fine-tuning methods (i.e. Groundhog, VCD,Chat-uniVi, GoundingGPT). The results are shown below.

Compared to these methods, it is clear to see that our approach still achieves the state-of-art results, demonstrating its superior performance and advantages in addressing the object hallucination issue in LVLMs.

Reference [1] Yang, Jianwei, et al. "Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v." arXiv preprint arXiv:2310.11441 (2023). [2] Zhang Y, Ma Z, Gao X, et al. Groundhog: Grounding large language models to holistic segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024: 14227-14238. [3] Liu, Fuxiao, et al. "Mitigating hallucination in large multi-modal models via robust instruction tuning." The Twelfth International Conference on Learning Representations. 2023. [4] Leng, Sicong, et al. "Mitigating object hallucinations in large vision-language models through visual contrastive decoding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [5] Jin, Peng, et al. "Chat-univi: Unified visual representation empowers large language models with image and video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [6] Li, Zhaowei, et al. "Groundinggpt: Language enhanced multi-modal grounding model." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. [7] Yuan, Yuqian, et al. "Osprey: Pixel understanding with visual instruction tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [8] Rasheed, Hanoona, et al. "Glamm: Pixel grounding large multimodal model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Thanks for your positive feedback and constructive comments. We would like to address your concerns in weaknesses one by one as follows.

W1: Lack of Novelty in Methodology.

Response: Thanks for your comments. We acknowledge that "utilizing a pretrained model for feature extraction and fine-tuning the vision-language model" is indeed a widely adopted paradigm for improving LVLM capabilities. However, the primary contribution of our research lies in identifying the specific module responsible for hallucination generation in LVLMs, rather than proposing a general approach for enhancing the LVLM performance.

Our preliminary experiments reveal that hallucinations primarily arise in the modal alignment module. To address this issue, we propose the PATCH strategy to enhance the alignment between visual and textual features. PATCH is a plug-and-play method that can be dynamically adjusted based on the requirements of specific applications. Furthermore, we add more baselines you mentioned (as below), and the experimental results consistently demonstrate superior performance of our method.

W2: Insufficient Ablation Study on Virtual Tokens.

Response: We are sorry for the unclear organization of our ablation study. We have conducted a thorough ablation study in our paper. For clarity, we explain the results of all ablation experiments here, and all of them are based on MiniGPT-v2. (1) origin: We perform the zero-shot inference with the prompt: <Img>[Image]</Img>[vqa][Question]. (2) origin + objs: We perform the zero-shot inference with the prompt: <Img>[Image]</Img>Objects:[Object][vqa][Question], where the object category information is included at the [Object] position, formatted as "category". (3) origin + objs & bboxes: We perform the zero-shot inference with the prompt: <Img>[Image]</Img>Objects:[Object][vqa][Question], where the object category and position information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}". (4) origin(sft) + objs & bboxes: We fine-tune on MiniGPT-v2 using LoRA with the prompt: <Img>[Image]</Img>Objects:[Object][vqa][Question], where the object-related information is included at the [Object] position, formatted as "category{<x1><x2><x3><x4>}". (5) PATCH: Our method using 20 virtual tokens based on MiniGPT-v2. (6) PATCH - bboxes: Our method using on 20 virtual tokens based on MiniGPT-v2 with only the object category information. (7) PATCH - bboxes & objs: Our method using on 20 virtual tokens based on MiniGPT-v2 without any object-related information.

Among these ablation studies, (1), (2), and (3) are proposed to evaluate the impact of different formats of detection results on MiniGPT-v2's inference. (4) is proposed to compare the effects between two different fine-tuning methods: our virtual tokens and LoRA. (5), (6), and (7) are proposed to evaluate the impact of different formats of detection results on PATCH.

Reference [1] Yang, Jianwei, et al. "Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v." arXiv preprint arXiv:2310.11441 (2023). [2] Zhang Y, Ma Z, Gao X, et al. Groundhog: Grounding large language models to holistic segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024: 14227-14238. [3] Wang, Peng, et al. "Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution." arXiv preprint arXiv:2409.12191 (2024). [4] Li, Bo, et al. "Llava-onevision: Easy visual task transfer." arXiv preprint arXiv:2408.03326 (2024). [5] Rasheed, Hanoona, et al. "Glamm: Pixel grounding large multimodal model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Dear Reviewer,

We sincerely appreciate your time and effort in reviewing our paper and offering valuable suggestions. Since there will be no second stage of author-reviewer discussions, and the discussion phase is drawing to a close, we would to confirm whether our responses have effectively addressed your concerns a few days ago. And we hope they have adequately addressed your issues. If you require further clarification or have additional concerns, please don’t hesitate to contact us. We are more than willing to continue our communication with you.

Best regards.