Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance

Thank you for your insightful feedback and acknowledgement of our extensive experiments and the overall clarity and structure of our paper. We detail our response as follows.

Q1: Using the DETR trained on MSCOCO may be unfair.

MSCOCO (train) is a widely-used open-source image-caption dataset frequently utilized for pre-training various vision encoders, including CLIP, the backbone of current LVLMs (e.g., LLaVA). While all vision encoders leverage data that includes MSCOCO training split, the difference in how to process and leverage these information determines model performance. This variability is evident in MARINE’s superior performance compared to LVLMs utilizing only the CLIP encoder. Additionally, our manuscript's Table 6 illustrates instances where MARINE with only DETR underperforms compared to MARINE employing only RAM++. Nonetheless, aggregating information from multiple visual encoders (as in MARINE) consistently achieves the highest performance.

Moreover, the vocabulary derived from MSCOCO effectively encompasses frequent objects common across diverse natural-image datasets, demonstrating strong generalizability to other evaluation data. This is confirmed by the substantial improvements in POPE results on A-OKVQA shown in Table 15, highlighting our method's capability to generalize effectively and outperform baselines beyond MSCOCO.

Q1.1: Comparison with VCD.

We acknowledge that VCD is a valuable work, but our approach addresses the hallucination problem from a different and effective perspective. VCD aims to reduce LVLMs’ over-reliance on language priors from LLM pre-training data by contrasting distorted visual inputs. In contrast, MARINE focuses on hallucinations arising from insufficient visual context. Thus, VCD and MARINE approach hallucination from complementary angles, and integrating both methods has the potential to achieve further performance improvements.

Q2: Relevant works on feature steering for hallucination reduction.

Thank you for pointing to these relevant lines of research on feature steering. We summarize them as follows and will include the discussion in our next revision.

Nullu identifies a “HalluSpace” by comparing truthful and hallucinated features, then projects model weights to the null space of those hallucination-prone directions, reducing object hallucinations with no extra runtime cost. VTI (Visual and Textual Intervention) learns “shift vectors” by analyzing how vision features change under corruption and how text features differ between hallucinated and correct outputs. It then applies these shifts at inference to stabilize LVLMs and reduce hallucinations.

Q3: More in-depth explanation of how this method differs from direct prompting.

We clarify key conceptual and empirical differences between MARINE and prompting-based methods, which we believe directly address this concern:

• Direct prompting relies solely on the model’s textual instruction-following capabilities, which can exacerbate hallucination issues in less capable models (e.g., LLaVA). Conversely, MARINE directly enhances visual understanding by integrating additional visual information, making it effective even for weaker models without incurring training overhead.
• Prompting methods cannot introduce new visual data and thus remain constrained by the original vision model’s capabilities. MARINE introduces novel visual information through an improved vision encoder, fundamentally enhancing the model's observational accuracy.
• Direct prompting and additional prompt methods require careful crafting and tuning specific to each task or dataset. MARINE, however, exhibits strong generalization capabilities across diverse models and datasets, effectively reducing hallucinations and improving reliability without additional training or manual prompt optimization.

Appendix B.1.1 further elaborates on these distinctions. In particular, we evaluated the direct prompting baseline using a highly detailed instruction explicitly guiding the model to describe only observable visual characteristics

As shown below, MARINE consistently outperforms the prompting baseline by significantly reducing hallucinations while maintaining or improving recall. Although prompting can improve recall in some cases, it often worsens hallucination metrics. MARINE achieves better overall reliability across all models.

Thank you very much for your insightful review and valuable feedback. We sincerely appreciate your recognition of the simplicity and thorough experimental validation of our MARINE approach.

Q1.1 Clarify the claim regarding misalignment.

In the original claim, by "visual encoder," we referred to the LVLM's visual encoder component (e.g., CLIP), highlighting that MARINE supplements its initially insufficient visual context. The mentioned misalignment issue refers that LVLMs typically employ a trainable linear alignment layer, which can potentially lose or distort important object-level information during the mapping from visual to textual representations. In contrast, MARINE directly extracts and utilizes explicit object-level details, thereby preserving critical visual information intact. We will clarify this point explicitly in the revised manuscript.

Q1.2. Clarify the claim regarding access to advanced models.

Thank you for pointing this out. We will clarify that MARINE inherently requires access to additional vision models to enrich the visual inputs in our revision. Additionally, the "advanced LLMs" that we referenced refer to closed-source models that are accessible only via paid APIs (e.g., GPT-4o), while the vision models are open-source. We will explicitly state this distinction in our revision.

Q2. Could you explain the concept and importance of "MARINE-Truth" clearly in the main paper?

We consider using the ground-truth object list as a variant of MARINE and denote it as MARINE-Truth. The performance of MARINE-Truth serves as a reference to MARINE’s best performance. However, the ground-truth object list may also contain noise and therefore occasionally underperforms MARINE. We will explain this in our revision.

Q3. In your latency measurements, why did you exclude the latency of additional forward passes through external models required by MARINE?

Thank you for raising this point. To clarify, we did include the latency of additional forward passes through external vision models in our measurements. These external vision models have negligible inference overhead compared to autoregressive models (e.g., LLMs, LVLMs). To illustrate clearly the impact of including or excluding the latency introduced by these external models, we present a detailed latency comparison in the table below. Specifically, the table includes scenarios both excluding visual prompt generation latency (Offline MARINE) and including it (Online MARINE):

As shown, processing images via additional vision models adds only a negligible overhead to the overall latency.

Q4. Can you compare MARINE against other SOTA VLM methods, especially BRAVE?

Thank you for pointing out these related works, we will include them in discussion in our next revision. Especially, BRAVE indeed shares a very similar intuition as ours for ensembling diverse visual information sources to improve model faithfulness, confirming the motivation of our work. Here, we compare MARINE against BRAVE on the POPE benchmark. As shown below, MARINE achieves comparable performance than BRAVE, while introducing no additional trainable parameters.

Note: We report BRAVE’s POPE$_\text{avg}$ score as stated in their paper. Their model and detailed evaluation results have not been open-sourced. For fair comparison, we adopt the same test set as LLaVA-v1.5.

Q5: Labels in figures.

Thank you for catching these. We will update them in our next revision.

Thank you very much for your thoughtful and constructive review. We appreciate your recognition of MARINE’s effectiveness and comprehensive evaluation. We provide detailed responses to your questions below:

Q1. Effect of sampling temperatures.

In our paper, we opted for greedy sampling (temperature = 0) to ensure deterministic behavior in LLMs, eliminating randomness and thereby facilitating more reliable comparisons. This setup is also consistent with our main baseline VCD, and is widely used in existing benchmark evaluations to ensure reproducibility.

Here, we conducted experiments using temperature = 0.6 and reported the mean ± standard deviation in the table below. As shown, MARINE consistently improves object hallucination metrics regardless of the sampling strategy.

Q2. Questions on guidance strength. Is mPLUG-OWL2 an outlier in Figure 3?

Increasing guidance strength generally improves model faithfulness across all evaluated models, indicated by a notable decrease in the CHAIR. However, the optimal guidance strength varies by model. mPLUG-OWL2 serves as example of the more advanced models that begin with an inherently lower CHAIR score, suggesting it captures visual information better than LLaVA. Thus, it benefits from guidance strengths other than strictly 1.0, where the visual guidance effectively complements rather than dominates its internal visual encoder. In contrast, earlier models like LLaVA rely significantly more heavily on the introduced visual guidance, as it can largely dominate their intrinsic visual grounding capabilities.

However, excessively strong visual guidance can overall harm a model’s ability to follow instructions accurately. This negative effect is illustrated in Figures 9 and 13 of our manuscript, where a guidance strength of 1 reduces the quality of model generations for tasks beyond visual grounding.

For simplicity and consistency, we adopted a universal strength of 0.7 across experiments. Nonetheless, tuning this hyperparameter for each base LVLM using a validation set could yield optimized results tailored to each model. We will include this discussion in our next revision.

Q3.1. Why the intersection-based method outperforms the union-based method?

The intersection-based method retains only visual signals consistently grounded across different vision models, while the union-based method includes all signals, even conflicting or incorrect ones. Intersection outperforming union indicates that precision is currently more critical than recall for LVLMs. In other words, intersection reduces false positives from visual guidance, whereas union increases true positives at the cost of more false positives. For example, LLaVA, one of the earliest LVLMs, is particularly prone to hallucinations with complex or noisy instructions and thus benefits greatly from intersection-based methods. Newer models, such as LLaVA-v1.5 and mPLUG-Owl2, are more robust but remain sensitive to partially incorrect inputs.

Q3.2 Why use different prompts in MARINE-intersec and MARINE-union?

We evaluated MARINE-union using the exact same prompt template as MARINE-intersec, which underperformed. We hypothesize this is because the original version provides more detailed information, whereas reusing the intersec-style prompt combined reliable grounding with potentially misleading false positives, thus weakening the visual guidance.

Note: Greedy and Intersec results are reported in Table 7 of the main paper.

Q4. More recent works included related work but not baselines.

The more recent works were concurrent to our project development, and thus we included them in discussion but not empirical comparison. In the following, we additionally include

Q5. Typos

Thank you for the catch. We will ensure to correct them in our next revision.

Q6. What is the "noise intensity" for DETR?

DETR outputs a confidence score for each object. We filter predictions using a threshold. lower thresholds allow more noisier detections, while higher ones yield fewer, more precise results. This threshold defines the noise intensity.

We sincerely appreciate your thoughtful review and encouraging feedback. Thank you for recognizing the clarity and practical design of our approach, our emphasis on mitigating hallucinations, and the strong empirical support we provided. Below, we provide detailed responses to your comments:

Q1. Discuss the limitations and the scope of the metrics used in the paper.

Thank you for your suggestion and for pointing out recent developments in hallucination evaluation. CHAIR and POPE are widely adopted and reliable metrics for evaluating hallucination, and we believe they offer reliable assessments within their respective scopes. Nonetheless, we acknowledge their inherent limitations: CHAIR depends on a predefined list of object classes and synonyms, which may struggle to detect uncommon objects or very nuanced attributes. POPE’s reliability, in turn, can be influenced by the segmentation tool used to obtain ground-truth objects. We will include a brief discussion of these limitations in the revised paper and also report results using ALOHa to complement our evaluation.

Q2. Use up-to-date automatic metrics such as ALOHa.

Thank you for pointing out the newer evaluation metrics. We will include them in discussion. We have incorporated ALOHa into our evaluation to better assess localizable object hallucinations. Specifically, we report both the object-level hallucination score ($ALOHa_0$) and the caption-level aggregated score ($ALOHa$). As shown in the table below, MARINE consistently reduces object hallucinations and outperforms baseline generations across all settings and metrics.

Note: For implementation details, we use MSCOCO ground-truth captions as references and enable reference object detection for more localizable and generalizable object hallucinations detection.

Q2.1. Evaluation in addition to automatic metrics

Thanks for the suggestion. In Table 3 of our manuscript, we did include GPT-4v evaluation, which compares the outputs of two LVLM assistants using GPT-4V as a judge. This "LLM-as-a-Judge" evaluation protocol has become widely accepted as a reliable proxy for human evaluation, particularly when large-scale human assessments are costly.

Although conducting extensive human evaluations is beyond the scope of our rebuttal timeline, we fully acknowledge its importance. We consider incorporating comprehensive human evaluations as future extensions of this research.

Q3. Is using synonym list a common practice with CHAIR?

Yes, this is standard practice in the original CHAIR paper [1] and its official implementation. Specifically, they use a synonym list from Lu et al. (2018) [2] to map words (e.g., “player”) to MSCOCO object categories (e.g., “person”).

[1] Object Hallucination in Image Captioning

[2] Neural Baby Talk

Q4. Suggested related work.

Thank you for your suggestion on this line of research on faithfulness evaluation for text-to-image generations, parallel to our focus on image-to-text generations. We gave the mentioned research a careful read and summarized them below.

TIFA focuses on evaluating text-to-image generation by automatically generating and answering questions derived from prompts, measuring faithfulness along several categories (objects, actions, attributes). DSG (Davidsonian Scene Graph) also assesses text-to-image alignment but emphasizes a structured decomposition of prompts into atomic propositions. VQAScore introduces a VQA-based metric that better captures compositional semantics (such as object relationships and attributes), providing insights for attribute and relationship level object hallucination evaluation. Finally, Gecko addresses text-embedding tasks: it uses LLMs to synthesize query–passage pairs (and carefully select hard negatives) in order to train a compact yet powerful universal representation model.

CAPTURE introduces a structured metric for evaluating detailed image captions by extracting and aligning objects, attributes, and relations across captions. We will include this in related work discussion in our next revision.