GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity

We appreciate the reviewer for taking the time to provide detailed and constructive feedback. We are grateful for the recognition of our motivation, the performance of our method, and the training-free design.

We also thank the reviewer for the insightful questions and are happy to clarify the points raised as follows:

A1. Attribute and relational hallucinations

We appreciate you raising this important point. We agree that attribute-level and relational hallucinations are critical challenges for real-world applications. As mentioned in our Limitations and Future Work section, our current study focuses on object existence hallucinations because publicly available token-level annotations and benchmarks for attribute and relational hallucinations in open-ended generation settings remain limited.

That said, we explored whether GLSIM can handle these more complex hallucination types. To this end, we generated captions with each LVLM for 500 randomly selected images from the MSCOCO validation set. We then used a powerful multi-modal model GPT-4o, to produce pseudo ground-truth annotations for two new categories:

• Attribute Hallucination: Incorrect properties of objects are described (e.g., “a red car” when the car is blue).
• Relational Hallucination: The relationship between objects is incorrectly stated (e.g., “a cat sitting on a table” when it is under the table).

We then computed token-level GLSIM scores and aggregated them by averaging across attribute–object (e.g., “a red car”) and object–relation (e.g., “a cat sitting on a table”) spans. These aggregated scores served as unsupervised estimates of the hallucination likelihood. Without any task-specific modification, GLSIM consistently outperformed baseline methods. We hope GLSIM can serve as a promising baseline for detecting more complex forms of hallucination.

• Attribute Hallucination

• Relational Hallucination

A2. Methodological advancement

We understand how your concern arises. While the idea of combining global and local signals may seem intuitive in hindsight, what distinguishes GLSIM is not only the formulation of these two signals but the methodological rigor with which we analyze and instantiate them. Below, we elaborate on the methodological significance of our work:

1. First, our motivation is original and offers a new lens through which to understand hallucination in LVLMs. Through detailed case studies in Section 4.1, we demonstrate, for the first time, that neither global nor local similarity alone is sufficient for reliable hallucination detection. This insight itself is a valuable contribution, as it challenges the prevailing reliance on single-perspective signals and motivates a more structured diagnostic approach. The impact of our work, therefore, goes beyond a specific method—it points the field toward a promising direction for hallucination detection in multimodal models and opens up room for more future work.

2. Second, what sets our method apart is the rigorous investigation of how each component, global and local similarity, should be designed and integrated optimally. We do not treat the combination as a black-box fusion; instead, we design, ablate, and validate every aspect of the formulation---this is both a new and non-trivial effort. For example, in Section 5.3 (Lines 293–315), we carefully explore the design of both local and global scores by evaluating different unsupervised grounding methods, token positions, and aggregation strategies. Lines 316–330 examine how to best integrate the global and local scores. We further ablate distance metrics (Appendix D.5) and analyze all layer pair combinations to identify meaningful patterns for the scoring function (Appendix E). This is also recognized by other reviewers:

   "The paper includes in-depth ablation studies that analyze the impact of global and local similarity and different design choices for them". Reviewer 5sQ6

   "The authors conduct thorough ablation studies on key design choices, including object grounding strategies and scoring variants, offering clear empirical justification for their final architecture (Section 5.3)." Reviewer S1xd

3. Lastly, we validate our design through extensive quantitative (Section 5.3, Appendix D, and Appendix E) and qualitative (Lines 147-167, Lines 299-302, and Appendix B) ablations, showing that the proposed global and local metrics capture interpretable signals critical for robust object hallucination detection. These results reinforce the value of our approach, and we hope they address the reviewer’s concerns regarding methodological significance.

A3. Evaluation with advanced models

Thank you for pointing this out. In response, we have conducted additional experiments using two recent and advanced LVLMs: Qwen2.5-VL-7B [1] and InternVL3-8B [2]. For additional evaluations on models such as InstructBLIP, Cambrian-1, and LLaVA-NeXT, please refer to Appendix D.1.

These results demonstrate that GLSIM consistently outperforms all baseline methods, further validating its effectiveness and generalizability on state-of-the-art models. Specifically, GLSIM improves over the previous top-performing method, SVAR, by +5.36% on Qwen2.5-VL-7B and +4.41% on InternVL3-8B in AUROC.

We will include these results in detail in the revised version of the paper.

A4. Comparison with external model-based methods

We appreciate the reviewer’s request for a more detailed comparison.

As reviewers 5sQ6 and mtgR noted, our method offers practical advantages over approaches that rely on external knowledge sources, such as LLMs and LVLMs. These methods typically rely on prompting an external LLM or LVLM with a triplet input consisting of the image, instruction, and the generated caption. This approach is inherently restricted to post-generation evaluation, as hallucinations can only be assessed after the entire caption has been produced by the base model. In contrast, GLSIM operates during the token decoding phase, enabling real-time hallucination detection at the token level. This makes GLSIM particularly well-suited for interactive or streaming applications, where immediate feedback is crucial. Moreover, external LVLM-based approaches often require multiple forward passes—one to generate the caption from the base model under evaluation, and at least one or more to evaluate it using external LVLMs. GLSIM requires only a single forward pass, with no additional queries or models needed, making it much more efficient at inference time.

Regarding reliability, external LVLMs may themselves be prone to hallucination. Our method avoids this dependency by operating entirely within the base LVLM, without introducing additional uncertainty from external evaluators. Below, we provide a quantitative comparison on the MSCOCO dataset using LLaVA-1.5-7B as the base model. The results show that GLSIM is significantly more efficient, requiring only 1.6 seconds per image for inference, compared to 9.3 seconds for the External-based method, which uses the more powerful LLaVA-1.5-13B model. This results in an efficiency gain of +82.8%. Despite this significant inference efficiency advantage, GLSIM still performs competitively on reliability metrics. Compared to External-based, GLSIM shows better performance across almost all evaluated metrics.

These findings support our claim that GLSIM is practical and reliable for real-world use. It achieves this without relying on any external models in a fully unsupervised, training-free manner, using only internal signals from the base model itself.

We will update the main paper to make this comparison clearer.

[1] Bai et al., "Qwen2.5-VL Technical Report," arxiv:2502.13923

[2] Zhu et al., "InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models," arxiv:2504.10479

This is an insightful experiment, I'm happy to raise my score from 2 to 4.

We sincerely appreciate the reviewer’s thoughtful and constructive feedback. We are encouraged by the recognition of our motivation, performance improvements, and comprehensive ablation studies.

We are also grateful for the insightful questions and are happy to clarify the points raised as follows:

A1. How are target objects selected for evaluation?

Thank you for raising this important point. As noted in Lines 226–227 of the main paper, we follow a verified and widely adopted evaluation protocol from prior works [1], where object mentions are matched against the ground-truth annotations provided in the MSCOCO and Objects365 datasets. This ensures consistency and comparability across different models and detection methods.

Specifically, we extract objects from the generated captions and compare them with ground-truth object classes using exact string matching with synonym expansion. This standardized evaluation is applied uniformly across all tested LVLMs and baselines in our experiments.

While our main results are based on a benchmark-grounded setup to ensure rigorous and fair evaluation, we note that GLSIM can be naturally extended to real-world scenarios with unrestricted vocabulary settings. In such cases, lightweight object extraction tools such as noun chunking or external taggers like spaCy [2] can be used to identify candidate noun or object phrases from generated text in real time. These phrases can then be processed by GLSIM’s mechanism, allowing for consistent and automated hallucination detection even without a fixed set of predefined object categories.

We will revise the main paper to provide a more detailed clarification.

A2. Performance on advanced models

Great point! To address this, we conducted additional experiments on two recent and advanced LVLMs: Qwen2.5-VL-7B [3] and InternVL3-8B [4]. For further evaluations on other strong models, including Cambrian-1, LLaVA-NeXT, and InstructBLIP, please refer to Appendix D.1.

We evaluate object hallucination detection performance on 2,000 randomly sampled images from the MSCOCO validation set and report the average AUROC (%) over three different random seeds. These results demonstrate that GLSIM consistently outperforms all baseline methods, further validating its effectiveness and generalizability on state-of-the-art models. Specifically, GLSIM surpasses the previous top-performing method, SVAR, by +5.36% on Qwen2.5-VL-7B and +4.41% on InternVL3-8B.

We will include these results in detail in the revised version of the paper.

A3. Clarification for Line 286

Thank you for your careful reading and for pointing out this typo and ambiguity. In Line 286, we intended to report the minimum and maximum AUROC improvements observed when comparing our local method ($s_{\text{local}}$) to prior local baselines (Internal Confidence, Contextual Lens) to highlight the effectiveness of our proposed object grounding approach for object hallucination detection.

Specifically:

• The minimum gain of +1.1% is observed when comparing $s_{\text{local}}$ (Top-1) to Contextual Lens on the LLaVA model.
• The maximum gain of +7.7% is observed when comparing $s_{\text{local}}$ (Top-K) to Internal Confidence on the Shikra model.

We acknowledge this was a typo and will revise the main paper to clarify the specific baselines used for comparison and improve overall clarity.

Thank you again for catching this detail. We appreciate your close attention.

[1] Rohrbach et al., "Object Hallucination in Image Captioning," EMNLP 2018

[2] Honnibal et al., "spaCy: Industrial-strength Natural Language Processing in Python"

[3] Bai et al., "Qwen2.5-VL Technical Report," arxiv:2502.13923

[4] Zhu et al., "InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models," arxiv:2504.10479

We appreciate the reviewer for the thoughtful comments and valuable feedback. We are encouraged by the recognition of the practicality and generalizability of our approach, as well as the comprehensiveness of our experiments.

Below, we address your key concerns.

A1. Detection threshold and the choice of layers

Thank you for this thoughtful observation!

In our evaluation, we use a threshold-independent metric, AUROC (Area Under the Receiver Operating Characteristic curve), which is standard in binary classification for comparing model ranking performance over all possible decision thresholds.

However, for deployment purposes, as you rightly noted, we do need to choose a specific detection threshold $\tau$ to convert continuous hallucination likelihood scores into binary decisions (hallucination vs. non-hallucination). In practice, this threshold can be set using a small labeled validation set (e.g., 100 samples), depending on downstream priorities (such as higher precision vs. higher recall). A common and effective strategy is to select the threshold that maximizes the F1 score on the validation set, which we also use for the case study.

Compared to fixed rule-based heuristics, our continuous scoring approach provides greater flexibility, making it well-suited for real-world scenarios where hallucination severity and uncertainty may vary across applications.

For the choice of layers $(l,l')$, as you helpfully pointed out, we refer to our analysis in Appendix E. Our layer-wise performance analysis shows that selecting middle-to-late layers (typically after 70% depth) for both image and text embeddings yields consistently strong and stable performance across different LVLMs. These layers tend to encode richer semantic information, and we observe low variance across models, which minimizes sensitivity and reduces the practical burden of hyperparameter tuning.

A2. How to mitigate borderline errors?

That’s an excellent point raised. We acknowledge that borderline errors are a valid concern in real-world deployment.

To mitigate such borderline misclassifications, a promising direction is to train a lightweight meta-classifier using pseudo-labels derived from GLSIM itself. Specifically, we can select a threshold on a held-out validation set that maximizes F1 score, which allows us to generate high-quality pseudo-labels despite lacking ground-truth object annotations. These labels then supervise a linear classifier whose input features include the GLSIM score as well as auxiliary uncertainty metrics like negative log-likelihood and entropy. This meta-classifier can learn an adaptive decision boundary, replacing the inflexible numeric threshold and allowing the system to output calibrated probabilities.

This strategy has proven useful in safety-critical applications where conservative behavior is preferred over false positives or negatives. In our context, such an adaptive scoring mechanism allows GLSIM to retain high precision and recall for confident cases, while flagging ambiguous ones for downstream handling (e.g., human-in-the-loop review or secondary verification). By supporting low-confidence abstention, the meta-classifier improves robustness near the threshold and makes GLSIM more practical for real-world deployment. We will include this discussion in the revised version to clarify how GLSIM can be made more resilient to borderline uncertainty -- thanks again for your insightful comment!

We appreciate the reviewer’s comments and suggestions. We are encouraged by the recognition of our method as practical, along with the acknowledgment of our comprehensive experiments and thorough ablation studies.

We also thank the reviewer for the interesting and insightful questions, which we are happy to address below:

A1. How can we extend GLSIM to detect other types of hallucinations?

Great point, and thank you for inviting our thoughts! We agree that attribute-level and relational hallucinations are important challenges for real-world applications. As noted in our Limitations and Future Work section, our current study focuses on object existence hallucinations because publicly available token-level annotations and benchmarks for attribute and relational hallucinations in open-ended generation settings remain limited.

That said, we believe GLSIM can be naturally extended to address these more complex types of hallucination, which are typically defined as follows:

• Attribute Hallucination: Incorrect properties of objects are described (e.g., “a red car” when the car is blue).
• Relational Hallucination: The relationship between objects is incorrectly stated (e.g., “a cat sitting on a table” when it is under the table).

To this end, we explored whether the current GLSIM score, without any task-specific modification, can capture attribute and relational hallucinations. We generated captions with each LVLM for 500 randomly selected images from the MSCOCO validation set. We then used a powerful multi-modal model, GPT-4o, to produce pseudo ground-truth annotations for both attribute and relational hallucinations. We then computed token-level GLSIM scores and aggregated them by averaging across attribute–object (e.g., “a red car”) and object–relation (e.g., “a cat sitting on a table”) spans. These aggregated scores served as unsupervised estimates of hallucination likelihood. While this is a simple implementation, GLSIM demonstrated meaningful detection capabilities without requiring task-specific modifications, and consistently outperformed baseline approaches.

Beyond methodology, one of the key challenges in this direction is the lack of comprehensive benchmarks for attribute and relational hallucinations at the token or span level. Given the open-ended nature of caption generation, constructing such annotations is non-trivial, especially for nuanced or fine-grained visual concepts. Developing such benchmarks would be a crucial step toward enabling fair and systematic evaluation in this area.

We view these as promising future directions and believe GLSIM can serve as a promising baseline for exploring more complex forms of hallucination detection.

• Attribute Hallucination

• Relational Hallucination

A2. Can GLSIM handle high-resolution images?

Good point! Yes, GLSIM is compatible with high-resolution inputs. We acknowledge that processing high-resolution inputs is critical for reliable real-world deployment. GLSIM remains effective in such scenarios because its local similarity component operates on patch-level visual tokens generated by the vision encoder.

In Section 5.2, we show that GLSIM outperforms other methods on the Objects365 dataset, which contains extremely high-resolution images—up to 8412×5608, significantly larger than the 640×480 resolution typical of MSCOCO.

To further demonstrate GLSIM’s robustness on high-resolution inputs, we conducted additional experiments on 500 randomly sampled images from the top 20% highest-resolution images in the Objects365 dataset. We evaluated our method using two advanced LVLMs: LLaVA-1.5-7B and LLaVA-NeXT, as well as Qwen2.5-VL-7B [1], a state‑of‑the‑art model that processes images at their native resolution, dynamically converting higher‑resolution inputs into thousands of visual tokens. As input resolution increases (e.g., from 224×224 to 1024×1024), Qwen2.5-VL adjusts its number of visual tokens accordingly, providing finer spatial granularity and enabling more precise object localization for GLSIM's local similarity component.

These results demonstrate that GLSIM consistently outperforms all baseline methods, confirming its effectiveness on high-resolution images.

[1] Bai et al., "Qwen2.5-VL Technical Report," arxiv:2502.13923