Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models

Thanks for your thoughtful feedback and insightful suggestions. We have carefully considered each of your suggestions and addressed them as follows：

Q1 About test-retest reliability

The overall idea is the testify to the consistency of evaluation results over different random seeds, which, however, involves two factors, including the randomness of the models and the metrics. It is unreasonable to attribute it directly to the metrics.

For Yes-or-No and caption benchmarks, the randomness originates from the model itself. For free-form benchmarks (GAVIE and MMHal) , randomness arises from both the model and the GPT scoring. We believe that the introduction of GPT randomness leads to the relatively lower test-retest reliability. To address this, we improve the GPT evaluation process by modifying the instable scoring tasks to more reliable judgment tasks, reducing the impact of GPT randomness on evaluation results.

To validate our analysis of randomness, we conduct repeated tests on the same model responses for MMHal and GAVIE, isolating GPT randomness, and calculate the correlation between the results. The correlation coefficients are 0.89 for MMHal and 0.88 for GAVIE, confirming that GPT scoring is the primary source of their inconsistency results in repeated tests.

Q2 About parallel-form reliability

During the construction of HQH, to improve validity, the authors use free-form VQA, and to improve test-retest reliability, the authors prompt GPT to only conduct binary classification.I wonder how HQH improves its parallel-form reliability, even compared with the free-form counterparts like MMHal and GAVIE.

We analyze that the relatively lower parallel-form reliability of MMHal and GAVIE arises from the same issue as their test-retest reliability: the inconsistency in GPT scoring for similar responses. Assigning accurate hallucination scores to model responses is a challenging task for current GPT, making it more susceptible to randomness and producing inconsistent scores.

To address this, we refine the GPT evaluation process by replacing the scoring process with simpler and more stable binary classification. This adjustment not only enhances test-retest reliability but also improves parallel-form reliability, ensuring greater consistency in evaluations across parallel tests.

Q3 About the evaluation metric of HQH

In Figure 4, we see that the authors provide a lot of textual information about the image for GPT. Do you still give the original image to GPT? If so, why did you choose to provide this textual information explicitly to GPT judgment?

We do not provide GPT with the original image, supplying only textual information. Similar to other LVLMs, GPT is also prone to hallucination issues, potentially generating text responses inconsistent with the visual input.

By providing detailed textual information, we aim to prevent the influence of visual input on hallucination evaluation, allowing GPT to focus solely on extracting key information from the comprehensive textual descriptions to assess whether the model responses align with the annotations. Such design not only reduces the risk of hallucinations caused by GPT’s visual perception but also enhances the accuracy and consistency of hallucination evaluations, making the results more reliable and credible.

Thanks for your thoughtful feedback and insightful suggestions. We have carefully considered each of your suggestions and addressed them as follows：

Q1 About the novelty

My main concern is the limited novelty. As the authors introduced in Sec.3, introducing psychometrics into AI is not a new idea, and the proposed metrics (Test-retest Reliability, Parallel-forms Reliability, Criterion Validity) are also proposed by previous works. So the evaluation part seems to evaluate the current benchmarks with an off-the-shelf idea, with limited novelty.

Introducing psychometrics into AI is an emerging trend, with only a few works [1, 2, 3] exploring this integration, each focusing on different aspects compared to our work. Specifically, [1] and [2] evaluate the psychological traits of LLMs using psychometric inventories and datasets, while [3] introduces the concept of construct-oriented evaluation from psychometrics into AI evaluation.

Our work uniquely combines psychometrics with hallucination benchmarks for LVLMs, to our knowledge, proposing the first quality measurement framework for AI benchmarks. We adapt and redefine psychometric metrics for AI benchmarks, forming a clearly defined and practical framework.

Q2 About the general applicability and relevance to hallucination

Following 1, the evaluation is a general idea that is unrelated to the Hallucination problem. We could apply it to other multi-modal benchmarks and have some similar conclusions. So I think it hardly provides some insight about the following study about hallucination.

As discussed in the Section 5, the principles underlying our benchmark quality measurement are general and can be extended to other multi-modal benchmarks, which we consider an advantage. Our focus on hallucination benchmarks stems from the fact that hallucination is currently a significant and critical issue for LVLMs. Additionally, hallucination benchmarks are highly diverse, encompassing various tasks and metrics, and their evaluation results are often inconsistent, making it difficult to determine which are more credible.

Through our quality measurement framework, we identify several specific issues with existing hallucination benchmarks. For example, in free-form benchmarks, the GPT-based hallucination scores lack sufficient accuracy and stability, while in caption benchmarks, the metrics are highly influenced by response length. We hope that our analysis can provide insights for improving and designing future hallucination benchmarks.

Q3 About the difference with common capability evaluation benchmark

Similarly, the proposed HQHBench also seems like a common MLLM capability evaluation benchmark (for example, a free-form version of MMBench or SeedBench), rather than a hallucination evaluation benchmark. If we treat all the wrong answers as hallucinations, we should not view hallucination as an independent research topic.

Hallucination evaluation and common capability evaluation benchmarks may share some overlapping perception tasks but their evaluation focus and design principles differ fundamentally. Hallucination evaluation emphasizes depth, employing fine-grained perception tasks to further analyze specific deficiencies across different hallucination types (e.g., attribution, existence) . In contrast, common capability evaluation benchmarks (e.g., MMBench, SeedBench) prioritize breadth, comprehensively assessing a model’s performance across perception and cognition tasks including logical reasoning, and numerical computation. While there is some overlap in perception tasks, they are not identical. Our HQH focuses exclusively on tasks that reflect cross-modal consistency, excluding others such as facial recognition or image segmentation, which fall outside the scope of hallucination evaluation.

The core of hallucination evaluation lies in determining whether a model's textual output aligns with its visual input, rather than merely judging the correctness of answers. HQH employs a free-form design to avoid the influence of potential response bias on evaluation results. For example, in closed-ended tasks, incorrect answers may result not only from hallucination but also from response bias, such as a tendency to answer "yes" or favor certain options. HQH evaluation focuses specifically on hallucination issues, providing targeted feedback for improvement, while common capability evaluation holistically assess a model's overall abilities across a wide range of tasks, which are complementary, addressing different evaluation goals.

Q4 About the capacity of GPT-3.5

Last, I have a concern about the evaluation quality of HQHBench, the GPT-3.5 seems not capable enough to understand the complex image with only a short caption and phrases with bbox.

In our criterion validity measurement, we examine the correlation between HQH evaluation using GPT-3.5 and human evaluation. The results show a correlation of 0.94, indicating that the evaluation quality of GPT-3.5 is close to human-level performance.

Q5 About the evaluation of new models

Besides, the MLLMs evaluated in the paper are quite out-of-date. There are many capable new models, such as LLaVA-Next/OneVision, Qwen2-VL, InternVL2, NVLM, etc.

Thank you for your suggestion. We have updated our evaluation on HQH to include several latest models. The results are presented in Table 1, showing that some of these new models perform exceptionally well, particularly GLM-4V and Qwen2-VL, with GLM-4V even surpassing GPT-4o. These results will be included in the revised manuscript.

Table1 Evaluation results of new models on HQH

Reference

[1] Pellert et al. AI psychometrics: Using psychometric inventories to obtain psychological profiles of large language models. OSF preprint 2023.

[2] Li et al. Quantifying AI psychology: A psychometrics benchmark for large language models. arXiv preprint 2024.

[3] Wang et al. Evaluating general-purpose AI with psychometrics. arXiv preprint 2023.

We have updated our evaluation results with InternVL2-8b as follows:

Table1 Evaluation results of new models on HQH

We will continue to update our evaluation results along with the development of LVLMs.

Thanks for your thoughtful feedback and insightful suggestions. We have carefully considered each of your suggestions and addressed them as follows：

Q1 About the discussion on benchmark improvement and integration

Essentially, the effort of curating a new benchmark is orthogonal to improving existing benchmarks. Even more I'd say they are additive to some extend, as this is a more efficient way to merge community efforts for a more comprehensive benchmark. I'd like to see authors' discussion on this point.

Thank you for your insightful suggestions. The two points you raised are indeed valid and worthy of further exploration. Our core contribution lies in introducing the concept of quality measurement. We believe that both existing and future benchmarks can benefit from improvements based on our HQM framework, which serves as one of the primary goals of our work. By incorporating the perspective of quality measurement, we aim to provide new insights and references for the enhancement and design of hallucination benchmarks.

Regarding your point about integrating improved benchmarks to form a more comprehensive and unified benchmark, we fully agree with its value. This approach not only facilitates the efficient consolidation of community efforts but also contributes to establishing a more valid and reliable benchmarking system. We plan to explore this direction further in the future and look forward to collaborating with the community to achieve this goal.

Q2 About the potential missing-annotation scenarios

Another issue is that the HQHBench is curated with the bless of Visual Genome annotation, therefore hallucination judgement can be done in a binary format against image information provided in the annotation. However, there are still image information not covered by the annotation. Such a missing-annotation scenario may cause a rich response capturing not-in-annotation fact be judged as hallucination. How can we address such scenarios?

While potential missing-annotation scenarios may exist, their impact on overall evaluation results is minimal. On one hand, our questions are designed to focus on specific aspects, such as the attributes of a particular object, rather than broad image captioning tasks, which constrains the model's responses, reducing the likelihood of overly divergent answers. On the other hand, the image information we utilize is highly comprehensive, covering nearly all major objects within the images.

Additionally, in our validity measurement, we compare the hallucination evaluation conducted by human evaluators based on visual image with those by GPT using textual annotations. The results show a strong correlation, indicating that potential missing-annotation scenarios are minimal and have limited impact on the overall evaluation results.

Q3 About the details of test-retest protocol

The test-retest protocol is a bit unclear. When we re-run the benchmark, do we require the model being evaluated to re-generate the responses? If not, why is there difference for Yes-or-No benchmarks? If yes, how do we distinguish whether the randomness comes from the benchmarking judge, or the models being evaluated?

Under the test-retest protocol, the evaluated model is required to regenerate its responses. For Yes-or-No and caption Benchmarks, the randomness originates from the models themselves. For free-form Benchmarks (GAVIE and MMHal) , randomness arises from both the models and GPT scoring. We believe that the introduction of GPT randomness leads to the relatively lower test-retest reliability. To address this, we improve the GPT evaluation process by modifying the instable scoring tasks to more reliable judgment tasks, reducing the impact of GPT randomness on evaluation results.

To validate our analysis of randomness, we conduct repeated tests on the same model responses for MMHal and GAVIE, isolating GPT randomness, and calculate the correlation between the results. The correlation coefficients are 0.89 for MMHal and 0.88 for GAVIE, confirming that the randomness of GPT scoring is the primary source of their inconsistency results in repeated tests.

Thanks for your thoughtful feedback and insightful suggestions. We have carefully considered each of your suggestions and addressed them as follows：

Q1 About the motivation

The motivation and method are inconsistent. The major motivation of this paper lies in that the evaluation metrics of existing hallucination benchmarks are insufficient. But the proposed solution is to curating a new high-quality benchmark and calculate “hallucination rate” by altering the GPT-scores and GPT-based binary choices. The proposed multi-fold validation criterions are not employed for HQH evaluation.

Our primary motivation stems from the observation that existing hallucination benchmarks yield inconsistent evaluation results, making it challenging to determine which results are more trustworthy. To address this issue, we introduce a quality measurement framework aimed at quantifying the reliability and validity of hallucination benchmarks. Based on our analysis of the quality measurement results of existing benchmarks, we improve the metric for open-ended VQA and propose our HQH. We further validate that, under the quality measurement framework, i.e. the mentioned "multi-fold validation criterions," HQH provides more reliable and valid hallucination evaluation.

Q2 About the potential data leakage

I'm wondering whether it is reasonable to use the validation set of Visual Genome dataset. It can incurs data leakage problem as existing LVLMs are using validation sets of open-source datasets for their training.

Thanks for raising this concern. To assess the potential risk of data leakage, we apply the Multimodal Leakage (ML) metric [1], which is designed to quantify the extent of data leakage in multimodal benchmarks. Specifically, ML calculates the difference in scores between an LVLM without visual inputs and its LLM base (without multimodal training) under the given benchmark. Higher ML value indicates more potential data leakage, as it suggests that the model performance without visual input surpasses that of its unimodal base, likely due to exposure to evaluation samples during multimodal training. Conversely, an ML value close to 0 indicates no data leakage.

We calculate ML for the top-performing models on our HQH benchmark, as shown in Table 1. For comparison, we include ML of other benchmarks as reported in [1]. The results show that HQH achieves the lowest average ML across models compared to other benchmarks, demonstrating that HQH has minimal data leakage.

Additionally, our HQH is relatively challenging, as most models perform poorly, which further supports that our benchmark effectively differentiates models and avoids inflating performance due to potential data leakage.

Table1 Multimodal Leakage↓ (%) of LVLMs on HQH and other benchmarks

*For closed-source LVLMs, we compare the results with those of GPT-4V and Gemini-Vision-Pro as reported in [1].

Q3 About the difference of GPT scoring and judgement

What are the key differences between asking a LLM to directly give a score and judging whether the answer includes hallucination? I cannot figure out their intrinsic differences here.

The key difference lies in the validity and reliability of the two metrics. Direct scoring requires LLMs to assign specific hallucination scores to LVLM responses, which current models often struggle to do accurately and consistently, due to their limited capability for precise quantification. In contrast, judging whether an LVLM response includes hallucination is a binary task, which is comparatively simpler and aligns better with the abilities of current LLMs.

In our criterion validity measurement, we calculate the correlations between human evaluation and both GPT scoring and GPT judgment separately. The results show that GPT judgment achieves a higher correlation with human evaluation, indicating that it provides a more accurate and reliable results.

Q4 About the contribution

The overall technical contribution is limited. The primary contributions of this paper include that existing evaluation metrics of hallucination benchmarks are instable under multi-fold validation, and curates a high-quality hallucination benchmark with “hallucination rate” as the evaluation metric.

Our core contribution lies in introducing a hallucination benchmark quality measurement framework and proposing an improved benchmark. To the best of our knowledge, our HQM is the first framework aimed at measuring the quality of AI benchmarks, which is an innovative integration of AI and psychometrics. We belief this framework will help to identify potential reliability and validity issues in existing benchmarks, providing valuable insights for current benchmark improvement and future benchmark construction within the community.

Reference

[1] Chen et al. Are we on the right way for evaluating large vision-language models? NeurIPS 2024.

Dear Reviewer 6UGx,

We sincerely appreciate your constructive suggestions and have carefully addressed each of them. As the discussion phase is coming to an end, we look forward to your feedback on our response and welcome any further discussion on points that may remain unclear regarding both our paper and our responses.

Thanks for your continued support of our work.

Best regards!

The authors

Thanks for your follow-up feedback!

Regarding the "quality measurement framework", it is inspired by psychometrics and generally designed to evaluate the quality of hallucination benchmarks, incorporating indicators for both reliability and validity. Using this framework, we have identified and analyzed some quality issues of existing hallucination benchmarks. Through our analysis, we develop HQH, which is verified to provide more stable and accurate evaluation results under this framework.

Regarding Q3, we agree with your point that scoring can distinguish the degree of hallucination. However, the current issue is that LLMs, including GPT, still have limitations in achieving precise and consistent scoring, as evidenced by our experiments. We believe that for a metric to be effective, it is more crucial to ensure the accuracy and consistency of the evaluation results compared to granularity. Therefore, binary classification is a more practical choice at present. In the future work, we will continue to explore new metrics that reflect the degree of hallucination with finer granularity.

Please let us know if you have any remaining concerns, or if you would consider updating your evaluation based on our response. We sincerely appreciate your support of our work!