Trust but Verify: Programmatic VLM Evaluation in the Wild

[Methodological contribution is weak .. DSG paper already provides a question-answering generation pipeline from the scene graphs] We respectfully disagree. Firstly, the goal of DSG is to evaluate text-to-image generation. It does so by generating templated binary (yes/no) QA pairs, of the format: Q: “Is there a <node>”? A: “Yes”, for every node in the scene graph constructed from the user prompt. The DSG score for a generated image is then computed as the fraction of the scene graph that can be externally verified (e.g. by a VLM) against the generated image.

On the other hand, PROVE seeks to evaluate image-to-text generation in the wild. As Figure 2 of our paper shows, realistically simulating in-the-wild usage requires evaluating VLM responses to free-form QA pairs that test a range of perception and reasoning abilities, rather than just binary questions. However, naively using an LLM to generate QA pairs at scale, as done in some prior works [A, B], often results in unreliable QA pairs that may be ambiguous, unanswerable, or impossible to reliably evaluate. We include more such examples from existing benchmarks in Figure 11 of the revised appendix.

Our main methodological contribution is to address this shortcoming by generating verification programs to detect and exclude invalid QA pairs. Note that programmatic verification of free-form QA pairs is far from trivial, as each QA pair requires a bespoke program (see https://prove-explorer-anon.netlify.app for examples). We achieve this by designing a rich scene-graph representation and minimal but highly expressive API to query the scene graph. We then prompt an LLM planner to generate a unique verification program using this API, which is when executed with the scene graph as input can be used to validate its corresponding QA pair. In fact, 28.1% of the original set of LLM-generated QA pairs (~8100 QA pairs!) are found to be unverifiable and filtered out by this method. This results in a high-quality and reliable evaluation dataset, as confirmed by our human study (L369) where subjects rate 95.9% of questions as relevant and answerable, and 98.2% of answers as correct.

To our knowledge, we are the first to propose such a programmatic verification strategy for validating LLM-generated visual question-answer pairs. Further, we also leverage this scene graph design to propose a novel method for jointly evaluating the helpfulness and truthfulness of free-form VLM responses at test-time, which leads to more interpretable and trustworthy evaluation than existing LLM-as-a-judge methods (see Figure 11 of the appendix).

[Comparison with existing metrics] As suggested, we compare our proposed hscore and tscore metrics to scores generated by an LLM judge (a LLaMA-3.1-Instruct (8b) model) that is provided the ground truth caption as context. We reuse the detailed prompt format introduced in MMHal-Bench [A] that includes several in-context examples, and ask the model to provide scores for both helpfulness and truthfulness. We then repeat the procedure described in L376-405 of our paper: we score responses from four models – LLaVA-1.5 (7b), LLaVA-Next (7b), GPT-4o-mini, and GPT-4o – via this method, and then measure how well these scores correlate with human judgements of the same set of responses.

Shown below are the Pearson correlations between each method (LLM-as-judge and Ours) with human judgements of helpfulness and truthfulness.

As seen, our proposed metrics correlation substantially better with human judgment compared to a pure LLM-as-judge approach. We will include this discussion in the paper, thanks!

[A] Sun, Zhiqing, et al. "Aligning large multimodal models with factually augmented rlhf." arXiv preprint arXiv:2309.14525 (2023).

[B] Liu, Fuxiao, et al. "Mitigating hallucination in large multi-modal models via robust instruction tuning." The Twelfth International Conference on Learning Representations. 2023.

[Evaluation requires dense scene graph and an executable program] This is incorrect. Our approach does not require scene graph and verification program annotations but rather generates them on-the-fly from dense image captions. As a result, our method can be readily generalized to other datasets with dense captions [A-E].

[Evaluation depends on caption quality / coverage ] We agree and have taken several measures to ensure quality and coverage:

– Quality: The DOCCI dataset, used in our benchmark, underwent a rigorous 3-stage human annotation process.

– Coverage: DOCCI captions average 136 words/caption, significantly exceeding other datasets and correlating with higher image recall.

– Postprocessing: As detailed in L228-245, we applied programmatic and text-based filtering to ensure QA pairs in PROVE are verifiable, unambiguous, and challenging.

While elements absent from the scene graph cannot be evaluated, PROVE’s scene graphs are significantly more detailed (40.9 ± 15.2 tuples per image) than those in existing benchmarks, providing critical context for evaluating free-form VLM responses.

[Is evaluation truly “in the wild”?] Our goal is to realistically simulate “in the wild” usage at scale. Existing efforts lack either scale (e.g., <100 examples in LLaVA-in-the-Wild, MMHalBench) or reliability (e.g., GAVIE, CIEM rely on unverified LLM-generated QA pairs). PROVE combines creative LLM-generated QA pairs with robust external verification, yielding a comprehensive and reliable evaluation testbed.

[Claim-based vs. graph-based evaluation] While graph-based verification is a generalization of claim-based verification, we disagree that it is inherently less flexible or generalizable. We first reiterate that our method does not require pre-annotated scene graphs or verification programs but generates them dynamically, and so is as widely applicable.

Next, we demonstrate how the added complexity of graph-based evaluation enhances reliability. To do so, we benchmark a claim-based LLM-as-judge evaluation strategy with LLaMA-3.1-Instruct (8b). We provide the model with the ground truth caption as context and prompt it to break down responses into atomic claims and score their helpfulness and truthfulness. We repeat the evaluation process for four models (LLaVA-1.5, LLaVA-Next, GPT-4o-mini, GPT-4o, L376-405) and measure Pearson correlation with human judgments:

As seen, while the observed correlation scores are still objectively far-from-perfect, they are relatively far superior to existing LLM-as-judge approaches.

[Analyzing modest tscore correlation] As recommended, we investigate the relatively low correlation observed for tscore and human judgement of truthfulness. For each QA pair, we compare human-assigned truthfulness scores to our proposed tscore (computed via image-text and text-text matching, see Eq. 2 of our paper). On average, tscore overestimates truthfulness by 11%, primarily due to the image-text matching; removing it causes tscore to underestimate truthfulness by 15%. However, removing image-text matching also reduces the Pearson correlation with human scores from 0.45 to 0.39, highlighting the complementary strengths of both components.

We sincerely hope that these clarifications persuade you to reconsider your evaluation, and would be happy to address any lingering concerns.

[A] Krause, Jonathan, et al. "A hierarchical approach for generating descriptive image paragraphs." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[B] Pont-Tuset, Jordi, et al. "Connecting vision and language with localized narratives." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020.

[C] Urbanek, Jack, et al. "A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[D] Garg, Roopal, et al. "ImageInWords: Unlocking Hyper-Detailed Image Descriptions." arXiv preprint arXiv:2405.02793 (2024).

[E] Awadalla, Anas, et al. "BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions." arXiv preprint arXiv:2411.07461 (2024)

[Concern about use of models] This is a valid concern towards which we have devoted considerable thought and effort.

Evaluating LLM-generated scene graph. The scene graphs in the DOCCI dataset are generated using Palm-2 (340B) following the DSG [A] method. A human study in DSG shows that these scene graphs achieve 92.2% precision and 100% recall on TIFA160, demonstrating high quality.

Evaluating model-based metric computation. Next, we study potential errors in text-text (via Sentence-BERT) and image-text (via OFA) matching and how those may propagate. We focus on truthfulness evaluation where we observe lower correlation with human judgment. For each QA pair, we compare human-assigned truthfulness scores to our proposed tscore (computed via image-text and text-text matching, see Eq. 2 of our paper). On average, tscore overestimates truthfulness by 11%, primarily due to OFA; removing image-text matching causes tscore to underestimate truthfulness by 15%. However, removing image-text matching also reduces the Pearson correlation with human scores from 0.45 to 0.39, highlighting the complementary strengths of both components.

To demonstrate robustness, we visualize the evaluation pipeline for various models and examples here: https://prove-explorer-anon.netlify.app.

Further, we note that our reliance on external models is limited to well-defined subtasks (e.g., tuple extraction, text matching) for which these models are specialized and continually improving. Unlike LLM-as-judge scoring, our method provides an interpretable and auditable trace of how responses are scored, which mitigates errors and enhances transparency.

In summary, while external models can introduce errors, we believe that these are largely mitigated by our robust approach. The near-perfect human ratings for relevance and correctness of QA pairs in our benchmark (L369-374) further validate this.

[Human study clarification] Human ratings for hscore and tscore are discrete (0/1 and 0/0.5/1.0, respectively, as noted in L377). We discovered a bug in our correlation computation after fixing which observe Pearson correlation scores of 0.81 (hscore) and 0.45 (tscore).

To contextualize these scores, we benchmarked an LLM-as-judge baseline (LLaMA-3.1-Instruct, 8b), following the MMHal-Bench protocol. Using the same prompts and procedures (L376-405), we evaluated responses from four models (LLaVA-1.5, LLaVA-Next, GPT-4o-mini, GPT-4o) for helpfulness and truthfulness, correlating these scores with human judgments:

As seen, our proposed metrics correlate substantially better with human judgment compared to a pure LLM-as-judge approach.

We sincerely hope that these clarifications persuade you to reconsider your evaluation, and would be happy to address any lingering concerns.

["this paper presents a way to evaluate image-captioning"] We believe that there has been a misunderstanding: PROVE is a benchmark to evaluate vision-language models (VLMs) in the wild, rather than image captioning models alone. As reviewer ubc1 succinctly summarizes:

This paper proposes a new benchmark which the authors constructed by turning detailed captions into scene graph representations, and generating QA pairs as well as the corresponding verification programs based on the scene graphs. It also proposes a programmatic evaluation of the VLMs’ responses on both helpfulness and truthfulness by comparing the predicted scene graphs and ground-truth scene graphs.

Concretely, we use human-generated image captions to construct a ''gold scene graph'' which serves two complementary purposes:

1. Dataset verification. For each image, we programmatically verify each question-answer pair generated by an LLM (GPT-4o) from its gold scene graph; specifically, that the question is even answerable for the image, and that the generated answer is accurate.

2. VLM Evaluation. At test time, we evaluate both the helpfulness and truthfulness of each free-form VLM response by comparing the response’s scene-graph representation with the gold scene graph. Such scoring provides a more reliable and interpretable scoring rationale than a pure ''LLM as judge'' approach.

We will rephrase the text to improve clarity, and welcome suggestions towards the same.

["Why does the performance across models in Table 1 look similar? .. GPT-4o should be better than other models"] To clarify, performance across methods is not identical: across surveyed methods, the largest performance gap is 6.5% (absolute) for hscore and 5.75% (absolute) for tscore. However, due to the fact that the two metrics tend to be negatively correlated (see Figure 4), the maximum average performance gap is relatively small (3.3% absolute).

Further, GPT-4o does indeed score highest on the (comprehension and reasoning focused) hscore measure, matching expectations. However, we find that it also tends to hallucinate more than some other methods, resulting in a relatively low tscore. It is precisely this ability to jointly evaluate helpfulness and truthfulness, which existing benchmarks lack, that motivates our work.

However, we agree that our continuous cosine similarity based metric is not conducive to fine-grained discrimination between methods. To address this, we propose a discrete version based on thresholding, defined as:

$$\text{hscore}^{\theta} (\hat{A})=\frac{\sum_{t \in g(A)-g(Q)} \mathbb{I}\big[\max_{t' \in g(\hat{A})} \text{sim}(t, t') > \theta\big]}{|g(A)-g(Q)|}$$ $$\text{tscore}^{\theta} (\hat{A}) = \frac{\sum_{t' \in g(\hat{A})} \mathbb{I}\big[\text{max}\big(\max_{t \in g(C)}(\text{sim}(t', t), p(I \models t')\big) > \theta\big]}{|g(\hat{A})|}$$

Recomputing (a subset of) methods in Table 1 with these updated measures gives us:

As seen, thresholding leads to a lower hscore and higher tscore across the board, more than doubling the maximum gap in average performance between methods to 7.5% (absolute), without changing the overall ranking -- GPT-4o still scores highest on helpfulness and LLaVA-1.5 (7b) on truthfulness We will be happy to revise our paper with these fine-grained results.

Fig. 5 correction. Thanks for bringing this to our notice. LLaVA-1.5 (7b)’s actual response to Q2 for the first image is ''The bricks at the bottom of the wall are white'', which results in the low score -- we will revise the paper to fix this. For Q1, GPT-4o’s answer ''BISTER'' is more similar to the ground truth answer ''BUSIER'' in Sentence BERT embedding space than LLaVA-1.5-7b’s (''MISTHI''), which is why it obtains a higher hscore.