VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR

We thank the reviewer for thoughtful comments and detailed technical questions. Below, we respond to each point.

Weakness 1 & Question 1: Suggested additional baselines
Due to rebuttal time constraints, we were unable to conduct a full-scale human preference study. Additionally, such studies are often subject to annotator disagreement and noise, and more importantly, they contradict our core motivation: building an annotation-free, scalable video-caption evaluation framework that avoids the cost and subjectivity of manual labeling.

Per reviewer suggestion, we experimented with GPT-4o (via OpenAI API) as a meta-evaluator on 150 examples from LongVideoBench, used in Fig. 3. For each video, GPT-4o was asked to rank five generated captions in terms of grounding and utility (the relevance to video content and questions).

However, the results revealed significant issues:

• GPT-4o indicated that 32% of the video clips contained captions completely irrelevant to the video content, making grounding-based ranking unreliable. This issue happens even if the model is not asked to acknowledge the "unrankability".
• Similarly, GPT-4o flagged 92% of captions as irrelevant to the associated task questions, making utility-based ranking equally unstable.

This suggests that even strong VLMs like GPT-4o fail to reliably evaluate captions on LongVideoBench—likely due to known challenges in long-video understanding. Although humans can often infer correct answers from these captions, current VLMs cannot. Due to this inconsistency, we decided not to include GPT-4o as a baseline in our experiments. Interestingly, despite the poor performance of VLMs in this task, they can still be used in VIBE to evaluate captions since we do not require complicated visual reasoning, but only ask the VLM to "guess" certain masked tokens and measure their confidence in next-token probability.

For the subset of examples where ranking was possible, the Mean Absolute Error (MAE) of the ranking was:

• Grounding MAE: 1.59
• Utility MAE: 1.92

These numbers reflect disagreement with VIBE as there are only 5 captions (ranking 1~5). To sum up, the experiments do not justify including GPT-4o as a reliable oracle.

Weakness 2: Multiple inference limitation
We acknowledge this limitation, as noted in the paper. However, we wish to clarify that VIBE operates at the token level—each evaluation is a single forward pass for one token, rather than a full sequence generation. Since sentence generation involves many such token-level steps, the relative cost of VIBE is comparable and not prohibitive in practice.
We will add a brief clarification to the limitations section to make this point more explicit.

Question 2: Implementation details
In all experiments (including the ablation in Appendix B), we generated five summaries per model using five temperature values: $0$, $0.5$, $1$, $1.25$, and $1.5$.
For each run, we held the prompt, model, and vLLM backend constant. The primary model used in the main paper was Qwen-2.5-72B-AWQ. For ablation studies, we also evaluated:

• InternVL-2.5-8B-MPO
• InternVL-3-38B These results are reported in Appendix B.

Question 3: Clarification on “VLM (CoT) Accuracy (xx%)”
Thank you for pointing this out. “VLM (CoT) Accuracy (xx%)” refers to a baseline where we prompt the same VLM—Qwen-2.5-72B-AWQ—to answer the 10 task questions from the user study using only the video input and chain-of-thought (CoT) prompting. The percentage reflects its accuracy on those questions.
We will clarify this definition in the main text in future revisions.

Question 4: Hardware details for reproducibility

Thank you for the reminder. We have updated the paper checklist to reflect this and added the following system information:

All experiments were conducted on four NVIDIA RTX 6000 Ada GPUs (48GB VRAM) using the vLLM backend.
The system was equipped with an Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz, featuring 64 cores (x86_64, 64-bit).
This setup handled all model variants—Qwen-2.5-72B-AWQ, InternVL-2.5-8B-MPO, and InternVL-3-38B—efficiently for both caption generation and VIBE evaluation.

We thank the reviewer for the thoughtful feedback and detailed technical questions. Below, we respond to each point.

Weakness 1: Comparison with traditional caption evaluation metrics
Traditional metrics such as ROUGE, BLEU, and CIDEr require gold-standard captions, which are unavailable and prohibitively expensive to produce—particularly for the datasets we use. More importantly, these metrics evaluate sentence-to-sentence (caption-to-caption) similarity, whereas VIBE evaluates caption-to-video alignment in a multimodal setting. Hence, VIBE focuses on a fundamentally different problem.

Gold-standard captions also introduce annotator bias and often lack the conciseness and utility needed for downstream tasks. Therefore, previous works need 3~5 human annotators per video. We believe comparing VIBE with caption-based metrics is not a meaningful apples-to-apples comparison.

Question 1: Why no direct comparison to BLEU or CIDEr?
We cannot provide a direct comparison to BLEU or CIDEr because our datasets do not include gold-standard captions, which these metrics require. This is precisely why we propose VIBE: to support evaluation in the absence of reference annotations.

Furthermore, even when available, high BLEU scores between candidate summaries do not guarantee correctness—if all captions are inaccurate, their mutual similarity is irrelevant. Our method instead evaluates how well a generated summary aligns with the video content, a dimension traditional caption-to-caption metrics fail to capture.

As suggested by Reviewer P4an, we have added an additional baseline using LLM-as-a-judge for caption selection. Specifically, we experimented with GPT-4o (via OpenAI API) as a meta-evaluator on 150 examples from LongVideoBench, used in Fig. 3. For each video, GPT-4o was asked to rank five generated captions in terms of grounding and utility (the relevance to video content and questions).

However, the results revealed significant issues:

• GPT-4o indicated that 32% of the video clips contained captions completely irrelevant to the video content, making grounding-based ranking unreliable. This issues happens even if the model is not asked to acknowledge the "unrankability".
• Similarly, GPT-4o flagged 92% of captions as irrelevant to the associated task questions, making utility-based ranking equally unstable.

This suggests that even strong VLMs like GPT-4o fail to reliably evaluate captions on LongVideoBench—likely due to known challenges in long-video understanding. Although humans can often infer correct answers from these captions, current VLMs cannot. Due to this inconsistency, we decided not to include GPT-4o as a baseline in our experiments. Interestingly, despite the poor performance of VLMs in this task, they can still be used in VIBE to evaluate captions since we do not require complicated visual reasoning, but only asking the VLM to "guess" certain masked tokens and measure their confidence in next-token probability.

For the subset of examples where ranking was possible, the Mean Absolute Error (MAE) of the ranking was:

• Grounding MAE: 1.59
• Utility MAE: 1.92

These numbers reflect disagreement with VIBE as there are only 5 captions (ranking 1~5). To sum up, the experiments do not justify including GPT-4o as a reliable oracle.

Weakness 2: On summary correctness
Regarding task correctness, we demonstrated in Fig. 4(a) and (b) that VIBE scores correlate with human task performance, as reflected by accuracy and utility ratings.

Our grounding score assesses factual correctness by measuring how token probabilities shift when the video is removed. Tokens whose probabilities remain unchanged are likely ungrounded (Eq. 3), suggesting the model generated them without relying on visual evidence. As a result, hallucinated content typically receives grounding scores near zero. Similarly, when a caption is factually incorrect—either irrelevant or contradictory—inputting the video tends to suppress the probability of those tokens, leading to zero or negative scores. This mechanism allows our method to identify ungrounded or inaccurate content.

Question 2: Clarification on score range and normalization
We believe the reviewer is referring to the utility score, as it has the closest range to [0, 1]. We also acknowledge that Fig. 3 had the axes flipped: the X-axis should be “Grounding Score $\log I(V; T)$” and the Y-axis should be “Utility Score $\log I(Y; T)$”. This has been corrected.

As noted in Line 132, pointwise mutual information (PMI) can range from $[−∞, ∞]$. While the scores may appear bounded in Fig. 3, this is not mathematically the case—both grounding and utility scores are unbounded in theory.

Regarding score behavior: the grounding score (Eq. 3) is computed at the token level, while the utility score considers the joint probability over multiple masked tokens. This means utility scores tend to be higher when more relevant keywords appear. We considered normalizing by the number of masked tokens, but doing so would break consistency with mutual information theory and would make $\alpha$ a function of the number of masked tokens.

Question 3: Influence of summary length on response time and accuracy
We computed the mean and standard deviation of word count for each method across datasets:

As shown, the summary lengths are relatively consistent across methods within each dataset. This is expected, as all methods select from the same candidate pool with comparable length distributions. Therefore, differences in task accuracy and response time found in Table 1 are unlikely to be explained by length alone.

As shown in Appendix Fig. 9(f), we conducted a correlation analysis between summary length and task accuracy across all three datasets:

• LongVideoBench: No significant correlation
• SUTD-TrafficQA: No significant correlation
• LearningPaper24: A weak positive correlation was observed (Spearman $r\_s(38) = 0.429,\ p = 0.006$)

These results suggest that while summary length may have some influence, particularly in the LearningPaper dataset, it is not a consistent or dominant factor. Instead, accuracy appears to be better predicted by the utility score we proposed. We conjecture that task accuracy depends more on the presence of informative content (e.g., task-relevant keywords) than on overall word count.

Regarding response time, we did not observe a significant correlation with summary length in any dataset. This may be due to the narrow length distribution across conditions, as all summaries were drawn from the same candidate pool. While it is intuitive that longer summaries require higher response time, we consider this effect an inherent property of human information processing and hence not a focus of our analysis.

We thank the reviewer for the insightful comments and for pointing to relevant related work. Below, we address the specific concerns raised.

Weakness 1: On grounding score and hallucination
The grounding score measures the logarithm of the "lift" in next-token probability with and without the video, given a masked summary (Eq. 3). Roughly speaking, hallucination can arise from two sources:

1. Internal model bias — originating from pretraining or fine-tuning;
2. External input — due to misleading or misinterpreted content from the video or summary.

If hallucination stems from (1), the model’s predictions likely remain unchanged regardless of the video, and the grounding score becomes ($\log 1 = 0$), indicating the summary is ungrounded.

If it arises from (2)—due to misleading content or VLM misinterpretation or mis-attribution of key frames—the summary is still grounded, because the video influences generation. The masked summary, with removal of meaningful keywords, should carry no new information and thus should not contribute to hallucination

The works referenced by the reviewer ([1], [2]) primarily examine the fundamental limitations of VLMs, which can be categorized into case (1). Since the grounding score also considers the accuracy (probability of answering the correct option) without video, this term acts as a normalization term to mitigate the internal model bias.

Weakness 2: Object-centric bias and framing of short clips
We agree that object-centric bias is a known limitation of some VLMs, but this issue stems from model pretraining, not from our proposed evaluation method. Recent VLMs are typically trained to mitigate this bias, and we do not observe it to significantly impact our results.

As noted in Lines 282–284, traffic events in SUTD-TrafficQA unfold over just a few seconds. In such settings, directly viewing the clip is often more efficient and informative than reading textual summaries—even when the sampled frames inputted by VLM capture the full event and the generated summary is accurate. This emphasizes VIBE’s value in scenarios where processing full videos is costly or time-constrained.

We thank the reviewer for the thoughtful and detailed feedback. Below, we respond to each concern individually.

Weakness 1: Missing coverage of VLMs in Related Work
We have added a subsection in the related work to address recent advancements in video-language models (VLMs). This new section surveys key developments in VLM architecture, training, and evaluation benchmarks--including models not directly used in the paper [1, 2, 3].

We also discuss how these models have been applied in related tasks such as video captioning [4], video understanding, and multimodal reasoning [5], to contextualize our approach within the broader landscape of VLM research. In the final version, we will further expand this subsection to include additional relevant works beyond the citations listed here.

[1] Nguyen, Thong, et al. "Video-language understanding: A survey from model architecture, model training, and data perspectives." arXiv preprint arXiv:2406.05615 (2024).

[2] Tang, Yunlong, et al. "Video understanding with large language models: A survey." IEEE Transactions on Circuits and Systems for Video Technology (2025).

[3] Li, Zongxia, et al. "A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges." arXiv preprint arXiv:2501.02189 (2025).

[4] Abdar, Moloud, et al. "A review of deep learning for video captioning." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

[5] Li, Yunxin, et al. "Perception, reason, think, and plan: A survey on large multimodal reasoning models." arXiv preprint arXiv:2505.04921 (2025).

Weakness 2: Clarification on the challenge and novelty
We appreciate the opportunity to clarify this. Our key challenge lies in evaluating video-language alignment in unannotated, real-world videos without human-annotated captions—a setting poorly addressed by existing metrics. Unlike prior work that assumes access to human-annotated gold-standard text (e.g., captions or scripts), our approach enables scalable evaluation directly from raw video and model-generated text.

This setting introduces both conceptual and technical innovations. Conceptually, it redefines the evaluation of "video captioning" without relying on human annotations. Technically, it grounds evaluation in mutual information approximations between modalities, as detailed in Section 4.

In contrast, prior works in the related work section applied mutual information with the token masking technique only to assess text-to-text summary quality.

Weakness 3: Mathematical derivation to justify approximation using pointwise mutual information

We derive the case where the approximation holds under the stated assumption, starting from the right-hand side of Eq. 3 and 4. Since both approximations follow nearly identical steps, we present the derivation of the grounding score as an example:

Since $T_{\mathrm{masked}}$ is a masked sentence with all keywords removed, we assume it contains no information and is independent of any other random variables. Thus, we can rewrite the expression as:

All steps follow from Bayes’ theorem and the assumption of independence.

Weakness 4: Metrics on redundancy reduction

Thank you for the suggestion. Recent works [1,2] on long video understanding with VLMs often generate clip-level captions and aggregate them for downstream tasks, resulting in a lengthy and verbose summary composed of many sentences. While we briefly mention this in the abstract, it is not the central focus of our study. Our current experiments emphasize downstream task performance rather than length reduction.

That said, we believe VIBE offers a principled approach to addressing verbosity—not by simply truncating output, which can compromise grounding and utility—but by selectively preserving the most informative content. We have noted verbosity evaluation as a promising direction in the future work section.

[1] Hur, Chan, et al. "Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[2] Wang, Ziyang, et al. "Videotree: Adaptive tree-based video representation for LLM reasoning on long videos." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

Weakness 5: Interpretation of SUTD-TrafficQA results
Thank you for raising this point. If the paper is accepted, we will expand the interpretation in the final version to further contextualize the results.

Regarding the specific findings on SUTD-TrafficQA: As shown in Table 1, while the accuracy of Video-Only and Max-G is similar, the key differentiator is response time (Video-Only: 48.63s; Max-G: 80.97s; Max-U: 137.27s). As discussed in Lines 272–275 and 297–298, this pattern is due to the short video durations (2–10 seconds) in SUTD-TrafficQA. In such settings, participants can extract relevant information directly from the video with low working memory demand. In contrast, text summaries require parsing language and mentally reconstructing events—particularly challenging for temporally and spatially grounded traffic event questions.

This result suggests a promising future direction: systematically identifying the boundary conditions under which summarization enhances human task performance versus when direct video consumption is more effective. Exploring this trade-off more systematically could inform adaptive interfaces that tailor content presentation to context. Despite this open question, the consistent utility-accuracy correlation trends across datasets (see Appendix Fig. 9 (b)) support the generalizability of our core findings. Moreover, video-to-text summarization methods offer clear value in scenarios where visual attention is constrained—such as accessibility applications, mobile multitasking, or hands-free settings—where direct video inspection may not be feasible and text can be converted to audio for easier accessibility.

Weakness 6: Elaboration on Applicable Scenarios
We appreciate the chance to elaborate on our evaluation setting. Our work focuses on real-world scenarios where videos often lack human-generated captions. This is typical in surveillance, driving, sports, and other domains where scalable annotation is infeasible. As such, conventional reference-based metrics (e.g., CIDEr, METEOR) are not applicable.
The datasets we selected reflect this challenge: they contain raw video and questions but no canonical ground-truth summaries. Our approach—VIBE—avoids reliance on expensive or biased annotations by directly measuring alignment between video and model-generated summaries. We believe this setup is not only practical but essential for general-purpose video understanding evaluation going forward.