CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding

Q1: Definition of Clue.

Thank you for your feedback. We will clarify the definition of "clue" in the paper. Specifically, "clue" refers to the temporal boundaries (represented by start and end timestamps) of the video segments that provide hints for correctly answering the question. We will include this definition in the relevant section of the paper to enhance clarity.

Q2: Adding other related benchmarks and models.

Thank you for your valuable comments. Regarding the missing benchmarks/datasets in Table 2, we have supplemented and discussed them in the revised manuscript. Specifically, we have added detailed discussions on EgoTimeQA, MultiHop-EgoQA, ReXTime, and E.T. Bench, and included the methods proposed by these papers in Table 3.

1. In the revised manuscript, we have added detailed discussions on EgoTimeQA, MultiHop-EgoQA, ReXTime, and E.T. Bench, focusing on comparing these datasets with CG-Bench. We highlighted that although these datasets share some similarities with CG-Bench in the QA and grounding settings, their videos are sourced from academic datasets and lack diversity in length, such as Ego4D, QVHighlight, and ActivityNet. In contrast, our CG-Bench primarily focuses on open-domain long videos, thereby introducing more diverse textual queries. Additionally, we have implemented more evaluation mechanisms to provide a more comprehensive assessment.
2. In Table 3, we have included the methods proposed by these papers and conducted a detailed performance comparison. We tested GroundVQA, GeLM, and ET-Chat on the Question Grounding and Long-MCQ tasks, with the results as follows:

Q11: Abalation studies of Subtitles.

We conducted an ablation study focusing on subtitles, with the results as follows (f-128 indicates using subtitles corresponding to uniformly sampled 128-frame timestamps, and full-video indicates using all subtitles from the full video):

We have updated these results in the revised manuscript. The experimental results indicate that although subtitles provide useful semantic signals, their benefit is significantly diminished when visual input is used. This also indirectly demonstrates that our benchmark is more inclined towards visual signals.

Q12: More explaination of Table 5.

In table 5, Bias is the absolute value of the difference between the human evaluation score and the model evaluation score (in percent). Trigger rate refers to the probability that the model triggers "visual clues required". Recall rate refers to the recall rate of the model triggering "visual clues required" given the manually annotated trigger annotations.

Q13: Curve of CRR for GPT4o.

The short-acc of each model is a model-related constant, leading to a strong correlation between the CRR curve and the long-acc curve. Both GPT-4o and Gemini exhibit a curve that initially declines before rising, which I believe is influenced by their respective training strategies. As commercial models, GPT-4o and Gemini possess extensive conversational capabilities, resulting in less stable performance on multiple-choice questions (MCQs), especially in sparse contexts. In contrast, Qwen2VL appears to be more benchmark-oriented, possibly undergoing comprehensive training for MCQs, enabling it to "guess" answers more effectively from visual information.

Q14: What is the "refusal rate" of Gemini.

When using Gemini, it sometimes rejects questions for some reason. There are two main reasons: 1. Gemini thinks the question involves privacy issues. 2. Gemini thinks there is not enough information to answer the question. According to our statistics, Gemini's rejection rate is about 31%.

Q15: Experiments of open-ended evaluation quality (L464-L466).

Thank you for comments. Your understanding is correct, we have added the explanation in the revised paper.

Q1: Process of collecting and filtering videos. Thank you for your valuable feedback. Our videos primarily come from Bilibili and YouTube. We initially constructed broad Level 1 and Level 2 video tags and used these tags for manual searches. During this process, we expanded the Level 2 tags and annotated Level 3 tags. In the manual filtering process, we applied the following criteria:

1. Videos must exhibit sufficient dynamism.
2. For knowledge-related videos, we retained those with some visual dynamism and excluded those that were purely speech-based.
3. We prioritized selecting the most recently uploaded videos to ensure they are as held-out as possible.
4. For each Level 3 tag, we retained only 1-2 videos.
5. We developed a checking program to ensure that the selected video IDs do not overlap with those in major existing video datasets, including coin, youcook2, activityNet, hacs, cinepile, Crosstask, finegym, finevideo, hdvila100m, hirest, howto100m, internvid, kinetics, Mira data, openvid1m, panda70m, queryd, qvhighlight, shot2story, sports1m, tapos, uvo, valor, vast, vidchapters, vitt, vript, youtubehl and yt-temporal-1b, MultiHateClip and ChinaOpen.

We will include these details in the revised version to enhance the understanding of the benchmark's composition and applicability.

Q2: Open-source benchmark data.

We will release the data within a few days. To avoid the issue of video download failure, we will provide a solution that can definitely download the video which will not bring license issues.

Q3: more examples and Hallusion question.

We have added more QA and QAC examples in the supplementary materials. Here is one example of Hallusion Question.

Question: When the protagonist of the video set the language for the game, which of the following options are correct? 
1. Thai is one of the languages selected by the protagonist 
2. Bulgarian is one of the languages selected by the protagonist 
3. Romanian is one of the languages selected by the protagonist 
4. Finnish was selected by the protagonist first and then cancelled 
5. Danish was selected by the protagonist first and then cancelled
Option: 
A. 1 and 3; 
B. 2 and 5; 
C. 2 and 4; 
D. 1, 3 and 5; 
E. 1, 2 and 4; 
F. 2, 3 and 5

Different from the general binary discrimination hallucination test questions, we adopted a multiple-choice setting to find multiple correct answers or multiple wrong answers from multiple statements. This significantly increases the difficulty and efficiency of the hallucination test.

Q4: Figure of clue coverage.

Thank you for your insightful comment. We discovered that the increase in clue coverage was due to a missing normalization step in our code when generating the plot. We have corrected this error and updated the figure in the revised version of the paper.

Q5: Translation of questions.

Yes, all questions are translated into English. Initially, we experimented with GPT-4 for translation, but we observed unstable results. Therefore, we ultimately used Google Translate for this task. After translation, we conducted a manual review of the QA pairs to ensure accuracy and maintain the integrity of the original expressions.

Q6: How to make sure the answers are grounded in visual content?

We require annotators to annotate questions that contain clues in the video, mainly using vision as the anchor. A small number of samples can use audio, audio + video, or subtitles as anchors to enhance the multimodal evaluation capabilities of the benchmark. During the Review Iteration process, we manually check whether most annotated questions are queries about vision.

Q7: Acc@IoU with τ=0.

Regarding acc@IoU when τ=0, it does not simply reduce to accuracy. Simple accuracy does not account for the model's ability to identify the clue interval. In contrast, acc@IoU with τ=0 requires the model to not only select the correct option but also produce a time interval that overlaps at least slightly (tIoU > 0, not >= 0) with the annotated clue interval.

Q8: Rephrasing the definition of CRR.

Thank you for the valuable suggestion. We have rephrased the description about CRR as the reviewer suggested. Please refer to the revised manuscript for this change.

Q9: Why GPT4o is a low hallucination model ?

Thank you for your comment. Low hallucination is a relative concept. We refer to GPT-4o as a low-hallucination MLLM because of its outstanding performance on several well-known benchmarks, such as OpenCompass and the lmsys leaderboard. In the revised manuscript, we will add relative descriptions to convey this more clearly.

Q10: why are mIoU scores >1 ?

Thank you for your reminder. This value is in percentage and we have modified it in the revised manuscript.

Q1: How to ensure the videos are held out?

This is indeed a good question. It is challenging to ensure with absolute certainty that the video data has not been used in model pre-training, especially for commercial models that may have utilized all the available online content, including validation and test sets of academic datasets. For our video selection, we focused on hindering overlapping with publicly available academic datasets. To achieve this, we implemented two main strategies:

1. We selected videos from the most recently uploaded content.
2. In addition, we developed a filtering tool and excluded all the videos that have already existed in major video datasets. This is done by checking whether their video IDs already exist in these datasets, including coin[1], youcook2[2], activityNet[3], hacs[4], cinepile[5], Crosstask[6], finegym[7], finevideo[8], hdvila100m[9], hirest[10], howto100m[11], internvid[12], kinetics[13], Mira data[14], openvid1m[15], panda70m[16], queryd[17], qvhighlight[18], shot2story[19], sports1m[20], tapos[21], uvo[22], valor[23], vast[24], vidchapters[25], vitt[26], vript[27], youtubehl[28], yt-temporal-1b[29], MultiHateClip[30] and ChinaOpen[31].

By this means, approximately 20M video IDs were excluded to ensure that our video data are held out to the largest extent.

Q2: Human performance on sparse uniformly sampling.

Thank you for your insightful comments. To address your suggestion, we conducted a quick human evaluation experiment under constrained visual conditions. We uniformly sampled 30 videos from CG-Bench, resulting in 296 questions. For each video, we uniformly sampled 128 frames and asked volunteers to perform a multiple-choice question (MCQ) testing. The resulting accuracy was 59.85%. This result indicates that our dataset is indeed challenging and that it is difficult to derive solutions from a limited number of frames. It also highlights that even the most advanced models, such as GPT-4o, have ample room for improvement in long video comprehension.

Q3: Performance grouped by video duration.

Thank you for your insightful suggestion regarding a finer-grained analysis. We have conducted the analysis by grouping videos based on their duration and assessing the long-acc performance of GPT-4o with 128 frames. The results are as follows:

These results indicate that the current model is facing undersampling issues, particularly with longer videos. We will release the code for this grouped analysis alongside our open-source materials to facilitate further exploration and validation by the research community.

Q4: Metric design and Acc@IoU with τ=0.

We acknowledge that the accuracy and tIoU metrics are not novel; however, they are widely accepted and reliable within the community. Our focus is on how combining these metrics can provide more insightful evaluations. Regarding acc@IoU when τ=0, it does not simply reduce to accuracy. Simple accuracy does not account for the model's ability to identify the clue interval. In contrast, acc@IoU with τ=0 requires the model to not only select the correct option but also produce a time interval that overlaps at least slightly (tIoU > 0, not >= 0) with the annotated clue interval. We have updated the corresponding part in the main manuscript for a clearer explanation.

Reference

[1] Tang Y, Ding D, Rao Y, et al. Coin: A large-scale dataset for comprehensive instructional video analysis.

[2] Zhou L, Xu C, Corso J. Towards automatic learning of procedures from web instructional videos.

[3] Caba Heilbron F, Escorcia V, Ghanem B, et al. Activitynet: A large-scale video benchmark for human activity understanding.

[4] Zhao H, Torralba A, Torresani L, et al. Hacs: Human action clips and segments dataset for recognition and temporal localization.

[5] Rawal R, Saifullah K, Basri R, et al. Cinepile: A long video question answering dataset and benchmark.

[6] Zhukov D, Alayrac J B, Cinbis R G, et al. Cross-task weakly supervised learning from instructional videos.

[7] Shao D, Zhao Y, Dai B, et al. Finegym: A hierarchical video dataset for fine-grained action understanding.

[8] Farré, Miquel and Marafioti, et al. FineVideo.

[9] Sun Y, Xue H, Song R, et al. Long-form video-language pre-training with multimodal temporal contrastive learning.

[10] Zala A, Cho J, Kottur S, et al. Hierarchical video-moment retrieval and step-captioning.

[11] Miech A, Zhukov D, Alayrac J B, et al. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.

[12] Wang Y, He Y, Li Y, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation.

[13] Carreira J, Zisserman A. Quo vadis, action recognition? a new model and the kinetics dataset.

[14] Ju X, Gao Y, Zhang Z, et al. Miradata: A large-scale video dataset with long durations and structured captions.

[15] Nan K, Xie R, Zhou P, et al. Openvid-1m: A large-scale high-quality dataset for text-to-video generation.

[16] Chen T S, Siarohin A, Menapace W, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers.

[17] Oncescu A M, Henriques J F, Liu Y, et al. Queryd: A video dataset with high-quality text and audio narrations.

[18] Lei J, Berg T L, Bansal M. Detecting moments and highlights in videos via natural language queries.

[19] Han M, Yang L, Chang X, et al. Shot2story20k: A new benchmark for comprehensive understanding of multi-shot videos.

[20] Karpathy A, Toderici G, Shetty S, et al. Large-scale video classification with convolutional neural networks.

[21] Shao D, Zhao Y, Dai B, et al. Intra-and inter-action understanding via temporal action parsing.

[22] Wang W, Feiszli M, Wang H, et al. Unidentified video objects: A benchmark for dense, open-world segmentation.

[23] Chen S, He X, Guo L, et al. Valor: Vision-audio-language omni-perception pretraining model and dataset.

[24] Chen S, Li H, Wang Q, et al. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.

[25] Yang A, Nagrani A, Laptev I, et al. Vidchapters-7m: Video chapters at scale.

[26] Huang G, Pang B, Zhu Z, et al. Multimodal pretraining for dense video captioning.

[27] Yang D, Huang S, Lu C, et al. Vript: A Video Is Worth Thousands of Words.

[28] Sun M, Farhadi A, Seitz S. Ranking domain-specific highlights by analyzing edited videos.

[29] Zellers R, Lu J, Lu X, et al. Merlot reserve: Neural script knowledge through vision and language and sound.

[30] Wang H, Yang T R, Naseem U, et al. Multihateclip: A multilingual benchmark dataset for hateful video detection on youtube and bilibili.

[31] Chen A, Wang Z, Dong C, et al. Chinaopen: A dataset for open-world multimodal learning.

Thanks for the explanation and additional experiments! Sorry that my initial review might be not clear enough regarding $\tau=0$ -- a naive dummy solution under such constraint is basically predicting full span for all time so it would ensure overlap>0 always holds. Any comments on that?

Q1: Reliability of interval annotation and Recall@IoU

Regarding the reliability of temporal interval annotations, this issue has been widely discussed in early temporal understanding works such as Charades-STA[1] and ActivityNet-Caption[2]. Due to the high-level semantic nature of events, there is often ambiguity in event boundaries, leading to low agreement among annotators. This is why we provide human performance on mIoU to relax the optimal target and make the evaluation more realistic. We agree that using Recall@IoU is another valuable metric to consider. We have conducted preliminary calculations, and the results are as follows:

However, we believe that mIoU still holds reference value, and we plan to retain both metrics. The related content has been updated in the manuscript. We appreciate the reviewer for the valuable suggestion.

Q2: Impact of prompts on outputing interval formats.

This sentence, "Any output that does not conform to this nested array format will be considered incorrect.", serves as a "warning" to the models in the prompt. Before using this sentence, the success rates for GPT-4o and Gemini in following the nested array format instructions were approximately 62.3% and 53.5%, respectively. After adding this sentence, the success rates improved to 91.0% and 82.4%. In actual testing, outputs that did not conform to the required format were not considered incorrect; instead, we requested the output again until the model produced the correct format. Therefore, all final predictions could be used to compute valid tIoUs.

Q3: License issues for releasing videos.

To avoid the issue of video download failure, we will provide a solution that can definitely download the video which will not bring license issues.

Reference

[1] Gao J, Sun C, Yang Z, et al. Tall: Temporal activity localization via language query.

[2] Iashin V, Rahtu E. Multi-modal dense video captioning.

Q7: QA examples filtered by the "small model & sparse frame check" step.

Regarding the quality control process, especially for the "small model & sparse frame check" step, we appreciate the reviewer’s request for more insights. In general, questions lacking temporal dynamics are easily addressed by sparse frame checks, while small models excel at solving the corresponding simpler questions. For instance:

Question 1: What color is the protagonist's cycling outfit in the video?
A. Red
B. Blue
C. Green
D. Black
E. White

Analysis: The protagonist is cycling in a first-person view, and the outfit appears throughout the video.

Question 2: What does the climbing wall in the video look like?
A. An indoor artificial climbing wall
B. A climbing wall with various color markers
C. An outdoor low-altitude climbing wall
D. A training wall with multiple climbing routes
E. A cliff face

Analysis: The climbing wall is a prominent target, and the distinctions between the options are very clear, requiring minimal comprehension.

These examples illustrate how the "small model & sparse frame check" step effectively filters questions that do not require extensive temporal understanding or complex reasoning.

Q1: Distribution of video resolution.

We have analyzed the distribution of video resolutions. Our findings show that there are 1,065 videos with a short side between 720 and 1080 pixels, 120 videos with a short side exactly 720 pixels, and 34 videos with a short side less than 720 pixels. This indicates that the majority of our videos are high-definition.

Q2: How to collect videos and ensure the videos are held out?

Thank you for your valuable feedback. Our videos primarily come from YouTube and BiliBili. We initially constructed broad Level 1 and Level 2 video tags and used these tags for manual searches. During this process, we expanded the Level 2 tags and annotated Level 3 tags. In the manual filtering process, we applied the following criteria:

1. Videos must exhibit sufficient dynamism.
2. For knowledge-related videos, we retained those with some visual dynamism and excluded those that were purely speech-based.
3. We prioritized selecting the most recently uploaded videos to ensure they are as held-out as possible.
4. For each Level-3 tag, we retained only 1-2 videos.
5. We developed a checking program to ensure that the selected video IDs do not overlap with those in major existing video datasets, including coin, youcook2, activityNet, hacs, cinepile, Crosstask, finegym, finevideo, hdvila100m, hirest, howto100m, internvid, kinetics, Mira data, openvid1m, panda70m, queryd, qvhighlight, shot2story, sports1m, tapos, uvo, valor, vast, vidchapters, vitt, vript, youtubehl, yt-temporal-1b, MultiHateClip and ChinaOpen. We will include these details in the revised version to enhance the understanding of the benchmark's composition and applicability.

Q3: Where are the subtitles from?

Our subtitles are downloaded directly from the video website. However, not all videos have subtitles. According to our statistics, 50.12% of videos have subtitles, mainly in English and Chinese, and a small amount of other languages.

Q4: Effect of frame sampling strategies.

We conducted relevant experiments. In order to speed up the testing process, we mainly tested GPT4o at 50 frames. The experiment was divided into three parts: low resolution, high resolution, and key frame extraction (by ffmpeg) + low resolution. The results are as follows:

The results show that higher resolution can bring some improvement, but keyframes have no significant effect. We have updated the table in the revised paper.

Q5: More explanation on Black-box evaluation

In general, providing a model with more information should enhance the accuracy of question answering. Ideally, the accuracy with the full video (long-acc) should be greater than or at least equal to the accuracy with only the clue interval (clue-acc). This is primarily because the human-annotated clue intervals may not always represent the optimal clue segments; for instance, the actual clues might extend beyond the annotated intervals. Content outside the clue interval may also indirectly and implicitly help in the reasoning process. A sufficiently sampled and sufficiently trained long-video MLLM should be capable of automatically identifying all useful relevant clues. Therefore, under the same sampling settings, CRR can be used to assess the extent to which the same model has been effectively trained on long contexts. For cross-model comparisons, CRR can evaluate how is alignment extent of different models between both short and long contexts. For example, with the same 32/128 sampling settings, the CRR for Gemini is 70.5, while for GPT-4o it is 81.1. This indicates that Gemini's performance in matching short and long contexts is weaker compared to GPT-4o.

Q6: Explanation on white-box evaluation without timestamp information.

In general, most existing MLLMs treat videos as a series of images. When multiple images are provided without the corresponding timestamps, the model lacks the temporal context necessary to process these images as a cohesive video. This absence makes it challenging for the model to perform comprehensive temporal dynamic modeling, leading to some uncertainty in the model's output and resulting in suboptimal performance. Adding additional temporal information helps improve tasks such as temporal localization.

Thanks the authors for the detailed response. The reviewer still have a few doubts, detailed below:

• If subtitles are not available for 50% of the videos, are Table 4 results on the whole dataset, or just the subset that have subtitles? If on the whole dataset, Table 4 results do not correctly reflect the impact of adding subtitle or subtitle timestamps.
• Black-box evaluation: If possible, can the authors provide examples where (1) the human-annotated clue intervals may not always represent the optimal clue segments and (2) content outside the clue interval may also indirectly and implicitly help in the reasoning process.
• White-box evaluation without timestamp information: the reviewer's confusion is mainly on how the models would respond for clue grounding when no timestamp information is provided to the model. Especially looking at the prompt in L1518, the models are required to respond in seconds. When only the frames are provided, it is nearly impossible for even humans to ground the clues in seconds. Hence, the reviewer thinks mIoU in this case is not very meaningful when timestamps are not provided.

Q1: Ablation studies on the subset with subtitles.

Our original intention was to establish a connection between Table 3 and Table 4 by presenting the performance metrics for the complete dataset. This dataset includes videos without subtitles; however, their inclusion does not affect the comparisons made in the ablation experiments.

After your valuable feedback, we realized that including the entire dataset obscures the true impact of different prompts and modalities on performance because the gain values are averaged. To address this, we recalculated the performance metrics using a subset of videos that include subtitles only, as detailed below: In the revised manuscript, we will replace Table 4 with this new table and move the original Table 4 to the appendix as an additional reference.

Q2: Examples for black-box evaluation

Generally speaking, the two situations you mentioned cannot be completely decoupled. To demonstrate this situation comprehensively, we found an example from our dataset. We also provide the corresponding online video link for your reference.

Example:

video_link: https://www.youtube.com/watch?v=WHAMiF-YJtQ

Question: What did the two protagonists do after dinner in Florence, Italy?

Choices: A. Watch the sunset over city B. Exchange gifts C. Hug and kiss D. Enjoy a man singing E. Drink together

Direct clue: [21:30, 21:45] This segment shows the two protagonists hugging and kissing after dinner in Florence. It is directly relevant to the question.

Indirect clues:

1. [[7:41, 7:50], [9:49, 9:54], [23:12, 23:16]] These intervals show moments where the protagonists share intimate moments, like kissing each other, drinking the same bottle of water, and being close to each other. These moments can imply a closer relationship, helping reason that after dinner, they would engage in physical affection (hugging and kissing).

2. [[11:12, 11:35], [12:51, 13:30], [16:17, 16:28], [20:15, 21:00]] These intervals show iconic landmarks and activities in cities like Venice and Rome. This is useful to help filter out irrelevant parts of the video, ensuring the answer stays focused on Florence. It also helps exclude other possible and similar scenarios in different cities.

Suboptimal clue: [21:36, 21:45]. This segment only shows the two protagonists hugging and kissing. However, the question asks "After dinner in Florence". The "dinner in Florence" part should also be included in the clue annotation.

Q3: how the models would respond for clue grounding when no timestamp information is provided to the model?

In the case where the testing without the timestamps (row F/F+S in table-4), we prompt the model that these images are sampled uniformly from the entire video and require it to output the normalized time interval between 0 and 1. During the evaluation, we convert the normalized output into an interval of seconds to calculate the tIoU.