InfiniBench: A Comprehensive Benchmark for Large Multimodal Models in Very Long Video Understanding

---

Q 3.2 (Continued)

Using 250 frames per video is not applicable for LLava-OV and InternVL, as they are standard video models not designed to handle too-long videos. Accordingly, we hit the maximum context length for these models, which prevents us from running on higher sampling rates.

---

Q4: Are there text scripts (screenplay) associated with all the videos (movies and TV shows)?

Yes, transcripts are available for all the movies and TV series included in our benchmark.

• We collected these transcripts from publicly available sources on the internet and will publish them on our website to ensure convenience and facilitate future research.

• However, as discussed earlier, we will not release the video content itself. Instead, we will refer users to the original sources, such as the MovieNet and TVQA datasets, where the videos can be independently accessed.

This approach ensures compliance with copyright and ethical standards while making the benchmark resources easily accessible for the research community.

---

Q3:

Q3.1: On GPT-4o evaluation, only 250 frames are selected. Are the 250 frames selected uniformly?

Yes, we sample 250 frames uniformly.

Q3.2: Have you tried to reduce the frame size and squeeze more frames into GPT-4o?

1. Influence of Input Frame Rate:

• Feeding more frames intuitively improves accuracy, but the degree of improvement varies across models.

• For instance, GPT-4o benefits the most from higher frame rates, while LLaVA-OV’s performance remains almost unchanged despite using an 8x higher frame rate.

2. Analysis of LLaVA-OV’s Behavior:

• The limited benefit of higher frame rates for LLaVA-OV may be attributed to its training strategy.

• LLaVA-OV is trained jointly on single images, multi-images, and videos.

• This strategy employs a balanced visual representation approach, aggressively downsampling video inputs to ensure parity with image-based scenarios.

• While effective for general tasks, this aggressive downsampling likely hurts LLaVA-OV’s ability to understand long videos, limiting its benefit from higher frame rates.

3. Skill-Specific Insights:

• Specific skills benefit more from higher frame rates. For example, the ``local vision+text'' skill improves most as it relies on short sequential shots.
• Increasing the frame rate reduces the chance of missing critical shots related to the answer, thereby boosting accuracy for such tasks.

The results demonstrate that while higher frame rates generally improve performance, the degree of improvement depends on the model’s design and training strategy. Models like GPT-4o, optimized for sequential inputs, show significant gains, whereas models like LLaVA-OV, which aggressively downsample videos, see minimal benefits.

---

Q1: Can the authors comment on the limited variety of TV shows? What about sports events like NBA, NFL, Tennis, etc.

We appreciate the reviewer's suggestion and agree that including a wider variety of video sources, such as sports events, could add value to future benchmarks. However, we argue that movies and TV shows are highly suitable and effective for assessing long-video understanding for the following reasons:

1. Diverse and Complex Contexts:

• Movies and TV shows are rich in content, featuring intricate character relationships, evolving storylines, and multi-layered themes. These elements introduce dynamic and complex reasoning challenges beyond the repetitive or monotonic scenarios often found in daily-life videos like vlogs or live streams.

• For instance, movies often include unexpected actions, rapid shifts in context, and non-linear narratives that demand a higher level of understanding, making them ideal for testing models' ability to handle long-term dependencies and reasoning tasks.

2. Variety of Skills Tested:

• Our benchmark is designed to include diverse questions that evaluate different levels of video understanding, from surface-level observations to deeper reasoning about characters, events, and causality.

• This holistic design ensures that the benchmark challenges models on a wide range of skills, as demonstrated by the low performance of current state-of-the-art models.

3. Limitations of Casual Videos:

• Vlogs and daily-life videos often revolve around singular, straightforward activities (e.g., riding a bicycle or walking through a park). These scenarios typically lack the nuanced interplay between characters and the layered storytelling in cinematic content.

• While casual videos may help test immediate perception tasks, they are less suited for evaluating the reasoning and complex relational understanding required for true long-video comprehension.

4. Storytelling Patterns:

• While movies and TV shows follow storytelling patterns, these patterns are not uniform and vary greatly across genres, cultures, and creators. This inherent variability further enriches the benchmark by introducing a broad spectrum of reasoning and comprehension challenges.

Acknowledgment and Future Work:

We acknowledge that including additional categories of videos, such as sports events, could further diversify the benchmark and make it more comprehensive. We plan to explore incorporating these sources in an extended version of the work or future studies.

---

Q2: Will all the videos be released to public? Are there any legal issues?

We appreciate the reviewer’s concern about copyright implications and ethical considerations related to using movies and TV shows in our dataset. Below, we clarify our approach and steps to ensure compliance with legal and ethical standards:

1. Use of Existing Datasets (e.g., MovieNet and TVQA):

• We rely on publicly available datasets like MovieNet and TVQA, which provide downsampled versions of video content.

• Both datasets are widely used in academic research and distribute videos in a legally compliant manner by heavily reducing the frame rate (e.g., 1 FPS). This downsampling transforms the videos into low-resolution derivatives primarily for research purposes, which may mitigate copyright concerns.

• While we cannot definitively state that downsampling resolves all legal and ethical issues, its widespread use in academia suggests it is generally accepted within the research community.

2. Benchmark Without Redistribution of Videos:

• Our benchmark does not redistribute video content directly. Instead, we provide annotations, question-answer pairs, and evaluation scripts. Users are required to independently download the videos from official sources (e.g., MovieNet or TVQA).

• This ensures we do not claim ownership of the original video content, nor do we host or distribute it ourselves.

• This approach aligns with practices used in existing benchmarks, where users are required to obtain the original images independently.

3. Ethical Considerations:

• We acknowledge that relying on copyrighted material, even in downsampled form, can raise ethical questions.

• We are committed to transparency and ensuring our benchmark is used responsibly. To that end, we will include a clear disclaimer stating:

1. The benchmark does not redistribute or modify original video content.

2. Users must adhere to the terms and conditions of the original video sources.

Conclusion:

While we cannot claim a definitive resolution of all legal or ethical concerns, our approach of using downsampled versions from widely accepted datasets and ensuring users obtain the videos independently reduces our direct responsibility for copyright compliance. If further legal concerns arise, we will explore ways to refine our approach in collaboration with legal experts and ensure the benchmark remains compliant and ethical.

---

Q1: Compare with MoVQA.

We appreciate the contributions of MoVQA and acknowledge its value in advancing video question-answering research. However, our benchmark is significantly larger in scale and broader in scope, covering a wider range of skills, video durations, and models.

[Key Differentiation Points:]

1. Scale:

   • Our benchmark is 5× larger than MoVQA, with 108.2K QA pairs compared to 21.953K QA pairs in MoVQA.

2. QA Diversity:

   • We support both multiple-choice questions (MCQ) and open-ended evaluations, while MoVQA is limited to MCQ only.

3. Video Sources:

   • Our dataset features videos from both TV shows and movies, whereas MoVQA focuses solely on movies.

4. Video Length:

   • The average video length in our benchmark is significantly longer (52.59 minutes) compared to MoVQA (16.53 minutes).

5. Model Coverage:

   • We evaluate 10 long-video models, whereas MoVQA evaluates only 4 models (MPlug-Owl, Otter, VideoChatGPT, and VideoChat).

[Comparison with Other Recent Benchmarks:]

In addition to comparing with MoVQA, we have expanded our evaluation to include other recent benchmarks, such as Video-MME, LVBench, LongVideoBench, and MLVU.

Here are some key highlights:

1. Video Length:

   • Most benchmarks focus on short videos, with an average length of around 10 minutes.

   • The exception is LVBench, which includes 1-hour-long videos, making it comparable in duration to our dataset.

2. Scale:

   • Our benchmark is 70× larger than LVBench.

3. QA Types:

   • We support both MCQ and open-ended QA, while other long-video benchmarks, such as LVBench and LongVideoBench, are limited to MCQ.

4. QA Resources:

   • Our QA resources include the video script and summary, providing additional context for question-answering.

5. Challenging Capabilities:

   • While most benchmarks, including LVBench and LongVideoBench, focus on visual understanding, our benchmark evaluates combined subtitle + visual understanding, making it more challenging.

Conclusion:

As shown in Table 1 of the revised paper, our benchmark surpasses MoVQA and other recent benchmarks in terms of scale, diversity, and the breadth of evaluation. We believe these advancements contribute significantly to the development of long-video understanding models.

---

Q2: The writing and paper organization needs refinement.

Thank you for highlighting this issue. We have carefully revised and polished the writing and organization of the paper to ensure clarity and improve readability. Please refer to the revised version for the updated changes.

Dear Reviewer iJs3,

We kindly ask if our response has addressed your concerns. Fortunately, we still have till December 3rd to discuss. Therefore, please feel free to share any additional questions or feedback, and we’ll be happy to provide further clarification.

Best regards, The Authors

---

Q8 (Continued)

Using 250 frames per video is not applicable for LLava-OV and InternVL, as they are standard video models not designed to handle too-long videos. Accordingly, we hit the maximum context length for these models, which prevents us from running on higher sampling rates.

---

Q9: Could authors explain how spoiler questions are generated and provide the prompt used?

The spoiler questions in our benchmark are not generated by GPT-4o or any other AI model. Instead, they are sourced from the IMDB website, where all questions and answers are created by humans.

Process for Spoiler Questions:

1. We scraped spoiler-related questions from IMDB, which are manually written and answered by human contributors.

2. To ensure quality and completeness, we filtered out any unanswered questions, keeping only those with valid human-provided answers.

3. We verify 10% of them and the human verification accuracy is 95.34%

This approach guarantees that the spoiler questions are realistic and reflect genuine human reasoning.

---

Q10: How can a model’s performance be lower than random?

We appreciate the reviewer’s insightful question. While it might seem counterintuitive, there are valid reasons why a model's performance can fall below random chance in a multiple-choice question (MCQ) setting.

Key Reasons:

1. Overfitting or Bias Towards Incorrect Patterns:

• Some models may overfit to spurious correlations or patterns in the training data, leading to systematic biases in their predictions.

• Instead of distributing predictions randomly across all answer choices, the model might consistently select incorrect answers due to these biases, resulting in performance worse than random.

2. Instruction-Following Issues:

• In some cases, the model might fail to correctly follow the format of the question or ignore the instruction to choose from the given options.

• This behavior can result in answers that are invalid or unrelated to the choices, further reducing accuracy.

---

Q11: For the human verification, how were human responses on open-ended questions evaluated?

For human verification of open-ended questions, we asked annotators to evaluate the model's responses on a five-point scale based on their correctness level relative to the ground truth (GT), similar to the evaluation process used by GPT-4o.

This approach ensures a consistent and fair evaluation of open-ended responses by aligning human judgments with the predefined GT criteria.

---

Q12: Do authors have evidence to the quality of TVQA annotations and summaries obtained from the web?

1. TVQA Annotations:

• The quality of TVQA questions and answers is well-documented in the TVQA paper (Section 3.2, Table 8).

• It explicitly mentions that “The negative answers in TVQA are written by human annotators. They are instructed to write false but relevant answers to make the negatives challenging.”

• This demonstrates that the questions and options are carefully crafted and verified by humans, ensuring their quality and relevance.

2. IMDB Summaries:

• According to IMDB’s contribution guidelines source: Contributors must follow strict instructions when submitting plot summaries.

• Each contribution is reviewed and approved by the IMDB team before publication.

• This rigorous process ensures that IMDB summaries are reliable, accurate, and already human-verified.

---

Q8 (Continued)

---

Q7: Why are multiple-choice questions evaluated by GPT not exact matching?

We appreciate the reviewer’s observation. The primary reason for using GPT-4o for evaluation instead of exact matching is to address cases where models do not strictly follow instructions and fail to output the selected choice directly.

Why GPT-4o Evaluation?

• The flexibility of GPT-4o allows us to focus on assessing the specific skill being tested rather than penalizing models for instruction-following errors.

• This approach ensures that the evaluation emphasizes the core reasoning or understanding ability of the model, rather than its adherence to formatting requirements.

Comparison of Evaluation Methods

To assess the impact of these two evaluation strategies—exact matching (restrictive) and GPT-4o evaluation (flexible)—we conducted experiments on two models: GPT-4o and MiniGPT4_video.

Findings:

• GPT: Performs equally well under both exact matching and flexible evaluation. GPT strictly follows the instructions and outputs the exact choices, showing no advantage from flexibility.

• MiniGPT4_video: Benefits significantly from the flexibility of GPT-4o evaluation. This model sometimes deviates from the instruction format but still demonstrates reasonable understanding of the question. Flexible evaluation allows us to capture its actual performance more effectively.

Conclusion and Future Reporting

• Inspired by this discussion, we will report results using both evaluation methods:

1. Exact Matching: A restrictive approach that directly evaluates instruction adherence.

2.GPT-4o Evaluation: A flexible approach that focuses on the skill being tested.

This dual reporting will provide a more comprehensive understanding of model performance, balancing strict evaluation with flexibility where appropriate.

This ablation only for 20 % of the benchmark

---

Q8: How does the number of video frames provided affect the model accuracy?

1. Influence of Input Frame Rate:

• Feeding more frames intuitively improves accuracy, but the degree of improvement varies across models.

• For instance, GPT-4o benefits the most from higher frame rates, while LLaVA-OV’s performance remains almost unchanged despite using an 8x higher frame rate.

2. Analysis of LLaVA-OV’s Behavior:

• The limited benefit of higher frame rates for LLaVA-OV may be attributed to its training strategy.

• LLaVA-OV is trained jointly on single images, multi-images, and videos.

• This strategy employs a balanced visual representation approach, aggressively downsampling video inputs to ensure parity with image-based scenarios.

• While effective for general tasks, this aggressive downsampling likely hurts LLaVA-OV’s ability to understand long videos, limiting its benefit from higher frame rates.

3. Skill-Specific Insights:

• Specific skills benefit more from higher frame rates. For example, the ``local vision+text'' skill improves most as it relies on short sequential shots.
• Increasing the frame rate reduces the chance of missing critical shots related to the answer, thereby boosting accuracy for such tasks.

The results demonstrate that while higher frame rates generally improve performance, the degree of improvement depends on the model’s design and training strategy. Models like GPT-4o, optimized for sequential inputs, show significant gains, whereas models like LLaVA-OV, which aggressively downsample videos, see minimal benefits.

---

Q5: [Copyrights] The copyright implications of using movies and TV shows and possibly releasing the dataset are not discussed and raise ethical concerns.

We appreciate the reviewer’s concern about copyright implications and ethical considerations related to using movies and TV shows in our dataset. Below, we clarify our approach and steps to ensure compliance with legal and ethical standards:

1. Use of Existing Datasets (e.g., MovieNet and TVQA):

   • We rely on publicly available datasets like MovieNet and TVQA, which provide downsampled versions of video content.

   • Both datasets are widely used in academic research and distribute videos in a legally compliant manner by heavily reducing the frame rate (e.g., 1 FPS). This downsampling transforms the videos into low-resolution derivatives primarily for research purposes, which may mitigate copyright concerns.

   • While we cannot definitively state that downsampling resolves all legal and ethical issues, its widespread use in academia suggests it is generally accepted within the research community.

2. Benchmark Without Redistribution of Videos:

   • Our benchmark does not redistribute video content directly. Instead, we provide annotations, question-answer pairs, and evaluation scripts. Users are required to independently download the videos from official sources (e.g., MovieNet or TVQA).

   • This ensures we do not claim ownership of the original video content, nor do we host or distribute it ourselves.

   • This approach aligns with practices used in existing benchmarks, where users are required to obtain the original images independently.

3. Ethical Considerations:

   • We acknowledge that relying on copyrighted material, even in downsampled form, can raise ethical questions.

   • We are committed to transparency and ensuring our benchmark is used responsibly. To that end, we will include a clear disclaimer stating:

4. The benchmark does not redistribute or modify original video content.

5. Users must adhere to the terms and conditions of the original video sources.

Conclusion:

While we cannot claim definitive resolution of all legal or ethical concerns, our approach of using downsampled versions from widely accepted datasets, combined with ensuring users obtain the videos independently, reduces our direct responsibility for copyright compliance. If further legal concerns arise, we will explore ways to refine our approach in collaboration with legal experts and ensure the benchmark remains compliant and ethical.

---

Q6: [Duplicates] Since the dataset has around 100 questions per video, it is likely that there are (near) duplicate questions.

We appreciate the reviewer’s suggestion and agree that addressing potential duplicates is an important aspect of the benchmark creation process.

[Deduplication Methodology]

To ensure the dataset is free from (near) duplicate questions, we implemented the following approach:

1. Encoding and Similarity Calculation:

   • We used M3-Embedding [1] to encode the questions and answer choices into vector representations.

   • Cosine similarity was then calculated to identify potential duplicates.

2. Thresholds for Evaluation:

   • To account for varying degrees of similarity, we evaluated three thresholds: 90%, 95%, and 98% cosine similarity.

3. Findings:

   • As shown in Figure X of the revised paper, the vast majority of questions across all skills are unique, with no duplicates detected.

   • Two skills, Temporal Questions and Character Actions, showed instances of potential duplicates.

[Analysis of Detected Duplicates]

Upon investigation, we found that the detected duplicates were false positives. For example, the following pair of questions was flagged as duplicates because they differ only by the words "before" and "after". However, this small difference completely changes the meaning of the question:

1. Q1: Did the event flashback to Phoebe completing a mile on a hippity-hop before turning thirty, happen before the event Monica makes breakfast with chocolate-chip pancakes?

2. Q2: Did the event flashback to Phoebe completing a mile on a hippity-hop before turning thirty, happen after the event Monica makes breakfast with chocolate-chip pancakes?

These examples highlight the importance of semantic context in evaluating question similarity, as minor lexical differences can significantly alter meaning.

Conclusion

Based on our analysis, we are confident that the dataset is free from true duplicates, and our pipeline effectively identifies and handles potential near-duplicates. The false positives flagged by the similarity detection process underscore the complexity of semantic evaluation, especially in nuanced question construction.

[1] Multi-Granularity, Multi-Linguality Multi-Functionality. "M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation."

---

Q4: [Multi-Modal Benchmark] It is unclear how much the benchmark relies on multimodal reasoning. It would be interesting to see an ablation that uses (1) No context, only the question itself (2) Only the question and subtitles (3) the question, subtitles and video frames.

To evaluate the contribution of multimodal reasoning, we conducted experiments on Qwen using four input variants:

1. Video + Subtitle
2. Video
3. Question + Subtitle
4. Question

Findings:

1. [Visual Clues]

   • Dropping subtitles did not significantly impact performance, demonstrating that our benchmark focuses on visual cues and that textual information alone is insufficient to answer most questions.

2. [Blind Models]

   • To assess the importance of visual inputs, we tested models using subtitles alone (termed Blind Models) without video frames.

   • Results showed that blind models achieved near-random performance, emphasizing the critical role of visual inputs in answering questions.

3. [Blind and Deaf Models]

   • Similarly, removing both video frames and subtitles (termed Blind and Deaf Models) resulted in random performance.

   • This further highlights the significant role of visual inputs, with textual inputs alone contributing minimally, as evidenced by the similarity in performance between blind and blind-and-deaf models.

Tables summarizing these findings have been included in the revised paper.

---

Q3: [Data Leakage] Most MLLMs likely know the plots of popular movies and shows because their summaries or transcripts were part of their training data. So, they may be able to answer the questions in the dataset without any context

[Blindness Experiment]

To genuinely assess the data leakage, we deliberately dropped the video and only fed the question and some context about the episode or the movie without any visual inputs.

For instance, here is the input prompt in the blindness case:

`` This is a question for a video from {show} {season_num} {episode_num}, use your knowledge to answer this question: {question} ''

We have conducted the blindness experiments using two models, Qwen and GPT-4o. As shown in the tables below, in most skills, the blind models' performance is too close to the random performance.

For instance, on the "global appearance" and the "scene transitions" skills, Qwen achieves 19.6 and 21, while GPT-4o achieves 20.8 and 22.5, approximately equal to the random performance of 17 for both skills.

[Data Leakage vs. Common Sense]

In contrast, only the blind Qwen on one skill, the ``Character Actions'', achieves closer performance than the Qwen, which takes the visual input, 36.6 and 36, respectively. This could be interpreted as the model using its common sense to answer the question. The choices in this skill contain valid actions, and only their order is wrong. Thus, we argue that the model could perform well using common sense to order the events.

To test our hypothesis, we assess the model performance on this skill as an open-ended question without choices. We leverage GPT-4o to score the models' outputs out of 5, where 0 is the worst and five is the best. The detailed prompt used while scoring is depicted in Figure 7. As expected, when we remove the visual input, the accuracy drops significantly from 0.79 to 0.003 as shown in the table below.

---

Q1: A human evaluation is performed on a subset of the data, but good human performance is no proof that questions are well-formed and free of hallucinations.

We appreciate the reviewer’s concern and would like to clarify the steps we have taken to ensure the quality and validity of our benchmark:

[Human Verification of 10% of the Data]

We conducted a human verification process on 10% of the dataset (10.8k questions) to assess the dataset's quality. By "dataset quality," we evaluate:

By saying dataset quality we mean:

1. Validity of the Questions: Ensuring that the questions are relevant to the video.

2. Correctness of the Answers: Verifying whether the answers provided for valid questions are accurate.

3. Plausibility of Question-Answer Pairs: Checking if the format and phrasing of the pairs are clear and free of vagueness.

This verification process ensures a holistic assessment of the benchmark’s reliability.

Results:

• On average, 95.8% of the questions were deemed correct and valid.

• A breakdown of accuracy per skill is provided below:

We are continuously working to verify and correct the rest of the dataset and estimate that full verification will require approximately 4249 human hours.

[Human Verification of 100% of the Data]

The full dataset consists of:

• 923 episodes, each requiring 3 hours on average for verification.

• 296 movies, each taking approximately 5 hours to verify.

To expedite this process, we have partnered with a data annotation company with ten full-time annotators working on the benchmark. Based on this setup, we estimate that the entire dataset will be fully verified and corrected within 20 working days.

We believe these efforts demonstrate our commitment to ensuring the benchmark is free of hallucinations and robust enough for long-video understanding tasks.

---

Q2: [Transcript Details] Could authors provide evidence of transcript quality? How accurate and complete are they? How much focus do they have on vision? Could authors provide examples?

1. Role of Transcripts:

• Transcripts are detailed documents created by movie or TV show writers. They provide comprehensive information beyond spoken dialogue, including:

1. Scene descriptions.
2. Context about settings, locations, and character actions.
3. Camera angles or shot compositions.

• Transcripts serve as blueprints for visual and narrative elements, helping us extract visual insights and design challenging, reliable benchmark questions.

2. Transcript Example:

Transcript of episode 1 season 1 of Friends TV shows:
**[Scene: Central Perk, Chandler, Joey, Phoebe, and Monica are there.]**
Monica: There's nothing to tell! He's just some guy I work with!
Joey: C'mon, you're going out with the guy! There's gotta be something wrong with him!
...
**(Ross gestures his consent.)**
Joey: Strip joint! C'mon, you're single! Have some hormones!
Ross: I don't want to be single, okay? I just... I just- I just wanna be married again!
**(Rachel enters in a wet wedding dress and starts to search the room.)**

As shown in the above example the transcript contains a lot of visual clues such as:

• Rachel enters in a wet wedding dress and starts to search the room.

• Scene: Central Perk, Chandler, Joey, Phoebe, and Monica are there.

• Ross gestures his consent.

3. Subtitle Example:

In contrast, the subtitle does not provide any visual clues therefore, we use it only during testing only.

Subtitle of the same episode
1
00:00:54,012 --> 00:00:57,641
There's nothing to tell.
It's just some guy I work with.

2
00:00:57,892 --> 00:00:59,962
You're going out with a guy.
There's gotta be something wrong with him.

By distinguishing between the roles of transcripts and subtitles, we ensure that the benchmark tests true long-video understanding during inference while using transcripts only for reliable benchmark construction.

Dear Reviewer oZkF,

We kindly ask if our response has addressed your concerns. Fortunately, we still have till December 3rd to discuss. Therefore, please feel free to share any additional questions or feedback, and we’ll be happy to provide further clarification.

Best regards, The Authors

Q3: (Continued)

After adding the new methods, a complete leaderboard can be seen in Table 5 of the paper.

---

Q4: In Table 1, the benchmark comparison is insufficient, especially regarding recent video benchmarks such as Video-MME and LongVideoBench. Additionally, the author's definition of "very long" is problematic - MLVU and MovieChat have only a 3-minute gap, yet MLVU is defined as very long. This is not reasonable.

We appreciate the reviewer's feedback and agree that our previous taxonomy, based on a hard threshold (e.g., 10 minutes), is not robust.

[Updated Taxonomy]

Inspired by this discussion, we have adopted a more adaptive and reliable categorization method using K-means clustering:

• We set $k=3$ to categorize benchmarks into three groups based on average video length.

• This method adaptively determines the categories without relying on arbitrary thresholds.

The updated categorization addresses inconsistencies in defining "very long" benchmarks and ensures a fair comparison.

[Expanded Comparison]

Additionally, we have included comparisons with recent benchmarks, such as Video-MME and LongVideoBench, in Table 1 of the revised paper.

We believe these changes improve the reliability and comprehensiveness of our benchmark comparison. Thank you for raising this important point!

---

Q5: How is GPT-4V's scoring aligned with human evaluation?

We conducted a human evaluation on 10% of the dataset to assess the alignment between GPT-4o's scoring system and human preferences.

Evaluation Methodology:

• We designed a simple GUI that displayed responses from two models side by side for each question.

• Annotators were asked to select which model provided the better response based on quality and relevance.

• We then measured the Pearson correlation between human preferences and GPT-4o's scoring or ranking systems.

Results:

• The correlation between human preferences and GPT-4o's scoring system was 96%, indicating that GPT-4o's scoring is highly reliable and closely aligned with human judgment.

These results validate the robustness of GPT-4o's scoring system as a reliable evaluation metric.

---

Q3: InfiniBench's testing does not cover current mainstream open-source models such as Qwen2VL, LLaVA-Onevision, and InternVL2

We have included three new recent long video models:

1. Qwen2VL
2. InternVL
3. LLava OV

Due to time constraints, these experiments were conducted on 20% of the dataset.

---

Q1: The benchmark only uses movies and TV shows, which is too limited. They should add more casual videos like vlogs and livestreams to make the testing more realistic.

We appreciate the reviewer's suggestion and agree that including a wider variety of video sources, such as vlogs or live streams, could add value to future benchmarks. However, we argue that movies and TV shows are highly suitable and effective for assessing long-video understanding for the following reasons:

1. Diverse and Complex Contexts:

• Movies and TV shows are rich in content, featuring intricate character relationships, evolving storylines, and multi-layered themes. These elements introduce dynamic and complex reasoning challenges beyond the repetitive or monotonic scenarios often found in daily-life videos like vlogs or live streams.

• For instance, movies often include unexpected actions, rapid shifts in context, and non-linear narratives that demand a higher level of understanding, making them ideal for testing models' ability to handle long-term dependencies and reasoning tasks.

2. Variety of Skills Tested:

• Our benchmark is designed to include diverse questions that evaluate different levels of video understanding, from surface-level observations to deeper reasoning about characters, events, and causality.

• This holistic design ensures that the benchmark challenges models on a wide range of skills, as demonstrated by the low performance of current state-of-the-art models.

3. Limitations of Casual Videos:

• Vlogs and daily-life videos often revolve around singular, straightforward activities (e.g., riding a bicycle or walking through a park). These scenarios typically lack the nuanced interplay between characters and the layered storytelling in cinematic content.

• While casual videos may help test immediate perception tasks, they are less suited for evaluating the reasoning and complex relational understanding required for proper long-video comprehension.

4. Storytelling Patterns:

• While movies and TV shows follow storytelling patterns, these patterns are not uniform and vary greatly across genres, cultures, and creators. This inherent variability further enriches the benchmark by introducing a broad spectrum of reasoning and comprehension challenges.

Acknowledgment and Future Work:

We acknowledge that including additional categories of videos, such as vlogs, live streams, or documentaries, could further diversify the benchmark and make it more comprehensive. We plan to explore incorporating these sources in an extended version of the work or future studies.

---

Q2: Without scripts or captions, the benchmark can't test how well AI models understand regular videos people watch and share online.

We appreciate the reviewer’s observation and would like to clarify an important distinction regarding the use of transcripts and subtitles in our benchmark:

[Transcript vs. Subtitles]

1. Role of Transcripts:

   • Transcripts are detailed documents created by movie or TV show writers. They provide comprehensive information beyond spoken dialogue, including:

     1. Scene descriptions.
     2. Context about settings, locations, and character actions.
     3. Camera angles or shot compositions.

   • Transcripts serve as blueprints for visual and narrative elements, helping us extract visual insights and design challenging, reliable benchmark questions.

   • Key Point: Transcripts are used only during the benchmark creation process to ensure robustness and question diversity, not during inference or evaluation.

2. Role of Subtitles:

   • Subtitles focus solely on translating spoken dialogue into text, typically extracted by transcribing the video's audio.

   • Subtitles are optional inputs for the AI model during inference, representing an additional modality when available.

3. Clarifying the Inputs:

   • During inference, the model's input consists of video frames and, optionally, subtitles, as shown in Table 3.

• Transcript Example:

Transcript of episode 1, season 1 of Friends TV shows:
[Scene: Central Perk, Chandler, Joey, Phoebe, and Monica are there.]
Monica: There's nothing to tell! He's just some guy I work with!
...
(Ross gestures his consent.)
Joey: Strip joint! C'mon, you're single! Have some hormones!
(Rachel enters in a wet wedding dress and starts to search the room.)

• Subtitle Example:

Subtitle of the same episode
1
00:00:54,012 --> 00:00:57,641
There's nothing to tell.
It's just some guy I work with.

2
00:00:57,892 --> 00:00:59,962
There's gotta be something wrong with him.

By distinguishing between the roles of transcripts and subtitles, we ensure that the benchmark tests true long-video understanding during inference while using transcripts only for reliable benchmark construction.

Dear Reviewer 147D,

We kindly ask if our response has addressed your concerns. Fortunately, we still have till December 3rd to discuss. Therefore, please feel free to share any additional questions or feedback, and we’ll be happy to provide further clarification.

Best regards, The Authors

---

Q1: The paper lacks discussion of related work, e.g., Video-MME, LVBench, and Long VideoBench.

We have expanded our evaluation to include other recent benchmarks, such as Video-MME, LVBench, and LongVideoBench.

Here are some key highlights:

1. Video Length:

• Most benchmarks focus on short videos, with an average length of around 10 minutes.

• The exception is LVBench, which includes 1-hour-long videos, making it comparable in duration to our dataset.

2. Scale:

• Our benchmark is 70× larger than LVBench.

3. QA Types:

• We support both MCQ and open-ended QA, while other long-video benchmarks, such as LVBench and LongVideoBench, are limited to MCQ.

4. QA Resources:

• Our QA resources include the video script and summary, providing additional context for question-answering.

5. Challenging Capabilities:

• While most benchmarks, including LVBench and LongVideoBench, focus on visual understanding, our benchmark evaluates combined subtitle + visual understanding, making it more challenging.

Conclusion:

As shown in Table 1 of the revised paper, our benchmark surpasses the recent benchmarks in scale, diversity, and breadth of evaluation. We believe these advancements contribute significantly to the development of long-video understanding models.

---

Q2: [The quality of the dataset] Most of the question-answer pairs are generated by GPT-4o. Although multiple information sources were used as input, it's difficult to guarantee the dataset's quality.

[Human Verification of 10%]

We have conducted a human verification of our benchmark for 10% (10.8k questions) to verify the dataset's quality. The verification of the 10% of the data takes around 400 human hours. The results show the average of correct questions is (95.8), and humanly correct the rest of the dataset. So the final humanly verified set is 100% accurate. The remaining set is now considered as weak labels with 96% expected accuracy which we believe can be a valuable resource for training. We are expanding the human verification process to cover the whole set which is expected to take around 4249 human hours.

The detailed accuracy per skill is reported in the table below:

[Human Verification of 100%]

We have 923 episodes, whereas one requires 3 hours on average for verification. In addition, we have 296 movies, each taking around five hours to be verified. We have a contract with a data annotation company, with 10 annotators working full-time on our benchmark. Therefore, all the data should be verified and ready in 20 working days.

---

Q3: [Dataset Leakage] Part of the data comes from IMDB content, which likely appeared multiple times in the training corpus of LLMs used by video models, potentially leading to dataset leakage issues.

[Blindness Experiment]

To genuinely assess the data leakage, we deliberately drop the video and only feed the question and some context about the episode or the movie without any visual inputs.

For instance, here is the input prompt in the blindness case:

`` This is a question for a video from {show} {season_num} {episode_num}, use your knowledge to answer this question: {question} ''

We have conducted the blindness experiments using two models, Qwen and GPT-4o. As shown in the tables below, in most skills, the blind models' performance is too close to the random performance.

For instance, on the "global appearance" and the "scene transitions" skills, Qwen achieves 19.6 and 21, while GPT-4o achieves 20.8 and 22.5, approximately equal to the random performance of 17 for both skills.

---

Q3: (Continued)

[Data Leakage vs. Common Sense]

In contrast, only the blind Qwen on one skill, the ``Character Actions'', achieves closer performance than the Qwen, which takes the visual input, 36.6 and 36, respectively. This could be interpreted as the model using its common sense to answer the question. The choices in this skill contain valid actions, and only their order is wrong. Thus, we argue that the model could perform well using common sense to order the events.

To test our hypothesis, we assess the model performance on this skill as an open-ended question without choices. We leverage GPT-4o to score the models' outputs out of 5, where 0 is the worst and 5 is the best. The detailed prompt used while scoring is depicted in Figure 7. As expected, when we remove the visual input, the accuracy drops significantly from 0.79 to 0.003 as shown in the table below.

---

Q4. Evaluate more long-video models (e.g., Qwen2VL) and different input frame rates (1, 8, 32, 128, and more).

We have included three new recent long video models:

1. Qwen2VL
2. InternVL
3. LLava OV

In addition, we tested the best-performing model on our benchmark, GPT-4o, with different input frame rates. Due to time constraints, these experiments were conducted on 20% of the dataset.

[Findings]

1. Influence of Input Frame Rate:

• Feeding more frames intuitively improves accuracy, but the degree of improvement varies across models.
• For instance, GPT-4o benefits the most from higher frame rates, while LLaVA-OV’s performance remains almost unchanged despite using an 8x higher frame rate.

2. Analysis of LLaVA-OV’s Behavior:

• The limited benefit of higher frame rates for LLaVA-OV may be attributed to its training strategy.
• LLaVA-OV is trained jointly on single images, multi-images, and videos.
• This strategy employs a balanced visual representation approach, aggressively downsampling video inputs to ensure parity with image-based scenarios.
• While effective for general tasks, this aggressive downsampling likely hurts LLaVA-OV’s ability to understand long videos, limiting its benefit from higher frame rates.

3. Skill-Specific Insights:

• Specific skills benefit more from higher frame rates. For example, the ``local vision+text'' skill improves most as it relies on short sequential shots.
• Increasing the frame rate reduces the chance of missing critical shots related to the answer, thereby boosting accuracy for such tasks.

The results demonstrate that while higher frame rates generally improve performance, the degree of improvement depends on the model’s design and training strategy. Models like GPT-4o, optimized for sequential inputs, show significant gains, whereas models like LLaVA-OV, which aggressively downsample videos, see minimal benefits.

Q4: (Continued)

Using 250 frames per video is not applicable for LLava-OV and InternVL, as they are standard video models not designed to handle too-long videos. Accordingly, we hit the maximum context length for these models, which prevents us from running on higher sampling rates.

---

Q5: Since most question-answer pairs are generated by GPT-4o, could this lead to inflated evaluation results for GPT-4o?

We appreciate the reviewer’s suggestion and share the concern about ensuring a reliable and unbiased evaluation. To address this point, we conducted the following experiments:

To address this point, we conducted several experiments:

1. [Reliable Benchmark]

• We performed a human evaluation of the generated question-answer pairs.
• The results show that the generated pairs align with human annotations by more than 95%, demonstrating that the benchmark is sufficiently reliable.

2. [Poor Performance]

• Despite GPT-4o being used to generate the question-answer pairs, its evaluation performance is far from acceptable.
• For example, it achieves only 35% accuracy on the "Character Actions" skill, while random performance is 16%. This indicates that the evaluation is not artificially inflated.

3. [Input Richness]

• An important question arises: If GPT-4o can generate reliable, human-like question-answer pairs, why does it perform poorly on these questions?
• The answer lies in the difference between the inputs used during data creation and evaluation:
• During data creation, we provide GPT-4o with the transcript and the video summary, which contain rich visual and contextual information, including semantics, event context, character personalities, and other specifics.
• To assess long-video understanding capabilities during the evaluation, we omit the transcript and the summary and provide only the video input.
• This discrepancy in input richness justifies the gap between GPT-4o’s performance during testing and the benchmark creation.

We hope these points clarify why the evaluation results are not inflated and further justify the reliability of the benchmark and experimental setup.

Q4: (Continued)

Dear Reviewer 6qyh,

We kindly ask if our response has addressed your concerns. Fortunately, we still have till December 3rd to discuss. Therefore, please feel free to share any additional questions or feedback, and we’ll be happy to provide further clarification.

Best regards, The Authors