Can LVLMs Describe Videos like Humans? A Five-in-One Video Annotations Benchmark for Better Human-Machine Comparison

Q11. In the qualitative results shown in Figures A12–A17, it would be helpful to highlight any hallucinated content in the model responses.

Response: We sincerely thank the reviewer for the valuable suggestion. In the revised Appendix E.4 of the paper, we have added more analyses, using different colors to annotate the types of errors made by the models. Specifically, we categorized the following five common types of errors:

• Omission: The model fails to describe key events or objects in the video. While this cannot be directly marked in the model's output, we have provided textual analyses of such omissions after the relevant examples.
• Misrepresentation: The description contains information inconsistent with the video content. These errors are marked in purple in the model outputs.
• Redundancy: The model repeats descriptions of the same event. These errors are marked in yellow in the outputs.
• Excessive Redundancy: The model overextends or speculates excessively, introducing unnecessary content. These errors are marked in green in the outputs.
• Hallucination Issues: The model includes content that is not present in the video. These errors are marked in red in the outputs.

By incorporating detailed error analyses and exploring potential causes, we aim to further enhance the value of FIOVA in evaluating LVLM performance and provide additional insights for model optimization. We appreciate the reviewer’s suggestion, which helps us present our research more comprehensively and explore broader research directions.

Q10. In Table A.3 in the Appendix, comparing the model's response with the GPT-synthesized human caption summary tends to produce higher metric values than directly comparing the model's response with a single human caption. Do the authors have any comments or insights regarding this observation?

Response: We sincerely thank the reviewer for their valuable observation. Below is our detailed analysis and explanation, we have added it in Appendix E.1:

1. GPT-Summarized Descriptions Improve Information Coverage

• Integration of Multi-Perspective Information: Each video in the dataset was annotated by five independent annotators who watched the entire video before providing detailed descriptions. Due to differing observational focuses, each annotator's description may highlight different aspects, such as:

  • One annotator may focus on the details of characters' actions;
  • Another annotator may emphasize the environment or background.

GPT-3.5-turbo aggregates these descriptions, integrating the multi-perspective information from the five annotators into a more comprehensive and diverse GT description.

• Broader Content Coverage: Compared to single annotator descriptions, GPT-summarized descriptions provide a more extensive representation of video content. For instance, in the case shown in Figure 4 (further analyzed in Appendix B.6), some annotators may focus on the actions of the child, while others may note background or secondary details. Through aggregation, these elements are comprehensively represented in the GT.

2. Reasons for Higher Metric Scores

• Higher Consistency Between Model Output and Aggregated GT: When compared with the aggregated GT, model outputs are more likely to align with certain annotators' observations, leading to higher metric scores (e.g., BLEU, METEOR). This is because:
  • Model outputs may randomly cover content described by multiple annotators;
  • The aggregated GT includes richer content details, reducing the likelihood of missing information.
• Limitations of Single Annotator Descriptions: Single annotator descriptions may only cover certain aspects of the video content. When compared with model outputs, these descriptions may suggest that the model has omitted certain details, leading to relatively lower scores.

3. Rationale for Using Aggregated GT

Using GPT-summarized descriptions as GT is both reasonable and necessary, for the following reasons:

• Improved Comprehensiveness: Aggregated GT maximally retains the diverse perspectives of the five annotators, providing a more complete representation of the video content;
• Reflection of Multi-Perspective Features: This aligns with FIOVA's core philosophy of evaluating models' semantic understanding through multi-perspective information;
• Enhanced Fairness in Evaluation: Multi-perspective GT avoids the biases that may result from individual annotators' preferences, enabling a more comprehensive evaluation of model performance.

Q9. It appears that the comparison of an LVLM against humans is conducted by comparing the LVLM’s response with the GPT-synthesized human caption summary in the main paper. However, in the image-language domain, having multiple human annotators for each image is not new (e.g., COCO captions include 5 human captions per image), and researchers often compare a model’s caption with each human caption individually, then aggregate the metric values (e.g., averaging the 5 metric values obtained by comparing with each human caption). Is there a reason for not taking this approach?

Response: We sincerely thank the reviewer for raising this very important question. Based on the reviewer's suggestion and considering the characteristics of FIOVA (each video is annotated by five human annotators), we have optimized the AutoDQ metric and proposed the FIOVA-DQ metric (see Common Concern 2: Propose a New Evaluation Method FIOVA-DQ). Its computational approach aligns with the key points raised by the reviewer. Below, we elaborate on the reasons behind our prior choice of using a synthesized GT for evaluation and provide an introduction to FIOVA-DQ to better address the reviewer's suggestion.

1. Reasons for Merging Multiple Annotators’ Descriptions

• Integration of Multi-Perspective Information: Each annotator may focus on different aspects of the video content (e.g., action details, background information, or event logic). By merging the descriptions from five annotators, the GT captures a broader semantic representation of the video, reducing the risk of missing details inherent in single annotator descriptions.
• High-Quality Reference Standard: Compared to using the description from a single annotator, the merged GT more accurately reflects the overall human understanding of video content, providing a higher-quality baseline for evaluating LVLMs.

2. Experimental Support for Individual Comparisons

As suggested by the reviewer, we conducted experiments in Appendices E.1 and E.2 that adopt the approach of individual comparisons. In these experiments, the LVLM-generated outputs are compared with each annotator’s description individually, and the results are then aggregated. This method offers additional insights and demonstrates the differences between individual comparison and merged description evaluations.

3. Optimization and Improvements with FIOVA-DQ

In the revised version of our paper, we have taken the reviewer’s suggestion into account and further optimized the AutoDQ method, proposing a new evaluation metric, FIOVA-DQ. The FIOVA-DQ metric is conceptually aligned with the idea of individual comparisons, as it incorporates the following features:

• Event Weighting Mechanism: During evaluation, FIOVA-DQ assigns weights to events based on their importance, as determined by the descriptions from all five annotators, thereby integrating the perspectives of multiple annotators into the evaluation process.
• Closer Alignment with Human Judgment: Compared to traditional metrics and the original AutoDQ, FIOVA-DQ better captures event importance and semantic consistency, producing evaluation results that are more aligned with human intuitions regarding description quality (see revised paper Sec 3.2 and Sec 4.1).

In conclusion, we believe that merging the descriptions from multiple annotators to construct the GT is an effective approach. At the same time, we have explored the method of individual comparisons in our experiments and incorporated the strengths of both approaches into FIOVA-DQ. This ensures a more comprehensive and robust evaluation framework.

Q8. AutoCQ only provides event-level evaluation. Why were the five dimensions (e.g., consistency, context, etc.) from Sec. 2.2 not considered in the model response evaluation metrics?

Response: We sincerely thank the reviewer for raising this important question. The five dimensions introduced in Sec. 2.2 (Consistency, Context, Correctness, Detail Orientation, Temporality) were specifically designed using GPT to assess the relative differences among human annotators' descriptions, rather than to serve as absolute metrics for model evaluation. Therefore, we did not incorporate them as direct metrics in the subsequent evaluation of model performance. Below are detailed explanations:

1. Core Objectives and Application Scenarios of the Five Dimensions

The core goal of these five dimensions is to measure the semantic differences among human annotators, rather than to judge the absolute quality of the descriptions. For example, a video may have descriptions with differing focuses (e.g., one focusing on background details and another on actions). Such variations reflect the natural diversity of human cognition and should not be viewed as strictly better or worse. To this end, we calculate the coefficient of variation (CV) for each dimension to quantify the relative consistency of descriptions. Importantly, CV emphasizes the overall distribution of descriptions rather than the absolute accuracy of individual scores. This calculation provides the foundation for grouping videos into categories (Groups A-H). Thus, these five dimensions are more suitable for relative evaluation rather than absolute performance assessment.

2. Why Were the Five Dimensions Not Directly Used for Model Evaluation?

Although these five dimensions effectively capture the variability among human annotations, there are limitations to applying them directly to model evaluation:

• Relative vs. Absolute Evaluation: The five dimensions are designed to assess the relative differences in human annotations, whereas model evaluation focuses on the absolute semantic quality of generated descriptions. For example, in measuring Consistency, a higher CV indicates significant variability in human annotators' styles; however, for models, we require explicit metrics to determine the quality of the generated descriptions.
• Suitability: Model evaluation typically relies on established absolute metrics (e.g., BLEU, METEOR) and event-level metrics (e.g., AutoDQ and FIOVA-DQ). These metrics are better suited for directly measuring semantic consistency, event coverage, and temporal coherence in model-generated descriptions.

3. Embedding Relative Evaluation Principles into the Framework

While the five dimensions were not directly used for model evaluation, we incorporated the relative evaluation principle into subsequent experiments. For instance, in the Batch Ranking experiment (Sec. 4.3 and Appendix C), we compared the consistency of descriptions between human annotators and model-generated outputs, thereby indirectly quantifying their relative differences.

In summary, the primary objective of the five dimensions is to capture the relative variability among human annotators, rather than to serve as absolute metrics for model performance. This design emphasizes the distribution and diversity of descriptions but is less suited for absolute semantic evaluation compared to traditional and event-level metrics (e.g., AutoDQ and FIOVA-DQ). By embedding the principles of relative evaluation into later experiments (e.g., Batch Ranking), we expanded their applicability. We sincerely thank the reviewer for their insightful feedback on this matter.

Q7. Did the human annotators watch the actual video files, or were they presented with sampled frames?

Response: We appreciate the reviewer’s attention to the annotation process. All videos in the FIOVA dataset were annotated by human annotators after watching the complete videos, ensuring both comprehensiveness and consistency in the annotations. Specifically:

1. Comprehensiveness

Annotators, by watching the full videos, could accurately capture all key events, dynamic relationships, and temporal sequences. This comprehensive annotation approach ensures that the core semantics of the video content are fully represented.

2. Consistency in Key Details

Viewing the entire video enabled annotators to understand complex spatiotemporal causalities and multi-agent interactions, resulting in semantically consistent and detail-rich descriptions. This approach effectively avoids semantic biases or missing details that might arise from fragmentary viewing.

3. Unified Training and Strict Guidelines

All annotators underwent standardized training to familiarize themselves with the annotation process and task objectives. They followed strict annotation guidelines, ensuring the quality and consistency of the descriptions and establishing the reliability of the dataset.

Through these measures, the annotation process for the FIOVA dataset not only guarantees multidimensional coverage of video content but also enhances the semantic reliability and applicability of the dataset.

Q6. When the SOTA models were evaluated, 8 frames per video were used. Were 8 frames sufficient for FIOVA?

Response: We appreciate the reviewer’s attention to the frame selection strategy. In our study, we chose to use 8 frames per video based on a careful consideration of balancing evaluation efficiency and information capture, which aligns with the standard experimental paradigm in the current field of video tasks. Below are the detailed explanations:

1. Consistency with Experimental Paradigm

The core design of FIOVA is to provide an open and high-quality evaluation benchmark for analyzing and comparing the long-video description capabilities of various LVLMs and human annotators. To build a reproducible and scalable evaluation platform, we adhered to standard paradigms widely adopted in the field, such as fixed-frame sampling. This consistency not only facilitates horizontal comparisons with existing works but also provides a reference framework for future studies.

2. Methodological Generality

Long-video tasks often involve processing a substantial amount of visual information. Selecting 8 frames as input effectively reduces computational cost while supporting large-scale experiments and benchmark construction. In fact, many related works (e.g., VideoGPT+ [1] and Emu-3 [2]) also adopt 8-frame inputs, highlighting the representativeness and generality of this setting in long-video understanding tasks. Moreover, current LVLMs often have frame count limitations; too many frames may overload the input capacity or degrade performance. The 8-frame setup accommodates the capabilities of mainstream models while avoiding information redundancy.

3. Fairness and Feasibility of the Evaluation Platform

In our study, all experimental results were derived from the 8-frame setting. This choice not only validated FIOVA’s evaluation capability under this configuration but also ensured fairness and feasibility within the evaluation platform. The selection of 8 frames strikes an optimal balance among semantic capture, experimental efficiency, and model constraints, making it a reasonable setting under the current standard experimental paradigm.

In the revised manuscript, we further clarify the experimental rationale behind the 8-frame setup and emphasize FIOVA’s flexibility and extensibility in Appendix D.2. By providing an open frame-setting option, future research can explore more sophisticated frame sampling strategies to further unlock the potential of models.

[1] Maaz M, Rasheed H, Khan S, et al. VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding[J]. arXiv preprint arXiv:2406.09418, 2024.

[2] Wang X, Zhang X, Luo Z, et al. Emu3: Next-token prediction is all you need[J]. arXiv preprint arXiv:2409.18869, 2024.

Q5. Line 306 states that each model was fine-tuned for video caption generation. Do you mean you further fine-tuned these models? On which dataset was the fine-tuning performed?

Response: Thanks for pointing this out. During the inference process, we did not fine-tune the LVLMs, and to ensure fairness in evaluation, we preserved the original configurations of the models as much as possible. We have revised this section in the updated manuscript to clarify this point and prevent any misunderstanding. We sincerely apologize for any confusion caused by the previous description.

Q4. How does the GPT-summarized caption affect the distributions shown in Figure 3?

Response: Thanks for raising this insightful question. We want to clarify that during the construction of the FIOVA benchmark, GPT-3.5-turbo was used twice in our study, and these two steps are entirely independent. There is no direct connection between the first step (illustrated in Fig. 3) and the second step (illustrated in Fig. 4). In other words, the distributions shown in Fig. 3 are solely based on GPT-3.5-turbo’s analysis of human annotations and are unrelated to the Groundtruth generation process. Below is a detailed explanation:

1. First Use of LLM: Analyzing Human Annotations for Grouping

• Context: This corresponds to Sec. 2.2: Caption Quality Assessment and Fig. 3.
• Objective: To quantify the consistency of human annotations based on semantic differences and calculate the Coefficient of Variation (CV) to group videos into categories A-H.
• Method: GPT-3.5-turbo analyzed the textual descriptions provided by five annotators for each video across five dimensions—contextual consistency, semantic coverage, linguistic fluency, as well as the length of descriptions. The CV for each dimension was calculated, and videos were grouped based on the resulting CV scores.

2. Second Use of LLM: Generating Groundtruth Descriptions

• Context: This corresponds to Sec. 2.3: Groundtruth Generation and Fig. 4.
• Objective: To generate a unified and comprehensive Groundtruth description by semantically integrating the multi-perspective descriptions provided by five annotators.
• Method: GPT-3.5-turbo combined the multi-perspective annotations from the five annotators, performing linguistic optimization and semantic integration to create a refined Groundtruth description.

To summarize, these two steps are entirely independent. The distributions in Fig. 3 reflect the semantic differences in human annotations analyzed by GPT-3.5-turbo. The purpose of this analysis was to evaluate the variability among annotators' descriptions across multiple dimensions (Consistency, Context, Correctness, etc.) and calculate CV scores for each video. These CV values measure the level of consistency in the human annotations and are used to group the videos.

On the other hand, Fig. 4 demonstrates how GPT-3.5-turbo generates Groundtruth descriptions through semantic integration. During this process, GPT-3.5-turbo does not introduce new modal information; it simply integrates and optimizes the language based on the textual descriptions provided by the five human annotators. This step primarily aims to create a unified Groundtruth reference to evaluate model performance on long video captioning tasks.

Q2 & Q3. What are the sources of the videos? How do you collect and process the videos in order to obtain and control the resulting varied video complexity?

Response: Thank you for raising concerns about the source of our videos. The videos in FIOVA are sourced from publicly available copyright-compliant platforms, consistent with the practices of most benchmark datasets. Furthermore, we have adopted standardized procedures to ensure the diversity of our dataset. Below, we provide detailed clarifications, and have added this information in Appendix B.1:

1. Legitimacy of Video Sources

The videos in the FIOVA dataset are all obtained from legal, publicly accessible copyright resources:

• Public Copyright Resources: All videos are selected from online platforms with clearly defined public copyright policies, such as YouTube and similar sources. We ensure that all videos comply with relevant copyright policies.
• Compliance Statement: We strictly adhere to the terms of use of the video source platforms and select only videos that can be used for non-commercial research purposes.

2. Diversity in Video Selection

To ensure the diversity of the dataset and its applicability to a wide range of use cases, we employed the following strategies during video selection:

• Coverage of Various Scene Types: The video content spans a wide range of themes, including daily activities, sports events, and natural landscapes, ensuring the evaluation capability of LVLMs across multiple scenarios.
• Diverse Dynamic Complexity: We specifically focused on videos with dynamic characteristics, such as short- and long-term temporal relationships, complex actions, and multi-agent interactions, to reflect the challenging scenarios that LVLMs may encounter in real-world applications.

3. Video Screening and Quality Control

To further ensure data quality, we implemented multiple rounds of rigorous quality control during the video screening process:

• Initial Screening: We selected videos that complied with public copyright requirements and demonstrated strong diversity.
• Manual Review: Each video was manually reviewed to ensure its clarity and suitability for video understanding tasks.

During video processing, we utilized pre-screening and grouping techniques to ensure diversity in video length, content, and event complexity. These methods were designed to comprehensively evaluate LVLM performance across videos of varying difficulty levels, achieving a robust assessment outcome.

Q1. Were the same five distinct annotators used to annotate all 3,002 videos? I assume the answer is 'No.'

Response: We appreciate the reviewer’s interest in the composition of annotators. As noted in the review, the entire dataset was not annotated by a fixed group of five annotators. Below, we provide a detailed explanation regarding this issue, and have added this information in Appendix B.2:

1. Annotator Assignment

Not all 3,002 videos were annotated by the same five annotators. To ensure diversity and operational feasibility, we adopted the following strategies:

• Multiple Groups of Annotators: The FIOVA dataset was annotated by multiple groups of annotators. Each video was annotated by five independent annotators, but these annotators were not fixed across all videos.
• Annotator Selection Criteria: All participating annotators underwent detailed training to ensure they understood the annotation guidelines and could deliver high-quality annotations.

2. Impact of Annotator Diversity on GT

We chose to use multiple groups of annotators rather than a fixed group of five to enhance the diversity and representativeness of the groundtruth (GT):

• Diverse Descriptive Styles: The involvement of different annotators helps incorporate a wider range of perspectives and descriptive styles, reducing biases introduced by a fixed group of annotators.
• Broader Semantic Coverage: The participation of multiple groups of annotators effectively improves the GT's coverage of video content details and enhances the diversity of descriptions.
• Improved Robustness of GT: The diverse sources of annotations make the GT more adaptable and generalizable, allowing it to better reflect model performance across a variety of scenarios.

3. Quality Control Measures

While the diversity of annotators may introduce variations in descriptive styles, we implemented a strict quality control process to ensure the reliability and consistency of the GT:

• Standardized Annotation Guidelines: All annotators followed the same annotation handbook and guidelines to ensure the consistency and comparability of annotations.
• Annotation Review: Each video’s annotations underwent quality checks to ensure the descriptions aligned with the video content and contained no obvious errors.
• Semantic Integration and Optimization: Using GPT-3.5-turbo, the five annotators' descriptions were consolidated to enhance linguistic consistency while retaining multi-perspective characteristics.

Weaknesses. LLM hallucination and detail omission issues may be present in the ground truth description of each video. Additionally, using an LLM instead of a VLM to summarize the five human captions is insufficient.

Response: Thanks for raising concerns about hallucinations, detail omissions, and using LLMs over VLMs in the groundtruth generation process. Below, we address these concerns in detail and clarify the reliability of FIOVA's GT methodology, supported by revisions in Appendix B.6 (see Fig. A7):

1. Accuracy and Diversity of the Five Annotators

• Accuracy Assurance: The initial annotations in FIOVA were provided by five independently trained annotators. Their descriptions, based on full video observations, ensured alignment with core events and causal logic in the videos.
• Diversity and Detail Coverage: Variations in annotators’ focus on aspects such as actions, background, or narrative are natural and enhance the richness of multi-perspective annotations. These variations are integral to FIOVA’s design, creating a benchmark that better reflects human video understanding.

To address concerns about the limited frame coverage in the original Fig. 4, we revised the example (Appendix B.6, Fig. A7) by expanding it to 35 frames sampled at 20-frame intervals. This expanded visualization confirms:

• All annotators captured the core events of the video (e.g., riding a bike, pretending to fall, lying on the ground, reaching out a hand).
• The temporal order and causal logic in the descriptions strictly align with the video content, demonstrating the accuracy of the annotations.

These results affirm the reliability of multi-perspective annotations in FIOVA, distinguishing it as a benchmark with robust human-understanding baselines.

2. Rationale for Choosing LLMs over VLMs

We opted to use GPT-3.5-turbo instead of vision-language models for text integration, to minimize the risk of hallucinations and preserve annotation authenticity.

• Reducing Multimodal Hallucination Problems: Multimodal models may exhibit alignment instability, potentially generating hallucinated content when interpreting videos. In contrast, LLM strictly processes text, focusing on semantic integration without introducing new content or subjective interpretations. This approach preserves the authenticity and diversity of human annotations while avoiding interference from multimodal alignment issues.
• Preserving the Diversity of Human Annotations: LLM consolidates annotations without altering critical information or suppressing annotator-specific features. For example, in resolving conflicting descriptions (“crying” vs. “smiling”), GPT-3.5-turbo integrates contextual cues, ensuring the final groundtruth aligns with the most plausible narrative logic.
• Efficiency in Text Integration: LLM effectively merges multi-annotator descriptions into coherent, semantically consistent groundtruth, validated by its application in related studies (e.g., Tarsier [1], MM-Ego [2]).

3. Analysis and Handling of Conflicts in Figure 4 (e.g., “Smiling” vs. “Crying”)

In Fig. 4, discrepancies in the boy’s expression (“crying” by Human3 vs. “smiling” by Human5) were resolved through GPT-3.5-turbo’s integration process:

• Contextual Analysis: By analyzing key actions (e.g., “getting off the bike,” “lying down,” “pretending to fall”), combined with Human1’s mention of “pretend” and Human5’s description of “smiling,” GPT-3.5-turbo identified “smiling” as more consistent with the video’s overall narrative.
• Conflict Resolution: Human3’s inference of “crying” was speculative, while Human5’s interpretation aligned better with the prank context and the video sequence, resulting in a semantically coherent GT.

This integration process illustrates how LLMs effectively handle annotation conflicts, producing groundtruths that maintain consistency and completeness.

4. Measures to Ensure GT Reliability

• Avoiding Hallucinations: Strictly limiting LLMs to text integration tasks and using carefully designed prompts minimized risks of hallucinations.
• Enhancing Detail Retention: Five independent annotators ensured comprehensive coverage of video content, minimizing omissions from any single perspective. Besides, manual review of all 3,002 videos identified and corrected minor redundancies, ensuring the groundtruth accurately reflects human annotations.

By combining the diversity of human annotations with LLM’s text consolidation capabilities, FIOVA delivers high-quality, semantically rich groundtruths. This approach ensures credibility and distinguishes FIOVA as a benchmark for long-video description tasks.

[1] Wang J, Yuan L, Zhang Y, et al. Tarsier: Recipes for training and evaluating large video description models[J]. arXiv preprint arXiv:2407.00634, 2024.

[2] Ye H, Zhang H, Daxberger E, et al. MM-Ego: Towards Building Egocentric Multimodal LLMs[J]. arXiv preprint arXiv:2410.07177, 2024.

3. Analysis of Human Evaluator Feedback

Moreover, by analyzing the criteria used by the 10 human evaluators for ranking, we identified the following major points of focus. These points illustrate the diversity and complexity of human evaluators’ considerations when assessing video descriptions:

• Details and Accuracy

  • Most evaluators emphasized accurate descriptions of video details, including event sequence, character actions, objects in the scene, and the presentation of key elements such as colors, movements, and expressions.
  • Example: Subject4 noted inaccuracies in A21, where Human4 described a flamingo as a goose. These inaccuracies significantly lowered their ratings.

• Completeness and Information Coverage

  • Many evaluators tended to assign higher scores to longer, more detailed descriptions, as these often covered more video content and provided richer information.
  • Example: Subject5 highlighted the importance of completeness, favoring longer descriptions. Similarly, Subject2 observed that more extensive descriptions often better captured video details.

• Coherence and Language Quality

  • Evaluators focused on the fluency, structure, and adherence to natural language conventions in descriptions.
  • Example: Subject8, under Mode A (text-only), particularly emphasized textual coherence and structure, stressing whether the text could help reconstruct the video. Subject10 pursued narrative quality in Mode A.

• Objectivity and Logical Consistency

  • Evaluators favored descriptions that were objective and aligned with the logical progression of events in the video, avoiding subjective judgments or deviations from the actual content.
  • Example: Subject4 remarked that Human1’s description in A19, which included "pretending to fall," may contain a subjective interpretation. Subject5 also stressed that descriptions should align with the video’s logical flow.

• Video-Text Alignment

  • In multimodal evaluation (Mode B), evaluators paid closer attention to whether the descriptions accurately reflected the video content, particularly the faithful representation of details.
  • Example: Subject7 and Subject9, under Mode B, prioritized accuracy and detail alignment between the video and the descriptions.

• Conciseness and Avoidance of Redundancy

  • Evaluators preferred descriptions that avoided redundancy while being detailed, particularly when summarizing complex video content.
  • Example: Subject9 pointed out that repetitive phrases in A22’s Human2 description negatively impacted its score. Subject3 noted that under Mode B, descriptions should remain concise while capturing all key actions.

4. Consistency with Research Trends: Standardization Facilitates Synergy

The adoption of 8 frames as a standard setting optimizes experimental efficiency while supporting cross-benchmark model comparisons:

• Ease of Model Comparison: Using a frame setting consistent with other mainstream benchmarks enables researchers to more easily compare model performance on FIOVA with results on other benchmarks. This consistency not only aids understanding but also highlights model generalizability and adaptability across benchmarks.

• Fostering Collaborative Development: Standardized experimental setups lower the barrier to replicating experiments and comparing results, thereby promoting collaborative research in video understanding.

• Enhanced Benchmark Adaptability: Consistency with the frame settings of mainstream algorithms allows FIOVA to seamlessly adapt to various architectures, further solidifying its relevance in the video captioning domain.

5. Potential for Future Extensions

We acknowledge that increasing the frame count (e.g., to 16 or 32 frames) could capture more video details and potentially improve the performance of certain models. However, varying frame counts is more suitable for ablation studies or algorithm-specific optimization rather than the core objectives of benchmark development. FIOVA is designed with high flexibility, allowing researchers to explore alternative frame settings within its framework to further uncover the potential of LVLMs.

In summary, the choice of 8 frames reflects a careful balance between semantic capture and computational efficiency while adhering to the experimental paradigms widely accepted in video understanding research. This design provides a unified and efficient evaluation framework, ensuring fair comparisons across models while leaving room for researchers to explore extended settings. We believe this decision underscores the scientific rigor and rationality of FIOVA’s benchmark design, further cementing its value in video captioning research. We look forward to future research extending the frame count to explore the untapped capabilities of LVLMs. Once again, we sincerely thank the reviewer for their insightful suggestions.

Q3. This paper provides an overview of performance metrics, but lacks detailed error analysis to explain the types of errors made by LVLM and the reasons behind them.

Response: We sincerely thank the reviewer for raising this important issue. Building on the reviewer's suggestion, we have added more examples and detailed analysis in Appendix E.4. Below are the specifics of our updates:

1. Error Type Categorization

We annotated the examples in Appendix E.4 with error categories and identified five common types of errors:

• Omission: The model fails to describe critical events or objects in the video. While this cannot be directly marked in the model's output, we provide textual analyses of such omissions after the relevant examples.
• Misrepresentation: The description contains information inconsistent with the video content. These errors are marked in purple in the model outputs.
• Redundancy: The model repeats descriptions of the same event. These errors are marked in yellow in the outputs.
• Excessive Redundancy: The model overextends or speculates excessively, introducing unnecessary content. These errors are marked in green in the outputs.
• Hallucination Issues: The model includes content not present in the video. These errors are marked in red in the outputs.

2. Potential Cause Analysis

(1) Architectural Limitations

• Cross-modal alignment issues: Current LVLMs face challenges in aligning video-text data effectively. For instance, Tarsier encodes each frame using separate visual encoders, while VideoLLaMA2 utilizes a shared visual encoder across all frames. These differences in alignment strategies impact the model's understanding of video content.
• Insufficient long-sequence modeling: Handling long videos with multiple events requires robust attention mechanisms. Current LVLMs often fail in this regard. For example, Video-LLaVA’s descriptions tend to focus only on initial scenes, omitting later parts of the video.

(2) Training Data Bias

• Inconsistent or insufficient data diversity: Training data with limited diversity may result in biased outputs. For instance, Video-LLaVA struggled with recognizing martial arts scenes (Fig. A23) compared to other LVLMs, likely due to a lack of relevant training examples.
• Hallucination issues: Models can learn and extrapolate hallucinated content from noisy training data. For example, in Fig. A20, VideoChat2 entirely misrepresents players and spectators in a baseball stadium as children, illustrating a significant discrepancy between output and video reality.

(3) Generation Strategy Issues

• Simplistic generation strategies: Using basic generation techniques, such as beam search, can lead to repetitive or incoherent descriptions. ShareGPT4Video, despite using high-quality training data, suffers from severe repetition due to the lack of constraints during generation.
• Weak constraints during generation: Insufficient semantic constraints in the generation process can result in hallucination or semantic errors.

3. Suggestions for Improvement and Optimization

(1) Model Optimization

• Enhancing detail capture: Improving attention mechanisms to better capture key events and details, such as incorporating hierarchical attention mechanisms for long-sequence modeling. For example, VideoLLaMA2 introduces the STC Connector to better handle spatiotemporal information, which future LVLM research can build upon to improve continuity.
• Improving semantic alignment: Leveraging cross-modal alignment constraints, such as visual-language consistency checks, can reduce semantic errors and hallucinations. LLaVA-NeXT-Video highlights the importance of maintaining consistency from cross-modal alignment to understanding.
• Implementing deduplication strategies: Introducing repetition detection mechanisms during generation to reduce redundant descriptions.

(2) Training Data Optimization

• Enhancing data diversity: Expanding training datasets to include diverse scenarios, particularly for complex events in long videos.
• Data cleaning: Removing hallucinated or incorrect examples from training corpora to improve data quality. ShareGPT4Video demonstrates the benefits of using high-quality video-text data but still shows room for improvement.

(3) Evaluation Method Enhancement

• Fine-grained error categorization: Incorporating fine-grained error categorization and statistical mechanisms within the FIOVA framework to better identify model weaknesses. For instance, while calculating FIOVA-DQ, similarity between annotators' events and LVLM-generated events could inform more precise error identification.

We thank the reviewer for this insightful suggestions, which have helped us present our research more comprehensively and explore broader research directions.

Q2. Since GPT-3.5-Turbo cannot directly see the video, induction based on human text order alone can easily bring errors such as illusions to groundtruth (see Fig.4).

Response: We appreciate the reviewer’s insightful comments on the GT generation process. First, we apologize for the ambiguities in Fig. 4 of the original paper, which may have led to misinterpretations. To address this, we have redrawn Fig. 4 for clarity. Due to space constraints in the main text, we have provided a more detailed explanation and analysis of this example in Appendix B.6 of the revised manuscript. Specifically, we use Appendix B.6 and Fig. A7 therein to demonstrate the accuracy of human annotations and the reliability of LLM-generated GT. Below is a detailed response.

1. Accuracy and Diversity of Human Annotations

• Annotation Accuracy: FIOVA’s initial annotations were completed by five independently trained annotators, carefully selected to ensure their descriptions aligned strictly with the video content.
• Diversity and Detail Coverage: Variations in focus (e.g., character actions, environmental details) among annotators reflect natural differences in perspective, enriching the dataset and offering a more comprehensive understanding of video content. These variations are a core feature of FIOVA, distinguishing it from existing benchmarks.

To address concerns regarding the limited frame coverage in the original Fig. 4 (6 frames), we redrew the example (Fig. A7 in Appendix B.6) by extracting 35 frames at 20-frame intervals. Comparing the descriptions of the five annotators with the video frames confirms:

• All annotators accurately captured core events (e.g., riding a bike, pretending to fall, lying on the ground, reaching out).
• Temporal order and causal logic in the descriptions strictly align with the video content, reflecting high annotation accuracy. These findings affirm the reliability of human annotations and their contribution to the multi-perspective richness of FIOVA.

2. Role of GPT-3.5-turbo

• Semantic Integration: GPT-3.5-turbo semantically integrates the descriptions from all five annotators, enhancing coherence and completeness without generating new content or altering the original meanings.
• Minimizing Interference: To maintain the authenticity of human annotations, we avoided multimodal models, which may introduce hallucinations during video-text alignment. Instead, carefully designed prompts guided GPT-3.5-turbo in consolidating human perspectives, preserving diversity while improving overall consistency. This approach has been validated in prior works like Tarsier [1] and MM-Ego [2].

This approach leverages GPT-3.5-turbo’s strengths in text processing to preserve human annotation diversity while enhancing the overall quality and coherence of the descriptions.

3. Issues and Improvements in Figure 4’s GT

(1) Redundancy in GT and Improvements

We appreciate the reviewer’s observation regarding redundancy in the GT. In rare cases, significant differences in style or length among annotators’ descriptions led GPT-3.5-turbo to misinterpret some details as repetitive events. To address this:

• Prompt Optimization: We refined the prompt design to clarify and avoid potential redundancy in outputs.
• Manual Screening and Re-experiments: We manually reviewed all 3,002 videos and reprocessed those with redundancy issues, ensuring the GT comprehensively represents human annotation information.

(2) Verification of Specific Actions (e.g., Smiling and Pointing at the Camera) In the revised GT (Appendix B.6, Figure A7), specific actions were validated:

• Pointing at the Camera: Most annotators (Human 1, 2, 4, 5) mentioned the “reaching out” action, with Human 2 and 4 specifically identifying it as pointing at the camera. Given this high agreement, GPT-3.5-turbo confirmed the action’s accuracy, which was also verified in the video.
• Smiling vs. Crying: Conflicting annotations ("smiling" by Human 5 and "crying" by Human 3) were resolved by analyzing video context. Human 5 interpreted the boy’s smile as part of a prank, consistent with Human 1’s mention of a “pretend” fall. GPT-3.5-turbo synthesized these perspectives to conclude the boy was smiling, aligning with the broader narrative logic.

By integrating five annotators’ descriptions, GPT-3.5-turbo reduces bias and enhances the accuracy and coverage of the GT, reflecting core video semantics. This multi-perspective integration establishes FIOVA as a robust and unique benchmark for video description tasks.

[1] Wang J, Yuan L, Zhang Y, et al. Tarsier: Recipes for training and evaluating large video description models[J]. arXiv preprint arXiv:2407.00634, 2024.

[2] Ye H, Zhang H, Daxberger E, et al. MM-Ego: Towards Building Egocentric Multimodal LLMs[J]. arXiv preprint arXiv:2410.07177, 2024.

Q1. The evaluation model has limitations. This article evaluates some selected LVLMs, but does not explore broader models, such as the business model Gemini-1.5-Pro, which has a strong understanding of long videos. Include more LVLMs for experiments and conduct in-depth analysis, preferably summarizing directions to improve the long video description capabilities of LVLMs.

Response: We thank the reviewer for the valuable suggestion, which provides an opportunity to further discuss the rationale for model selection and the generalizability of FIOVA's evaluation framework. Below is our detailed response:

1. Rationale and Constraints of Model Selection

In this study, we primarily evaluated six representative open-source LVLMs, such as Video-LLaVA and VideoChat2. This selection was based on the following considerations:

• Reproducibility: The core goal of FIOVA is to establish a high-quality evaluation environment that supports extensive research on LVLM performance. Using open-source models ensures reproducibility, enabling the academic community to conduct comparative experiments and improvements based on FIOVA.
• Academic Value: Analyzing open-source models provides direct guidance for model optimization and methodological improvements, which commercial models cannot readily achieve. For instance, insights into limitations revealed in this study can directly drive optimization within the open-source community.
• Academic Recognition: Evaluating open-source models has been widely adopted in academic research, as demonstrated in studies like LVLM-eHub [1], where open-source models are exclusively used for performance analysis.

While we acknowledge that commercial models (e.g., Gemini-1.5-Pro) may exhibit strong capabilities in long-video understanding, selecting them poses the following challenges:

• Closed Source and High Cost: Commercial models are often closed-source and incur significant API costs, limiting their accessibility in academic research.
• Limited Optimization Guidance: The black-box nature of commercial models makes it difficult to use FIOVA’s evaluation results for model optimization, which is a core goal of this study.

2. Validity of the Current Model Selection

Despite not including commercial models, the six open-source LVLMs we selected possess the following characteristics, which comprehensively reveal the strengths and limitations of current LVLM technology in long-video description tasks:

• Architectural Diversity: The selected models span diverse architectural designs. For example, LLaVA-NEXT-Video employs the AnyRes technique to process high-resolution images into sequences consumable by pre-trained Vision Transformers (VIT), a dominant approach for video LVLMs. VideoLLaMA2 integrates custom spatiotemporal convolutional modules to capture relationships over time within videos. This diversity allows us to analyze performance differences in semantic consistency, detail coverage, and language fluency across various designs.
• Task Coverage: The selected models are tailored for long-video description tasks, exhibiting different capabilities in complex semantic understanding, multi-event tracking, and temporal reasoning. Analyzing these models enables us to identify key bottlenecks in current technology and guide future model design.

Furthermore, our primary focus is on establishing an open evaluation framework that provides guidance for improving and optimizing LVLMs, rather than testing specific model performance. The selection of these open-source models demonstrates FIOVA’s utility as a high-quality evaluation environment.

3. Scalability of FIOVA

FIOVA aims to provide a high-quality evaluation environment, designed with comprehensiveness and generalizability in mind to support various model types (e.g., open-source and commercial). We plan to open-source the FIOVA dataset and evaluation toolkit, offering researchers flexible tools to evaluate a broader range of models, including commercial ones. This will enable researchers to use FIOVA and its resources for further studies and analyses tailored to their research goals.

Although this study did not include commercial models, the systematic analysis of open-source models validated FIOVA’s effectiveness as an evaluation framework. FIOVA’s open-source design and high generalizability also offer extensive possibilities for future research.

[1] Xu, Peng, et al. "Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models." arXiv preprint arXiv:2306.09265 (2023).

Q5. There are still many typos in the text, one of the most obvious being that the evaluation metric proposed by Wang et al. (2024) is AutoDQ, not AutoCQ.

Response: We sincerely thank the reviewer for pointing out the spelling issues with great attention to detail. In response, we have meticulously reviewed and revised the entire manuscript to eliminate any typos. Additionally, we would like to clarify and address the ambiguity caused by the naming of AutoDQ.

AutoDQ (Automatic Description Quality), originally proposed by Wang et al. (2024), is an evaluation metric designed to comprehensively and automatically assess the quality of generated textual descriptions. In our initial submission, we renamed the metric as “AutoCQ” (Automatic Caption Quality) to better align it with the Video Caption task in our study. This renaming was intended to emphasize its relevance and applicability to video description tasks.

However, while the renaming aimed to enhance task-specific clarity, it may have inadvertently introduced inconsistency with the original literature and created terminological confusion. Following the reviewer’s constructive suggestion, we acknowledge that adhering to the original naming convention is crucial for academic dissemination and clarity. Therefore, in the revised manuscript, we have reverted all instances of “AutoCQ” back to “AutoDQ,” making the text, tables, and references consistent with the original naming in the literature.

Additionally, we conducted a comprehensive spelling and language check throughout the manuscript to ensure textual accuracy and consistency.

It is important to note that, during the rebuttal process, we optimized and extended AutoDQ to better suit the unique characteristics of FIOVA. In the revised manuscript, we have introduced this enhanced metric, FIOVA-DQ, which incorporates new evaluation dimensions and demonstrates improved applicability to the long video description task.

Q4. This paper provides an overview of performance metrics, but lacks detailed error analysis to explain the types of errors made by LVLM and the reasons behind them.

Response: We sincerely thank the reviewer for raising this important question. We acknowledge the limitations of current traditional metrics (e.g., METEOR, BLEU, GLEU) and automated evaluation methods based on GPT models (e.g., AutoDQ) in assessing complex video description tasks. Based on the reviewer’s suggestions, we have optimized the original AutoDQ method and proposed a new evaluation metric, FIOVA-DQ (see Common Concern 2: Propose a New Evaluation Method FIOVA-DQ), which better integrates human cognition and evaluation needs. Below is our detailed response:

1. Reasons for Choosing Traditional Metrics

In this study, we selected traditional text generation metrics (e.g., METEOR, BLEU, GLEU) as baseline evaluation tools for the following reasons:

• Broad Academic Acceptance and Generalizability: These metrics are widely used in text generation tasks and are recognized as standardized evaluation methods. Employing these metrics ensures the comparability of our experimental results with existing studies, thus enhancing the academic value of our research.
• Foundational Linguistic Quality Assessment: These metrics remain effective in evaluating the basic linguistic structure, syntactic correctness, and alignment with reference descriptions. For long video description tasks, they provide a reliable basis for assessing the linguistic quality of generated texts.

Nonetheless, we acknowledge the limitations of traditional metrics:

• Inability to Capture Event-Level Semantic Consistency: Traditional metrics rely primarily on textual matching, making it difficult to capture event-level semantic relationships between descriptions and video content, such as temporal consistency and causal logic.
• Limited Focus on Fluency and Detail Coverage: Traditional metrics cannot comprehensively evaluate the fluency and completeness of descriptions. These limitations have been discussed in detail in Sec 4.2.

2. Introduction and Complementation of AutoDQ

To address the limitations of traditional metrics, we introduced the AutoDQ method, which offers the following advantages:

• Event-Level Semantic Analysis: AutoDQ enables fine-grained evaluations of semantic consistency between generated descriptions and video content by extracting and matching events. For instance, it assesses a model’s ability to handle event sequences, logical relationships, and causal chains in multi-event long video descriptions.
• Adaptability to Complex Scenarios: AutoDQ effectively captures the completeness and consistency of multi-event descriptions in long video tasks, particularly in scenarios where traditional metrics lack discriminatory power.

Our experimental results demonstrate the advantages of AutoDQ in:

• Event-Level Differentiation: Variations in AutoDQ scores across different models reflect their differing capabilities in capturing event semantics.
• Applicability to Complex Scenarios: AutoDQ effectively handles semantic decomposition for long videos, capturing the completeness and consistency of multi-event descriptions.

3. Introducing the New Evaluation Metric FIOVA-DQ

While AutoDQ exhibits suitability for fine-grained semantic analysis, we recognize its limitations in reflecting human preferences and capturing the importance of key events. Hence, we proposed an optimized evaluation metric, FIOVA-DQ, with the following features:

• Integration of Human Cognition via Weighted Design: FIOVA-DQ incorporates the descriptions of five annotators and weights model-generated events based on their alignment with human annotations. Specifically, importance weights are derived by ranking event consistency with human annotations (see revised paper Sec 3.2 and Sec 4.1).
• Enhanced Judgment of Human-Like Descriptive Capabilities: Compared to traditional metrics, FIOVA-DQ aligns more closely with human judgments of description quality, particularly excelling in assessing the consistency and fluency of multi-event long video descriptions.

In summary, we conclude:

• Traditional metrics provide a foundational method for linguistic quality assessment, facilitating comparisons with existing research.
• AutoDQ and FIOVA-DQ introduce event-level analysis, addressing the limitations of traditional metrics in capturing semantic consistency and event importance.
• The optimized FIOVA-DQ incorporates human cognition into the evaluation framework, ensuring fairness and scientific rigor in model assessments across multiple dimensions.

Q3. While this work has adopted multiple metrics to demonstrate the video caption performance, it lacks analysis of how those metrics align with human preference.

Response: We sincerely thank the reviewer for raising this important question. We believe that evaluation metrics serve as tools to quantify model performance and compare the results against human capabilities (human baselines). Leveraging the unique characteristics of FIOVA, we believe its contributions at the data, metric, and experimental levels ensure alignment between model performance and human preferences. Below is our detailed response:

1. Clarifying the Role and Purpose of Evaluation Metrics

• Quantifying Model Capability: Evaluation metrics (e.g., BLEU, METEOR, AutoDQ) are tools for quantitatively analyzing model performance, objectively reflecting differences between generated descriptions and human annotations. From an academic perspective, these metrics are widely used in video description tasks and facilitate standardized model evaluation.
• Indirectly Reflecting Human Preferences: While these metrics do not directly capture human subjective preferences, they provide critical insights into the alignment between models and human understanding by assessing aspects such as semantic consistency, event coverage, and linguistic quality.

2. How FIOVA Reflects Human Preferences in Its Design

• High-Quality Human Baseline: Each video in FIOVA is annotated by five independent annotators, ensuring the groundtruth is multi-perspective, comprehensive, and high-quality. This multi-annotator strategy captures the diversity of human understanding of video content while minimizing potential biases or omissions from individual annotators.
• Balancing Diversity and Consistency: By integrating the descriptions from five annotators, FIOVA reflects human expectations for both detail-oriented and consistent video descriptions. Thus, aligning evaluation metrics with groundtruth inherently simulates a comprehensive human judgment of video understanding capabilities.

3. Diverse Evaluation Metrics and Further Optimization of AutoDQ

(1) Combining Traditional Metrics with Semantic Consistency Metrics in the Original Study

• Foundational Role of Traditional Metrics: BLEU, METEOR, and GLEU are widely accepted in text generation research and effectively measure the text-matching quality between generated and reference descriptions. However, these metrics have limitations in capturing complex video semantics and human preferences.
• Introduction of AutoDQ: To address these limitations, we incorporated AutoDQ as an event-level semantic consistency evaluation tool:
  • Event-Level Semantic Analysis: AutoDQ extracts and matches events between model-generated texts and video content, providing fine-grained evaluations of semantic consistency.
  • Applicability to Complex Scenarios: AutoDQ captures the completeness and consistency of multi-event descriptions in long videos, particularly supplementing traditional metrics in scenarios where they lack discriminatory power.

(2) Further Optimization of AutoDQ to Incorporate Human Preferences

As detailed in Common Concern 2: Propose a New Evaluation Method FIOVA-DQ, we enhanced AutoDQ by weighting event importance based on the descriptions from five annotators. This optimization improves its ability to evaluate human-like descriptive capabilities and led to the development of the FIOVA-DQ metric (see revised paper, Sec 3.2 and Sec 4.1). Compared to traditional metrics, the optimized AutoDQ (FIOVA-DQ) aligns more closely with human judgments of description quality while better reflecting FIOVA's core value as an evaluation framework.

Therefore, through its multi-annotator baselines, diverse evaluation metrics, and the optimized FIOVA-DQ, FIOVA effectively aligns model performance with human preferences:

• At the data level: Multi-annotator annotations reflect human expectations for detail-oriented and consistent descriptions.
• At the metric level: FIOVA-DQ, combined with traditional metrics, provides comprehensive evaluations from text matching to semantic consistency.
• At the experimental level: Comprehensive experiments reveal the similarities and differences between model and human video description capabilities, offering valuable insights for future research.

Q2. Describe Videos like Humans might be an interesting evaluation setting. However, it does not stand alone as a task. It would be meaningful to include further analysis to show the correlation between performance of Describe Videos like Humans and other video understanding tasks (VideoQA, etc.).

Response: Thanks for raising this important question. "Describe Videos like Humans" is a characterization of our research goal for the Video Caption task, not a new task. Below is our detailed response:

Clarifying the Task Scope

We would like to clarify that the core task studied in this work is Video Captioning, a classic and widely recognized video understanding task in academia. "Describe Videos like Humans" specifically encapsulates our research goal for Video Captioning, emphasizing that generated video descriptions should match human-level performance in terms of detail, completeness, and consistency. Therefore, "Describe Videos like Humans" is not a novel task but a refinement of the Video Captioning goal through the FIOVA benchmark, aligning Video Captioning capabilities with human standards and revealing the strengths and limitations of LVLMs in describing complex and long videos.

Relation Between Video Captioning and Other Video Understanding Tasks

Video Captioning and other video understanding tasks (e.g., Video Question Answering, VideoQA) require deep comprehension of video content but differ in their goals and evaluation methods:

• Video Captioning:

  • Objective: Generate natural language descriptions that comprehensively cover key events, scenes, and details of the video.
  • Requirements: Temporal reasoning (capturing event sequences and causal relationships) and language generation capabilities.

• VideoQA:

  • Objective: Answer questions related to video content, focusing on specific semantic reasoning (e.g., causal relationships of events, character intentions).
  • Requirements: Accurate semantic understanding and logical reasoning based on question context.

Although their goals differ, both tasks require:

• Semantic Understanding: Modeling the core semantics of video content.
• Spatio-Temporal Reasoning: Capturing temporal sequences and spatial relationships of events.
• Multimodal Integration: Processing both visual and textual information from videos.

Connection Between Video Captioning and VideoQA

The MSVD-QA dataset [1] is a representative VideoQA dataset based on the existing Microsoft Research Video Description (MSVD) dataset. As mentioned in related works, this dataset contains approximately 120K sentences describing 1,970 video clips. In MSVD-QA, question-answer (QA) pairs are generated based on these descriptions. While this dataset was initially created for video description, its large size has also made it useful for VideoQA. It contains 1,970 video clips and approximately 50.5K QA pairs.

Similarly, FIOVA primarily supports the Video Captioning task but can also be extended to support VideoQA in future research.

Research Value of FIOVA

We believe FIOVA provides a novel evaluation perspective for Video Captioning:

• It focuses on the fine-grainedness and semantic consistency of generated descriptions, offering an important entry point for evaluating LVLMs' human-like video understanding capabilities.
• FIOVA's dataset, with multi-annotator labels and long video coverage, strongly supports Video Captioning, particularly in capturing complex video semantics.

In conclusion, "Describe Videos like Humans" is a further articulation of the research goal for Video Captioning, not a new task. The FIOVA benchmark, through its novel evaluation perspective, effectively measures the gap between algorithms and human video understanding and description capabilities, embodying the goal of "Describe Videos like Humans."

[1] Chen D, Dolan W B. Collecting highly parallel data for paraphrase evaluation[C]//Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. 2011: 190-200.

Q1. Some settings are not easy to follow, like the batch ranking in Sec 4.3.

Response: We appreciate the reviewer’s observation regarding the lack of clarity in the current description of the “Batch Ranking” settings. We acknowledge that the existing explanation was overly concise and did not fully capture the detailed operational process or its core significance. In the revised manuscript, we have added Fig. A8 in Appendix C and provided a detailed explanation of the Batch Ranking calculation steps. Below, we outline the specific process:

The Batch Ranking procedure involves three main steps (corresponding to Algorithms A1, A2, and A3 in Appendix C). Using one video as an example, the calculation process is as follows:

Step 1: Evaluating human caption consistency (Algorithm A1 in Appendix C)

• Input: Five independent human-generated textual descriptions.
• Evaluation dimensions: We designed five evaluation dimensions—consistency, context, correctness, detail orientation, and temporality—and utilized an LLM to assess them (as detailed in Sec 2.2). Additionally, we included caption length as a sixth dimension.
• Output:
  • The coefficient of variation (CV) is computed for each dimension across the five human-generated captions, representing the consistency of human descriptions.
  • The average CV across the five dimensions is calculated to determine the overall human consistency score for the video. A higher score indicates greater stylistic variation among the five human annotators for that video. Based on these CV scores, the entire FIOVA dataset is categorized into A-H groups (see Sec 2.2).

Step 2: Evaluating LVLM model consistency (Algorithm A2 in Appendix C)

• Input: Six textual descriptions generated by LVLMs.
• Evaluation dimensions:
  • Traditional metrics (e.g., BLEU, METEOR, GLEU).
  • Event-level metrics (AutoDQ), which emphasize semantic consistency.
  • The newly proposed FIOVA-DQ, optimized for the characteristics of the FIOVA dataset.
• Output:
  • The CV is computed for each evaluation dimension across the six model-generated descriptions.
  • The average CV across all dimensions is calculated as the overall model consistency score for the video. A higher score indicates greater stylistic variation within the LVLM ensemble when describing the video content.

Step 3: Comprehensive ranking and difference analysis (Algorithm A3 in Appendix C)

• Human ranking:
  • Videos are ranked based on the overall human consistency scores calculated in Step 1.
  • The relative position of the current video in the human baseline ranking is recorded.
• Model ranking:
  • Videos are ranked based on the model consistency scores calculated in Step 2.
  • The relative position of the current video in the model baseline ranking is recorded.
• Difference analysis:
  • The difference between the rankings of the same video in the human and model baselines is compared. This quantifies the stylistic differences between human annotators and the LVLM ensemble when describing the same video. It also evaluates whether humans and LVLMs adopt similar strategies in their understanding and description of video content.

In the revised manuscript, we have supplemented Appendix C with detailed visualizations (Fig. A8) and textual explanations. Additionally, we reorganized Sec 4.3 to clearly articulate the core concept, specific operations, and analytical conclusions of Batch Ranking.

The core design of Batch Ranking lies in quantifying the ranking differences in consistency between humans and models. This provides insights into their respective performance gaps in long-video description tasks. We believe that Batch Ranking offers a novel perspective for evaluating the semantic capabilities of LVLMs and enriches the analytical methodologies in the long-video description domain. We hope this design will inspire and support future research in this field.

Q4. Sec 2.2 shows that GPT-3.5-turbo is adopted to assess the quality of video caption, while that does not sound make sense to me. How can GPT-3.5 (a legacy text-only model) evaluate dimensions like Context, Correctness for video captions without accessing the visual parts? Can you provide more evaluation samples?

Response: We sincerely thank the reviewer for raising valuable questions regarding our evaluation method. Our goal in this step is to use GPT-3.5-turbo to analyze the differences in video descriptions provided by multiple human annotators and to group videos accordingly. In other words, the purpose of using GPT-3.5-turbo in our study is to evaluate the relative differences in human annotations (e.g., consistency, semantic coverage, and linguistic fluency), rather than to assess the absolute correctness of the annotations. We believe GPT-3.5-turbo is well-suited to achieving this goal. Besides, we have provided more examples in Appendix B.5 for reference. Below is our detailed response:

1. Clarifying the Evaluation Goal and the Applicability of GPT-3.5-turbo

(1) The Core Goal of the Evaluation

Before the annotation process began, we carefully selected and trained the annotators, requiring them to follow annotation guidelines, watch the videos attentively, and provide complete descriptions. Consequently, we believe the descriptions accurately reflect the video content as much as possible. The purpose of this step is not to determine the absolute correctness of the annotations but to analyze the relative differences in semantic expression among multiple annotators, such as contextual consistency, detail coverage, and linguistic style. By calculating the coefficient of variation (CV) based on these differences, we grouped the videos to facilitate further analysis of description consistency and complexity.

(2) Why We Chose GPT-3.5-turbo Instead of a Multimodal Model

We opted for the single-modal GPT-3.5-turbo rather than a multimodal model (e.g., GPT-4) for the following reasons:

• Avoiding Multimodal Hallucinations: Multimodal models may introduce visual hallucinations when aligning textual and video content, which could lead to evaluation results deviating from true semantics. For instance, multimodal models may "guess" events not present in the video, thus compromising the accuracy of the analysis.
• Focusing on Linguistic Variations: GPT-3.5-turbo exclusively analyzes textual semantics and linguistic style differences, capturing the semantic characteristics of human annotations in a more focused manner.

Moreover, we designed refined prompts (see Appendix D 1.1) to guide GPT-3.5-turbo to analyze description texts from dimensions such as contextual consistency and detail coverage. These prompts were tailored to the task characteristics of our study and optimized through multiple experiments to ensure the reliability of the outputs.

(3) Validation of GPT-3.5-turbo's Text Analysis Capability

Although GPT-3.5-turbo is a single-modal model, its performance in text understanding and semantic analysis tasks has been widely recognized. Similar methods have been successfully applied in studies such as VideoLLaMA2. This demonstrates that GPT-3.5-turbo is well-suited to our objective of capturing semantic differences in the textual modality.

Based on GPT-3.5-turbo's analysis results, we grouped video descriptions into Groups A to H to study the consistency and diversity of descriptive styles. Among these, Group A (high consistency) features highly consistent descriptions with concentrated semantic distributions, while Group H (high variability) exhibits significant differences in descriptive styles and semantic distributions.

2. Specific Case Analyses in Revisied Paper

To further validate the rationale of our evaluation method, we have included evaluation samples from different groups in the revised version to better demonstrate GPT-3.5-turbo's applicability in semantic analysis (see Appendix B.5).

The text analysis capability of GPT-3.5-turbo enables it to efficiently evaluate the semantic differences in human annotations, making it suitable for text-based grouping tasks. By presenting examples of high consistency and high variability groups, we have validated the method's rationale and effectiveness. We sincerely thank the reviewer for their feedback, which has prompted us to further refine our evaluation design and research presentation.

Q4. Traditional metrics are not well-suited for current LVLMs due to the nature of long responses. Additionally, is AutoDQ sufficient for evaluating dense captioning?

Response: Thanks a lot for raising this important question. First, we adopted traditional metrics (e.g., BLEU, METEOR, GLEU) due to their representativeness and general applicability. Additionally, we incorporated AutoDQ, which evaluates video understanding by disentangling event-level semantics, offering a more fine-grained perspective compared to traditional metrics. AutoDQ demonstrates the ability to evaluate complex video description tasks, especially for dense, long video descriptions. Building on this, and in response to the reviewer’s suggestions, we optimized AutoDQ by leveraging FIOVA’s unique characteristics (i.e., descriptions from five human annotators per video) and proposed a new metric, FIOVA-DQ (see Common Concern 2: Propose a New Evaluation Method FIOVA-DQ). Below is our detailed response:

Rationale for Choosing Traditional Metrics

In this work, we selected traditional text generation metrics (e.g., BLEU, METEOR, GLEU) for the following reasons:

• Widespread Use: Traditional metrics are widely adopted in the text generation domain and have become standard methods for model evaluation. Their use ensures that our results can be directly compared with prior studies, thereby enhancing the academic value and comparability of our experiments.
• Language Quality Evaluation: These metrics remain effective for evaluating fundamental aspects of textual outputs, such as linguistic structure, syntactic correctness, and alignment with reference descriptions. For long video description tasks, they provide a reliable foundation for assessing the linguistic quality of generated texts.

We also acknowledge the limitations of traditional metrics in long video description tasks, which are discussed in detail in Section 4.2. For instance:

• Inability to Capture Semantic Consistency: Traditional metrics are primarily text-matching-based and often fail to capture the event-level semantic relationships between descriptions and video content, such as temporal consistency and causal logic.
• Lack of Focus on Fluency and Detail Coverage: These metrics cannot comprehensively assess the fluency of descriptions or the completeness of detailed coverage.

Introduction of AutoDQ as a Complementary Metric

To address the limitations of traditional metrics, we introduced AutoDQ, a recently proposed evaluation metric (July 2023) that provides fine-grained assessment and analysis from an event-level perspective. Compared to traditional metrics, AutoDQ offers several advantages:

• Event-Level Semantic Analysis: AutoDQ evaluates the fine-grained semantic alignment between generated descriptions and video content by extracting and matching events. For instance, it can assess the model’s performance in capturing event sequences, logical relationships, and causal chains in multi-event long videos.
• Suitability for Complex Long-Video Scenarios: AutoDQ effectively captures the completeness and consistency of multi-event descriptions in long videos. It provides targeted evaluation, especially in scenarios where traditional metrics lack discriminatory power.

In our experiments, AutoDQ demonstrated the following capabilities:

• Discriminatory Power at the Event Level: AutoDQ scores reveal differences in models' abilities to capture event-level semantics. For instance, when describing multi-event long videos, AutoDQ identifies strengths and weaknesses in models’ handling of event order, logical relationships, and semantic coherence.
• Applicability to Complex Scenarios: AutoDQ effectively disentangles semantic complexities in long videos, capturing the completeness and consistency of multi-event descriptions. It complements traditional metrics in scenarios where they fall short.

Although AutoDQ has shown considerable applicability, we acknowledge its room for improvement. In response to the reviewer’s suggestions, we further optimized AutoDQ during the rebuttal process by incorporating FIOVA’s characteristics.

Introduction of a New Evaluation Metric

We improved AutoDQ by weighting events based on the descriptions from five human annotators. This enhancement strengthens AutoDQ’s ability to evaluate human-like descriptive capabilities, leading to the proposed new metric, FIOVA-DQ (see revised paper, Sec 3.2 and Sec 4.1). Compared to traditional metrics, the optimized AutoDQ aligns more closely with human intuitive judgments of description quality.

Q3. Do the models that perform better on FIOVA also perform better in human evaluations?

Response: Thanks for propose this insightful question. We believe that the design of FIOVA ensures alignment between model performance and human evaluation. Below is our detailed response:

Providing a High-Quality Human Baseline with Multi-Annotator Strategy

• Multi-Annotator Strategy: Each video in the dataset is annotated by five independent annotators. This multi-annotator strategy not only captures the diversity of semantics but also minimizes the potential biases and omissions introduced by individual annotators. By integrating the descriptions from all five annotators, FIOVA establishes a comprehensive, objective, and multi-perspective human baseline.

• Representativeness and Comprehensiveness: The groundtruth in FIOVA aggregates multi-perspective descriptions from five annotators, fully representing human understanding of video content, including complex spatiotemporal relationships and multi-event semantic capture. Consequently, the alignment of model descriptions with the groundtruth reflects the degree to which the models align with human video comprehension capabilities.

Ensuring Consistency Between Evaluation Metrics and Human Judgments

• Combining Traditional Metrics with Semantic Consistency Evaluation: We employed traditional text-matching metrics such as BLEU, GLEU, and METEOR, which are widely used in video description tasks to assess the semantic consistency between generated and reference descriptions. In addition, we introduced event-level semantic evaluation metrics (AutoDQ), which focus on fine-grained analyses of the alignment between video content and semantic descriptions. This provides a more precise assessment of how well the generated descriptions align with the actual video content.

• Optimizing AutoDQ: As detailed in Common Concern 2: Propose a New Evaluation Method FIOVA-DQ, we further improved the calculation of AutoDQ in the revised version by incorporating weightings based on the five annotators’ descriptions. This enhancement strengthens its ability to assess human-like descriptive capabilities and led to the proposed modified metric, FIOVA-DQ (see Sec 3.2 and Sec 4.1 of the revised paper). Compared to traditional metrics, this optimized AutoDQ aligns more closely with human intuitive judgments of description quality.

In conclusion, through the multi-annotator strategy that provides a high-quality human baseline, diverse evaluation metrics, and the optimized AutoDQ metric, FIOVA ensures a robust alignment between model performance and human evaluation.

Q2. There are no experiments that demonstrate why the authors' dataset is superior to other datasets. For instance, is the FIOVA dataset better than the DREAM-1K dataset used in Tarsier for evaluation?

Response: We sincerely thank the reviewer for raising the important question about the uniqueness of the FIOVA dataset, as this feedback has been highly valuable in refining our work. We believe that FIOVA offers substantial advantages over existing benchmarks. In addition to the representative benchmarks listed and compared in Table 1 (e.g., HowTo100M, ACAV, YT-Temporal-180M, HD-VILA-100M, Panda-70M, MSVD, LSMDC, MSR-VTT, DiDeMo, ActivityNet, YouCook2, VATEX), FIOVA also includes the features of the DREAM-1K dataset, which the reviewer specifically highlighted.

Below is our detailed response:

1. The core objective of FIOVA distinguishes it from DREAM-1K, emphasizing its unique contributions.

FIOVA is designed as a high-quality benchmark to evaluate LVLMs' performance in complex long-video understanding and generation tasks. Compared to DREAM-1K, FIOVA offers distinct advantages in several critical aspects:

• Video length and complex scenarios:
  • FIOVA: The average video length is 33.6 seconds, significantly longer than DREAM-1K's 8.9 seconds. This design increases the difficulty for LVLMs in handling cross-frame information integration and capturing spatiotemporal causal relationships, aligning better with real-world application demands.
  • DREAM-1K: The shorter videos primarily focus on single- or multi-event action detection, making it more suitable for localized dynamic understanding tasks.
• Multi-annotator labeling:
  • FIOVA: Each video is annotated by five independent annotators, providing multi-perspective semantic coverage and significantly reducing the bias introduced by single annotators.
  • DREAM-1K: Videos are annotated by a single annotator, which, while excelling in fine-grained action labeling, lacks the ability to provide multi-perspective descriptions.
• Diversity of themes and scenarios:
  • FIOVA: Covers a wide range of themes (e.g., long-video narratives, complex dynamic scenarios), better aligned with real-world application needs.
  • DREAM-1K: Primarily focuses on short video action events (single or multiple), suitable for validating localized dynamic analysis.

2. FIOVA focuses on evaluation capabilities that differ significantly from DREAM-1K.

The uniqueness of FIOVA extends beyond its dataset design to its ability to enhance LVLM evaluation:

• Long-video semantic understanding:
  • DREAM-1K’s shorter videos limit the validation of models’ capabilities in complex narratives and multi-event scenarios. In contrast, FIOVA’s long-video design allows testing models’ ability to understand complex spatiotemporal causal relationships.
• Comprehensive multi-annotator perspectives:
  • FIOVA leverages multi-annotator annotations to create high-quality human benchmarks, offering more reliable diversity and detailed coverage. This design also lays the foundation for fine-grained comparisons between models and humans.
• Broader domain applicability:
  • Through its diverse themes and multi-event design, FIOVA evaluates models’ generalization and real-world applicability, whereas DREAM-1K primarily focuses on action-event detection tasks.

These multidimensional contributions have been recognized by multiple reviewers:

• Reviewer SaKT: Highlighted that FIOVA provides a wealth of materials, making it a valuable resource for evaluating LVLMs’ video comprehension capabilities.
• Reviewer CA5F: Acknowledged that FIOVA is more comprehensive than previous benchmarks, focusing on complex spatiotemporal relationships in long videos and providing realistic and challenging test cases for LVLM evaluation.
• Reviewer pxyN: Emphasized that FIOVA uniquely and comprehensively represents human understanding in video description tasks, advancing video-language model development.

To further highlight FIOVA’s uniqueness, we have added a direct comparison between FIOVA and DREAM-1K in the revised manuscript (Table 1).

Once again, we thank the reviewer for the valuable feedback on the uniqueness of the FIOVA dataset. We believe the clarifications and improvements outlined above effectively demonstrate FIOVA’s critical contributions to evaluating long-video understanding tasks.

Q1. The overall paper reads more like a collection of experimental analyses rather than a benchmark.

Response: We sincerely thank the reviewer for raising an important point about balancing experimental analysis and the positioning of the FIOVA benchmark. We firmly believe that FIOVA not only satisfies the core requirements of a high-quality benchmark but also provides deep insights through experimental analysis, which have been highly appreciated by the other three reviewers. Specifically, Reviewer SaKT noted that FIOVA offers extensive materials, making it a valuable resource for evaluating the video comprehension capabilities of LVLMs; Reviewer CA5F recognized FIOVA as being more comprehensive than previous benchmarks, providing realistic and challenging test cases for LVLM evaluation; and Reviewer pxyN highlighted FIOVA as a unique and important benchmark that advances video-language model development, representing a comprehensive understanding of video description tasks.

Below is our detailed response:

1. FIOVA fully satisfies the core elements of a high-quality benchmark.

The purpose of a benchmark is to provide a standardized framework for evaluating model performance, typically requiring the following three key elements:

• High-quality dataset: A benchmark must include diverse scenarios and tasks to support comprehensive model performance evaluation.
• Systematic evaluation methods: A standardized evaluation framework and metrics are essential for objectively comparing different models.
• Introduction of new insights: Experimental analysis should reveal model characteristics and limitations, offering directions for future research.

FIOVA has been meticulously designed to adhere to these principles:

• Dataset (see Section 2): FIOVA contains 3,002 long videos (average duration 33.6 seconds), each annotated by five independent annotators. These annotations enhance semantic diversity and detailed coverage, forming the foundation for a robust human benchmark.
• Evaluation methods (see Section 3): We conducted a systematic evaluation of six representative open-source LVLM models, all of which are state-of-the-art in the field. We combined traditional metrics (e.g., BLEU, GLEU, METEOR) with event-level semantic metrics (AutoDQ) to validate model performance from multiple dimensions. Based on reviewer feedback, we also introduced a novel evaluation metric, FIOVA-DQ, tailored to the unique characteristics of FIOVA.
• New insights (see Sections 4.2 and 4.3): Our experiments reveal the strengths and weaknesses of LVLMs when handling complex long videos, such as their trade-offs between accuracy and completeness and their significant gaps compared to human benchmarks. These insights provide valuable guidance for future model development.

These contributions have been positively acknowledged by the other three reviewers, who recognized our work on dataset quality, evaluation methods, and comprehensive analysis supporting the benchmark construction.

2. The length of experimental analysis is justified by the need to validate the benchmark’s evaluative capabilities.

The reviewer noted that the experimental analysis occupies a significant portion of the paper. We argue that experimental analysis plays a crucial role in benchmark-type papers. The following points illustrate the close connection between experiments and the benchmark:

• Experiments validate the benchmark’s effectiveness:
  • Our experiments demonstrate that FIOVA effectively reveals model performance and limitations in complex scenarios, confirming its value as a benchmark.
  • For instance, by analyzing the deficiencies of LVLMs in consistency and detail coverage for complex video scenarios, we validated both the dataset design of FIOVA and its relevance to model evaluation.
• Experimental analysis provides long-term value to the field:
  • The experimental results directly support the positioning of FIOVA as a benchmark. For example, our findings show that LVLMs tend to adopt uniform strategies when handling complex situations, despite differences from the diverse strategies employed by humans. By uncovering these behavioral patterns, we provide directions for model optimization and improvement.

We believe these improvements and clarifications make FIOVA’s positioning as a comprehensive benchmark even more explicit. Once again, we thank the reviewer for the valuable feedback, which has helped us refine the paper and better present its academic value.