This paper introduces a scalable, semi-automatic pipeline for constructing the Neptune benchmark dataset, aimed at evaluating long video understanding capabilities. A new evaluation metric, GEM, is proposed, demonstrating its advantages over traditional rule-based evaluation metrics.

The paper introduces NEPTUNE, a new benchmark for evaluating multimodal understanding of long videos. It addresses the limitations of current video datasets and models, which often focus on short clips and can be solved by image models applied per frame. The authors propose a semi-automatic pipeline that leverages large Video Language Models (VLMs) and Large Language Models (LLMs) to automatically generate dense, time-aligned video captions and challenging question-answer-decoy sets for video segments up to 15 minutes long. The paper also discusses the related works in video question answering, the motivation behind the NEPTUNE dataset, and the detailed pipeline for dataset creation. It concludes with the limitations of the dataset and potential areas for future improvement, such as reducing biases introduced by the models used in the creation process and enhancing the dataset with additional annotations like temporal grounding or entity labels.

The paper introduces a new benchmark dataset called Neptune, designed to assess multimodal, high-level understanding of long videos. The key contributions of the paper are as follows:

1. Semi-Automatic Pipeline: The authors propose a scalable, semi-automatic pipeline for generating challenging question-answer-decoy sets for long videos, leveraging large video language models (VLMs) and large language models (LLMs) to automatically generate dense, time-aligned video captions and tough QAD sets for video segments up to 15 minutes long.
2. Neptune Dataset: The dataset, Neptune, covers a broad range of long video reasoning abilities and includes a subset that emphasizes multimodal reasoning. It consists of 3,268 QAD annotations for 2,405 videos.
3. Evaluation Metrics: Given the limitations of existing metrics for open-ended question answering, the authors introduce a new model-based metric called GEM (Gemma Equivalence Metric) to score open-ended responses on Neptune, offering a static and reproducible evaluation method.
4. Benchmark Evaluations: The paper presents benchmark evaluations that reveal current open-source long video models perform poorly on Neptune, especially on questions of testing temporal ordering, counting, and state changes, indicating a significant gap between open-source and proprietary models like Gemini-1.5 and GPT-4.

In summary, the paper contributes a new benchmark dataset and metric aimed at advancing the field of long video understanding through multimodal reasoning, with the goal of spurring the development of more advanced models in this area.

We thank the reviewers for their insightful comments, highlighting the strengths of our dataset and suggesting areas for improvement. Here, we address points raised by multiple reviewers.

Comparison to other Long Video dataset pipelines

We highlight novelties of our dataset, and its creation pipeline compared to both previous and concurrent works $1-11$ .

1. Reducing Manual Effort: Our pipeline is semi-automatic, unlike manual pipelines $4,5,3,11$ . In LVBench $3$ and VideoMME $11$ , even the video selection is done manually, and for MoVQA $1$ , only the decoys are generated automatically. We asked raters to propose QAs (but not decoys) for videos from scratch, and found the time taken (19.03 minutes), longer than simply discarding or correcting QAs generated automatically (10.32 minutes), i.e. showing our pipeline almost halves rater effort. Manually proposing decoys is even more challenging, both in time as well as effort (as it requires more creativity to come up with incorrect answers). The closest to our pipeline is concurrent work VideoVista $6$ , which uses GPT4 to generate QADs automatically. However the performance of GPT4 and Gemini-1.5 on this dataset is close to saturated (98% on some categories).
2. Scalability to generic YouTube videos: Our pipeline can be applied to any generic YouTube video. This is unlike EgoSchema $2$ , which relies on human generated captions, SFD $9$ , which requires movie titles, loglines and synopses (human-written), or MLVU $4$ , which re-uses annotations from existing datasets for many of their tasks. This makes the dataset scalable, as YouTube has a constantly growing set of videos. Potential further use cases could be applying this pipeline to generate large quantities of training data.
3. High Quality Video Captions using careful multi-stage prompting. A key novelty is the ability to generate high quality video captions, using a multi-stage process (Sec 4.2) which involves generating shot-level captions, clustering shots into segments, and adding visual support captions. Examples of caption quality are provided in the supplementary (Fig. 13 and Sec. F), showcasing details such as visual support, numerous events, mood and atmosphere, details from the ASR, and even high level feelings and emotions.

Clarifications on the GEM metric

Training: We train GEM by fine-tuning the open source Gemma model on BEM, a dataset of general answer equivalence ratings. Therefore, GEM is not specific to Neptune and generally applicable to other answer equivalence problems as well.
Comparison to human assessments: To assess GEM (Sec. 5.1, appendix D.1, D.2), we manually annotate an answer equivalence dataset by sampling 97 question-answer pairs from Neptune. We then generate 3 candidate answers per question by prompting VideoLLAVA, Gemini-1.5-Pro and MA-LMM to write a free-form answer for each question without looking into the decoys or ground truth. We then used human raters to manually score these responses, resulting in a dev set with 291 QA-pairs that allows us to compare GEM to human performance. GEM metric reaches an F1-Score of 71.2, which very closely approximates the performance of the much more expensive, closed-source Gemini model (F1-Score 72.8). We will move some details from the appendix to the main paper.

Additional Frame Experiments

We provide additional frame experiments for both Gemini-1.5-Pro and new open-sourced models below. We find that Gemini saturates at around 50 frames while open-sourced models have varying saturation points, VideoLLaMA2 (8 frames), MiniCPM (50 frames) and LLaVA-OneVision (100 frames). For both MiniCPM and LLaVA-OneVision we were unable to fit more frames into the context window.

This is an interesting experiment which we will add to the paper. Note that these are uniformly sampled frames, and hence cover the full span of the video.

We also provide frame ablations comparing Neptune to other datasets in our response to reviewer jd8F.

$1$ * MoVQA, Zhang et al.

$2$ EgoSchema, Mangalam et al.

$3$ * LVBench, Wang et al.

$4$ * MLVU, Zhou et al.

$5$ MMBench-Video, Fang et al.

$6$ * VideoVista, Li et al.

$7$ * ReXTime, Chen et al.

$8$ * LongVideoBench, Wu et al.

$9$ * Short Film Dataset, Ghermi et al.

$10$ Towards long-form video understanding, Wu et al.

$11$ * Video-MME, Fu et al.

$12$ CinePile, Rawal et al.

$13$ Perception Test, Pătrăucean et al.

* (unpublished or concurrent work)

We thank the reviewers for their helpful comments and insightful questions which have helped strengthen our paper! We summarize the improvements we have made as part of the rebuttal here:

1. Comparisons to additional open source models: We evaluated three more recent open source models: LLaVA-OneVision, MiniCPM and InternVL2, giving a more comprehensive and up-to-date view of the state of the art on our dataset.
2. More detailed ablation with different number of frames: We added an ablation going from 1 to 150 frames in 6 steps. Besides Gemini, which we used in the original ablation in the paper, we also ablated on 3 open source models, providing insights into each model’s ability to reason with limited information and their saturation points.
3. Experimental comparison with three other video understanding benchmarks: We compare Neptune against Video-MME, CinePile and Perception Test using Gemini-1.5-Flash with different numbers of frames, showing the model performance and saturation characteristics on each benchmark.
4. An additional metric that gives equal weights to different task types. This allows for a more balanced view of model performance and removes skew of the metric to the most prominent task types.
5. A breakdown of per-task accuracy. We have added a table that breaks down performance of all benchmarked models per task and allows detailed insights into each model’s strengths and weaknesses.

Additionally, we will include clarifications that reviewers requested, e.g. the comparison of our pipeline to pipelines used for other benchmarks, and details on our evaluation of the GEM metric. Again, we sincerely thank the reviewers for taking the time to review our paper and providing us with their valuable feedback.

This paper introduces NEPTUNE, a benchmark designed to evaluate multimodal understanding of long videos, addressing the limitations of existing datasets that primarily focus on short clips. The paper contributes a semi-automatic pipeline leveraging large video language models (VLMs) and large language models (LLMs) to generate dense, time-aligned video captions and challenging question-answer-decoy (QAD) sets for video segments up to 15 minutes long. Additionally, it proposes a new evaluation metric, GEM (Gemma Equivalence Metric), to assess open-ended responses in a static and reproducible manner. Reviewers commend the dataset's focus on long video reasoning, its innovative annotation pipeline, and the introduction of GEM, which tackles the limitations of prior evaluation metrics. However, concerns about the lack of comparisons with recent benchmarks (e.g., MLVU, Video-MME) and state-of-the-art open-source models (e.g., InternVL, LLaVA-OneVision) are raised. Reviewers also note potential biases in dataset creation due to reliance on VLMs and LLMs, limited novelty in the annotation pipeline, and insufficient evaluation against human judgment for GEM. While the contributions are valuable, the absence of more substantial comparisons and deeper analysis leaves room for improvement.

Reject