Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs

We would like to begin by expressing our sincere gratitude for your thorough review of our paper. We greatly appreciate your suggestions, which are crucial in improving the quality of our paper. The questions you raised are insightful, and we believe we have carefully clarified and addressed them as follows.

W1&Q1: While the synthetic nature is a strength in terms of scalability and avoiding data leakage, it also limits the ecological validity of the benchmark. Performance on VNBench doesn't guarantee real-world performance. The authors acknowledge this, but a correlation computation between the given benchmark and other real-world ones is needed.

R1&A1: In response to this concern, we add a new section, "cross-task correlation analysis," in Appendix B.2 to provide a detailed correlation computation between VNBench and other real-world benchmarks. Specifically, we evaluated 6 models with varying frame input sizes (16, 32, 48, 64, 96, and 128 frames) as discussed in Section 6. These models were assessed on several real-world video understanding benchmarks, including VideoMME [1], MLVU [2], and LongVideoBench [3].

Table: Evaluation results on VNBench and real-world benchmarks.

For each benchmark, we calculated the correlation coefficients of the corresponding performance vectors. The results show that all benchmarks achieved correlation coefficients above 0.75. Notably, VNBench demonstrated correlation coefficients higher than 0.9 with MLVU and VideoMME, two commonly used real-world benchmarks. This indicates that VNBench can, to a certain extent, reflect a model's ability to process real-world videos. We show the cross-task correlation matrix below and visualize the correlation matrix in Figure 15.

Table: Cross-task correlation matrix among VNBench and real-world video understanding tasks.

[1] Fu, Chaoyou, et al. "Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis." arXiv preprint arXiv:2405.21075 (2024).

[2] Zhou, Junjie, et al. "MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding." arXiv preprint arXiv:2406.04264 (2024).

[3] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." arXiv preprint arXiv:2407.15754 (2024).

W2&Q2: The reliance on CLIP for filtering needle images introduces a potential bias. The performance of the CLIP model itself could influence the results, especially on synthetic videos. A sensitivity analysis or exploration of alternative filtering methods would be valuable.

R2&A2: We conducted a human evaluation on 300 sampled cases and compared the results with those obtained using the CLIP thresholding strategy before we chose to adapt this CLIP-based filtering method, which will be shown in the Appendix of our revised manuscript. Only 2 out of 300 samples showed differing conclusions, demonstrating the effectiveness of the CLIP filtering method.

W3: Wrong single quotation marks in L18, P1.

R3: Thank you for your suggestion. We have addressed the issue and made the necessary corrections in the revised version.

Q3: How robust are the results to variations in the types and complexities of the "needles"?

A3: Thank you for your question. In Appendix F.2, we include an additional set of experiments where the needle content is replaced with simpler alternatives, such as fruit images or fixed subtitles. The results, presented in Table 8 and below (SRC for VNBench and NEW for the alternative one), show a strong correlation between the test results, suggesting that the use of simple and easily recognizable needles ensures the robustness of our results.

Additionally, during the design of VNBench, each task type includes three subcategories corresponding to different needle complexities and types. In these cases, as shown in Table 2, the complexity of the needles introduces some variability in the results. This is because the differences between subcategories are not limited to needle content but also extend to needle formats, which test diverse aspects of video understanding. For instance, subtitle needles assess localized recognition, while image needles evaluate global semantic comprehension. This variability highlights the ability of VNBench to probe different facets of model performance.

Q4: Could the VideoNIAH framework be adapted to evaluate other multimodal tasks beyond video understanding?

A4: Thank you for your insightful question. The VideoNIAH framework can indeed be adapted to evaluate a broader range of multimodal tasks beyond video understanding. For instance, in high-resolution image understanding, localized edits could be introduced by inserting small image patches as needles. Similarly, in the context of video and audio analysis, a short audio segment could be inserted into a video as a needle to evaluate multimodal integration.

We plan to further explore and extend the applicability of the VideoNIAH framework in future work to enhance its versatility and support a wider range of multimodal evaluation scenarios.

Dear Reviewer 9VkG,

We have carefully reviewed your comments and suggestions on our manuscript and have submitted our rebuttal addressing the points raised. We highly value your feedback and would be very grateful if you could take a moment to review our responses.

We sincerely appreciate your time and consideration in providing further feedback.

Best regards,

Authors

Dear Reviewer 9VkG,

We sincerely appreciate your time and efforts in reviewing our paper.

We have carefully addressed the suggestions and concerns raised by the reviewers. In particular, we have added more detailed revisions based on your feedback, including a robust analysis on needle content, needle type, and sample numbers, as well as a correlation analysis between VNBench and real-world benchmarks.

As the rebuttal deadline approaches, we kindly request that you provide your feedback on the revised version of the paper at your earliest convenience. Your input is invaluable, and we greatly appreciate your timely response.

Thank you once again for your time and thoughtful review. We look forward to hearing from you soon.

Best regards,

Authors

We would like to begin by expressing our sincere gratitude for your thorough review of our paper. We greatly appreciate your suggestions, which are crucial in improving the quality of our paper. The questions you raised are insightful, and we believe we have carefully clarified and addressed them as follows.

W1: While VNBench assesses important aspects of video understanding, it may not cover all possible video-related tasks and scenarios.

R1: As stated in our paper, we acknowledge that VNBench is not a fully comprehensive benchmark. The goal of the proposed VideoNIAH method is to decompose the capabilities of video understanding models, enabling focused evaluation of specific abilities, such as temporal ordering (ordering task) or spatio-temporal coherence perception (counting task). By decoupling different aspects of video understanding, we aim to better identify the particular shortcomings of existing models.

On the other hand, the three sub-tasks of VNBench (retrieval, ordering, and counting) align with key capabilities in current video understanding. This is further supported by the strong correlation between VNBench evaluation results and those of several real-world benchmarks. In Appendix B.2, we provide a detailed analysis of VNBench’s correlation with real-world video benchmarks VideoMME[1], MLVU[2], and LongVideoBench[3]. Our results show that VNBench closely aligns with the evaluation of these real-world benchmarks. Moreover, the performance across VNBench’s three sub-tasks offers valuable insights into where models may be lacking in specific abilities.

Table: Cross-task correlation matrix among VNBench and real-world video understanding tasks.

[1] Fu, Chaoyou, et al. "Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis." arXiv preprint arXiv:2405.21075 (2024).

[2] Zhou, Junjie, et al. "MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding." arXiv preprint arXiv:2406.04264 (2024).

[3] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." arXiv preprint arXiv:2407.15754 (2024).

W2&W3: The overall tasks in the proposed evaluation framework are relatively simplistic and with low cost. Models may develop strategies to perform well on the tasks without truly understanding the video content, thus compromising the reliability of the evaluation. After targeted training, models can achieve high scores with relative ease, which may not accurately reflect their ability to handle more complex and diverse real-world video understanding scenarios.

R2: First, the goal of the proposed VideoNIAH method is to assess specific aspects of a model’s capabilities, as a tool for analyzing strengths and weaknesses. We do not encourage models to simply "optimize" for scores on distribution-matching data.

Additionally, VNBench is categorized according to different capability dimensions, allowing us to focus on evaluating particular aspects we aim to explore, such as temporal and spatio-temporal coherence. The tasks in VNBench are designed to capture what we consider essential capabilities for current video understanding. As a result, VNBench exhibits a relatively low bias towards real-world scenarios, as shown in the correlation analysis in R1.

Moreover, all benchmarks, including VNBench, have inherent biases. Typically, when training large multimodal models, we do not rely on a single benchmark metric as the evaluation criterion. Instead, we adopt a multi-benchmark approach for comprehensive evaluation. This means that VNBench is generally used in conjunction with other benchmarks to provide a more accurate evaluation of a model’s true capabilities. The strength of VNBench lies in its ability to decouple specific model capabilities, enabling targeted optimization of particular skills.

Thank the author for the rebuttals. My main concerns have been addressed, so I maintain my score as 6. Looking forward for your open-sourced codebase and datasets.

We would like to begin by expressing our sincere gratitude for your thorough review of our paper. We greatly appreciate your suggestions, which are crucial in improving the quality of our paper. The questions you raised are insightful, and we believe we have carefully clarified and addressed them as follows.

W1: Although the authors mentioned the proposed framework is inspired by language model evaluation. It is still not very clear about the motivation for proposing such a synthetic framework for video benchmarking and why the proposed one is necessary or important.

R1: The motivation for proposing VideoNIAH is to address the inefficiencies in evaluating video models during iterative development due to the high cost of constructing datasets and the difficulty in isolating specific skills.

VideoNIAH introduces several key differences compared to previous video benchmarks:

• Skill-Specific Evaluations: By inserting multiple spatio-temporal "needles" and generating corresponding query-response pairs based on predefined rules, VideoNIAH enables the precise identification of a model's weaknesses in specific video understanding capabilities. The queries in VideoNIAH focus on specific aspects of video understanding, such as temporal perception, chronological ordering, and spatio-temporal coherence, allowing for more granular evaluation compared to benchmarks that assess general video understanding capabilities.

• Less Data Leakage: Synthetic video generation provides more control over the construction of the benchmarks and eliminates the risk of data leakage.

• Scalability and Efficiency: VideoNIAH automates the generation of query-response pairs using predefined rules, minimizing manual labor compared to benchmarks that require extensive manual annotation and data filtering. It allows for the use of diverse video sources with flexible lengths, offering significant scalability and efficiency.

W2: The proposed framework mainly adopts subtitles and unrelated images as the "needle", it is better to discuss whether there are other types of "needle" that can be used.

R2: We include the results of VNBench-Act in Appendix E. VNBench-Act is a synthetic benchmark constructed using the VideoNIAH method. In VNBench-Act, we aim to evaluate the insertion of short video clips containing continuous natural frames from Something-Something V2[1]. This ensures that the inserted needle not only carries static image-level semantics but also conveys short-term action meanings.

Table: Evaluation results on VNBench-Act, which use short action video clips as needles.

It is worth noting that when the needles are extended to short video clips, the task becomes more challenging, especially for ordering tasks. This highlights limitations in modeling temporal action sequences effectively.

[1] Goyal, Raghav, et al. "The" something something" video database for learning and evaluating visual common sense." Proceedings of the IEEE international conference on computer vision. 2017.

W3: A small number of video samples and the types of selected video datasets are used to build the benchmark. In addition, only three tasks are adopted to evaluate the video MLLMs. It is better to make the benchmark more scalable and diverse with comprehensive evaluation capabilities.

R3:

Thank you for your constructive feedback. As detailed in Appendix F.3, we double the number of test samples using the VideoNIAH framework. We evaluated the impact of the increased sample size on the average accuracy across the three VNBench tasks: Retrieval, Ordering, and Counting. Our analysis in Table 9 and below (SRC for VNBench and MORE for the expanded one) indicates a relatively high correlation coefficient between the results of the original dataset and the enlarged benchmark, demonstrating the robustness of our benchmark with respect to the current sample size.

Additionally, we appreciate your suggestion regarding diversity. In future work, we plan to further expand the types of tasks included in VNBench to enhance its comprehensiveness and evaluation capabilities.

Q1: The paper mentioned the used 'needles' are not related to the original videos. Did you conduct the similarity calculation between each video frame and the candidate "needle" before inserting them?

A1: As mentioned in Section 3.3, we utilized the CLIP model to calculate the similarity between the inserted needle and all frames of the video haystack, and applied a threshold-based strategy to filter relevant samples. Additionally, we conducted a human evaluation on 300 sampled cases and compared the results with those obtained using the CLIP thresholding strategy, which will be shown in the Appendix of our revised manuscript. Only 2 out of 300 samples showed differing conclusions, demonstrating the effectiveness of the CLIP filtering method.

Q2: Inserting the unrelated visual "needle" may not make the benchmark very challenging. Did you try the "needles" that are similar to the original video frame to a certain extent but are from a different category or situation? Maybe it will make the benchmark more challenging.

A2: We appreciate the reviewer’s suggestion, but we would like to clarify a few points.

First, the goal of VideoNIAH is to decouple image recognition from video-specific temporal abilities. For this reason, we specifically chose unrelated "needles" to evaluate a model’s targeted video capabilities without introducing additional complexity related to visual similarity.

Second, the benchmark was designed with graded difficulty within tasks and variability across task types, which ensures that it remains challenging. Even state-of-the-art closed-source models, such as GPT-4 and Gemini 1.5 Pro, do not achieve perfect performance, further highlighting the benchmark’s challenge.

Furthermore, inserting similar video frames could indeed increase the task difficulty by making the target distinction harder. However, this would complicate our ability to isolate and explore the specific video understanding capabilities we aim to assess. In this case, the performance degradation on the benchmark would result from more complex factors, making it less effective for evaluating the targeted abilities.

Thank you for your response. My concerns have been addressed. I'm willing to raise my rating to 6.

We would like to begin by expressing our sincere gratitude for your thorough review of our paper. We greatly appreciate your suggestions, which are crucial in improving the quality of our paper. The questions you raised are insightful, and we believe we have carefully clarified and addressed them as follows.

W1: The rule-based automation process for dataset curation is unclear. For example, Figure 1(b) lacks clarity in illustrating this process, which is essential to understanding the scalability benefits of this approach, which can be improved.

R1: The rule-based automation process involves designing query templates based on the task type and needle type. For example, for a task of the 'order' type and a needle of the 'subtitle' type, the query might ask, 'What is the order of subtitles appearing in the video?' The corresponding options and ground truth are directly determined by how we insert the needles and their content, which is known in advance.

Since the questions depend solely on the type and method of needle insertion, generating query-response pairs can be fully automated based on predefined rules. We update Figure 1(b) to include some example rule types. However, we recommend readers refer to Appendix A.4 and Figure 13 for more details on the rules to generate queries and responses.

W2: While the method is scalable and captures the temporal dependency well, the scope of video understanding through this framework seems limited. For example, understanding actions.

R2: We include the results of VNBench-Act in Appendix E. VNBench-Act is a synthetic benchmark constructed using the VideoNIAH method. In VNBench-Act, we aim to evaluate the insertion of short action video clips containing continuous natural frames from Something-Something V2[1]. This ensures that the inserted needle not only carries static image-level semantics but also conveys short-term action meanings.

Table: Evaluation results on VNBench-Act, which use short action video clips as needles.

It is worth noting that when the needles are extended to short video clips, the task becomes more challenging, especially for ordering tasks. This highlights limitations in modeling temporal action sequences effectively.

[1] Goyal, Raghav, et al. "The" something something" video database for learning and evaluating visual common sense." Proceedings of the IEEE international conference on computer vision. 2017.

W3: Given the nature of the benchmark, the sampling strategy of the video MLLM I believe plays a significant role in its performance. Clarification on how this issue is addressed, or suggested solutions, would strengthen the framework’s reliability.

R3: Thank you for your insightful comment. We acknowledge that the sampling strategy of the video MLLM can indeed influence performance. To address this, we set the interval for inserting the needle to 1 second, ensuring that if the model samples at 1 frame per second (fps), it will capture the needle in the input.

Currently, some models (e.g. Video-LLaVA, VideoChat2, ST-LLM, .etc) do not operate at 1fps, primarily due to computational limitations, which is a inherent shortcoming for achieving better video understanding ability. Dense sampling is fundamental for comprehensive video understanding, while sparse sampling can lead to the neglect of crucial video details.

On the other hand, we observe that current state-of-the-art models, such as GPT-4 and Gemini 1.5 Pro, operate at 1fps, which means no needles are missed during VNBench evaluation. However, these models still struggle to achieve optimal performance, suggesting advanced improvement on video understanding in future.

W4: Can the authors explain if each cell in Figure 8 corresponds to a single test sample? It would be better to enhance this image with a heat map to better understand how the depth of the needle impacts the MLLMs performance. In addition, is there a reason that a depth higher than 10% was not considered?

R4: Thank you for your valuable suggestions. In the original figure, each cell indeed corresponds to a single test sample. We revise Figure 8 into heatmaps to show the modal ability more accurately. Now, we construct 32 samples, each corresponding to needle depths ranging from 0% to 90% (at 10% intervals) and haystack durations from 10s to 180s (at 5s intervals) with the same needle content. For each cell position, we report the average accuracy across the 32 samples.

Additionally, the depth range in the original figure was incorrectly labeled as 0-9; the correct range is from 0% to 90%, with intervals of 10%.

W5: Section 6: The mention of the "training recipe" lacks context on what is being trained, creating some confusion. A clearer introduction to this section would aid comprehension.

R5: We have provided details of the training process in the appendix F, including the model architecture, training data, and training hyperparameters. We also add more details in appendix F, including input format and loss functions.

Q1: Counting E2 Task: What factors might explain the poor performance of Gemini and GPT-4 on the counting E2 task (Figure 5)?

A1: The poor performance of Gemini and GPT-4 on the Counting E2 task can be attributed to two main factors related to the task's requirements for both temporal and spatial co-occurrence:

1. Poor temporal co-occurrence understanding: Both Gemini 1.5 Pro and GPT-4 performed suboptimally on other tasks that only require temporal co-occurrence awareness, such as Counting-E1 and Counting-I. This suggests that the models may struggle with tracking temporal relationships effectively.
2. Weak spatial co-occurrence perception: We conduct a simple experiment to evaluate GPT-4o's ability to perceive spatial co-occurrence. Specifically, we select an image and randomly attached 0 to 4 smaller fruit images to different regions, which can be regarded as a image-level NIAH test. We then asked GPT-4o to predict the number of attached fruit images on 100 curated samples, and it achieved an accuracy of 72/100. In the Counting E2 task, where 4 different clips in the video sequence each have image needles (needle number randomly choice from 0 to 4 per clip) attached, the accumulated accuracy would be 0.72^4≈0.26. This probabilistic estimation indicates that spatial co-occurrence understanding might also be a limiting factor for these models on this task.

Q2: Consistency Check: Could the authors verify the data in Figure 5 and Table 2? For instance, Video-LLaVa and VideoChat2 seem to perform better than Gemini on the counting E2 task in Figure 5, but Table 2 suggests otherwise.

A2: We appreciate the reviewer’s suggestions. During the plotting process, a small portion of the data (specifically, counting-edit-level2 and counting-insert for certain models, e.g., Video-LLaVA, VideoChat2) was mistakenly mixed. We correct the results in Figure 5 of the manuscript. Additionally, all numerical results were already provided in Table 11 of Appendix G. Thanks for you advice!

Q3: Haystack Length Variation: In Section 5, where the "Effect of Needle Position" is discussed, the authors mention that the haystack is fixed. Could the authors clarify how the length of the haystack is adjusted in this scenario?

A3: We selected a fixed video segment and randomly cropped it to obtain haystacks of varying lengths. For the position test, we used the same cropped video segment for haystacks of the same length, altering only the depth of inserted needle .

Q-IV: The performance drop to 0% on Counting-E2 for GPT-4 is especially puzzling, given its apparent advantage in sampling strategies over other models. Further clarification on this discrepancy would be valuable.

A-IV:

Thank you for your insightful comment. The poor performance of GPT-4 on the Counting-E2 task indeed worths further investigation.

• In our experiments, the input prompts were kept consistent across all models. For both the GPT-4 series and Gemini 1.5 Pro, we applied the same frame sampling strategy and query format. Therefore, we believe there were no issues in the testing procedure that could have influenced the results.

• Interestingly, we found that all models performed poorly on Counting-E2, with even the best model (LLaVA-OneVision-72B) achieving only 14.6% accuracy on this task, which is lower than best accuracy on other sub-tasks. While counting is relatively easy for humans, it proves more challenging for multimodal large language models at the spatial level, as noted by Molmo[1]. We will also conduct more comprehensive experiments and analysis on the counting ability in video MLLMs base on the VideoNIAH framework.

[1] Deitke, Matt, et al. "Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models." arXiv preprint arXiv:2409.17146 (2024).

Q-V: Figure 5 contains a few inconsistencies that should be addressed. For example, some plots are missing models that could provide a more comprehensive view. Ensuring consistency between plots and tables is essential.

A-V: Apologized for the mistakes! We modify Figure 5 in the revised version. And we also conclude all model's performance in Table 11 and Figure 5. Once again, thank you for your thorough review, which has been a great help in improving our paper.

Our conclusions are as follows:

1. When the conditional probability Acc.|Consist. is significantly higher than 25% (roughly equivalent to random guessing), the accuracy can effectively reflect the model's ability to handle the task. In Counting-E1 and Counting-I, the results from the model performance comparison are valuable.
2. When the conditional probability Acc.|Consist. fluctuates around 25% or falls below 25%, comparisons of accuracy have limited significance. In this case, the conclusion is that the models are unable to solve the task, meaning that all models perform unsatisfactorily on Counting-E2. This represents a common issue across all models and indicates an area where VideoMLLMs need optimization.
3. For GPT-series models, we find that compared to Gemini 1.5 Pro and LLaVA-OneVision series models, their Consistency Rate on unsolvable tasks (e.g., Counting-E2) is very low (5.3% for GPT-4o, 3.6% for GPT-4-turbo). This suggests that these models are highly sensitive to the order of options, resulting in a 0.0% accuracy in circular evaluation. While other models also fail to provide correct answers (reflected by the conditional probability Acc.|Consist. around 25%, roughly equivalent to random guessing), their higher Consistency Rate leads to slightly higher Acc. values. However, as concluded in point 2, such comparisons are not significant. This phenomenon can also be validated in Table 7, where GPT-4o shows performance closer to Gemini 1.5 Pro with fewer evaluation iterations.
4. The low Consistency performance of GPT-series models is seen only in tasks they cannot solve. On tasks they can handle, their Consistency Rate is high (in Table 12, 13).

The above conclusions suggest that for VNBench, comparisons at high Acc. levels reflect the models' abilities, while low Acc. levels indicate a common issue for all Video MLLMs. In fact, this conclusion holds true for most multi-choice QA benchmarks. Meanwhile, inspired by the reviewer's advice, we believe that Acc.|Consist. may also be an important metric in circular evaluation.

[1] Wei, Sheng-Lun, et al. "Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models." arXiv preprint arXiv:2406.03009 (2024).

[2] Wang, Haochun, et al. "Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models." arXiv preprint arXiv:2402.01349 (2024).