Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos

Thanks for your valuable comments. We respond to your concerns below,

1. Motivation of annotating audio text description: We appreciate your insight regarding the integration of audio modalities. Currently, our approach correlates speech semantics with visual content, serving as a foundational step towards multi-modal alignment. While we recognize the importance of incorporating auditory context for comprehensive multimedia understanding, this integration remains part of our future work. Our focus on visual-oriented alignment in the present study provides a structured basis upon which we aim to build more complex multi-modal reasoning capabilities.

2. Absence of a Self-Developed Model: Actually, we introduce the SUM-Shot models, explicitly designed for the Shot2Story benchmark. Unlike traditional holistic models that treat videos uniformly, SUM-Shot leverages structured prompting and a multi-shot framework to emphasize temporal sequences and contextual relationships between shots. This specialization allows the model to maintain narrative coherence and accurately track multiple events and actions across different shots. Our experimental results in Table 3 show that SUM-Shot significantly outperforms standard models, effectively handling the complexities inherent in multi-shot video narratives and showcasing the specific advantages of our benchmark.

3. Optimality of using LLMs for annotation and reasoning tasks: Thank you for addressing this important consideration. Our primary motivation is to demonstrate that the detailed summaries generated through our approach are versatile and effective across multiple downstream tasks. This effectiveness inherently signifies that our annotations are of high quality, providing a robust foundation for various applications in multi-modal understanding. To achieve this, we employed an off-the-shelf LLM for reasoning tasks.

   We acknowledge that a more collaborative approach—such as utilizing distinct specialized models for annotation and reasoning—could mitigate biases and further enhance multi-modal alignment, developing and integrating such systems falls outside the scope of our current work. Nevertheless, we recognize the value of this approach and consider it a promising direction for future research.

Thanks for your constructive feedback. We respond to your concerns as below

1. Baseline models across all tasks: We appreciate your observation regarding the consistency of baseline models across the three tasks. To clarify, we employed the same set of baseline models—MiniGPT-4 and Video-Chat2—across all tasks, including single-shot video captioning, video summarization, zero-shot question answering, and multi-shot video question answering, as detailed in Table 2 to Table 5. This consistent application ensures comparability and demonstrates the robustness of our Shot2Story dataset. The experimental progression follows a hierarchical training approach: video summarization models are pretrained on single-shot captions (Table 2), and subsequently, question-answering is performed with the video summaries from models in Table 3. This inherited and progressive methodology underscores the utility of our multi-shot annotation data in enhancing large video-language model training and highlights the high quality of our annotations. The consistent use of baseline models across all tasks facilitates clear and uniform observation of performance improvements, validating the effectiveness of our dataset in advancing multi-shot video understanding.
2. Annotation of transition between shots: We acknowledge the importance of capturing meaningful transitions and relationships between shots to enhance multi-shot temporal reasoning in large language models. Our dataset annotations have actually annotated detailed descriptions of shot transitions and progressions. These include explicit annotations of camera movements such as "camera zooms in/out," "camera pans," and "video cuts/shifts to," as well as progressive narrative phrases like "as the video progresses," "the video continues," and concluding statements like "video wraps with." These annotations go beyond simple transition words by providing specific information about the nature of the transitions and the continuity between shots, thereby ensuring a coherent temporal flow and interconnectivity. This structured approach effectively captures the temporal dynamics essential for developing robust multi-shot temporal reasoning capabilities in LLMs.
3. Understanding simpler and longer videos: We have conducted experiments using the ActivityNet-QA dataset, which comprises much longer videos with an average duration of approximately 10 minutes and contains continuous shots with minimal variations. As shown in Table 4, our models, trained exclusively on the Shot2Story multi-shot videos, achieved superior performance in generating video summaries compared to existing instruction-tuned models. This indicates that our models can effectively handle both simpler and longer videos without being adversely affected by potential biases introduced during training. Our approach to processing long videos involves utilizing the same pipeline as for multi-shot videos, to approximate detect shot boundaries (due to motions and variations), resulting in only a few shots (considering the much longer durations, compared to our multi-shot videos in averaged 17 seconds). These findings demonstrate that training on the Shot2Story dataset enhances the model's capacity to understand and process videos of varying lengths and complexities, thereby mitigating concerns about potential biases.

Thanks for the comments. Please check our replies to your concerns below:

1. Scientific novelty and contribution: We would like to clarify that ICLR does consider benchmark and dataset papers for publication, as highlighted in the Call for Papers for ICLR 2025. Benchmarks play a pivotal role in advancing research domains by providing standardized datasets that facilitate consistent evaluation and comparison of different models. Our contribution lies in the creation of Shot2Story, a large-scale, meticulously curated benchmark that addresses the complexities of multi-shot video understanding through detailed shot-level captions, comprehensive video summaries, and a challenging question-answering (QA) component.

2. Novelty of Shot2Story: Shot2Story distinguishes itself by integrating multiple modalities—combining audio narratives with visual content—to provide a richer and more nuanced understanding of videos beyond mere textual descriptions. This multimodal approach allows models to leverage both auditory and visual information, thereby facilitating more accurate and contextually relevant video summaries and QA responses. Additionally, Shot2Story introduces tasks that assess models on both captioning and QA, highlighting their ability to comprehend and generate descriptions as well as to reason about the content in a question-answering framework. Our benchmark also showcases superior zero-shot video question-answering performance, underscoring its potential to drive advancements in video understanding.

3. Architecture design for multi-shot video understanding: Beyond providing a comprehensive benchmark, we introduce the SUM-Shot models specifically designed for multi-shot video understanding. Unlike traditional holistic models that process videos uniformly, SUM-Shot utilizes structured prompting and a multi-shot framework to emphasize the temporal sequence and contextual relationships between shots. This design enables the model to maintain narrative coherence and accurately track multiple events and actions across different shots. Our experiments demonstrate that SUM-Shot models significantly outperform standard models, effectively handling the complexities of multi-shot video narratives.

Dear Reviewer G1Gp,

We deeply appreciate the constructive comments and the comprehensive review you’ve provided. As the rebuttal period concludes soon, we would like to confirm whether our responses have addressed your concerns. If any further clarification is necessary, we would be happy to discuss additional details with you.

Kind Regards!

Thank you for your comments. We reply to your concerns as following,

1. Common pitfalls in the generated summaries: Our dataset predominantly features human-centric activities due to our video filtering process, which selects videos rich in visually related audio information, and the sourcing from HDVILA, which primarily curates content related to human activities. Consequently, our annotations tend to highlight salient events and large-scale objects essential to the video’s storyline, mentioning smaller objects only when they directly contribute to the narrative. Common pitfalls in the generated summaries include the omission of minor yet contextually important details and a bias towards emphasizing more prominent actions or objects, potentially overlooking less conspicuous elements.

2. Challenges posted by shot transitions and multi-event progression: Shot2Story introduces unique and complex challenges that stem from its emphasis on shot transitions and multi-event progression, setting it apart from existing benchmarks like MSRVTT and ActivityNet.

• Taking question-answering task as example, unlike MSRVTT-QA, which poses questions such as "Who do three judges talk to?" (Figure 19) , or ActivityNet-QA's general inquiries like "What is the person in the video doing?" (Figure 20), that primarily assess the model's ability to understand interactions at one timepoint, Shot2Story-QA delves deeper into the temporal and causal dynamics of multi-shot videos, by posing "Temporal-related" and "Multi-shot Holistic Understanding" questions (two of our QA question types).
• For instance, a temporal-related question like "What is the man's immediate action after handling the skewer?" (Figure 21) requires the model to not only identify specific actions but also comprehend the sequence and progression of events within the video. This necessitates a sophisticated understanding of narrative flow and the ability to link consecutive actions meaningfully.
• Moreover, multi-event progression within a single video requires models to accurately capture and represent multiple concurrent events, ensuring that each event is understood in its proper temporal context. Shot2Story-QA's multi-shot holistic understanding question "How does the setting change from the start to the end of the video?" requires the model to understand the multi-shot progression and the event within each shot.

Shot2Story-QA provides a more rigorous and nuanced evaluation framework that pushes the boundaries of current video understanding and summarization models beyond the capabilities assessed by benchmarks like MSRVTT-QA and ActivityNet-QA.

3. Information redundancy when compiling multi-shot summaries: Regarding potential information redundancy when compiling multi-shot summaries, we define redundancy based on the temporal and contextual relevance of shots. Specifically, if two shots with similar content are not consecutive, their inclusion in the summary is not deemed redundant but rather indicative of a coherent event progression. To prevent immediate repetition, our data filtering process ensures that consecutive shots do not contain similar content, as detailed in Lines 129-130 of the manuscript. By maintaining diversity between consecutive shots and preserving the narrative flow across non-consecutive similar shots, we minimize information fragmentation and ensure comprehensive coverage of the video’s storyline. This approach allows us to capture essential elements without unnecessary repetition, thereby enhancing the quality and cohesiveness of multi-shot summaries.

4. Video hallucination: Thank you for highlighting the issue of hallucinated details in video summaries. Although mitigating hallucinations was not the primary focus of our study, we acknowledge its importance and have conducted an analysis of the hallucination patterns observed in our models. We identified two main types of hallucinations: (1) Multimodal Alignment Errors, where the model incorrectly associates visual elements with textual descriptions, such as inaccurately highlighting the "black" color in Figure 5, and (2) Hallucinations with Repeated Details, where similar visual contexts in different shots lead to redundant or incorrect descriptions. As shown in Figure 21, "woman holding a bowl with almonds" only appears in the last shot, while the model erroneously describes the woman as "holding something with almonds" in both the initial and ending shots of Figure 21 SUM-shot models. Moving forward, we plan to explore advanced multimodal alignment techniques in future research.

Dear Reviewer 5bm5,

We sincerely appreciate the time and effort you've dedicated to reviewing our paper, as well as your insightful and constructive feedback. As the rebuttal period is drawing to a close, we kindly ask if our responses have sufficiently addressed your concerns. Please feel free to reach out if further discussion or clarification is needed; we are more than willing to engage with you.

Kind Regards!