TeaserGen: Generating Teasers for Long Documentaries

We appreciate the reviewer for the thoughtful review and constructive comments. Please find our response below.

[Weakness 1]:

We would like to highlight that we also introduce TeaserGen-LR, a learning-based end-to-end architecture designed for this task. In this paper, we focus on the descriptive and precise aspects of teaser generation and present several metrics to evaluate its performance. To ensure factual accuracy, particularly in educational documentaries, we advocate for extracting visual content rather than generating it. Creativity in our approach arises from engaging narration rather than visual content. Although this paper does not explore music or sound effects, our publicly available dataset, which spans multiple modalities, provides a solid foundation for future research in areas such as long-form video-to-audio generation[1,2,3] audio separation [4], and trailer generation [5]. In this paper, we limit our scope more on the non-creative aspects of the task of teaser generation in generating only the narration and the accompanying visuals, and we believe our proposed dataset can enable future research on modeling the creative aspects of this topic.

[Weakness 2]:

We thank the reviewer for raising this concern. As mentioned in the discussion section, we are interested in generating visually-aligned sound effects and background music as a next step. We also note that this is an active area of research [2, 6, 7]. Given the complexity of this task, we chose to focus on narration and visual generation in this paper as an initial step toward the ultimate goal of generating teasers with visually-aligned sound effects and background music. We evaluate the output from the first stage, i.e the generate narrations in our subjective test. Our subjective (see the attached table) test shows that generated narrations are coherent, informative, and engaging. In our approach, we sought to mitigate these losses by fine tuning the pretrained model in Table 9 (see the attached table). However, we recognize that these strategies may not fully eliminate the issue, therefore, we proposed a learning based method to directionally mapping narrations to visual contents. The F1 score of the learning-based approach TeaserGen-LR is higher than the F1 score of the interval-based approach TeaserGen-PT.

Table 9:

[Weakness 3]:

We observe that scene changes occur more frequently "within sentences" in the ground truth teaser, and thus we leverage the narration to ensure continuity and coherence of the narrative in the proposed two-stage system. On average, there are 2.84 scene changes per sentence, meaning scene changes typically happen within sentences. To ensure the flow of generated scripts, we design careful prompts. For example, we convert the summary of each chunk into a story using prompts such as: “Rewrite the paragraph into an engaging story opening in 10 sentences or less, keeping all names and avoiding the use of pronouns.” Our subjective test in Section 5.5 (see the attached table) demonstrates that our generated narrations are coherent, informative, and engaging.

For the narration-visual matching process, to avoid frequent scene changes and overly-fragmented visuals, we have proposed an interval-based binary search algorithm in TeaserGen-PT that does not allow selecting video clips shorter than 3 sec.

[1]: Polyak et al., Movie Gen: A Cast of Media Foundation Models, ECCV, 2020 (https://arxiv.org/abs/2410.13720).

[2]: Tian et al., VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling, Arxiv, 2024(https://arxiv.org/abs/2406.04321).

[3]: Li et al., MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization, Arxiv, 2024 (https://arxiv.org/abs/2410.12957).

[4]: Huang et al., High-Quality Visually-Guided Sound Separation from Diverse Categories, Arxiv, 2024 (https://arxiv.org/abs/2308.00122).

[5]: Argaw et al., Towards automated movie trailer generation, CVPR, 2024 (https://arxiv.org/abs/2404.03477.)

[6]: Kang et al., Video2Music: Suitable music generation from videos using an Affective Multimodal Transformer model, Expert Systems with Applications, Arxiv, 2023 (https://doi.org/10.1016/j.eswa.2024.123640).

[7]: Lee et al., Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition for Foley Sound, Arxiv, 2024 (https://arxiv.org/abs/2408.11915).

[8]: Lin et al., UniVTG: Unified Video-Language Temporal Grounding, ICCV, 2023 (https://arxiv.org/abs/2307.16715)

[9]: Son et al., CSTA: Spatiotemporal Attention for Video Summarization, CVPR, 2024 (https://arxiv.org/abs/2405.11905)

[10]: He et al., Align and Attend: Multimodal Summarization with Dual Contrastive Losses, CVPR, 2023 (https://arxiv.org/abs/2303.14150)

[Weakness 4 and Question 2]:

We would like to highlight that we conducted an ablation study on the diffusion prior in Section 5.6, Table 6 (attached below). Our ablation study indicates that applying the diffusion prior leads to lower repetitiveness and higher language-vision correspondence, as reflected by higher CLIP and VTGHLS scores. The decreased repetitiveness may result from the sampling process in the diffusion prior, which is a generative model that produces diverse image embeddings for the same input text embeddings. The increased CLIPScore likely stems from the diffusion prior's ability to bridge the gap between textual embeddings and visual embeddings. Comparing the two results of TeaserGen-LR using beam search decoding, we also observe that applying the diffusion prior results in a higher scene change rate, possibly because the diffusion prior further encourages scene transitions.

To demonstrate the robustness of our model, we also report zero-shot examples on TED Talks and old movies, which are showcased on our demo page (https://54321anonymous.github.io/ICLR2025/). Our training samples include documentaries from various genres, such as nature scenes and social events. We observe that the model can effectively generate teasers for TED Talks and old movies, even though it has never been trained on these domains, highlighting the robustness of the proposed methods. Below are the objective results of the zero-shot examples. We further compare TeaserGen-PT and TeaserGen-LR with two extra video summarization models: CSTA[9] and A2Summ[10] in zero-shot scenarios in the table below.

[Weakness 4 & Question 1]:

We utilize a video temporal grounding pretraining model, UniVTG [8], to generate a score curve that measures frame importance (defined by its likelihood of being included as a highlight in video highlight detection) in addition to text-visual correspondence. For each sentence $s_i$​, we generate audio using a pretrained text-to-speech model (https://platform.openai.com/docs/guides/text-to-speech). Our objective is to determine a threshold on the score curve, where the x-axis represents video time, and the y-axis represents the score. The goal is to select the start and end times of video clips such that the scores within these clips exceed the threshold. Multiple clips may satisfy this criterion, and we ensure that the total length of the selected clips matches the duration of the generated audio. Furthermore, we ensure each selected clip exceeds a minimum length and resolve overlaps with preceding sentences. As the threshold is lowered, the total length of the selected video clips increases monotonically. To maintain computational efficiency, we employ a binary search to determine the optimal threshold.

[Question 3]:

Thank you for pointing that out! We recognize the importance of addressing potential risks associated with copyrighted content, as noted in our ethics statement: "Our generative model, trained on copyrighted material, carries a risk of generating content that could potentially infringe on copyright." To address this concern, we have taken several steps to mitigate potential risks. First, we do not release raw video data from the dataset to avoid unauthorized redistribution of copyrighted content. Instead, we provide publicly accessible YouTube links along with our annotations, ensuring that users can access the data within the bounds of the original platform's terms of service. Furthermore, our use of the dataset complies with fair use principles, focusing on research purposes such as analysis, annotation, and model development. We are also open to feedback on additional steps we can take to improve transparency and ethical considerations in this area.

We appreciate the reviewer for the thoughtful review and constructive comments. Please find our response below.

[Weakness 1 & Question 1]:

Thank you for pointing that out! $s_{i}​$ is a sequence of language tokens, and S = {$s_{1}$, $s_{2}$ ... $s_{m}$} represents a sequence of sequences of language tokens. We use TTS to generate the audio waveform for each sentence and denote $\tau_{i}$ as the length of the generated audio waveform for the sequence of language tokens $s_{i}$

[Weakness 2 & Question 2]:

Yes. We conducted an ablation study on matching score function in section 5.6 and also proposed a learning based approach TeaserGen-LR that leverages transformer architecture. We conduct ablation study on TeaserGen-LR by experimenting with decoding strategy and diffusion prior module.

In Table 3(we attached table below), we report the objective result of using CLIPScore as the matching metric.

In Table 3 (see the table below), we also report the ablation study on TeaserGen-LR, experimenting with the decoding strategy and the diffusion prior module.

[Weakness 2 & Question 3]:

During our experiments, we aimed to avoid having clip lengths that were too short, as this would make the output overly fragmented. However, setting the minimum clip length too high would exclude too many clips, thereby sacrificing audio-visual correspondence. We conducted an ablation study on different combinations in the following table for TeaserGen-PT with textual narration being the query. We observe that selecting 3 as our minimum clip length and setting overlapping tolerance being 1 results in a scene change rate that is closest to the ground truth and provides higher audio-visual correspondence.

[Weakness 3 & Question 4]:

No, we did not experiment with higher frame rates due to limited computing resources. We follow LfVS[1] and UniVTG[2] to use fps=1, and we find it to be working well in practice. Moreover, from Figure 1, we can clearly observe that a frame rate of 1 fps is sufficient to capture most scene changes. We believe that by increasing the frame rate, we will be able to capture better motion and achieve prediction accuracy, but it would also introduce some optimization challenges as well.

[Question 9]:

We follow TGT[4] and LfVS [1] to use L2 loss in the embedding space. We are mapping textual embedding to image embedding with transformer architecture. The L2 loss is the MSELoss between predicted image embedding and ground truth embedding rather than in the pixel space.

We would like to point out that we did not frame the textual narration-visual matching task as a generative task. Rather, we framed it as a retrieval task where we aim to retrieve the most relevant frames to accompany the LLM-generated textual narrations.

[Weakness 7 & Question 10]:

Here are the details of dataset splitting:

Our training samples include documentaries from various genres, such as nature scenes and social events. We also report zero-shot examples on TED Talks and old movies, which are showcased on our demo page(https://54321anonymous.github.io/ICLR2025/). We report the objective evaluation results on zero-shot examples in the table below. We observe that the models can effectively generate teasers for TED talks and old movies as well, while the models have never been trained on these domains, which showcases the robustness of the proposed methods.

[Weakness 8 & Question 11]:

The number of trainable parameters of TeaserGen-LR without the diffusion-prior model is 20.4M, where TeaserGen-LR with the diffusion-prior model has 549.8M trainable parameters. All experiments were done with one single NVIDIA RTX A6000 GPU.

[Weakness 8 & Question 12]:

We would like to point out that there are on average 2.84 scene changes per sentence, i.e., scene changes usually happen within sentences. Thus, our proposed method relies on textual narration in ensuring the overall coherence of the narrative, where the visuals are selected to accompany the LLM-generated textual narration. We do not want that high scene change rate as it will lead to fragmented, unengaging visuals. As shown in Table 4 and Table 5 (We attached the table below), we conduct subjective evaluation on both textual narrations and generated teasers. The subjective test shows that our generated teaser narrations are coherent, informative, and engaging, and that teasers generated by TeaserGen are overall coherent.

Table 4:

Table 5:

[Weakness 2 & Question 13]:

As detailed in Section 5.1, we introduce two independent methods, TeaserGen-PT and TeaserGen-LR, to ensure consistent matching of visual content. Specifically, TeaserGen-PT employs CLIP-ViT-B/32 to extract visual and sentence embeddings, chosen to match the embedding size of the pretrained video temporal grounding model. In contrast, TeaserGen-LR utilizes CLIP-ViT-L/14, which aligns with the embedding size required by the pretrained diffusion prior.

Reference:

[1] Argaw et al., Scaling up video summarization pretraining with large language models, CVPR, 2024(https://arxiv.org/abs/2404.03398)

[2] Lin et al., UniVTG: Unified Video-Language Temporal Grounding, ICCV, 2023 (https://arxiv.org/abs/2307.16715)

[3] Chin-Yew Lin., ROUGE: A Package for Automatic Evaluation of Summaries, Text Summarization Branches Out, 2004. (https://aclanthology.org/W04-1013.)

[4] Argaw et al., Towards automated movie trailer generation, CVPR, 2024(https://arxiv.org/abs/2404.03477.)

[Weakness 3 & Question 5]:

Yes, we propose different methods to address the issue of repetitive scenes for TeaserGen-PT and TeaserGen-LR and enhance diversity and temporal coherence.

1. TeaserGen-LR: In section 4.2.2, we incorporate diffusion prior to encourage diversity within one sentence. In addition, diffusion prior helps to bridge the gap between textual embedding and frame embeddings. The transformer model can capture temporal coherence.
2. In addition, for the proposedTeaserGen-PT model, Our interval based approach aims to select intervals(video clips) by thresholding for each sentence. Therefore, no repeated frames will be selected within one sentence embedding.

[Weakness 4 & Question 6]:

The estimated VTGHLS value of 0.64 on the ground truth is computed as follows: We first match the ground truth teaser back to body contents. To determine the lowest distance that can be considered as close, we randomly pick 15 documentaries with teaser and body content frame pairs. We calculate the 20% L2 distances between each teaser frame and each body frame and consider it as the lowest distance threshold. We then extract the CLIP feature and select top 20 images with the highest cosine similarity to find those close in semantic meanings. Then we calculate the pixel-by-pixel L2 distance between the teaser frame and those 20 images. We pick the one with the lowest distance and smaller than the selected threshold. Then we leverage a pretrained VTGHL model to calculate the audio-visual correspondence between matched frames(in body contents) and corresponding textual narrations plus the importance of the frame. Moreover, we would like to point out that the optimal threshold for TeaserGen-PT is determined using binary search to maximize visual-textual narration correspondence while maintaining the desired video length. When the threshold is lowered, the total length of the selected video clips increases monotonically. We leverage binary search to ensure computational efficiency.

[Weakness 5]:

We would like to point out that decoding techniques like beam search are proposed module and we conduct ablation study on decoding method in table 3(We attached table below). An advanced decoding method, such as beam search, results in a higher F1 score and reduced repetitiveness.

[Weakness 7 & Question 7]:

We believe that using a deeper and more advanced transformer architecture would likely lead to a better performance in capturing complex visual and temporal patterns. However, due to the limited computing budget, we did not experiment with different network architecture in this work.

[Weakness 6 & Question 8]:

We conduct a subjective evaluation of our generated narration in Section 5.5 by assessing the organization, informativeness and engagingness on a Likert scale from 1 to 5. Our carefully designed prompts are considered consistent, informative, and engaging by human evaluators. As mentioned in section 2, teaser generation is unlike traditional summarization tasks, teaser generation requires handling longer input sequences, higher compression rates, and ensuring the textual narration remains engaging and story-driven.

We report common text summarization evaluation metrics, specifically the average F1 score of ROUGE, in the table below. We compare our LLM-generated narration with the audio transcription of the ground truth teaser, which includes dialogues. Additionally, for reference, we compare the performance of TeaserGen narration with fully extractive narration from the main content and a naïve summarization approach.

[Weakness 1 & Question 1]
The authors addressed my concern.

[Weakness 2 & Question 2]
Why do some metrics (e.g., REP) vary so significantly across experiments?Any thoughts/rationale? I recommend that the authors provide a more in-depth analysis of the results, offering insights into the underlying reasons for the observed performance trends. This would enhance the clarity and impact of the findings by connecting the outcomes to the design choices and experimental setups.

[Weakness 2 & Question 3]

The authors addressed my concern.

[Weakness 4 & Question 6]

• The explanation of "20% L2 distances" is unclear. Does this mean the lowest 20% of distances or a specific percentile? The terminology requires clarification.

• The rationale for selecting 15 documentaries to calculate the threshold is not explained. Why this specific number? Is it statistically justified or arbitrary?

• Similarly, why are the top 20 images chosen based on cosine similarity? Is this number optimal, and if so, how was it determined?

• The impact of varying thresholds on model accuracy or other performance metrics is not discussed. A quantitative description of how binary search improves the process would strengthen the argument.

• How the model calculates audio-visual correspondence and the frame's importance is not described. This makes it challenging to assess the validity of the approach.

• The phrase "we calculate the pixel-by-pixel L2 distance" is slightly redundant (L2 is already a element-wise operation), and its necessity after selecting semantically similar frames is unclear.

• The response does not discuss the limitations or potential sources of error in the process, such as misalignment in teaser-body content matching or issues with the pretrained VTGHL model.

• The response does not provide enough evidence the effectiveness of the approach. For example, it would be helpful to include examples (qualitative) or data supporting the 0.64 VTGHLS value.

[Weakness 5]

What was the number of beams utilized, and what impact did it have on computational overhead?

[Weakness 7 & Question 7]

My concern was addressed, I will not consider this for the final decision.

[Weakness 6 & Question 8]

• The reported ROUGE scores (ROUGE-1, ROUGE-2, and ROUGE-L) are extremely low, with values like 0.14, 0.02, and 0.12 for TeaserGen. This suggests limited overlap between the generated and reference text. However, the discussion does not critically analyze these results or explain why such low scores are acceptable. Why is there no significant improvement in ROUGE scores despite claims of improved narration quality?

• The narrative around the evaluators ("our carefully designed prompts") lacks transparency about the evaluation process and does not discuss possible weaknesses or biases in human evaluators. Typically, multiple prompts are utilized, and the results are averaged to determine a statistical trend in the model's performance.

[Question 9], [Weakness 7 & Question 10] & [Weakness 8 & Question 11]:

The authors addressed my concerns.

[Weakness 8 & Question 12

• In Table 4, TeaserGen-LR using beam search scores low on coherence (2.84) and realness (2.64–2.71), suggesting a gap between textual narration coherence and its translation into a cohesive visual experience.
• While Table 5 highlights strong subjective scores for narrations (organization, informativeness, and engagingness), these results do not directly resolve the issue of disjointed visuals caused by high scene change rates. The coherence of narration does not inherently guarantee cohesive visuals if the scene changes outpace the narration's logical flow.

[Weakness 2 & Question 13]

• How do TeaserGen-PT and TeaserGen-LR handle embedding mismatches or inconsistencies in practice?
• What mechanisms are in place to update or adapt the approach as new models or embeddings become available?

Decision: My primary concern with this approach pertains to the 0.64 threshold, as I find insufficient evidence to support the implications of this design decision.

Thanks for your constructive feedback, we would like to first address your primary concern and then answer your questions one by one.

The goal of the proposed searching algorithm is to maximize the total video-narration correspondence while maintaining the desired length. For each sentence in narration scripts, we generate a score curve (with the x-axis representing video time and the y-axis representing the score). This curve reflects frame importance (defined by its likelihood of being included as a highlight in video highlight detection) and text-visual correspondence to the sentence.

During the search process, the algorithm selects a threshold such that the intersections of the score curve and the threshold determine the start and end times of video clips. These clips are included if their scores exceed the threshold, and multiple clips can satisfy this condition. The search starts at the highest point of the score curve, progressively lowering the threshold in small steps (0.001) until the total duration of the selected clips matches the duration of the generated audio waveform. This final threshold is identified as the optimal threshold of that sentence.

For each sentence in the generated narrations, we associate it with a score curve and determine an optimal threshold through a search process described above. We then calculate the sentence-level VTGHLS by averaging the VTGHLS values of all frames within that sentence whose VTGHLS exceeds the threshold. Finally, we report the VTGHLS in Table 3 by averaging the sentence-level VTGHLS values across all sentences in the generated scripts of all documentaries in the test set.

To improve computational efficiency, we take advantage of the monotonic increase in the total length of selected video clips as the threshold decreases. Instead of linearly decreasing the threshold, a binary search algorithm is applied to quickly identify the optimal threshold.

The estimated VTGHLS value of 0.64 for the ground truth is provided solely for comparison purposes and is not used at any stage of our training or inference pipeline. It does not correspond to the optimal threshold shown in Figure 2.

Below are the details of how the value 0.64 is calculated:

We first match the ground truth teaser frames back to the main documentary. For each sentence in ground truth narration scripts, we generate a score curve and average the VTGHLS of those frames within that sentence and get sentence-level VTGHLS. Then, we average the sentence-level VTGHLS values across all sentences in the narration scripts of all available ground truth teasers to obtain the estimated ground truth VTGHLS. Since we require exact matches, we cannot rely solely on semantic meaning. To minimize matching errors, we experimented with several methods:

1. Pixel-by-pixel comparison: Directly compare each frame in the ground truth teaser with all frames in the main documentary using pixel-by-pixel L2 distance.
2. Semantic matching with top 10 candidates and then pixel-by-pixel comparison: For each frame in the ground truth teaser, find the top 10 closest images in semantic meaning (i.e., the 10 images with the highest cosine similarity) and then compare those 10 images with frames in the main documentary using pixel-by-pixel L2 distance.
3. Semantic matching with top 20 candidates and then pixel-by-pixel comparison: For each frame in the ground truth teaser, find the top 20 closest images in semantic meaning (i.e., the 20 images with the highest cosine similarity) and then compare those 20 images with frames in the main documentary using pixel-by-pixel L2 distance.

We found that semantic matching with the top 20 candidates resulted in the least matching error. This was verified by manually checking the matched frames in the main documentaries against the frames in the ground truth teasers that we aimed to search for.

[Weakness 2 & Question 2]

We would like to point out that we discuss how our method addresses the issues of repetitiveness and fragmentation in Section 5.4 and in the ablation study. Here we provide two concrete examples below.

We define REP as $1 - \frac{\text{Number of Unique Frames}}{\text{Total Number of Frames}}$. For example, CLIP-NN has 92.73 % repetitiveness. This is because, for each sentence, CLIP-NN generates a sentence embedding and solely relies on it to select the number of frames that match the desired length. As a result, it frequently selects the same frames, leading to high repetitiveness. In contrast, TeaserGen-PT accounts for previously selected intervals when choosing video clips for the current sentence, reducing repetition. TeaserGen-LR goes further by leveraging a diffusion prior and smoothing mechanisms, both of which are shown to effectively mitigate repetitiveness in our ablation study.

We define scene change rate as $1 - \frac{\text{Number of Consecutive Frames Within the Same Scene}}{\text{Number of Frames}}$. For example, UniVTG[2] has a high scene change rate due to two limitations. First, UniVTG[2] can only process video frames of less than 10 minutes, requiring us to divide long documentaries into several short chunks and recombine them later. Second, for each chunk, UniVTG[2] selects keyframes without considering temporal continuity, resulting in high fragmentation. In contrast, TeaserGen-PT focuses on selecting important intervals, which helps reduce fragmentation. Additionally, TeaserGen-LR leverages beam search to incorporate temporal information, further improving scene continuity. By addressing both repetitiveness and fragmentation, our methods demonstrate clear advantages over baseline approaches.

[Weakness 5]:

As we shown in Section 4.2.2, we use a beam size of 5. All of our experiments were conducted on a single A6000 GPU.

[Weakness 6 & Question 8]:

1. We would like to clarify that this is a generative task, and the reference text—the audio transcription of the ground truth teaser—is a mixture of dialogue and narration. ROUGE scores measure the overlap between our generated narrations and the ground truth teaser audio transcription. However, our generated narrations intentionally exclude dialogue and include an engaging ending question, which inherently reduces overlap and leads to lower ROUGE scores. To address this limitation, we also conduct subjective evaluation metrics to better assess the quality of our generated narrations, as ROUGE alone does not fully capture the nuances of narration quality in this context.

2. To ensure a thorough and unbiased evaluation process, we took the following steps:

We compared naive summarization and finely-tuned scripts using specific GPT prompts. For naive summarization, we used the system prompt, "You are a narrator for this story." Since GPT cannot process an entire long documentary at once, we divided it into 10 chunks. For each chunk, GPT was prompted with the instruction: "Summarize the paragraph in one sentence." For the finely-tuned scripts, we used the same system prompt, "You are a narrator for this story," and divided the documentary into 10 chunks as well. For each chunk, we started with the same summarization step as naive summarization, instructing GPT to "Summarize the paragraph in one sentence." After generating these 10 summary sentences, we combined them into a single summarized paragraph. This combined paragraph was then further refined using an additional prompt: "Rewrite the paragraph into an engaging story opening in 10 sentences or less, keeping all names and avoiding being replaced by pronouns." Finally, we added another step to enhance the narrative by asking GPT: "Given the title {video_title} and the provided summary, formulate one thought-provoking and concise question that relates directly to the summary."

As shown in section 5.5, to assess the performance, we created two versions (A and B), each containing 5 pairs of results generated using the prompts for naive summarization and finely-tuned scripts. We recruited 20 participants, with 11 evaluating version A and 9 evaluating version B. The results were presented in a randomized order to ensure participants were unaware of which prompt generated each result, minimizing bias. The results of the subjective evaluations are presented as the average score, accompanied by the corresponding confidence interval to ensure a comprehensive and statistically robust evaluation.

[Weakness 8 & Question 12]:

We propose several modules to address the issue of disjointed visuals, and these modules have proven effective, significantly outperforming the baseline models by a wide margin.

[Weakness 2 & Question 13]:

TeaserGen-PT and TeaserGen-LR present two general paradigms for solving teaser generation tasks. TeaserGen-PT employs an interval-based approach with video-grounding backbone that can be updated. In contrast, TeaserGen-LR uses a learning-based approach that learns direct mapping across modalities.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We have carefully reviewed your comments and addressed your concerns and questions in our responses above. Additionally, we have incorporated the suggested revisions into the updated PDF. If there are any further questions or concerns, we would be happy to address them.

We thank the reviewer for the thoughtful review. We believe our tasks are novel and have wide applications. We are encouraged by the reviewer’s positive feedback on our model performance. We address the reviewer’s comments below.

[Weakness 1]

We conduct a subjective evaluation of our generated narration in Section 5.5 by assessing the organization, informativeness and engagingness on a Likert scale from 1 to 5. Our carefully designed prompts are considered consistent, informative, and engaging by human evaluators. As mentioned in section 2, teaser generation is unlike traditional summarization tasks, teaser generation requires handling longer input sequences, higher compression rates, and ensuring the narration remains engaging and story-driven.

We report common text summarization evaluation metrics, specifically the average F1 score of ROUGE[6], in the table below. We compare our LLM-generated narration with the audio transcription of the ground truth teaser, which includes dialogues. Additionally, for reference, we compare the performance of TeaserGen narration with fully extractive narration from the main content and a naïve summarization approach.

[Weakness 2]

As shown in Table 3 (see table below), we have compared our method to a query-based summarization model CLIP-IT [3]. We also conducted an experiment using CLIPScore to denote the highlight score (TeaserGen-PT-CLIP). Additionally, as discussed in the paper, the recent state-of-the-art trailer generation model TGT[4] and the state-of-the-art multimodal video summarization model LFVS[5], along with their code, have not made their training datasets open-source, making direct comparison impossible. To demonstrate the effectiveness of our proposed model, we further evaluate it against two extra SOTA video summarization models: CSTA[1] and A2Summ[7]. (Note that CSTA[1] cannot process videos with more than 5,000 frames, so we chunked the long videos and recombined them after processing. Similarly, A2Summ[7] cannot handle the long documentaries in our dataset, so we divided them into 10 chunks and recombined them afterward.) We observe that TeaserGen achieves a higher F1 score than CSTA[1], UniVTG[2], and CLIP-IT[3] and A2Summ[7]. In addition, TeaserGen demonstrates a scene change rate more closely aligned with the ground truth and TeaserGen-LR has the highest CLIPScore.

[1] Son et al., CSTA: Spatiotemporal Attention for Video Summarization, CVPR, 2024 (https://arxiv.org/abs/2405.11905)

[2] Lin et al., UniVTG: Unified Video-Language Temporal Grounding, ICCV, 2023 (https://arxiv.org/abs/2307.16715)

[3] Narasimhan et al., Clip-it! language-guided video summarization, CoRR, 2021(https://arxiv.org/abs/2107.00650).

[4] Argaw et al., Towards automated movie trailer generation, CVPR, 2024 (https://arxiv.org/abs/2404.03477.)

[5] Argaw et al., Scaling up video summarization pretraining with large language models, CVPR 2024(https://arxiv.org/abs/2404.03398)

[6] Chin-Yew Lin, ROUGE: A Package for Automatic Evaluation of Summaries, Text Summarization Branches Out, 2004. (https://aclanthology.org/W04-1013.)

[7] He et.al., Align and Attend: Multimodal Summarization with Dual Contrastive Losses, CVPR, 2023 (https://arxiv.org/abs/2303.14150)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We have carefully reviewed your comments and addressed your concerns and questions in our responses above. Additionally, we have incorporated the suggested revisions into the updated PDF. If there are any further questions or concerns, we would be happy to address them.

We thank the reviewer for the thoughtful review. We answer the reviewer’s comments below.

[Weakness 1]

Although there are only 1.2K documentaries, each documentary is over 30 minutes long, resulting in 600 hours of content. We compare our dataset with MovieNet[1] and TGT[2] in table below, we can see our dataset is the first and largest public available teaser/trailer-body contents dataset.

[Weakness 2]

We conduct a subjective evaluation of our generated narration in Section 5.5 by assessing the organization, informativeness and engagingness on a Likert scale from 1 to 5. Our carefully designed prompts are considered consistent, informative, and engaging by human evaluators. As mentioned in section 2, teaser generation is unlike traditional summarization tasks, teaser generation requires handling longer input sequences, higher compression rates, and ensuring the narration remains engaging and story-driven.

We report common text summarization evaluation metrics, specifically the average F1 score of ROUGE[3], in the table below. We compare our LLM-generated narration with the audio transcription of the ground truth teaser, which includes dialogues. Additionally, for reference, we compare the performance of TeaserGen narration with fully extractive narration from the main content and a naïve summarization approach.

[Weakness 3 & Question 2 ]

We would like to argue that our method has broad applications in content where narration is the primary medium for conveying information, with visuals serving as a complementary and engaging element. This approach is highly applicable to documentaries, TED Talks, and educational content such as lectures in high impact application fields. In the table below, we can see that TeaserGen performs well in zero-shot scenarios. We also compare TeaserGen-PT and TeaserGen-LR with two extra video summarization models: CSTA[6] and A2Summ[8]. Note that CSTA[6] cannot process videos with more than 5,000 frames, so we chunked the long videos and recombined them after processing. Similarly, A2Summ[8] cannot handle the long documentaries in our dataset, so we divided them into 10 chunks and recombined them afterward. Since A2Summ[8] is an extractive method and ground truth is unavailable in zero-shot scenarios, we leveraged the average teaser length of approximately 1.3 minutes. For each chunk, we selected 8 key frames (fps=1) and 1 key sentence to summarize the content effectively. We observe that TeaserGen achieves a relatively higher F1 score and also provides narrations with higher audio-visual correspondence that are effective in engaging the audience. The generalizability of our proposed models can be further demonstrated by samples on our demo page(https://54321anonymous.github.io/ICLR2025/).

[Weakness 4 & Question 3]

As shown in Table 3 (see table below), we have compared our method to a query-based summarization model CLIP-IT [4]. Additionally, as discussed in the paper, the recent state-of-the-art trailer generation model TGT[2] and the state-of-the-art multimodal video summarization model LFVS[5], along with their code, have not made their training datasets open-source, making direct comparison impossible. To demonstrate the effectiveness of our proposed model, we further evaluate it against two extra SOTA video summarization model: CSTA[6] and A2Summ[8]. (Note that CSTA[6] cannot process videos with more than 5,000 frames, so we chunked the long videos and recombined them after processing. Similarly, A2Summ[8] cannot handle the long documentaries in our dataset, so we divided them into 10 chunks and recombined them afterward.) We observe that TeaserGen achieves a higher F1 score than CSTA[6], UniVTG[7], and CLIP-IT[4] and A2Summ[8]. In addition, TeaserGen demonstrates a scene change rate more closely aligned with the ground truth and TeaserGen-LR has the highest CLIPScore.

[1] Huang et al., MovieNet: A Holistic Dataset for Movie Understanding, ECCV, 2020. https://arxiv.org/abs/2007.10937.

[2] Argaw et al., Towards automated movie trailer generation, CVPR, 2024 (https://arxiv.org/abs/2404.03477.)

[3] Lin, ROUGE: A Package for Automatic Evaluation of Summaries, Text Summarization Branches Out, 2004. (https://aclanthology.org/W04-1013.)

[4]Narasimhan et al., Clip-it! language-guided video summarization, CoRR, 2021 (https://arxiv.org/abs/2107.00650).

[5] Argaw et al., Scaling up video summarization pretraining with large language models, CVPR, 2024 (https://arxiv.org/abs/2404.03398)

[6] Son et al., CSTA: Spatiotemporal Attention for Video Summarization, CVPR, 2024(https://arxiv.org/abs/2405.11905)

[7] Lin et al., UniVTG: Unified Video-Language Temporal Grounding, ICCV, 2023 (https://arxiv.org/abs/2307.16715)

[8] He et al., Align and Attend: Multimodal Summarization with Dual Contrastive Losses, CVPR, 2023 (https://arxiv.org/abs/2303.14150)