REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing

We thank the reviewer for appreciating the significance of our proposed task, as also noted by reviewers yMCw and QQGE. We also appreciate your encouraging feedback on our extensive experiments, which reviewer tTNr similarly highlighted.

[Weakness 1]:

Note:

* ETS (extractive-then-smoothing) first extracts two interview segments, then uses GPT-4o to smooth them into a coherent output. To ensure fair comparison in downstream teaser generation, ETS is given prior knowledge that teasers typically contain two interview segments (based on the oracle average). Its closeness to the ground truth stems from this privileged information, effectively biasing it toward the oracle.

** Orcale: In our setting, the outputs generated by our dataset preprocessing pipeline, which includes pretrained models, are treated as oracle annotations representing an upper bound.

From the table, we can see that REGen-IDQ is the closest to the ground truth distribution and is comparable to A2Summ[1]. In addition, we would like to point out that the position of the quotation marks and whether the retrieved content fits the context are also important in addition to matching the overall distribution. We verify our performance through subjective evaluations (Please see table 3 and table 5) and present qualitative examples on our demo page (Please see section 3).

In addition, we provide two comparison plots on our demo page (Section 0): (1) the number of quotes per documentary, where the x-axis represents the number of quotes and the y-axis shows the count of documentaries; (2) the mean difference(can be negative) in quote count per documentary between the REGen system and the ground truth, with the x-axis as the mean difference and the y-axis as the number of documentaries.

[Weakness 2]:

Thank you for pointing that out! Even though the recall is relatively low, our model is still able to retrieve quotable interviews that support surrounding claims while maintaining a cohesive narrative flow. This is further supported by our subjective evaluation (see Tables 3 and 5), where the effectiveness of interview insertion is significantly higher than the baseline. We also provide qualitative examples on our demo page (see Section 3). We hypothesize that this is because some interviews convey similar ideas—for example, multiple interviewees may be included to support the same claim. As a result, their representations in the latent embedding space are very similar, making it challenging to retrieve the exact original interview segment. However, this does not undermine the role of quotable interviews, which is to support the claims while preserving the overall narrative flow.

[Weakness 3]:

We will expand the quote retrieval and narration-matching sections in our camera-ready version. Specifically, we will elaborate on the quote retriever using the architecture shown in Figure 2. We will also expand the explanation of how narration-flow matching works. Please see our demo page for Figure 1.

Reference:

[1]He et al., Align and Attend: Multimodal Summarization with Dual Contrastive Losses, CVPR, 2023

The additional experiment solving weakness 1 is encouraged to have a deeper analysis in the final paper, such as adding a comparison on the newly provided Quote Distribution Plot between baselines and the proposed method.

Meanwhile, the main issues from the weaknesses of my concern have been solved. I will raise my initial rating to 4.

Thank you for this valuable suggestion! We will include the Quote Distribution Plots in the final paper and provide a more in-depth analysis using both plots.

We thank the reviewer and sincerely appreciate your acknowledgement of the strong motivation and significance of our work—also highlighted by reviewer yMCw—as well as the thorough experiments demonstrating our framework’s effectiveness. We address your comments point by point below.

[Question 1]:

We run speaker diarization on a single A6000 GPU, and fine‑tune LLaMA and train the retriever (batch size 64) on a single A100 GPU. The entire system was trained in under one day. To build the candidate pool, we first separate each video’s audio into music, dialogue, and sound effects—providing a clean dialogue track that can enhance WhisperX’s accuracy. WhisperX then processes this track to identify speaker turns and determine start‑end timestamps for each quotable segment (the most time‑consuming step, yet still 50× faster than real time). Next, we extract textual and visual features for every interview segment using our feature extractor.

Once the candidate pool is assembled, there’s no need to rerun diarization or feature extraction. Retrieval over L quotes and N candidates incurs O(L · N) time, so computation scales linearly with archive size and can become challenging for very large video collections. In our human‑annotated test set, each video contains on average three quotes—equating to roughly one second of retrieval time per video when inserting three video segments.

[Question 2, Weakness 1 and Weakness 2]:

As discussed in section 6, we identify two primary constraints. First, the limited context window can be addressed by adopting a more powerful LLM with a larger context length; we chose LLaMA for its strong open‑source performance, and our architecture readily accommodates such upgrades. Second, in lecture settings where the instructor’s voice is present (as discussed in the section 6), we can use OCR‑based frame comparison to detect slide transitions and segment long lectures into smaller chunks. This variation affects only the preprocessing stage, and our two‑stage method remains effective once quotable segments are extracted. We hope this work will serve as a foundation for hybrid abstractive–extractive summarization that supports both factual grounding and narrative coherence, with significant potential in educational domains.

We thank the reviewer for their constructive feedback. We’re encouraged that the review find our documentary‑teaser framework effective and our hybrid extractive–abstractive approach inspiring. We answer the reviewer’s comments below.

[Weakness 1]:

We propose a new retrieval-embedded generation framework that enables an LLM to quote multimodal resources while maintaining a coherent narrative. Our framework supports long-to-short video editing by generating short videos that feature a coherent narrative with embedded video insertions extracted from a long input video.

[Weakness 2]:

Thank you for pointing that out. Following TeaserGen [1], we split the transcribed text into smaller segments to generalize more effectively to extremely long videos.

[Weakness 3]:

We would like to clarify that the overlap ratio in Table 6 is used to measure the overlap between the content quoted by the large language model and the ground truth interviews. It is defined as {#overlap words} / {#words in matched interviews}. This metric is not used to evaluate narrative coherence. When training our retriever, we use joint optimization with a multitask loss: L = $L_{gen}$ + α · $L_{ret}$. Here, $L_{gen}$ encourages the model to attend to the context window around the masked segment, while $L_{ret}$ helps differentiate relevant quotable segments from other candidates.

[Question 1]:

Speaker diarization is effective for content such as news and documentaries, where multiple speakers are involved. However, in lecture videos, a key constraint is that there is usually only one speaker—the instructor—making speaker diarization less applicable. To address it, we detect slide transitions using OCR to compare frames and identify the start and end timestamps of each slide. This allows us to segment long lectures into smaller chunks. Importantly, these adjustments only affect the data preprocessing step. Once quotable video segments are extracted, our two-stage method remains effective.

[Question 2]:

Yes, it could. We intend to alleviate this issue in our script generation stage and retrieval stage. In our script‑generation stage, we leverage an LLM to preserve temporal information. In our retrieval stage, the query embedding is generated from the surrounding context, thereby retaining temporal cues. In the future, we can incorporate these contextual embeddings into the representations of quotable interview segments in our candidate pool to ensure temporal coherence and maintain logical consistency.

Reference:

[1] Xu et al., TeaserGen: Generating Teasers for Long Documentaries, ICLR 2025.

We thank the reviewer for their thoughtful feedback and are encouraged by their recognition of our task’s novelty, the elegance and rigor of our hybrid abstractive‑extractive framework, and the clarity of our presentation. We hope this work meaningfully advances grounded and coherent long-to-short video editing. We answer the reviewer’s comments below.

[Weakness 1]:

Our system demonstrates robust speaker diarization performance on our documentary dataset, achieving a narrator‑prediction F1 score of 71.6%, start‑time correctness of 93.5%, end‑time correctness of 94.9%, and transcription accuracy of 94.9%. We thank the reviewer for pointing out the dependency of our framework on diarization quality. Although we acknowledge that alternative methods may further improve segment extraction on other datasets, we evaluated our framework’s robustness by injecting controlled noise: we introduced varying proportions of segments labeled as “narrator” into the set of quotable interview segments. Using a fixed random seed, we then measured recall under different noise levels and reported in the table below; as expected, recall declines slightly as more incorrect segments are added.

[Weakness 2]:

One possible reason is the preprocessing step used for lecture videos. As we mentioned in the limitations section, the current preprocessing method results in overly fragmented outputs in lecture videos. We anticipate better performance by segmenting lecture videos based on slide transitions—for example, using OCR to detect transition points or leveraging built-in slide metadata that may accompany lecture recordings.

[Weakness 3]:

Our pipeline begins by inputing raw videos: we first apply ASR and speaker diarization to convert audio into structured transcripts. Drawing on our video‑editing experience, we then use an LLM to infill and draft the script—essentially outlining where interview clips or other footage should be inserted—and follow up with a learned retriever to find the most appropriate video segments for each insertion placeholder. This modular design not only mirrors real‑world editing workflows but also maximizes data efficiency by allowing each component to be trained with specialized and smaller datasets.

[Question 1]:

In quotable interview segments, the visual modality often provides background context. For instance, an interview filmed outdoors differs from one conducted indoors. We observe that in current quotable segments—primarily interviews—the text modality carries more information than the visual modality. However, our framework is designed to generalize to broader cases where the visual modality contributes more substantially to retrieval. For example, when retrieving natural view videos without narration, visual information becomes important for accurate retrieval.

[Question 2]:

We observe GPT-4o can only produce generalized insertions, which are not ideal as a query for retrieving a quotable segment. In contrast, our proposed model is trained specifically to learn to generate insertions from the context that includes detailed information that can be used as effective queries. Moreover, as the quotable segments (oftentimes an interview in our case) might be spoken by a different person other than the narrator, our proposed method can better capture the nuanced style of these quotable segments. We show an example on our demo page. GPT-4o’s output is sometimes unreliable, making postprocessing challenging. While we believe GPT-4o could be improved with more contextual guidance in future work, our second-stage retriever remains effective.

[Question 3]:

We run speaker diarization on a single A6000 GPU, and fine‑tune LLaMA and train the retriever (batch size being  64) on a single A100 GPU. The entire system was trained in under one day. To build the candidate pool, we first separate each video’s audio into music, dialogue, and sound effects—providing a clean dialogue track that can enhance WhisperX’s accuracy. WhisperX then processes this track to identify speaker turns and determine start‑end timestamps for each quotable segment (the most time‑consuming step, yet still 50× faster than real time). Next, we extract textual and visual features for every interview segment using our feature extractor.

Once the candidate pool is assembled, there’s no need to rerun diarization or feature extraction. Retrieval over L quotes and N candidates incurs O(L · N) time, so computation scales linearly with archive size and can become challenging for very large video collections. In our human‑annotated test set, each video contains on average three quotes—equating to roughly one second of retrieval time per video when inserting three video segments.