ScreenWriter: Automatic Screenplay Generation and Movie Summarisation

Thank you for your review. Here are our responses.

The system primarily combines existing techniques like keyframe extraction, speaker diarization, and face recognition into a single pipeline.. clarify the contribution of each component and identify their novelty.

The two main novel components are the algorithms for scene-break detection and for character name assignment. The design of these algorithms, particularly the former, differs substantially from any existing algorithms for the same tasks.

Provide a comparison to publicly available models like GPT-4 and Gemini.

We compare only to open-source models. We do not know the training setup or internals of closed-source proprietary models, so we would argue there is little to be learnt by comparison with them. We have added a line explaining this reasoning to the Summarisation paragraph in Section 5.

It is not clear how the commercial system (Otter AI) is used in comparison. Thank you for this comment. What we compare to is called Otter, a publicly available model that simply produces a short textual description of each scene. We use the code from https://github.com/Luodian/Otter. We mistakenly referred to this as ‘ Otter AI’ at some points in the paper. Thank you for pointing this out. We have now corrected it.

It seems this paper only uses text and audio as input, and does not consider image and visual input. Is my understanding correct?

No, our paper uses video and audio input. There is no text input required. This is in fact one of the major contributions of our model.

Consider expanding the evaluation by including additional metrics, especially end2end user evaluation.

Thank you for this suggestion. Unfortunately, are very expensive to obtain. The automatic metrics we use in their place have been shown to correlate with human judgements. We will pursue human evaluation in future work.

Thank you for your review. Here are our responses.

Most components of the proposed framework are adopted from existing works, lacking academic and technological contributions.

The scene-break detection algorithm and the algorithm for assigning character names to speaker IDs, which are essential components of our model, are both highly novel.

The screenplay generation process seems somewhat unnecessary. If the transcript is summarized and then processed by the summarizer alongside visual information in parallel, why is a screenplay still needed?

The transcript identifies speakers only by a randomly generated ID, e.g. “SPEAKER08: Now it places the lotion in the basket”, whereas the automatic screenplay we produce has character names, e.g. “Jame Gumb: Now it places the lotion in the basket” (from Fig 3). Additionally, the transcript does not include scene breaks, which are needed for hierarchical summarisation. The benefits both of scene breaks and name assignments are shown empirically in Table 4.

Additional baselines are needed, such as SceneDetection. Thank you for this suggestion. We have added SceneDetect and another existing method as comparisons for the scene segmentation results.

Table 1 in the paper has been updated with these results. We are working on adding additional baselines and will update the paper as soon as they are ready.

A discussion should be included of MM-VID, MM-Narrator and MM-ScreenPlayer.

We have added this to the related work section.

Some typo: 'sctructure' in line 67 should be 'structure', 'movide' in line 317 should be 'movie'.

Thank you, we have fixed these.

It's confusing that In the updated version of the paper, the author directly includes the name in the paper. This violates the double-blind rule.

Thank you for your review. Here are our responses.

Why not just purchase screenplays?

As you mention, the actual dialogue often deviates significantly from the official screenplay. We have observed this in our dataset, for example in V for Vendetta (2004) and Somethings’ Gotta Give (2003). There are also long videos for which screenplays are not available, such as documentaries. We are working on producing some generated summaries for documentary videos, which we will update the paper with as soon as they are ready. Additionally, $25 is not an insignificant amount to pay per video if one wants to purchase screenplays.

A proper literature survey on the multimedia content processing and computer vision areas, and comparison with SOTA models, needs to be done.

We have added further related work on video captioning and LLMs for video understanding. We have also added two more comparison models for the scene segmentation task, which are now in Table 1.

What is the impact of name assignment errors on the quality of the screenplay generation and on the summarization? This is tested in the 'w/o names' ablation setting in Table 4, where we see a moderate degradation in summary quality. We observe that the LLM is sometimes able to put the correct name in the summary from the dialogue only, but that this is sometimes inaccurate.

There should be an evaluation comparing the generated screenplay with the original screenplay, perhaps using some of the text processing metrics used for the summary comparisons. Thank you for the suggestion. Unfortunately, it is very difficult to automatically evaluate the quality of the generated screenplay. Text comparison metrics such as ROUGE or BLEU are designed for comparing much smaller texts of a couple of sentences, and struggle even with paragraph-length comparison. We have added several example generated screenplays to the supplementary material which can be evaluated qualitatively. Instead, we choose to indirectly evaluate the screenplay via the summarisation task.

What would be the quality of the summary generated by the name-only prompt of the current leaders of the ChatBot Arena (say ChatGPT-4o, Gemini 1.5 and Claude 3.5)? if it is substantially better than the proposed method, then the case for the proposed research will be weakened. Thank you for the suggestion. We make the deliberate choice to compare only to open-source models. We do not know the training setup or internals of closed-source proprietary models, so we would argue there is little to be learnt by comparison with them. We have added line explaining this reasoning to the Summarisation paragraph in Section 5.

The name-only prompt results are actually quite impressive, why does the quality degrade when more information is provided i.e. full script and whisperx script. When prompted with the name only, the model very likely effectively regurgitates an existing online summary. When given a very long prompt that includes dialogue and event descriptions, the model tries to actually summarise the information given, during which it can make mistakes. This is evidenced, for example, by the fact that, in the name-only setting, it simply repeats the same information over and over when forced to give a longer output. This suggests that it has a fixed-length answer it is retrieving, rather than dynamically generating the summary in response to the given information. We have added this analysis to the the Summarisation paragraph in Section 5.

Thank you for your review. Here are our responses.

Compare scene-break alg to Scene Transition Graph by IBM from 1997.

We have added this comparison as you suggest.

We have updated the paper with these results.

There should be a more in-depth discussion of existing methods for movie summarization.

We are not aware of any existing summarisation methods that (a) use only the video as input and (b) operate on very long videos, e.g. movie length. We have already included some discussion of long summarisation using text instead of video. In response to your comment, we have added references to the literature on summarising short videos: Sridevi & Kharde (2020); Seo et al. (2022); Zhou et al. (2018b) and Yang et al. (2023), which differ from our work in that the small video size allow them to use end-to-end models.

The contribution in terms of algorithms is limited.

We strongly disagree with the claim that the algorithms we propose are not novel. There are indeed other attempts to solve the same problems, such as segmenting a video into scenes, but the internal design of our scene segmentation algorithm is very different from existing methods. We are grateful for the suggestion to discuss some of these existing methods in our paper. The use of the minimum description length principle in particular is highly novel, allowing our algorithm, among other things, to automatically determine the number of scenes. While the problem of scene segmentation is not novel, our proposed solution is.

The movie application is exactly where these kind of techniques for automated objective textual summarization seem less interesting as you can find a movie summary for every movie at the IMDB website and if not, likely the actors can also not be found there.

There is an advantage to being able to generate the summary, as you can control the length and detail. The summaries you mention on the IMDB website are mostly restricted to 239 characters–about 1/10th the length of the summaries we generate. Additionally, long-form video-only summarisation is a useful testbed for developing video understanding systems in general.

Personalized summaries of movies are in my opinion more interesting. Thank you for the suggestion–we have added this as a future work.

A typical summary should not contain any spoilers where the current system doesn't take that into account.

We are generating Wikipedia summaries, which include all plot information, including spoilers. These are detailed summaries of what happens in a movie, and test the models' understanding capabilities. We are not generating synopses, which are intended to entice viewers to watch a show without revealing too much of the plot.

Many early works use the same datasets as us.

The MovieSumm dataset was only released in 2024.

Ref [Abhimanyu Dubey] is not proper.

Thank you, we have fixed this.