Frame-Level Captions for Long Video Generation with Complex Multi Scenes

General Response to All Reviewers:

1. The first trial of the frame-level caption method, representing a significant attempt in dataset annotation paradigms. This approach offers unique advantages:

   • First, the annotation method is simple, requiring only uniform frame sampling without considering clip, shot, or scene boundaries. This greatly reduces annotation difficulty while capturing natural transitions in the dataset, serving as a crucial technique for advancing multi-shot/scene generation scenarios.
   • Second, unlike direct frame-level image caption strategies, our core innovation involves using state-of-the-art Vision-Language Models (VLM) to first establish global understanding of multi-shot long videos before annotating individual frames. This fundamentally differs from traditional pipelines that process and caption single clips separately, providing substantial significance.
   • Finally, fine-grained frame-level captions can be flexibly rewritten and summarized into clip/chunk/shot/scene/video-level captions using LLM models based on specific usage scenarios. Particularly in chunk-level autoregressive inference, frame-level captions can form chunk prompts of arbitrary lengths without concerns about shot or scene boundaries—an issue that cannot be resolved using shot-level or scene-level captions directly.

2. This paper introduces semantic confusion in multi shot autoregressive long video scenarios, going beyond the commonly discussed error accumulation problem. More text descriptions do not improve generation quality; instead, they cause semantic confusion during generation. This occurs because each chunk requires its own dedicated prompt rather than a global video-level prompt—a critical insight not addressed in previous autoregressive works focused primarily on simple actions and single shots. To quantify this issue, we introduce the 'confusion degree' metric.

3. This paper also addresses how to effectively utilize frame-level captions through our proposed Frame-Level Attention Mechanism. By restricting each frame/chunk/shot to attend only to its corresponding prompt while maintaining global information interaction through visual self-attention, we find this approach can still achieve overall understanding. This concept demonstrates remarkable foresight, as evidenced by its subsequent adoption in Seedance—the current SOTA closed-source video generation model released after our work. Seedance employs a similar mechanism where multi-modal visual-text attention operates exclusively within shots, while global information interaction through temporal self-attention (visual only) prevents inter-shot semantic confusion. This convergence confirms our approach's forward-looking and heuristic value, particularly for future advancements in multi-shot/scene long video generation.

4. Regarding our proposed PMWD, we emphasize it represents just one optional inference method after frame-level training—not a core contribution of this paper. While the original PMWD shows significant advantages in addressing error accumulation, simpler diffusion-forcing sliding window inference methods could also be viable if they solve the error accumulation issue (e.g., using distribution matching techniques like DMD used in Causvid or self-forcing). Notably, PMWD's inherent parallelization capability provides distinct advantages with sufficient GPU resources, and if combined with compressed representations of infinite historical frames, offers substantial future potential.

5. Open Source Commitment: We will promptly release our dataset processing code, model training/inference/evaluation code, and part of multi-shot video data with frame-level caption annotations.

6. More clear implementation details: As described in line 297 and the caption of Table 1, we present a comparative analysis of models trained on the Wanx2.1 dataset using either global video-level or frame-level prompts under identical training and inference configurations. All experiments were conducted using our proposed Parallel Multi-Window Denoising (PMWD) method. A 5-second video corresponds to 21 latents, which matches the frame length used during Wanx training; therefore, we follow diffusion forcing protocols by treating these 21 latents as a sliding window unit. After each generation step, the window slides by 1/3 of its length (7 latents). For the FIFO approach, we adhere to the original paper's configuration with a slide increment of 1 latent per step. While these implementation details do not affect the conclusions, we apologize for not clearly presenting them. We will provide more detailed explanations in the main text and supplementary materials according to the reviewers' suggestions, and ensure reproducibility through open-source code.

Finally, we sincerely thank all reviewers for their meticulous reviews and kindly request consideration of the heuristic contributions this paper offers to the field.

Response To Reviewer uVip

1. The technical contribution is limited

We are the first to propose frame-level captioning within the comprehensive understanding of entire videos rather than isolated video clips, leveraging the robust comprehension capabilities of Vision-Language Models (VLMs). The key distinctions and significance have been detailed in our general response. Notably, Temporal MultiDiffusion[2] cannot generate specific chunk-level prompts without frame-level annotations.

2. The problem setting is confusing.

Our paper title clearly states "Long Video Generation with Complex Multi Scenes", and we have repeatedly explained the multi-shot long video scenarios throughout the manuscript. Furthermore, our test set incorporates complex narrative plots and multi-shot scenes. However, we will further refine our paper based on your valuable feedback. We sincerely apologize for any confusion caused and will thoroughly address this issue in subsequent revisions.

3. The paper lacks any ablation experiments to validate its design.

In Table 1, we systematically validate the effectiveness of frame-level prompts versus video-level prompts in mitigating semantic confusion while maintaining comparable quality levels. In Table 2, we demonstrate the efficacy of PMWD in addressing error accumulation during autoregressive long video generation through direct comparisons with state-of-the-art methods including Diffusion Forcing and FIFO.

General Response to All Reviewers:

1. The first trial of the frame-level caption method, representing a significant attempt in dataset annotation paradigms. This approach offers unique advantages:

   • First, the annotation method is simple, requiring only uniform frame sampling without considering clip, shot, or scene boundaries. This greatly reduces annotation difficulty while capturing natural transitions in the dataset, serving as a crucial technique for advancing multi-shot/scene generation scenarios.
   • Second, unlike direct frame-level image caption strategies, our core innovation involves using state-of-the-art Vision-Language Models (VLM) to first establish global understanding of multi-shot long videos before annotating individual frames. This fundamentally differs from traditional pipelines that process and caption single clips separately, providing substantial significance.
   • Finally, fine-grained frame-level captions can be flexibly rewritten and summarized into clip/chunk/shot/scene/video-level captions using LLM models based on specific usage scenarios. Particularly in chunk-level autoregressive inference, frame-level captions can form chunk prompts of arbitrary lengths without concerns about shot or scene boundaries—an issue that cannot be resolved using shot-level or scene-level captions directly.

2. This paper introduces semantic confusion in multi shot autoregressive long video scenarios, going beyond the commonly discussed error accumulation problem. More text descriptions do not improve generation quality; instead, they cause semantic confusion during generation. This occurs because each chunk requires its own dedicated prompt rather than a global video-level prompt—a critical insight not addressed in previous autoregressive works focused primarily on simple actions and single shots. To quantify this issue, we introduce the 'confusion degree' metric.

3. This paper also addresses how to effectively utilize frame-level captions through our proposed Frame-Level Attention Mechanism. By restricting each frame/chunk/shot to attend only to its corresponding prompt while maintaining global information interaction through visual self-attention, we find this approach can still achieve overall understanding. This concept demonstrates remarkable foresight, as evidenced by its subsequent adoption in Seedance—the current SOTA closed-source video generation model released after our work. Seedance employs a similar mechanism where multi-modal visual-text attention operates exclusively within shots, while global information interaction through temporal self-attention (visual only) prevents inter-shot semantic confusion. This convergence confirms our approach's forward-looking and heuristic value, particularly for future advancements in multi-shot/scene long video generation.

4. Regarding our proposed PMWD, we emphasize it represents just one optional inference method after frame-level training—not a core contribution of this paper. While the original PMWD shows significant advantages in addressing error accumulation, simpler diffusion-forcing sliding window inference methods could also be viable if they solve the error accumulation issue (e.g., using distribution matching techniques like DMD used in Causvid or self-forcing). Notably, PMWD's inherent parallelization capability provides distinct advantages with sufficient GPU resources, and if combined with compressed representations of infinite historical frames, offers substantial future potential.

5. Open Source Commitment: We will promptly release our dataset processing code, model training/inference/evaluation code, and part of multi-shot video data with frame-level caption annotations.

6. More clear implementation details: As described in line 297 and the caption of Table 1, we present a comparative analysis of models trained on the Wanx2.1 dataset using either global video-level or frame-level prompts under identical training and inference configurations. All experiments were conducted using our proposed Parallel Multi-Window Denoising (PMWD) method. A 5-second video corresponds to 21 latents, which matches the frame length used during Wanx training; therefore, we follow diffusion forcing protocols by treating these 21 latents as a sliding window unit. After each generation step, the window slides by 1/3 of its length (7 latents). For the FIFO approach, we adhere to the original paper's configuration with a slide increment of 1 latent per step. While these implementation details do not affect the conclusions, we apologize for not clearly presenting them. We will provide more detailed explanations in the main text and supplementary materials according to the reviewers' suggestions, and ensure reproducibility through open-source code.

Finally, we sincerely thank all reviewers for their meticulous reviews and kindly request consideration of the heuristic contributions this paper offers to the field.

Response To Reviewer r4uF

1. The paper shows plausible quality on complex long video generation with dynamic scenes. But it is still necessary to include the base performance of general T2V performance by only global text prompt on the entire Vbench benchmark. How does the introduction of frame-level dataset & attention influence the foundational ability of the original backbone model (Wan2.1)?

We also show the Vbench quality result on 5s video in Tab. 1 and Tab. 2. The introduction of frame-level dataset & attention does not influence the foundational ability of the original backbone model (Wan2.1). Through testing on other metrics in Vbench 1.0 and 2.0, performance is more influenced by the learning rate during the finetuning stage and the quality of the training dataset itself. While we collected a large number of datasets, as this is research work, we did not carefully select high-quality data. As a result, after training on our dataset, both video-level prompts and frame-level prompts experienced some performance loss, but they remain comparable when compared directly.

2. The release of the dataset significantly influences the broad contribution of this paper.

We will promptly release our dataset processing code, model training/inference/evaluation code, and high-quality single and multi shot video data with frame-level caption annotations.

3. The therotical/practical insight of comparing frame-level attention VS video-level attention is suggested. What does this new fine-grained dataset and approach modify the pretained representation/distribution of the original backbone model? For example, the comparison of the token-level attention mapping and according discussion will be very helpful

This is an excellent suggestion. However, our analysis indicates that the impact is not significant. The functionality of each attention head remains largely unchanged, with some heads still responsible for global information and others for local information. In the text-visual cross-attention component, there is a noticeable hard mask effect, but this is primarily due to our implementation rather than being learned. Since we cannot include images here, we will present the experiments in subsequent supplementary materials.

4. The dataset is mainly annotated by Gemini. But the quality of the annotation itself is lack of manual validation. How good the quality of the dataset is crucial. Even a portion of evaluation is suggested.

Even with the most advanced Gemini Pro, its comprehension ability is still far below that of humans. We will include a detailed analysis of this aspect in the supplementary materials.

Good paper, I will keep my score.

General Response to All Reviewers:

1. The first trial of the frame-level caption method, representing a significant attempt in dataset annotation paradigms. This approach offers unique advantages:

   • First, the annotation method is simple, requiring only uniform frame sampling without considering clip, shot, or scene boundaries. This greatly reduces annotation difficulty while capturing natural transitions in the dataset, serving as a crucial technique for advancing multi-shot/scene generation scenarios.
   • Second, unlike direct frame-level image caption strategies, our core innovation involves using state-of-the-art Vision-Language Models (VLM) to first establish global understanding of multi-shot long videos before annotating individual frames. This fundamentally differs from traditional pipelines that process and caption single clips separately, providing substantial significance.
   • Finally, fine-grained frame-level captions can be flexibly rewritten and summarized into clip/chunk/shot/scene/video-level captions using LLM models based on specific usage scenarios. Particularly in chunk-level autoregressive inference, frame-level captions can form chunk prompts of arbitrary lengths without concerns about shot or scene boundaries—an issue that cannot be resolved using shot-level or scene-level captions directly.

2. This paper introduces semantic confusion in multi shot autoregressive long video scenarios, going beyond the commonly discussed error accumulation problem. More text descriptions do not improve generation quality; instead, they cause semantic confusion during generation. This occurs because each chunk requires its own dedicated prompt rather than a global video-level prompt—a critical insight not addressed in previous autoregressive works focused primarily on simple actions and single shots. To quantify this issue, we introduce the 'confusion degree' metric.

3. This paper also addresses how to effectively utilize frame-level captions through our proposed Frame-Level Attention Mechanism. By restricting each frame/chunk/shot to attend only to its corresponding prompt while maintaining global information interaction through visual self-attention, we find this approach can still achieve overall understanding. This concept demonstrates remarkable foresight, as evidenced by its subsequent adoption in Seedance—the current SOTA closed-source video generation model released after our work. Seedance employs a similar mechanism where multi-modal visual-text attention operates exclusively within shots, while global information interaction through temporal self-attention (visual only) prevents inter-shot semantic confusion. This convergence confirms our approach's forward-looking and heuristic value, particularly for future advancements in multi-shot/scene long video generation.

4. Regarding our proposed PMWD, we emphasize it represents just one optional inference method after frame-level training—not a core contribution of this paper. While the original PMWD shows significant advantages in addressing error accumulation, simpler diffusion-forcing sliding window inference methods could also be viable if they solve the error accumulation issue (e.g., using distribution matching techniques like DMD used in Causvid or self-forcing). Notably, PMWD's inherent parallelization capability provides distinct advantages with sufficient GPU resources, and if combined with compressed representations of infinite historical frames, offers substantial future potential.

5. Open Source Commitment: We will promptly release our dataset processing code, model training/inference/evaluation code, and part of multi-shot video data with frame-level caption annotations.

6. More clear implementation details: As described in line 297 and the caption of Table 1, we present a comparative analysis of models trained on the Wanx2.1 dataset using either global video-level or frame-level prompts under identical training and inference configurations. All experiments were conducted using our proposed Parallel Multi-Window Denoising (PMWD) method. A 5-second video corresponds to 21 latents, which matches the frame length used during Wanx training; therefore, we follow diffusion forcing protocols by treating these 21 latents as a sliding window unit. After each generation step, the window slides by 1/3 of its length (7 latents). For the FIFO approach, we adhere to the original paper's configuration with a slide increment of 1 latent per step. While these implementation details do not affect the conclusions, we apologize for not clearly presenting them. We will provide more detailed explanations in the main text and supplementary materials according to the reviewers' suggestions, and ensure reproducibility through open-source code.

Finally, we sincerely thank all reviewers for their meticulous reviews and kindly request consideration of the heuristic contributions this paper offers to the field.

Response To Reviewer Tfgw

1. Further explanation for "automatically choosing between shared or independent descriptions based on the degree of visual change" and "adaptive frame-level annotation".

Unlike direct frame-level image caption strategies, our core innovation involves using state-of-the-art Vision-Language Models (VLM) to first establish global understanding of multi-shot long videos before annotating individual frames by providing all the keyframes evenly sampled from the entire video. A video may contain shot transitions, scene changes, significant action variations, or subtle modifications. Our concept of "adaptive or automatically" refers to the VLM's capability to recognize these scenarios and generate appropriate frame-level annotations. We apologize for the lack of clarity in our original presentation and will revise this explanation in the main text according to your suggestions.

2. Why do you use DF training when the inference only handles unified noise level per step?

The focus of this paper is on the impact of video-level prompts and frame-level prompts on the model, so the main difference during training is whether frame-level prompts are used. However, since our test scenario involves multi-shot autoregressive long videos, we modified the video-level timestep flow matching to diffusion forcing's frame-level timestep. This modification enables direct comparison with the two state-of-the-art approaches— sliding window inference of diffusion forcing and FIFO inference. Notably, even though PMWD utilizes video-level timesteps, diffusion forcing remains compatible with this configuration after training.

3. Comparison with baseline long video generation methods should be included. The field of autoregressive long video generation is extremely cutting-edge, and fair comparisons are challenging due to open-source and resource constraints. The LCT and Mask2DiT methods you mentioned are both ByteDance works that have not been open-sourced. In particular, different research groups utilize varying datasets and training schemes, making direct comparisons difficult. Therefore, we reproduced diffusion forcing training using the same internal data and identical settings, and compared against the most mainstream approaches of sliding window and FIFO. We also experimented with the history noise scheme from diffusion forcing v2, but observed negligible performance differences. Our experiments have clearly and fairly validated our conclusions regarding frame-level semantic confusion and the effect of frame-level caption.

4. Why are there temporal jittering in the generation results? Is it a drawback of the PMWD inference strategy?

Temporal jittering occurs because sliding window-based methods only have 21 latents, resulting in a very short visible range. Even with overlap, the maximum overlap is 20 latents. As a training-free method, PMWD solves the problem of error accumulation but still has issues with consistency and temporal jittering. While this is indeed a drawback of the current most basic PMWD implementation, the method has significant potential with its parallel characteristics. It will have greater room for improvement if combined with longer historical context and framepack context compression.

5. Further explanation for "automatically choosing between shared or independent descriptions based on the degree of visual change" and "adaptive frame-level annotation".

Unlike direct frame-level image caption strategies, our core innovation involves using state-of-the-art Vision-Language Models (VLM) to first establish global understanding of multi-shot long videos before annotating individual frames by providing all the keyframes evenly sampled from the entire video. A video may contain shot transitions, scene changes, significant action variations, or subtle modifications. Our concept of "adaptive or automatically" refers to the VLM's capability to recognize these scenarios and generate appropriate frame-level annotations. We apologize for the lack of clarity in our original presentation and will revise this explanation in the main text according to your suggestions.

General Response to All Reviewers:

1. The first trial of the frame-level caption method, representing a significant attempt in dataset annotation paradigms. This approach offers unique advantages:

   • First, the annotation method is simple, requiring only uniform frame sampling without considering clip, shot, or scene boundaries. This greatly reduces annotation difficulty while capturing natural transitions in the dataset, serving as a crucial technique for advancing multi-shot/scene generation scenarios.
   • Second, unlike direct frame-level image caption strategies, our core innovation involves using state-of-the-art Vision-Language Models (VLM) to first establish global understanding of multi-shot long videos before annotating individual frames. This fundamentally differs from traditional pipelines that process and caption single clips separately, providing substantial significance.
   • Finally, fine-grained frame-level captions can be flexibly rewritten and summarized into clip/chunk/shot/scene/video-level captions using LLM models based on specific usage scenarios. Particularly in chunk-level autoregressive inference, frame-level captions can form chunk prompts of arbitrary lengths without concerns about shot or scene boundaries—an issue that cannot be resolved using shot-level or scene-level captions directly.

2. This paper introduces semantic confusion in multi shot autoregressive long video scenarios, going beyond the commonly discussed error accumulation problem. More text descriptions do not improve generation quality; instead, they cause semantic confusion during generation. This occurs because each chunk requires its own dedicated prompt rather than a global video-level prompt—a critical insight not addressed in previous autoregressive works focused primarily on simple actions and single shots. To quantify this issue, we introduce the 'confusion degree' metric.

3. This paper also addresses how to effectively utilize frame-level captions through our proposed Frame-Level Attention Mechanism. By restricting each frame/chunk/shot to attend only to its corresponding prompt while maintaining global information interaction through visual self-attention, we find this approach can still achieve overall understanding. This concept demonstrates remarkable foresight, as evidenced by its subsequent adoption in Seedance—the current SOTA closed-source video generation model released after our work. Seedance employs a similar mechanism where multi-modal visual-text attention operates exclusively within shots, while global information interaction through temporal self-attention (visual only) prevents inter-shot semantic confusion. This convergence confirms our approach's forward-looking and heuristic value, particularly for future advancements in multi-shot/scene long video generation.

4. Regarding our proposed PMWD, we emphasize it represents just one optional inference method after frame-level training—not a core contribution of this paper. While the original PMWD shows significant advantages in addressing error accumulation, simpler diffusion-forcing sliding window inference methods could also be viable if they solve the error accumulation issue (e.g., using distribution matching techniques like DMD used in Causvid or self-forcing). Notably, PMWD's inherent parallelization capability provides distinct advantages with sufficient GPU resources, and if combined with compressed representations of infinite historical frames, offers substantial future potential.

5. Open Source Commitment: We will promptly release our dataset processing code, model training/inference/evaluation code, and part of multi-shot video data with frame-level caption annotations.

6. More clear implementation details: As described in line 297 and the caption of Table 1, we present a comparative analysis of models trained on the Wanx2.1 dataset using either global video-level or frame-level prompts under identical training and inference configurations. All experiments were conducted using our proposed Parallel Multi-Window Denoising (PMWD) method. A 5-second video corresponds to 21 latents, which matches the frame length used during Wanx training; therefore, we follow diffusion forcing protocols by treating these 21 latents as a sliding window unit. After each generation step, the window slides by 1/3 of its length (7 latents). For the FIFO approach, we adhere to the original paper's configuration with a slide increment of 1 latent per step. While these implementation details do not affect the conclusions, we apologize for not clearly presenting them. We will provide more detailed explanations in the main text and supplementary materials according to the reviewers' suggestions, and ensure reproducibility through open-source code.

Finally, we sincerely thank all reviewers for their meticulous reviews and kindly request consideration of the heuristic contributions this paper offers to the field.

Response To Reviewer uVip

1. Why only Gemini Pro 2.5 was used for prompt generation. How will the quality metrics value change with other VLM use?

Unlike frame-level image captioning schemes, our approach utilizes Vision-Language Models (VLM) to first establish global understanding of the entire video before annotating individual frames. This represents a global-to-local annotation methodology that enables comprehensive global comprehension along with ID understanding and memory capabilities. These requirements place extremely high demands on VLM performance, which is why we selected the state-of-the-art Gemini Pro model available at the time. While other VLM models could potentially be used, they would likely yield inferior results due to the inherent challenges of long video understanding. The cost of annotating the entire dataset using VLMs is extremely high. Therefore, we did not conduct additional VLM comparisons and only used the most advanced VLM model available. We will include comparisons of different VLMs in the supplementary materials.

2. More clear discription of Confusion Degree

The traditional Video-Level Video-Text Similarity may assign higher scores even when content from different scenes are inappropriately generated in one frame, not considering temporal and semantic inaccuracies. A frame should only contain content specified in its corresponding prompt. If a frame contains content that should not appear in it, we define this as confusion. For example, when all content is generated in every frame and they merge together. In implementation, if the similarity between prompt_i of frame_i and the visual content of another frame_j is greater than the similarity between prompt_i and prompt_j, it indicates that frame_j may contain redundant content that should belong to prompt_i. This is specifically defined by the frame-level text-text similarity and frame-level text-frame similarity in Equation 3.

3. Can we use PMWD for non-overlapping frame shifts? What will be the effect of such operations?

If PMWD is used for non-overlapping frame shifts, it would result in completely discontinuous video with each segment generated independently. The overlapping regions represent the information from other segments visible to each segment. While this approach shows significant potential, it needs to be combined with infinite context and compression technologies like framepack or memory techniques to improve implementation consistency in future work.

4. Computational cost evaluation

For autoregressive long video generation, such as generation of 1 minute video, the computation cost and memory usage of PMWD are the same as those of sliding window inference in diffusion forcing in sequential mode inference. This is because all inference methods use the same sliding window length (21 latents): for diffusion forcing, the sliding is 7 latents per step, while for PMWD it is a 7-latent overlap, resulting in the same additional overhead. However, in parallel mode, PMWD can utilize more GPU resources to generate each sliding window in parallel, where the total generation time is inversely proportional to the resources used—a trade-off.