Context Preserving Autoregressive Frame Generation for Bounded Video

We appreciate reviewer for the valuable comments. Our responses are as follow:

W1. Contribution

- Novelty issue regarding on autoregressive diffusion process

Our work focuses on bounded video generation which have unique challenge, in terms of causality between frames. Specifically, for open-ended video generation in previous works, next frame is predicted solely based on preceding frames. However, bounded video generation requires to consider not only the past context, but also the future context which is imposed by ending frames. If these two contexts are not properly integrated, discontinuity emerges in the output video due to the difference of the contexts.

In this paper, we present a new problem Context Gap, and propose a novel method to guide autoregressive diffusion process to reduce the context discrepancy. Specifically, we leverage contextual information of a video sequence, which is not addressed in the prior work [1]. Moreover, our method is the first approach to capture and edit the context of frame, which is novel for video generation domain.

---

W2. Analysis on bias

According to the reviewer's concern, we conducted an analysis for the bias of model. Following the previous work [6], we measure Relative MSE to compare the model bias for two cases, flipped frames and reversed time schedule. Specifically, for predicted clean frames $\hat{z}(t)$ for forward direction video, we obtain the noise prediction using flipped $\hat{z}(t)$ to investigate the model's bias for flipped frames. For reversed time schedule, we utilized $\hat{z}(t)$ while the noise are added by the defined diffusion time. We present the comparison of Relative MSE in Table below.

Table 1 Related MSE for each diffusion time step (T=64)

The results shows that the mode bias is severe for flipped frames. Specifically, the error escalates in the middle of diffusion generation process, where the context started to be formed. In contrast, the model exhibits relatively low bias for reversed time schedule across the whole time steps. The results verifies a superiority of our approach, avoiding the severe bias of model against frames in flipped order.

[6] Kim, Jihwan, et al. "Fifo-diffusion: Generating infinite videos from text without training." Advances in Neural Information Processing Systems, 2024

---

W3. Discussion on quantitative comparison

We will modify the table according to the comments. Moreover, according to the concerns of the reviewer, we briefly analyze the underperforming metrics. First, regarding on the visual quality, the main difference arises from the initialization of noisy new frames. Specifically, FIFO initialize the frames adding noise in the original video frame (start condition). However, our method initialize frames based on the predicted outputs of preceding frames near large diffusion time, where the predicted clean frames contain only low frequency components. Hence, the visual information provided in the initialization induces the slight gap of visual quality. We kindly remind the reviewer that FIFO is a method for open-ended video. Second, regarding on the MAE(loop), the ending condition is imposed near large diffusion time in the proposed bounded diagonal denoising. In other words, the first frame in the start context has smallest noise level, while the first frame in the ending context has largest noise level. This gap induces the slight differences in looped video.

---

W4. Issues regarding on SVGD

Relation of SVGD with main algorithm

We propose a SVGD-based guidance method to extract contextual information of video frames. Specifically, bounded video generation have unique challenge, in terms of causality between frames. Bounded video generation requires to consider not only the past context, but also the future context which is imposed by ending frames. If these two contexts are not properly integrated, discontinuity emerges in the output video due to the difference in the contexts. To address the context gap, we extract the SVGD-based context gradient, and guide the diffusion process to generate smooth transition between the two sequences.

According to the reviewer's concern, we provide the ablation study on SVGD and mixing, which are the components to enhance the smooth transition between the two context. We compare temporal coherency in both pixel level and semantic level, using Temporal Flickering(pixel) and Background consistency(CLIP ViT-B/16) respectively. In particular, we measure the metric for frames located in 32-35 indices to focus on the context gap. The baseline is naive application of bounded autoregressive denoising, without any techniques proposed in this paper. The results are presented in Table 2.

Table 2 Ablation study of each components for context gap. (32-35) refers the frame indices in the middle of the video.

The result demonstrates the effectiveness of each component in our method for the context gap. Without mixing and context gradient, the context gap is not fully addressed. Specifically, both metric evaluated in the middle of the video(33-35 frames) clearly shows the reduced discontinuity by the proposed components. In particular, temporal flickering is more sensitive to show the discontinuity by measuring the pixel-level difference, compared to the background consistency which measures the semantic continuity.

---

W5. Comparison with training-based method

We appreciate the reviewer's comments. Unfortunately, due to the limited time and resources, we could not conduct the comparison with training-based method.

Q1. Bounded Video Generation and Video Interpolation

We appreciate the reviewer's comment. Our work follows the task definition and experimental settings suggested in the previous work [7]. Video interpolation focuses to generate frames to connect the boundary with shortest path. In contrast, bounded video generation aims to produce videos with plausible motions while accommodating varying degrees of contextual deviation.

[7] Feng, Haiwen, et al. "Explorative inbetweening of time and space." ECCV, 2024.

Q2. $\tilde{z}_j$ in Algorithm 1

We denote the diffusion vector $z$ as $x$ for forward direction and $y$ for backward direction, as mentioned in the paragraph in page 5 of main paper. Accordingly, $\tilde{z}$ refers the vector for frame initialization obtained by equation (5) in manuscript. We will resolve the confusing notation in the final version.

Q3. Failure case

As discussed in W3, the bounded diagonal denoising induces timestep gap between the first frames of starting context and ending context. This induces slight discontinuity at the end of looping video, which is reflected in the MAE(loop).

Suggestion: Recent model

Our method is applicable only to video model that supports distinct time embedding for each frame. In case of Videocrafter2, the model duplicates the time embeddings for each frame, which enables distinct time condition for each frame. Recent models such as cogVideo(mmDiT), Wan/Hunyuan(DiT) do not support the distinct time embeddings, limiting compatibility with our approach. We also tried CausVid, however, the reversed frame generation is not supported, which is the key observation in section 3.1 in our paper. We believe it is due to the difference of model knowledge that is induced by the different training method, which would be future research direction.

We appreciate your thoughtful comments. Our responses are as follows:

Video interpolation aims to connect frames with shortest path. Specifically, in case of identical bound, video interpolation connects the two identical bounds with same frames, which is shortest path. In contrast, bounded video generation aims to generate video with plausible motions while satisfying the boundary. I believe that effectively managing context deviation within videos is a critical factor for bounded video generation.

We believe that the degradation in the mentioned metrics are due to the different initialization of the frames. Specifically, the mentioned methods provide more visual information to the frames. FIFO(FVD_{16}, VID) and ViBiD(MAE, Loop) initialize frames with original images which contains high frequency information. In contrast, our method use predicted frames which only includes low frequency information, to consider context. Accordingly, our method provide less prior information to the frames, which leads to the degradation of metric.

Moreover, following the reviewer's comment, we provide human evaluation to verify a improvement in visual quality and the motion dynamics. 50 videos are evaluated by 22 participants. We compare our method with bounded video generation methods, TRF and ViBiD, while FIFO is excluded as it is an open-ended video. Preferences were rated on a scale of 0 to 5. The results demonstrate that our method improvements for both visual quality and motion dynamics.

Table 1 Human evaluation

We appreciate the reviewer's comments. Unfortunately, due to the limited time and resources, we could not conduct the comparison with training-based method. Recent models such as Wan/Hunyuan(DiT) do not support the distinct time embeddings, which requires new architectural modification for the fine-tuning. Moreover, fine-tuning requires to construct text-video paired dataset, which was no able to conduct during limited rebuttal period.

Regarding on errata, we will modify our manuscript in final version.

We would like to emphasize the contribution of our method extends beyond the training.

• We explore context of video frames, which is underexplored.
• We propose method to capture and edit frame context, which is novel in video generation domain.
• We present a novel problem context gap, and successfully mitigated by the proposed method.

We appreciate reviewer for the valuable comments. Our responses are as follow:

W1. Contribution

1. Novelty issue regarding on autoregressive diffusion process

Our work focuses on bounded video generation which have unique challenge, in terms of causality between frames. Specifically, for open-ended video generation in previous works, next frame is predicted solely based on preceding frames.

In contrast, bounded video generation requires to consider not only the past context, but also the future context which is imposed by ending frames. If these two contexts are not properly integrated, discontinuity emerges in the output video due to the difference of the contexts. In this paper, we present a new problem Context Gap, and propose a novel method to guide autoregressive diffusion process to reduce the context discrepancy.

2. Novelty issue on SVGD

We propose a SVGD-based diffusion guidance method to extract contextual information of video frames, which is fundamentally different from prior work~[3] in both motivation and mechanism. The primary goal of [3] is to preserve inter-frame relationships after video editing, where the authors employ a Gaussian kernel based on the L2 difference between frames. This approach relies solely on visual similarity, without considering the temporal causality between frames.

In contrast, our method mainly aims to capture contextual information of video frames. To consider temporal causality, our method presents a moving average instead of gaussian kernel. The moving average monotonically decreases with respect to the distance in frame index, which reflects temporal dependency between frames. Overall, our paper focuses on the emergence of frame context and propose a novel method to adjust context for bounded video generation, which is underexplored in previous works.

---

W2. Evaluation on semantic consistency

We kindly remind the reviewer that the prompts are randomly generated by LLM, without consideration on an irreversibility of motions. Regarding on the metric, temporal consistency is evaluated in both pixel level and semantic level consistency. Specifically, "Background consistency" measures cosine similarity between CLIP visual features(CLIP ViT-B/32), while "Temporal flickering" measures MAE difference of pixel values. According to reviewer’s concern, we provide `Subject Consistency’ which measures cosine similarity between [CLS] token features of DINO. In Table 2, the result shows that our method outperforms the previous methods in Subject Consistency, by improved semantic consistency. Moreover, we add Imaging Quality of VBench, to additionally evaluate visual quality of videos. Our method also outperforms the previous approach in both metrics.

---

W4. Errata

We will modify the main paper in final version

---

(W3/Q1/Q2). Further analysis

1. Stability analysis

Following the reviewer's comment, we provide stability analysis of our method for the variable length of video. Specifically, we focus on the visual quality and context deviation, which is directly affected by the video length. For the measurement of context gap, we obtain the MAE difference between the frames in the transition of the two sequences(MAE between 33,34 frames). In Table 2, we provide FID and context gap for various video length.

Table 2 Ablation study for variable length

As shown in Table 2, our method retains the performance in varying video length, in terms of both temporal coherency and visual quality.

2. Further study on context gap

2.1. Varying context mismatch

According to the reviewer’s concern, we investigate the performance of our method to the varying degree of context gap. To quantify the context gap for a given motion, we independently extended input frames using bidirectional denoising, and calculated the MAE between the final frames of each direction. Then, we measure the MAE difference in the middle of video generated by our method to the corresponding motion. A scatter plot would be appropriate for this analysis. However, considering the conference guideline, we present statistical values instead. We grouped the instances into 7 distinct partitions according to the context gap of original motion. Then, we measure mean and standard deviation of each groups, as shown in Table 3.

Table 3 Varying degree of context gap. Context Gap is unnormalized MAE difference in pixel values.

The result shows that our method improve the gap for all groups. However, in the partition 7, the result shows increased mean and deviation, which indicates the presence of discontinuous video in our result.

2.2. Formation of frame context and context gap

We appreciate the valuable comment. The context gap emerges by the causality of frames which is unidirectional. The next frame is generated only considering the past context, not the future context. Therefore, the context of frame deviates as the dependency to the initial frame become weak, which is also demonstrated in section 4.3.2 of main paper. According to the causal structure of frames, the video model has bias of generating frames in forward direction. We believe the the model's bias for the unidirectional frame forms the context gap in video generation. To support our claim, we present the bias analysis for video model.

Following the previous work [4], we measure Relative MSE to compare the model bias for two cases, flipped frames and reversed time schedule. First, we investigate the model's response for flipped frame (reversed causality). Next, we explore the model's output for reversed time scheduling with forward frame, which demonstrates the potential of our approach to indirectly extract the reversed causality from model's knowledge.

Specifically, for predicted clean frames $\hat{z}(t)$ for forward frames, we obtain the noise prediction using flipped $\hat{z}(t)$. For reversed time schedule, we utilized $\hat{z}(t)$ while the noise are added by the reversed diffusion time. We present the result of Relative MSE in Table 4.

Table 4 Average of related MSE for each diffusion time steps

The results shows that the mode bias is severe for flipped frames. Specifically, the error starts to escalate from the middle of diffusion generation process, which indicates the formation of context. Simultaneously, the prediction of model contains the change of contents, which is demonstrated by high value of Related MSE. The analysis shows the unidirectional causal structure within the model, which causes the context gap.

Moreover, the model exhibits relatively low bias for reversed time schedule across the whole time steps. The results verifies a potential of our approach to indirectly extract reversed causality, leveraging the unidirectional bias of video model.

[4] Kim, Jihwan, et al. "Fifo-diffusion: Generating infinite videos from text without training." Advances in Neural Information Processing Systems, 2024

Dear Reviewer PqNB,

Thank you for your participation in the review process. Please engage in the discussion phase by following these guidelines:

• Read the author rebuttal;
• Engage in discussions;
• Fill out the "Final Justification" text box and update the "Rating" accordingly.

The deadline is Aug 8, 11.59pm AoE.

Thanks,

AC

We appreciate the fruitful comments. The response is as below:

Recent models

Our method is applicable only to video model that supports distinct time embedding for each frame. Specifically, Videocrafter2 model duplicates the time embeddings to the dimension of frame length, which enables distinct time condition for each frame. Recent models such as cogVideo(mmDiT), Wan/Hunyuan(DiT) do not support the distinct time embeddings, limiting compatibility with our approach. We also tried CausVid, however, the reversed frame generation is not supported, which is the key observation in section 3.1 in our paper. We believe it is due to the difference of model knowledge that is induced by the different training method, which would be future research direction.

---

Evidence for reversed frame generation

To support the reversed frame generation in the selected model, we conduct an analysis on the model’s prediction. Following the previous work[1], we present the Relative MSE, as shown in Table 1.

Table 1 Related MSE for each diffusion time step ($T$=64)

The results demonstrates that the reversed frame generation is supported, showing low relative MSE. In contrast, the value is high for flipped frame, which verifies the severe bias for the prediction on flipped frames. Specifically, the error in flipped frame starts to escalate from the middle of diffusion generation process, which indicates the formation of frame context.