Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing

Anonymous link for additional experiment results

https://anonymous.4open.science/r/additional-exp-results-for-anonymous-github-F6EB/readme.md This includes: Table_R1, Table_R2, Figure_R1, and Figure_R2

$~$

Q1: Evaluation on VBench

VBench is primarily designed for text-to-video evaluation. For our assessment, we selected four metrics: aesthetic quality, imaging quality, motion smoothness, and temporal flickering. The first two measure spatial (appearance) quality, and the last two assess temporal consistency. We compared Ca2-VDM and OS-Ext on the Skytimelapse test set.

As shown in Table_R2, Ca2-VDM achieves comparable performance in both appearance quality and temporal consistency. Given our primary focus on efficiency, we conclude that Ca2-VDM matches the bidirectional baseline while being more efficient in both computation and storage for autoregressive video generation.

$~$

Q2: The claim on applicability of the method to long-term or live-stream video generation is not validated.

Long-term video generation remains an open problem, with numerous challenges to be solved, including short-term frame quality, long-term content consistency, and generation efficiency.

For our Ca2-VDM, as claimed in the introduction, the primary focus is to improve the generation efficiency (both computation and storage) of auto-regressive video generation. Providing superior long-term video generation performance (surpassing SOTAs) is not the primary focus of Ca2-VDM.

For the visual quality, we achieve comparable visual quality to SOTA methods, as evidenced by the FVD results on MSR-VTT and UCF-101 datasets (Tables 1 and 2), as well as the results on VBench (Table_R2).

For the long-term quality, we conducted additional experiments to show long-term content drift, as in Figure_R1. In fact, OS-Ext and our Ca2-VDM show comparable visual quality. Both models exhibit a similar degree of error accumulation over time.

$~$

Q3: Essential References Not Discussed

Hoogeboom, Emiel, et al. "Autoregressive Diffusion Models." in ICLR, 2022

Thank you for the suggested reference. This work proposes Autoregressive Diffusion Model (ARDM), which is built on autoregressive models and incorporates techniques from discrete diffusion models. It offers efficient training without the need for causal masking and parallel generation at test time.

In contrast, our Ca2-VDM is built on continuous latent diffusion models for video generation, and enables efficient autoregressive generation with causal attention and cache sharing. We will add more discussion in the revised paper.

$~$

Q4: Clarity: Confusion in the Introduction about the temporal attention

Thank you for your suggestion. We agree that temporal attention does not necessarily have to be bidirectional. Our introduction aims to highlight that, in current video diffusion models, temporal attention is commonly used and is typically applied in a bidirectional manner, both in UNet and DiT structures. We will improve the writing of the introduction to avoid any potential misleadings in the revised paper.

$~$

Q5: Quality: Unsatisfactory temporal consistency, limited qualitative results.

We would like to clarify that the causal attention is not intended to improve the temporal consistency. The causal attention is designed to combine with the kv-cache queue and cache sharing to improve the generation efficiency.

For temporal consistency, as evidenced in Table 3 and Figure 5, our Ca2-VDM offers extendable conditions and achieves better temporal consistency than fixed-length condition SOTA models (StreamT2V, Gen-L-Video, and OS-Fix). We also conducted an additional evaluation in terms of frame-differencing, as shown in Figure_R2. The results show that our method has good temporal consistency, while the other three have periodic content mutations (at the edge of each autoregression chunk), especially GenLV.

Due to limited computational resources and the tight schedule of the rebuttal period, we regret that currently we cannot conduct additional large-scale training to further improve the quality of videos in the supplementary material.

$~$

Q6: Originality: Concerns about the contribution

We would like to clarify that applying auto-regressive modeling and kv-cache to VDMs is not trivial (as acknowledged by Reviewer-dKiu). Our original contribution includes: 1) kv-cache queue boosted with cyclic-TPE and 2) cache sharing

1. KV-cache queue with cyclic-TPE: It addresses the cases where positional indices grow beyond the training length while keeping the training-inference alignment. More detailed analysis can also be found in the Q4 of Reviewer-19uZ.
2. Cache sharing: It enables the model to store only KVs of clean frames, leading to much less GPU memory usage than concurrent work (e.g., Live2diff). More detailed analysis can also be found in the Q4 of Reviewer-7MWe.

Anonymous link for additional experiment results

https://anonymous.4open.science/r/additional-exp-results-for-anonymous-github-F6EB/readme.md This includes: Table_R1, Table_R2, Figure_R1, and Figure_R2

$~$

Q1: Essential References Not Discussed

Yin, Tianwei, et al. "From slow bidirectional to fast causal video generators." CVPR 2025.

Thank you for your suggestion. This work also introduces causal attention into autoregressive video diffusion models (AR-VDMs). They use distributed matching distillation to align the visual quality of causal student model to the bidirectional teacher model.

This referenced work is a concurrent work with our paper. I.e., Our initial version of Ca2-VDM was developed concurrently with it.

Our approach distinguishes itself through the kv-cache queue (with cyclic-TPE) and cache sharing mechanisms that provide a both computational and storage efficient framework for (AR-VDMs). We will include this referenced work and add additional discussions in our revised paper.

$~$

Q2: Any negative impact or instability observed during finetuning from bidirectional VDM ?

We did not observe any obvious negative impact or instability during our fine-tuning.

Indeed, as verified by the aforementioned reference work (Yin, Tianwei, et al, 2025), distilling the knowledge from a pre-trained bidirectional teacher model to the causal student model brings quality improvement over directly fine-tuning. Additionally, as discussed in our "Possible Future Directions" section of the Appendix (Sec. F), pretraining the causal attention from scratch might have potential improvements. We will explore these two strategies in our future work.

$~$

Q3: Content drift over longer generated videos

As discussed in our Introduction section, bidirectional models are also capable of generating longer videos beyond the training video length, and this is not a unique strength of causal models. Also, we did not claim long video quality improvement over the bidirectional baseline (OS-Ext) as our contribution.

We conducted additional experiments to show long-term content drift. As shown in Figure_R1, OS-Ext and our Ca2-VDM show comparable visual quality. Both models exhibit a similar degree of error accumulation over time.

Nevertheless, long-term video generation remains an open problem. Both models face challenges in this regard. We can apply the aforementioned distillation technique to enhance the visual quality.

$~$

Q4: GPU memory usage between Ca2-DM and baselines

We conducted empirical GPU memory statistics, as shown in Table_R1

We compare Ca2-VDM with a concurrent work, Live2diff [1]. It stores the kv-cache for every denoising step (with different noise level $t$ and thus different KV features) , which costs much more GPU memory than ours. Live2diff uses StreamDiffusion [2]'s pipeline denoising, which puts frames with progressive noisy levels into a batch and generates one frame each autoregression step. So, its batch size in the model forward shape is equal to the denoising steps, i.e., $B=T$.

Note that cache sharing is an inherent property of Ca2-VDM, and it cannot be evaluated without cache sharing. Instead, we demonstrate that Ca2-VDM’s KV-cache memory cost is independent of denoising steps, as its fixed shape $(1,25, hw, C)$ ensures stable memory usage. In contrast, Live2diff's memory scales with $T$ (e.g., from 1.42 GB at $T=4$ to 17.70 GB at $T=50$), confirming that cache sharing saves $T \times$ GPU memory. As a result, Ca2-VDM requires only 0.86 GB (w/ PE) or 0.77 GB (w/o PE), with the difference due to spatial KV-cache for prefix-enhancement (PE).

While Live2diff uses distillation (e.g., LCM) to reduce $T$, existing few-step acceleration methods still require at least 4 steps to generate frames with acceptable quality. This means we can save at least 4 x GPU memory.

In addition, Live2diff cannot run 50-step denoising with KV-cache when GPU memory is limited or for high-resolution videos. This prevents proper evaluation of the teacher model before distillation, further restricting its applicability.

[1] Xing, Zhening, et al. Live2diff: Live stream translation via uni-directional attention in video diffusion models. 2024.

[2] Kodaira, Akio, et al. Streamdiffusion: A pipeline-level solution for real-time interactive generation. 2023.

$~$

Q5: How much is the model sensitive to cyclic TPEs

The cyclic TPE in Ca2-VDM is specifically designed to enable the model to generate videos that exceed the training length. In other words, a direct comparison of model performance with and without cyclic TPEs is not feasible. Also, it is not intended to enhance visual quality.

In terms of the model's sensitivity to cyclic TPE, we discussed the impact of training with cyclic TPE in the "Limitations" of Appendix (Sec. F). It requires the model to learn all the possible situations during training and making it difficult to converge.

Anonymous link for additional experiment results

https://anonymous.4open.science/r/additional-exp-results-for-anonymous-github-F6EB/readme.md This includes: Table_R1, Table_R2, Figure_R1, and Figure_R2

$~$

Q1: Additional metrics for per-frame perceptual evaluation

We conducted additional evaluations on the VBench (https://github.com/Vchitect/VBench) benchmark. It is primarily designed for text-to-video evaluation. For our assessment, we selected four metrics: aesthetic quality, imaging quality, motion smoothness, and temporal flickering. The first two measure spatial (appearance) quality, and the last two assess temporal consistency. We compared Ca2-VDM and OS-Ext on the Skytimelapse test set, as shown in Table_R2.

The results show that Ca2-VDM achieves comparable performance in both appearance quality and temporal consistency. Given our primary focus on efficiency, we conclude that Ca2-VDM matches the bidirectional baseline while being more efficient in both computation and storage for autoregressive video generation.

$~$

Additional evaluation on temporal consistency

We also conducted an additional evaluation in terms of frame-differencing, as shown in Figure_R2. The results show that our method has good temporal consistency, while the other three have periodic content mutations (at the edge of each autoregression chunk), especially GenLV.

$~$

Q2: GPU memory usage for w/ cache sharing vs. w/o cache sharing

Cache sharing is integral to Ca2-VDM due to the clean prefix condition design, meaning it cannot be evaluated without cache sharing. Instead, we compared Ca2-VDM with a concurrent work, Live2diff [1], which also uses kv-cache during autoregressive video generation. We conducted empirical GPU memory statistics, as shown in Table_R1

Live2diff stores the kv-cache for every denoising step (with different noise level $t$ and thus different KV features) , which costs much more GPU memory than ours. Live2diff uses StreamDiffusion [2]'s pipeline denoising, which puts frames with progressive noisy levels into a batch and generates one frame each autoregression step. So, its batch size in the model forward shape is equal to the denoising steps, i.e., $B=T$.

Benefited from cache sharing, Ca2-VDM’s KV-cache memory cost is independent of denoising steps, as its fixed shape $(1,25, hw, C)$ ensures constant memory usage. In contrast, Live2diff's memory scales with $T$ (e.g., from 1.42 GB at $T=4$ to 17.70 GB at $T=50$), confirming that cache sharing saves $T \times$ GPU memory. As a result, Ca2-VDM requires only 0.86 GB (w/ PE) or 0.77 GB (w/o PE), with the difference due to spatial KV-cache for prefix-enhancement (PE).

While Live2diff uses distillation (e.g., LCM) to reduce $T$, existing few-step acceleration methods still require at least 4 steps to generate frames with acceptable quality. This means we can save at least 4 x GPU memory.

In addition, Live2diff cannot run 50-step denoising with KV-cache when GPU memory is limited or for high-resolution videos. This prevents proper evaluation of the teacher model before distillation, further restricting its applicability.

[1] Xing, Zhening, et al. Live2diff: Live stream translation via uni-directional attention in video diffusion models. 2024.

[2] Kodaira, Akio, et al. Streamdiffusion: A pipeline-level solution for real-time interactive generation. 2023.

$~$

Q3: Unsatisfactory qualitative results of supplementary videos.

We acknowledge that the current qualitative results are not ideal. However, our primary focus is on improving the generation efficiency of video diffusion models, providing a computational and storage efficient framework. The visual quality improvement over the bidirectional attention baseline (OS-Ext) is not claimed as our contribution.

Due to limited computational resources and the tight schedule of the rebuttal period, we regret that we could not conduct additional large-scale training to further improve the qualitative results in the supplementary material. Instead, we conducted experiments on the Sky-Timelapse dataset, as shown in Figure_R1. We can observe that Ca2-VDM achieves comparable qualitative performance to OS-Ext, both in terms of single-frame quality and long-term visual content drift (with a similar degree of error accumulation during long-term video generation).

Nevertheless, we can still apply distillation techniques to improve the quality, e.g., distilling from a bidirectional teacher model to enhance the generation results of our causal generation model (as the student). Additionally, as discussed in the "Future Directions" section of the Appendix (Sec. F), pretraining the causal attention from scratch might have potential improvements.

Anonymous link for additional experiment results

https://anonymous.4open.science/r/additional-exp-results-for-anonymous-github-F6EB/readme.md This includes: Table_R1, Table_R2, Figure_R1, and Figure_R2

$~$

Q1: Unsatisfactory supplementary video quality

We acknowledge that the current qualitative results are not ideal. However, our primary focus is on improving the generation efficiency of VDMs, providing a computational and storage efficient framework.

Due to limited computational resources and the tight schedule of the rebuttal period, we regret that we can not conduct additional large-scale training to further improve the qualitative results in the supplementary material. Instead, we conducted experiments on SkyTimelapse, as in Figure_R1. It shows that Ca2-VDM achieves comparable qualitative performance to OS-Ext, both in terms of single-frame quality and long-term visual content drift (with a similar degree of error accumulation).

In addition, we can still apply distillation techniques to improve the quality, e.g., distilling from a bidirectional teacher model to enhance the visual quality of Ca2-VDM (as the student). Also, as discussed in the "Future Directions" of Appendix (Sec. F), pretraining the causal attention from scratch might have potential improvements.

$~$

Q2: Essential References Not Discussed

Modular approaches (ControlNet or T2I-Adapter style methods) for domain shift

ControlNet and T2I-Adapter are methods designed to introduce additional conditional signals to VDMs. They provide structural guidance like edges, depth maps, or segmentation maps.

However, our work does not focus on domain adaptation or structure-guided generation. Instead, we generate future frames autoregressively conditioned on previous frames, rather than conditioning on external guidance to synthesize realistic frames from sketches or animate realistic videos.

$~$

Q3: Highlight the differences between existing works about cache sharing

In our work, "cache sharing" specifically refers to sharing kv-cache across denoising steps, not for "sharing the cache across autoregressive steps". To this end, we are the first to introduce cache sharing to autoregressive VDMs.

To the best of our knowledge, there is also a concurrent work Live2diff [1] that introduces kv-cache to autoregressive VDMs. However, it stores kv-cache for every denoising step (with different noise level $t$ and thus different KV features), which costs much more GPU memory than ours.

In contrast, Ca2-VDM enables cache sharing and only stores the KVs of clean frames. This is boosted by our "clean prefix" conditional frames, the corresponding timestep sampling strategy, and the training objectives (cf. Eq.2).

We further provide GPU memory comparison in Table_R1. More detailed analysis can be found in the Q4 of Reviewer-7MWe.

$~$

Q4: The design of Ca2-VDM compared to existing techniques

We would like to clarify our original contribution. Our model is the first one to introduce 1) kv-cache queue boosted with cyclic-TPE and 2) cache sharing to VDMs.

1. KV-cache queue with cyclic-TPE: It's non-trivial to apply LLMs' kv-cache to VDMs. In LLMs, all tokens from the beginning are maintained during all autoregression steps. However, in VDMs, early conditional frames can be (or should be) removed due to the large memory cost for visual data, as the appearance and motion of new frames are primarily influenced by the most recent frames. Due to the kv-cache queue, early TPEs have been bound to previous KV-caches and can not be reassigned (as discussed in Figure 4(c)). This motivates us to propose Cyclic TPEs to keep training-inference alignment while enabling generation beyond the training length.

2. Cache sharing: As answered in Q3.

$~$

Q5: Exploration on distributed generation strategies

Thank you for your suggestion. We surveyed some related works. E.g., Video-Infinity [2] offers a distributed approach across multiple devices, where spatial modules operate independently, whereas temporal modules synchronize context via collaborative communications.

We plan to incorporate this kind of method into Ca2-VDM as future work to improve the scalability of video generation.

$~$

Q6: More evaluations for visually consistent transitions

We evaluated Ca2-VDM and OS-Ext on the Skytimelapse test set using VBench evaluation, as shown in Table_R2. The last two metrics (motion smoothness and temporal flickering) measure consistent transitions. It shows that Ca2-VDM achieves comparable performance with OS-Ext.

$~$

Q7: Writing improvement

Thank you for your suggestion. We will revise the paper to improve the writing and incorporate the discussions and clarifications in the above answers.

$~$

[1] Xing, Zhening, et al. Live2diff: Live stream translation via uni-directional attention in video diffusion models. 2024.

[2] Tan, Zhenxiong, et al. Video-infinity: Distributed long video generation. 2024.