Lifelong Learning of Video Diffusion Models From a Single Video Stream

Before addressing specific concerns, we note that several reviewers have entirely missed the Google Drive links [a,b,c,d] (Section 5 Footnotes 2,3,4,5) containing video samples from Lifelong and Offline Learning models for every dataset (lower quality videos viewable online here). Some reviewers also missed the reference to the quantitative results for two more baselines in the appendix for every dataset (Section 5 Line 398, Appendix D). These oversights have led to lower scores and skepticism regarding our main claim—that lifelong learning of video diffusion models is not only feasible but can achieve performance comparable to offline training given the same number of gradient steps. We kindly ask that you visit these resources should you question the validity of our claim and re-evaluate your scores in light of these results.

To substantiate our main claim even further, we have incorporated new results on a real-world driving dataset despite operating within the limits of resources available to an academic lab. Please refer to the rebuttal addressed to Reviewer KLJW for the experiment setup and qualitative and quantitative results. These results support the validity of our claim on complex real-world video data.

We now address specific questions and concerns.

The paper does not provide comparisons with world models and video prediction models, despite the fact that the proposed datasets share strong similarities with them.

We agree that our lifelong learning of autoregressive video generative models is related to those two problems. However, it is difficult to directly compare our lifelong models to the baselines from those problems. World models require action conditioning, and the introduced video datasets do not contain actions. Video prediction models are incapable of generating multiple data samples, so it's not straightforward to compare against them via video generative modeling metrics. However, given the relevance to our problem setup, we will cite these world model papers in the camera-ready draft.

While continual learning and lifelong learning are important topics, the relevance of the proposed single long, continuous video streams seems more crucial in game generation settings, such as in Genie. The necessity of applying lifelong learning to general video generation remains unclear.

We are glad that the reviewer finds single long, continuous video streams valuable to game generation settings and highlight that the proposed datasets (ex. Lifelong PLAICraft) could also benefit the game generation community. As the reviewer noted earlier, autoregressive video modeling is intimately related to world modeling since we can view the former as the latter marginalized over actions. The key motivation behind this work is to make progress toward lifelong learnable vision-based embodied AI whose world model can be updated in real-time. We believe that agents with updatable world models that are combined with planners, for example a language model as in GenEx, is key to having flexible embodied AI agents that are not constrained to the behaviors associated with pretraining checkpoints or can only be updated periodically. We chose to work with video modality in the beginning because it is much easier to generate datasets that work purely on video modalities, and we expect action conditioning to not significantly perturb our results.

Please include video samples in the supplementary material rather than providing Google Drive links.

Thank you for your comment. We will update this for the camera-ready draft.

---

We appreciate that the reviewer believes that our experimental results are robust and that they strongly support our claim. We hope that our rebuttal experimental results, which further support the fact that lifelong learning of video diffusion models is possible for complex, real-life datasets, addresses the reviewer’s concern that the previously covered scenarios are relatively narrow. In addition, we hope that our rebuttal’s clarification on the motivation of our work being the first step toward lifelong learnable vision-based embodied AI clarifies the value of our work. Thank you again for your engagement and we would be delighted to have further discussions and address any other questions.

Before addressing specific concerns, we note that several reviewers have entirely missed the Google Drive links [a,b,c,d] (Section 5 Footnotes 2,3,4,5) containing video samples from Lifelong and Offline Learning models for every dataset (lower quality videos viewable online here). Some reviewers also missed the reference to the quantitative results for two more baselines in the appendix for every dataset (Section 5 Line 398, Appendix D). These oversights have led to lower scores and skepticism regarding our main claim—that lifelong learning of video diffusion models is not only feasible but can achieve performance comparable to offline training given the same number of gradient steps. We kindly ask that you visit these resources should you question the validity of our claim and re-evaluate your scores in light of these results.

To substantiate our main claim even further, we have incorporated new results on a real-world driving dataset despite operating within the limits of resources available to an academic lab. Specifically, we evaluated Offline and Lifelong Learning on a continuous video stream comprising 550K training and 20K test frames, recorded from a single car’s dashcam over multiple driving sessions totalling 8 hours. We refer to this dataset as Lifelong Drive and will release it upon acceptance. The dataset consists of 1 to 40-minute driving sessions, with transitions occurring when the car is started and parked. Fade-in and out animations are applied at the video concatenation boundaries to ensure smooth session transitions. Frames are encoded into 4×64×64 latents using the Stable Diffusion encoder. Other experiment details are identical to the Lifelong PLAICraft experiment.

Lifelong Drive qualitative results are presented here, and the quantitative results are below. The samples produced by Offline and Lifelong Learning are qualitatively indistinguishable and quantitatively comparable.

In the style of Appendix E, we also report p-values from two-sided T-tests where the null hypothesis is that there is no difference between the test stream performance metrics from Offline and Lifelong Learning. The tests fail to reject the null hypothesis that the performances of the two algorithms are not different ($\alpha=0.05$).

We now address specific questions and concerns.

There are no improvements to experience replay nor ablations on forgetting.

Please refer to the last and first points of our rebuttal addressed to Reviewer gPNY.

There are no ablations on long horizon stability nor sensitivity to different training conditions.

We present new qualitative results that show the similarity in the quality of Lifelong and Offline learning's long video samples here. Our original results also show that Lifelong and Offline Learning perform comparably across datasets, highlighting the former's robustness to training conditions.

The absence of experiments on how the model adapts to distribution shifts or scales with increasing data exposure weakens the validity of its claims.

Distribution shift is synonymous with nonstationarity. Lifelong Bouncing Balls (O), (C) results in Appendix D localizes the effect of color nonstationarity on all baselines, while Lifelong PLAICraft measures the effect of distribution shift in multiple timescales. Appendix G shows the forgetting behaviors associated with distribution shift. Lastly, Appendix F illustrates how model performance on test stream scales with increasing data exposure for all datasets.

FVD, minADE, ColorKL are standard for assessing generative video models.

We note that minADE (trajectory metric) and ColorKL (our original metric) as video metrics are not standard.

---

We appreciate that the reviewer finds our experiments well-structured and our findings interesting. We hope that our rebuttal results and analysis from the appendix will help the reviewer share Reviewer YRxK’s sentiment that our results strongly support the claim that training autoregressive video diffusion models from a single video stream can be as effective as offline training given the same number of gradient steps.

Before addressing specific concerns, we note that several reviewers have entirely missed the Google Drive links [a,b,c,d] (Section 5 Footnotes 2,3,4,5) containing video samples from Lifelong and Offline Learning models for every dataset (lower quality videos viewable online here). Some reviewers also missed the reference to the quantitative results for two more baselines in the appendix for every dataset (Section 5 Line 398, Appendix D). These oversights have led to lower scores and skepticism regarding our main claim—that lifelong learning of video diffusion models is not only feasible but can achieve performance comparable to offline training given the same number of gradient steps. We kindly ask that you visit these resources should you question the validity of our claim and re-evaluate your scores in light of these results.

To substantiate our main claim even further, we have incorporated new results on a real-world driving dataset despite operating within the limits of resources available to an academic lab. Please refer to the rebuttal addressed to Reviewer KLJW for the experiment setup and qualitative and quantitative results. These results support the validity of our claim on complex real-world video data.

We now address specific questions and concerns.

The only baseline is the iid method and no ablation study is present (related: how critical is the replay buffer?).

We summarize the importance of replay buffer in the main text (Section 5.4, Line 398) where we point the readers to Appendix D where, for all datasets, we report and elaborate on the quantitative results for two additional baselines: naive sliding window lifelong learning that ablates replay loss from the replay objective (No Replay) and unlimited memory experience replay (Full Replay). Additional replay buffer size results for Lifelong 3D Maze are presented in Table 8. Furthermore, Appendix F and G respectively showcase how fast different baselines can learn to perform well on the test stream and how much forgetting affects the baselines’ final models on the train stream for all datasets.

The paper demonstrates its finding on a single architecture. Would the choice of the underlying architecture affect the results?

Given the related work [1] that demonstrates how both U-Net and Transformer-based non-generative models can be online learned on a data stream made from stitching short, unrelated videos, we expect the architecture to not significantly affect the results. We underscore that we have successfully demonstrated that it is possible for lifelong learned video diffusion models with tens of millions of parameters to achieve a performance good enough to be comparable to offline learning across multiple datasets and two parameter sizes, a finding that was not previously evident.

This work does not claim or introduce any technical novelties, apart from applying an existing technique to a somewhat different context.

We believe that the machine learning community will nonetheless benefit from being aware of our novel investigation into a different kind of video generative model training regime for both its carefully designed datasets and its surprising findings. Our streaming learning setup is particularly relevant to the development of lifelong learnable vision-based embodied AI that can update its generative predictive model in real time. In addition, although our focus is on investigating a new problem setup, no prior work has introduced batch-level duplication of the latest sliding window with different noising levels for better estimation of the streaming diffusion loss in Equations (3) to our knowledge.

---

We appreciate that the reviewer finds the key result of the paper interesting. We hope that our video model samples, additional baselines, analysis, and results will help the reviewer share Reviewer YRxK’s sentiment that our results strongly support the claim that training autoregressive video diffusion models from a single video stream can be as effective as offline training given the same number of gradient steps.

[1] Carreira, J., King, M., Patraucean, V., Gokay, D., Ionescu, C., Yang, Y., Zoran, D., Heyward, J., Doersch, C., Aytar, Y., Damen, D., and Zisserman, A. Learning from one continuous video stream, 2024.

Before addressing specific concerns, we note that several reviewers have entirely missed the Google Drive links [a,b,c,d] (Section 5 Footnotes 2,3,4,5) containing video samples from Lifelong and Offline Learning models for every dataset (lower quality videos viewable online here). Some reviewers also missed the reference to the quantitative results for two more baselines in the appendix for every dataset (Section 5 Line 398, Appendix D). These oversights have led to lower scores and skepticism regarding our main claim—that lifelong learning of video diffusion models is not only feasible but can achieve performance comparable to offline training given the same number of gradient steps. We kindly ask that you visit these resources should you question the validity of our claim and re-evaluate your scores in light of these results.

To substantiate our main claim even further, we have incorporated new results on a real-world driving dataset despite operating within the limits of resources available to an academic lab. Please refer to the rebuttal addressed to Reviewer KLJW for the experiment setup and qualitative and quantitative results. These results support the validity of our claim on complex real-world video data.

We now address specific questions and concerns.

The t in Equation (3) for lifelong learning exactly matches the total duration of the training video. Does i in Equation (2) cover the entire duration of the training video despite being randomly selected? Ensuring this would make the comparison fair.

Thank you for the great question. Equation (3)’s t matches the number of training frames observed so far by the model. Equation (2)’s i covers the entire duration of the training video in our experiments to ensure that the comparison is fair. We will clarify this in the camera-ready draft.

If the training steps exceed the actual t in the training video, there is no difference between the lifelong and offline losses. How does the comparison between the two methods hold under these much longer training steps?

Our lifelong learning setup requires at least one minibatch index at every timestep to be a real-time video frame, Lifelong Learning and Offline Learning cannot be compared once we have sequentially performed a single gradient step for all sliding windows of the video stream.

Will the same outcome apply for larger video U-Nets?

While larger models could not be evaluated in this preliminary investigation due to compute restrictions, given Deepmind’s recent work [1] that shows that online learned discriminative models with 8 and 350 million parameters can match IID-trained model performance, we expect the same outcome to apply to larger video diffusion models. We note that our diffusion models have the same parameter count as the original paper's models [2]. Regardless, we have shown that lifelong learning can achieve a performance comparable to offline learning across a wide range of datasets.

Recent papers demonstrate that offline training for autoregressive diffusion can generate robust infinite single-game simulations for complex datasets. These works are missing from the paper.

We believe that lifelong learning of advanced diffusion models capable of robustly generating long videos is an exciting next step and have included these references in the updated paper. We note that even the largest of these models, Genie 2, only stably generates videos up to a minute [3].

---

We appreciate that the reviewer believes that the lifelong learning of video diffusion models is novel and can make a valuable contribution to the literature. We hope that our qualitative results that highlight indistinguishableness of lifelong and offline learning samples and comparison fairness clarifications will help the reviewer share Reviewer YRxK’s sentiment that our results strongly support the claim that training autoregressive video diffusion models from a single video stream is not only possible but can also be as effective as offline training given the same number of gradient steps.

[1] Carreira, J., King, M., Patraucean, V., Gokay, D., Ionescu, C., Yang, Y., Zoran, D., Heyward, J., Doersch, C., Aytar, Y., Damen, D., and Zisserman, A. Learning from one continuous video stream, 2024.

[2] Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C., & Wood, F. (2022). Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35, 27953-27965.

[3] Genie 2: A large-scale foundation world model. (2025, March 25). Google DeepMind. deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/