Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion

We thank the reviewer for their valuable feedback. We are encouraged by the overall positive assessment, with reviewers agreeing that our work addresses a "critical and long-standing problem" [Vwja] and is "very well motivated" [2kqq]. The method is "clearly presented" [9RYz], "intuitive" [N5mF], and the paper is “well-written and easy to follow” [2kqq]. Our "quantitative results are solid" [9RYz] and were praised as a "thorough empirical section" [Vwja], demonstrating "broad success across VBench, human preference studies, and efficiency benchmarks" [N5mF].

We will now address the concerns and questions raised to further clarify our contributions.

• Gradient truncation: We implemented more heavily-optimized activation checkpointing and additional training parallelism to support Self Forcing training without gradient truncation. However, we find that models trained without gradient truncation do not produce better results than models with truncation. We hypothesize that models already learn representations that capture long-range dependencies during the pretraining stage, so that enabling long-range gradient flow during post-training becomes unnecessary. We will add those discussions in the final version of the paper.

• Comparison of DMD/SiD/GAN: We find GAN training is generally less stable and requires more hyperparameter tuning efforts than DMD/SiD. DMD is preferred over SiD when using a larger teacher model. Unlike SiD, DMD does not require backpropagating through the teacher and is therefore much more efficient. We will add a discussion in the final version.

• Hyperparameter sensitivity: Our method is not sensitive to the mentioned hyperparameters. The maximum KV-cache size is determined by the base model, and we always apply truncation to the last denoising step to minimize memory consumption. We do not observe better results with full backpropagation (without truncation), despite larger memory consumption. The number of diffusion steps is a hyperparameter that controls the quality/speed trade-off. Our method does not fail catastrophically when using less denoising steps. For example, we trained a 3-step model and it obtains a VBench total score of 84.26, only slightly lower than the 4-step version (84.31).

• Varying noise intensities: We interpret the question as follows: instead of using clean self-generated context frames as conditioning (as in the current Self-Forcing algorithm), can we add noise to self-generated context frames during training and inference (maybe with progressive intensity levels similar to rolling diffusion), and does it further reduce error accumulation?

  We do not find adding noise to context frames helpful in our case. Intuitively, when there is a train-test distribution mismatch (e.g., in teacher forcing), adding Gaussian noise to both distributions makes them have larger overlap and thereby mitigates cumulative errors. However, since our method does not have a train-test distribution mismatch, adding noise to the context is no longer necessary and only introduces drawbacks as discussed in L45-47.

We thank the reviewer for their valuable feedback. We are encouraged by the overall positive assessment, with reviewers agreeing that our work addresses a "critical and long-standing problem" [Vwja] and is "very well motivated" [2kqq]. The method is "clearly presented" [9RYz], "intuitive" [N5mF], and the paper is “well-written and easy to follow” [2kqq]. Our "quantitative results are solid" [9RYz] and were praised as a "thorough empirical section" [Vwja], demonstrating "broad success across VBench, human preference studies, and efficiency benchmarks" [N5mF].

We will now address the concerns and questions raised to further clarify our contributions.

• Long video generation: The goal of this paper is not to improve long video generation quality in the extrapolation setting. Our work aims to address exposure bias and to reduce error accumulation for videos generated within the training context length. This is a distinct problem from extrapolation (i.e., generating videos substantially longer than the training context), where we do not claim strong performance, as discussed in our limitations (L318-320). Extrapolation introduces a second distributional mismatch (of video length), in addition to the distribution mismatch that we address (ground-truth vs. generated history). While resolving exposure bias is a necessary step for long video generation, it is not sufficient by itself. To truly enable high-quality long video generation, future work needs to either scale up the video length during training, or investigate orthogonal techniques that enable better extrapolation.

• Visual comparisons with DF/TF: We will add additional qualitative comparisons with DF/TF in the final version of the paper (we are not allowed to add them in rebuttal due to NeurIPS policy restrictions). Visually, videos generated with DF/TF tend to exhibit significantly more error accumulation and often suffer from over-saturation over time.

• MALT Diffusion reduces error accumulation by adding noise to context frames during training. This was also used in earlier works including Diffusion Forcing and GameNGen, and we have some discussions in L44-47. We will add citations to MALT Diffusion in the final version.

• Memory usage: We use mostly bfloat16 precision for both training and inference. The peak allocated memory during training (batch size 1) is 53.2GB. During inference, it is 23.9GB Inference without text encoder offloading and 13.0GB with offloading. We will add these details in the final version.

Thanks for the response. I think this is a good paper on video diffusion literature. Please revise the manuscript accordingly (e.g., adding memory consumption) if the paper is accepted. I've raised my score.

We thank the reviewer for their valuable feedback. We are encouraged by the overall positive assessment, with reviewers agreeing that our work addresses a "critical and long-standing problem" [Vwja] and is "very well motivated" [2kqq]. The method is "clearly presented" [9RYz], "intuitive" [N5mF], and the paper is “well-written and easy to follow” [2kqq]. Our "quantitative results are solid" [9RYz] and were praised as a "thorough empirical section" [Vwja], demonstrating "broad success across VBench, human preference studies, and efficiency benchmarks" [N5mF].

We will now address the concerns and questions raised to further clarify our contributions.

• Presentation versus TF/DF: We believe comparing Self-Forcing directly to Teacher-Forcing (TF) and Diffusion-Forcing (DF) is useful for understanding our core contribution, as the fundamental axis of comparison is the source of the conditioning context during training. As illustrated in Figure 1, these methods represent three distinct choices for this context:

  • Teacher Forcing: Conditions on clean, ground-truth frames.
  • Diffusion Forcing: Conditions on noisy, ground-truth frames.
  • Self Forcing (Ours): Conditions on the model's own previously generated frames.

  Although TF/DF were originally introduced for pre-training and Self-Forcing is described as a post-training algorithm for practical considerations, we argue that algorithms used at different stages can still form valid comparisons. As an analogy, while today GANs are often used only for post-training/distillation and diffusion models for pre-training, they are still routinely compared as alternative generative modeling paradigms.

  Moreover, the application of TF/DF is not limited to pre-training. Our key baseline, CausVid, applies DF in a post-training phase with a DMD objective. Our ablation studies (Table 2) apply all three paradigms in the same post-training setting. This ensures an apples-to-apples comparison to isolate the impact of the conditioning strategy—the central variable our paper investigates.

• Long video generation: The goal of this paper is not to improve long video generation quality in the extrapolation setting. Our work aims to address exposure bias and to reduce error accumulation for videos generated within the training context length. This is a distinct problem from extrapolation (i.e., generating videos substantially longer than the training context), where we do not claim strong performance, as discussed in our limitations (L318-320). Extrapolation introduces a second distributional mismatch (of video length), in addition to the distribution mismatch that we address (ground-truth vs. generated history). While resolving exposure bias is a necessary step for long video generation, it is not sufficient by itself. To truly enable high-quality long video generation, future work needs to either scale up the video length during training, or investigate orthogonal techniques that enable better extrapolation.

• Visual results not convincing enough: The visual results in Fig. 7 show much better quality than CausVid (less oversaturation). While videos obtained with Wan/SkyReels/MAGI-1 indeed sometimes have visually comparable quality, those methods are 150-400x slower than us in terms of latency. So even same quality is still a big win for us.

• Moving training details: We thank the reviewer’s suggestion and will move the training objective descriptions to the main paper in the final version.

• Training rollout: We generate 21 latent frames (corresponding to 81 video frames) during training, which is aligned with the setting of the base model. The number of rollouts is 21 divided by the chunk size (which equals to 3 or 1 in our experiments). We will add these details in the final version.

Thank you for your continued engagement and insightful follow-up questions. We appreciate the opportunity to provide further clarification.

• You are absolutely correct that fully parallel, non-autoregressive diffusion models (like the base model) do not suffer from exposure bias. However, the central motivation for adopting an autoregressive (AR) framework is to enable applications that are fundamentally impossible with parallel generation, as discussed in our introduction (L18-28). These include:

  • Real-time streaming: Where video frames must be generated and displayed sequentially with minimal latency.
  • Interactive applications: Where user input can alter the course of the video during its generation.

  Once we commit to an AR paradigm to unlock these benefits, exposure bias immediately becomes a critical issue, even when generating videos within the original context length. Previous AR Diffusion models have been trained with TF/DF, which suffer from error accumulation even within training length due to exposure bias. Our Self-Forcing method addresses this fundamental problem for AR models.

• Clarifications about KV caching:

  • KV caching's standard role is to make AR generation efficient, not specifically about extrapolation. This holds true both in our work and in its standard, default usage in LLMs, where it prevents quadratic complexity during AR generation.
  • While there have been prior works that use KV caching for video models (CausVid/MAGI-1), using KV caching during training is indeed one of our main contributions. This is a departure from standard practice where it is an inference-only optimization.
  • We aditionally propose a rolling KV cache algorithm (abstract and figure 3) specifically for the extrapolation scenarios. This algorithm aims to improve the efficiency of extrapolation (orthogonal to quality) and is also orthogonal to our main contribution (addressing exposure bias by performing rollout with KV caching during training, within length limit).

We sincerely appreciate this discussion and will integrate these clarifications into the final manuscript to make these contributions clearer to all readers. We are happy to answer any remaining questions.

We thank the reviewer for their valuable feedback. We are encouraged by the overall positive assessment, with reviewers agreeing that our work addresses a "critical and long-standing problem" [Vwja] and is "very well motivated" [2kqq]. The method is "clearly presented" [9RYz], "intuitive" [N5mF], and the paper is “well-written and easy to follow” [2kqq]. Our "quantitative results are solid" [9RYz] and were praised as a "thorough empirical section" [Vwja], demonstrating "broad success across VBench, human preference studies, and efficiency benchmarks" [N5mF].

We will now address the concerns and questions raised to further clarify our contributions.

• Novelty: We agree with the reviewer and acknowledge in our paper (L49, L70-74) that training with self-rollout was explored in early RNN+GAN-based models, including InfiniteNature. However, as Reviewer N5mF kindly noted, our work is the first to successfully adapt this principle to modern Autoregressive Diffusion Transformers. This adaptation is highly non-trivial, requiring a careful design for combining a few-step diffusion, KV caching, and gradient truncation. Successfully bridging this established concept to the AR-DiT architecture is a core technical contribution of our work, leading to significant gains in generation quality over older paradigms.

• Improvement over CausVid: We want to clarify that Self-Forcing has significantly better quality than CausVid-Wan-1.3B implemented by the original author under the same-speed setting. This advantage is quantitatively supported by VBench scores (Total Score: 84.31 vs. 81.20), user preference study (66.1% preference), the qualitative examples in the appendix (Fig. 7), and the html website in the supplementary material. We respectfully hypothesize that the reviewer's observation that our results “look on par with CausVid” may refer to a different implementation of CausVid built upon a larger closed-source model that runs significantly slower (9.4 FPS for 360P, compared with ours 17FPS for 480P). Our paper provides an apples-to-apples comparison against the public CausVid implementation to fairly evaluate the specific impact of our training paradigm on the same 1.3B architecture.