SF-V: Single Forward Video Generation Model

We thank the reviewer for the constructive comments. The detailed responses regarding each concern are listed below.

Q1. Reproducing the results.

A1. We plan to make our code and checkpoint public for the reproducing of our work.

Q2. Training data.

A2. Our dataset contains around 1M video clips of various lengths. As we mentioned in the paper (Line 191-192), our model is trained for 50K iterations with a total batch size of 32. For each training sample, we randomly select 14 consecutive frames for a video temporally downsampled to 7 FPS. We agree our dataset is far less the one used for SVD training. However, in this work, we fine-tune SVD for efficient generation instead of learning motion from scratch, which makes our model requires less data. We humbly believe it is an advantage of our approach that we can require much less data than SVD while being able to fine-tune the model to single-step.

On the other hand, we do expect our model can benefit from more high-quality data training and longer training iterations. To validate the assumption, we run the following experiment.

• First, we randomly selected 700K videos out of the 1M dataset and use that to train our model. Our method achieves FVD of $275.7$ with 700K videos while $180.9$ using 1M videos. We also provide the qualitative results in Fig. C of the attached one-page PDF.

• Second, we report the performance of the model under various training iterations. Due to the limit of computation, we report results for training the model within 50K iterations. In the following table, we show how the FVD score changes along the training process.

| Iters (K) | 20 | 30 | 40 | 50 | | --- | --- | --- | --- | --- | | FVD | 542.2 | 479.5 | 217.1 | 180.9 |

Q3. Discriminator initialization.

A3. We use the encoder of the UNet to initialize the backbone for its strong video understanding ability and we fix the weights of the backbone during training. Such design can further facilitate faster training and fewer memory consumption.

To further demonstrate the effectiveness, we train the discriminator backbone together with the added heads without initializing it from the pre-trained UNet encoder. We observe a significant drop of performance, i.e., the FVD on UCF-101 is 1490.2 (without initialization from UNet) vs. 180.9 (with initialization of UNet.). We also provide qualitative results in Fig. C of the attached one-page-PDF.

Q4. Distillation loss.

A4. There are two main reasons for not using the teacher-student distillation loss in ADD [26].

• First, to obtain distillation target in ADD [26], during training, the denoising process for the teacher model is required. Thus, the teacher model needs to run multiple inference steps. Such a training process can drastically increase the training time.

• Second, distillation loss in ADD [26] needs to be calculated in the pixel space instead of latent space. Decoding the latents and calculating the gradients with the video VAE decoder will introduce significant computation overhead for video models, leading to out-of-memory issue for SVD. Thus, we use the reconstruction objective instead of the distillation loss in our approach to make the model training feasible.

Q5. Generalization ability.

A5. As shown in the Supplementary Materials, we qualitatively evaluate our method on images of various styles and objects, demonstrating the strong generalization ability of our method for video generation.

As for the quantitative evaluation metric, we chose FVD on UCF-101 as it is the most adopted benchmark for video generation. For example, it is adopted in SVD[14] and AnimateLCM [21]. We fully agree with the reviewer that more benchmark datasets are helpful for video model evaluation and it is an exciting direction for future work.

We thank the reviewer for the constructive comments. We provide detailed response in the following.

Q1. About novelty.

A1. We agree with the reviewer that certain design components, e.g., adversarial training and constructing the discriminator initialized from the diffusion model, have been explored in different manners in other tasks. However, we would also like to kindly emphasize that distilling a video diffusion model into single-step with the similar performance as 25-steps has not been studied in the literature.

In the following, we would like to kindly clarify that the novelty comes from the design of the whole training pipeline that is tailored for solving a specific challenging task, instead of the individual component like adversarial training.

Our framework is inspired by LADD [27], which distills an image diffusion model into fewer steps. Here, we highlight major differences compared with LADD.

• First, we extend the image latent adversarial distillation from image to the video domain by incorporating spatial and temporal heads to achieve one-step generation for video diffusion models. The training computation to distill a video model is much larger than distilling an image model. Therefore, we carefully design our discriminator, which is not used in the image domain, to enable the stable training. Notably, our discriminator only uses certain part of the UNet that we find important. Such design is different from exiting arts that re-use the whole UNet. In this way, we can significantly reduce the training computation for adversarial training, and train the video model much faster and consume much less GPU memory.

• Second, based on the EDM-framework, we observe that sampling $t^\prime$ using a discretized lognormal distribution provides more stable adversarial training compared to the logit-normal distribution used in LADD.

• Third, unlike LADD, we utilize real video data instead of synthetic data for training, removing the storage and computation burden for calculating and saving the synthetic data.

• Fourth, we employ reconstruction objective (Eq. 9 in the main paper) along with R1 gradient penalty (Eq. 11 in the main paper) to stablize the training of the video diffusion model.

In we only focus on how we design the discriminator (e.g., re-using some weights from the generator and fixing them), indeed, they do not look fancy and seem like lacking novelty. However, we humbly think it is important to look at how such design, when smoothly combined with techniques (like how to sampling the noise), can solve an important task.

We hope such an explanation would make sense to the reviewer.

Q2. Comparison with the approaches that built for distilling image diffusion models [24, 25, 26, 27, 28].

A2. Directly taking existing step-distillation methods for distilling text-to-image models to video generation is not sufficient to achieve reasonable results. As suggested by the reviewer, we compare our approach with the image-based distillation methods.

• First, DMD [25] and ADD [26] are too computationally costly to implement in video generation tasks. Both of these works require loss calculation in the pixel space instead of latent space. Forwarding and backwarding through the VAE decoder while training will introduce a huge GPU memory overhead and slow down the whole training process.

• Second, since LADD [27] outperforms SDXL-Lightning [28] by significant margin [27], we try our best to implement LADD [27]. We also implement UFOGen [24]. We replace the generator in LADD and UFOGen with SVD. Please kindly notice that simply re-using the discriminator from LADD and UFOGen leads to the out-of-memory issue, since the computation in the video model is much larger than the image model. Here we replace the discriminator from LADD and UFOGen to the one proposed in our work.

We report the FVD score on UCF-101 in the following table. Additionally, we provide qualitatively comparisons between our method and LADD and UFOGen in the Fig. C of the attached one-page-PDF.

| Method | UFOGen | LADD | Ours | | --- | --- | --- | --- | | FVD | 1917.2 | 1893.8 | 180.9 |

From these results, we can see that, for distilling video diffusion models, our approach performs much better than exiting step-distillation methods built upon image-based-diffusion models.

Q3. Importance of the proposed training regime

A3. We are not sure whether we fully understand this concern, yet, please allow us to try to answer them in three different perspectives.

• First, as the experiments shown in A2, our adversarial training regime is very important, as it achieves better results than the ones proposed in LADD [27] and UFOGen [24], which are two methods for distilling image-based-diffusion models.

• Second, the reviewers suggest apply our approach for two work: SVD[14] and AnimateLCM [21]. Actually, our video model is fine-tuned from SVD [14]. In Tab.1 of the main paper, we show single-step model achieves similar results as 25-steps SVD, and better results than 8-steps AnimateLCM [21].

• Third, directly applying our training pipeline to the image-based-diffusion model is non-trivial. For example, It requires the changing of our discriminator, the noise sampling process, and the changes of adversarial losses. We humbly think such experiments are beyond the scope of this work.

Q4. Discussion about limitations

A4. Thank you for the suggestion. We observe that when the given conditioning image indicates complex motion, e.g. running, our model tends to generate unsatisfactory results, e.g. blurry frames, as shown in the Fig. D of the attached one-page PDF. Such artifacts are introduced by the original SVD model, as can be observed in Fig. D of the attached one-page PDF. We believe a better text-to-video model can solve such issue and are interested in trying our step-distillation approach to such a model when there is an open-sourced one.

Dear Reviewer FT7s,

Thank you for reading our rebuttal and providing response!

We would like to apologize that our previous response in Q1 regarding the comparison between our approach and LADD does not contain the detailed explanation regarding the experiments that we have done. In the following, we connect our previous experiments along with the difference between our work and LADD.

First, in Tab. 2 of the main paper, we conduct ablation experiments with different discriminator heads settings. We show that, using spatial and temporal heads achieves better generation quality than only using spatial head (which is the design from LADD). Additionally, our discriminator only uses certain part of the UNet that we find important, while LADD does not. In this way, we can significantly reduce the training computation for adversarial training, and train the video model much faster and consume much less GPU memory. By directly using the discriminator design from LADD, we notice the out of GPU memory issue on 80G Nvidia A100 GPU.

Second, based on the EDM-framework, we observe that sampling $t^\prime$ using a discretized lognormal distribution provides more stable adversarial training. In Fig. 6 and Fig. 7 of the main paper, we show the importance of our proposed noise sampling schedule for achieving the better performance.

Third, we utilize real video data instead of synthetic data for training, removing the computation burden for calculating and saving the synthetic data. For instance, generating 1M videos using SVD requires approximately 3K GPU hours.

Fourth, we employ reconstruction objective (Eq. 9 in the main paper) along with R1 gradient penalty (Eq. 11 in the main paper) to stablize the training of the video diffusion model. Without R1 gradient or the reconstruction objective, we observe frequent training divergence.

Besides, as mentioned in our answer for Q2, we use the proposed training regime from LADD [27] combined with the discriminator proposed in our work and conduct quantitative and qualitative comparison experiments. Our method significantly outperforms LADD [27] training regime in the video distillation task, demonstrating the effectiveness of our proposed training regime.

Thanks again for the feedback from the reviewer. We will add the above discussion to the revised paper and hope it can be helpful to understand the difference between our work and LADD. We would deeply appreciate it if the reviewer could reconsider the score and we are always willing to address any of your further concerns.

Best,

Authors

We thank the reviewer for the constructive comments. The detailed responses regarding each concern are listed below.

Q1. About improving the writing for the preliminary section.

A1. Thanks for the suggestion! We will revise the writing of the manuscript.

Q2. About the approach is simple.

A2. We agree with the reviewer that our approach, built upon adversarial training, is relative straightforward and simple to re-implement. On the other hand, as high-lighted by the reviewer, our approach actually works, which we believe further demonstrate the value of this paper - simple yet effective. In fact, our approach also outperforms other distillation based methods like latent consistency model.

Additionally, this is the first paper showing that a single forward video diffusion model can obtain similar quality as a 25-step model. Such results and findings could inspire later research and efforts to develop more advanced approaches to further improve the generation quality.

Q3. Inference time for image encoding and encoding.

A3. We calculate sampling time for each component when generating $14$-frames videos at a resolution of $1024\times 576$. We use A100 GPU as the testbed to get the latency.

• SVD use both CLIP image encoder and VAE encoder to encode the first frame of the video as the condition input:
  • The runtime for CLIP image encoding is 0.03 s.
  • The runtime for VAE image encoding is 0.01 s.
• The latency for forwarding the UNet once is 0.30 s.
• The latency for VAE decoding the latent to get videos is 2.32 s.

Q4. Training instabilities of GANs.

A4. We agree with the reviewer that applying GAN during training is non-trivial. To avoid the mode collapse and achieve the stabilized training, we carefully design our training pipeline. For instance, we use hinge loss as the adversarial training. Additionally, we employ reconstruction objective (Eq. 9 in the main paper) and R1 gradient penalty (Eq. 11 in the main paper) during training to make the training process more stable. Moreover, based on the EDM-framework, we observe that sampling $t^\prime$ using a discretized lognormal distribution provides more stable adversarial training.

We have also tried to re-train our model multiple times using the same dataset and approach. The training is stable across different runs and we can re-produce very similar results.

We thank the reviewer for the constructive comments. The detailed responses regarding each concern are listed below.

Q1. About generating real world videos.

A1. Thanks for the suggestions. In this work, we fine-tune SVD, which is an image-to-video model, into single sampling step. Our model can take arbitrary given image as input and generate the corresponded video.

In the main paper (Line 221-222 and Fig. 5), we show the results of using the real images as input to generate the real world videos. The saved videos can be found in our Supplementary Materials. Additionally, we compare our generation results with other approaches.

We also provide more examples of using real images as input to generate real world videos. The examples are shown in the Fig. A in the attached one-page PDF.

Q2. About some videos with laminar motion.

A2. Thank you for noticing the laminar motion for some videos. We would like to kindly mention that this paper aims to improve the sampling efficiency of the pre-trained SVD model, instead of improving the motion quality of SVD. In fact, we have shown in the main paper (Fig. 5) and the Comparisons section in the webpage of the Supplementary Material, the motion quality of our approach is similar to the motion of 25-steps of the original SVD model.

In fact, we use training videos with large motion to fine-tune SVD. Some examples of training videos are shown in the Fig. B of the attached one-page PDF. Nevertheless, improving the motion of SVD without modify its whole architecture (e.g., image-based encoder), is still challenging, which also beyonds the scope of this paper. We further provide more examples in Fig. C of the attached one-page PDF, demonstrating that for the same conditioning image, running SVD with 25 steps can also generate videos with laminar motion.

One reason of that SVD generates videos with laminar motion is due to the SVD can only synthesize short clips. We believe that in the future, with more open sourced video generation models synthesizing minutes-long videos, the motion can be significantly improved. By that time, we would be very interested in applying our approach to those models.

Q3. About the training challenges of using GAN, such as mode collapse and instability.

A3. We agree with the reviewer that applying GAN during training is non-trivial. To avoid the mode collapse and achieve the stabilized training, we carefully design our training pipeline. For instance, we use hinge loss during adversarial training. Additionally, we employ reconstruction objective (Eq. 9 in the main paper) and R1 gradient penalty (Eq. 11 in the main paper) during training to make the training process more stable. Moreover, based on the EDM-framework, we observe that sampling $t^\prime$ using a discretized lognormal distribution provides more stable adversarial training.

We have also tried to re-train our model multiple times using the same dataset and approach. The training is stable across different runs and we can re-produce very similar results.

Q4. standard video datasets like UCF101 used in evaluation

A4. We agree with the reviewer that UCF-101 is a standard dataset that should be used in the evaluation.

In the paper, we mainly use UCF-101 to obtain the quantitative results. For example, in Line 211 of main paper, we explain the details of how we use UCF-101. In Tab.1, Tab.2, and Tab.3, we show the calculated FVD on UCF-101 to compare our approach with others and ablating the design principles of our approach.