Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation

We are grateful for your recognition of our novel contributions and promising results. We follow your advice to conduct the following set of experiments.

Weakness 1: Perceptual loss instead of adversarial loss. Following your suggestion, we conducted experiments comparing perceptual loss with our approach, using ModelScopeT2V teacher on the WebVid mini val set. The results are summarized in the table below. Our observations are: The adversarial loss outperforms using 1 and 2 sampling steps, benefiting low-step sampling by producing sharper details. Both losses perform similarly using 4 and 8 steps. We do not observe apparent training instability using adversarial loss, thanks to the well-trained feature extractor DINO v2 and the discriminator gradient penalty loss. We will add more analysis and details in the revised version.

Question 1: Results on image-to-video and human video generation. Following your advice, we conduct the following image-to-video distillation experiment using Stable Video Diffusion (SVD). The quantitative results on MSRVTT are listed below. We make the following observations. We outperform Euler and AnimateLCM on FVD across most sampling steps. Compared to AnimateLCM, which finetunes the entire SVD (1.6B parameters), we achieve better performance by training only 5.9% of parameters using a lightweight LoRA (95M parameters).

We will add these results to our revised manuscript and make our SVD-based MCM checkpoint publicly available.

Due to limited rebuttal time, we demonstrate our method's compatibility with ControlNet for controllable human video generation. Examples are provided in the attached PDF. This integration highlights our MCM’s versatility and potential for broader applications. We will include additional results in the revised version.

Limitation 1: High-resolution video generation evaluation. Following your advice, we conducted high-resolution video generation evaluations at 768x768 resolution, using our AnimateDiff-based MCM on the WebVid mini val set. The results below demonstrate that our MCM achieves state-of-the-art performance in high-resolution video generation. We've included promising examples of these high-resolution outputs in the PDF attached to our general response.

Thank you for recognizing our MCM’s contributions and providing constructive feedback. We respond to your concerns below.

Weakness 1: Motion jittering. Thank you for the feedback! The motion jittering is largely inherited from the teacher model (the same phenomenon was also reported in AnimateDiff-Lightning [35]). To mitigate this, we could use the following methods.

• Increase the hyperparameter $\lambda_{\text{real}}$ in mixed trajectory distillation, so that MCM could learn more information from real videos instead of jittering teacher output.
• Apply additional temporal losses to maintain stable video outputs, such as constraints on the brightness or optical flow changes.

We will add additional discussion in the revised version.

Weakness 2: Fig. 6 caption. Our Fig. 6 shows only our MCM results adapted to different image dataset styles, not two-stage results. The first and second rows represent the first and last video frames, respectively, similar to Fig. 5. We'll clarify this in the revision.

Question 1: Watermark in “Ours w/ WebVid” in Fig. 6. Thank you for noticing the watermark. This is because all WebVid videos contain “shutterstock” watermarks. We use WebVid for a fair comparison with ModelScopeT2V-based video diffusion distillation methods, such as DDIM, DPM++, and LCM. We will elaborate more on this in the revised version.

Thank you for identifying our key strengths and providing insightful comments. We address your concerns point by point.

Weakness 1: Gaps in training and inference distillation inputs. Thank you for raising the concern! There exist two major gaps in the mixed trajectory distillation.

• Distribution mismatch gap. During training, we assume all inputs are noisy low-quality videos; during inference, the input will be noisy high-quality videos.
• Information leakage gap. During training, the original consistency distillation takes as input noisy videos that contain ground-truth video information; during inference, the sampling will start from pure noise, containing zero signal.

Our mixed trajectory distillation simultaneously addresses these two problems. We will make this clear in the revised version.

Weakness 2: Related work in decoupled appearance and motion. Thank you for the suggestions! While disentangle appearance and motion is well-established in video understanding, our work addresses unique challenges in video diffusion distillation.

• Video diffusion models suffer from long sampling time due to the additional temporal dimension.
• Popular open-source video datasets contain low-quality frames, such as watermarks, motion blur, and low resolution.
• Our MCM simultaneously achieves video diffusion acceleration and frame quality improvement by leveraging additional high-quality image data.
• We introduce disentangled motion distillation and mixed trajectory distillation to overcome specific challenges in video diffusion models.

We will expand our related work section to include additional references and contextualize our contributions in video diffusion distillation.

Weakness 3: Analysis on learnable motion representation. Thank you for the advice! Our Table 5 shows that most representations, except latent low-frequency components, reduce FVD and improve CLIPSIM at low sampling steps. This indicates that disentangling motion from the raw latent space for consistency learning enhances video quality. The learnable motion representation performs best, outperforming handcrafted ones. We attribute this to its ability to:

• Adaptively learn optimal motion features, capturing complex motion patterns that generic handcrafted representations might miss.
• Better separate motion from appearance, reducing conflicts in learning high-quality frames.

This learned disentanglement allows for more effective motion consistency modeling while preserving frame appearance quality. We will elaborate more on this in the final version.

The authors have addressed most of my concerns. However, the response to Weakness 2 is still very sketchy. I suggest the authors provide more detailed discussions in the revised paper. I decide to keep my initial positive rating.