IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis

7. Potential Limitations: Are there any specific scenarios or types of video content where the IV-Mixed Sampler might underperform? If so, how might these limitations be addressed?

Thank you for your interesting question. IV-mixed Sampler is effective only if the quality of the images generated by IDM is higher than that of the video frames generated by VDM. Consequently, the algorithm may underperform if the VDM itself outperforms the IDM. For example, in the case of Mochi-1-preview, if IDM SD V1.5 is not strong enough, it can simply be replaced with a stronger IDM, such as SD XL.

[1] https://github.com/Vchitect/VBench.

1. Vbench is needed for a comprehensive evaluation.

For a comprehensive evaluation of both visual quality and semantic consistency, we further assessed the performance of IV-mixed Sampler on VBench [1], with the results presented in the above table. From the above table, it is clear that IV-mixed Sampler outperforms vanilla sampling across most metrics, particularly on the multiple objects, where IV-mixed Sampler improved the performance of Animatediff from 36.88% to 58.46%. Additionally, the average scores of IV-mixed Sampler exceeded vanilla sampling by a margin of 6.5 points on Animatediff, fully demonstrating the effectiveness of IV-mixed Sampler.

2. Need to provide more compelling examples to showcase improvements

Thank you for pointing this out. We provide additional compelling examples of IV-mixed Sampler on different models on the anonymous website: page.

3. Figure 3's illustrations are not clear.

Thank you for your useful suggestion. We have redrawn Figure 3 (see the revised version) to use a more representative case, highlighting the low temporal coherence of IDM alone and the low quality of VDM alone.

4. The method's compatibility with state-of-the-art video generation models is limited. This restricts its potential applications and prospects in the field.

Thank you for your comment. Our proposed IV-mixed Sampler is compatible with the SOTA VDMs. In particular, IV-mixed Sampler, which relaxes the dependency on VAE, shows significant performance gains on the state-of-the-art VDM (12b parameters) Mochi-1 (see Appendix H in our revised version and the anonymous website page). For example, for the prompt "A serene forest clearing at dawn, where deer graze peacefully while golden rays of sunlight pierce through the mist-laden trees", IV-mixed Sampler not only improves visual quality but also makes the deer in the video walk very naturally, which is really amazing!

Therefore, we believe that IV-mixed Sampler, now capable of improving video quality regardless of whether the IDM and VDM use the same or different VAEs, holds significant potential and promise in this field.

5. What are the detailed reasons that we can't use different VAEs for IDM and VDM when adapting the proposed method? Do you have experiment results about this?

This is an interesting question. In this revised version, we find it is possible to relax the dependency on VAE. We can simply edit the video frames by transferring $x_t$ to $x_0$ in a small number of steps of DDIM and DDIM-Inversion. Details can be found in Appendix H.

Furthermore, for the IV-mixed Sampler presented in the original version, if the latent spaces of the VAEs in IDM and VDM are different, IV-mixed Sampler will not function properly. We provide experimental results demonstrating this by applying SD XL as the IDM to Animatediff (SD V1.5, Motion Adapter V3), as shown in Fig. 10 of the revised version. As illustrated in Fig. 10 of the revised version, the resulting video frames are entirely black and cannot be recognized by the naked eye.

6. How does the IV-Mixed Sampler perform as the length and complexity of the video sequences increase? Are there any scalability issues that the authors have identified?

IV-mixed Sampler continues to perform well. Specifically, our proposed IV-mixed Sampler shows significant performance gains on Mochi-1 for long video sequences (84 frames, fps=30). Note that the number of frames in Animatediff and Modelscope-T2V is 16. For further details, please refer to Appendix H in our revised version and the anonymous website page.

We present the framework of IV-mixed Sampler on Mochi-1 in Figure 11 of the revised version, focusing on converting the video latent $x_t$ to $x_0$ using one-step sampling, enhancing it with the ControlNet Tile (SD XL), and adding a certain amount of random noise before passing it through Euler's inversion (similar to DDIM-Inversion). This approach allows the video latent to be converted back to $x_t$, thereby improving video fidelity through IDM.

1. Why do the paper keep mentioning OpenAI's strawberry in Abstract and Intro, which is closed-sourced and no paper or technical detais was released? How is this important or even related to motivating the paper?

Thank you for this question. We mention OpenAI's Strawberry because it exemplifies the inference scaling laws in large language models (LLMs). This motivates us to focus on improving the inference paradigm for video diffusion models, which could lead to significant performance gains. In our revised version, we would like to follow your suggestion to remove the mention of OpenAI's Strawberry from the abstraction and do not keep mentioning it. Instead, we focus on IDM's ability to enhance the visual quality of VDM and emphasize the inference-heavy aspects in the revised version.

2. Figure 2 is confusing and should not be put in the intro. There is no detailed explanation of “R-”, “I-” and “V-”.

We apologize for any distress caused by the figure. It has been moved to the ablation experiments section in the revised version. The meanings of “R-”, “I-” and “V-” are as follows:

• "R-": Forward noise addition using random noise addition.
• "I-": Forward noise addition using DDIM Inversion with IDM.
• "V-": Forward noise addition using DDIM Inversion with VDM.

We have added those explanation in the revised version.

3. The use of "go", "back", "begin", "end" etc in the equations are also confusing.

We apologize for any confusion caused by these definitions. Note that "go" and "back" denote the DDIM and DDIM-Inversion, respectively. We have renamed "begin" and "end" to t = 0 and t = 1 to more clearly indicate their meaning in the revised version. Specifically:

• $\omega_{go}^{t=0}$represents the value of $\omega_{go}$ at t = 0 (when DDIM is performed).
• $\omega_{back}^{t=0}$ represents the value of $\omega_{back}$ at t = 0 (when DDIM-Inversion is performed).
• $\omega_{go}^{t=1}$ represents the value of $\omega_{go}$ at t = 1 (when DDIM is performed).
• $\omega_{back}^{t=1}$ represents the value of $\omega_{back}$ at t = 1 (when DDIM-Inversion is performed).

4. Plots like Fig 6 are also hard to read.

Thank you for pointing this out. In the revised version, we have replaced Fig. 6 with three histograms and simplified some cases. Fig. 6 now more effectively illustrates the impact of IDM dynamic CFG scale, VDM dynamic CFG scale, and $\rho$ on the quality of the final composite video.

5. The results of the model is decent but not very strong. For example, FreeInit is significantly better than ours for UMT-FVD on ModelScope-T2V

IV-mixed Sampler outperforms both vanilla sampling and other comparison algorithms across nearly all models and metrics, indicating strong performance, and IV-mixed Sampler outperforms FreeInit on Modelscope-T2V using a stronger IDM. The performance of IV-mixed Sampler largely depends on the generative capacity of the IDM and the VDM. In the previous version of this study, we used SD V1.5 as the IDM on Modelscope-T2V, but now, we choose RealVis-V6.0-B1 [1], which is less sensitive to resolution, as the IDM. We then adjust the CFG scales $\omega_\textrm{go}$ of the IDM and VDM to 2 ($\omega_\textrm{back}=-2$) and 3 ($\omega_\textrm{back}=-3$), respectively, and present the results in the above table. From these results, we observe that IV-mixed Sampler using RealVis-V6.0-B1 as the IDM outperforms FreeInit across all metrics, which highlights the potential of our algorithm.

[1] https://huggingface.co/SG161222/Realistic_Vision_V6.0_B1_noVAE.

5. Does this method still work in scenarios using distilled models? E.g. 1 step or 2 step generation.

Yes, IV-mixed Sampler still work using distilled models. To verify the effectiveness of IV-mixed Sampler on the distillation-based accelerated sampling model, we apply IV-mixed Sampler to the motion consistency model (MCM) [2]. We use the Animatediff-laion version from the official MCM library and set the number of sampling steps to 4. After extensive empirical experimentation, we found that the sampling mechanism shown in Fig. 9 of the main paper can enhance MCM performance by introducing IDM. Specifically, we used three checkpoints: the VDM w/o distillation, the VDM w/ distillation, and the standard IDM. During each sampling step with the VDM w/ distillation, IV-mixed sampling is performed using the VDM w/o distillation and the standard IDM. It is important to note that if the VDM w/ distillation is used for IV-mixed sampling (i.e., replacing the VDM w/o distillation with the VDM w/ distillation), it will not work due to its LCMScheduler at each step, which is non-deterministic and would disrupt the semantic information provided by the IDM.

We present the results in the above table, and we can find that IV-mixed Sampler significantly improves the performance of MCM in all metrics.

[1] I4VGen:ImageasStepping StoneforText-to-Video Generation, Arxiv 2024.

[2] Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation, NeurIPS 2024.

[3] https://huggingface.co/OzzyGT/SDXL_Controlnet_Tile_Realistic.

1. How would this method also be used in cases where there is temporal downsampling in the video latent space

Thank you for your interesting question. In our revised version, we relaxed the dependency of IV-mixed Sampler on VAE, allowing it to operate on different VAEs for IDMs and VDMs (see the performance on Mochi-1 in the anonymous website page.

Furthermore, we can apply IV-mixed sampler by treating the VDM itself as an IDM. To improve the visual quality of the synthesized samples from the IDM (i.e., the VDM), high-quality images (with data of $b \times c \times 1 \times h \times w$) can be used by the DPO to fine-tune the VDM.

2. The method's relevance may decline in 6+ months as public video datasets (e.g., OpenVid10M) and video generation models (e.g., Mochi-1) improve.

The potential of IV-mixed Sampler will remain strong beyond 6+ months. The concerns you raise are very practical. However, even the 12b open-source large-scale VDM Mochi-1 still does not match the visual quality (the video suffers from severe distortion) of middle-scale IDM (such as SD XL). Our proposed IV-mixed Sampler, which we have slightly modified to eliminate dependency on VAE, shows significant performance gains on Mochi-1 (see Appendix H in our revised version and the anonymous website page. For example, for the prompt "A serene forest clearing at dawn, where deer graze peacefully while golden rays of sunlight pierce through the mist-laden trees", IV-mixed Sampler not only improves visual quality but also makes the deer in the video walk very naturally, which is really amazing!

The changes made to Mochi-1 are shown in Figure 11 of the revised version, focusing on converting the video latent $x_t$ to $x_0$ using one-step sampling, enhancing it with the ControlNet tile (SD XL) [3], and adding a certain amount of random noise before passing it through Euler's inversion (similar to DDIM-Inversion). This approach allows the video latent to be converted back to $x_t$, thereby improving video fidelity through IDM.

Therefore, we believe that IV-mixed Sampler, which is now capable of improving video quality whether the IDM and VDM use the same VAE or different VAEs, still holds great potential for future development.

3. What is the variance of the quantitative metrics (e.g. FVD in Tables 1 and 2)?

Thank you for your meaningful suggestion. We have included the standard deviations for Tables 1, 2, 3, and 4 in the revised version. Specifically, we sampled and evaluated each configuration three times (set seeds 41, 42, and 43), and found that the standard deviations of all results are on the order of two decimal places. This suggests that the improvement from the IV-mixed sampler is significent.

4. It would be nice to have more video samples to look at (e.g. on an anonymous website).

Thank you for your valuable advice. We have presented the results of IV-mixed Sampler on different models on the anonymous website: page.