ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler

W1. Notations: Lines 247 - 256 (Section 3.2, last frame guidance with DDS) was kind of confusing at first, with many similar looking notations without predefinitions, and mixed use of notations as functions and variables.

A. In response to the reviewer’s request, we modified the mixed use of notations as functions and variables. We further revised confusing definitions. Please refer to the revised Section 3.

W2, Q1. I am not fully convinced if the proposed bidirectional sampling itself is a powerful method compared to existing works, and suspect the superior performance could have come from the advanced manifold guidance methods, which, is hard to say to be a contribution. Could the authors provide experiments on this, to ablate and show the importance of bidirectional sampling?

A. Thank you for your constructive suggestions that highlight our contributions. First, we would like to inform the reviewer that we have already demonstrated the superior performance of our bidirectional sampling method, referred to as “Ours (Vanilla),” compared to time reversal fusion (TRF) across all experiments and metrics. Unlike Generative Inbetweening, which requires fine-tuning of the diffusion model, our method operates entirely in a training-free manner.

Moreover, we have conducted additional ablation studies and included further visualizations to demonstrate the effect of each building block. Please refer to the additional project page (https://vibid.github.io/supple/) and the Appendix B.2.

As illustrated in the first section of our anonymous webpage, the vanilla TRF model exhibits unnatural motion in videos featuring a coffee machine and a child kicking a ball. In contrast, our vanilla model produces more natural motion and visually vivid frames compared to the TRF vanilla model. We also agree that when advanced guidance techniques, such as CFG++ and DDS, are applied to TRF and our method, video quality of both methods improves. Nonetheless, the videos generated by the TRF model with these guidance techniques remain inferior to our model. These qualitative observations are consistent with the quantitative analysis presented in the Table 5.

Specifically, as shown in the second section of our anonymous project page, the goldfish positioned on the far left in the vanilla TRF model disappears in the intermediate frames. In contrast, the fish exhibits natural movement in the videos generated by our vanilla model. Additionally, while the camel's shape appears blurred and its movement lacks smoothness in videos produced by the vanilla TRF model, our vanilla model demonstrates improved clarity in the camel's form. Furthermore, when advanced guidance techniques are applied, the videos display enhanced quality in the camel's leg shapes and movements.

Q2. There are three steps, (1) denoising (w/ c_start), (2) renoising, (3) denoising (w/ c_end). I am specifically curious of this process in that the (2) renoising step could possibly remove the information from (1) denoising, and the conditioning could have possibly be injected only at (3), making the steps (1) and (2) redundant. A further analysis / clarification on this part, that the proposed method surely takes both start / end frame conditioning.

A. The renoising step is not redundant but rather a critical component of bidirectional sampling. During the renoising step, a small amount of additional stochasticity is introduced (Eq. 8), which plays a key role in preventing mode collapse problem by facilitating the effective integration of information from both the forward and backward paths.

For easy understanding, we additionally conducted a simple comparison using only the (3) denoising step (w/ $c_{end}$) as suggested by Reviewer. As shown in the Table, our method outperforms the approach using only the (3) denoising step (w/ $c_{end}$). This ablation study confirms that our slight renoising step retains the information from the (1) denoising step.

W1, Q1. The improvement of contribution from bidirectional sampling seems not obvious with using SOTA sampling approaches including DDS and CFG++ from Table. 1. The author should further verify the improvement of each building blocks used in the paper, i.e., bidirectional sampling, DDS and CFG++, e.g., providing more visualization as well as adding DDS and CFG++ to other baselines and show the improvement.

A. Thank you for the constructive suggestion that highlights our contributions. Firstly, reviewer is kindly reminded that, as shown in the Table 1, we have already demonstrated the superior performance of our bidirectional sampling, referred to as “Ours (Vanilla)”, compared to the time reversal fusion (TRF) across all experiments and metrics. Notably, Generative Inbetweening conducts additional fine-tuning on diffusion model to improve the performance, whereas our method is conducted efficiently in training-free manner.

Moreover, we have conducted additional ablation studies and included further visualizations to demonstrate the effect of each building block. Please refer to the additional project page (https://vibid.github.io/supple/) and the Appendix B.2.

As illustrated in the first section of our anonymous webpage, the vanilla TRF model exhibits unnatural motion in videos featuring a coffee machine and a child kicking a ball. In contrast, our vanilla model produces more natural motion and visually vivid frames compared to the TRF vanilla model. We also agree that when advanced guidance techniques, such as CFG++ and DDS, are applied to TRF and our method, video quality of both methods improves. Nonetheless, the videos generated by the TRF model with these guidance techniques remain inferior to our model. These qualitative observations are consistent with the quantitative analysis presented in the Table 5.

Specifically, as shown in the second section of our anonymous project page, the goldfish positioned on the far left in the vanilla TRF model disappears in the intermediate frames. In contrast, the fish exhibits natural movement in the videos generated by our vanilla model. Additionally, while the camel's shape appears blurred and its movement lacks smoothness in videos produced by the vanilla TRF model, our vanilla model demonstrates improved clarity in the camel's form. Furthermore, when advanced guidance techniques are applied, the videos display enhanced quality in the camel's leg shapes and movements.

Q2-1. The practical use the proposed approach seems not convincing. The proposed approach still requires 192s to generate 25 frames, which is at least two times longer than directly inferencing a video generation model with similar computational cost.

A. The reviewer is kindly reminded that the video interpolation problem is a bit different from video generation. Video generation is more focused on “fake” video generation based on conditions, whereas video interpolation technique could be applied to interpolate between “real” video frames. Therefore, reviewer is kindly reminded that in this real-world scenario applying video generation model instead of interpolation is not a viable option.

Moreover, we emphasize that our method's inference time, which is only 2× that of direct video generation, is significantly more efficient compared to other VDM-based methods (e.g., TRF with 4.5× and Generative Inbetweening with 12.5× inference time of direct video generation).

Q2-2. How to keep the consistency between frames when the required length of the output video is beyond 25?

A. Thanks for the constructive comments that highlight our advantages. Note that recursive sampling can be used to ensure consistency when the desired output video length exceeds the trained configuration. Specifically, once 25 frames are generated at the first stage, we can further improve the temporal resolution between frames by recursively applying our sampler.

If reviewer’s intension is to extend beyond the two end frames, we would like to kindly remind the reviewer that the suggested problem is not video interpolation; rather, it is a video extrapolation problem which is beyond the scope of this paper and other existing video interpolation methods.

W1, Q1. The method is training-free. In general, training-based method can be more effective

A. We agree that DynamiCrafter and Generative Inbetweening are training-based and fine-tuning-based approaches, respectively. Nonetheless, our method achieves superior video interpolation performance despite its training-free manner, as demonstrated in Table 1. More specifically, due to the efficiency of our training-free approach, our method significantly outperforms other SVD-based methods in inference time. Specifically, compared to Generative Inbetweening, our method achieves approximately 6× faster inference, as shown in Table 2.

W2, Q2. Ablation study is suggested to assess the effect of pre-trained video generation models. Will better video generation models result in better performance?

A. In response to the reviewer’s suggestion, we conducted simple ablation study to assess the effect of VDMs. In this study, we employed SVD-XT, a fine-tuned version of the original SVD. To evaluate the impact of video diffusion models (VDMs), we conducted a comparative analysis between SVD-XT and SVD. Please refer to the revised Appendix B.4. As shown in the table, we see the performance increment with better video generation models.

W3, Q3. Discussion on the limitation and future work.

A. Per the reviewer’s suggestion, we have outlined the limitations and future works in the revised Appendix B.5. As discussed in response to Q1 from Reviewer VMQD, we have not considered the incorporation of text conditioning, as our method is based on SVD. However, when bidirectional sampling is applied to other I2V diffusion models, such as DynamiCrafter, we believe that it becomes feasible to guide actions based on text conditions. Extending our method to control additional text condition is a promising direction, and we are actively considering this for future work.

W1. The experimental results presented lack PSNR scores.

A. In response to the reviewer's request, we additionally provide PSNR scores of our method and baselines. Please refer to the revised Appendix B.1. As shown in the table, our model demonstrates superior performance in both PSNR and SSIM across the DAVIS and Pexels datasets.

W2. I would appreciate a more comprehensive video comparison in the ablation study, in addition to the provided static comparison on a single scene.

A. Per the reviewer’s suggestion, we included more comprehensive video comparison of the ablation study in our project page. Please refer to the additional project page for more comprehensive visualization (https://vibid.github.io/supple), and Appendix B.2 for evaluation.

Additionally, more comprehensive video comparisons are available in the second section of our anonymous webpage. The goldfish positioned on the far left in the vanilla TRF model disappears in the intermediate frames. In contrast, the fish exhibits natural movement in the videos generated by our vanilla model. Additionally, while the camel's shape appears blurred and its movement lacks smoothness in videos produced by the vanilla TRF model, our vanilla model demonstrates improved clarity in the camel's form. Furthermore, when advanced guidance techniques are applied, the videos display enhanced quality in the camel's leg shapes and movements.

Q1. How to incorporate text conditioning into your framework so users can control the action in the generated frames?

A. Currently, our method employs Stable Video Diffusion (SVD), which replaces text embeddings with the CLIP image embedding of the conditioning image. Thereby, the incorporation of text conditioning is limited to control video content through text conditioning. However, when bidirectional sampling is applied to other I2V diffusion models that originally utilize text conditioning, such as DynamiCrafter, it becomes feasible to guide actions based on text conditions. Extending our method to control additional text condition is a promising direction, and we are actively considering this for future work. We have detailed the future work in the revised Appendix B.5.

Thank you again for the encouraging comments. In response to the reviewer's suggestion, we have included the PSNR and SSIM scores for FILM in the revised Appendix B.1.

As shown in the table, FILM achieves the highest PSNR and SSIM scores compared to all video generative models. This is because FILM is trained using PSNR-oriented losses, such as the L1 loss computed between the ground truth and output frames, which prioritizes pixel-wise reconstruction accuracy. In contrast, our method is specifically designed to generate perceptually natural and smooth motion between consecutive frames, rather than focusing on precise pixel-wise accuracy.

Despite the FLIM's numerical advantages of PSNR and SSIM, it is evident that FILM fails to predict dynamic motion between two bounding frames, as demonstrated in the videos available on our webpage (https://vibid.github.io/). For example, in the cases of a camel and a dog, FILM distorts the appearance of these subjects. This issue arises due to its reliance on PSNR-oriented losses and a network architecture based on an optical flow estimation module, which may compromise the perceptual quality. This is also confirmed in LIPS and FID score in the following table. To clarify these distinctions, we have revised the paper to ensure readers can better understand the differences.

Could you provide the scores of FILM? If the video generative models are not competitive with traditional frame interpolation methods, it's totally understandable. You may just have to make it clear in your paper that your method is not designed for accurate reconstruction. If you can modify your paper accordingly to make readers easily understand the difference, I am more than happy to increase my rating to encourage this kind of generative works with amazing visual results.