FrameBridge: Improving Image-to-Video Generation with Bridge Models

There are only about 2 days left for the review discussion. I wonder whether the authors still want to submit their rebuttal.

We appreciate the reviewer for recognizing the experimental results in our work and the instructive comments. We are glad to engage in a further discussion and address the reviewer's concerns.

Reply to W1: Differences between FrameBridge and diffusion-based I2V models

We thank the reviewer for carefully reviewing our method and providing detailed feedback. We will clarify the differences between FrameBridge and diffusion-based I2V models below.

At the beginning, we want to emphasize that since the sampling and training process are closely related for bridge models and diffusion models, we should take the sampling algorithm into account when discussing the two types of models. No matter if we use SAF or not, the sampling process of FrameBridge is different from that of diffusion-based I2V models.

To be more specific, diffusion models generate data from noise with the backward diffusion SDE

$\mathrm{d}x_t = \left[f(x_t, t) - g(t)^2 \nabla_x \log p_{diff, t}(x_t, y)\right]\mathrm{d}t + g(t) \mathrm{d}w, \quad x_T \sim \mathcal{N}(0, I),$

while the sampling process of bridge models is equivalent to solve the backward bridge SDE

$\mathrm{d}x_t = \left[\tilde{f}(x_t, t) - \tilde{g}(t)^2 \left(\frac{1}{2}\nabla_x \log p_{bridge, t}(x_t, y, x_T) - \tilde{h}(x_t, t, y)\right)\right]\mathrm{d}t + \tilde{g}(t) \mathrm{d}w, \quad x_T = y.$

Here $f, g, \tilde{f}, \tilde{g}, \tilde{h}$ can be directly calculated given the bridge and diffusion schedule, and $p_{diff, t}, p_{bridge, t}$ are the marginal distribution of forward diffusion and bridge process. Similar to diffusion models, diffusion bridge models estimate the bridge score function $\nabla_x \log p_{bridge, t}(x_t, y, x_T)$ with the help of a neural network $x_\theta(x_t, t, y, x_T)$ (we use the data-prediction bridge models [1, 2] as an example for our illustration) trained with denoising bridge score matching (DBSM) loss, and the training loss is:

$L_{bridge}(\theta) = \mathbb{E} \left[ \lVert x_\theta(x_t, t, y, x_T)- x_0\rVert^2 \right]$

SAF modifies the input parameterization of $x_\theta(\cdot)$, but does not change the sampling process and the learning target $L_{bridge}(\theta)$ (i.e., we still train the model $x_\theta$ to estimate the bridge score function and sampling with the backward bridge SDE from a deterministic point $x_T$). Therefore, it is not exactly that "the only difference between the proposed method and the standard diffusion model is the prediction types". The advantages of bridge models over diffusion models lie not only in training but also in the sampling process, and the latter may even be more crucial.

More intuitively speaking, both FrameBridge with and without the proposed SAF technique generate videos from a deterministic prior, thus following a frames-to-frames process, which is naturally suitable for I2V generation tasks as demonstrated by our experimental results. FrameBridge with SAF only modifies the input of bridge models and still sample videos by solving backward bridge SDE with the estimated score of bridge process. This makes it inherently different comapred with diffusion-based I2V models.

We appreciate the reviewer's suggestion to use pseudo code to illustrate the differences between bridge models and diffusion models, and we will add that in the final version of our paper later.

Reply to W2: Main contributions of our work

We want to emphasize that our main contribution is to extend bridge models to video domain (specifically, to Image-to-Video generation), and design techniques (e.g., SAF and neural prior) which further improves bridge models.

• SNR-Aligned Fine-tuning: For video generation tasks, it is a widely adopted method to fine-tune a pre-trained T2V base model for other tasks. To the best of our knowledge, it is the first trial to fine-tune a pre-trained diffusion model to a bridge model, which is different from previous works of bridge models in image domain [2, 3].

• Neural prior: Bridge models for image-to-image translation tasks can directly use the condition image as the prior of bridge models, which is not the case for I2V generation. To obtain a prior in the same Euclidean space as the target, we first use the replicated image condition and then explore to generate the prior with another single-step regression model (i.e., neural prior).

Reply to W3: FVD learning curve

As suggested by reviewer ScY5, we use the new-FVD (content-debiased FVD [4]) to evaluate the video quality generated by I2V models.

To comapre FrameBridge with baseline models, we provide the new-FVD curve of DynamiCrafter and FrameBridge on MSR-VTT. (Due to the limiation of computational resources, we can not reproduce the training procedure of SEINE, and use the released model for evaluation)

Table: new-FVD on MSR-VTT at different training iterations

Since we use the same model structure and batch size for the FrameBridge and DynamiCrafter model, it can be directly concluded that FrameBridge generate better videos than its diffusion counterpart during the whole training process and it converges faster.

Reply to W4: Ablation for neural prior

To address the reviewer's concern, we report the FVD for diffusion models (with gaussian prior) using $F_\eta$ as a condition:

Comparing the second and last row of the table, it can be seen that bridge models can utilize the neural prior $F_\eta$ better by setting it as both the prior (i.e. the start point of sampling process) and model condition, and the prior plays an import role in bridge models.

Reply to W5: Evaluation metrics

We evaluate the new-FVD value of FrameBridge, DynamiCrafter, and SEINE on MSR-VTT:

Moreover, we adopt other reviewer's suggestions to provide evaluations on the advanced benchmark VBench [5] for video generation.

Table: Scores of VBench-I2V (higher the better)

FrameBridge outperforms other baselines on most of the dimensions, demonstrating the higher overall quality of videos generated by bridge models.

About dynamic degree: There is usually a trade-off between dynmaic motion and condition alignment [13] for I2V models, and the high dynamic degree scores of some baseline models are at the cost of condition and temporal consistency. FrameBridge can reach a balance demonstrated by the multi-dimensional evaluation on VBench-I2V. Moreover, the motion dynamic of FrameBridge is not low compared with other models on VBench-I2V leaderboard.

Table: Scores related to motion

*: We directly use the evaluation results provided by VBench-I2V leaderboard.

[1] Kaiwen Zheng*, Guande He*, Jianfei Chen, Fan Bao, and Jun Zhu. Diffusion Bridge Implicit Models. arXiv preprint arXiv:2405.15885, 2024.

[2] Linqi Zhou, Aaron Lou, Samar Khanna, and Stefano Ermon. Denoising diffusion bridge models. In ICLR, 2024.

[3] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima Anandkumar. I2sb: Image-to-image schrodinger bridge. In ICML, 2023.

[4] Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang. On the Content Bias in Fréchet Video Distance. In CVPR, 2024.

[5] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive Benchmark Suite for Video Generative Models. In CVPR, 2024.

Thanks for the author's rebuttal and further experiments and discussions. I have read the comments from the reviewers and the author's responses.

1. For my first concern, it is not well addressed. The explanation of the authors does not reduce my concern. The re-parameterization trick proposed in the paper will influence the calculation of the score function. Please provide proof of the equivalence of score function when using the re-parameterization.

2. For my requests for checking the pseudo-code of training/inference, the authors only responded with "they will". I am confused about why they do not want to show the comparison. It is the most straightforward way to do so.

3. Additionally, the comments shared by other reviewers that the visual quality of generated videos is bad also raise my concerns about the visual quality. The demo video presented in the paper and pages is not convincing. It has a large gap to many open-sourced image-to-video diffusion models. How can we expect it to work better in practice?

Overall, I appreciate the efforts the authors have paid to the rebuttal, but considering the above concerns I maintain my rate for rejection.

We appreciate the reviewer's recognition of the motivation and underlying ideas behind our approach. To address the reviewer's concerns, we provide more experimental results and clarifications of our method below.

Reply to W1: Discussions about related works

[1] is one of the representative works to manipulate diffusion schedules in the inference stage, with the attempt to align the marginal distributions between training and inference stages. In the field of video diffusion models, [2, 3] share the same underlying philosophy and use a noisy latent related to image conditions as the starting point of sampling process. FrameBridge has two main distinctions compared to these works:

1. [1, 2, 3] are all diffusion models, and the starting point of sampling process is a Gaussian distribution with certain variance scale. However, FrameBridge follows the framework of bridge models [4] and the sampling process begins with a deterministic latent $z_T$.

2. [1, 2, 3] focus on the sampling process of a diffusion model and do not modify the training process. These methods are applied to already trained diffusion models. In contrast, FrameBridge trains an I2V bridge model with the denoising bridge score matching loss (DBSM) [4, 5], which is fine-tuned from a T2V diffusion model or trained from scratch and outperforms its diffusion counterparts trained with the same method.

Meanwhile, we also want to clarify that SAF aims to align the marginal distribution of input latents in the fine-tuning stage with that of the pre-trained diffusion model. The objective of SAF is to effectively fine-tune a pre-trained diffusion model to a bridge model. Specifically, in I2V generation tasks, it fine-tunes a T2V diffusion model to an I2V bridge model.

We hope the above clarification can address the reviewer's concerns and we are willing to have a further discussion.

Reply to W4: Dynamic degree of FrameBridge and FrameBridge-NP

We would like to gently point out that "the transformation starting point of all frames comes from the transformation of the same frame" is a misunderstanding of our method. The above evaluation results has shown that FrameBridge will not limit the dynamic degree of generated videos, we will have a further discussion about that.

Theoretically, I2V bridge models will not limit the motion of the generated videos. Formally speaking, the latent of a bridge process at timestep $t$ (i.e., $z_t$) has the same marginal distribution as $a_t z_0 + b_t z_T + c_t \epsilon$, where $a_t = \alpha_t (1 - \frac{SNR_T}{SNR_t}), b_t = \frac{\alpha_t}{\alpha_T}\frac{SNR_T}{SNR_t}, c_t = \sigma_t \sqrt{1 - \frac{SNR_T}{SNR_t}}$ (Here $\alpha_t, \sigma_t$ is the schedule of a diffusion process used to construct the bridge process [4], and $SNR_t = \frac{\alpha_t^2}{\sigma_t^2}$.) Although at timestep $T$, $a_T = 0, b_T = 1, c_T = 0$, and $z_T$ is a deterministic point, the coefficient $c_t > 0$ at $t \in (0, T)$. Intuitively, different from the sampling process of a diffusion model (where the noise scale is monotonically decreasing), the noise scale of a bridge latent $z_t$ will first increase to a high point and then decrease during the whole sampling process. Notably, the sampling process of a bridge model is also a iterative process, instead of a one-step transformation.

Moreover, when neural prior is not applied, the latent $z_T$ is obtained by replicating the first frame. If we use neural prior, $z_T$ will be replaced by a regression model $F_\eta(z^i, c)$ ($z^i$ is the first frame and $c$ is additional conditions), it will definitely contain more information about motion than simply replicating $z^i$. So, neural prior do not limit the dynamics of generated videos and will add to the dynamics on the contrary.

Reply to W2 and Question

We will update our project page later and show more cases generated by a FrameBridge model trained longer (i.e. 100k iterations).

Meanwhile, we thank the reviewer to point out the typo and will carefully check our writing. We will upload our revised version of the paper later.

[1] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common Diffusion Noise Schedules and Sample Steps are Flawed. In WACV, 2024.

[2] Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint arXiv:2312.07537, 2023.

[3] Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation. arXiv preprint arXiv:2402.04324, 2024.

[4] Linqi Zhou, Aaron Lou, Samar Khanna, and Stefano Ermon. Denoising diffusion bridge models. In ICLR, 2024.

[5] Zehua Chen, Guande He, Kaiwen Zheng, Xu Tan, and Jun Zhu. Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis. arXiv preprint arXiv:2312.03491, 2023.

[6] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, Ying Shan. VideoCrafter1: Open diffusion models for highquality video generation. arXiv preprint arXiv:2310.19512, 2023.

[7] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors. In ECCV, 2024.

[8] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive Benchmark Suite for Video Generative Models. In CVPR, 2024.

[9] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In ICLR, 2024.

[10] Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models. arXiv preprint arXiv:2311.16933, 2023.

[11] Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, and Jun Zhu. Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model. In NeurIPS, 2024.

Reply to W3: Video quality of FrameBridge

In the first version of our paper, constrained by the very limited computational resources, we fine-tune a FrameBridge model from videocrafter-256 [6] for 20k iterations and compare it with a DynamiCrafter-256 [7] model trained with the same budget, aiming to demonstrate the advantage of I2V bridge models over their diffusion counterparts. To address the reviewer's concerns, we train a FrameBridge model for longer time (100k iterations) and compare it with baselines. Adopting the suggestion of reviewer pN4E, we provide the evaluation results on VBench-I2V [8], which is widely acknowledged as a reliable and comprehensive benchmark for video generation.

Table: Scores of VBench-I2V (higher the better)

The above table shows that the overall video quality of FrameBridge are higher than the baselines.

About dynamic degree

There is usually a trade-off between dynmaic motion and condition alignment [11] for I2V models, and the high dynamic degree scores of some baseline models are at the cost of condition and temporal consistency. FrameBridge can reach a balance demonstrated by the multi-dimensional evaluation on VBench-I2V. Moreover, the motion dynamic of FrameBridge is not low compared with other models on VBench-I2V leaderboard:

Table: Scores related to motion

*: We directly use the evaluation results provided by VBench-I2V leaderboard.

We would like to clarify that the training budget, the dataset size, the number of model paramters and the video resolution are all crucial factors for the quality of generated videos. Due to the limitation of computational resources, currently we can only train our models at 256 $\times$ 256 resolution with certain model size and training iterations. Moreover, the training dataset of some other models may consist of closed-source datasets and we only have access to the open-source ones. Nevertheless, we have made an effort to train an I2V bridge model under these constraints which outperforms the diffusion counterparts. The main target and contribution of our work is not to train a SOTA I2V diffusion model, but to propose a bridge-based I2V method and demonstrate the effectiveness with resonable and fair experimental results. We hope the evaluation results of both traditional metrics (FVD, IS, CLIPSIM, PIC) and comprehensive video benchmark (VBench-I2V) can address the concerns of the reviewer.

1. Distorting

You claim that "The transformation starting point of all frames comes from the transformation of the same frame." in my review is a misunderstanding. But I think you are distorting my review.

In your pipeline, z_0 = Bridge Process( Duplicate( ) ). Before the Bridge Process, all information is the same at the frame level. In diffusion models, z_0 = Diffusion Process( Random Sample( ) ). Before the Diffusion Process, all information is different at the frame level. Therefore, Diffusion Models may have more dynamics among frames, which aligns with the phenomenon in this paper. In my reviews, the "transformation" is your Bridge Process (I never say it is a one-step transformation, stop distorting please), and "the same frame" means the input of the Bridge Process. Is that clear? If it is not true, please fix your pipeline figure.

2. Ignoring

Weakness 2: some videos on the website can not be played. It seems that authors still modify the website when the review period starts.

Your response: we will update our project page later and show more cases generated by a FrameBridge model trained longer (i.e. 100k iterations).

If my statement is wrong, please feel free to correct it. Or if you want to skip this weakness, please do not say "Reply to W2". Are authors allowed to make unlimited changes to the anonymous website after reviewers have started reviewing the manuscript?

3. Video Quality

In my opinion, it is a reasonable weakness. I have read many generative-related papers in ICLR 25, many reviewers challenge the generated quality even if the method beats the baseline line methods overall. If ACs think reviewers should not do that, please stop all of them. If other submitting papers face this problem, why this article can be a special case?

Do the authors still plan to submit a rebuttal?

Due to the constraints of time and computational resources, we are unable to provide the results of user study currently. We will add it in the subsequent discussion period.

We hope the above experimental results and discussions can address the reviewer's concerns. We would be glad to have a further discussion.

[1] Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models. arXiv preprint arXiv:2311.16933, 2023.

[2] Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation. arXiv preprint arXiv:2402.04324, 2024.

[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. arXiv preprint arXiv:2311.15127, 2023.

[4] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models. arXiv preprint arXiv:2311.04145, 2023.

[5] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation. arXiv preprint arXiv:2311.17117, 2023.

[6] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive Benchmark Suite for Video Generative Models. In CVPR, 2024.

[7] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors. In ECCV, 2024.

[8] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. In NeurIPS, 2024.

[9] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.

[10] Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In ICCV, 2023.

[11] Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint arXiv:2312.07537, 2023.

[12] Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, and Jun Zhu. Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model. In NeurIPS, 2024.

Reply to W2: Evaluation of I2V models

As suggested by reviewer yg8s, we evaluate baselines and FrameBridge on an advanced I2V benchmark, i.e. VBench-I2V[6], and shows the results below:

Table: Scores of VBench-I2V (higher the better)

• The above table demonstrates the high level overall quality of videos generated by FrameBridge, and FrameBridge outperforms all the other baseline models across most dimensions of video quality (especially for those related to I2V Consistency). Considering the training budget and dataset size, FrameBridge is a promising method for I2V generation.

• About dynamic degree: There is usually a trade-off between dynmaic motion and condition alignment [13] for I2V models, and the high dynamic degree scores of some baseline models are at the cost of condition and temporal consistency. FrameBridge can reach a balance demonstrated by the multi-dimensional evaluation on VBench-I2V. Moreover, the motion dynamic of FrameBridge is not low compared with other models on VBench-I2V leaderboard.

Table: Scores related to motion

*: We directly use the evaluation results provided by VBench-I2V leaderboard.

We appreciate the reviewer for acknowledging the novelty and the potential of our approach. To address the reviewer's concerns, we provide more experimental results and discussions about related works below:

Reply to W1: Comparisons and discussions for more baselines

1. Comparisons with more baselines:

We add SparseCtrl [1] and ConsistI2V [2] as baselines and provide the evaluation results of zero-shot metrics on MSR-VTT and UCF-101. We also adopt the suggestions of other reviewers and include the results of some other classic I2V diffusion models (SVD [3] and I2VGen-XL [4]).

Table: Zero-shot metrics on MSR-VTT

Table: Zero-shot metrics on UCF-101

*: We directly use the data from [7].

**: Since the I2V model of SVD generate videos with 14 frames (all the other models in the above tables generate videos with 16 frames), the PIC metric (average of similarity between each frame and the image condition) of SVD may be overestimated as the image condition is used as the first frame.

Besides, we provide some additional clarification below:

• We do not add Animate Anyone [5] as it is an image animation method designed for character animation and requires additional conditions (i.e., pose sequence), which is a different setting from FrameBridge and other baseline models. Nevertheless, we will discuss this work later.

• For ConsistI2V, we use the officially released model weights. Although their model is trained with much more budget than our FrameBridge model, the video quality of FrameBridge is comparable to theirs.

Table: Training budget and dataset size for different models

• We will provide the comprehensive evaluation results on VBench [6] in the following part.

2. Discussions about previous works

• I2V diffusion models with additional fusion module: Some previous works [1, 5, 8, 9] have explored to enhance T2I or T2V diffusion models with additional modules to achieve controllable video generation. It is a constructive methodology as how to integrate conditions into the denoising network effectively is crucial for I2V and other controllable video generation tasks. In our work, we aim at replacing the conditional diffusion process with a frames-to-frames bridge process, and trains a bridge denoising network to sample videos. Our FrameBridge is parallel to these works, and we can freely choose the structure of the denoising network (e.g., in our experiments, we use the same network structure as DynamiCrafter [7] which also utilizes a lightweight projection module to process the image condition).

• Noise manipulation for video diffusion models: Several works have explored to improve the uninformative prior distribution of diffusion models. PYoCo [10] proposes to use correlated noise for each frame in both training and inference. ConsistI2V [2], FreeInit [11], and CIL [12] present training-free strategies to better align the training and inference distribution of diffusion prior. These strategies focus on improving the noise distribution to enhance the quality of synthesized videos, while they still suffer the restriction of noise-to-data diffusion framework, which may limit their endeavor to utilize the entire information (e.g., both large-scale features and fine-grained details) contained in the given image. In contrast, we propose a data-to-data framework and utilize deterministic prior rather than Gaussian noise, allowing us to leverage the clean input image as prior information.

We appreciate the reviewer's advice and will add these discussions in the final version of our paper later.

Reply to W2: Motion scale of generated videos

Samples with larger movements: We will provide more samples with larger movements on our project page.

Metric of dynamic degree: We use VBench-I2V [5] to evaluate the dynamic degree of generated samples. As mentioned above, although the dynamic degree of DynamiCrafter is higher than FrameBridge, it is at the cost of temporal consistency and motion smoothness. Moreover, the motion dynamic of FrameBridge is not low compared with other models on VBench-I2V leaderboard.

Table: Scores of VBench-I2V

*: We directly use the data provided by VBench-I2V leaderboard.

We would like to clarify that the motion scale of generated videos are influenced by several experimental factors independent of models and algorithms. For example, training dataset, text prompts and fps condition used when sampling, and data preprocess strategy (e.g., frame stride of training data) all play a role in determining the motion scale of videos generated by I2V models. Meanwhile, there is usually a trade-off between dynmaic motion and image alignment [8] for I2V models, and our FrameBridge aims at reaching a balance demonstrated by the multi-dimensional evaluation on VBench-I2V.

Reply to W3: Sampling steps

Justification for using 250 steps: Using more timesteps will guarantee the covergence of the solution for SDEs/ODEs and reduce numerical errors caused by the sampler. In our experiments, we focus on comparing the I2V generation capabilities between bridge-based I2V models and their diffusion counterparts, so we use a sufficiently large timestep to fully demonstrate the generative capability of different models and reduce the impact of numerical errors.

Quality vs. Number of timesteps: We evaluate the impact of sampling timesteps for quality on MSR-VTT datasets as below: (We will add figures to illustrate that in our final version of paper later)

Table: FVD on MSR-VTT (lower the better)

FrameBridge has lower FVD (better video quality) compared with its diffusion counterparts using different sampling timesteps.

Table: PIC on MSR-VTT (higher the better)

As shown, for FrameBridge, the similarity between generated video frames and image conditions is higher than that of diffusion-based I2V models across different sampling timesteps. Furthermore, the similarity remains relatively stable as the sampling timesteps are reduced, whereas the similarity of diffusion models decreases rapidly.

The wall-clock time required for sampling is directly proportional to sampling timesteps. FrameBridge and DynamiCrafter use networks with the same structure, so their sampling speeds are identical when using the same number of timesteps. We will compute the time required per step for SEINE and compare it with that of FrameBridge/DynamiCrafter later in the final version of our paper.

Reply to W4: Clarification of neural prior

• The default FrameBridge does not include a neural prior. Neural prior is a technique to further enhance FrameBridge by replacing the prior of bridge models with the results of another one-step regression model.

• Theoretically, the prior of bridge models can be chosen freely and replicating the first frame (i.e., image condition) is a convenient and effective practice justified by our experiments. The goal of experiments for FrameBridge-NP training from scratch on UCF-101 is to demonstrate that utilizing one-step regression neural prior (although it is a relatively vague and over-smooth intermediate result) is a useful method to improve the performance of default FrameBridge. Due to the limitation of computational resources, we can only fine-tune T2V diffusion models to I2V bridge models with the proposed SAF technique and are unable to train the neural prior model from scratch on WebVid dataset.

Reply to W5: Clarification of experimental setups

We want to clarify that the experimental setups in Table 1 and Table 2 are completely different.

All the baseline models and FrameBridge models in Table 1 are fine-tuned from pre-trained diffusion models, trained on large dataset (WebVid2M or WebVid10M) consisting of text-video pairs.

All the baseline models and FrameBridge models in Table 2 are trained-from scratch on relatively small-scale dataset (UCF-101) consisting of class-labeled videos. So, models like DynamiCrafter, I2VGen-XL can not be directly applied to this setting.

Ablation studies for Table 1: Part of the original Table 1 can be seen as the results of an ablation study.

• Results of FrameBridge (w/o SAF) vs. FrameBridge (w/SAF) show that FrameBridge with SAF technique performs better than those without SAF.
• Results of FrameBridge (w/o SAF) vs. DynamiCrafter show that bridge process is more suitable for I2V generation than diffusion process (as the two models have the same network structure and are trained with the same budget).

We hope these results and clarifications will address your concerns, and we are willing to have a further discussion.

[1] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors. In ECCV, 2024.

[2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. arXiv preprint arXiv:2311.15127, 2023.

[3] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models. arXiv preprint arXiv:2407.15642, 2024.

[4] Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation. arXiv preprint arXiv:2402.04324, 2024.

[5] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive Benchmark Suite for Video Generative Models. In CVPR, 2024.

[6] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models. arXiv preprint arXiv:2311.04145, 2024.

[7] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction. In ICLR, 2024.

[8] Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, and Jun Zhu. Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model. In NeurIPS, 2024.

We thank the reviewer for their constructive and practicable suggestions to improve the experiment part of this paper. In response, we have included additional results to address the reviewer’s concerns:

Reply to W1: Evaluation of baselines

DynamiCrafter: In the first version of our paper, we do not use the original weights of DynamiCrafter for evaluation as there are some differences between the setting in our work and DynamiCrafter [1].

• Training and sampling settings: Following previous works of Image-to-Video (I2V) generation [2, 3, 4], the target of FrameBridge is to generate videos with condition image as the first frame. In contrast, DynamiCrafter adopts a different setting and use a randomly selected frame from the whole video as image condition and there may be discrepency between the beginning of generated videos and image conditions.
• Training budget: For the original weights of DynamiCrafter, the I2V diffusion model is fine-tuned from a T2V diffusion model for 100k iterations, and FrameBridge is fine-tuned from the same T2V diffusion model for 20k iterations because of our very limited computational resources. Therefore, to compare FrameBridge and DynamiCrafter under the same experiment setting and budget, we reproduce DynamiCrafter at 256 $\times$ 256 resolution for 20k iterations with their official implementation, except that image conditions are sampled as first frames of video data.

In this rebuttal, to address the reviewer's concerns, we evaluate the original weights of DynamiCrafter and compare it with a FrameBridge model trained longer (i.e. 100k iterations):

Table: Metrics on MSR-VTT

Moreover, as suggested by Reviewer mfhQ, we also compare DynamiCrafter (original weights at 256 $\times$ 256) and FrameBridge (100k iters) on VBench-I2V [5]. (The model ranked first on VBench-I2V is the 576 $\times$ 1024 version of DynamiCrafter)

Table: Scores of VBench-I2V (higher the better)

FrameBridge outperforms DynamiCrafter-256 at different dimensions except dynamic degree. However, the higher score may be caused by more flickering and less temporal consistency (which can be justified by the temporal flickering score and motion smoothness score). We will have a further discussion about the dynamic degree of FrameBridge latter.

Other baselines: We adopt the reviewer's suggestions and add SVD[2], I2VGen-XL[6] as baselines. Below we provide the evaluation of sample results with 50 sample steps on MSR-VTT (we will add both results on MSR-VTT and UCF-101 in the final version of our paper):

Table: Metrics on MSR-VTT (50 sample steps)

*: Considering the resolution of released model, the inference of I2VGen-XL requires much computational resources, and there is no official evaluation result on MSR-VTT [6], we use the result reported in [1].

**: The I2V model of SVD only conditions on the input image and does not use text prompts, so we don't evaluate the CLIP similarity between video frames and text prompts. Besides, SVD generates 14 frames and all the other models in the table generate 16 frames. This may lead to higher average similarity between generated frames and the condition image (i.e. higher PIC) of SVD as all models use the condition image as the first frame in generated videos.