FrameBridge: Improving Image-to-Video Generation with Bridge Models

Dear Reviewer 2c9b,

We sincerely appreciate your recognition of the strengths and effectiveness of our work and the valuable suggestions to help us improve that. We are happy to have a discussion and hope it could address your concerns. Tables are provided in https://framebridge-icml.github.io/rebuttal-demo-page/.

Contributions of FrameBridge

1. Reply to W1

We would like to gently point out that the contribution of our work is not limited and our FrameBridge is a novel bridge-based I2V framework:

(1) SAF technique: Applying bridge to I2V generation is not trivial. As shown by Table 4 of our paper, directly fine-tuning T2V diffusion to I2V bridge may cause suboptimal video quality, and we propose SAF technique to align the two process. As far as we know, it is the first trial to fine-tune bridge models from diffusion models. A concurrent work [13] also shows the importance of aligning different processes in fine-tuning. Table 3, 4 in our link provide more evidence for the effectiveness of SAF.

(2) We propose neural prior to further improve the performance of bridge model. As bridge-models benefit from informative priors, it is well-motivated and effective to further investigate the choice of prior.

(3) We provide theoretically sound proof and derivations for all the techniques we proposed.

In conclusion, we are the first to apply bridge models to I2V generation, which may not achieve superior generation quality without the two novel proposed techniques (i.e., SAF and neural prior).

2. Discussions about other related works

We thank the reviewer for pointing out the related works we unintentionally left out, and will add citations and discussions in our revised paper. VideoElevator [14] proposed to improve the T2V generation quality with T2I diffusions, achieves high spatial quality and motion smoothness at the same time. Noise Calibration [15] proposed to enhance the video quality of SDEdit [16] with iterative calibration of initial noise. These are valuable works investigating into the improvement of T2V diffusion models, laying foundations for better I2V generation by building enhanced base models and providing viable techniques for improving diffusion-based I2V generation.

Qualitative Comparisons

Reply to S1 and W2: We thank the reviewer for the instructive and valuable suggestions and add a later academic baseline ConsistI2V (TMLR 2024) [7] in our qualitative comparisons. Although we fine-tune a FrameBridge model from CogVideoX, we choose CogVideoX-2B as base model and fine-tune for several hundred GPU hours, which is $< 1\%$ of that used to train CogVideoX-I2V (fine-tuned from CogVideoX-5B), HunyuanVideo and other industry-grade models. We provide more qualitative comparisons in https://framebridge-icml.github.io/rebuttal-qualitative/, and pre-trained models are specified. Please feel free to tell us if you still have any concern, and we are willing to provide more experimental evidence or have a further discussion.

Comparing FrameBridge with Inference-Time Noise Manipulation

Reply to W3: If we understand correctly, the provided baseline is a inference-time noise manipulation which only changes the prior when sampling and do not modify the diffusion process. (We are not certain if we have understood correctly. Feel free to point it out if there is misunderstanding and we are willing to provide more experimental resutls.) This baseline will generate almost static videos, and thus the quality is much inferior to FrameBridge and diffusion I2V models.

Table 1. Zero-shot metrics on MSR-VTT and VBench-I2V score.

Due to character limit, we kindly invite you to read our response to Revewer fXNN (Comparison between bridge model and inference-time noise manipulation part), where we provide more intuitive explanations. In case that you are interested in the comparison between bridge models and flow-matching (which shares some similarities with the mentioned the baseline), we kindly invite you to read our response to Reviewer g6Xo (Comparing bridge and flow-matching part).

Other Discussions

Reply to S2: Thanks for the suggestion and we will revise that in out paper later. (In this response, we temporarily align the decimal places with the current version.)

Due to the character limit, we can not elaborate on all the details. Please feel free to tell us if you still have any concern or question. We are willing to continue to have a kind and thorough discussion.

References: See response to Reviewer g6Xo.

Dear Reviewer g6Xo,

We sincerely appreciate your acknowledgement of the strengths of our work and instructive suggestions to help us improve that. We hope the following discussions could address your concerns and questions. Tables are provided in https://framebridge-icml.github.io/rebuttal-demo-page/.

Ablations between noise-to-data and data-to-data

We thanks the reviewer's suggestion and provide the experimental results of this ablation.

1. For the setting of fine-tuning, we provide the results in Table 3, 4. Here we use the same base model, network architecture, training data, training budget for noise-to-data and data-to-data models. As we need to fine-tune bridge models from pre-trained diffusion models, it is important to align the two process with our proposed SAF technique. A concurrent work [13] reaches similar conclusions when fine-tuning from flow-matching to diffusion process.

2. For the setting of training-from-scratch, since VDT-I2V has the same network architecture (Latte-S/2), training dataset, training budget as FrameBridge, we would like to gently point out that Table 3, 5 in the paper can offer experimental evidence for the ablation between noise-to-data and data-to-data. We appreciate the reviewer's question which help us to identify the lack of clarity in the setup (details are now dispersed in Sec. 5.2, App. D.3.3, D.3.4) and will revise it for better clarity.

Does data-to-data learn a shotcut to more static motion?

This is a thoughtful question. Bridge models will not learn a shortcut to more static motion. Due to character limit, we kindly invite you to read our response to Reviewer fXNN for more discussions (Reply to Q2 part).

Comparing noise-to-data and data-to-data

We wold like to offer more intuitive comparisons between the two frameworks.

1. When considering the differences between noise-to-data and data-to-data frameworks, we suggest to view from the perspective of forward-backward process instead of pointwise denosing. Theoretically, denoising score matching guarantees perfect score function for each $(z_t, t)$. But practically the score function is not perfect for each point, and we should consider the complexity for modeling the whole process.

2. Differences in process: For noise-to-data I2V generation, the image can only be injected into the model by network conditioning. For data-to-data I2V generation, the whole diffusion process is changed to a bridge process and now we can utilize the image condition from two aspects (network conditioning and process). As shown by Figure 2 in our paper, the marginal distribution $z_t$ in bridge process carries more information than in diffusion. In alignment with our experiments, modeling the whole bridge process with bridge score function could be easier than modeling the whole diffusion process with diffusion score function.

We hope the discussion could provide some intuitive insights for the readers and are willing to have a further discussion if there are other cocerns or questions.

Comparing bridge and flow-matching

We conduct experiments with 3 flow-matching models and compare them with diffusion and FrameBridge:

1. Vanilla flow-matching: The noisy latent takes the form of $z_t = (1-t)z_0 + t \epsilon, \epsilon \sim \mathcal{N}(0, I)$
2. Coupling flow-matching with $\sigma=1.0$ [18]: $z_t = (1-t)z_0 + t (z^i + \epsilon)$
3. Coupling flow-matching with $\sigma=0$ [19]: $z_t = (1-t)z_0 + t z^i$

Results are shown in Table 12 of our link. We compare bridge and flow-matching in both methods and experiment results as below:

1. The algorithm and performance of Vanilla flow-matching is quite similar to that of diffusion.
2. Coupling flow-matching with $\sigma=0$ will generate almost static video. In [19], it is used to refine the results of a diffusion, and need condition augmentation, which is not suitable for single-stage I2V.
3. Coupling flow-matching with $\sigma=1$ has better performance than diffusion models. Different from bridge models, the prior is still a Gaussian (with non-zero mean), while the prior of bridge is a deterministic point. Empirically, as shown by the experiment results, brige models can utilize the condition more effectively.

Due to character limit, we can not elaborate on all the details. Please feel free to tell us if you still have any concern and we are willing to have a further discussion.

References (Due to character limit, we can only provide abbreviated forms of some works where there is no ambiguity):

[1] VBench++

[2] Stable Video Diffusion

[3] CogVideoX

[4] Conditional Image Leakage

[5] I^2SB

[6] DDBM

[7] ConsistI2V

[8] RPGDiffusion

[9] MovieDreamer

[10] VideoGen-of-Thought

[11] VideoDirectorGPT

[12] VIDIM

[13] SANA-Sprint

[14] VideoElevator

[15] Noise Calibration

[16] SDEdit

[17] Flow Matching for Generative Modeling

[18] Stochastic Interpolants with Data-Dependent Couplings

[19] Boosting Latent Diffusion with Flow Matching

Dear Reviewer WAVy,

We sincerely appreciate your recognition of the strengths of our work and providing valuable suggestions to help us improve it. We hope the following discussions can address your concerns. Tables and demos are provided in https://framebridge-icml.github.io/rebuttal-demo-page/.

Performance on VBench-I2V

1. Reply to W5

We find there are discrepancies between the results of Table 2 in our paper and the latest version of VBench-I2V [1] and thank the reviewer for bringing this to our attention. We present the modified version in Table 1. Note that two FrameBridge models still achieve higher total score than baselines.

2. Reply to W4

It is a valuable consideration and we will provide our empirical results and theoretical discussions. Before that, we would like to gently point out that the much higher DD of SparseCtrl may lead to inferior overall video quality as shown by Table 1 and 6. We kindly invite you to read the discussions in our response to Reviewer fXNN (Reply to Q2 part) due to the character limit.

We would appreciate it if you could kindly take the time to read that, and it may also provide intuitions about why the samples can "deviate" from the prior, which is also appicable to the scenarios involved in W1 and W3.

Implementation Details

Reply to W1

Firstly, we would like to calrify that neural prior predicts the mean video conditioning on the first frme instead of "mean image". (i.e., Frames can be different.) We list the model input, condition, priors of diffusion and bridge models below.

Here $F_\eta(z^i, c)\in \mathbb{R}^{F \times h \times w \times d}$ has the same shape as video latents and we do not concatenate $F_\eta(z^i, c)$ with the original image when it serves as a prior (as illustrated in our pseudo code, line 846), and this design is not a hack. However, both diffusion and bridge model should take original image as an input to ensure it is learning the conditional score function. Blurry prior will not degrade quality of final videos (as shown by our experiments) and bridge models here can be seen as a refiner. (e.g., Bridge can be applied to deblurring [5].)

Experimental Design

1. Reply to W2

To address the concern, we conduct ablations of SAF in zero-shot setting and provide results in Table 3, 4. Neural prior is designed for training from scratch and the ablation setting is aligned with the main experiments in our paper. We would like to gently point out that the ablation with VideoCrafter is already provided in the Table 1 of paper (line 342, 344, 345). We thank the reviewer's suggestion and will improve the clarity of ablation studies. For Figure 6, the first point is evaluated when the model is fine-tuned for 3k steps. When initialized, both models generate noise and thus we omitted it. To address the concern, we provide the CD-FVD curve before 3k in Figure 1.

2. Reply to W3

This is a thoughtful suggestion to help us show the effectiveness of FrameBridge more thoroughly. Due to the constraints of time and computational resources, we temporarily provide the experiment results with 24 frames in Table 5. Conducting experiments with long video requires significant computational resources: The base model should be large enough to have capability to deal with long video and training with more frames will also add to computational complexity. As far as we know, baselines and benchmarks under this setting (I2V for longer than 16 frames) is not mature enough. Taking all the above into account, we compare FrameBridge with a diffusion I2V model fine-tuned from the same CogVideoX-2B, and find FrameBridge outperforms the diffusion counterpart. Please feel free to tell us if you still have concerns and we are willing to provide more experimental evidence and discussions.

Other Discussions

1. Reply to Q1: This is an insightful question and we think the input/output parameterization is not the most important difference. We kindly invite you to read our response to Reviewer g6Xo (Comparing noise-to-data and data-to-data part.)

2. Reply to Q2: We kindly invite you to read our response to Reviewer fXNN (Reply to Q1 part).

3. Additional references: We thank the reviewer for pointing out the related works we unintentionally left out, and will add citations and discussions in our paper later.

4. More demos of FrameBridge: We provide more demos generated by FrameBridge in Table 9-11, which we hope could address the concern of dynamic degree.

Due to the word limit, we could only elaborate on some important points. Please feel free to tell us if you still have any concerns and we would be happy to have a kind and thorough discussion further.

References: See response to Reviewer g6Xo.

Dear Reviewer fXNN,

We sincerely appreciate your acknowledgement of our methods and proposed techniques as "well-motivated", "practical and effective". We are happy to engage in a thorough discussion and hope it will address your concerns and questions. Tables and demos are provided in https://framebridge-icml.github.io/rebuttal-demo-page/.

Reply to Q2

Firstly, we will offer some empirical solutions to improve the dynamic degree of FrameBridge. The limited dynamic degree is a well-studied issue for diffusion I2V models [2,3,4]. We adapt two techniques used in diffusion I2V models to FrameBridge: (1) Add frame-stride condition; (2) Add noise to image condition. Demonstrated by our experments on FrameBridge-VideoCrafter, we find that these techniques are also effective for FrameBridge, and can improve the DD score from 35.77 to 48.62, which is much higher than the diffusion counterpart (38.69 for DynamiCrafter). (See Table 2,7,8. -MI for Motion-Improved).

Secondly, we would like to mention that the potential limited dynamic degree is an empirical issue instead of the inherent weakness of bridge-based methods. Just as diffusion models, if we have the perfect bridge score function and a perfect sampler for the backward bridge SDE (Eq. 5 in the paper), we can sample from the true data distribution, which is a mathematically well-established conclusion in [6]. It may be counterintuitive. To understand it more intuitively (but not that rigorously), we would like to provide discussions from three aspects:

1. Marginal: Consider the marginal $z_t=a_t z_0 + b_t z_T + c_t \epsilon$, where $a_0=b_1=1, a_1=b_0=c_0=c_1=0$. Different from the diffusion process, the coefficient of noise $c_t$ first increases then decreases when $t$ varies from $T$ to $0$ in the sampling process. So, the latent $z_t$ will continuously become noisier in the first half part of sampling process.

2. Forward-Backward SDE: In [6] a forward diffusion process

$$\mathrm{d}z_t = f(t)z_t \mathrm{d}t + g(t)\mathrm{d}w,$$

is used to construct the forward bridge process

$$\mathrm{d}z_t=[f(t)z_t + g(t)^2 \nabla_{z_t} \log p_{T,diff}(z_T|z_t)]\mathrm{d}t+g(t)\mathrm{d}w,$$

Compared with diffusion process, the additional term $g(t)^2 \nabla_{z_t} \log p_{T,diff}(z_T|z_t)$ gradually "pushes" $z_t$ to the value $z_T$ and finally the prior $z_T$ will become a fixed point. Reversely, compared with diffusion backward process, the backward bridge process will additionally "pull" $z_t$ away from $z_T$.

3. Comparison between bridge model and inference-time noise manipulation: In FrameInit, the prior $z_T$ is constructed by combining low-frequency information from a static video with Gaussian noise. (Similar to FrameBridge at first glance.) Then, $z_T$ is denoised with a trained diffusion model. Although $z_T$ has been influenced by the low-frequency components of a static video, the diffusion process will sample from it in the same way as a Gaussian noise, since the training process of diffusion model is "ignorant" of the noise manipulation and still learns the score function of a standard diffusion process $z_t=\alpha_tz_0+\sigma_t\epsilon$, which may cause train-test mismatch and limit the dynamic degree. (This may also explain why the Dynamic Degree score of ConsistI2V is lower.) However, for bridge model, we change the prior $z_T$ along with the diffusion process. The model is "aware" of that and will learn the score function of a bridge process $z_t=a_0z_0+b_tz_T+c_t\epsilon$ with bridge score matching loss.

Reply to Q1

This is a valuable question, and multi-shot/scene I2V generation is a highly demanding task for I2V generation without mature benchmarks or baselines currently. It will also be challenging for FrameBridge as repeating the first frame as prior may not be a proper design. Inspired by RPGDiffusion [8] and other previous works about long video generation [9,10, 11] , we propose a possible hierarchical solution as below:

1. Use LLM planner to divide the whole video into several sub-video clips $v_1, v_2, ..., v_M$, and generate image prompts for the start of each clip.

2. Generate the start of each clip $i_1, i_2,..., i_M$ with a T2I diffusion model.

3. For $v_t, 1 \leq t \leq M - 1$, construct a neural prior by interpolating between $i_t, i_{t + 1}$ and train a FrameBridge for video interpolation (which is also a typical task for I2V generation [12]) to generate it. Generate $v_M$ with the static prior constructed by $i_M$ using FrameBridge.

Our consideration is to generalize the neural prior with LLM planner and use FrameBridge to leverage the locally reliable prior.

Other Discussions

The line 168 is a typo and should be $(z_T | z^i, c)$. We thank the reviewer to point it out and will revise it later.

Due to word limit, we can not elaborate on all the details and please feel free to tell us if you still have any concern.

References: See response to Reviewer g6Xo.