Author Response to Reviewer f9AW

We thank reviewer f9AW for the valuable and constructive comments. We address the concerns as follows.

Q1: Claim. The conditional image and the diversity of content changes are, in fact, a matter of trade-off. The essence of image-to-video generation is to leverage the conditional image to enhance video quality. Proposition 1 and the Training Strategy proposed by the authors are essentially fine-tuning techniques that favor the diversity of motion.

We appreciate the reviewer for the comment but disagree with our highest respect. While the conditional image for video quality and motion diversity can be a trade-off, we emphasize that our results simultaneously achieve dynamic motion and high video quality, without sacrificing either aspect. As demonstrated in Table 1, we achieve superior FVD and IS metrics compared to all baselines, indicating higher video quality. The user study in Figure 9 further confirms that our method outperforms all baselines in overall aspects, including video quality and dynamic motion. The results in Rebuttal Table A show that our method achieves a lower motion score error than all baselines, underscoring that our method helps to control motion degrees more precisely.

Q2. Recent studies show incorporating additional guidance signals can achieve significant motion and control the motion degree. From a subjective standpoint, not all applications require substantial motion.

We clarify that our method is orthogonal to the approaches that introduce additional signals and our method enables them control motion degrees more precisely. We validate this by consistently achieving a lower error between the motion score of the generated video and the given desired motion score, regardless of the input motion score in SVD (see Rebuttal Table A). More details are provided in Common Concern 1. DragAnything, DragNUWA, MotionCtrl, and ToonCrafter are novel controllable methods that achieve great performance on controllable video generation. We will include a discussion with these methods in the Related Work section in the final version

Q3: Ablation studies.

We did a systematic analysis of each hyperparameter in our proposed strategy in the original paper.

For the start time $M$ in Analytic-Init, the qualitative and quantitative results are reported in Figure 4, Table 7 and Table 8. As discussed in lines 144-146 and lines 159-162 of the original paper, an appropriate $M$ can enhance performance by increasing motion without compromising other performance. A too-small $M$ delivers poor visual quality due to the training-inference gap. Analytic-Init helps to mitigate it, especially with a smaller $M$.

For the two hyperparameters in the training strategy, the qualitative results and corresponding analysis are shown in Figure 5 and described in lines 183-195 in the original paper. Specifically, higher noise levels, such as those produced by a concave function ($a$ < 1), enhance dynamic motion but reduce temporal consistency and image alignment. For the maximum noise level $\beta_m$, a higher $\beta_m$ enhances dynamic motion but decreases temporal consistency and image alignment, while a lower $\beta_m$ leads to less motion.

Q4: In Figure 9, was a user study not conducted on the Analytic-Init? What does '-tune' signify—does it mean finetuning (training strategy)? If so, why does it appear in the inference phase?

The inference strategy in Figure 9 is denoted as Analytic-Init. The < method >-tune represents a naively finetuned baseline using the same training setup and dataset as our TimeNoise training strategy, ensuring a fair comparison. We validate our inference strategy on both the original baseline and a naively finetuned baseline. The < method >-tune baseline is crucial because I2V-DMs often use private datasets with varying data filtering, resolutions, and unknown training settings. Controlling these variables ensures a fair comparison. We will rename < method >-tune as < method >-naive-tune and label the inference strategy as Analytic-Init for clarity in the final version.

Q5: Insights into the selection of hyperparameters

As discussed in lines 184-185 of the original paper, firstly, the principle of $\mu(t)$ is to increase monotonically with time step. This ensures that higher noise levels are favored at later time steps, which carries a higher risk of leakage. To formalize this, we define $\mu(t)$ as a power function: $\mu(t)=2t^a-1, a>0$. This choice allows us to flexibly tune $a$ to achieve various monotonic behaviors, where smaller values of $a$ indicate that more of the later time steps will sample higher noise levels. Specifically:

• $a=1$ corresponds to a linear increase,
• $a<1$ indicates a concave function, suggesting a faster increase in noise levels over time, and
• $a>1$ represents a convex function, indicating a slower increase in noise levels over time.

For the choice of the maximum noise level $\beta_m$, the only principle is that it should be greater than 0. We refer to popular choices of maximum noise levels in VE-SDE[1,2] (e.g. 80-100) and experiment with parameters around these values.

[1] Score-based generative modeling through stochastic differential equations.

[2] Elucidating the Design Space of Diffusion-Based Generative Models.

Dear Reviewer f9AW,

Thank you for your feedback. We are pleased to hear that our responses have addressed most of your concerns. We believe the additional experiments, analysis, and explanation have significantly improved the quality and clarity of our submission. We hope that you and other reviewers may regard this as a sufficient reason to raise the score.

Best, Authors

Author Response to Reviewer hjyn

We sincerely thank Reviewer hjyn for the constructive and valuable comments. The concerns are addressed as follows.

Q1: Issues with derivation.

We clarify that our objective is to optimize $\mu_p$ and $\sigma_p^2$, the parameters of $p_M(X_M)=N(X_M; \mu_p, \sigma_p^2 I)$, rather than $\mu_q$ and $\Sigma_q$ (as stated in Proposition 1, line 155 of the original paper). Note that $\mu_q$ and $\Sigma_q$ represent the expectation and covariance matrix of the forward marginal distribution $q_M(X_M)$, which is determined by the forward diffusion process and is independent of the $\mu_p$ and $\sigma_p^2$ in $p_M(X_M)$. Given this independence, the second and third terms in Equation (7) can be disregarded for the purpose of optimization. Therefore, the optimization objective can be formulated as:

We will improve the writing of Appendix A in the final version.

Q2: The authors say that with previous works the models could in principle just output the conditioning image several times and still lead to lower error or improved quality metrics like FID and IS. However, they still use the same metrics for their experiments. Perhaps quantifying amount of motion or even better, quantifying realism of motion directly is important.

We clarify that models that repeatedly output the conditioning image might achieve a lower training loss, as defined by Eq. (2) in the original paper, rather than a lower evaluation error or improved video quality metrics like FVD and IS. In fact, as demonstrated in Table 1 of the original paper, such repetitive behavior can lead to poorer FVD and IS, since these metrics can evaluate the dynamic qualities of videos and static conditioning images lack the inherent dynamics. It's important to note that FVD and IS are video evaluation metrics based on an action classification model rather than image-based metrics like FID and IS, which can reflect video dynamics and video quality.

As suggested, we also incorporate the motion score metric to quantify the amount of motion, which is computed by obtaining dense optical flow maps with RAFT [1] between adjacent frames, calculating the magnitude of the 2D vector for each pixel, and summing these magnitudes across the frame. As demonstrated in Rebuttal Table F, our approach yields substantially higher motion scores compared to all baseline methods. We will add this in the final version. Quantifying the realism of motion is indeed crucial, yet, to the best of our knowledge, there currently exists no specific automatic quantitative metric to evaluate this aspect directly. Our user study, which covers overall aspects, takes this into account. We welcome any suggestions from the reviewers regarding additional metrics for evaluating motion realism.

[1] Raft: Recurrent all-pairs field transforms for optical flow.

Rebuttal Table F. Motion score results on the ImageBench dataset. < Method >-naive-tune represents a naively fine-tuned baseline using the same training setup and dataset as our TimeNoise training strategy, ensuring a fair comparison.

Q3: Explanation of Figure 9.

Figure 9 shows user preference percentages for our methods versus baselines. The left side displays baselines, while the right side shows our inference (Analytic-Init) and training (TimeNoise) strategies. Orange indicates the preference for our strategies and blue for the baselines. For instance, in the first row at the top left, 84.2% preferred our inference strategy over the original DynamiCrafter's output (15.8%) from an overall perspective. The < method > and < method >-tune denote the original model and naively finetuned versions, serving as baselines. The < method >-tune uses the same training settings and dataset as our TimeNoise strategy, ensuring a fair comparison. This is crucial because I2V-DMs often use private datasets with varying data filtering, resolutions, and unknown training settings. Controlling these variables ensures a fair comparison.

In summary, we validate our inference strategy against the original method and a naively finetuned version, and our training strategy against a naively finetuned version. As discussed in the original paper (lines 238-245), our strategies significantly enhance video dynamism and natural motion, maintaining image alignment and temporal consistency, thus achieving superior results overall. We will rename < method >-tune as < method >-naive-tune for clarity and improve the caption of Figure 9 in the final version.

Author Response to Reviewer NwFJ

We thank Reviewer NwFJ for the valuable comments.

Q1: Add comparisons.

Q1-1: Add comparison with related noise initialization methods.

(1) As suggested, we add comparisons with FreeInit[a], Progressive Noise[b] and FrameInit[c]. Our Analytic-Init outperforms all baselines across all metrics, as demonstrated in Rebuttal Table C. Specifically:

• compared with FrameInit: we achieve much higher motion scores. This is because FrameInit duplicates the conditional image into a static video and uses it as coarse layout guidance,which can limit motion degree.
• compared with FreeInit: we reduce inference time by more than half, while achieving a slight enhancement in performance. This is because FreeInit's iterative refinement process, involving the sampling of a clean video, adding noise, and regenerating the video, increases its overall inference time.
• compared with Progressive Noise: we achieve better performance. This is because our inference framework generates videos from an earlier time step, while the baseline can not handle this training-inference gap.

(2) Additionally, we clarify that our initialization method enhances temporal consistency (referred to as 'low frequency' in related work), consistent with findings in prior work ( see 5th vs 9th columns in Figure 4 of the original paper). Our high-motion capability (referred to as 'high frequency' in related work) is improved by starting the generation process at an earlier time step rather than initialization, avoiding the unreliable late-time steps of I2V-DMs.

Thank you for highlighting these important related works. We will incorporate the above discussions in the final version.

Rebuttal Table C. Comparison of different noise initialization methods. Output MS is the motion score of the generated videos, computed via an optical flow network. User Rate. indicates the average rank assigned by users across all methods from overall aspects. Inference Time refers to the duration required to generate a video.

Q1-2: Add comparison with PIA.

As suggested, we apply our inference strategy to PIA. Our method consistently enhances the motion degree, regardless of the initial motion level in PIA, as demonstrated in Rebuttal Table D in the reponse PDF. This improvement suggests the presence of conditional image leakage in PIA. It is important to note that our method is orthogonal to approaches that introduce additional signals, such as PIA and SVD, and enables them control motion more precisely (see Common Concern 1).

Q2: The authors claim models learn a shortcut by copying the image, yet they still produce outputs with motion.

We clarify that the static video example in Figure 2 is a schematic representation. We aim to highlight that these models tend to generate videos with reduced motion due to over-reliance on the conditional image at large diffusion timesteps (see lines 6, 34, 103 in the original paper). In some cases, this over-reliance might lead to nearly static videos, as exemplified in Figure 2. We repeat the experiments three times using different random seeds and conducte a statistical analysis using a two-sample t-test between the baselines and ours. As demonstrated in Rebuttal Table E in the response PDF, our method achieved higher motion scores with a significance level of p < 0.05 compared to all baselines. We will make it clear in the final version.

Q3: The principle for motion score chosen. Plot the input motion score against the output motion score.

For the input motion score chosen, we select a range of motion scores between 5 and 200 on the ImageBench. A score of 20 achieves high visual quality, including high temporal consistency and natural motion, and is used as a baseline. In the remaining experiments (Table 1, Figure 7, Figure 9), we directly use this 20 motion score for the baseline SVD, SVD-tune, and SVD with Analytic-Init. As suggested, we report the input motion scale against the output motion scale in Rebuttal Table A and Figure 1 in the response PDF. The results show that our method enables SVD control motion degrees more precisely by consistently achieving a lower error between the motion score of outputs and input motion score. See more detailed analysis in Common Concern 1.

Q4: The motion score in Figure 3 and Figure 7

In Figure 3, we use the default motion score of 127 in SVD, which tends to show camera movement with poor temporal consistency. In Figure 7, as detailed in Q3, we use a motion score of 20 for its high visual quality. As discussed in Common Concern 1, our method consistently achieves a lower motion score error, regardless of the input motion score in SVD.

Q5: How does TimeNoise affect inference time?

During inference, we use the conditional image with a fixed noise level across all time steps, similar to CDM[1], which does not affect inference time. This is because, during training, the model learns to handle conditional images with varying noise levels. In our initial experiments, we tested noise levels at 0, 5, and 10, finding that levels 0 and 5 exhibit similar performance, while a level of 10 sacrifices image alignment and temporal consistency. Based on these findings, we directly use the clean conditional image across all time steps for simplicity. We will include these details in the final version.

[1] Cascaded diffusion models for high-fidelity image generation.

Thank you for the feedback and for updating the score. We greatly appreciate your patience and the multiple rounds of communication, as your engagement has significantly enhanced the quality of our work.

We appreciate the reviewer’s quick feedback. Below, we address the further concerns.

Q1: Details about the experiments across motion scores.

Q1-1: Please detail the computation of the motion score

We compute the motion score by resizing the video to $800\times450$, extracting dense optical flow maps using RAFT [1] between adjacent frames, calculating the magnitude of the 2D vector for each pixel, averaging these magnitudes spatially, and then summing them across all frames.

[1] Raft: Recurrent all-pairs field transforms for optical flow.

Q1-2: Why it would make sense to directly compare the input/output motion scores.

Training SVD involves learning a conditional probability distribution $p(X|y)$, where $y$ represents the motion score condition. During inference, the generated output $X' \sim p(X|y)$ is expected to match the given condition $y$. Therefore, the error between the motion score of generated output $y'$ and the given input score $y$ can reflect the discrepancy between the generated outputs and the target distribution in terms of motion magnitude.

We will include the above details in the main paper in the final version.

Q2: Comparisons with PIA: "it categorizes motion degree into three discrete levels (small/moderate/large) labeled as 0/1/2" -- this is not true. The motion value from PIA comes from pixel-level L1 distance in the HSV space which obviously has more than three levels.

Thanks for highlighting the misunderstanding about PIA. The default three levels (small / moderate / large) of motion degree in PIA correspond to three predefined input affinity scores $S _ {in} =$ {$s _ {in}^i$}$_ {i=1}^n$ , which are negatively correlated with the motion score, where $n$ is the frame and $s _ {in}^i\in[0,1]$. Consistent with the experiments from Q1, we evaluate the precision of motion control by computing the error between the input and output affinity scores. We implement the output affinity scores $S _ {out} =$ {$s _ {out}^i$}$_ {i=1}^n$ following the guidelines for calculating the input affinity scores as described in the PIA paper's Inter-frame Affinity section (3.2). Then error between them is computed as:

As shown in Rebuttal Table G, our method consistently outperforms the baseline by achieving a lower affinity score error, regardless of the input affinity score. We will include the above experiments in the main paper in the final version.

Rebuttal Table G. The error comparison between PIA and PIA with our Analytic-Init. The three motion degree levels of Input Motion Level correspond to three predefined input affinity scores.

Author Response to Reviewer iEBW

We sincerely thank Reviewer iEBW for the recognition of our work and for providing constructive comments.

Q1: Details about the ImageBench dataset.

Our ImageBench dataset is designed based on two key aspects: breadth of categories and logical complexity. For breadth, we include popular categories such as nature, humans (both single and multi-person), animals, plants, buildings, food, arts, and vehicles. For complexity, we incorporate elements like numerals, colors, and complex scenes. Following these principles, we collected 100 images from various sources and T2I models like SDXL[1] and UniDiffuser[2]. We will add this in the revised version.

[1] Sdxl: Improving latent diffusion models for high-resolution image synthesis.

[2] One transformer fits all distributions in multi-modal diffusion at scale.

Q2: While performing the user study, what qualities are evaluated by the users?

As noted in lines 226-228 of the original paper, users were asked to conduct pairwise comparisons between our method and the baselines, assessing dynamic motion, temporal consistency, image alignment, and overall performance.

Q3: Add the evaluation about the text alignment.

We appreciate the constructive feedback provided. Following the suggestion, we add text alignment evaluation using the CLIP-text score[1] and a user study in Rebuttal Table B. Since SVD does not utilize text prompts as input, we conducted these evaluations only for VideoCrafter1 and DynamiCrafter. The results demonstrate that our approach achieves comparable text alignment performance with the baselines, suggesting that our strategy does not adversely affect text alignment quality.

[1]: Learning Transferable Visual Models From Natural Language Supervision

Rebuttal Table B. Text alignment performance comparison on the ImageBench dataset. < Method >-naive-tune and < Method >-TimeNoise represent finetuned models with naive strategy and our training strategy respectively, using the same setup and dataset for a fair comparison. User preference indicates the percentage of users who preferred our method over the baselines. DC. and VC. denote DynamiCrafter and VideoCrafter1.

Q4: Future work on improving the realism of generated videos.

We appreciate the insightful comments. We think there are two promising ways for enhancing the realism of the generated videos. One promising direction is to explore advanced model architectures, such as the Transformer architecture. By refining the architecture to better capture long-range dependencies and spatial-temporal coherence, we may enhance the realism and consistency of the generated videos. Another possible aspect is to scale up the diffusion model. Scaling the model can involve increasing the depth and width of the network, as well as using larger datasets to improve quality.