Dear Reviewer NpS9,

Thank you for your thoughtful review and for recognizing the strengths of our work. We appreciate your specific feedback and are committed to addressing each of your concerns to improve the clarity and rigor of our paper.

Weakness 1

Reviewer's Comment: "The paper feels rushed. (The caption of the teaser figure misplaced the text for sub-figures (B) and (C))."

Response: We apologize for the typo in the caption of the teaser figure, which misplaced the text for subfigures (B) and (C). We will correct this in the revised version. If there are other areas where the presentation could be improved, we welcome your specific suggestions to improve the clarity of our paper.

Weakness 2

Reviewer's Comment: “What is the novelty of the proposed method given the "Make Pixels Dance" (CVPR'24) paper”

Response: We understand your concern about the novelty of our method compared to the paper "Make Pixels Dance" [1] (PixelDance). Although both papers use state triplets, approach is significantly different:

• PixelDance: Utilizes ⟨first frame, text prompt, last frame⟩ triplets in a custom architecture requiring training from scratch, aimed at enhancing T2V model training and improving video dynamics.
• State & Image Guidance: Employs a training-free scheme with novel inference mechanics for existing T2V models like VideoCrafter2 [2]:
  • State Guidance: Conditions on ⟨first state prompt, transition state prompt, last state prompt⟩ for dynamic video scene generation.
  • Image Guidance: Conditions on ⟨prompt, reference⟩ duplet for Zero-Shot I2V generation.
  • Combination of State Guidance and Image Guidance: Conditions on ⟨first frame, text prompt, last frame⟩ triplet for Zero-Shot II2V generation.

While our Zero-Shot II2V pipeline is comparable to the II2V regime of PixelDance, the motivations and implementations differ significantly. For a detailed comparison with "Make Pixels Dance", please refer to Appendix A.5 (lines 892-896 and Figure 8).

Weakness 3

Reviewer's Comment: “It is extremely difficult to follow the experimental results and appreciate the contributions.”

Response: We understand the importance of clarity in the presentation of experimental results and would like to address your concerns:

1. References to different methods: Due to space limitations in the tables, we have included references to the methods in the text preceding each table. Specifically:
   • Table 3: The methods are listed on lines 395-396 and 403-405.
   • Table 5: Methods are listed on lines 462-463.
   • Table 6: Methods are listed on lines 484-485.
2. Ablation studies in main text: Including ablation studies in the main text is standard practice to highlight the impact of different components of the method. For example, recent papers such as "Ground-A-Video" [3] (ICLR'2024), "ControlVideo" [4] (ICLR'2024), and the aforementioned "Make Pixels Dance" [1] (CVPR'2024) include ablation studies in their experimental sections. We believe that this placement helps readers to better understand the papers.
3. Supplementary Material Navigation: Guidance for navigating the Supplementary Files is provided in Appendix A.7. We aim to make it as user-friendly as possible and will consider further improvements.
4. The bold numbers in Table 3 highlight the performance improvements when using State Guidance (SG) over standard inference. Other methods are included for reference to show that even commercial models and models with multi-prompt inference may not perform as well.
5. Table 5 shows the performance of our method with different "thresholds" (hyperparameter values) introduced in formula (6). We compare these variations with each other and with TI2V-Zero to show the flexibility and effectiveness of our approach (with the best results in bold).

We hope these clarifications have addressed your concerns. We are committed to improving the presentation and clarity of our work and look forward to your response. Please don't hesitate to contact us if you have any further questions.

Thank you again for your valuable feedback.

---

[1] Zeng, Yan, et al. "Make Pixels Dance: High-Dynamic Video Generation." (CVPR’2024).

[2] Chen H. et al. "VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models." (CVPR’2024).

[3] Jeong H., Ye J. C. "Ground-a-video: Zero-shot grounded video editing using text-to-image diffusion models." (ICLR’2024).

[4] Zhang Y. et al. "ControlVideo: Training-free Controllable Text-to-Video Generation." (ICLR’2024).

Dear Reviewer EnJp,

Thank you for your valuable feedback. We would like to address some of your concerns and initiate a discussion.

Regarding your points related to the CogVideoX paper (Weakness 1, Weakness 4, and Question 3), we would like to clarify that CogVideoX was released only 1.5 months before the submission deadline. According to general submission guidelines, it is acceptable not to consider such recent developments in our initial submission. Nevertheless, we are currently conducting detailed experiments with this model and will provide thorough responses within a week.

Thank you for your understanding and patience. Responses to all other weaknesses and questions (Weaknesses 2, 3 and Questions 1, 2) are presented below.

Weakness 2

Reviewer's Comment: “The paper lacks key information on how the transition order is maintained…”

Response: The transition order is maintained by scheduling frame-wise hyperparameters $\gamma_{is}^f$ ,$\gamma_{ts}^f$, and $\gamma_{fs}^f$ as detailed in Eq. 2. These hyperparameters influence each frame's state:

• $\gamma_{is}^f$ is highest at the first frame and decreases over time.
• $\gamma_{ts}^f$ peaks in the middle and is lowest at the beginning and end.
• $\gamma_{fs}^f$ is lowest at the first frame and increases towards the final frame.
• We make sure that $\gamma_{is}^f +\gamma_{ts}^f+\gamma_{fs}^f=1$ for all frames $f$ to keep the data distribution consistent.

This guidance schedule ensures the sequential alignment from the initial state, through the transition, to the final stage. We also utilize a Guidance Interval to enhance video consistency. For more details, please refer to Lines 226-252 and Table 1.

Weakness 3

Reviewer's Comment: “As mentioned in the limitation section in the supplementary, the method introduces additional hyper-parameters...”

Response: Our method indeed introduces two new hyperparameters: guidance schedule and guidance interval $\xi$. These parameters significantly enhance control over video generation. The paper contains a detailed analysis of each hyperparameter and provides recommendations for their optimal values.

Guidance Schedule:

• Dynamic T2V Task: A variety of schedules were tested, and the Negative Linear schedule shows best scene dynamics compared to other methods (see Table 4).
• Zero-Shot II2V Task: The Partial quadratic schedule ensures better consistency with reference images.

Guidance Interval $\xi$:

• Dynamic T2V Task: This parameter balances video dynamism and the connectedness of initial and final states. Our experiments show that $\xi=0.95$ yields optimal results (see Lines 412-419, Table 4).
• Zero-Shot I2V Task: It balances image similarity and amount of motion, with $\xi \in [0.9, 0.98]$ being preferred (see Lines 457-460, Figure 4, Table 5).
• Zero-Shot II2V Task: It ensures video quality and consistency, with $\xi=0.5$ being optimal based on our experiments (see Appendix A5, Lines 869-878, Table 8).

While these hyperparameters add complexity, they are essential for ensuring higher control and quality in video generation.

Question 1

Response: As shown in Section 3.4, State Guidance (Eq. 3) with Image Guidance (Eq. 5) enables reference images as state descriptors. In our Zero-shot II2V pipeline, this applies when references are available for both start and end frames, as per Eq. 7. For scenarios with only one reference image and prompts for each state, we adjust Eq. 7 as follows:

Only end frame:

$\hat{\epsilon_\theta}^f(z_t,\langle p_{is},p_{ts}, \langle p_{fs},i_{fs}\rangle\rangle)=(w+1)\tilde{\epsilon_\theta}^f(z_t,\langle i_{is},p_{ts}, \langle p_{fs},i_{fs}\rangle\rangle)-w\epsilon_\theta^f(z_t, \varnothing)$

$\tilde{\epsilon_\theta}^f(z_t,\langle p_{is},p_{ts},\langle p_{fs},i_{fs}\rangle\rangle)=\gamma_{is}^f\epsilon_\theta^f(z_t,p_{is})+\gamma_{ts}^f\epsilon_\theta^f(z_t, p_{ts})+\gamma_{fs}^f\bar{\epsilon_\theta}^f(z_t, p_{fs}, i_{fs})$

Only first frame:

$\hat{\epsilon_\theta}^f (z_t,\langle \langle p_{is}, i_{is} \rangle, p_{ts}, p_{fs}\rangle)=(w+1)\tilde{\epsilon_\theta}^f(z_t,\langle \langle p_{is},i_{is}\rangle, p_{ts}, p_{fs}\rangle)-w\epsilon_\theta^f(z_t, \varnothing)$

$\tilde{\epsilon_\theta}^f(z_t,\langle\langle p_{is}, i_{is}\rangle, p_{ts},p_{fs}\rangle)=\gamma_{is}^f\bar{\epsilon_\theta}^f(z_t, p_{is},i_{is})+\gamma_{ts}^f\epsilon_\theta^f(z_t,p_{ts})+\gamma_{fs}^f\epsilon_\theta^f(z_t,p_{fs})$

Question 2

Response: Our inference pipelines introduce an inductive bias that improves morph transform generation. We define a dynamic video scene as a trajectory with three states: initial state, transition state, and final state. The initial and final states are nearly static, resembling scenarios the T2V model is typically trained on. The guidance schedule ensures smooth transitions between these states, maintaining coherence and quality in morph transform.

Dear Reviewer EnJp,

We would like to supplement our previous response by addressing Weakness 1, Weakness 4, and Question 3. To provide a comprehensive answer, we conducted additional experiments using the CogVideoX-5B $1$ model with both short and detailed prompts (all experiments are conducted with frozen random state, DDIM sampler with 50 steps, for both with State Guidance we user Negative linear guidance schedule and $\xi = 0.95$). The detailed prompts were generated from the original short prompts and state prompts in our Dynamic T2V benchmark utilizing the official CogVideoX prompt enhancer available at: https://huggingface.co/spaces/THUDM/CogVideoX-5B-Space

Table A. Quantitative evaluation of CogVideoX on Dynamic T2V scenes bench.

In addition, we conducted a side-by-side user study comparing standard inference of CogVideoX and inference with State Guidance on short and detailed prompts. The results are summarized in Table B below:

Table B. Qualitative evaluation of CogVideoX on Dynamic T2V scenes bench.

Results analysis:

1. Our findings indicate that default CogVideoX-5B does encounter issues with generating dynamic video scenes on short prompts, which are elaborated on in our Weakness 1 response.
2. Table A confirms that utilizing enhanced prompts improves performance in Dynamic T2V generation.
3. Table A and Table B show that State Guidance improves dynamic video scene generation for both short and enhanced prompts.

Weakness 1

Reviewer's Comment: “The T2V limitation mentioned in L202 is not convincing enough…”

Response: We conducted additional experiments with CogVideoX that reveal that:

1. DiT-based Architectures: While DiT-based T2V architectures like CogVideoX exhibit superior spatial-temporal modeling and flexible text conditioning, our experiments indicate that the default CogVideoX still struggles with generating dynamic video scenes (see Table A). This issue likely stems from pre-trained models not being extensively trained on dynamic scenes since it is not common in training data. However, we show that incorporating State Guidance (SG) significantly enhances CogVideoX’s capability in dynamic video scene generation (see Tables A and B).

2. Detailed Prompts: Our results (Table A) show that using detailed prompts improves Dynamic T2V generation quality. It's important to note that enhancing prompts is orthogonal to applying State Guidance. Our findings (Tables A and B) demonstrate that combining enhanced prompts with State Guidance further advances dynamic video scene generation.

We hope these additional insights address your concerns effectively.

Weakness 4

Reviewer's Comment: “… there are strong models taking much more descriptive prompt as input for video generation…”

Response: We have addressed this by providing additional experimental results in Table A and Table B above. For further details, please refer to the Results Analysis and the response to Weakness 1.

Question 3

Reviewer's Comment: “Have the authors try a more detailed caption vs. the simple ones used in the paper? Will that lead to better motion?”

Response: Thank you for your question. In response, we conducted additional experiments and confirm that using more detailed prompts leads to better motion quality. For more details, please refer to the Results Analysis and the response to Weakness 1.

---

[1] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. Yang, Zhuoyi and Teng, Jiayan and Zheng, Wendi and Ding, Ming and Huang, Shiyu and Xu, Jiazheng and Yang, Yuanming and Hong, Wenyi and Zhang, Xiaohan and Feng, Guanyu and others. arXiv preprint arXiv:2408.06072

Dear Reviewer sBeJ,

Thank you for your valuable feedback. We would like to offer a brief response to your concerns regarding our submission and to initiate a discussion. Note that the Weakness 4 and Weakness 6 responses will be further supplemented by the results of the experiments we are currently working on.

Weakness 1

Reviewer's Comment: “In Sec. 3, the definition of diffusion models might be incorrect”

Response: Unfortunately, it is difficult to localize the incorrectness of the definition. Could you please specify this comment?

Weakness 2

Reviewer's Comment: “The idea of state guidance seems similar to the deforum-like technique used in the stable diffusion user community”

Response: State Guidance differs from Deforum in the following ways:

1. Models and Objectives:

• Deforum: Uses Image Diffusion models to animate single static images.
• State Guidance: Utilizes Video Diffusion models specifically designed for generating dynamic video scenes.

2. Generation Process:

• Deforum: Iteratively transforms frames by adding and removing noise with Text-to-Image models.
• State Guidance: Applies a concept guidance strength schedule over time to ensure smooth transitions between initial, transition, and final states.

Additionally, our method incorporates Image Guidance and combines it with State Guidance to enhance video quality and consistency (see Lines 100-107).

Weakness 3

Reviewer's Comment: “Entries in the proposed dynamic scenes benchmark consist of three states tailored for the proposed framework…”

Response: Our Dynamic Scenes Benchmark comprises 106 single prompts describing video scenes with significant changes, compatible with all standard Text-to-Video (T2V) models. The state prompt triplets are specifically designed for our State Guidance approach, utilizing pre-trained models and requiring structured input.

Regarding experimental consistency, as shown in Table 3, we evaluated standard pre-trained T2V models using their typical inference method and our State Guidance approach without altering their architectures or weights. State Guidance enables models to efficiently incorporate state prompt triplets, significantly enhancing their ability to generate dynamic video scenes.

Additionally, Table 3 includes results from two closed-source commercial T2V models and two models that use multiple prompts from large language models (LLMs), which also struggle with dynamic scene generation, confirming the reliability and relevance of our benchmark.

Weakness 4

Reviewer's Comment: “The paper does not mention how generated results were selected…”

Response: We generated all videos using a fixed random seed to ensure consistency, and preliminary tests showed negligible metric variation with different seeds. While reporting mean and standard deviation is ideal, the computational demands of diffusion-based video generation make it challenging and it is uncommon in the current literature.

To address your concern, we will include the mean, standard deviation, and success rates for key results in Table 3 for both LaVie and VideoCrafter2 with and without State Guidance. We hope this addition meets your expectations.

Weakness 5

Reviewer's Comment: “For II2V experiments, the paper does not compare with SEINE”

Response: Due to page limitations, we were initially unable to include this comparison. We recognize its importance and will incorporate it in the revised manuscript. Below is the comparison, where M stands for Metamorphosis, A for Animation, and O for Overall performance.

Weakness 6

Reviewer's Comment: “The scale of user study seems relatively limited and the design of user study seems flawed”

Response: Thank you for your feedback. We are revising the study to improve its robustness and will include the updated design in our revised submission next week.

Comment 1

Reviewer's Comment: “The paper coined a new term II2V, which is actually frame inbetweening/interpolation”

Response: We define II2V as generating transitions between two input reference frames (see Lines 49-50). This term aligns with related concepts like I2V and Dynamic T2V, ensuring consistency and clarity within our framework.

---

Please let us know if our answers meet your expectations or if there are other areas that need clarification. We look forward to your continued feedback. Thank you again and we hope to hear from you soon.

Dear Reviewer sBeJ,

Thank you for providing response to our brief response and accepting our responses to W2/W5. We would like to supplement our brief response with detailed responses to W1, W3, W4, and W6.

Weakness 1

Reviewer's Comment: “In Sec. 3, the definition of diffusion models might be incorrect…”

Response: If we understand you correctly, your comment is related to the connection between the Noise Conditional Score Network (NCSN [2]) and the Diffusion Model (DDPM [3]). The training goals of both methods can be mathematically linked. In particular, the denoising goal in DDPMs can be interpreted as a form of score matching under certain conditions.

Section 2.2 of [1] states that the NCSN [2] and DDPM [3] ($L_{simple}$) goals are equivariant and both are a weighted sum of denoising score matching goals. Therefore, the NCSN objective simplifies to the DDPM objective. Both objectives minimize the expected squared error between the true noise $\boldsymbol{\epsilon}$ and the model's noise prediction $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$, so they're mathematically equivalent.

---

[1] Song Y. et al. “Score-based generative modeling through stochastic differential equations” (ICLR’2021)

[2] Song Y., Ermon S. “Generative modeling by estimating gradients of the data distribution (NeurIPS 2019)

[3] Ho J., Jain A., Abbeel P. “Denoising diffusion probabilistic models” (NeurIPS 2020)

Weakness 3

Reviewer's Comment: “… What happens if the three states are concatenated with temporal adverbials or conjunctions?”

Response: We would like to clarify that the prompt triplet model does not introduce additional information beyond the original prompt. Instead, it explicitly separates the initial and final states from the original prompt, with the transition prompt either duplicating the original prompt or converting it into an action state. For instance, the original prompt "Empty glass fills with water" is transformed into the triplet: <"An empty glass", "A glass is filling with water", "A glass with water">.

For experimental rigor, we tested VideoCrafter2 with triplet prompts concatenated using " , then " (results in the table below). This performed worse than using the original prompt directly:

Weakness 4

Reviewer's Comment: “The paper does not mention how generated results were selected...”

Response:

1. Selection of generated results: All videos were generated with a fixed random seed for consistency. Preliminary tests showed minimal metric variation with different seeds.
2. Reporting mean+std: We evaluated the metrics from Table 3 for both LaVie and VideoCrafter2, with and without State Guidance, using five different random seeds. The results show low standard deviations, indicating robustness:

The non-overlapping value intervals further affirm the consistency of our results.

3. Reporting success rate of each generation: Unfortunately, computing the success rate of each generation requires user involvement. However, our quantitative TextSim and qualitative TA metrics provide equivalent information.

Weakness 6

Reviewer's Comment: “The scale of user study seems relatively limited and the design of user study seems flawed…”

Response: We have improved our user study methodology and conducted it on a larger set of users. Additionally, we have added the option "Equal" to each question to account for instances where users are unable to prefer one option over the other.

Updated Instructions:

• Question 1 (TA): "Which video better reflects the actions described in the text description?"
• Question 2 (MQ): "Which video is more dynamic (has more action and events, including simultaneous events)?"

For each side-by-side comparison, between 50 and 67 users participated, with each pair of videos assessed by at least 5 unique users. Below are the updated user study results:

Dear ICLR Review Committee,

As the author of this paper, I would like to express some concerns with the review process and the quality of the review comments.

Initially, the paper received ratings of 5, 5, and 5. After extensive discussion with Reviewer sBeJ and conducting additional experiments, Reviewer sBeJ appreciated the significance of our work and increased their rating to 8. It is important to note that this reviewer had the highest confidence rating of 5 among all reviewers. However, it appears that the Area Chair (AC) based the final rating predominantly on the opinions of Reviewer EnJp and Reviewer NpS9, who showed limited engagement during the review process and had lower confidence ratings of 3.

I would also like to highlight three specific points related to reviewers' comments during the discussion:

1. Definition of Diffusion Models:

   • AC noted a concern regarding the definition of diffusion models which was not updated in the revised manuscript. We wrote about this on December 3; however, at that point, the submission editing window was closed. Therefore, we were talking about updating the manuscript after acceptance.

2. Detailed Prompts and SOTA Models:

   • Reviewer EnJp expressed uncertainty about the limitations of SOTA models needing detailed prompts, arguing that extending the prompt is not difficult. It is important to note that by July 1, 2024, no open-source DiT for video generation used an alternative text conditioning mechanism to frame-wise cross attention and detailed prompts. Additionally, Reviewer EnJp did not provide other models, other than CogVideoX, to support their claim, rendering this concern less applicable.

3. Comparison with "Make Pixels Dance" (CVPR'24):

   • Reviewer NpS9 raised an issue about not referring to and comparing with the "Make Pixels Dance" paper. This comparison was, in fact, included in the original submission. During the rebuttal, we clarified this and directed the reviewer's attention to the relevant section in the appendix, providing additional explanations about the differences between the methods.