VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis

We genuinely thank the reviewer for leaning towards acceptance of our work. Below, we address the concerns in more detail.

(W.1 & W.2) Table 2 lacks a setting of Baseline+TAR and classic metrics evaluation such as FVD.

We sincerely thank reviewer JuSf for their constructive feedback and have included the Baseline + TAR results and added FVD evaluation, as shown below. Employing TAR alone can induce more dynamics, leading to improved MT-Score. Meanwhile, it achieves a lower FVD score compared to the baseline. This highlights the importance of TAR in synthesizing faithful long dynamic videos.

(W.3) When comparing VSTAR with FreeNoise in a mult-prompt setting, noticeable flickering in FreeNoise exhibits the implementation is not aligned with the original paper. (Although this does not affect my recognition of the effectiveness of VSTAR in the mult-prompt setting.)

We appreciate the reviewer for their careful reading. We used the official implementation of FreeNoise, available here. We observed that it encounters challenges in the multi-prompt long-video setting, particularly when dynamic visual evolution is required. The flickering issue can also be observed from their provided examples, e.g., "A boy/man/old man in grey clothes running in the summer/autumn/winter".

(Q.1) Concurrent work FreeLong

We thank the reviewer for bringing this work to our attention. Indeed, FreeLong also notes a similar pattern in the temporal attention visualization. However, instead of directly regularizing temporal attention as VSTAR does, FreeLong performs transformations and high/low-pass filtering in the frequency domain. While this approach is insightful, it introduces challenges in determining appropriate parameters for the high/low-pass filters, making the process less intuitive. It is worth mentioning that VSTAR uses a single hyperparameter, i.e., the Gaussian standard deviation $\sigma$ in the regulation matrix, to control the regularization strength and thus the video dynamics, without the need for frequency domain transformations, being simple yet effective. Additionally, our VSTAR can also transfer the dynamics from a given reference video, compared to FreeLong. Nevertheless, it would be interesting to further experiment and test the complementarity of both approaches when FreeLong is open-sourced. We once again thank the reviewer for their constructive feedback and the valuable pointer.

Hi,

First, I check the G.5 on your anonymous website again. Flickering also happens in VSTAR. There may be some errors on the web. Therefore I download the gif from the website. Then I find the frames are not well ordered in .gif. Could you check it and directly upload the original three videos in G.5 as a .zip file (through the OpenReview system)?

Second, the official implementation of FreeNoise does not support the three prompts. In your implementation, the third prompt never happens, which is not aligned with the same demo shown by FreeNoise. Still, I will keep the positive score. However, if you are very confident about your implementation, you can upload the related code thus we can check it.

We greatly appreciate the reviewer for taking the time to provide such a thorough and detailed evaluation, as well as their overall positive assessment. Below, we address the individual concerns in detail:

(Q.1) Clarification on per-layer temporal attention analysis, i.e, resolution and location in UNet.

Following the common practice of attention visualization using mean averaging[1,2], we performed our visualization by averaging across all layers at different resolutions within the UNet.

(Q.2) "max" operation in Eq. (3)

Our aim of using the max operation is to provide an automatic scaling mechanism in response to the Softmax operation in the original attention calculation. Otherwise, the values of the designed regularization matrix might not match the original attention values well, making it less effective or causing it to be overly applied. Despite the maximum being scaled based on the original attention, the regularization strength can be flexibly controlled by adjusting the $\sigma$ to control the relative attention among frames (i.e., the shape of the regularization matrix), thus the video dynamics. As shown in Fig. 9, reducing $\sigma$ results in stronger regularization, leading to more dynamic content.

(Q.3) By looking at the results presented in Figure 8, it appears that the dynamics transfer also includes the style transfer. Intuitively, the attention map was used to regularize the interactions among frames to ensure that the evolving content was consistent with the real videos. With that being said, should we assume that these "dynamics" also model the video motions? If that is the case, the style transfer appears to be confusing here. Please also clarify what are the captured dynamics by using the default $\Delta_A$.

We thank the reviewer for this thoughtful observation. Based on our extensive experiments, we did not observe that the dynamics transfer includes a style transfer effect. To further illustrate this, we have provided more samples using different random seeds here. It can be seen that the scenes look quite different, and the third sample even contains trees and a lake.

From a technical point of view, the proposed regularization is applied to temporal attention, which only models the correlations between different frames and doesn't affect the style of the generated frames. Our designed $\Delta_A$ promotes stronger correlations with neighboring frames, while correlations decrease for more distant frames. This mimics the natural behavior of real videos with uniform visual progression, without special visual effects such as Flash Zoom or Hyperzoom.

(Q.4) Based on the discussions from Sec.5, there are limitations to applying the TAR to existing T2V models, depending on how the temporal attention was designed. Please offer more comments on Transformer-based architecture, which frequently uses positional encodings.

We thank the reviewer for the insightful comments. Indeed, transformer-based architectures often rely on positional encoding, which largely constrain their generalization ability for longer video synthesis. Similar observation has been identified for LLMs, as mentioned in L523. Furthermore, recent T2V models such as Open-Sora-Plan and CogVideoX even utilize 3D full attention without explicitly modeling the temporal dimension, rendering VSTAR inapplicable in this case. Nevertheless, VSTAR can be used to improve other T2V models, e.g., VideoCrafter and ModelScope. We once again thank reviewer dcFR for their constructive feedback and have extended our discussion in the revised manuscript.

[1] Hertz, Amir, et al. "Prompt-to-Prompt Image Editing with Cross-Attention Control." ICLR 2023

[2] Chefer, Hila, et al. "Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models." ACM Transactions on Graphics (TOG) 42.4 (2023): 1-10.

We thank the reviewer for the overall positive assessment. In what follows, we address the individual concerns in detail:

(W.1) Details on VSP: Do all frames share one interpolated text embedding or does each group share a different embedding?

To clarify, each frame has a unique textual embedding, obtained by interpolating between the textual embeddings of key frames. These embeddings are then utilized by the model via cross-attention mechanism to guide the synthesis process. To illustrate this, we have visualized the interpolation process through color coding in Fig. 2. We thank the reviewer for pointing this out, and we have updated Fig.2 and text in Sec. 3.4 for improved clarity and also included these details in the Sec. C of the Supplementary Material.

(W.2) Introducing a Toeplitz matrix to temporal attention could help to improve temporal dynamics. There is concern that hard temporal modification may break the original motion.

It is a very good question, which we also asked ourselves when we were designing the solution. We agree that, in theory, extreme regularization could potentially disrupt the original motion. Thus we carefully designed the regularization matrix as a Toeplitz matrix, with values along the off-diagonal direction following a Gaussian distribution to explicitly avoid abrupt change. As a result, in our experiments we did not observe evident abrupt changes in motion. Also, as shown in Fig. 9, users have the flexibility to control the strength of the regularization by adjusting the standard deviation ($\sigma$) of the Gaussian, which we have visualized in Fig. S.13. This allows users to customize the extent of visual changes based on their preferences. As mentioned in L485, we find that $\sigma_{64}=1$ strikes a good balance between dynamic changes and temporal coherency.

Meanwhile, we agree with the reviewer that sometimes the motion may look less natural, which is an inherited issue from the pretrained T2V models, as VSTAR is training-free. Nevertheless, we are optimistic that the community will open-source better and larger models trained on more data, further enhancing the motion quality. We thank reviewer 4ewj for the constructive feedback, and we have extended our limitation discussion in Sec. E of the Supplementary Material to address this.

(W.3) The proportion of the "less structured" temporal attention maps in video diffusion models, based on my personal experience only some cases and layers do not satisfy the band-matrix pattern.

We thank the reviewer for sharing their experiences. They are quite align with ours on some models, e.g., LaVie and AnimateDiff, exhibit more structured temporal attention for shorter videos (e.g., 16 frames). However, their attention maps degrade significantly for longer videos, such as those with 32 frames, as illustrated in Fig. 10. This phenomenon was commonly observed across layers and was easily noticeable when generating longer videos. We attribute this to their positional encoding of the frame index, which restricts their ability to generalize effectively for longer video synthesis beyond the trained number of frames, e.g., 16 frames, as discussed in Sec. 5.

(W.4) Other metrics like CHScore

We appreciate reviewer 4ewj's pointer and added the CHScore evaluation below. It can be seen that VSTAR achieves the best vision-language alignment and MTScore, while achieving competitive CHScore. For reference, we also computed the CHScore of dynamic videos, yielding 10.96, which is very close to the scores achieved by VSTAR. We have included some CHScore for real and generated videos here.

We hope our responses have addressed the reviewer’s concerns, and we are happy to provide further clarifications if there are any additional questions.

We sincerely thank the reviewer for the positive assessment of our work. In what follows, we address the individual concerns in detail:

(W.1) The side effect of TAR: may reduce motion amplitude.

We thank reviewer Apsy for the engaging question. In our experiments, we didn't observe TAR to reduce the motion amplitude. Instead, we saw that TAR enables synthesis of more dynamic content. For instance, from more visual examples provided in G.3 of the supplementary anonymous Github link, we can see that the corgi in VSTAR's result has a more dynamic walk. To further consolidate our observations, we conducted a quantitative evaluation using MTScore - a new metric specifically designed to measure metamorphic amplitude, the results can be found in Table 1. Using TAR improves MTScore from 0.401 to 0.448. We also added samples with and without using TAR as well as their individual MTScores here. Our approach demonstrates superior performance compared to other methods, achieving a higher MTScore, which reflects more pronounced and dynamic transformations over time. These results highlight VSTAR’s capability to generate long videos with dynamic content.

(W.2) Further ablation on TAR. Does VSP module is used in Fig.9? How do the TAR module and attention map change with the introduction of VSP module?

To clarify, the VSP module is indeed utilized in the results presented in Fig. 9. To further analyze the role of VSP, we have provided the attention map visualization with VSP only here. These visualizations demonstrate that using VSP alone does little to improve the temporal attention structure, and the video output shows minimal visual evolution, even when the prompts describe visual changes. In contrast, when combined with TAR, the temporal attention becomes more structured, and the results align with the desired content. This highlights the crucial role of TAR in generating long dynamic videos. We have incorporated these discussions in Sec. A.5 of the manuscript accordingly.

(W.3) Temporal consistency. Does the introduction of VSP lead to a decrease in temporal consistency?

We thank the reviewer for raising this insightful question. VSP's impact on temporal consistency can vary depending on the prompts used. When the prompts describe numerous abrupt or significant changes over a relatively short video, it may compromise temporal consistency. For instance, as shown in the example here, when the described subject remains the same (e.g., A Ferrari) while undergoing progressive seasonal changes, the result exhibits good consistency. However, when the prompt instruct the subject to change completely (e.g., switching from a Ferrari to a truck or limousine), as expected, we observe a degradation in temporal consistency. To generate more reasonable subprompts, as described in Sec. C, L1182 of the Supplementary Material, we explicitly instruct the LLMs to consider coherence between subprompts. Additionally, we interpolate between the textual embeddings of several key prompts, rather than generating prompts for each frame individually. This ensures smooth transitions in the textual embeddings, avoiding abrupt changes. Nevertheless, when the described visual changes are too extensive, it may create an impression of reduced consistency.

We hope our responses have addressed the reviewer’s concerns, and we are happy to provide further clarifications if there are any remaining questions.