MAVIN: Multi-Action Video Generation with Diffusion Models via Transition Video Infilling

Thank you for your valuable and constructive feedback. We would like to respond to your concerns individually as follows:

Weaknesses

1. Despite being similar to video interpolation, they address different tasks and scenarios. The major difference in our transition infilling task is that the infilling span is long (16 frames), and its content is uncertain and ambiguous. Thus, it cannot be determined by calculation-based algorithms. Video interpolation algorithms focus on limited and unambiguous contents in between adjacent frames, and are not tasked to interpolate an entire video sequence with large motion gaps. In the AIGC era, this task is newly made possible by generative models. And compared to video prediction. The proposed task is bi-directional conditioned. The generated content should not only consider the preceding content but also how to smoothly transit into the following one. The additional constraint also makes it more difficult than causal prediction.
2. The technique itself is not a new one, but it stands as a purpose-driven solution tailored to address the unique challenges of our task. Although various existing methodologies could be adapted, we found the BFG approach particularly essential for this specific scenario. Our strategy of encoding and concatenating the two edge frames as a guidance signal is also a deliberate design choice that mitigates the lack of guidance encountered during unsupervised training (where no textual description of the target action is provided).
3. We borrow the idea and make adaptations to our transition-infilling task.
4. We recognize the limitations of our work. Resource constraints and the lack of suitable existing datasets have hindered our ability to scale up at this time. Nonetheless, we have taken the crucial first step of proposing the task and validating the feasibility of our approach. We believe that multi-action video generation will and needs to be paid more attention in the future. While pioneering this direction and proposing a perfect solution is impracticable, we hope such a bi-directional unsupervised training pipeline can be valued and spur further expansion on this topic.

Questions

1. "PDF" in line 215 refers to the probability density function.
2. Since the effect of GFM is rather strong, when it is applied, the gap of w/ and w/o BFG is mitigated. However, if GFM is not applied, removing BFG results in major performance degradation (please see the last two rows of the ablation results). In other words, adding BFG during training or not has big differences.
3. We understand the concern. For long durations, there will be problems as you described. However, the duration of the infilling target in the test data is not that long and the transition basically happens throughout the intermediate clip. Especially for the test-manual, which is hand-made to ensure so. Within the limited infilling duration (16 frames) if the tiger rises too rapidly (e.g. using only 4 frames) it will seem unnatural and also lead to a low CLIP-RS score. Otherwise, the model and gt are both trying to perform the same action through the entire target clip; a little out-sync in this case is totally normal and does not affect the evaluation too much. Baseline methods either perform too many unnecessary actions or generate irrelevant actions/contents, which can be captured by the gt-based metrics.

Thank you for your valuable and constructive feedback. We would like to respond to your concerns individually as follows:

Weaknesses

1. We recognize the limitations of our work. Resource constraints and the lack of suitable existing datasets have hindered our ability to scale up at this time. Nonetheless, we have taken the crucial first step of proposing the task and validating the feasibility of our approach. We believe that multi-action video generation will and needs to be paid more attention in the future. While pioneering this direction and proposing a perfect solution is impracticable, we hope such a bi-directional unsupervised training pipeline can be valued and spur further expansion on this topic.
2. We understand the concerns regarding the baseline comparison. However, making a totally fair comparison is rather impractical. As highlighted, SEINE and DynamiCrafter are large-scale, general-purpose, and data-heavy models, while MAVIN is designed to be lightweight, task-specific, and data-efficient. In such a case, further fine-tuning SEINE and DynamiCrafter on our animal-specific benchmark could indeed lead to a more unfair condition, because their models are already extensively pre-trained on vast amounts of data for transition tasks. If we further fine-tune them on the WHOLE set of our training data, the overwhelming pre-training and data superiority could make the methodology layer negligible. Instead, our current comparison neutralizes their model's generality aspect with the fact that our model is low-resource trained on less than 1-hour total video duration, whereas theirs is about 52,000 hours in total. Considering the scope differences, the low-resource task-specific setup makes the comparison relatively more fair. And from a more practical and realistic aspect, they did not open-source the training code but only inference scripts, nor do we have the resources to fine-tune a foundation model at that scale.
3. While these ideas themselves are not new in the field, they are purpose-driven and deliberately designed for the proposed task. They are necessary components that we explored aiming to make the approach work, rather than finding innovational improvement on existing methods.

Question

Most current interpolation and prediction models adopt the BERT-like approach, which works well for predicting short missing content. However, for our task, we need to generate entire segments of video while considering the content and coherence of both preceding and succeeding video segments. The BERT approach makes the model prone to relying on the nearest unmasked tokens to generate content, thus failing to train the model's ability to handle larger gaps without reference. In the prediction stage, when the model needs to independently generate 16 consecutive frames without reference, it becomes inefficient. This can be observed from the baseline model Seine's generation performance; once the number of infilling frames increases, its generated content cannot maintain consistency and relevance. In our approach, masking a consecutive subsequence of training data forces the model to acquire the ability to generate coherent long-range content, thereby making the content-ambiguous transition generation possible. We appreciate the suggestion for additional background and will ensure to include a more comprehensive discussion to better contextualize our method and highlight its novelty.

Thank you for your valuable and constructive feedback. We would like to respond to your concerns individually as follows:

Weakness

We recognize the limitations of our work. Resource constraints and the lack of suitable existing datasets have hindered our ability to scale up at this time. Nonetheless, we have taken the crucial first step of proposing the task and validating the feasibility of our approach. We believe that multi-action video generation will and needs to be paid more attention in the future. While pioneering this direction and proposing a perfect solution is impracticable, we hope such a bi-directional unsupervised training pipeline can be valued and spur further expansion on this topic.

Questions

1. The test-auto samples one frame from every four frames in the original video, resulting in the whole video spanning four seconds. The motion range of the video is therefore larger and more challenging. MAVIN achieves more noticeable performance superiority to baseline methods under this setup due to the increased complexity.
2. Eq5 and Eq6 are used for inference noise initialization. The blurred images in Fig1 correspond to the Eq6 output.
3. In fact, we had tried this method in early experiments. Not only did it fail to bring improvement, but it also resulted in a decline. Mixing two edge images featuring two different action states leads to a naive mixture that corrupts either image's signal, therefore as a noise initialization, it may confuse the model. Instead, we use either one of the edge images (the nearest) and adjust the residual of it. We found this approach to be working great. As for the comparison of keeping the low-frequency band vs the whole frequency band, we suggest referring to the FreeInit paper where detailed analyses and comparisons were given.
4. CLIP-RS can be used with any similarity-based metric, such as optical-flow-RS. However, currently, there is no particularly effective method to characterize motion smoothness effectively; it is mainly observed visually. The naturalness and smoothness difference between baseline modes and MAVIN, when assessed by eyes, was quite obvious. The baseline methods show noticeable abruptness and inconsistency. Please refer to the video examples in the supplementary material to see the comparison.

Thank you for your valuable and constructive feedback. We would like to respond to your concerns individually as follows:

1. We recognize the limitations of our work. Resource constraints and the lack of suitable existing datasets have hindered our ability to scale up at this time. Nonetheless, we have taken the crucial first step of proposing the task and validating the feasibility of our approach. We believe that multi-action video generation will and needs to be paid more attention in the future. While pioneering this direction and proposing a perfect solution is impracticable, we hope such a bi-directional unsupervised training pipeline can be valued and spur further expansion on this topic.

2. For the baselines mentioned, we appreciate you pointing out these relevant works. We didn't include them because our work was actually done before their release. We would like to include these baselines in the next updated version of our paper. Here we show a couple of quantitative results of these two models' ability in connecting action transitions. The videos are updated in the supplementary material.

   As can be seen from the result, they face the same fetal issues as Dynami-Crafter and SEINE have -- the generated video is rather a smooth scene transition than a natural action transition that obeys the physical law. This is because these methods only consider image conditioning, not video conditioning, thereby neglecting temporal coherence and naturalness. As far as we know, we are the first to implement the video-based transition generation. And we do hope our exploration and introductory verification can draw more attention to this topic.

I appreciate the authors' detailed replies that clarifies much confusion. I raise my score. I agree that video transition (scene transition) is another important and unexplored task in video completion. I also agree the novelty of this work.

Thank you for your valuable and constructive feedback. We would like to respond to your concerns individually as follows:

1. Multi-action generation is relatively understudied, and existing works all failed to consider this aspect. Our proposed method distinguishes itself by decomposing multi-action generation into a single-action generation and a connection-infilling task, with the latter being a novel task that has not been explored. We will state it more clearly in the revision.

2. The related works we reviewed include video generation, video interpolation, and scene transition. Despite their relevance, none of them address the same problem we are solving. For instance, video interpolation fails to infill consecutively long and uncertain video content, video prediction is not conditioned on bi-directional constraints, and scene transition only considers image borders, neglecting temporal coherence constraints in video transition. We will also improve the Introduction and Related Work sections to more clearly convey these core ideas.

3. The manual data ensures action transitions, considering at least two distinct actions. On the other hand, the automatic data, filtered via optical flow thresholds, selects significant movements but does not always ensure two distinct actions.

4. The supplementary material includes more examples. We will analyze and select some most representative failure examples to include together in the appendix in our next modified version to better illustrate our strengths and limitations. We appreciate this valuable and insightful suggestion.