Motion Graph Unleashed: A Novel Approach to Video Prediction

Question 1: More details about the setup of the datasets such as KITTI and Cityscapes that are not originally used for testing video prediction.

Answer 1: We follow the experiment setup from previous works which originated from Wu, Yue, et al. "Future video synthesis with object motion prediction". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. We use the data preprocessing code from this repo, https://github.com/hzwer/CVPR2023-DMVFN/blob/main/utils/, to generate the train/val data.

---

Question 2: The results on Cityscapes and KITTI datasets based on LPIPS, SSIM, PSNR are needed to be explained in detail in the evaluation methods either in the main text or Appendix.

Answer 2: Thank you for the suggestion. We will add the calculation details in the appendix.

Question 1: Do the authors see a future where such work might be useful for improving video generative models?

Answer 1: Yes. This work uses video prediction task as an example to validate the efficiency and effectiveness of motion graph as a comprehensive video motion representation. In the future, we will explore motion graph's potential as an efficient motion representation tool, developing advanced motion graph-based systems for long-term video generation.

---

Question 2: Can the authors explain what the weight components are for? It seems that the dynamic vectors can represent motion even without the weighted component. If it does represent something meaningful, can the authors show what they represent after training? For example, might it correspond to the change in depth?

Answer 2: After training, the graph construction process uses weights from the cosine similarity scores between pairs of image patches to determine their connectivity. Similarly, the weights output by the motion upsampler function as confidence scores for each predicted motion vector. This weighting strategy offers several advantages:

a) Graph Construction Robustness: In the motion graph, each image patch ideally connects to $k$ patches in the subsequent frame. If a patch is occluded in the next frame, selecting the top $k$ connections could lead to random, inaccurate connections. The weighting strategy helps identify and minimize these erroneous connections.

b) Selective Information Aggregation: During graph learning, this approach allows the model to focus on high-confidence correspondences and reduce the influence of ambiguous connections, enhancing the overall learning effectiveness.

c) Enhanced Frame Synthesis: During the synthesis of future frames, multiple source pixels may correspond to a single pixel in the future frame. The weighted approach allows for strategic aggregation of these source pixels, improving the quality of the resulting image.

---

Question 3: How does the motion graph approach compare to recent transformer-based video prediction methods in terms of efficiency and performance?

Answer 3: We appreciate the reviewer initiating this discussion. Both the motion graph approach and transformer-based video prediction methods (e.g. MaskViT) share the fundamental goal of predicting unknown relational information by modeling existing temporal correspondences between adjacent frames. However, significant differences in efficiency, performance, and adaptability set them apart:

a) Efficiency: Transformer-based methods create a complete graph for all feature patches between two adjacent frames, employing complex models that require a large number of parameters. This often leads to high computational demands, making these methods less suitable for high-resolution video processing. In contrast, the motion graph approach constructs a sparser graph, with edge weights determined by feature similarities from a CNN-based image encoder. This results in a more lightweight model. Furthermore, motion graphs leverage Graph Convolutional Networks (GCNs) composed of stacked linear layers, enabling efficient learning from the graph structure.

b) Performance: The efficiency differences make direct performance comparisons with transformer-based methods challenging. Currently, few transformer-based approaches are tested on the high-resolution datasets discussed in this manuscript, leaving performance comparisons inconclusive.

c) Adaptability to Different Video Resolutions: A major drawback of transformer-based methods is their lack of flexibility with varying input resolutions during testing. They struggle to process videos of different resolutions effectively. Conversely, our motion graph approach employs a CNN-based image encoder and motion upsampler, offering greater adaptability to various video resolutions. This adaptability enhances the suitability of the motion graph method for real-world applications.

---

Question 4: How sensitive is the method to the choices made in graph construction (e.g., number of neighbors, edge types)?

Answer 4: Our ablation study (detailed in Table 7) reveals a monotonic performance increase as the value of $k$ increases, although the rate of gain diminishes with larger $k$ values. This suggests that performance is more sensitive to changes in $k$ when $k$ is small, as it more closely resembles an optical flow-based method. However, as $k$ increases and motion information approaches saturation, the impact of $k$ lessens. Additionally, Table 12 in Appendix A.4 demonstrates that including both spatial and backward edges enhances model performance.

---

Question 5: How does the motion graph approach scale with longer video sequences? Is there a practical limit to the number of frames it can handle efficiently?

Answer 5: Longer video sequence can add more nodes and edges to the motion graph, which will result in an increase in the parameter number and computational overhead of motion graph learning. However, in terms of graph construction and feature extraction, the impact of video sequences length will be less, since we extract each image frame individually through an image encoder. Lastly, the practical limit to the number of frames a motion graph approach can handle efficiently is typically dependent on the hardware constraint. The available GPU memory significantly influences how many frames can be effectively managed.

Q1. Motivation and occlusion / out-of-view cases.

A1: As a non-generative model, our proposed video prediction system may face challenges with occlusions that require the generation of unseen objects. However, it excels in scenarios with scenarios involving occlusions of known objects, as showcased in the third column of Figure 6, where our method outperforms existing SOTA methods. Notably, in scenes with partial obstructions—such as the white wall behind the cyclist—our system adeptly employs multiple motion vectors per pixel to reconstruct occluded areas. This feature also supports precise management of object expansion due to perspective projection, as exemplified by the green car in Figure 6's first column.

Moreover, our approach is adept at managing objects exiting the camera’s view. By explicitly modeling the motion of image patches, the motion graph predicts when features are about to leave the scene. Thus, any image patches projected to move out of view are not included in the final prediction. This is demonstrated in the 1st, 3rd, and 4th columns of Fig 6, where objects like a cyclist and the front of a blue truck are shown as moving out of frame. Our method successfully and uniquely captures and represents these movements, unlike how the other methods evaluated.

---

Q2. Video prediction evaluation setting.

A2: Thank you for initiating this discussion. Our research on video prediction has identified key differences between systems designed for short-term and long-term predictions. Short-term systems typically use fewer frames to predict the immediate next few frames, while long-term systems are tasked with forecasting an extensive sequence of future frames. The design logic and objectives of these systems, thus, vary significantly. For a more detailed comparison, we have outlined these distinctions in the table below:

In this study, we emphasize our method's advanced motion modeling and significant reduction in computational costs, essential for short-term prediction of high-resolution videos. Our system is tailored for short-term video prediction, with evaluations conducted along this line of work. We plan to further exploit the motion graph's potential as an efficient motion representation tool and develop advanced, motion graph-based systems for long-term video generation in our next work.

---

Q3: Comparison with MMVP.

A3: We highlight in the manuscript three major differences to MMVP.

The advantages of our method are evidenced by its performance in various tests:

a) Our sparser motion representation allows the system to process videos with higher resolutions using fewer computational resources. For instance, Table 6 indicates that GPU consumption for our method is 52% of that required by MMVP.

b) Our graph-based motion prediction module aggregates relational information from motion embeddings more efficiently, resulting in a lighter and more effective model. As shown in Tables 4 and 6, our method requires only 21% of the model size of MMVP while improving the LPIPS metric by 22%.

c) Pixel-level image warping maintains critical appearance information from existing frames, significantly boosting detail recovery. Figure 5 demonstrates this by comparing the clarity of details like the horse’s facial features, the rider’s face, and the athlete’s feet, with our method outperforming others in detail retention.

---

Q4: Comparison with other motion-cues-guided video prediction methods.

A4: Although MotionRNN also utilizes motion vectors, there are several fundamental differences:

a) MotionRNN employs local $3\times3$ motion filters, which are limited in handling large-scale motion;

b) MotionRNN does not account for the interaction of motion across different spatial and temporal regions, a feature that our motion graph learning process emphasizes;

c) MotionRNN performs feature-level warping and our approach conducts pixel-level warping, providing finer detail.

These distinctions contribute to the lesser robustness of MotionRNN in handling high-resolution videos with complex motion scenarios, as detailed in Tab 3 of the manuscript. We have reviewed various motion-based video prediction methods in Sec 2. Moving forward, we plan to conduct a further literature review to expand and update this section accordingly.

---

Q5: Lack of component-wise ablation study.

A5: The components of motion upsampling, motion decoding, and image warping modules are each essential to the functionality of the system. So we have opted not to isolate each component for ablation studies. Instead, we focus the ablation analysis on the composition of motion features: please see details in Table 8 of the manuscript. Appendix A.4 also provides additional ablation studies, with Table 12 specifically exploring the effects of various interaction modules.

---

Q6. Presentation of this complex method.

A6: Thank you for this comment. To more effectively present our proposed method, for it to be more comprehensible and accessible, in the revised manuscript we worked on the clarity of descriptions and incorporated additional visual explanations.

Question 1: Which backbone are used to extract features during motion graph construction. Does the method robust to different backbones?

Answer 1: ResNet was used as the backbone of our image encoder to extract features during motion graph construction. In the manuscript submission, Figure 7 of Appendix A.2 illustrates the network architecture of the image encoder. Our experiments demonstrate the robustness of our design across different image encoder architectures. We selected a comparatively lightweight model to ensure both model compactness and system computational efficiency.

---

Question 2: Does the prediction quality decrease with respect to resolutions? is it possible to analyze prediction quality with large resolution settings.

Answer 2: To evaluate the model's performance on high-resolution images, we tested our method on the SJTU4K dataset, which has a resolution of $2160\times3840$. Due to the absence of publicly available data splits for direct comparison with existing state-of-the-art (SOTA) methods, we created our own training and validation splits, allocating 80% of the sequences to training and 20% to validation.

While the model's performance on the SJTU4K dataset surpasses its performance on KITTI, which has a much lower resolution, it does not match the performance on Cityscapes, which also has a smaller resolution than SJTU4K. Our findings suggest that factors such as motion complexity, frame rate, and scene complexity have a greater influence on prediction performance than frame resolution alone.