E-Motion: Future Motion Simulation via Event Sequence Diffusion

Q1: Event V.S. High-Speed Camera

There are basically three reasons for event data outperforming high-speed cameras:

(1) Data characteristics. Event cameras record only dynamic intensity changes and have extremely high-temporal resolution, which means they capture a wealth of motion information. Compared to RGB data from evenly spaced exposures and the often difficult-to-obtain optical flow data, event data has a natural advantage in the task of predicting future motion information. Moreover, due to the concise structure of data, events can be sensed in low latency, which is also a strength of event data.

(2) Measurement requirements. For capturing high-temporal resolution motion information with a high-speed camera, we usually need to enhance the lighting condition or highlight our target, since the exposure time of a high-speed camera is very short. Otherwise, the captured images will be blurry and dark, making the content difficult to discern.

(3) Cost. The acquisition and usage costs of high-speed cameras far exceed those of event cameras. Theoretically, the smallest time interval for data generation by an event camera is 1 $\mu s$, which corresponds to a maximum temporal resolution of 1e6 frames per second. In practice, many studies e.g.[1,2] have confirmed that event cameras can easily achieve nearly 1,000 frames while maintaining good semantic information. To achieve the same frame rate, the computational and storage costs for high-speed cameras are significantly higher than those for event cameras. Moreover, event cameras have a larger dynamic range compared to conventional cameras, allowing them to capture object motion effectively even under poor lighting conditions, as shown in Fig. 1 of the uploaded PDF file.

[1] Tulyakov, Stepan, et al. "Time lens: Event-based video frame interpolation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[2] Tulyakov, Stepan, et al. "Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Q2: Event-based Design

In fact, our proposed method takes full advantage of the characteristics of event data. Specifically, as detailed in Section 4.1 (Lines 153-165) of the main text, under the High-Temporal Resolution Guided Sampling section, we exploit the high temporal resolution and flexibility of event data by segmenting the temporal bin into multiple sections, which containing rich motion information from the events. During the reverse diffusion process, we replace pure Gaussian Noise with these high-temporal resolution voxels in each denoising phase. This strategy enables us to achieve a high-temporal resolution diffusion process by utilizing these detailed event representations.

Q3: Reinforcement Learning Incorporation

Reinforcement learning is a training strategy that stabilizes and enhances the results generated by diffusion. Diffusion models are trained with decoupling steps from a continuously linked probability flow. Since the model cannot accurately estimate the score function at each step, there must be accumulated errors during the reverse diffusion steps. RL-based methods can take the accumulated error into consideration since the reward is directly modeled on the final reconstructions. Furthermore, Figure 10 in our Appendix demonstrates the effectiveness of using reinforcement learning for motion alignment. Notably, as shown in Figure 10(b), even extensively pre-trained video diffusion models can yield unstable and flawed outcomes in challenging tasks like motion prediction. However, after applying motion alignment through RL, our model delivers more stable results (Figure 10(c)), highlighting the importance of incorporating RL-based motion alignment.

Q4: Experiment Settings

Yes, the “T” and "S" indeed represent the temporal and spatial layers. We have added relevant annotations to the table. We will carefully review the figures and tables in the main text and make corrections in subsequent versions. Thank you for pointing this out.

Dear Reviewer vmNW

Thanks for your time and effort in reviewing our manuscript and the favorable recommendation. In our previous response, we addressed your remaining concerns directly and comprehensively. We are looking forward to your further feedback on our responses.

Best regards,

The authors

Dear reviewer vmNW,

Thanks for your time and effort in reviewing our paper. In our previous response, we provided detailed and comprehensive explanations to resolve your concerns. We are looking forward to and sincerely appreciating your further feedback.

Best regards,

The authors

Dera reviewer,

your questions seem to have been addressed by the authors. Can you please comment to confirm?

Q1: Event-specific Design

The authors want to note that the proposed method indeed makes event-based designs. Specifically, during the high temporal resolution guided sampling stage in Section 4.1 of the main text, our method fully leverages the high temporal resolution and flexible sampling capabilities of event cameras. We divided the temporal bin of event data into multiple sections and achieved a high-temporal resolution diffusion process by prompting those high-temporal resolution event representations.

Moreover, due to the large and diverse training datasets and extensive training, those RGB-based modules (VAE and CLIP) have very strong generalization ability. Although it may not outperform some event-based designs on specific datasets, its generalization and robustness make it not fail in most scenes. This is also why we retained the original architecture and adjusted certain weights to adapt the SVD to event data. Besides, the authors also want to note that it's ineffective to only change some parts of a large generative model, since the diffusion U-Net is trained with the perception of original VAE, CLIP models generation distributions. We also have experimentally validated that after changing those modules. The experimental results are shown in the following table, where we feed features from different clip models (Event-trained or RGB-trained) to the SVD U-Net. Note that all CLIPs are fed with event voxels. Even further fine-tuning the SVD with plenty of data, the resulting diffusion model with Event-trained is still underperformed.

Table: Experimental results of SVD fed with different CLIP inputs, where both CLIP models are fairly perceived with event data.

Moreover, in the future, the authors will also try to design and adapt more event-specific designs with the diffusion model, with large datasets and an extensive training process.

Q2: RGB-Domain Metrics & Raw Data Metrics

That's a good question for generative models. Here the authors merge those two questions and answer them together. The conclusion goes first that metrics on the raw space are inappropriate for measuring and training generative models.

The authors want to first note that perceptual metrics (or the metrics operated in low-dimensional data manifold) are essential for the generative model. In other words, the measuring and training of the generative model especially for the diffusion model must (or to be more effectively) be carried out on the data distribution manifold (or we can say in the perceptual space) [1]. The underlying reason is quite simple usually the target data distribution is not continuous on the raw space. If we utilize a metric such as MSE (as mentioned in the third question), it will make the generated data approach the GT data in the raw space. However, in the raw space, the data distributed near GT samples, e.g., blurring or noised versions of GT samples are not in the original GT data distribution. The target data distribution contains nearly all clear and meaningful samples. Thus, considering that if we can find the manifold of data, the nearby samples of a target sample must be clear event data showing the content with similar semantic meaning. Then, we should use perceptual metrics, which can act as a measurement tool for semantic and high-level meaning similarity.

Moreover, just as we discussed the aforementioned question, the current RGB-based model has a strong generalization ability, which ensures there with less probability to be failure on the perception in various scenarios. Thus, the authors adopt the aforementioned FVD, FID, and LPIPS as our measure metrics.

Finally, to address your concern, we also show the quantitative results for MSE and PSNR metrics in the table below. Our method also outperforms SOTA methods in terms of pixel-level metrics in the raw space.

Table: Quantitative comparison between SOTA methods.

[1]Song, Yang, et al. "Consistency models." arXiv preprint arXiv:2303.01469 (2023).

Dear Reviewer AE3a

Thanks for your time and effort in reviewing our manuscript. In our previous response, we addressed your concerns directly and comprehensively. We very much look forward to your further feedback on our responses. Let us discuss.

Best regards,

The authors

Q1: Additional Analysis w.r.t. Baselines

There are indeed some conflicts between different metrics, e.g., FID, IOU, and FVD, because only FVD can comprehensively evaluate the both spatial and temporal distribution alignment between the generated samples and GTs. FID and IOU can only measure the per-frame (spatial) distributions, which neglects the spatial-temporal consistency. However, through showing those metrics, we want to illustrate a comprehensive evaluation of different methods. To further resolve your concern, we draw a sample metric distribution graph, comparing the proposed method with SimVP, which performs best in terms of aIoU. The results are shown in Fig. 5(a) and Fig. 5(b) of the uploaded PDF file, It can be seen that our method overall outperforms SimVP, but SimVP achieves excessively high scores on certain samples, such as the blank scene shown in Fig. 5(c).

As shown in Lines 187-190 in the main text, we utilize the FVD and SSIM as metrics to reward the motion alignment reinforcement learning process. Thus, it's plausible that the trained metrics are increased. Moreover, as aforementioned, the FVD is the most principle metric for evaluating the generation results. Further, we actually have experimentally tried to utilize a mixture of all metrics to model the reconstruction reward. However, the performance results are much worse than the method we ultimately used, as shown in the following table. It may be due to that sometimes the optimization of pixel-level metrics of MSE and PSNR may be contradictory to perceptual metrics. Thus, such a mixture of metrics makes the objective hard to optimize. Meanwhile, due to the target data distribution, it's more plausible to optimize the process on the data manifold (perceptual space) than the raw space. We also refer the reviewer to the second reply for the Q2 of reviewer AE3a.

Table: Ablation Study of reward metrics

Q2: Related Work

Thanks for indicating such wonderful work. The authors will definitely include more discussions with related works in the main body of the paper in the final version. Moreover, we write some discussions as below shown:

"In recent years, the development of multimodal diffusion technology has advanced rapidly. Researchers are dedicated to applying the powerful generative capabilities of diffusion to different modalities with unique advantages, such as optical flow and depth. Saxena et al. [A1] were the first to apply diffusion models to optical flow and depth estimation. For the characteristics of training data, they introduced infilling, step-rolling, and L1 loss during training to mitigate distribution shifts between training and inference. To address the lack of ground truth in datasets, they also used a large amount of synthetic data for self-supervised pretraining, enabling the diffusion model to acquire reliable knowledge. Chen et al. [A2] utilized the motion information embedded in control signals such as edges and depth maps to achieve more precise control over the text-to-video (T2V) process. They used pixel residuals and optical flow to extract motion-prior information to ensure continuity in video generation. Additionally, they proposed a first-frame generator to integrate semantic information from text and images. Unlike them, we focus on exploring the rich motion information contained in event data and use it to conditionally achieve more precise control over the generation of future motions. Furthermore, we also investigate the significant role of reinforcement learning in video diffusion and the task of motion estimation."

[A1] The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation, NeurIPS 2023.

[A2] Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models, 2023.

Dear Reviewer Yrkm

Thanks for your time and effort in reviewing our manuscript and the favorable recommendation. In our previous response, we addressed your remaining concerns directly and comprehensively. We are looking forward to your further feedback on our responses.

Best regards,

The authors

Dera reviewer,

your questions seem to have been addressed by the authors. Can you please comment to confirm?

Dear reviewer Yrkm,

Thanks for your time and effort in reviewing our paper. In our previous response, we provided detailed and comprehensive explanations to resolve your concerns. We are looking forward to and sincerely appreciating your further feedback.

Best regards,

The authors

Q1: Performance in Challenging Visibility

Benefiting from the unique characteristics of event cameras, our event-based video diffusion framework can handle future motion estimation issues to some extent. To further address your concern about algorithm performance on challenging visibility scenes, we conducted experiments across various scenarios, as shown in Fig.1 of the uploaded PDF file. Specifically, we first illustrate the poor exposure scene in which a car is passing through a tunnel (Fig.1(a)). The proposed method clearly predicts the car's contour, whereas the contour is difficult to discern from the RGB data and even some frames of the GT event data.

Furthermore, for the object occlusion, the experimental results shown in Fig.1(b) illustrate a scenario where a person is passing through an occluding object. Our method successfully estimates the future motion of the object despite the occlusion. Similarly, Fig.1(c) demonstrates that when a bicycle enters an occlusion, our method provides a more accurate prediction of the bicycle's motion after it enters the occlusion.

Q2: Generalization Ability on Other Datasets

We further validate the generalization ability of the proposed algorithm on other datasets, i.e., CRSOT[1], VisEvent([54] in paper), and hs-ergb[2]. The following table presents the test results, the consistent performance of our method indicates strong generalization. Fig.1 and Fig.2 in the uploaded PDF demonstrate that our method outperforms existing methods across various scenarios from these datasets.

[1] Zhu Y, Wang X, Li C, et al. Crsot: Cross-resolution object tracking using unaligned frame and event cameras[J]. arXiv preprint arXiv:2401.02826, 2024.

[2] Tulyakov, Stepan, et al. "Time lens: Event-based video frame interpolation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

Q3: Limitation & Failure Cases

Our method is relatively limited in the following scenarios:

(1) Complex background scenarios. Event cameras may capture incomplete textures in certain situations, leading to poorer prediction performance, as shown in Fig. 3(a) of the uploaded PDF. Complex backgrounds can reduce the clarity of the target object, resulting in worse outcomes, especially in cases where the camera lens is shaking.

(2) Heavily overlapped object scenarios. When objects overlap, their motion becomes quite complex, and due to the edge-focused characteristics of event cameras, understanding such motion is challenging, as shown in Fig.3(b) and Fig.3(c). When people overlap, their footsteps often become chaotic, leading to less accurate predictions.

However, we also note that those scenes (extreme complexity or occlusion) are also very challenging for traditional RGB-based vision. We expect that in future research, a diffusion model that perceives different modalities may help to address those problems.

Q4: Computational Resource & Scalability

The following table compares the computational resources of our method with SOTA methods. The powerful generative capability and high fidelity of diffusion models lead to the cost of substantial computational resource consumption. As shown in the following table, our parameter count and FLOPs significantly exceed those of traditional models. However, we believe this trade-off is necessary because of the powerful learning capability of large models in the real world. Taking the future motion estimation task as an example, our diffusion-based method significantly surpasses traditional methods in understanding and learning motion.

For a salable model size for different inference environments, there indeed are many works [1] indicating that the diffusion model can be applied to quantization or other acceleration techniques for speeding up the inference process. The authors will also add further illustrations in the final version.

[1]So, Junhyuk, et al. "Temporal dynamic quantization for diffusion models." Advances in Neural Information Processing Systems 36 (2023).

Q5: Different demographic groups & Potential misuse

Since the event sensors have a higher dynamic range and without color information, the authors believe it has better equality for different skin color races. Moreover, because it contains less texture information, the individual's privacy can be better protected. We will add further discussions in the final version.

Q6: Regulatory and Compliance Issues

For applying the proposed algorithm on the robotic or automotive platform, such a system must have an event sensor first. Moreover, sufficient computational resources are also necessary for network running. We will add further discussion in the final version of the paper. Thanks for the advice.

Q7: Source Code

In the supplementary materials file provided, we include the hyperparameters and core components required for training the network. We will also make the code open-source in the future to facilitate replication of our methods by other researchers.

Dear Reviewer 8wZD

Thanks for your time and effort in reviewing our manuscript. In our previous response, we addressed your concerns directly and comprehensively. We very much look forward to your further feedback on our responses. Let us discuss.

Best regards,

The authors