Bridge Frame and Event: Common Spatiotemporal Fusion for High-Dynamic Optical Flow

1. It would be nice if some visualization or numerical results could be presented to show the time-continuous optical flow estimation effects, not limited by the used dataset, for analysis. Some time-dense event-based optical flow estimation methods could be compared.

2. It would be nice if some frame-event fusion methods as discussed in the related work section, such as the cross-modal attention module and existing multimodal fusion methods, could be compared and analyzed in the experiments as well, which will better show the superiority of the proposed fusion solution.

3. If it is possible, a table can be added to compare the properties of existing event-based optical flow estimation datasets and your proposed one.

4. More time-dense-relevant results like inference time, latency, inference time of the proposed model could be discussed and analyzed.

• The method appears highly engineered; however, the language used to describe implementation details is vague. The paper lacks a detailed description of the method. The authors are encouraged to provide additional method details or introduce their approach mathematically in the main paper. For example, in Line 289, it is unclear how $p$ is obtained. Similarly, the reference to "$\mathbf{y}$ as the pre-classified boundary class label" could be clarified with more specific details. Additionally, any statistics on the collected dataset would be valuable.

• The compared state-of-the-art methods were re-trained (Line 428). However, in optical flow estimation, performance is highly dependent on the quantity of pre-training data. A model pre-trained on extensive datasets like FlyingThings, Sintel, KITTI-2015, and HD1K may achieve strong performance in high-dynamic scenes. If this is the case, the proposed method’s advantages may diminish. Additionally, the authors are encouraged to explore more recent unimodal methods beyond GMA, such as Flowformer, Flowformer++, and VideoFlow.

• In the DSEC benchmark, one test sequence is recorded at night. It would be beneficial for the method to be evaluated on the nighttime sequence within the DSEC benchmark to assess its performance.

1. The whole framework looks complex, including six losses. How to train the proposed method? I suggest that the authors can provide a detailed description.
2. Efficiency comparison of the proposed method, including runtime and model size.
3. The introduction about the proposed dataset is relatively short. The authors should add the details of this dataset.

1. An established method [1] is closely related to the study of this paper, please present the differences with it.

2. $L_{flow}^{pho}$ includes both self-supervised and supervised parts, which maintains the learning capability of frame-based and event-based optical flow estimators. Whereas $F^f$ is the optical flow from frames, it is necessary to explain how Eq(10) can be used for event-based optical flow.

3. Event-based correlations are computed in Section 3.3 for estimating continuous optical flow, and implementation details need to be clarified. a) Are correlations computed from the first event slice to the subsequent, or adjacent slices? b) Are multi-frame continuous optical flows starting from frame 1, or between adjacent slices? c) How many continuous optical flows are estimated by the GRU and how are they accumulated to obtain the final optical flow result for comparison? Does the aggregation process need to deal with the occlusion of invisible points? d) The performance of temporally continuous optical flow is only represented by the TEPE metric in the ablation experiment section and is not compared with existing methods such as BFlow[2]. In addition, the differences between the image event correlation fusion of this paper and BFlow need to be clearly stated.

4. The overall training process uses K-means clustering and EKF tracking as well as image gradients and Canndy contours several times. It is necessary to clarify whether these are required to ensure gradient backpropagation in model training.

5. Slow-DSEC and Fast-DSEC are obtained by adding various degrees of blur effect to DSEC. Event-KITTI is a synthetic version of the KITTI 2015 dataset with simulated events, but significant blurring can be observed in Table 5 of Supp, which is inconsistent with the original KITTI dataset. The implementation details of adding blur effect need to be clarified, whether based on multi-frame blending or on motion aggregation. Both KITTI and DSEC have publicly available leaderboards. At least submission on a real-captured event dataset, DSEC, is necessary.

6. Lack of explanation of training details, including hyperparameters, training dataset, etc. Do the compared methods take the same training setup as the method in this paper? The compared image-based optical flow estimation methods are outdated. Since this paper introduces the transformer for correlation fusion, at least some transformer-based new methods such as GMFlow[3] should be compared.

[1] Exploring the Common Appearance-Boundary Adaptation for Nighttime Optical Flow, ICLR 2024 [2] Dense continuous-time optical flow from event cameras, TPAMI 2024 [3] GMFlow: Learning Optical Flow via Global Matching, CVPR 2022