Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes

We thank the reviewer for affirming our contributions: the pioneering exploration of the appearance-boundary complementarity between frames and events, clear motivation, and abundant experiments.

Regarding the raised comments, we summarize them into five aspects: difference from ABDA-Flow, clarification of the visualized images, comparison experiments with ABDA-Flow, generalization under other challenging conditions, and explanation of the performance degradation of some methods after visual enhancement.

Q1: Differences between the proposed method and ABDA-Flow [1].

A: It is emphasised that our method is different from ABDA-Flow in three typical aspects: task, motivation and framework.

• Regarding the task, ABDA-Flow focuses on spatial degradation in low-light scenes caused by the global exposure of frame cameras, which leads to texture loss in dark regions. In contrast, our method additionally addresses temporal degradation in high-speed scenes, where fixed exposure times cause motion blur in fast-moving objects.

• Regarding the motivation, while ABDA-Flow leverages the high dynamic range of event cameras to mitigate spatial degradation, it does not resolve temporal degradation. Our work explores the appearance-boundary complementarity between frames and events to handle both spatial degradation in low-light scenes and temporal degradation in high-speed scenes.

• Regarding the framework, ABDA-Flow employs a unidirectional transfer from the event to the frame domain to enhance spatial quality, leveraging the high dynamic range of events. In contrast, our proposed method exploits the appearance-boundary complementarity between frames and events and introduces a bidirectional complementary fusion strategy, effectively addressing both types of degradation.

Overall, the task with temporal and spatial degradation in high-speed and low-light scenes, the motivation with frame-event appearance-boundary complementarity , and the framework with a bidirectional complementary fusion strategy and diffusion-based module jointly constitute a significant innovation of our method.

Q2: Clarification of the visualized images in Fig. 6.

A: Thanks to the reviewer for pointing out this issue. As described in the supplementary materials, the HS-FEFD and LL-FEFD datasets used for generalization testing in Fig. 6 of our paper were collected by our frame-event dual-mode system and subsequently processed. Regarding the appearance of similar images in related studies, it's possible that the authors used the same data source as ours to construct their own datasets. We can confirm that the datasets used in our generalization experiments were collected by ourselves in real-world scenes. Further details about these datasets will be made public upon acceptance of the paper.

Q3: Additional comparison with ABDA-Flow [1].

A: We did not compare with ABDA-Flow before because the author did not open source the code. During the rebuttal period, we contacted the author by email to obtain the source code of ABDA-Flow and conducted additional comparative experiments. As shown in the table below, we can observe the following:

• ABDA-Flow performs well in low-light scenes but fails in high-speed scenes since the method is designed to focus on nighttime conditions.
• Our proposed method achieves state-of-the-art performance in both high-speed and low-light scenes.

Q4: Generalization of the proposed method for other challenging conditions.

A: It is worth noting that our method aims to address the optical flow estimation problem in high-speed, low-light scenes. Our main idea is to exploit the the complementarity between frames and events in appearance saturation and boundary completeness to obtain better optical flow in high-speed and low-light scenes.

Following the reviewer's comment, we conduct additional experiments on rainy, foggy, large-displacement, and overexposed scenes. For rainy and foggy scenes, we use the Weather-KITTI2015 (Rain-KITTI, Fog-KITTI) and Weather-GOF (Rain-GOF, Fog-GOF) introduced by Zhou et al. [2], with synthetic events generated using the v2e model [3]. For large displacements, we constructed the LD-DSEC dataset by sampling one frame every three frames and re-segmenting events from the original DSEC, using the method of Ce Liu et al. [4] to generate optical flow ground truth. For overexposed scenes, we built the OE-DSEC dataset by selecting frames with intense car headlights or streetlights from DSEC. Each dataset includes 800 randomly selected images for generalization testing. We additionally include CH$^2$DA-Flow [2], an unsupervised method for optical flow in adverse weather, as a comparison baseline.

As shown in the table below, we can observe the following:

• From the overall perspective, our proposed method performs well in large displacement and overexposed scenes, but perform not so well in the rainy and foggy scenes.
• In large-displacement and overexposed scenes, our method significantly outperforms others. The fixed temporal resolution and exposure time of frame cameras cause temporal discontinuity and texture loss in bright regions, while event cameras, with their high temporal resolution and dynamic range, capture continuous motion and preserve scene details under such challenging conditions.
• In rainy and foggy scenes, our proposed method performs poorly, since raindrops and fog not only blur the frame image but also introduce a large number of noise points in the event stream, making it difficult for our method to obtain good visual features.

In summary, our proposed method performs better under conditions where we can obtain good visual features through the appearance-boundary complementarity between frames and events. However, under conditions where both frame cameras and event cameras are limited, we will explore solutions such as domain adaptation in the future.

Q5: Explanation of the performance degradation of some methods after visual enhancement.

A: Actually, we mentioned the relevant explanation in lines 34-35 of the paper. The visual enhancement such as motion deblurring and low-light enhancement indeed improves the apparent visual effect of frame images. However, such improvement may have a negative effect on optical flow estimation since visual enhancement may destroy the pixel matching relationship between adjacent frames, resulting in incorrect optical flow results.

Reference

[1] Hanyu Zhou, et al. Exploring the Common Appearance-Boundary Adaptation for Nighttime Optical Flow. ICLR 2024.

[2] Hanyu Zhou, et al. Adverse Weather Optical Flow: Cumulative Homogeneous-Heterogeneous Adaptation. TPAMI 2024.

[3] Yuhuang Hu, et al. v2e: From video frames to realistic dvs events. CVPR 2021.

[4] Ce Liu, et al. Human-Assisted Motion Annotation. CVPR 2008.

Thank the reviewer for your positive feedback. We are glad that you increased your score and recommended acceptance. We will update all the comments in the revised version.

We thank the reviewer for affirming our contributions: the pioneering exploration of the appearance-boundary complementarity between frames and events, well-designed modules, and convincing analysis experiments.

Regarding the raised comments, we summarize them into six aspects: explanation of the usage of the Sobel operator, details of constructing the 4D cost volume, clarification of dataset construction and additional comparison experiments, additional quantitative generalization experiments, comparison of computational efficiency, and explanation of some notations.

Q1: Explanation of the usage of the Sobel operator.

A: Thank you for your detailed review comments. Actually we did not perform ablation experiments on the threshold of the Sobel operator because the Sobel operator was only used to obtain boundary maps in the analysis experiment in Fig. 3. In practice, we set the threshold of non-maximum suppression in the Sobel operator to 50 to get a good boundary map visualization result. If necessary, we will provide boundary map visualizations for different thresholds in the final version of the paper to validate our choices.

Q2: Details of constructing the 4D cost volume.

A: The method we use to construct the 4D cost volume is similar to RAFT [1]. The difference between the two lies in the input. RAFT uses two consecutive frames as input to calculate correlations. Our input $x_{fusion}$ is the visual feature obtained by fusing the features of two consecutive frames and the features of the event stream at the corresponding timestamps through Attention-ABF module. Then, we calculate correlations between the fusion visual features $x_{fusion}$ at two timestamps to obtain the 4D cost volume as the motion feature.

Q3: Clarification of dataset construction and additional experiments on the original DSEC [2] dataset.

A: Actually we mentioned the relevant explanation in lines 196-197 of the paper. The HS-DSEC is obtained by applying motion blur to daytime segments from the original DSEC dataset, and the LL-DSEC is composed of nighttime segments from the original DSEC dataset. Moreover, in order to verify the generalization of our proposed method, we conducted some additional experiments to compare the performance of our proposed method with other comparison methods in optical flow estimation on the original DSEC dataset. Specifically, we randomly separated 3888 images from the original DSEC dataset as the training set to train our proposed method and all comparison methods, and the remaining 432 images were used as the test set to test the optical flow estimation performance metrics of each method. As shown in the table below, our proposed method achieves the state-of-the-art performance on the original DSEC dataset for optical flow estimation.

Q4: Additional quantitative generalization experiments on HS-FEFD and LL-FEFD.

A: Following the reviewer's comment, we conduct additional quantitative generalization experiments on HS-FEFD and LL-FEFD for all methods. As shown in the table below, owing to the advantages of event camera to capture clear and complete boundaries in high-speed and low-light scenes, and the strong generalization and robustness of diffusion models, our proposed method achieves the best performance.

Q5: Comparison of inference time between the modules in our proposed method and other optical flow methods.

A: In order to comprehensively test the computational efficiency of the proposed method, we conducted some additional experiments on images from KITTI with a resolution of 1242×375 and DSEC with a resolution of 640×480 to test the number of parameters, memory consumption, and inference time of the proposed method and the comparison methods, using a single RTX 3090 GPU as the inference platform.

As shown in the table below, we can observe the following:

• The computational efficiency of each module in our proposed method is within a reasonable range.
• Resulting from the additional introduction of event modality and the nature of iterative denoising of diffusion models, although the overall metrics of our method are slightly higher than those of existing methods, it achieves a significant improvement in performance.

Q6: Explanation of the meaning of $t$ in the TVM-MCA module.

A: $t$ denotes the denoising time step, which acts as a controller, enabling the denoising module to perform adaptive denoising behavior at different denoising time steps. This is achieved by encoding the denoising time step $t$ into the time embedding $e_t$ and fusing it with motion/visual features $x_{cv}/x_{fusion}$ via a cross-attention mechanism, which is subsequently served as a conditioning signal for the denoising process.

Reference

[1] Zachary Teed, et al. Raft: Recurrent all-pairs field transforms for optical flow. ECCV 2020.

[2] Mathias Gehrig, et al. Dsec: A stereo event camera dataset for driving scenarios. RAL 2021.

[3] Shihao Jiang, et al. Learning to estimate hidden motions with global motion aggregation. ICCV 2021.

[4] Xiaoyu Shi, et al. Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation. CVPR 2023.

[5] Mathias Gehrig, et al. E-raft: Dense optical flow from event cameras. 3DV 2021.

[6] Mathias Gehrig, et al. Dense continuous-time optical flow from event cameras. TPAMI 2024.

[7] Ao Luo, et al. Flowdiffuser: Advancing optical flow estimation with diffusion models. CVPR 2024.

We thank the reviewer for affirming our contributions: the pioneering exploration of utilizing dual modal features to guide diffusion models for optical flow estimation, detailed ablation studies, and extensive experiments.

Regarding the raised comments, we summarize them into three aspects: implementation and computational efficiency of the method, generalization under other challenging conditions, and clarification of some notations.

Q1: Implementation and computational efficiency of our proposed method.

A: The framework seems complex, but is simply modularized into two components: Attention-ABF and MCIDD. The former performs appearance-boundary complementary fusion of frames and events through cross-attention mechanism and self-attention mechanism, and the latter adopts cross-attention mechanism to aggregate information from multiple conditions, including time steps, visual features, and motion features, which is subsequently used to guide the optical flow denoising process.

In order to compare the computational efficiency of these modules with different optical flow methods, we conduct some additional experiments on images from HS-DSEC and LL-DSEC with a resolution of 640×480 to test the number of parameters, memory consumption, and inference time of the methods, using a single RTX 3090 GPU as the inference platform.

As shown in the table below, we can observe the following:

• The computational efficiency of each module in our proposed method is within a reasonable range.
• Resulting from the additional introduction of event modality and the nature of iterative denoising of diffusion models, although the overall metrics of our method are slightly higher than those of existing methods, it achieves a significant improvement in performance.

Q2: Generalization of the proposed method for other challenging conditions.

A: It is mentioned that our method is designed to solve the problem for optical flow estimation in high-speed and low-light scenes. Our main idea is to exploit the the complementarity between frames and events in appearance saturation and boundary completeness to obtain better optical flow in high-speed and low-light scenes.

Following the reviewer's comment, we conduct additional experiments on rainy, foggy, large-displacement, and overexposed scenes. For rainy and foggy scenes, we use the Weather-KITTI2015 (Rain-KITTI, Fog-KITTI) and Weather-GOF (Rain-GOF, Fog-GOF) introduced by Zhou et al. [1], with synthetic events generated using the v2e model [2]. For large displacements, we constructed the LD-DSEC dataset by sampling one frame every three frames and re-segmenting events from the original DSEC, using the method of Ce Liu et al. [3] to generate optical flow ground truth. For overexposed scenes, we built the OE-DSEC dataset by selecting frames with intense car headlights or streetlights from DSEC. Each dataset includes 800 randomly selected images for generalization testing. We additionally include CH$^2$DA-Flow [1], an unsupervised method for optical flow in adverse weather, as a comparison baseline.

As shown in the table below, we can observe the following:

• From the overall perspective, our proposed method performs well in large displacement and overexposed scenes, but perform not so well in the rainy and foggy scenes.
• In large-displacement and overexposed scenes, our method significantly outperforms others. The fixed temporal resolution and exposure time of frame cameras cause temporal discontinuity and texture loss in bright regions, while event cameras, with their high temporal resolution and dynamic range, capture continuous motion and preserve scene details under such challenging conditions.
• In rainy and foggy scenes, our proposed method performs poorly, since raindrops and fog not only blur the frame image but also introduce a large number of noise points in the event stream, making it difficult for our method to obtain good visual features.

In summary, our proposed method performs better under conditions where we can obtain good visual features through the appearance-boundary complementarity between frames and events. However, under conditions where both frame cameras and event cameras are limited, we will explore solutions such as domain adaptation in the future.

Q3: Clarification of $K_v, V_v$ and $K_m, V_m$.

A: I apologize for not clarifying how we obtain $K_v, V_v$ and $K_m, V_m$. Specifically, we first flatten the visual feature $x_{fusion}$ and motion feature $x_{cv}$ into tokens, and consequently project them into $K_v, V_v$ and $K_m, V_m$ with the same dimension as $Q_v$ and $Q_m$ by learnable linear layers, respectively. Therefore, $K_v$ is actually not identical to $V_v$, and likewise $K_m$ to $V_m$.

Reference

[1] Hanyu Zhou, et al. Adverse Weather Optical Flow: Cumulative Homogeneous-Heterogeneous Adaptation. TPAMI 2024.

[2] Yuhuang Hu, et al. v2e: From video frames to realistic dvs events. CVPR 2021.

[3] Ce Liu, et al. Human-Assisted Motion Annotation. CVPR 2008.

[4] Shihao Jiang, et al. Learning to estimate hidden motions with global motion aggregation. ICCV 2021.

[5] Xiaoyu Shi, et al. Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation. CVPR 2023.

[6] Mathias Gehrig, et al. E-raft: Dense optical flow from event cameras. 3DV 2021.

[7] Mathias Gehrig, et al. Dense continuous-time optical flow from event cameras. TPAMI 2024.

[8] Ao Luo, et al. Flowdiffuser: Advancing optical flow estimation with diffusion models. CVPR 2024.

We thank the reviewer for affirming our contributions: the pioneering exploration of diffusion models for event-based optical flow, promising experimental results, clear and helpful graphical illustrations.

Regarding the raised comments, we summarize them into three aspects: the difference from FlowDiffuser, clarification of some concepts, and methodological details.

Q1:Differences between the proposed method and FlowDiffuser.

A: It is emphasized that our method is different from FlowDiffuser in three typical aspects: task, motivation and framework.

(1) Regarding the task, FlowDiffuser is designed for optical flow estimation in conventional scenes. In contrast, our proposed method explores the temporal and spatial degradation of frame images caused by motion blur in high-speed scenes and texture loss in low-light scenes, which in turn leads to inaccurate optical flow estimation.

(2) Regarding the motivation, FlowDiffuser introduces diffusion models with promising performance for optical flow estimation to obtain a more generalizable and robust model. In addition, we introduce event modality and exploit the appearance-boundary complementarity between frames and events to solve the temporal and spatial degradation of frame images in high-speed and low-light scenes.

(3) Regarding the framework, FlowDiffuser simply designs an optical flow iterative denoising decoder based on diffusion models. Differently, we propose Attention-ABF and MCIDD. The former adopts cross-attention module to focus on the appearance information of the frame modality and the boundary information of the event modality, respectively, and subsequently fuse them through the self-attention module to obtain better visual features. The latter integrates information from time steps, visual features, and motion features to guide the denoising process.

Overall, the task with optical flow estimation in high-speed and low-light scenes, the motivation with frame-event appearance-boundary complementarity, and the framework with diffusion models guided by fused frame and event features jointly constitute a significant innovation of our method.

Q2: Clarification of “appearance saturation”.

A: We apologize for causing your misunderstanding on the concept of appearance saturation.

In this paper, appearance saturation refers to the abundance of appearance texture information within a visual modality. It reflects the degree of spatial variation in pixel intensity caused by fine-grained textures, shading, and color details. Dense appearance saturation refers to the abundance of fine-grained textures, shading, and color details, while sparse appearance saturation is the opposite. Modalities with dense appearance saturation, such as frame images, preserve abundant structural and photometric cues, which are helpful for establishing correlations between two consecutive frames and further estimating optical flow. In practice, appearance saturation can be quantified by measuring the variance of local gradient magnitudes, emphasizing the presence of detailed visual patterns:

where $I$ denotes frame image or event frame constructed from original event stream.

In addition, we also provide the definition of boundary completeness. Boundary completeness denotes the continuity and integrity of object boundaries within a modality. It evaluates how well the modality captures clear, coherent, and complete boundary structures. Dense boundary completeness means that the visual modality has clear, continuous, and complete boundaries, while the boundaries of a visual modality with sparse boundary completeness may be blurred, interrupted, or even missing. For modalities with dense boundary completeness, such as events, the captured boundaries are clear and tightly connected in space, which plays an important role in obtaining optical flow with clear and complete boundaries. To quantitatively calculate boundary completeness, we first obtain the boundary map through the Sobel operator, and then calculate the ratio of the number of pixels of connected boundaries in the boundary map to all boundary pixels as the boundary completeness:

where $L_c,L_i$ denote the pixel length of connected boundaries and all boundaries, respectively. $C,N$ are the number of connected boundaries and the number of all boundaries, respectively.

Due to motion blur in high-speed scenes and insufficient illumination in low-light scenes, frame cameras with long imaging time and low dynamic range have dense appearance saturation but sparse boundary completeness, while event cameras with short imaging time and high dynamic range have dense boundary completeness but sparse appearance saturation. Therefore, we exploit the complementarity between frames and events in appearance saturation and boundary completeness to design a fusion module to obtain better visual features.

Q3: Explanation of the design of using cross-attention mechanism between time embedding and motion/visual features.

A: It is worth noting that we actually apply the cross-attention mechanism between time embedding $e_t$ and motion/visual features $x_{cv}/x_{fusion}$,

rather than between time embedding and two modality features. Rather than serving as a bridge between the two modalities, the time embedding acts as a controller, enabling the denoising module to perform adaptive denoising behavior at different time steps. Specifically, in the initial stage with more noise, the module should remove as much noise as possible, while in the later stage with less noise, it should consider preserving the optical flow details while removing noise. Based on this motivation, we use the cross-attention mechanism between time embedding and motion/visual features in order to enable the model to learn to adaptively extract features according to the denoising time step. The features are subsequently fused to guide the denoising process, thereby realizing the regulation of the denoising behavior by the diffusion time step.