Event-Driven Dynamic Scene Depth Completion

Minor Weakness1:

Although our method assumes reasonably synchronized input, it is explicitly designed to handle the challenges of imperfect RGB-D alignment in high-speed or dynamic scenes. In such cases, event data which is naturally asynchronous and high-temporal-resolution, plays a key role in correcting spatial misalignments and compensating for motion blur or sparse/degraded depth near dynamic objects.

To this end, we introduce event-guided deformable convolution, where the event stream dynamically generates the offset fields and modulation weights. This allows the model to adaptively sample features from more accurate spatial positions, even in the presence of RGB-D misalignment. In other words, instead of requiring perfectly aligned input, our method leverages the asynchronous nature of events to mitigate such issues.

We will add a discussion of this point in the revision, to clarify that our approach is particularly beneficial in non-ideal, high-motion scenarios, and offering future directions such as noise-aware offset estimation or sensor calibration refinement for further robustness. Thank you for the insightful suggestion.

Minor Weakness2:

We would like to emphasize that the EventDC benchmark already includes a diverse set of environmental conditions to facilitate comprehensive evaluation. Specifically:

1. Scene diversity: Our dataset covers both indoor and outdoor scenarios, ranging from confined corridors to large open areas.

2. Lighting variation: We include sequences captured in daytime, nighttime, and low-light conditions, making the benchmark suitable for evaluating models under challenging illumination.

3. Depth range: The effective depth spans from approximately 0.1 meters to 100 meters, thus covering both close-range tasks (e.g., indoor navigation or robotic grasping) and long-range perception (e.g., outdoor driving or surveillance).

Question1:

Yes, we will release the proposed datasets upon paper acceptance. In addition, we are glad to share the corresponding preprocessing tools to further support and benefit the research community.

Question2:

We agree that discussing failure cases can provide a more complete understanding of our method's limitations. EventDC, while effective in high-motion scenarios, may underperform in the following typical cases:

1. Extremely low-motion or static regions: Since event data is inherently triggered by intensity changes, regions with little or no motion (e.g., static background or slowly moving objects) generate very few events. This limits the contribution of event-guided modules in such areas, making the model rely primarily on RGB and sparse depth.

2. Repetitive or low-texture areas: In scenes with repetitive patterns or low texture (e.g., blank walls or road surfaces), events may be ambiguous or noisy, leading to inaccurate offset estimation in deformable convolutions.

3. Sensor artifacts or miscalibration: Although our method is designed to handle RGB-D misalignment, large-scale calibration errors or noisy depth input may still affect performance, especially in complex outdoor scenes.

Thank you again for these valuable comments. We will carefully consider incorporating these discussions in the revision.

To Weakness1 and Question1:

The rationale behind the current layout is as follows:

1. Depth branch (S) as the main stream: In depth completion tasks, the sparse depth input is typically treated as the primary modality, with RGB (I) and Event (E) streams serving as auxiliary sources. Therefore, we place the S branch at the center of the diagram to emphasize its dominant role in predicting the dense depth map.

2. Symmetry with the decoder: The depth features extracted from the S branch are progressively refined through the decoding path. To clearly illustrate this symmetric encoder-decoder structure, we align the S branch horizontally with the decoder layers.

3. Cross-modal guidance via EMA modules: The EMA modules fuse features from the I and E branches to enhance the S branch at multiple stages. This hierarchical guidance flows into the next layer of the S branch. Maintaining the current top-down layout (I at top, S in center, E at bottom) allows us to clearly visualize this progressive refinement process.

According the reviewer's suggestion, we do explore alternative layouts, including swapping the positions of the S and E branches. However, we find that these configurations introduce more visual clutter, particularly due to increased crossing of connection lines, which makes the figure harder to follow. After comparison, we determine that the current configuration offers better visual balance and a more intuitive interpretation of the network architecture.

Nevertheless, we agree that Figure 2 can benefit from improved clarity and thank the reviewer for this suggestion. We will revise the figure with a cleaner layout and clearer labeling to enhance readability and ensure that the design intentions are more easily understood.

To Weakness2 and Question2:

We will adopt Δ𝑝 as the unified notation for the offset throughout the paper. This would help avoid confusion and improve the overall readability of the mathematical formulations.

To Weakness3, Question3, and Limitation1:

As shown in Equation (1), the weight term of the deformable convolution in DCNv2 is represented as 𝑤·Δ𝑚, where 𝑤 denotes the learnable convolution kernel and Δ𝑚 is the modulation scalar that adjusts the importance of each sampled position in the offset domain. The only difference in the deformable convolution weights between the encoder and decoder lies in the treatment of Δ𝑚:

In the encoder, Δ𝑚 is set as an identity matrix and does not participate in learning;

In the decoder, Δ𝑚 is adaptively learned based on event information.

The reasons for such design are as follows:

As illustrated in Fig. 2, the EMA modules in the encoder primarily adjust the pixel distributions of RGB and depth features individually using event information. They further align the RGB-D features under the constraint of Equation (6), which helps mitigate misalignment caused by fast ego-motion. In other words, EMA focuses on learning the offset. Furthermore, 𝑤 is automatically learned by the network and Δ𝑚 is fixed as a non-learnable identity matrix.

The LDF of the decoder pays more attention to depth recovery around dynamic object regions after EMA-based alignment. As a result, it not only learns the offsets near moving objects but also aims to learn the contribution of each position through Δ𝑚. This helps regulate the weighted summation process and promotes more accurate depth estimation around dynamic areas.

At the code level, making Δ𝑚 learnable in the encoder is straightforward to implement. As shown in Table R2, two sets of comparison experiments are conducted based on the reviewer's suggestion. The results indicate that a learnable Δ𝑚 in the encoder has little impact on performance. We thank the reviewer for this valuable suggestion and will clarify this point in the final paper.

Table R2: RMSE Results on EventDC-Real.

To Weakness4, Question4, and Limitation2:

1. Focusing on the EventDC-Real dataset (i.e., Fig. 4):

The event stream reveals that dynamic regions include not only the truck but also background elements such as shaking trees. Depth unreliability often occurs both within these dynamic areas and at their boundaries with the background. As shown in Fig. 4, our proposed method gives better results in event-indicated areas and their transitions to the background. We will clarify that this outcome aligns well with the main objective of our work. Moreover, we will follow the reviewer’s suggestion to include more representative visualization results on EventDC-Real to further highlight improvements in dynamic regions.

2. Considering the full range of datasets (i.e., Figs. 4, 5, 6, and 7(b)):

The dynamic car in Fig. 5, the moving person and motorcycle in Fig. 6, and the person, bicycle, and car in Fig. 7(b) all demonstrate that our method consistently performs better around dynamic objects. These results strongly support that our method aligns well with the main goal of enhancing depth estimation in dynamic scenes.

To Weakness5, Question5, and Limitation3:

As shown in Table 6, we ablate our EventDC by comparing its performance with (EventDC-iv) and without (EventDC-ii) the event stream. The corresponding RMSEs are 638.3 mm and 685.9 mm, respectively. This indicates a reduction of 47.6 mm when incorporating the event stream. The result demonstrates the effectiveness of leveraging event information.

Following the reviewer's suggestion, we construct EventDC-x by removing only the event stream from EventDC-ix, while retaining both the EMA and LDF modules with deformable convolutions. As reported in Table R3, excluding the event stream increases the RMSE and MAE by 15.2 mm and 7.5 mm, respectively. This further confirms the contribution of the event stream in enhancing depth estimation accuracy.

Table R3: More ablations on EventDC-Real.

We thank the reviewer for this suggestion, and will include the ablation study in our final paper.

Thanks to the author for the reply. Most of my concerns have been resolved. I would like to raise the score.

To Weakness4:

Indeed, in real-world dynamic scenes with fast and complex motion, both RGB frames and sparse depth may become unreliable due to motion blur, temporal misalignment, or sensor noise. This is precisely the motivation behind leveraging event streams, which provide asynchronous, and high-temporal-resolution measurements that are inherently robust to fast motion and extreme lighting conditions. Event data captures pixel-level brightness changes at microsecond latency, making it resilient to motion blur and misalignment that commonly affect RGB and depth sensors.

In our framework, events are used to guide the spatial sampling of features through event-driven deformable convolutions. The offsets and modulation weights in these convolutions are derived from event features. For example, as illustrated in Figure 2, the event-centric EMA module adjusts the pixel distributions of RGB and depth features using event information. This enables more precise RGB-D alignment (see Figure 7(a)). The design remains effective even when RGB and depth inputs become unreliable. This is due to the microsecond-level temporal resolution and high motion sensitivity of event streams in dynamic environments.

Although RGB and sparse depth provide appearance and geometric priors, events offer fine-grained motion cues that help localize trustworthy structures and correct ambiguous or missing depth information in challenging scenarios. This synergy enables our model to better recover complete and reliable depth maps under conditions where traditional sensors struggle.

Finally, we would like to further clarify our current focus:

Our work focuses on highly dynamic scenes where RGB-D misalignment and local depth degradation commonly occur near moving objects or during fast camera motion. Although RGB and sparse depth are generally informative in most parts of the scene, we explicitly target regions where they become less reliable due to motion blur or latency mismatch. Thus, the proposed setting does not assume that both RGB and sparse depth are globally unreliable. Instead, it accounts for their reliability being spatially and temporally variant, which is common in real-world dynamic scenarios. We thank the reviewer for this insightful question, and will add the discussion into the final paper.

To Weakness5:

Thank you for raising this important concern. It is true that DCN is a widely used module. However, the contribution of this work does not lie in merely using DCN. Instead, it lies in how DCN is adapted and leveraged specifically for event-guided depth completion in highly dynamic scenes.

First, both the EMA and LDF modules are carefully designed to tackle the unique challenges posed by RGB-D misalignment and unreliable depth around moving objects in dynamic environments. In particular:

(a) The EMA module uses event features to dynamically generate DCN offsets and weights. This enables spatially adaptive feature alignment across RGB and depth inputs. To the best of our knowledge, this is one of the first methods proposed to use events to guide deformable alignment for depth completion in dynamic scenes.

(b) The LDF module further refines depth estimations around moving objects by learning motion-aware masks from events, which is crucial in scenes with locally degraded or missing depth.

Second, our integration with event data is notably different although DCN is a standard building block. This includes our design of the supervision and training strategy. For example, how the offset is driven by event features. In contrast to conventional DCN usage, which typically operates in a generic spatial context, our formulation is event-centric. It uses the microsecond-level temporal cues provided by event streams.

Finally, the novelty of our work is not limited to architecture, but also includes the problem setting, i.e., event-guided depth completion in dynamic RGB-D scenes, and the benchmark contribution (EventDC dataset), which enables a more rigorous study of this underexplored problem.

We hope this explanation clarifies the novelty and motivation of our design. We will revise the manuscript to better highlight these aspects. Thank you again for the constructive feedback!

To Weakness6:

Thank you for raising this important concern. We would like to clarify that our comparison setting is fair and rigorous. All compared methods fall under the category of depth completion, which take both color images and sparse depth as inputs, not image alone. Similarly, our EventDC method also takes color images and sparse depth as inputs, with the addition of event streams.

Our EventDC benchmark comprises three distinct subsets, i.e., EventDC-Real: Real-world RGB-D-event recordings without any synthetic motion blur; EventDC-SemiSyn and EventDC-FullSyn: Semi-synthetic and fully synthetic datasets with simulated motion blur to emulate high-dynamic scenes.

This setup ensures that our evaluation covers a wide spectrum of motion conditions, from real-world clean input to extreme motion-degraded cases. In particular, EventDC-Real does not contain any artificially induced blur, and yet our method still outperforms image-based baselines. This provides strong evidence that our model is effective even without relying on synthetic motion artifacts.

For EventDC-SemiSyn and EventDC-FullSyn, motion blur is intentionally added to simulate fast motion scenarios. These scenarios are the core focus of our work. The goal is to benchmark the robustness of RGB-D and event-based methods under such conditions. This is not intended to bias the results against image-based methods, but to fairly reflect real-world challenges where event streams offer the greatest benefit.

Furthermore, all methods are evaluated on the exact same inputs to ensure fairness in the comparison. The advantage of our model under motion blur conditions arises from its ability to leverage microsecond-resolution event data, which is inherently lacking in image-based models.

Additionally, to further support the fairness of our evaluation, we provide the following results on a version of EventDC-FullSyn where no motion blur is applied to color images:

Table R4: Results on EventDC-FullSyn without motion blur.

We appreciate the reviewer’s comment in helping us to improve the clarity and completeness of our work. We will add these results in the revision.

1. Weakness (pipeline latency) and Question (trade-off design):

We fully agree that the task of predicting dense depth from event camera-based sparse measurements in highly dynamic environments is particularly difficult. Our paper proposes a framework to solve this very challenging task for the first time. We hope our work serves as a foundation for future research and encourages continued advancements across multiple aspects.

Currently, our method primarily focuses on addressing RGB-D misalignment and improving depth accuracy around moving objects. However, we acknowledge that the present latency (41.5 ms) remains insufficient for the demands of highly dynamic scenarios. Following the suggestion of the reviewer, we explored several trade-off designs by reducing the channels of EventDC to half or one-quarter at each encoder stage, and by replacing the encoder with the lightweight EfficientNet-B0. The results in Table R1 show significantly reduced latency at the trade-off of some increase in error. Nonetheless, the performance remains superior to the suboptimal Prompting method.

We thank the reviewer for this valuable suggestion, and we will incorporate these additional results in our final paper.

Table R1: Complexity comparisons on EventDC-Real.

2. Limitation:

Yes, the current latency of 41.5 ms (~20.1 FPS) on EventDC-Real does not meet the real-time threshold. Since the color camera in EventDC-Real operates at 30 FPS, our model currently lags in a temporal gap of around 10 frames or approximately 300 ms of unprocessed event information. Moreover, a frame rate closer to 60 FPS (~16.6 ms latency) is often desirable for more responsive performance in high-dynamic environments. We will include Table R1 in the final paper to improve clarity. In future work, we plan to explore more efficient architectural designs. We will also investigate optimization strategies to reduce latency and move closer to real-time performance.

3. Paper Formatting Concerns:

We thank the reviewer for checking our paper meticulously. We will correct the typos (Table 1 FLIR BFS-U3-31S4C, Line 236 Prompting), proofread and spellcheck the paper in the final version.

Finally, please let us know if you have any further questions or suggestions that could help us improve the paper!