EAG3R: Event-Augmented 3D Geometry Estimation for Dynamic and Extreme-Lighting Scenes

W1: My main concern is ...... follow the generative model from Gehrig et al. [21]. W3: Related to the originality concern, ...... is more practical and system-oriented. Q1: The main components ..... with pointmap-based methods?

Reply: We thank the reviewer for the feedback and would like to clarify the novelty of EAG3R. While based on existing frameworks, our work is the first to integrate asynchronous event streams with the pointmap-based paradigm, enabling robust, pose-free 4D reconstruction in extreme lighting conditions—where prior methods typically fail. This integration demanded resolving several key challenges, which form our core contributions.

---

1. Solving The Challenge of Extreme-Light Conditions for Geometric Tasks

• Challenge: To enable the framework under extreme lighting conditions, a common workaround for low-light scenarios is to apply image enhancement. However, we found this insufficient for geometric estimation.

• Why Enhancement Fails: Preprocessing with RetinexFormer ("LightUp") yields minimal or negative impact on downstream performance due to artifacts and failure to guide multimodal fusion.

• Our Contribution: We introduce a Retinex-inspired SNR estimation module, not just for enhancement but to compute a spatially-varying confidence map. To the best of our knowledge, this is the first time a Retinex-based method has been used to generate a confidence map to guide the adaptive fusion of RGB and event data for a 3D geometry task, trusting events more in dark/noisy regions and RGB more where signals are clear.

---

2. Designing an Efficient and Effective Event Adapter

• Challenge: Events are sparse and asynchronous; adapting them effectively without excessive cost or large paired datasets is non-trivial.

• Why Naive Solutions Fail: Using a heavy RGB encoder with zero convolution initialization is both inefficient and ineffective.

• Our Contribution: We propose a lightweight Swin Transformer-based event adapter, pretrained via self-supervised learning on large-scale event-only datasets. This allows efficient transfer of motion and edge priors, enabling fast convergence and strong generalization under data scarcity.

---

3. Achieving Robust, Principled Feature Fusion

• Challenge: RGB and event data differ significantly in density, noise characteristics, and timing. Simple fusion (e.g., concatenation) or nonlinear fusion (attention) is inadequate.

• Our Contribution: A multi-stage adaptive fusion strategy that:

  • Uses cross-attention within the encoder, where event features query RGB features at multiple levels, learning nonlinear, context-aware relationships.
  • Combines this with SNR-guided feature aggregation, followed by a learnable nonlinear projection for further refinement. This ensures robustness to local noise and lighting variation.

Q2: The event-based photometric loss ...... make it stable and effective in this new context?

Reply: We thank the reviewer for the insightful comment. We emphasize that adapting event-based photometric consistency loss to a global optimization framework is not a trivial integration but a conceptual and methodological innovation. Our $L_{event}$ is not a minor extension for local tracking—it is specifically designed for the more complex task of global 4D reconstruction.

Our contributions are highlighted by three fundamental differences: application domain, motion modeling, and feature point strategy.

---

1. Application Domain: From Local 2D Tracking to Global 4D Reconstruction

• Prior work (e.g., Gehrig et al. [21]) focuses on local 2D feature tracking, operating on small patches in the image plane.
• In contrast, EAG3R performs global 4D scene reconstruction. Our $L_{event}$ serves as a supervisory loss within a global optimization (similar to bundle adjustment), jointly refining:
  • Per-frame depth maps ($D^t$)
  • 3D camera poses ($P^t$)
  • Camera intrinsics ($K^t$)

This moves from localized 2D motion to global scene-level optimization.

---

2. Motion Modeling: From Optical Flow to Geometric Projection

• Previous work models motion via 2D optical flow, which is unreliable in low-light where RGB quality degrades.
• Our approach bypasses this by computing the motion field $\Delta u_{\text{cam}}$ geometrically from estimated 3D structure and camera poses:

$$\Delta u_{\text{cam}}^{t \rightarrow t'} = \pi(K^{t'} (R^{t \rightarrow t'} (D^t(x) \cdot (K^t)^{-1} \tilde{x}) + T^{t \rightarrow t'})) - x$$

• This physically grounded projection provides robust supervision even when RGB fails, linking observed events directly to 3D geometry.

---

3. Feature Strategy: High-Confidence Constraints for Global Optimization

• We focus $L_{event}$ on Harris corners—sparse but stable points with strong gradients—introducing high-quality constraints into the global optimization.
• We compared this with:
  1. Harris corners (ours)
  2. SuperPoint (learned detector)
  3. Random sampling

• Random sampling introduce noisy gradients by selecting unreliable regions, degrading optimization. Network-based detectors are computational expensive. Our Harris-based method ensures stable, targeted supervision.

---

In summary, EAG3R significantly extends previous event-based tracking by redefining the application scope, introducing a geometry-driven motion formulation, and applying a tailored feature strategy—transforming local principles into a globally optimized 3D reconstruction system.

W2: The experiments are c ...... f robustness. Q3: Your evaluation is ...... caused training to fail?

Reply: We thank the reviewer for the valuable feedback on dataset diversity.

Please refer to our reply to our reply to reviewer jzhF W3 for reasons for choosing MVSEC and additional experiments on other datasets.

Why V2E generated datasets fail?

We determined that the failure of V2E) synthesis in low-light conditions stems from two compounding issues within the V2E pipeline: degraded frame interpolation and unstable event triggering.

1. Low-Light Frame Interpolation The V2E process relies on optical flow-based interpolation to generate high-framerate video. However, low-light footage suffers from high noise and low contrast, which severely degrades the quality of the optical flow estimation. This leads to inaccurate or artifact-ridden interpolated frames, making the synthesized brightness changes unreliable for the subsequent event generation step.

2. Unstable Log-Intensity Event Triggering

V2E generates events when the change in log-intensity at a pixel crosses a set contrast threshold, $C$:

$\Delta L(x, y, t) = \log(I(x, y, t)) - \log(I_{\text{ref}}(x, y)) \geq \pm C$

In low-light conditions, pixel intensities $I(x, y, t)$ are very low and close to zero. This makes the $\log(I)$ operation numerically unstable and highly sensitive to noise. As a result, the V2E process either fails to generate meaningful events from subtle motion or produces a stream of spurious events triggered by random noise.

W1: I find the paper somewhat difficult to read. ...... In my view, more intuition would make the paper easier to read.

Reply: We thank the reviewer for their valuable feedback on improving the readability of our paper. We appreciate the opportunity to clarify our design process, particularly for the event adapter and feature fusion strategy. Here we provide more intuitions and experimental considerations for our architecure and loss design, we will also revise out paper accordingly.

1. Design Rationale for the Hybrid Architecture

Our hybrid architecture design seeks an optimal balance between performance and efficiency. We initially evaluated an approach that duplicated the heavyweight RGB ViT encoder, but experiments showed it incurred significant computational overhead for limited performance gain. As shown in the table below, our final proposed architecture significantly outperforms this initial design.

We ultimately chose a lightweight Swin Transformer, pre-trained on large-scale event datasets, as the event backbone. The core advantage of this strategy is its ability to effectively transfer powerful, general-purpose features (e.g., motion patterns) from event data. This facilitates efficient model convergence and stronger generalization, especially in data-scarce, low-light scenarios.

2. Design Rationale for the Event-Based Photometric Loss ($L_{event}$)

Our event-based photometric consistency loss ($L_{event}$) is designed to provide a robust spatiotemporal supervision signal, particularly in scenarios where traditional image-based cues fail due to low light. Its effectiveness and stability stem from two key design choices:

• Geometrically-Derived Motion: To circumvent the unreliability of 2D optical flow in low light, the motion field is generated directly via geometric projection from the global 3D state (depth map $D^t$ and camera transformations $P^t, K^t$). This method establishes a direct, physically consistent constraint between the 3D geometry being optimized and the event data.

• Scale-Invariant Formulation: To address the unknown and variable contrast sensitivity threshold ($C$) of event cameras, we normalize both observed and predicted brightness-change patches to a unit $L^2$ norm. This makes the optimization objective focus on the consistency of gradient directions rather than their absolute magnitudes, ensuring robustness to sensor variations and stability during optimization.

W2: Some of the algorithmic choices ......, the authors could train their own relevant feature detector).

Reply: We thank the reviewer for the constructive comment regarding ablation on the choice of Harris corners. We apologize if our rationale was not sufficiently articulated in the submission. Below, we provide both theoretical justification and empirical validation.

Theoretical Motivation

Our use of Harris corners is guided by two key considerations:

• Signal Quality: Events are triggered by brightness changes, and Harris corners naturally lie in high-gradient regions (e.g., edges, textured areas), which produce dense and reliable event streams. These regions are ideal for enforcing event-based supervision tied to motion dynamics.

• Numerical Stability: The event loss in our Levent formulation depends on the image gradient (via brightness constancy). In flat, low-texture areas, gradients approach zero, making the event loss unstable or uninformative. Harris corners ensure we apply the loss only where it is physically meaningful and robust.

While learned detectors like SuperPoint can also identify salient features, Harris offers a simpler, interpretable, and gradient-aligned solution that matches the assumptions behind the event loss.

Empirical Validation

To validate this design choice, we performed an ablation study on pose estimation for night1 sequences comparing three patch selection methods:

1. Harris corners (ours)

2. SuperPoint corners (learned detector)

3. Random sampling

Results show that Harris-based selection achieves the best trade-off between performance and stability. SuperPoint yields comparable but slightly lower results with higher computational cost, while random sampling degrades performance due to noisy gradients in textureless regions. This confirms the effectiveness and efficiency of our choice.

W3: The experiments have a wide breadth (i.e. across many domains), but are quite shallow (i.e. not much testing within each domain). This is probably explained by the relative low availability of event + RGB datasets.

Reply: We thank the reviewer for the valuable feedback on dataset diversity. We first provide an overview of mainstream event-RGB datasets and explain our selection rationale. We then present additional experiments on MVSEC Indoor and M3ED, which cover a broad range of environments and motion patterns.

The reason for using MVSEC

Our choice of the MVSEC dataset was guided by the strict requirements of our task: robust 3D geometry estimation in dynamic scenes under extreme lighting. This demands datasets with aligned RGB, event data, and accurate ground truth for both depth and pose — a combination that is rarely available. As shown in the table below, few existing datasets satisfy the necessary conditions for evaluating dynamic reconstruction in challenging lighting.

Scalability of EAG3R

To demonstrate EAG3R’s scalability, we conducted additional experiments on MVSEC indoor and M3ED datasets, covering diverse environments (indoor, outdoor, night, HDR), sensor platforms (drones, robots, cars), and motion types, including complex aerial and ambulatory trajectories.

Models were trained under normal lighting and evaluated in low-light/HDR conditions in a zero-shot setting, confirming EAG3R’s robustness beyond vehicle-centric scenes.

Performance in HDR environments (robot dog)

To evaluate the model's performance in HDR scenarios, we conducted tests on the demanding pose estimation M3ED robot dog dataset featuring intense lighting changes. As shown in the table below, EAG3R's pose estimation accuracy is generally superior to both the MonST3R baseline and its scene-finetuned version, with substantial improvements across all key metrics.

Performance in high-speed outdoor drone scenarios

To further test the model's robustness under high-speed motion and complex lighting, we performed evaluations on M3ED high-speed drone dataset. EAG3R demonstrated robust performance in these highly challenging scenarios. For the demanding pose estimation, EAG3R shows clear improvements over the baseline.

Performance in high-speed indoor drone scenarios

We also evaluated the depth estimation performance on the indoor sequence from the MVSEC dataset. As shown in the table below, our proposed method, EAG3R, achieves the best performance across all key metrics compared to the MonST3R baseline and its finetuned variant.

W1: Limited dataset diversity W2: General-purpose dynamic reconstruction L1: Analysis on out-of-distribution motion patterns

Please refer to our reply to jzhF W3.

W3: Ablation completeness: ... clarify practical trade-offs. Q2: Do you have a runtime analysis ... assess practical feasibility?

Reply: We thank the reviewer for suggesting runtime analysis.

1. Lightweight event fusion

• Event Adapter: Uses efficient Swin Transformer, freezing the image encoder and updating only the adapter/decoder, minimizing computation.
• Enhancement Module: Shallow, Retinex-inspired design ensures high efficiency.

2. Runtime analysis

• Table 1: For single image-event pairs, EAG3R adds ~0.4 GB VRAM, ~0.11 TFLOPs, and ~1.2s inference time over MonST3R, with potential CUDA optimization.
• Table 2: For global optimization (sequence length 20), event-guided approach adds ~1 GB VRAM overhead compared to baseline without event loss.

Our design introduces minimal computational overhead while maintaining efficiency.

W4: Event data limitations:...... isn’t deeply evaluated. Q1: The method assumes perfectly time-synchronized, ...... the effect of event noise on your method? L2: Reliance on event data quality ...... noisy scenarios would be valuable.

Reply: We thank the reviewer for their comment on event data quality challenges. Our response covers two aspects:

1. MVSEC Noise: MVSEC’s real-world data, collected with a commercial DAVIS346 sensor, includes inherent noise and ~90% sparsity. EAG3R’s strong performance on this noisy input demonstrates its robustness.

2. Noise and Sparsity Simulations:

   • Noise: We injected 5%, 10%, and 15% Poisson noise into event streams to mimic sensor noise, testing EAG3R’s performance.
   • Sparsity: We simulated low-contrast scenes by randomly dropping 5%, 10%, and 15% of events to evaluate robustness.

These tests confirm EAG3R’s resilience to practical noise and sparsity challenges.

Modern event cameras (e.g., Prophesee GenX320) have higher dynamic range and lower noise than the DAVIS346 used in MVSEC, suggesting EAG3R’s performance could improve further with newer hardware.

W5: Although the paper claims ...... confidence intervals, or p-values.

Reply: We thank the reviewer for suggesting statistical analysis. While large-scale 3D vision models like ours, DUSt3R, and MonST3R typically report single-run results due to high computational costs, we conducted 4 additional independent training runs from random initialization to validate robustness. Results show stable performance in depth estimation, with low standard deviation across key metrics.

Q3: Can you include a qualitative failure ...... with event data? Q4: Could you provide examples of situations ...... boundaries of robustness.

Reply: We thank the reviewer for their insightful comments on our method’s operational boundaries. Our analysis of EAG3R’s performance on challenging MVSEC and M3ED datasets clarifies its robustness:

1. MVSEC Performance: In nighttime sequences, large motion (e.g., sharp turns) and sparse co-located RGB/event signals (e.g., dark, static areas) challenge performance. However, EAG3R’s degradation is less severe than MonST3R, showing the advantage of event-augmentation.

2. M3ED Generalization: Tests on diverse indoor and M3ED datasets (e.g., drone and quadruped robot motion) confirm EAG3R’s robustness across complex trajectories, outperforming baselines and avoiding overfit to MVSEC’s vehicular patterns.

3. Failure Cases: Extreme conditions like complete darkness or heavy event noise may cause failures due to limited information.

In summary, while aggressive motion and sparse events pose challenges, EAG3R consistently outperforms pure visual methods and generalizes well across diverse real-world scenarios.

Q5: How does EAG3R handle regions with few or no events, such as textureless walls or static background areas in dynamic scenes? Does the SNR-aware fusion default to purely RGB features in such regions, and if so, is there a risk of low-quality depth in underexposed areas?

Reply: Reply: We thank the reviewer for their insightful comment on our fusion design’s performance. EAG3R’s multi-stage fusion robustly handles event-sparse regions by:

• Well-lit static areas: Few events but high-quality RGB input; SNR-aware fusion prioritizes reliable image features.

• Underexposed static regions: Low event and RGB quality; early-stage cross-attention fusion uses broader image context to enhance weak local patches.

• Extreme cases(e.g., night sky): Unreliable depth estimates are masked out in benchmarks like MVSEC, minimally impacting metrics.

EAG3R’s fusion, combining early cross-modal interaction and SNR-aware aggregation, ensures resilience in low-activity regions, avoiding catastrophic failure under challenging lighting.

L3: Scalability to longer sequences/larger scenes: the method performs global optimization over per-frame pointmaps, camera poses, and intrinsics, which is computationally intensive. While the paper reports 300 optimization iterations, it doesn’t quantify how this scales with longer sequences.

Reply: We appreciate the reviewer’s comment on scalability. Our EAG3R framework ensures scalability for long video sequences using a sliding window optimization strategy, like MonST3R, computing pairwise pointmaps and loss terms only within the window, avoiding the costly fully connected graph of all frame pairs.

• Constant Problem Size: The sliding window ensures that the optimization problem's size and complexity remain constant at each step, regardless of the total length of the video sequence.
• Decoupled Iteration Cost: Because the optimization problem is fixed in size, the number of iterations required for convergence does not need to increase for longer videos. The cost per iteration is effectively decoupled from the total sequence length.

The computational cost for processing a long video scales linearly with frame count, not quadratically, ensuring high scalability. Supplementary experiments on an A100 GPU confirm this linear relationship between sequence length and cost.

Q1: The method relies on Harris corners...... to quantify bias.

Reply: We thank the reviewer for the valuable comment. While Harris corners may overlook texture-sparse regions, our design deliberately uses event loss as a sparse, high-confidence supplement, not the primary geometric constraint. These regions are already addressed by our global optimization, which integrates alignment, smoothness, and optical flow losses to ensure spatial-temporal consistency.

1. Why choose Harris corners Under extreme lighting where optical flow supervision becomes unreliable (due to degraded RAFT performance), the event loss becomes crucial. However, applying it indiscriminately in low-gradient regions like walls can lead to unstable or uninformative gradients. Hence, we restrict it to Harris corners—locations with strong gradients and dense, reliable event activity—maximizing both signal quality and numerical stability.

2. Comparing Harris corner-based path selection with other feature selection approaches To assess potential bias, we conducted an ablation comparing Harris-based and random patch sampling. As is summarized in the following Table, random sampling caused noticeable degradation, as many sampled patches fell into low-texture regions, introducing noisy gradients that hindered convergence. This empirically confirms that our targeted strategy enhances effectiveness while avoiding spurious influence from weak or invalid supervision.

Q2: Experiments only use MVSEC, more validation is necessary.

Reply: Please refer to our reply to reviewer jzhF W3.

Q3: The paper claims dynamic scene support but doesn’t quantify errors for moving objects (e.g., vehicles).

Reply: We thank the reviewer for the insightful question regarding evaluation on moving objects. The main challenge lies in the lack of dense annotations (e.g., masks) for dynamic objects in current datasets, which makes direct evaluation difficult. Nonetheless, we provide several indirect but effective metrics in the paper to assess performance in dynamic scenes.

• On one hand, in dynamic environments, the accuracy of monocular pose estimation serves as a strong indirect quantitative indicator of the model's ability to disentangle ego-motion from object motion. Poor handling of moving objects would result in incorrect camera motion estimates. As shown in Table 2, our model significantly outperforms the baseline in low-light dynamic settings.
• On the other hand, we quantify the video depth metric in Table A1 of the Appendix (below), which evaluates the depth estimation accuracy across all pixels in the video sequence. This naturally includes dynamic objects in the scene, thus indirectly reflecting the model’s performance on moving objects.

Q4: The SNR-aware fusion assumes linear weightings, but lighting nonlinearities (e.g., saturation) may invalidate this. Introduce nonlinear fusion models (e.g., attention-based) and compare via ablation.

Reply: We thank the reviewer for this insightful comment regarding our feature fusion strategy. We would like to clarify that our framework does not limit fusion to a single linear step. Instead, we adopt multi-stage, nonlinear and adaptive mechanisms to robustly integrate RGB and event features:

• Nonlinear fusion in the encoder: We embed a multi-level cross-attention mechanism within the encoder, where event features query RGB features across resolutions. This enables content-aware, nonlinear fusion early in the network, allowing dynamic emphasis on salient information from both modalities.
• Learnable nonlinear projection after SNR-guided aggregation: The initial SNR-based weighting is followed by a learnable nonlinear projection applied to concatenated features. This step adaptively reweights and transforms the representation, helping address challenges like local lighting nonlinearity.

To directly address the reviewer's suggestion, we conducted new ablation experiments on depth estimation comparing different fusion strategies: linear addition, attention-based fusion at the final stage, and our proposed multi-stage fusion. The results show that our approach outperforms all other choices.

Q5: Ignoring negative events (only using positive polarity) might lose intensity decrease signals in sudden lighting changes.

Reply: We appreciate the reviewer’s concern. However, we would like to clarify that our method does not ignore negative events. Both positive and negative polarity events are used throughout our pipeline. To avoid any further confusion, we will revise the manuscript to clearly state that both polarities of events are utilized during feature extraction and fusion.

Paper Formatting Concerns: The reference formatting is non-standard. For instance, some journal names are presented in full, while others use abbreviations.

Reply: We thank the reviewer for your thorough review and valuable feedback. We sincerely apologize for the inconsistencies in the citation formatting. In the revised version of the paper, we will carefully review and standardize all references to ensure consistency.