Scalable Benchmarking and Robust Learning for Noise-Free Ego-Motion and 3D Reconstruction from Noisy Video

Dear Reviewer nb61,

Thank you for your thoughtful feedback and recognition of our work. Below, we provide detailed responses to your comments and concerns. The revised manuscript has been uploaded, with all major changes highlighted in blue for clarity.

Q1: The problem formulation and cross-referencing can be simplified to improve readability.

A1: Thank you for the suggestion! We have streamlined the problem formulation and reduced the usage of cross-referencing in the revised paper.

Q2: Depth perturbation types can be extended.

A2: Thank you for the suggestion! We agree that depth noise types can be further extended. While our ultimate goal is to develop a generator that accurately models real-world depth noise, this work establishes an important foundation by highlighting how simplified depth perturbations, including missing data, noise, and range limitations, effectively expose critical weaknesses in existing methods, motivating future advancements in this domain.

Q3: The benchmark can be extended to more diverse scenes.

A3: Thank you for the suggestion! As our noisy data synthesis pipeline is scene-agnostic, extending the benchmark to include perturbations in more diverse 3D scenes is straightforward. We have included the discussion in our revised paper.

Q4: The evaluation of CorrGS can be extended to more diverse real-world scenes, e.g., ScanNet++.

A4: Thank you for the suggestion! We would like to clarify that existing real-world RGB-D dense SLAM datasets, such as ScanNet++, are nearly noise-free, which is why we did not evaluate the real-world robustness of CorrGS on this dataset. Instead, we self-collected scenes under dynamic illumination and rapid motion to demonstrate its performance in the main paper. We agree that extending the evaluation to more real-world scenes with natural perturbations is valuable, and we have included this discussion in the future work paragraph of our paper.

Dear Reviewer nb61,

We sincerely thank you for the insightful further feedback!

We fully agree that quantitative evaluations of CorrGS on additional real-world data are essential to showcasing the robustness of our method. To address this, we have included two qualitative RGB-D rendering results from our CorrGS model, captured in a natural outdoor setting featuring a large rock on grass near a paved surface. These results were obtained both in the morning and at night, under a 10x faster motion setting, to further challenge and evaluate our method's performance.

Our model achieved an average PSNR of 28 dB for RGB and an average depth L1 loss of 1.80 cm for the morning sequence, and a PSNR of 34 dB for RGB with an average depth L1 loss of 2.17 cm for the night sequence. These results demonstrate the capability of our approach to accurately reconstruct scene geometry while preserving high-quality appearance details. Furthermore, we observed that our method is robust to illumination changes, even under very low-light conditions at night. This highlights the effectiveness of our model in handling complex lighting variations that are typical in real-world scenarios.

We acknowledge that obtaining quantitative evaluations with ground truth in real-world settings remains a challenge due to the lack of accurate reference data for RGB-D comparisons. However, we believe the results presented here serve as an effective indication of our method's robustness and practical applicability.

Moreover, the range limitations of our RealSense D435i camera (0.3–3.0 m) restricted us from capturing larger scenes, which we plan to explore in future work. Nonetheless, the new results, now included in Section H.5 of the appendix, further demonstrate the effectiveness of our method in real-world scenarios, even under challenging conditions.

Warm regards,

The Authors of Submission 696

Q5: Can CorrGS handle high-frequency dynamics, and does it include adaptations tailored to specific noise profiles?

A5: Thank you for the valuable question!

• Handling high-frequency dynamics. CorrGS is designed to handle highly dynamic changes by employing online learning to adapt between the current perturbed observation and the historical map, which is maintained to be consistent and clean. In our experiments, we demonstrated CorrGS’s capability to tackle real-world scenarios, including high-dynamic video with varied illumination (e.g., transitions between darkness, normal lighting, and overexposure), depth noise, and motion blur in RGB-D data. These results are detailed in the Sec. 5.3 of the main paper and Sec. H.3 of the appendix.
• Adaptation to specific noise. We did not explicitly adjust CorrGS for specific noise types in our experiments. Instead, CorrGS uniformly approaches denoising as a problem of learning the transformation between the perturbed observation and the rendered (clean) frame of the historical map, without tailoring it to individual noise profiles.

Q6: Any envisioned adjustments for extending Robust-Ego3D and CorrGS from indoor to outdoor?

A6: Thank you for the thoughtful question. Extending beyond static indoor scenes, outdoor environments present unique challenges that warrant careful attention:

• 1) Dynamic elements. The presence of moving objects, such as vehicles and pedestrians, complicates consistent ego-motion estimation and 3D reconstruction.
• 2) Unbounded spatial scales. Outdoor scenes often span large, unbounded spaces, posing difficulties for existing online 3D reconstruction methods, even without external disruptions.
• 3) Evolving visual disruptions. Factors like shifting shadows, changing weather, and varying lighting conditions add complexity to visual processing and reconstruction accuracy.
• 4) RGB-D sensing limitations. Commercial RGB-D cameras, such as the Intel RealSense D435i (that we use) with a depth range of up to 3 meters, impose significant constraints on dense 3D reconstruction in outdoor environments. To address these challenges, future work must explore strategies for both benchmarking and model design that effectively mimic these effects in controlled environments and tackle them systematically. This would involve designing datasets and models that account for dynamic, large-scale, and visually complex outdoor scenarios, thereby enhancing the robustness and applicability of our methods.

Dear Reviewer ccbx,

Thank you for your thoughtful feedback and recognition of our work. Below, we provide detailed responses to your comments and concerns. The revised manuscript has been uploaded, with all major changes highlighted in blue for clarity.

Q1: Can CorrGS be extended to tackle the challenge of depth perturbations?

A1: Thank you for the insightful suggestion. While this work primarily focuses on RGB appearance denoising, we recognize that depth noise presents a significant challenge in certain scenarios. For example, as shown in Fig. H42 and Fig. H44 of the appendix, severe depth noise can emerge under strong overexposure, occasionally leading to minor pose estimation inaccuracies. Recovering from depth perturbations is currently beyond the scope of the CorrGS model. However, we view this as an important avenue for future work and have included a discussion on this topic in the Sec. 6.3 of the main paper. We appreciate your suggestion and believe it highlights a valuable direction for extending the capabilities of CorrGS.

Q2: What is the chosen loss function for the PQV step, and what are the insights behind its design?

A2: Thank you for the question. In the PQV step, we use the standard RGB-D rendering loss (Eq. G34 in the Appendix) to evaluate the quality of the pose. The purpose of the PQV step is to identify and reject degenerate cases where correspondence-based pose initialization becomes unreliable. These degenerate cases typically occur when the two viewpoints are either very close together or identical. In such scenarios (see Fig. G34. of the Appendix), it is challenging to accurately estimate the relative pose by optimizing the transformation between the point clouds of the two viewpoints, as the lack of sufficient variation undermines correspondence-based methods. Please see Sec. G.3.7 of the Appendix for more details.

Q3: Comparison of CorrGS to classical non-neural SLAM methods?

A3: Thank you for the valuable suggestion. While our work primarily focuses on dense Neural SLAM methods for photorealistic 3D reconstruction, we have extended our comparison to include two classical non-neural SLAM methods as suggested:

• ORB-SLAM3: CorrGS significantly outperforms ORB-SLAM3 under fast-motion scenarios in the 10× speed-up motion setup of our Robust-Ego3D benchmark (ATE RMSE: 13 cm of ORB-SLAM3 vs. 0.45 cm of our CorrGS), demonstrating superior robustness and accuracy. Additionally, ORB-SLAM3 loses tracking under combined fast-motion and dynamic illumination conditions, whereas CorrGS successfully handles these challenging perturbations (see Table 6 in the main paper).

• RTAB-Map: This dense non-neural SLAM method struggles to maintain tracking in the same 10× speed-up motion setup.

These results underline the robustness and adaptability of CorrGS compared to classical approaches in challenging scenarios.

Q4: How does the pose quality verification step perform under varying noise levels, and does it use thresholds?

A4: Thank you for the valuable question!

• Performance under pose noise: We thank the reviewer for highlighting the importance of robustness analysis under varied pose noise levels. The pose quality verification (PQV) step enables strong performance under varying noise conditions, effectively handling pose initialization errors with up to 40° rotation errors and 1.0m translation errors between views. To illustrate this, we have included new Figures H45 and H46 in Sec. H.4 of the Appendix. These figures detail the behavior of pose estimation error across diverse initial pose noise values, emphasizing the framework's resilience to both rotational and translational discrepancies. We hope this additional analysis clarifies the robustness of our method.
• Thresholding: There is no fixed hand-crafted threshold for PQV. Unlike threshold-dependent methods, PQV employs a dynamic comparison of RGB-D rendering losses. It contrasts the losses from correspondence-guided initialization against propagated poses (e.g., extrapolated or copied from prior frames). This mechanism ensures adaptive performance by selecting the pose with the lower rendering loss, eliminating the need for predefined thresholds or hyperparameters. The adaptive nature of PQV allows it to maintain resilience against noisy inputs.

Dear Reviewer ccbx,

We are pleased to hear that our responses have clarified your questions. Thank you again for your recognition of our work and for your thoughtful feedback and effort during the review process!

Warm regards,

The Authors of Submission 696

Dear Reviewer 5SrC,

Thank you for your thoughtful feedback and recognition of our work. Below, we provide detailed responses to your comments and concerns. The revised manuscript has been uploaded, with all major changes highlighted in blue for clarity.

Q1: Depth perturbation types and restoration can be extended.

A1: Thank you for the suggestion! 1) We agree that depth noise modeling can be expanded for more extensive study. While our long-term goal is to develop a generator that replicates real-world depth noise, this work lays the foundation by showing how simplified depth noise modeling reveals key vulnerabilities in existing methods, aiming to inspire further advancements in this area. 2) While this work primarily focuses on RGB appearance denoising, we recognize that depth noise presents a significant challenge in certain scenarios. For example, as shown in Fig. H42 and Fig. H44 of the appendix, severe depth noise can emerge under strong overexposure, occasionally leading to minor pose estimation inaccuracies. Recovering from depth perturbations is currently beyond the scope of the CorrGS model. However, we view this as an important avenue for future work and have included a discussion on this topic in the Sec. 6.3 section of the main paper. We appreciate your suggestion and believe it highlights a valuable direction for extending the capabilities of CorrGS.

Q2: Synthetic noise generation might not capture the unpredictable nature of some real-world scenarios.

A2: Thank you for the valuable feedback! We completely agree with the reviewer on this important point. Our long-term vision is to develop a noise generator capable of accurately replicating the complex and unpredictable nature of real-world noise. While this remains an ambitious goal, our current work takes an essential first step by highlighting that even simplified, individually modeled noise can expose significant vulnerabilities in existing dense SLAM methods. We hope this serves as a call to action for the community to prioritize research in this critical area. Additional insights on the value of using synthetic perturbations for benchmarking model robustness are provided in Section B.5 of the Appendix.

Q3: Suggestion on proposing new metrics for robustness evaluation.

A3: Thank you for the valuable suggestion! To ensure consistency and facilitate direct comparisons between the standard noise-free settings and the proposed noisy settings, we primarily adopt standard SLAM evaluation metrics used in clean settings. However, we recognize the importance of developing specialized metrics to address the unique challenges of noisy scenarios and will have included this as a promising direction for future research in our paper.

Q4: How does CorrGS handle extreme cases when the initial model is significantly misaligned?

A4: Thank you for the insightful questions! CorrGS relies on a well-defined noise-free state, derived from clean RGB-D observations in the initial video frames, as a reference for reconstructing 3D maps and poses from noisy video. Without such initial observations, the noise-free state becomes ambiguous, undermining both the reconstruction process and alignment of noisy frames. In cases where the initial internal model is significantly misaligned or inaccurate, CorrGS struggles to restore a noise-free 3D representation. Instead, it may produce noisy outputs or fail to reconstruct the 3D structure entirely. This limitation, which arises from the assumption that historical frames are available to establish a reference state, has been explicitly noted in the Future Work section. Establishing a robust initialization remains a critical prerequisite for our approach.

Q5: How sensitive is CorrGS to extremely bad initial pose estimates?

A5: We thank the reviewer for highlighting the importance of robustness analysis under varied pose initialization errors. The pose quality verification (PQV) step enables strong performance under varying noise conditions, effectively handling pose initialization errors with up to 40° rotation errors and 1.0m translation errors between views. To illustrate this, we have included new experimental results (please see Figures H45 and H46 in Sec. H.4 of the Appendix). These figures detail the behavior of pose estimation across diverse noise scenarios, emphasizing the framework's resilience to both rotational and translational discrepancies. We hope this additional analysis clarifies the robustness bound of our method.

Dear Reviewer 5SrC,

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

Thank you for your recognition of our work. In response to your comments, we have made the following revisions to our manuscript:

• We have clarified the scope of our depth noise modeling and highlighted its foundational role in exposing key vulnerabilities of existing methods. A discussion on extending CorrGS to handle depth perturbations is now included in Section 6.3.
• We acknowledge the limitations of synthetic noise and have elaborated on its role as a first step toward benchmarking robustness. Insights are provided in Section B.5 of the Appendix.
• While we maintain standard SLAM metrics for consistency, we have identified the development of robustness-specific metrics as a key future direction.
• We have explicitly noted CorrGS’s reliance on a noise-free reference state for alignment and reconstruction and addressed its limitations under extreme misalignment in the Future Work section.
• We have expanded our analysis of pose initialization errors, including new results in Appendix Section H.4, which demonstrate CorrGS’s resilience to rotational and translational noise.

We hope these revisions address your concerns and welcome any further feedback.

We look forward to actively participating in the Author-Reviewer Discussion session and welcome any additional feedback you might have on the manuscript or the changes we have made.

Thank you for your constructive comments!

Warm regards,

The Authors of ICLR Submission 696

Dear Reviewer iqVc,

Thank you for your thoughtful feedback and recognition of our work. Below, we provide detailed responses to your comments and concerns. The revised manuscript has been uploaded, with all major changes highlighted in blue for clarity.

Q1: More insight into why CorrGS achieves better performance under noisy conditions.

A1: Thank you for the comment! CorrGS outperforms prior methods through two key innovations: 1) Correspondence-guided pose learning, which enhances geometric alignment under dynamic motion, and 2) Correspondence-guided appearance restoration learning, which enables effective appearance restoration for mapping. By leveraging a progressively updated, clean 3D Gaussian Splatting representation rendered into RGB-D frames, CorrGS aligns noisy observations to achieve robust pose estimation and photorealistic reconstruction, even under severe noise and motion. These advances are validated by our experiments, demonstrating superior noise resilience and dynamic robustness.

Q2: For the introduction, it is suggested to further highlight the common challenges noise introduces to models.

A2: Thank you for the valuable feedback! We have revised the introduction to emphasize the key challenges noise introduces to models.

Q3: It is suggested to further emphasize the importance of noise resilience for the model in the abstract and introduction.

A3: Thank you for the valuable suggestion! We have revised the abstract and the introduction to further highlight the importance of noise resilience, as robust performance in real-world scenarios demands handling imperfect data caused by environmental factors, sensor limitations, and dynamic changes.

Q4: More analyses of specific vulnerabilities of existing models across different types of noise.

A4: Thank you for the valuable suggestion! We have conducted comprehensive benchmarking analyses of model vulnerabilities under various types of noise, using perturbation settings two orders of magnitude larger than previous works. These results, encompassing both experimental and theoretical insights, are detailed in Section 4 of the updated main paper. Additionally, we have provided extensive quantitative and qualitative analyses in the Appendix (Sections D, E, and F, pages 39–64) to explore model vulnerabilities in greater depth. We encourage the reviewer to refer to these sections for more insights.

Q5: What is the rationale behind the problem formulation, and is the section redundant?

A5: Sorry for the confusion. To clarify, the problem formulation section was intended to provide a structured overview of the key challenges addressed in our work: generating noisy video data from clean 3D models with specific poses and perturbations, developing a model capable of estimating clean 3D reconstructions and poses from noisy video inputs, and establishing an evaluation framework for perturbation-conditioned feedback to enhance model robustness. While these elements do align with standard ML workflows, we included the section to clarify how our contributions interconnect. However, we agree that merging this section with others would streamline the presentation, and we have revised the content and merged it with following sections accordingly.

Dear Reviewer iqVc,

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

In response to your comments, we have made the following revisions to our updated manuscript:

• We have clarified how CorrGS achieves better performance under noisy conditions by leveraging correspondence-guided pose learning and appearance restoration to enhance geometric alignment and mapping.
• We have revised the introduction and abstract to further emphasize the challenges posed by noise and the importance of noise resilience in real-world scenarios.
• We have conducted comprehensive analyses of model vulnerabilities under various noise types, benchmarking with perturbations two orders of magnitude larger than prior works. These results are detailed in Section 4 and the Appendix.
• We have streamlined the problem formulation section by merging it with subsequent sections for improved readability.

We hope these revisions adequately address your concerns.

We look forward to actively participating in the Author-Reviewer Discussion session and welcome any additional feedback you might have on the manuscript or the changes we have made.

Thank you once again for your valuable feedback!

Warm regards,

The Authors of Submission 696

Dear Area Chair uUps,

Thank you for the reminder about the response period. We would like to express our sincere gratitude to you and the reviewers for your time and valuable feedback. We are actively preparing our responses and will engage promptly within the specified timeframe.

Best regards,

The Authors

Dear reviewers, this is a reminder that November 26 is the Last day for reviewers to ask questions to authors.

We are pleased to report that our responses and revisions have addressed all concerns raised by participating reviewers (Reviewer ccbx and Reviewer nb61). Additionally, we are delighted that Reviewer nb61 increased the evaluation score from 6 to 8 during the rebuttal stage.

We have summarized the key improvements and clarifications below. The revised manuscript has been uploaded, with all major changes highlighted in blue.

• CorrGS Model Robustness and Real-World Applicability

  • Conducted extensive robustness analyses of CorrGS under varied pose initialization errors and included additional experiments demonstrating adaptability to diverse pose noise conditions, addressing concerns from Reviewers ccbx and 5SrC.
  • Enhanced qualitative results with new outdoor sequences to highlight CorrGS’s potential in real-world applications, addressing Reviewer nb61’s feedback.*
  • Expanded comparisons to include classical non-neural SLAM methods (e.g., ORB-SLAM3 and RTAB-Map) to highlight CorrGS’s superiority, as suggested by Reviewer ccbx.

• Clarity and Readability Enhancements

  • Streamlined problem formulation and reduced cross-references to improve flow, addressing suggestions from Reviewers iqVc and nb61.
  • Clarified the rationale behind the chosen loss function for the PQV step in CorrGS, based on Reviewer ccbx’s suggestion.
  • Provided additional insights into why CorrGS achieves superior performance under noisy conditions, addressing Reviewer iqVc’s feedback.
  • Highlighted the critical importance of resilience to noise for 3D reconstruction models in the introduction and abstract, incorporating Reviewer iqVc’s suggestion.

We recognize the reviewers’ valuable suggestions for extending benchmarks to more diverse environments, improving depth noise modeling, and developing robustness-specific metrics. These ideas will guide our future research efforts.

We appreciate the thoughtful discussions and are encouraged by the positive recognition of our work. The reviewers’ acknowledgments of our contributions and their constructive input have been instrumental in refining our paper.

Thank you once again for your valuable feedback and support. We look forward to advancing this line of research in collaboration with the community.

Warm regards,

The Authors of Submission 696