RoDyn-SLAM: Robust Dynamic Dense RGB-D SLAM with Neural Radiance Fields

Thanks for the positive and detailed review as well as the valuable suggestions for improvement. We would like to address the reviewer's concerns as follows:

W1: More substantial architectural changes details.
Thanks for the suggestion. We have added the more substantial architectural details for our SLAM system in Appendix A.1 in the revised paper.

W2: More experiments in static scenes.
Thanks for the suggestion. To better demonstrate the system performance of pose estimation and reconstruction in a static scene, we choose the real-world RGB-D dataset [1] rather than the synthetic datasets [2,3]. Compared with the current the state of the art neural rgb-d SLAM, we evaluate the pose estimation and reconstruction results on three represented sequences. Additional detailed comparison results are available in Table 9 in Appendix A.5.

Compared with our baseline methods Co-SLAM [4], our method does not compromise the performance of the original SLAM methods in terms of tracking and mapping in static scenes. In fact, it achieves competitive results.Notably, our proposed optimization algorithm is not restricted to a specific slam system. Thus, it can also be applied to other neural rgb-d slam methods to improve the data association between the inter-frame.

Compared with the E-SLAM [5], our SLAM system requires less storage space and achieves real-time running at 10fps, despite several hours of training time needed for E-SLAM. Consequently, our method strikes a well-balanced compromise between accuracy and speed, demonstrating competitive performance of pose estimation when compared to state-of-the-art methods.

[1] J ̈urgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In IROS, 2012.

[2] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint, 2019.

[3] Dejan Azinovi ́c, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In CVPR, 2022.

[4] Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam. In CVPR, 2023.

[5] JMohammad Mahdi Johari, Camilla Carta, and Franc ̧ois Fleuret. Eslam: Efficient dense slam system based on hybrid representation of signed distance fields. In CVPR, 2023.

W3: GBA defination.
Yes, your assumption is correct. We do not contribute to the loop closure methods with neural radiance fields. In this paper, GBA represents an operation that samples rays from entire keyframes management set and performs a global bundle adjustment to reduce the pose estimation errors and obtain a more consistent implicit map representation. We are also considering incorporating a loop closure module in future work to implement GBA in loop closure and reduce accumulated errors.

Thanks for the insightful and detailed review as well as the valuable suggestions for improvement. We would like to address the reviewer's concerns as follows:

W1: Motivation of neural RGB-D SLAM over dense RGB-D SLAM in dynamic scene.
Thanks for the suggestions. We have revised the introduction to motivate the benefits compared to existing dynamic dense RGB-D SLAM methods. All the modified text is highlighted in blue in the revised version.

Compared with the existing dense RGB-D dynamic SLAM methods, the key advantage of neural RGB-D SLAM lies in its neural implicit representation method. The neural scene representations have attractive properties for mapping in dynamic scenes, including improving noise and outlier handling, geometry estimation capabilities for unobserved scene parts, high-fidelity reconstructions with reduced memory usage, and the ability to generate high-quality static background images from novel views. Moreover, neural radiance fields also open up possibilities for the subsequent development of dynamic SLAM, such as distilling perceptive information and accomplishing object-based tracking or reconstruction.

Finally, please allow us to briefly introduce our motivation and contribution to building Rodyn-SLAM system. Existing neural RGB-D SLAM methods rely on a static environment assumption and do not work robustly within a dynamic environment due to the inconsistent observation of geometry and photometry. To tackle this problem, we propose a dynamic object identification method and filter the invalid sample ray to recover the static scene map. To further improve the accuracy of pose estimation in dynamic scenes, we propose novel and robust pose optimization methods utilizing edge projection loss to enhance data association between convisible frames.

W2: More experiment results with stronger baselines.
Thanks for the suggestion. Due to the current limitation of NeRF-based SLAM methods in handling dynamic scenes, we propose a general robust dynamic Nerf-based SLAM method RoDyn-SLAM, which constitutes the innovation of this paper. We have selected state-of-the-art NeRF-based SLAM methods as our benchmark for comparison, serving as robust baselines. Following your suggestions, we have added the results of MID-Fusion [1] and Droid-SLAM [2] in Table 2 and Table 3 in the revised paper, respectively. Due to the advanced factor graph optimization algorithm and extensive training data, Droid-SLAM has achieved superior pose estimation results in dynamic scenes compared to Nerf-based SLAM methods.

Bundle-SDF [3] and vMAP [4] are object-based neural SLAM methods rather than a general nerf-based SLAM method. It relies on external conditions, such as the availability of pose prior or the requirement for the camera to remain stationary. Thus, we do not add the comparison results of these methods in the revised paper. The more detailed analysis is summarized as:

• Bundle-SDF is designed for estimating the object pose and reconstructing the object mesh. Our approach aims to estimate the camera pose in dynamic scenes by leveraging a static background. The objectives of these two methods are distinct. Additionally, the validation datasets used for Bundle-SDF require the camera to remain stationary, allowing for the pose estimation of moving objects. The datasets we employed for evaluation do not meet these conditions.

• vMAP primarily emphasizes joint reconstruction of objects, relying on estimated poses from other methods i.e. ORB-SLAM3 to provide the initial pose. Therefore, we believe that the direct comparison of pose optimazation is unfair. We attempted to make a comparison of reconstruction performance with vMAP given GT pose on TUM RGB-D dynamic sequence. However, the final reconstruction results for dynamic scenes were notably blurred. Actually, due to the absence of joint optimization between the camera pose and implicit map, the dynamic objects can not be filtered, resulting in significant blurriness and floaters in the reconstructed scene.

[1] Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew Davison, and Stefan Leutenegger. Mid-fusion: Octree-based object-level multi-instance dynamic slam. In ICRA, 2019.

[2] Zachary Teed and Jia Deng. DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. In NeurIPS, 2021.

[3] DBowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas M ̈uller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In CVPR, 2023.

[4] Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison. vmap: Vectorised object mapping for neural field slam. In CVPR, 2023.

Thanks for the positive and detailed review as well as the valuable suggestions for improvement. We would like to address the reviewer's concerns as follows:

W1: Lack introduction in Section 3.2 eq (4).
Thank you for pointing it out. We have added the introduction of the meaning of variables $j$ and $k$ in Equation 4 to address confusion about the expression‘s significance. $j$ and $k$ stand for the keyframe ID, illustrating the optical flow mask warping process from the k-th to the j-th keyframe. All the modified text is highlighted in blue in the revised version.

W2: Typos in Fig 1.
Thank you for pointing out the spelling mistake in Fig 1. It is a typo and has been corrected. We have reviewed this paper to reduce occurrences of spelling issues in the revised paper.

Q1: The project going next on the (implicit) mapping.
It's my honor to discuss this question with you. We appreciate your opinion, and this is a question we have been contemplating recently as well. Actually, for the dynamic SLAM, it is crucial to identify the dynamic objects and enhance the data association in the tracking or mapping process. We follow this principle in designing the Rodyn-SLAM system, providing a specific technical solution based on neural RGB-D SLAM. While the method is relatively universal, minor adjustments (not based on depth information) may be necessary for different sensor inputs.

In a sense, the coupled inertial measurement is also an observation to enhance the data association utilizing IMU preintegration theory in optimization process. However, it is insufficient to rely only on inter-frame constraints from IMU if the visual tracking part is not adjusted. Although the modern visual-inertial solution can improve the robustness and accuracy compared with the pure visual solution in dynamic environments, the precision may not satisfy the purpose of separate mapping.

Formulate (implicit) mapping as a separate problem from pose tracking has a latent problem in which the tracking pose may not be consistent with the current mapping results. Certainly, without considering the above practical problems, we can also conduct research on the separate mapping in dynamic scenes. Since the background motion has been provided, independently estimating the pose and reconstructing the geometry structure of dynamic objects becomes a crucial and outstanding problem. It seems that employing object-based neural implicit mapping and object-level tracking could offer improved solutions to the aforementioned issues.

Thanks. As the Reviewer HXpV submitted the review comments very late (on the last day of author-reviewer discussion period), please allow some time for a more comprehensive reply.

We thank the reviewer for the valuable suggestions and address the reviewer’s concerns as follows:

Q1: Testing dataset.
Following traditional dynamic RGB-D SLAM methods [1,2,3], we assess the pose estimation results and reconstruction quality on the TUM RGB-D and BONN RGB-D datasets. As indicated in these papers, these datasets are sufficient to evaluate the performance of dynamic SLAM system. Additional evaluation results are presented in Appendix A.2, specifically in Tables 7 and 9 of the revised paper.

Q2: ATE meaning.
We have illustrated the meaning of ATE in the original paper. Please refer to the "Metric" part in the Experiment section for more details.

Q3: Comparison with ORB-SLAM3 and DVO SLAM.
As our method is a neural-based dynamic SLAM method, we selected the traditional dynamic SLAM methods, such as ReFusion [1], DynaSLAM [2], and MID-Fusion [3], as the baseline methods instead of the general static SLAM methods. ORB-SLAM3 [4] and DVO SLAM [5] rely on a static environment assumption and do not work robustly within a dynamic environment due to the inconsistent observation of geometry.

DynaSLAM has showcased competitive pose estimation results in comparison to DVO-SLAM. We add the comparison results of DVO-SLAM in Table 2 in the revised paper, cited from DynaSLAM. Please refer to the original DynaSLAM paper for more details.

ORB-SLAM3 primarily introduces multiple map fusion and tightly integrated visual-inertial fusion techniques via MAP estimation. However, it still depends on the assumption of a static environment, making it similar to ORB-SLAM2 [6] in this regard. We have included the ORB-SLAM2 pose estimation results in Table 2, cited from ReFusion. To address your concerns, we also evaluated the results of ORB-SLAM3 and included them in Table 10 in Appendix A.6.

Our comparison results with ORB-SLAM3 and DVO SLAM are as follows:

Q4: Degenerate case for computing the fundamental matrix.
Actually, the issue of degeneracy in fundamental matrix computation is a fundamental problem in multi-view geometry and should be investigated as a standalone problem. This is not the contribution of our paper. We employ an engineering trick to address this problem, similar to approaches used in ORB-SLAM2 and ORB-SLAM3. We compute the fundamental matrix $\mathbf{F}$ and homography matrix $\mathbf{H}$, utilizing the RANSAC method and reprojection error to score the computed matrices $SF$ and $SH$. Then, we determine the ratio of scores with the equation $rh = SH/(SH+SF)$. If the value of $rh$ is bigger than 0.5, we choose the homography matrix for the pose estimation; otherwise, we choose the fundamental matrix.

[1] Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguere, and Cyrill Stachniss. Refusion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting residuals. In IROS, 2019.

[2] Berta Bescos, Jose M Facil, Javier Civera, and Jose Neira. Dynaslam: Tracking, mapping, and inpainting in dynamic scenes. RAL, 2018.

[3] Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew Davison, and Stefan Leutenegger. Mid-fusion: Octree-based object-level multi-instance dynamic slam. In ICRA, 2019.

[4] Carlos Campos, Richard Elvira, Juan J Gomez Rodrıguez, Jose MM Montiel, and Juan D Tardos. Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE TRO, 2021.

[5] Christian Kerl, J ̈urgen Sturm, and Daniel Cremers. Dense visual slam for rgb-d cameras. In IROS, 2013.

[6] Raul Mur-Artal and Juan D Tardos. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE TRO, 2017.

Q5: Computing resources and Computation times analysis.
We have reported the time consumption of the tracking and mapping process in Section 4.4. Please refer to Table 5 in the original paper. Our system outperforms E-SLAM [7] in terms of time consumption and maintains a comparable level with Co-SLAM [8]. Therefore, our system can run in real-time like Co-SLAM, which is reported as a real-time SLAM method in paper [8]. In terms of computing resources, we run our RoDyn-SLAM on an RTX 3090Ti GPU, taking roughly 4GB of memory in total (please see "Implementation detail" part in the Experiment section).

Contribution. Please allow us to briefly review the contribution of our Rodyn-SLAM. We propose a robust dynamic RGB-D SLAM with neural implicit representation. Existing neural RGB-D SLAM methods rely on a static environment assumption and do not work robustly within a dynamic environment due to the inconsistent observation of geometry and photometry. To tackle this problem, we propose a dynamic object identification method that combines the optical flow and semantics prior to generating the motion mask and filters the invalid sample ray to recover the static scene map. To further improve the accuracy of pose estimation in dynamic scenes, we propose a divide-and-conquer pose optimization algorithm utilizing edge projection loss to enhance data association between convisible frames. Extensive experiments are conducted on the two challenging datasets, and the results show that RoDyn-SLAM achieves state-of-the-art performance among recent neural RGB-D methods in both accuracy and robustness.

[7] JMohammad Mahdi Johari, Camilla Carta, and Franc ̧ois Fleuret. Eslam: Efficient dense slam system based on hybrid representation of signed distance fields. In CVPR, 2023.

[8] Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam. In CVPR, 2023.