DG-SLAM: Robust Dynamic Gaussian Splatting SLAM with Hybrid Pose Optimization

Q1: Technical novelty.

A1: Thank you for your advice. We aim to clarify your concerns from the following perspectives: (1) As dynamic 3D Gaussian-based SLAM remains in its nascent stages of research and diverges markedly from traditional approaches, this paper presents preliminary explorations in this domain. We are the first work aimed at addressing the challenge of robust pose estimation for 3D Gaussian-based SLAM in dynamic environments. We hope to design an innovative dynamic SLAM system to better utilize 3D Gaussian explicit representation. (2) We have discovered that directly employing a 3D Gaussian representation for pose optimization in dynamic environments tends to result in convergence to local optima, and the solution process is notably unstable. Thus, we need to build a relevant robust tracking front end. (3) We design the motion mask generation method to filter out the invalid optical flow correspondence in Droid-VO to improve the robustness of the original tracking process. Furthermore, we have observed that Gaussian representations, when initialized with stable values, demonstrate improved convergence behavior. (4) Based on these observations and analyses, we propose a hybrid optimization approach incorporating a coarse-to-fine pose optimization strategy. Specifically, we employ a rectified confidence matrix within the droid-slam framework to provide initial pose estimates, which are subsequently refined using a global Gaussian map. This suite of designs significantly enhances the accuracy and robustness of pose estimation for GS-based SLAM in dynamic environments.

Q2: The meaning and motivation of designing a hybrid coarse-to-fine strategy.

A2: We sincerely appreciate the reviewer's meticulous and detailed review of our work, which helped us improve the quality of this paper. It should be noted that the coarse-to-fine approach is an optimization thought that has proven effective in enhancing the accuracy and robustness of pose estimation for complex tasks. However, how to achieve precise and robust pose optimization during the coarse and fine stages continues to be an open and challenging problem.

We have introduced a hybrid pose optimization strategy specifically tailored for Gaussian-based SLAM systems to effectively tackle pose estimation challenges in dynamic environments, which also represents a central innovative contribution of our paper. Specifically, in the coarse-optimized stage, we have implemented a strategy that filters out unreliable and inaccurate optical flow estimations within the existing confidence matrix component of the original droid-slam system. This strategy removes inaccurate matching correspondences, thereby providing a relatively precise initial pose estimate. During the fine-optimized stage, in addition to utilizing motion mask generation methods to filter out invalid sampled points, we have designed a novel point management technique to maintain the Gaussian map. This encompasses adaptive point addition and effective outlier removal strategies. These elements are critical in ensuring that our SLAM system maintains high levels of accuracy and robustness in dynamic environments.

Additionally, the coarse-to-fine pose optimization strategy forms a compact and integral feedback mechanism within the entire SLAM system. Improved pose optimization precision facilitates the generation of more accurate motion masks, which in turn enhance the accuracy of the initial pose estimates obtained during the coarse optimized stage and lead to better pose optimization results in the fine optimized stage.

Furthermore, the keyframe selection strategy, which is predicated on optical flow motion during the coarse-optimized stage, further enhances the efficiency and accuracy of pose optimization in the fine-optimized stage. This integrated approach ensures a compact and effective optimization process throughout the whole SLAM system.

W1: More experements results.

A1: Thank you for your suggestion. We have added experiments comparing ORB-SLAM3[2] and Refusion[1]. As shown in Table 4 (one-page global PDF), our method presents a more robust tracking capability.

W2: Comparing with MonoGS.

A2: Thank you for your suggestion. Actually, we have already compared our method with MonoGS[3] in the initial draft. MonoGS is the name of the open-source framework and we used the name of its paper, namely GS-SLAM, as shown in Table2,3,4. We will make modifications to this issue to prevent misunderstandings.

W3: Related work.

A3: Thank you for your suggestion. We will discuss and cite Crowd-SLAM[4] and DDN-SLAM[5] in the revised version.

W4: Real-world scenes experiments.

A4: Extending dynamic 3DGS to large-scale outdoor scenarios has always been a challenging problem. As mentioned in the conclusion part, the problem of loop closure for handling large-scale scenes is an interesting direction for future research. Therefore, we did not provide experimental results for outdoor environments.

However, we provided experimental results for real-world scenes, as shown in Fig. 2 and Tab. 3(one-page global PDF). We conduct experiments on 3 real-world sequences and all results indicate the superiority of our method.

W5: Lack certain optimization for the mapping system.

A5: We have designed an adaptive Gaussian point addition and pruning strategy for the mapping system. We will perform geometric consistency verification on map points through depth warp. Consequently, artifacts can be eliminated to a certain extent. Of course, the constructed Gaussian map may still contain a small number of artifacts, and we are considering how to better resolve this issue.

Q1: The role of the semantic mask. According to Table 6, the segmentation mask is much more important than other components.

A1: In response to your observation that the segmentation mask seems to play a much more significant role than other components, we believe it relates to the dynamic object classes in the validation datasets, such as TUM and BONN. The semantic prior predominantly features dynamic object classes, such as humans, enhancing semantic segmentation performance. For non-rigid objects like humans, the results of depth warp fusion are less accurate compared to semantic segmentation, which is reasonable.

Following your suggestion, we consider that SAM methods can indeed generate more accurate segmentation masks, provided that they receive inputs that are precise and unambiguous, such as points, boxes, or masks. While SAM methods could serve as an alternative approach to semantic segmentation, they still do not capture the true motion of objects. Therefore, a geometric consistency module is necessary for additional verification. Moreover, the inference speed of SAM methods is slower compared to current semantic segmentation approaches. Your suggestion has opened up new avenues for our research, and we are considering how best to integrate the SAM method into our SLAM system to improve the accuracy of our motion segmentation.

Q2: More implementation detail.

A2: Thank you for your helpful advice. We will enhance the description of the experimental details in the implementation section. We utilize OneFormer to generate prior semantic segmentation. For the depth wrap mask, we set the window size to 4 and the depth threshold to 0.6. For keyframe selection, we adopt the keyframe selection strategy from DROID-VO based on optical flow.

Q3: To what extent can the method proposed in the paper tolerate motion?

A3: As you mentioned, removing a large number of features could compromise tracking accuracy. Therefore, instead of employing a feature-based SLAM method as the front end, we utilize dense optical flow tracking from droid-SLAM for motion estimation. This choice enables our method to better tolerate dynamic environments. It can operate robustly and accurately in both low and high dynamic sequences of public datasets (TUM and BONN), as well as in real-world settings (Self-collected dataset). However, if occlusions are extensive, covering more than two-thirds of the entire image, the system is still prone to track loss. This issue remains a significant challenge for current SLAM systems and is yet to be resolved.

Q4: The definition of the dynamic object with respect to the semantic mask and the running time of the semantic model.

A4: Thank you for your suggestions. It is important to note that semantic segmentation is just one of the methods we use to generate motion masks; we also employ geometric consistency checks with depth warping operation. For different real-world applications, we typically predefine certain dynamic categories and initially obtain motion masks through semantic segmentation. In the vast majority of daily scenarios, humans are generally considered dynamic objects. Of course, the special scenario you mentioned, involving wax figures, may indeed struggle to balance predetermined motion priors with actual movements. However, such scenarios are relatively rare compared to typical applications, and we have not yet considered these cases in our approach. Regarding the real-time aspect of semantic segmentation you mentioned, it is essential to emphasize that our method does not focus on the specific architecture of the semantic segmentation network used, but rather on the design of information fusion methods. With ongoing advancements in these research fields and improvements in computing power, the processing speeds for semantic segmentation are expected to increase, ensuring they do not become bottlenecks for our method.

Q1: The details of the motion mask generation strategy.

A1: During our experiments, we observed that inaccuracies in pose estimation, coupled with noise in the captured ground-truth depth values, can result in unstable motion segmentation outcomes when relying solely on a single warp mask. However, by integrating observations from depth warps across multiple frames, a more accurate motion mask can be constructed. This approach significantly enhances the reliability and precision of the motion segmentation process, as shown in Fig. 1 (one-page global PDF).

Q2: The design criteria of the three gaussian point deletion methods.

A2: Thank you for your suggestions. We aim to clarify your concerns from the following perspectives:

(1) The opacity value denotes the importance of each gaussian point along each ray during the rendering process. Our opacity verification operation is similar in spirit to the original 3D GS. Since we implement an adaptive dense control algorithm to regulate the density of the Gaussian map, we contend that the deletion of non-essential Gaussian points can facilitate the incorporation of new Gaussian points and enhance the convergence of pose optimization.

(2) Through the visualization of reconstructed Gaussian maps, we have observed that the Gaussian spheres with large scale do not effectively represent the local details of objects; instead, they tend to cause blurring or shadowing during the rendering process. This results in highly unstable pose optimization, which in turn leads to the failure of camera tracking.

(3) The third criterion for evaluation arises because we have found that the rasterization of 3DGS imposes no constraints on the Gaussians along the viewing ray direction, even when a depth observation is present. This does not pose a problem when a sufficient number of carefully selected viewpoints are available. However, in continuous SLAM, this lack of constraint leads to numerous artifacts, which complicates tracking. Consequently, it becomes necessary to remove these Gaussian points.

For each added Gaussian point, only a sufficient number of observations can confirm that the point is not an artifact. Based on this design principle, we select half the window size as the empirical threshold for point deletion operations. If a point is observed too infrequently, we consider it not robust enough and, therefore, it needs to be removed promptly.

Q3: The details of the mapping processing running time on BONN dataset.

A3: Your understanding is correct. The extended execution time of the mapping process on the Bonn dataset is closely linked to the point management strategy. More specifically, it is intricately associated with the extent of camera movement within the dataset.

Due to the relatively large movements of the camera in the Bonn dataset, an increased number of Gaussian points are inserted during the mapping process to maintain tracking stability. Consequently, this leads to a noticeable augmentation in processing time during the mapping phase, as shown in Tab.2 (one-page global PDF). Even when addressing complex pose estimation issues in dynamic scenes, the runtime of our mapping process is approximately the same as that of other GS-based SLAM methods.

We sincerely appreciate the reviewer's meticulous and detailed review of our work, which helped us improve the quality of this paper. It should be noted that some of the issues you raised are typically treated as foundational definitions in the 3D Gaussian Splatting paper and traditional SLAM literature. To avoid unnecessary repetition, we have omitted some of the common expressions and explanations of symbols in our previous papers.

A1: (1)Eq.2 comes from the 3DGS[1] paper. $R$ is the viewing transformation and $J$ is the Jacobian of the affine approximation of the projective transformation. (2) $I_{m\times n}$ represents a matrix of the same size as the image, filled with ones. (3) Eq.8 is derived from DROID[2], where $p_i$ means a grid of pixel coordinates. (4) $S_i$ in l.192 and $R_i$ in l.193 are scaling vector and rotation matrix respectively. (5) In Eq.13, $\alpha_{i}$ means the opacity, which has been defined in l.125, and $S$ means the scaling vector. Other threshold parameters can be find more details in l.244.

A2: (1)"finishing pose estimation robustness" means achieving robust pose estimation. (2)“invalid zones” means dynamic object regions, illegal depth value regions, and regions with less rendering opacity. (3)"furnishing an initial pose estimate" represents that the coarse stage provides a better initial camera pose for the fine stage.

A3: We will modify it to "visual SLAM with dynamic objects filter".

A4: This has been explained in 3DGS[1]. The raycasting will sort all the Gaussian points based on their depth value.

A5: For the defination of $I_{m\times n}$, please refer to A1. The intersection operation signifies that for each element in the matrix $I_{m\times n}$, we assess whether its warp depth meets a specified threshold and subsequently modify the corresponding value at that position.

A6: The mean of 3D Gaussian points in space means the center coordinates of the Gaussian points. Thus, we use $\mu$ to represent the coordinates and mean. Their meanings are consistent.

A7: We parametrize the camera pose using quaternion and translation. To efficiently execute the pose optimization process, we have computed the Jacobian of the designed loss function to these two optimization variables and have implemented modifications in CUDA.

A8: We employed the dynamic point density management strategy, which determines whether to insert new Gaussian points according to the dynamic radius. The dynamic radius is determined based on color gradient, as explained in l.202 - l.204.

A9: $S_i$ refers to the scale of each Gaussian point, initialized by the mean distance of three nearest neighbor points. This operation originated from the original 3DGS[1] paper.

A10: $\lambda_1$-$\lambda_3$ are the loss weight, determined through grid search. We show the details of these parameters in l.242.

A11: Thank you for your advice. We will update this expression in the revised paper.

A12: We utilized the GT point cloud provided by the Bonn to qualitatively and quantitatively evaluate the reconstruction results, where "quantitative analytical experiments" refer to the results in Tab.1.

A13: The values presented in Tab. 2 and 3 correspond to the Absolute Trajectory Error (ATE) metrics, as mentioned in the Metrics section of the experiments (l.233). ATE quantifies the average absolute trajectory error between the estimated and ground truth poses across all measurement locations. Further details can be found in the Evo evaluation tool.

A14: Thank you for your suggestion. In the experimental section, we have added and compared some classic dynamic SLAM methods: Co-Fusion[4], MID-Fusion[5], and EM-Fusion[6]. The results are shown in Tab. 1(one-page PDF). Our method has showcased superior pose estimation results in comparison to classic dynamic SLAM methods. We will also discuss these methods in related work.

A15: It means “Map point deleting strategy” mentioned in l.214, Sec 3.4.

A16: In Table 5, we present the number of iterations of the tracking and mapping process for different SLAM methods. The experimental results show that our method can achieve superior results with fewer iterations. What's more, together with Table 7, Table 5 also demonstrates the competitiveness of our method in time consumption.

A17: We divided some default dynamic object classes based on semantic prior, such as humans. However, the semantic mask requires prior knowledge of motion categories and faces generalization challenges. Thus we use the depth warp mask to identify undefined dynamic objects such as balloons and boxes.

"encompass major motion categories" means the dynamic object category contained in Tum is human. Therefore, we opted to perform ablation studies on BONN. It contains moving balloons and boxes, which can show the effectiveness of our depth warp mask.

A18: STD means Standard Deviation of Absolute Trajectory Error. This concept has been mentioned in l.234. Further details can be found in the Evo evaluation tool.

A19: The semantic segmentation process can be considered as a part of data preprocessing. Thus, it has not been included in the system runtime calculations. To address your concern, we have tested the inference time of the Oneformer[7] method we adopted on an A6000 GPU, which is 163ms for every frame. It should be noted that our approach does not focus on the specific semantic segmentation network used, but rather on the fusion method itself.

A20: Thank you for your helpful advice. We will enhance the description of the experimental details in the implementation section. We utilize OneFormer[7] to generate prior semantic segmentation. We reported the run-time in A19.

A21: Extending dynamic 3DGS to large-scale scenarios has always been a challenging problem. As mentioned in GS-SLAM[8], the problem of loop closure for handling large-scale scenes is an interesting direction for future research. Thus, we believe it is still a challenge worth solving in dynamic SLAM.

Q4: Running time issue.

A4: It should be noted that our approach does not focus on the specific semantic segmentation network used, but rather on the fusion method itself. All the baseline methods in Table 7 in our main paper do not include the time for semantic segmentation. Semantic segmentation is typically employed as a preprocessing operation and serves as an input to the system. Most of the dynamic SLAM methods also did not include the time for semantic segmentation instead including it as part of the preprocessing step. Consequently, Table 7 captures the total runtime of our SLAM system, and the time taken for semantic segmentation is not included. Considering the reviewer's concerns about the runtime of the semantic segmentation network we utilized, we have tested the inference time of the Oneformer [7] method, which is 163ms for every frame. Correspondingly, the running time including semantic segmentation is 245.57ms for tracking.

To avoid potential ambiguity, we will provide a detailed description of the running time per frame for the semantic segmentation method employed in the revised paper.

Note that DG-SLAM is not optimized for real-time operation. As the first system to propose dynamic Gaussian splatting for SLAM, this paper primarily focuses on designing an effective approach to achieve robust tracking and mapping. With ongoing advancements in these research fields and improvements in computing power, the processing speeds for semantic segmentation are expected to increase, ensuring they do not become bottlenecks for our method.

Q5: Discuss more limitations and assumptions.

A5: Our approach has a certain tolerance for the accuracy of semantic segmentation. When the semantic segmentation model generates incomplete or incorrect segmentation masks for fewer frames, our system can still perform accurate and robust camera tracking. This is attributed to the coarse-to-fine pose optimization strategy where the motion estimation from the dense optical flow can tolerate some segmentation failures. Thus the coarse stage can provide a robust initial pose for the fine stage.

Even when addressing complex pose estimation issues in dynamic scenes, the runtime of our mapping process is approximately the same as that of other GS-based SLAM methods, as shown in Table 2 (one-page global PDF). The system runtime reported in the original paper was evaluated on a 3090ti GPU. With ongoing advancements in GPU computational capabilities, we believe our method is capable of real-time computation, enabling the live processing of sensory data streams.

Hope our response helps the reviewer's final recommendation. Thank you!

Q1: Notation issue.

A1: We are grateful for the reviewer's valuable comments. We will revise and refine the notations in the revision to eliminate any potential ambiguities.

Q2: Rotation representation and constraints within the SO(3) group.

A2: We utilize the quaternion as the rotation representation in the optimizer to facilitate pose optimization. Throughout the optimization process for the rotational variables, we compute the derivative of the final loss directly with respect to the current quaternion representation. This chain rule differentiation process is structured into two stages. In the first stage, the derivative from the quaternion to the rotation matrix is calculated using automatic differentiation within Pytorch, which allows for the automatic capture of optimization variables. In the second stage, the derivative of the final loss with respect to the rotation matrix is manually computed using the Jacobian matrix. This calculation is implemented using CUDA code during the Gaussian rendering process. Upon obtaining the optimized quaternion, we normalize it to achieve a unit quaternion. This entire optimization process ensures compliance with the constraints on the SO(3) group.

Q3: More experiments and consistent comparison.

A3: Thank you for your suggestions. To address the reviewer's concern, we provide the running time of tracking and mapping on the TUM dataset, as shown in Tab. 1. All the experiments were conducted with 20 iterations for tracking and 40 iterations for mapping. We have also added a comparison with the classical methods: Co-Fusion and MID-Fusion, where the running time is cited from the original papers.

Table1: Run-time comparison on TUM f3_walk_static. * denotes including semantic segmentation time.

Classical methods utilize an efficient variant of the Iterative Closest Points (ICP) algorithm for camera tracking and avoid a training process for mapping. Moreover, the implementation of classical methods has been optimized through the use of a C++ framework, resulting in faster runtime compared to neural SLAM methods. Currently, both NeRF and GS-based SLAM methods exhibit a certain gap in runtime when compared to traditional approaches. Addressing this discrepancy is a challenge that the research community continues to actively pursue.

Dear Reviewer UDnM,

We appreciate your time for reviewing, and thanks again for the valuable comments and suggestions. As the discussion phase is nearing its end, we wondered if the reviewer might still have any concerns that we could address. We believe our responses on consistent comparison, rotation representation, running time, and limitations addressed all the questions/concerns, and we hope that our work’s impact and results are better highlighted with our responses.

It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you！

Best wishes,

Authors