GS-CPR: Efficient Camera Pose Refinement via 3D Gaussian Splatting

Thank you for your positive comments, including the recognition that the proposed method is simple yet effective and achieves more accurate results than state-of-the-art NeRF-based methods. Below, we address your concerns and respond to your questions.

-(a) Scalability: We think the large-scale and unbounded scene reconstruction is an open problem in the community. Most of the structure-based methods that rely on scene geometry have similar problems under large-scale scene relocalization. However, visual relocalization methods that do not rely on scene representation have better scalability but much lower accuracy [1]. Currently, several concurrent studies [2][3][4] are exploring large-scale 3DGS reconstruction. These advancements hold the potential to be integrated into our existing framework in the future. In this paper, we demonstrate the high performance of our approach across three widely used public datasets, highlighting its effectiveness. We’d love to further explore this challenging scenario in our future works.

-(b) Advantages of 3DGS in the relocalization task. Our framework leverages novel view synthesis (NVS) from 3DGS to render images that provide more robust and sufficient 2D-3D correspondences for RANSAC + PnP, outperforming traditional reference image matching, particularly in scenarios with sparse reference databases. Unlike the traditional visual localization pipeline, which is constrained to a fixed set of predefined reference images, our 3DGS-based NVS can generate ANY view based on the pose estimator's predictions. By utilizing 3DGS instead of sparse point clouds for scene representation, our method employs a learning-based approach to adapt the domain of rendered images to the query's exposure conditions, ensuring closer alignment with the query and offering greater flexibility.

-(c) Compared with HLoc: HLoc still has a marginal advantage regarding translation accuracy on the Cambridge dataset. We updated the results of HLoc (SP + SG) on Cambridge landmarks in Table 3 in the rebuttal version. However, this paper focuses on comparisons within the state-of-the-art neural render pose (NRP) estimation approaches. We conduct experiments on the 7Scenes dataset using the open-source HLoc toolbox [5] with SfM-generated ground truth (GT) to compare our approach against two baselines: HLoc (SP + SG) + NetVLAD (k=1) and ACE + HLoc (SP + SG) with nearest pose-based retrieval (k=1).

The results show that pose-based image retrieval from ACE cannot get better results and our ACE + GSLoc is more accurate.

-(d) Evaluation metrics: We have supplemented the evaluation with both accuracy and pose errors for the 12Scenes dataset and updated Table 4 accordingly. For the Cambridge Landmarks dataset, we report only pose errors to ensure a consistent and fair comparison, as previous studies provide only this metric.

-(e) Differences between 3DGS SLAM: We believe our framework and 3DGS SLAM are solving different problems, please refer to our response to reviewer tSXc (a) for more details. In addition, for pose refinement, current popular 3DGS SLAM systems primarily leverage photometric loss to optimize both the map and the camera's trajectory iteratively with differentiable rendering, which is more similar to iNeRF and iComMa. However, our framework enables one-shot absolute pose refinement.

[1] Sattler, Torsten, et al. "Understanding the limitations of cnn-based absolute camera pose regression." CVPR 2019.

[2] Yan, Yunzhi, et al. "Street gaussians for modeling dynamic urban scenes." ECCV 2024

[3] Liu, Yang, et al. "Citygaussian: Real-time high-quality large-scale scene rendering with gaussians." ECCV 2024

[4] Liu, Xi, et al. "3dgs-enhancer: Enhancing unbounded 3d gaussian splatting with view-consistent 2d diffusion priors." arXiv 2024

[5] Sarlin, Paul-Edouard, et al. "From coarse to fine: Robust hierarchical localization at large scale." CVPR 2019.

-(f) More visualization results: In the new rebuttal version, we provide match visualizations of multiple scenes as well as examples of retrieved images in Figure 10. This would better complement the results in (c) Compared with HLoc. We also give you detailed information about each image, which will help you reproduce the results easily. These examples illustrate the advantages of using 3DGS render images from SCR methods’ prediction to perform matching. The key insight is that both image retrieval and ACE pose-based retrieval are restricted to identifying queries within a limited reference pool. In contrast, our approach theoretically allows for an unlimited reference pool.

Thank you for your follow-up questions. We appreciate the opportunity to address your concerns.

While we acknowledge that 3DGS has limitations regarding large-scale scalability, we view this as an open problem within the field. Additionally, we compare HLoc in Tables 2 and 3 of this paper, but our primary focus of this paper is comparing with NeRF-based methods, such as NeFeS and CrossFire. If scalability is considered a significant limitation, it is worth noting that all previous neural render pose (NRP) estimation methods share similar constraints. Large-scale scenes also pose challenges for traditional HLoc, which relies on the SfM sparse point cloud model.

Why is HLOC not compatible with ACE? We never state or show that HLoc is not compatible with ACE. We just show that ACE cannot get better-retrieved images than traditional image retrieval methods because they can only retrieve images from the same pool. The key insight is that both image retrieval and ACE pose-based retrieval are restricted to identifying queries within a limited reference pool. In contrast, our approach theoretically enables an unlimited reference pool by leveraging high-quality rendered images from 3DGS.

Thank you very much for your response. We will add a description in the limitation section like "This paper demonstrates the effectiveness of our framework on commonly used datasets and benchmarks. However, reconstructing high-quality 3DGS models for large scenes remains a significant challenge. Exploring the application of this framework to large-scale scenes for accurate visual camera relocalization is a promising avenue for future work." in the final version.

-(c) Combine hot approaches: Our approach goes beyond simply combining 3DGS and MASt3R. As outlined in Section 3.2, our method leverages the matching components of MASt3R to eliminate the need for training extra feature descriptors to match image pairs with a sim-to-real domain gap—a common limitation of other NeRF-based pose estimation techniques. However, relying solely on MASt3R with reference images fails to deliver accurate metric translation due to the lack of scale information and cannot build 2D-3D matches for absolute pose estimation. This limitation arises because MASt3R is unable to generate metric 3D points within the pre-built global coordinate system. For instance, [1] addresses this problem in robotics tasks by incorporating a depth camera. To resolve this challenge, 3DGS serves a critical function by rendering metric depth and constructing 3D geometry, enabling accurate 2D-3D matching. Besides, the rendered view generated by 3DGS from SCR/APR poses aligns much better than normal image retrieval from fixed reference images. This integration is important in recovering precise scale and achieving robust and accurate pose estimation with sufficient matches. By combining the strengths of these components, our framework addresses current limitations.

-(d) Feature matching comparison: We compare three different matching methods in Table 7 and discuss this in Section 4.4. That shows that MASt3R has higher accuracy than LoFTR and DUSt3R.

-(e) Figure 3 modification: We appreciate your feedback and revised Figure 3 to more clearly highlight the distinction between GSLoc_rel and GSLoc.

[1] Jiao, Jianhao, et al. "LiteVLoc: Map-Lite Visual Localization for Image Goal Navigation." arXiv 2024.

Thank you for your positive comments on our evaluation results. We primarily address your concerns below.

-(a) Difference between GS-SLAM: SLAM and camera relocalization are fundamentally different tasks with distinct objectives. Compared to GS-SLAM, our framework focuses on determining the pose of a single RGB query image relative to a pre-built map without relying on any prior frame information. SLAM involves concurrently building a map of an unknown environment and estimating the camera's motion across sequential frames, assuming smooth input and a continuous camera trajectory. Consequently, camera poses are determined relative to previous frames. However, this causes issues such as drift. Popular GS-SLAM systems primarily address this tracking challenge by leveraging photometric loss to optimize both the map and the camera's trajectory iteratively like iNeRF [1] and iComMa [2] with differentiable rendering. However, our framework enables one-shot absolute pose estimation.

Unlike SLAM, where the map evolves alongside the camera trajectory, our task aims to estimate the camera pose within a global pre-established map. Our framework is explicitly designed for the (re)localization task, achieving more accurate results than other state-of-the-art neural render pose (NRP) estimation pipelines.

Table 1. We list the main differences between SLAM and Relocalization.

-(b) Novelty: The scope of this paper is focused on comparisons within the state-of-the-art NRP estimation approaches such as NeFeS, CrossFire, and other pose-established methods like SCR and APR. Existing NeRF-based approaches typically rely on training scene-specific descriptors, as seen in methods like CrossFire and NeFeS. In contrast, our framework eliminates the dependence on extra feature training through the integration of MASt3R, while simultaneously reducing the domain gap between rendered images and query images by introducing the ACT module in 3DGS. Using 3DGS instead of sparse point clouds for scene representation enables the domain shift of the rendered image according to the query's exposure through a learning approach, offering greater flexibility. As demonstrated in Table 8, Figure 5, and Figure 9, the off-the-shelf Scaffold-GS approach alone does not achieve optimal results. By incorporating ACT and semantic filtering during 3DGS training, we significantly improve rendering quality and achieve superior performance. These modules make our framework easier to deploy and achieve higher accuracy than other NRP methods.

[1] Yen-Chen, Lin, et al. "inerf: Inverting neural radiance fields for pose estimation." IROS 2021

[2] Sun, Yuan, et al. "icomma: Inverting 3d gaussians splatting for camera pose estimation via comparing and matching." arXiv 2023

Thank you very much for your positive feedback on our paper, acknowledging the novelty of our approach, advantageous performance, excellent presentation, and good writing. We primarily address your concerns below.

-(a) Necessity: Enhanced Matching through Neural View Synthesis (NVS): Our framework uses NVS to render images that provide matches for RANSAC + PnP. Unlike HLoc, which is limited to a predefined set of reference images, 3DGS-based NVS is not constrained by the reference images database, instead rendering virtually unlimited views. This is especially beneficial when the reference database is sparse. Our framework can thus ensure closer alignment with the query. We conduct experiments on the 7Scenes dataset using the open-source HLoc toolbox [1] with SfM-generated ground truth (GT) to compare our approach against two baselines: HLoc (SP + SG) + NetVLAD (k=1) and ACE + HLoc (SP + SG) with nearest pose-based retrieval (k=1). The parameter k=1 is selected to ensure a fair comparison, as our framework is designed to match a single rendered image with the query.

The results show that pose-based image retrieval from ACE cannot get better results than our ACE + GSLoc.

-(b) Novelty: The significant performance improvements obtained by GSLoc, compared to other neural render pose (NRP) estimation methods were not directly from off-the-shelf Scaffold-GS, which yields suboptimal results, as shown in Table 8, Figure 5, and Figure 9 of our paper. We significantly enhance the rendering quality for localization by incorporating ACT and semantic filtering during 3DGS training. Additionally, MASt3R eliminates the necessity of training scene-specific features. The integration of these modules collectively leads to substantial improvements over state-of-the-art NRP estimation pipelines, highlighting the effectiveness and flexibility of our proposed approach.

-(c) 3DGS Rely on Dense View: We thank the reviewer for mentioning the issue, we perform experiments without using dense view 3DGS reconstruction. To evaluate the robustness of our approach under sparse view conditions, we uniformly sample 1/3, 1/10, and 1/20 of the training images (2000 frames) on Stairs of 7Senes to construct the 3DGS model and evaluate localization accuracy on this reduced dataset. We provide results here, demonstrating the effectiveness and robustness of our method even with sparse training views.

-(d) Aachen-day-night: we’d love to further explore this challenging scenario in our future works. Our intuition about current 3DGS/NeRF is that they cannot handle the day-night gap very well. Several concurrent studies [2][3][4] are exploring large-scale reconstruction and lighting changes for 3DGS. These advancements hold the potential to be integrated into our existing framework in the future. However, we hope our strong results in commonly used datasets are enough to show the advantages over existing NRP methods.

-(e) HLoc setting in Table 2: We report the HLoc results using DenseVLAD as the image retrieval module (k=20), combined with SuperPoint and SuperGlue, based on the official open-source implementation provided in [5].

-(f) Runtime: We rely on SCR/APR methods to provide initial poses, enabling fast inference. We provide runtime clarification of GSLoc and GSLoc_rel on RTX 4090 below.

GSLoc: 3DGS rendering (10ms) + MASt3R inference (70ms) + MASt3R matching (40ms)+ RANSAC+PnP (50ms) ~ 170ms

GSLoc_rel: 3DGS rendering (10ms) + MASt3R inference (70ms) ~ 80ms

Total runtime: SCR + GSLoc: ACE inference (17ms) + 170ms

APR + GSLoc: Marepo inference (7.4ms) + 170ms; DFNet inference (2ms) + 170ms

SCR + GSLoc_rel: ACE inference (17ms) + 80ms

APR + GSLoc_rel: Marepo inference (7.4ms) + 80ms; DFNet inference (2ms) + 80ms

Thanks for catching the runtime calculation issue, we will update the runtime data in Table 6.

[1] Sarlin, Paul-Edouard, et al. "From coarse to fine: Robust hierarchical localization at large scale." CVPR 2019.

[2] Yan, Yunzhi, et al. "Street gaussians for modeling dynamic urban scenes." ECCV 2024

[3] Liu, Yang, et al. "Citygaussian: Real-time high-quality large-scale scene rendering with gaussians." ECCV 2024

[4] Chen, Yingshu, et al. "StyleCity: Large-Scale 3D Urban Scenes Stylization with Vision-and-Text Reference via Progressive Optimization." arXiv 2024

[5]Brachmann, Eric, et al. "On the limits of pseudo ground truth in visual camera relocalization." ICCV 2021

Thank you for your follow-up questions. We are happy to address your concerns.

-(g) Fair comparision: On 7Scenes, we use the same SfM model for 3DGS training and HLoc relocalization with the same SfM GT. In the future, we will release all of our COLMAP models, pre-trained 3DGS models, and our code for reproducibility.

-(h) More visualization results: In the new rebuttal version, we provide match visualizations of multiple scenes as well as examples of retrieved images in Figure 10 (page 18). We also give you detailed information about each image, which will help you reproduce the results easily. These examples illustrate the advantages of using 3DGS render images from SCR methods’ prediction to perform matching. The key insight is that both image retrieval and ACE pose-based retrieval are restricted to identifying queries within a limited reference pool. In contrast, our approach theoretically allows for an unlimited reference pool.

Thank you very much for your positive feedback on our paper. We appreciate your comments on recognizing the novelty as well as the promising performance of our approach. We primarily address your concerns below.

-(a) W1: Here, we supplement the GLACE + GSLoc results, where GSLoc significantly improves GLACE accuracies in two of the three datasets (7scenes and 12scenes), demonstrating the effectiveness of our method. On indoor scenes, the initial accuracy of ACE and GLACE is very close. The accuracy after refinement of GLACE is also very close to that of ACE with GSLoc refinement. On the Cambridge dataset, we achieve comparable results with an advantage on rotational error for GLACE.

We conducted additional experiments with ACE+NeFeS and GLACE+NeFeS using the official open-source implementations. Our findings indicate that integrating NeFeS does not improve the accuracy of ACE or GLACE; in fact, it leads to a decline in performance. The quantitative results are presented below.

Table 1. Average results across 7Scenes.

Table 2. Average results across 12Scenes.

Table3. Average results across Cambridge Landmarks.

-(b) W2: Regarding the reliability of the depth map rendered from 3DGS, in our framework, we report the average Rel (Relative error) and $\delta_{1.25}$ (percentage of pixels where predicted depth lies within a factor of 1.25 of ground truth depth) across 17K test frames of 7Scenes using Kinect RGBD depth as GT.

The above table shows that depth maps rendered from 3DGS are accurate. For qualitative results, please refer to Figure 7 and Figure 8 in our paper. Since RANSAC can inherently provide robustness to some reasonable degree of noise, our evaluation results show that our 3DGS rendered depth maps can achieve “sufficient” robustness for high-accuracy pose prediction when combined with PnP+RANSAC. For example, we achieve 100% query accuracy of 5cm, 5deg on 7scenes datasets and nearly 99% accuracy on 2cm and 2deg on 12scenes datasets, demonstrating the reliability of our approach.

Thank you for your positive comments, including the recognition that our proposed method is simple yet effective and achieves state-of-the-art results. Below, we address your concerns and respond to your questions.

-(a) Minor writing problem: We modified the minor writing issues on ‘\cite’ and punctuation marks in the rebuttal version. Thank you for your valuable feedback on these writing details.

-(b) Eq. (3): Thanks for pointing out the typo here. The correct formula is:

Our code is implemented correctly. We appreciate the reviewer for catching this typo.

-(c) Limitation: One limitation of our method lies in its dependency on the accuracy of the initial pose estimates provided by the pose estimator. When the initial pose is highly inaccurate, the overlap between the rendered images and the query image is insufficient to establish reliable 2D-3D correspondences for accurate pose estimation. Examples of failure cases are provided in Appendix A.5 of the rebuttal version. Following Section 4.5 of NeFeS [1], we conducted quantitative experiments to evaluate the limitations of our framework. Specifically, we introduced random perturbations to the ground truth poses of test frames on the ShopFacade scene, applying fixed magnitudes of rotational and translational errors independently. The results after pose refinement using GSLoc are presented below.

Table 1: Average rotation error after refinement by GSLoc

Table 2: Average translation error after refinement by GSLoc

Our framework can improve the accuracy when rotation < 50$^\circ$ and translation < 8 meters, respectively. In contrast, NeFeS [1] achieves accuracy improvements only for rotational errors under 35$^\circ$ and translational errors up to 4 meters. These findings highlight that our method significantly expands the optimization range.

-(d) Title: Thank you for your suggestion regarding the title. After careful discussion, we have decided to change the title and framework name from GSLoc to GS-CPR (Camera Pose Refinement) to better reflect our focus on camera pose refinement. We incorporated this change in the rebuttal version. We will keep using GSLoc in the remaining discussions with reviewers for clarity.

[1] Chen, Shuai, et al. "Neural Refinement for Absolute Pose Regression with Feature Synthesis." CVPR. 2024.