Enhancing Visual Localization with Cross-Domain Image Generation

1 Limited discussion of failure cases.

Thank you for your valuable suggestions. We observe two main failure cases in our current method. First, in scenes like Piatrium_Night that contain extremely dark regions, the fine-tuned image editing model sometimes fails to produce realistic textures. This highlights the limitation of fine-tuned InstructPix2Pix in handling low-light appearances. Second, when significant static geometry changes are present in the query images (such as structural modifications), our cross-domain localization performance may degrade. In the future, this issue could be addressed by incorporating static geometry editing into the cross-domain image generation process, thereby enhancing data diversity and improving robustness in such scenarios. We have updated the manuscript to include the discussion of failure cases.

2 No analysis of computational costs.

We apologize for the missing details. All experiments were conducted on a single NVIDIA RTX 4090 GPU. Fine-tuning the image editing model for each scene takes approximately 10 hours on average. Cross-domain 3DGS training and image generation require at most 4 hours per scene. Visual localization training takes roughly 16 hours per scene. During inference, the localization method MS-T with the proposed positional attention mechanism runs at 32.51 ms per frame. We have updated the manuscript to include the computational costs of the experiments.

3 Is there any possibility to include more comparison methods?

1. For the cross-domain visual localization task, our work primarily focuses on Absolute Pose Regression (APR) methods, and the baselines we compare against are the state-of-the-art APR methods provided by the 360Loc benchmark. To the best of our knowledge, there are currently no other advanced methods targeting this task.

2. For the image generation task, we have included comparative experiments with the advanced GS-W [1], which is designed for unconstrained scenes. Since GS-W does not address the long-tail data distribution issue, we conduct quantitative experiments on the Atrium_Day scene, which contains sufficient data. Additionally, we extend its training iterations to 140,000 for the large-scale evaluation scene.

As shown in the table, our method significantly outperforms GS-W while using fewer iterations (60,000). This demonstrates that our method has stronger scene modeling capabilities for large-scale scenes.

[1] Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections [ECCV 2024]

1. Limitation results on 360Loc.

Thank you for your valuable suggestion. Our work targets the challenging task of cross-domain visual localization, which requires datasets containing query images captured by various types of cameras. To the best of our knowledge, 360Loc is the only existing benchmark with cross-camera query images. Therefore, we have exclusively selected 360Loc as our evaluation benchmark.

2. Why not use the cross-domain image generation for SCR methods? Is the proposed method can enhance localization accuracy of SCR methods?

Thank you for your valuable question. We would like to clarify the following points: 1) The baseline methods we evaluated follow the 360Loc benchmark, which does not include SCR methods for comparison. 2) In theory, our method could enhance the accuracy of SCR by using cross-domain 3DGS to render paired RGB and depth images. 3) Adapting SCR to cross-domain tasks is more complex, as SCR relies on camera intrinsics and distortion parameters, such as scene coordinate calculations, reprojection, and pose solving via PnP. However, existing SCR methods are designed for pinhole cameras. Adapting SCR to 360° and fisheye cameras requires deriving tailored methods. These enhancements to SCR are beyond the scope of our work, but it is a promising direction for future research.

3. More quantitative results to validate the effectiveness of the fine-tuning strategy.

Thank you for your valuable suggestion. To validate the effectiveness of our fine-tuning strategy, we conducted quantitative experiments on the atrium scene. We employed image similarity metrics SSIM and LPIPS to evaluate the similarity between the images generated and the ground truth.

As shown in the table, the image similarity significantly improved after fine-tuning, which demonstrates the effectiveness of the proposed fine-tuning strategy.

4. The proposed Cross-Domain 3DGS is engineered.

Our method builds upon Scaffold-GS but introduces key innovations to address challenges like appearance variations, dynamic object interference, and long-tail data distributions. We propose a photometric modeling scheme combining histogram-based embeddings and compensation to handle appearance changes, incorporate photometric priors for dynamic object uncertainty, and fine-tune a text-guided image editing model to augment sparse datasets in long-tail distributions. These improvements form the core innovation of our Cross-Domain 3DGS.

5. The proposed text-guided image editing model seems an application of InstructPix2Pix. The original contribution of the paper is not clear.

To address the long-tail distribution issue in the dataset, we use InstructPix2Pix to transform daytime images into sparse nighttime images for data augmentation. However, directly using the pre-trained model leads to domain inconsistencies and hallucinations, as shown in Figure 7. To overcome this, we propose a fine-tuning strategy. We use 3DGS to render daytime images corresponding to sparse nighttime images, creating a domain shift dataset that enables InstructPix2Pix to learn domain-consistent transformations. Additionally, to mitigate hallucinations due to the sparse domain shift dataset, we generate a scene prior dataset by rendering daytime images that cover the entire scene, enriching the model with more scene priors.

6. It is suggested to rewrite the Introduction to clearly express the challenge and contributions, provide an overview of the entire method, and modify Fig. 2.

1. Thank you for your constructive suggestions. We have revised the Introduction to better highlight the core challenges of cross-domain visual localization and our contributions. We begin by emphasizing the challenges in cross-domain localization, followed by the motivation for introducing 3DGS to address these issues. We then discuss the technical challenges in designing cross-domain 3DGS and the solutions we developed. Finally, we discuss the issues encountered when training visual localization methods with data generated based on 3DGS and how our method is designed to overcome these training issues.
2. We have rewritten a more structured overview of our method and modified Figure 2 to more clearly express the design of cross-domain 3DGS and the training of the visual localization method. We have also updated the paper to include the Impact Statement.

7. In Tables 1,2, which method does the paper use, and how many generated images are used?

1. Sorry for the unclear details. The method used in the paper is MS-T.
2. We provide the detailed counts used for each scene (including self-rotated images and ground truth). | Atrium | Concourse | Hall | Piatrium | |--------|-----------|-------|----------| | 17,430 | 14,730 | 16,200 | 18,960 |

We have updated the paper to include these details.

1. What is the APR method used in the proposed approach—PN, MS-T, or other methods?

Sorry for the unclear details. Our proposed approach employs the MS-T method for absolute pose regression (APR).

2. The ablations are mainly evaluated based on rendering quality. It would be beneficial to compare localization performance to demonstrate the relationship between rendering quality and localization performance.

We sincerely appreciate your insightful suggestion. To evaluate the impact of rendering quality, we conduct a quantitative experiment on the Atrium scene. Specifically, we compare the original Scaffold-GS (which produces lower-quality renderings compared to our cross-domain 3DGS) combined with our proposed data generation and positional attention mechanism.

As shown in the table, localization performance drops when using the lower-quality images generated by Scaffold-GS. This result demonstrates a positive correlation between rendering quality and localization accuracy.

1. Limited discussion of related work

Thank you for your valuable suggestion. We have expanded the related work section in the revised manuscript by incorporating the recommended references to better highlight the novelty of our proposed method. The revised portion of the related work is shown below:

Recent advances in neural rendering have enabled the 3D reconstruction from unconstrained photo collections. Neural Rerendering in the Wild [1] combines traditional 3D reconstruction with neural networks to handle unconstrained scenes. Extensions to the NeRF [2,3] further address challenges in uncontrolled scenes by embedding appearance information and transient uncertainty. In addition, methods such as Ha-NeRF [4] and Neural Scene Chronology [5] focus on capturing temporal variations by employing modules for appearance hallucination and temporal step function encoding. However, their slow training and rendering make large-scale scene modeling and data generation time-consuming. Additionally, the limited parameters of NeRF hinder its ability to effectively represent large outdoor scenes. Recently, 3DGS-based methods have garnered attention due to their faster optimization and rendering efficiency compared to NeRF. Among them, SpotLessSplats [6], GS-W, WE-GS, and WildGaussians have shown potential in modeling appearance variations and dynamic objects in wild scenes. However, they cannot explicitly control photometric properties and are constrained by the performance bottlenecks of pre-trained detectors. Existing methods are also not suitable for scenes with long-tail distribution problems. In this paper, we propose a method that models appearance variations by explicitly encoding photometric histograms, mitigates the impact of dynamic objects without relying on pre-trained detectors, and employs a fine-tuned image editing model to effectively address the long-tail distribution problem.

Several studies have explored cross-domain visual localization under pinhole camera settings. To enhance localization performance, [7] uses invertible generators to produce synthetic images, while [8] converts nighttime images to a more discriminative daytime representation. Other methods focus on learning domain-invariant features to bridge the gap between varying environmental conditions [9]. Additionally, several works [10,11] advocate for disentangling image representations into separate codes that isolate place-specific cues from appearance and occlusion factors, ensuring reliable place recognition. However, these methods are limited to single-camera localization and rely on image retrieval-based localization. Compared to APR methods, such approaches suffer from significantly higher computational costs and storage requirements due to the need to construct and maintain a retrieval database. This paper enhances APR-based cross-domain localization, including cross-camera scenarios, by proposing a novel cross-domain image generation method. Unlike the prior method with limited image augmentation, our method facilitates diverse cross-domain image generation.

[1]Neural Rerendering in the Wild [CVPR'19]

[2]NeRF in the Wild [CVPR'21]

[3]Ha-NeRF [CVPR'22]

[4]Neural Scene Chronology [CVPR'23]

[5]NeRF On-the-go [CVPR'24]

[6]SpotLessSplats [TOG'25]

[7]Adversarial training for adverse conditions: Robust metric localisation using appearance transfer [ICRA'18]

[8]Night-to-Day Image Translation for Retrieval-based Localization [ICRA'19]

[9]Retrieval-based localization based on domain-invariant feature learning under changing environments [IROS'19]

[10]Adversarial feature disentanglement for place recognition across changing appearance [ICRA'20]

[11]Place Recognition under Occlusion and Changing Appearance via Disentangled Representations [ICRA'23]

2. Figure 5 layout should be improved with a horizontal arrangement: GT, 360, fisheye3, fisheye2, fisheye1, pinhole.

Thank you for your valuable suggestion. We have revised the layout of Figure 5 to present the images in a horizontal arrangement as suggested. The manuscript has been updated accordingly.