Weakly-supervised Camera Localization by Ground-to-satellite Image Registration

We thank Reviewer-db9o for the valuable comments. Please find our response below, and let us know your further questions.

1. problem setting.

Thank you for suggesting the semi-supervised application. Our method can be easily adapted to address the scenario when a small amount of training data with accurate pose labels is available by adding additional supervision to the network with this amount of data.

Below, we show the performance of our method when fine-tuned with different portions of the current training dataset with accurate pose labels. Data amount = 0 indicates the original weakly supervised pipeline where no such data is available.

The results show that fine-tuning leads to better performance when the data amount with accurate pose labels is over 50%. However, when the data amount is smaller, the fine-tuning leads to inferior performance. We suspect this is because the model overfits the limited training data with accurate labels, causing a loss of general inference ability on other data.

We agree a properly designed training method should address this problem, and we would like to explore this further in our future work.

Apart from this semi-supervised scenario, we believe the original weakly-supervised application addressed in this paper is also of high importance. It avoids additional effort to select the portion of accurate data, as most localization algorithms and noisy GPS sensors themselves do not provide a measure of whether its prediction is accurate or not.

2. Consider larger errors for the training images.

We are sorry that we did not really understand the question. Let us try our best to explain something according to our understanding, and please let us know your further questions.

When the location error of training images is larger, we can increase the coverage of the corresponding satellite image to make it still cover the query image's local surroundings. Our framework is still applicable.

When the satellite image is an incorrect match of the query (the actual location of the camera is outside the region covered by the satellite image, and this is not known), our method still compares the similarity between the ground and satellite image across locations on the satellite image, and find the location corresponds to the largest similarity (though the largest similarity is very small).

When the satellite image covers the local surroundings of the ground camera, while due to time differences between the two-view images, the ground structure might be changed or the illumination is significantly different (day and night), our method behaves similarly as above. It still compares the similarity between the two view images across possible locations on the satellite image and finds the location corresponds to the largest (though very small) similarity.

3. The motivation part could be improved.

Thanks for the suggestion. We have rewritten the introduction to make the motivation clearer. The revised content is marked as blue.

4. RTK is not explained when it first appears.

We have added the full description of RTK in the abstract: Real Time Kinematics (RTK)

5. How to obtain more training data with less accuracy requirement on the label?

A consumer-level mobile phone with built-in GPS, or a consumer-level noisy GPS equipped in a vehicle with a camera, can help to obtain more training data.

The papers below suggest that GPS-enabled smartphones are typically accurate to within a 4.9m radius under open sky and within around 20m in dense high rise environments. The experiments conducted in this paper set the GPS error to be within a 56m x 56m region (a 28m radius). Thus, it is easy to use consumer-level smartphones to collect more data for network training.

[1]. Menard T, Miller J, Mowak M, Norris D. Comparing the GPS capabilities of the Samsung Galaxy S, Motorola Droid X, and the Apple iPhone for vehicle tracking using FreeSim_Mobile. 14th International IEEE Conference of Intelligent Transportation Systems. 2011, Oct. Washington, DC. [2]. Mok E, Retcher G, Wen C. Initial test on the use of GPS and sensor data on modern smartphones for vehicle tracking in dense high rise environments. Proceedings of the Ubiquitous Positioning, Indoor Navigation, and Location Based Services 2012. 2012, Oct. Helsinki, Finland.

We thank Reviewer-mQu3 for the valuable comments. Please find our response below, and let us know your further questions.

Suggestion: Improve the introduction and make contributions clear to representation learning.

Thank you very much for the suggestion. We have rewritten the introduction part and labeled the revised text as blue.

Incorrect statements:

1: "Neural networks are inherently sensitive to rotations." Equivariant networks address this problem.

This description is avoided after we rewrite the introduction.

2: "synthesized overhead view image" gives the reader the impression that satellite images are always "overhead view" orthographic images, which of course is not the case

Thank you for the suggestion. We have added a footnote in this paper, clarifying that the satellite images used for ground camera localization are roughly orthographic.

3: "magnitude of the rotation" ?

We have changed this description to "magnitude of the rotation angle".

4: "the equivariance property of convolution to translations ..."

We have changed the description to "the equivariance property of the convolution operation to translations".

5: Eq 2 describes the correlation coefficient, but later on page 14, a convolution was used

We have changed the terms to make them consistent.

Questions:

1: Parameterization of \mathbf{R} in Eq. 1 and Eq. 1 needs more precise writing.

Thanks for the suggestion. We have changed Eq.1 to use the original angle representation instead of $\mathbf{R}$ in the loss. The revised content is marked as blue.

2: Would it not be benificial to minimise $\mathbf{R}^{-1}\mathbf{R}$ → $\mathbf{I}$ ?, as $\mathbf{R}$ is SO(3)

Thanks for the suggestion. We make our network directly output the angle instead of a rotation matrix and thus do not need to apply this constraint to the network output.

3: Satellite image resolution is 512 x 512. What if satellite images are orders of magnitude larger?

In this work, we consider refining a coarse location of a ground image by ground-to-satellite registration. Thus, given this coarse location and its statistical error, we can extract a satellite image that covers the local surroundings of the query image. In this work, the satellite image coverage is around 100m x 100m (with a resolution of 512 x 512).

When the location ambiguity of a ground image is significantly large, for example, at the city or state level, cross-view retrieval is usually applied. We discuss this line of work in the first paragraph of the Related Work section.

We thank Reviewer-9PqU for the valuable comments. Please find our response below, and let us know your further questions.

1: More Justification on the BEV synthesis in Sec. 3.

Our method does not aim to learn viewpoint invariant features robust to parallax and occlusion. Instead, we want to explore the co-visible regions between the two views and weaken other regions' effect in similarity matching. The co-visible regions mainly concentrate on the ground plane and thus are handled by the ground plane's homography projection. For regions that are not co-visible, we resort to the confidence map to lower their weights in similarity matching. The appearance discrepancy for the co-visible areas is handled by the end-to-end training.

We discussed why using this simple projection instead of a powerful yet complex cross-view transformer for overhead-view synthesis in Sec. A of the appendix. The main reason is a balance between batch size and the model complexity. A large batch size is important in self/weakly supervised learning by contrastive learning. We also visualized the projected and corresponding satellite feature maps in Sec. B. The ground structures co-visible by both views are aligned well. More discussions are presented in Sec. C.

We follow previous works for data augmentation: none is applied to ground images, while satellite images are randomly rotated.

2: Similar to a missing reference OrienterNet (CVPR23)?

Thank you for pointing out this accidentally missed reference. We have added it back to the paper.

Both works synthesize an overhead-view feature map from the ground-view image and match it against the reference satellite feature map for translation estimation. The differences are:

1. OrienterNet rotates the synthesized overhead view feature map K times and then matches them to the reference satellite image. To avoid the additional search space caused by rotation and obtain continuous rotation estimates, we present a pose regressor for rotation estimation.
2. Our pose regressor leverages the sensitivity of a normal neural network to the rotation of input signals, and we introduce a self-supervised method for this pose regressor training. Our rotation estimation performance is significantly better than OrienterNet.
3. Our method does not require ground truth location labels of ground images for network training.

3: Why precise supervision is difficult? Precise reconstruction can be done from a mobile phone and then register a sequence of images to the satellite images.

Precise reconstruction can be obtained from a sequence of ground view images by SfM. However, the GPS of a mobile phone is often noisy. Thus, the absolute geo-location of the reconstructed 3D structure is unknown. Registering reconstructed 3D structures to a satellite image requires complex engineering techniques. From our investigation, there is no readily available technique for this. This effort can be avoided by using the method proposed in this paper. The original noisy GPS signals of ground images can be directly used for network training.

4: Inferior performance to OrienterNet?

OrienterNet reports its performance on Test-2 of the KITTI dataset with a single query image, and the model is trained on the original training dataset of KITTI (the same setting as ours). The comparison results are as follows:

The results of OrienterNet are from Tab.3 of its original paper (OrienterNet(c)).

Except for the longitudinal pose estimation performance, which is inherently ambiguous in ground-to-satellite image matching, our method achieves significantly better lateral pose and orientation estimation than OrienterNet, even though OrienterNet leverages the ground truth poses for training. In contrast, our method does not have access to this.

5: Better written with deeper insights.

Thank you for the suggestion. We have revised our introduction, marked as blue, to convey this paper's idea better.

6: Performance with supervised training.

The performance of our method with full supervision is as follows:

The performance improves significantly for the same-area evaluation (Test-1), while it suffers slight degradation on the cross-area evaluation (Test-2). This is similar to the observation on other fully-supervised methods: models tend to overfit on the training area when ground truth poses are available and further demonstrates that the proposed weakly-supervised approach exhibits strong generalization ability.