Scaling Image Geo-Localization to Continent Level

We thank the reviewer for their detailed and constructive review. We appreciate that they acknowledge a "leap from previous state-of-the-art" in terms of scale, and that our hybrid method "appears to be original" and that its "synergy overcomes the individual limitations of each approach". The reviewer kindly points to a "strong supplementary material" and experiments that suggest "strong generalization capabilities" to "completely unseen geographic areas" and "a different data domain (pedestrian-view Trekker dataset)". We address their questions and suggestions to improve the paper.

Limited reproducibility: The most significant weakness is that the research relies on a massive, proprietary dataset (Google StreetView) and secondly, the authors state they will not release the pre-trained model weights, while they provide a reasonable ethical justification for not releasing the model. Still, this is a major shortcoming of the work. We acknowledge these concerns. We are however unable to release the data, since we do not own it, nor the models weights, due to privacy concerns. We however commit to releasing the training and evaluation code to make reproduction of this work easier. The supplementary material provides detailed statistics on our data distribution, data processing steps, and implementation details, which make it possible to reproduce our method on a different (sufficiently large) dataset. Our focus is on demonstrating how far we can push the envelope in large-scale geolocalization given sufficient data and resources.

The training requires massive computational resources (128 TPUv2s) making it inaccessible for most researchers to replicate or build upon from scratch. We note that our models are trained on TPUv2, an architecture that was originally released in 2017 (the latest architecture is TPUv6e). With the scalable implementation described in the paper, our model can be trained on the smaller BEDENL dataset (used in the majority of experiments) with only 16 TPUv2, which are equivalent to 4 A100s: this is comparable to the VRAM requirements of recent works [1]. Exactly reproducing our results requires a setup equivalent to 26 A100 GPUs with 80GB VRAM: we acknowledge that this is large, but not necessarily out of reach for academic research labs.

Could the authors provide a learning curve showing how the final model performance (e.g., Recall@1@100m) varies with the size of the training dataset? We agree with the reviewer that this ablation would provide valuable insights, and ease the barrier on reproducing our results. There are two ways to reduce the size of the training dataset: we can either (a) reduce the total number of cells (i.e., geographical coverage), or (b) reduce the density of images within a cell, while maintaining the spatial coverage. An ablation like (a) is analogous to the BEDENL split we use for a large part of our evaluations. Recall is comparable to EuropeWest, suggesting that performance remains robust when increasing the size of the database. This can be attributed to the global discriminativeness of the prototypes. However, the impact of global negatives from classification will diminish as the map size is reduced, as it becomes more likely that hard negatives are part of the batch. As for (b), it is implicitly covered by Figures 3 (main paper: "recall vs training data density") and 7 (supplementary: "localization vs data density"). Classification is susceptible to the density and diversity of ground samples because it learns to remember content observed in each cell, and we would therefore expect a significant performance drop when significantly reducing data density. Cross-view retrieval is less sensitive to this because it is forced to encode visual information from the aerial tiles, and our hybrid method is able to combine the best of both modalities.

Consequently, we claim that our approach can provide substantial improvements even at a smaller scale, thanks to its hybrid nature. This increased robustness is what enables us to run on more cells, with a lower density of images per cell.

Are failures modes due to visual ambiguity, lack of data, or aerial-ground mismatch? This is difficult to answer, as we can only partially back up our intuitions with quantitative data. Figs. 4 and 8 clearly show that our method mostly fails in rural areas with low temporal coverage. In such cases, the method largely relies on cross-view retrieval. Visual ambiguity is also a problem, especially given large occluders, such as the truck covering almost the entire image in Fig. 13 (bottom row): these cases are very difficult to solve. The increased performance on highways (which have better data coverage) suggests that sparsity is the larger problem. Aerial-ground mismatches from the large domain gap were significantly reduced by our method thanks to the hybrid training and global discriminativeness.

Could confidence estimation or uncertainty modeling help reject low-quality predictions? Such predictions could certainly help in filtering false-positive predictions, e.g. unlocalizable images. Implicitly, the ViT already learns which parts of the image are useful for localization. Adding confidence estimation and / or uncertainty modeling during training could help to further downweight unlocalizable images, which yield noisy gradients. During evaluation, the similarity score could be used as a prediction confidence. We believe that all these topics are exciting future research directions.

Could the authors clarify whether the "Prototypes-Only" model was trained independently, or if its results were obtained by omitting aerial features at inference time from the full hybrid model? Please let us clarify: Ours (cell prototypes) and Ours (full) are two different models, trained independently. We evaluate them under different conditions: in Table 1-a ("ground retrieval"), we use either model to do VPR. In Table 1-c ("cell prototypes"), Ours (full) omits the aerial features at inference time, and in Table 1-d ("hybrid") we use the aerial features. We will make this clear in the revision.

[Continued:] ["Ours (full)" being better than "Ours (cell prototypes)" in Table 1-c] would suggest that joint training with aerial supervision improves prototype quality, even when not used at test time, which would be an interesting insight worth highlighting more directly. Correct. This is backed by experiments: the ablation study in Table 7 (supplementary) clearly shows that adding ground-aerial contrastive supervision during training greatly improves the recall rate with just prototypes (i.e., discarding aerials at test time). The aerial data smooths and stabilizes the prototypes, making them more robust to sparsity, both spatially and temporally. We thank the reviewer for pointing this out and we will highlight this insight more prominently in the revision.

Could the authors comment on the feasibility of scaling the method to an entire continent (e.g., all of Europe) or even globally? We performed most of our experiments on 128 TPUv2 (i.e., older-gen) to keep experimentation reasonably fast, and run an extensive set of ablations. With the training recipe and scalable implementation described in the paper, it is possible to run all experiments with half the amount of TPUs on EuropeWest, and just 16 TPUv2 for BEDENL, which was our primary dataset during exploration. Consequently, without any adjustments, the method could scale up to twice the area of EuropeWest (basically all of Europe, excluding Russia). Note that this of course depends on how the data is distributed: some countries are more densely sampled than others. Scaling up beyond that is tricky as the amount of descriptors stored per device scales linearly. Potential techniques to mitigate this include parameter sharing (e.g. setting the first $k$ dimensions of all descriptors in a country identical), multi-stage training (e.g. alternating between cross-country classification and in-country localization), or low-dimensional projections of cell descriptors. For the latter, one could encode prototypes with a location encoder (similar to GeoCLIP [11]), yet provide the network some flexibility to implicitly store / remember information per cell explicitly.

We kindly thank the reviewer for the insightful and constructive feedback. The reviewer acknowledges the "novel architecture and training scheme for visual localization at continent scale", supported by a "detailed quantitative analysis, including a careful ablation study of key design decisions". We also appreciate the comment stating that "the supplementary material is very detailed", recognizing that we invested extra time and effort. We thank them for suggesting improvements in the readability and motivation of the approach (with which we agree).

I think it is necessary to motivate the problem setting a bit more in the paper. We agree that the motivation section should be reworded, and that use-cases such as robotics or augmented reality are not a great fit. However, global, prior-less geo-localization is an important problem, enabling us to localize any images without GPS, such as older / historical images, videos (which usually lack a GPS tag), or images where the EXIF metadata was accidentally stripped, i.e. after image transformations. It can be applied to arbitrary imagery, such as images distributed in the media for validation, and enables criminal investigations or could support the detection of AI generated images. Finally, models trained for large-scale geolocalization can learn high-level semantics (geographical/cultural patterns, see Fig. 14) and could be a valuable pre-training for other tasks (though this is outside the scope of our work).

The dataset cannot be released due to a licensing issue. This will make it difficult to reproduce the paper given the relatively unique problem setting, which requires ground-level panoramas in addition to aerial imagery. We acknowledge these concerns. We are however unable to release the data, since we do not own it, nor the models weights, due to privacy concerns. We however commit to releasing the training and evaluation code. Alongside the detailed description of the data distribution and processing steps in the supplementary, we believe that this should make it possible to reproduce our work on a different (sufficiently large) dataset.

We note that ground-level panoramas are not strictly required. We stitch the images into panoramas and sample crops from them as a simple data augmentation technique, to generate a wide variety of view points and focal lengths. We believe that sufficient diversity in standard imagery would work equally well. Our requirements on aerial imagery are less strict than Fervers et al [29], where the best model was trained on 20cm resolution tiles, compared to 60cm for our method.

[...], the method relies on high-quality aerial imagery for the cross-view matching component. Wouldn't the areas which are clearly visible in aerial imagery be precisely those ones with good GNSS availability? While it is true that these use cases overlap, there are plenty of tasks / data sources where GPS is not available at all, see discussion above.

Dimensionality reduction can be helpful in reducing memory footprint, while approximate nearest neighbor search can be very fast on modern systems while still achieving recall rates of 0.9 @ 10 or similar for 1B+ datasets. We agree with the author that the mentioned statement did not take into account recent advancements in efficient image retrieval applicable during evaluation, and we will tone this down. However, the compression in database size via prototypes would also increase throughput and reduce memory when paired with dimensionality reduction and/or ANN. Furthermore, such techniques are not applicable during training, while we show that compression through cell prototypes reduces memory to a point where we can utilize global negative proxies for supervision, which boosts accuracy.

We contacted the authors of SALAD and asked for variants with lower dimensionality, however, these models are not available anymore. For the final paper version, we consider adding another baseline with smaller descriptors such as EigenPlaces [18] or CosPlace [17], or SALAD with reduced dimensionality if the authors can retrain the models. However, as illustrated in Table 9 (supplementary), we expect them to be significantly weaker than our classification-only baseline evaluated for VPR.

A more detailed explanation of how localization works as "a combination of aerial and cell code retrieval" (cf. L210) would be helpful, [...]. We agree that this section could be clearer. we train the network to find the optimal cell $j*$ by maximizing the similarity between the query and the representations in each modality: $j^* = argmax_{j} (z_i^Q)^T z_j^A$ and $j^* = argmax_{j} (z_i^Q)^T z_j^P$. To combine the search in both modalities, we look for the cell $j^*$ with the highest combined similarity, which is achieved by summing the individual similarities: $j^* = argmax_{j} ((z_i^Q)^T z_j^A + (z_i^Q)^T z_j^P)$. Due to the linearity of the dot product, this is equivalent to first summing the modality-specific representations of the cell and then calculating the similarity with the query: $j^* = argmax_{j} (z_i^Q)^T (z_j^A + z_j^P)$. If we define $z_j^{CELL} = z_j^A + z_j^P$, then the problem becomes $j^* = argmax_{j} (z_i^Q)^T z_j^{CELL}$. Note that all embeddings $z$ are L2-normalized here. We will use some of the extra page afforded in the revision to clarify this, adding an additional figure outlining our training and inference pipeline.

Is the Trekker dataset publicly available? The Trekker dataset is not publicly available, as it has the same legal requirements as the StreetView data used for training. As such, it is subject to takedown requirements, which need to be honored in a timely manner after being triggered. This requires a significant infrastructure investment, as the data owner needs to ensure that all copies are deleted.

We acknowledge that there are alternative ways to make the data "available": for instance, platforms like Kaggle or HuggingFace allow submissions on hidden test sets, and we are currently evaluating this type of scenario. This is, however, not without problems: these competitions still typically require a fair amount of public "training" data, or participants must resort to blind leaderboard probing.

In Eq. (1) it would be helpful to define the Q/A/P superscripts. Indeed this would improve readability, and we acknowledge that this was a point of confusion with multiple reviewers. We will incorporate this feedback in the revision and improve the explanation of Eq. (1).

We kindly thank the reviewer for the positive feedback, stating that the method is a "conceptually simple yet powerful hybrid approach", and highlighting that the proposed combination of aerial descriptors and cell prototypes is "novel" and "sensible". The reviewer also finds that the "training pipeline is interesting" and recognises the "significant engineering effort" and a "very complete evaluation". We address their questions.

Although the paper is, in general, very well written and easy to follow, this reviewer would appreciate more explicit or detailed information about the training and inference process. We thank the reviewer for this valuable feedback. We agree that, while the method is conceptually simple, the paper may not make this sufficiently clear. We will use the extra page afforded by the final version to outline both training and inference in a new pipeline figure that should help convey this message. Notably, we will highlight (a) how aerial and cell prototypes are merged before inference, and (b) the different losses and modalities used during training, summarizing Eqs. (1) and (2).

We also acknowledge that some terms may impair readability, e.g. "cell codes" and "prototypes" (which are the same thing).

The paper proposes to sum cell prototypes and aerial descriptors to obtain the final cell descriptors. However, the model is not trained to match the sum of these descriptors. Indeed, we never train the network to match the (weighted) sum of aerial embeddings and prototypes. However, we train the network to find the containing cell in each modality, i.e. we learn: $j^* = argmax_{j} (z_i^Q)^T z_j^A$ and $j^* = argmax_{j} (z_i^Q)^T z_j^P$. To combine the search in both modalities, we look for the cell $j^*$ with the highest combined similarity, which is achieved by summing the individual similarities: $j^* = argmax_{j} ((z_i^Q)^T z_j^A + (z_i^Q)^T z_j^P)$. Due to the linearity of the dot product, this is equivalent to first summing the modality-specific representations of the cell and then calculating the similarity with the query: $j^* = argmax_{j} (z_i^Q)^T (z_j^A + z_j^P)$. If we define $z_j^{CELL} = z_j^A + z_j^P$, then the problem becomes $j^* = argmax_{j} (z_i^Q)^T z_j^{CELL}$. Note that all embeddings $z$ are L2-normalized here. We will use some of the extra page afforded in the revision to clarify this, adding an additional figure outlining our training and inference pipeline.

They train on images from 2017,18,19,20,21,22,24 and evaluate on 2023, whereas a more realistic situation would be to train on 2017-2023 and evaluate on 2024. The explanation is very simple: we started this project in late 2024, so we did not have a full year of data for 2024 at the time (StreetView data needs to be filtered and posed, which takes time). We chose 2023 for evaluation because it gave us the best possible coverage at the time, allowing us to evaluate on most areas we train on. We did not think to reevaluate this decision later on. We agree that we may somewhat overestimate the model's robustness against long-term temporal changes, but we believe this is a minor factor.

SALAD with its biggest descriptors is unreasonable to run (5TB), but what about SALAD with smaller descriptors? (Table 1) We contacted the authors of SALAD, who declared that the weights of the more compact models (described in the SALAD paper) had unfortunately been lost.The model would thus have to be retrained. However, we refer to our evaluation in Table 9 (supplementary material) which shows that our retrieval baselines significantly outperform SALAD with large descriptors (D=8448). A smaller SALAD (D=2176) would likely be even weaker. (Please refer to our response to R1 / f1mU for more details.)

The model is never trained to match ground-ground descriptor and the training loss does not include a component that brings together two similar ground locations. However the model achieves outstanding metrics on the VPR modality. Can the authors give details or intuitions about this? Classification serves as a proxy task for image retrieval, as it puts all image embeddings belonging to the same class close to each other [17]. As we note in the paper (L81), some of the strongest image retrieval models are trained with a classification loss [17, 18].

Can the authors give more details on what Figure 5 (main paper) and Figure 12 (supplementary) represent? We apologize for the lack of clarity, which we will improve in the final version of the paper.

Fig. 5 shows 36 query images sorted by their rank, which describes the position of the first positive cell (within 200m of the query) in the list of database cells sorted by descriptor similarity (from the hybrid database). In other words, a query with rank K has K-1 false positive predictions.. A larger rank corresponds to a lower localization accuracy. Most queries with low rank (but not all) contain a fair amount of structure, whereas images showing mostly trees or sky are difficult to localize.

Fig. 12 illustrates the localization error obtained from different variants of our model. We show a ground-level query (which is repeated 3 times), and the aerial tile for the cell the query belongs to in a). The 3 columns (prototypes, aerial, full) denote the model used for retrieval, with “full” being the hybrid (aerial + prototypes) database. Overlaid on each query image, we report the distance to the top-1 nearest neighbor (i.e., its error). The border colors help illustrate this error, where green means low error. Calling the first column "input" is misleading, we will correct this (e.g. replacing it with “cell”).

Table I (c): what’s the difference between Ours (cell prototypes) and Ours (full)? Is the training process where Ours (cell prototypes) ignores the aerial edges, and ours (full) is the full training but not summing the aerial descriptors? Yes, Ours (cell prototypes) is a pure classification method without aerial edges, and Ours (full) is the full training (with aerial edges), but ignoring aerial descriptors during evaluation. We will try to highlight this setup in the final paper version.

We appreciate the reviewer's warm reception of our paper, who recognized its "high" novelty, scope ("enables meaningful evaluation of geolocation methods at a scale relevant for real-world deployment"), ("very high") performance, applicability ("suitable for large-scale deployment"), and generalization capabilities (to a subset of modalities or new viewpoints). We answer their questions.

What is the status (and upper bound) of the current state-of-the-art? Please note that we flag the top row in Table 1 as OOM primarily to drive home the point that retrieval is an inefficient approach to solve geolocalization at this scale. Moreover, in Table 9 (supplementary) we show that pre-trained SALAD is significantly weaker than our best model in the VPR evaluation, even out-of-domain (i.e. cross-area). In-domain, this gap would only increase. Thus, SALAD with FAISS / ANN would in all likelihood be a weaker baseline than our best VPR method. We reached out to the authors of SALAD, but unfortunately the weights of the smaller SALAD variants are currently unavailable. Retraining SALAD on our datasets would be insightful, but this requires defining positive anchors i.e. spatially close pairs and careful engineering & tuning, which is beyond the scope of the rebuttal.

It is unclear whether baselines are treated in the same way as the proposed approach. Table 1 shows that classification-based baselines underperform significantly, which is somewhat surprising. The classification baselines labeled as Hierarchical and Haversine rely on the cross-entropy loss, which we indeed found to significantly underperform the multi-similarity loss, on which the third baseline relies: Ours (prototypes-only).Note that we consider it a baseline because our main contribution is on hybrid geolocalization. We will clarify this in the final version.

All classification baselines were trained on the exact same data, and we carefully tuned them. We observed that standard softmax-based classification results in significant overfitting, which we were not able to resolve with a lower or learned temperature. Our intuition is that visual aliasing in combination with low data diversity leads to this behaviour. The network merely remembers "viewpoints" rather than learning more robust patterns. To reduce this, we mask out cells that are spatially too close to the query (L144), which improves accuracy. We also observed that our loss is more robust to unlocalizable images, likely because of supervising absolute- instead of relative similarities.

More implementation details of our baselines are listed in the supplementary, section C.1.

...methods with alternative training paradigms like GeoCLIP [11] or the diffusion-based method in [15] can be included. We thank the reviewer for suggesting additional baselines. We did re-implement GeoCLIP [11] during our research, but failed to obtain meaningful results with it. GeoCLIP was designed for much coarser localization (>10km) and its positional encoding is thus not suitable to capture high-frequency details required for fine-grained geolocalization. To alleviate this, we: (a) increased the size of the location encoder, and (b) replaced the positional encoding with SIREN [C]. Even then the method lagged behind the already weak classification baselines. While we believe that it should be possible to solve this problem, it requires a non-trivial amount of research and we opted to not pursue it further. We did not include these results in the paper because they are not fully-realized, but we will make a best-faith effort to further develop this baseline for the camera-ready if the paper is accepted.

The diffusion-based approach by Dufour et al [15] is also designed for much coarser global localization and, like GeoCLIP, would likely require significant adjustments to work well in our setting.

[A]: This work suggests a Search-within-Cell (SwC) scheme, where coarse classification is refined with retrieval and clustering to database images within this cell. This is orthogonal to our approach.

[B]: This work is closely related to ours, and we thank the reviewer for pointing it out. Notably, they also use a contrastive loss (ArcFace) for classification, which outperforms cross-entropy losses. This agrees with our findings. We applied ArcFace and CosFace on classification in an earlier stage of our project but we observed that they were not as effective as the multi-similarity loss (baseline Ours (prototypes)). Due to time and resource constraints, we are not able to rerun these experiments in the short rebuttal period but we will add them in the final version.

[B] also introduces a mixture-of-experts approach to mitigate visual aliasing. In our work, we avoid visual aliasing by masking out negatives that are spatially close to the query, which is conceptually simpler. The suggested "mixed pipeline", which performs retrieval on the images from the top-k classes (=cells), would also apply to our method, and is thus orthogonal.

The impact of the work would be increased if more cross-domain/cross-dataset evaluation is provided based on popular datasets. Public datasets are designed for a very different problem: Finding patterns that are common in a certain geographic area (e.g. country, city). While such cues are powerful, they do not support fine-grained localization, i.e. finding the right corner of a street. Therefore, while the end goal is similar, comparing our method to public works on these benchmarks is of limited value.

While we agree that further evaluation on a wider variety of real-world sensors and viewpoint conditions would be valuable, we believe the proposed datasets, IM2GPS3K and YFCC4K, are not suitable:

i) They are noisy, with a very large ratio of unlocalizable images (faces, food, indoor scenes). Results would be hardly comparable without extensive manual filtering. Failures due to unlocalizable images or different intrinsics would be indistinguishable. Nevertheless, we consider adding an ablation of FoV vs. recall on our dataset to show the models robustness to varying intrinsics. .

ii) The granularity gap with literature is too large, even for in-domain images, and the results would likely just highlight this fact, which is of little value. We believe that retraining public baselines on our (cleaner & larger) data, as we do in Table 1, is more interesting.

iii) As the reviewer notes, our evaluation is thorough: we report the performance against viewpoint changes (Table 4), data sparsity & diversity (Table 3), geographical variations (Figs. 5 & 8), and even population density (Fig. 9), which clearly illustrate where our method improves or fails. We agree that alternative devices are missing in this evaluation, but to the best of our knowledge, there is no public dataset with sufficient quality and coverage.

The selection threshold of negative prototypes is missing. We exclude any negative cells within $d=200m$ to address visual aliasing, i.e., queries that observe content over multiple cells. Without this threshold, the classification accuracy drops significantly. We apologize for this oversight, and will add a more extensive sensitivity analysis to the paper.

In Eq. 2, the loss includes two similarity terms between the ground and aerial embeddings… Does it impact the training, or could it be removed to simplify the loss? Indeed, one of the terms is implicitly optimized from another anchor in the batch. Excluding this term would certainly change the relative weight between contrastive and global terms. We have not conducted an ablation of this on our full model, however, in the SALAD aerial baseline, we found that this "bidirectional" variant slightly boosts accuracy over the unidirectional variant (where the term $z_i^A$ is excluded). Fervers et al [29] also uses a bidirectional loss.

Aerial image source and characteristics are underspecified. In general, we prefer the highest quality and most recent capture available for each location, determined by the geometric resolution of the aerial tiles. The imagery is owned by Google and not available externally. The orthophotos have a resolution of 10cm or 25cm (which we downsample to 60cm), stitched from oblique views using a high-quality digital surface model. The data was captured over the last 10+ years, and is the same data visible on Google Maps / Google Earth at the highest zoom level. We will add these details to the paper.

Reproducibility remains a concern. We acknowledge these concerns. We are however unable to release the data, since we do not own it, nor the models weights, due to privacy concerns. We however commit to releasing the training and evaluation code to make reproduction of this work easier. The supplementary material provides detailed statistics on our data distribution, data processing steps, and implementation details.

extensive [TPU] compute resources[...] With the scalable implementation described in the paper, our model can be trained on the smaller BEDENL dataset with only 16 TPUv2 (an older-gen model originally released in 2017), which are equivalent to 4 A100. This is comparable to the VRAM requirements of recent works [1].

It would be very beneficial if the authors could provide some way for the community to submit solutions (e.g., code and trained model) for the evaluation on the composed dataset. There are indeed ways to do this, as platforms like Kaggle or HuggingFace allow submissions on hidden test sets. We thank the reviewer for this suggestion, and we are actively evaluating similar scenarios. However, we note that these competitions still typically require a fair amount of public "training" data, or participants resort to blind leaderboard probing. This is difficult, as the data owners have to comply by law with takedown requirements.

[C] Implicit Neural Representations with Periodic Activation Functions, Sitzmann et al, NeurIPS, 2020.