LocDiffusion: Identifying Locations on Earth by Diffusing in the Hilbert Space

"This paper proposes a novel spherical positional encoding-decoding scheme to address non-linearity issues, enabling effective optimization and training for geolocation tasks within a diffusion-based model. Additionally, it introduces a new conditional latent diffusion model, LocDiffusion, which achieves state-of-the-art performance in image geolocalization."

This work proposes a diffusion model for image geolocalization (given an image, estimate where in the world the photograph was taken). Most work in this domain is based on either classification (partitioning the world into bins) or image retrieval. The main claim is that it overcomes limitations in spatial resolution caused by the partitioning scheme or the spatial distribution of gallery images. An encoding space is proposed that enables diffusion modeling on the sphere with the goal of reducing artifacts caused by modeling the sphere as a linear space.

1. Authors proposed a spherical positional encoding-decoding framework, which encodes points on a spherical surface into a Hilbert space of Spherical Harmonics coefficients.

2. Authors proposed a UNet architecture to learn conditional diffusion by minimizing a latent KL-divergence loss and train a LocDiffusion model that addresses the image geolocalization task via generation.

This paper tries to address the task of image geolocalization that estimates the geographic location where an image was taken. This paper introduces LocDiffusion, a novel generative approach to image geolocalization that leverages diffusion models to predict locations on Earth from images. The key innovation is a new Spherical Harmonics Dirac Delta (SHDD) encoding-decoding framework that enables diffusion in a Hilbert space while preserving spherical geometry. The authors also propose a Conditional Siren-UNet architecture for learning the conditional backward diffusion process.

This paper introduces a latent diffusion model called LocDiffusion for image geolocalization, aiming to predict precise geolocation on Earth from images with arbitrary resolutions. It proposes a novel spherical position encoding method, Spherical Harmonics Dirac Delta (SHDD) Representation, which encodes geocoordinates on Earth's spherical surface into a Hilbert space of Spherical Harmonics coefficients. The model then decodes these geolocations via a mode-seeking approach. A SirenNet-based architecture is used to learn the conditional backward process in the latent SHDD space by minimizing a latent KL-divergence loss. Experiments are conducted on the Im2GPS3k and YFCC-26k datasets, with comparisons to state-of-the-art methods like GeoCLIP.

General Response 5. Ablation studies In Appendix A.6 we have added more ablation experiments to help understand the behavior of the proposed model. There are three sets of ablation experiments:

(1) Ablation on replacing SHDD with other location encoding/decoding methods

Theoretically we argued in Section 4.1 that existing location encoding methods suffer from sparsity and non-linearity so that they are not suitable for location decoding. This is supported by ablation studies. Please see Table 6 and Table 7 in Appendix A.6. To summarize, we replace SHDD encoder/decoder with two commonly used location encoders (rbf and Sphere2Vec) and train a neural network to decode the diffusion output back to locations. After the replacement, the geolocalization performance is reduced by half on coarser scales, and on finer scales (1km and 25km) the accuracy drops to 0. We further demonstrate that such performance drop comes partially from the non-linearity of these location encodings (another reason is that the neural decoder is not 100% accurate). Adding a small Gaussian noise (variance = 0.01) to a non-SHDD location encoding, the spatial drift of the decoded location can be as large as 102.4km (rbf) and 89.1km (Sphere2Vec), while for SHDD encoding this drift is 5.3km. That means, in the case that the diffusion generated output is not noiseless (which is almost certain), SHDD suffers less.

(2) Ablation on replacing SHDD KL-divergence loss with other losses

We try to train LocDiffusion with alternative losses, such as L1/L2 and cosine distance loss. Please see Table 8 in Appendix A.6. The findings are very interesting: L1/L2 losses are much worse than SHDD KL-divergence, whereas cosine distance loss yields fairly good results on coarser scales. Our hypothesis is that the cosine distance loss is mathematically similar to the SHDD KL-divergence loss (SHDD KL-divergence is the sum of exponential element-wise multiplications, while cosine similarity is the sum of raw element-wise multiplications; the difference is that exponential terms are non-negative and the probability mass is more concentrated, i.e., it should be better than cosine losses on fine-grained scales, but not very different on coarser scales). This may indicate that we can use cosine loss as a computationally friendly alternative as long as we do not insist on high performance on low-error scales. This is also an interesting future work.

(3) Ablation on CS-UNet modules

We have explored many alternatives of the CS-UNet architecture, but unfortunately we did not record them for ablation studies. Due to the limited time given for rebuttal and the long training time and various combinations of alternative modules, we have not been able to conduct a very thorough ablation study on this part. But according to our experience, one of the most important factors is the bottleneck dimension of the CS-UNet: shrinking the bottleneck width w of the CS-UNet seems to help alleviate model overfitting (we can adopt a lower dropout rate) and slightly boost performance, but it makes the model more difficult to train. We managed to do this experiment and reported the results in Table 9 in Appendix A.6.

General Response 4. Computational complexity

The computational complexity of LocDiffusion, like most diffusion models, lies mainly in the diffusion process. The SHDD encoding/decoding is deterministic and learning-free, which is very fast. In fact, if the training data are fixed, we only need to construct a look-up table. Therefore, the time complexity of LocDiffusion is at the same level as the specific diffusion model we employ (for example, in this paper we used the classic DDPM model without sampling acceleration). We potentially can apply more efficient diffusion approaches such as stable diffusion or consistency diffusion.

The space complexity depends on the input and output dimensions of the diffusion model, i.e., the dimensions of SHDD encodings. As we have discussed in General Response 1 and General Response 3, it is both possible to reduce the dimensionality of SHDD encodings by low-pass filtering, and recommended not to use very high dimensional SHDD encodings for the sake of computational stability.

For details on the training and inference time/space costs, please see Appendix A.5. The inference time of LocDiffusion is about 0.05s per query on a single NVIDIA RTX5500 24G GPU, while GeoCLIP about 0.02s. Therefore in practice, the computational complexity of LocDiffusion should not be a huge concern. If we need to scale up, we can apply the well-established sampling acceleration techniques developed for diffusion models.

General Response 3. Addressing the quadratic complexity of SHDD encoding

The Legendre polynomial degree $L$ controls the sizes of spherical harmonics wavelengths and has a significant impact on the resolution (See Figure 4(a) in Appendix A.3). Several reviewers are rightfully concerned that the dimension of SHDD encoding scales quadratically. We developed a low-pass filtering technique to effectively reduce the dimensionality.

Briefly speaking, we find that not all $(L+1)^2$ dimensions are equally important in the computation of KL-divergence and in the decoding. See Figure 4(b) in Appendix A.3 for an illustration. Just like in signal processing, the Fourier features that have very small magnitudes can be discarded by a low-pass filter without significantly changing the shape of the signal; we applied the similar thing to the SHDD encoding. We used $L=47$, got a 2403-dimensional vector, and only kept the 1024 dimensions whose spherical Harmonics terms have top absolute coefficient values. In this way we were able to increase the spatial resolution of LocDiffusion and saw significant performance improvements especially for scale 1km and 25km.

Another possible approach is analogous to nested dropout. For each pass we only evaluate a fixed-size random subset of all the dimensions in hope that as training epochs increase, all dimensions are visited and learned.

General Response 2. Clarification on the contribution of the paper

Our goal in this research is not to beat and replace retrieval/classification based models; instead, we wish to provide a more spatially generalizable alternative and align it with previous works that try to overcome the resolution limitations of bins and galleries. More importantly, apart from the specific application to geolocalization, the design of SHDD encoding/decoding techniques itself may open up research opportunities in other applications where decoding locations is required (as far as we know, it is the first deterministic, learning-free spherical location decoding method).

Here we demonstrate that 1) how existing approaches rely on co-locations between training and testing data and why it may be problematic, 2) how our approach does not suffer from this problem (as demonstrated in Table 2 and Appendix A.4) and 3) by hierarchically using our model as the first stage of retrieval we can improve SOTA approaches (e.g., GeoCLIP) at all scales.

First, we'd like to illustrate that the existing retrieval process used by GeoCLIP (as in Table 1) implicitly relies on a strong assumption that many test images co-locate with the retrieval gallery. To demonstrate, we take the MP16 gallery used by GeoCLIP and compare the locations in the gallery with the locations in the Im2GPS3K test set. We measure how closely test locations align with the gallery locations by counting the number of gallery locations that are within 1km/25km from a given test location. There are 63.5% test locations that can find at least one location in the MP16 gallery that is less than 1 km away, and 95.2% less than 25 km away. This means the “correct answer” is almost always present in the gallery – this is a very strong spatial inductive bias. If we do not have access to such a gallery, for example if we use a set of evenly spaced locations on Earth as the gallery, the two numbers drop to 0.1% 1km away and 38.9% 25 km away, respectively. A detailed analysis can be found in Appendix A.4. This can be very problematic since in real-world settings, such a well-aligned gallery is hard to obtain. GWS15K is an actual example. In the GeoCLIP paper, the low-error resolution performance on GWS15K is significantly worse (0.6 on 1km, 3.1 on 25km) than on Im2GPS3K (14.11 on 1km, 34.47 on 25km). However, both GeoCLIP and PIGEON did not publicize this dataset, and to construct it using Google APIs takes longer time beyond the rebuttal period. We are happy to include this result in the camera-ready version if this paper gets accepted.

Second, we’d like to explain why our LocDiffusion model does not have this issue. SHDD encoding is an implicit representation of a continuous function. One can exactly evaluate the function values at any point on the sphere without the need of any gallery or spatial bins; the only factor that restricts the accuracy of the evaluation is the computational setting (i.e., using how many digits of floats). Instead, retrieval-based methods can only be applied to locations that exist in the gallery. In this sense, our claim in the paper that our method can achieve arbitrary spatial resolution can be confusing comparing to the spatial resolution of SHDD decoding we discussed in General Response 1. We sincerely apologize for the overload of concepts and we will replace the “arbitrary spatial resolution” term with “can be evaluated at arbitrary locations”. The benefit of being independent of a pre-defined gallery is stronger spatial generalization. Table 2 in the main paper and Figure 5 in Appendix A.4 clearly demonstrate that LocDiffusion performs stably well no matter whether the SHDD decoding is evaluated on MP16 gallery points or on evenly distributed grid points.

Finally, as we have also discussed in General Response. 1, **the best strategy may be a hierarchical combination of real-valued diffusion and discrete retrieval/classification (like that of PIGEON): ** use diffusion in coarser scales (>200km) to accurately narrow down the region the image may locate and for further fine scales (< 25km) use a locally trained retrieval model. This will help avoid retrieving from an extremely large gallery that might introduce noise retrieval results because of the needs to be both global and spatially fine-grained. The performance gain of such a combination is presented in the updated Table 1.

General Response 1. Improving the performance of LocDiffusion

As all the reviewers have noticed and brought up in the comments, the performance of LocDiffusion compared with existing retrieval-based methods such as GeoCLIP and PIGEON was not ideal, especially on low-error scales such as 1km and 25km. We were also aware of this and managed to improve it to be on pare with or even outperform SOTA methods. Please see our updated Table 1 in the revised PDF.

The major factor that constrained the performance of LocDiffusion in the previous experiment results is the Legendre polynomial degree $L$. As we have discussed in Section 4.2 and Appendix A.3, the spatial resolution of SHDD decoding, i.e., the variance of SHDD decoding introduced by using a truncated spherical harmonics vector, is dependent on $L$. See Figure 3 in Appendix A.3 for an illustration. In our previous experiment results, while we were aware that larger $L$ brings better performance (See Figure 4(a) in Appendix A.3), we only used $L = 31$ (i.e., SHDD encoding dimension = 1024) due to limited computational resources. The spatial resolution is around 320km. It is therefore very difficult to achieve high performance on fine-grained scales such as 1km and 25km because the intrinsic variance of the decoded locations is much larger.

To overcome this, we developed a dimension reduction technique to effectively use larger $L$. Briefly speaking, we find that not all $(L+1)^2$ dimensions are equally important in the computation of KL-divergence and in the decoding. See Figure 4(b) in Appendix A.3 for an illustration. Just like in signal processing, the Fourier features that have very small magnitudes can be discarded by a low-pass filter without significantly changing the shape of the signal; we applied the similar idea to the SHDD encoding. For example $L=47$ correspond to a 2403-dimensional vector, and we only kept the 1024 dimensions whose spherical Harmonics terms have top absolute coefficient values. In this way we were able to increase the spatial resolution of LocDiffusion and achieve further performance improvements.

However, further increasing $L$ leads to numerical instabilities because the coefficient values get too large and overflow when we calculate their exponentials for probability computation. As shown in Figure 4(b) in Appendix A.3, 2500 is the rough upper-bound of $L$ where we can stably compute the KL-divergence loss and decode an SHDD encoding. Therefore we stopped at 2403 dimensions. Whether we can overcome this hurdle and use even higher spatial resolutions is an interesting future research problem. However, the spatial resolution of $L=47$ is around 200 km, which is still significantly larger than 1km and 25km. Thus, while the performance of LocDiffusion on these two scales is close to GeoCLIP but can not beat it yet.

Given the success of PIGEON (which is a hierarchical geolocalization framework), we explore combining LocDiffusion and GeoCLIP: since LocDiffusion is a generative model and outperforms GeoCLIP on coarser scales (> 200 km) while GeoCLIP outperforms on finer scales (<25km), we can use it as the first stage of geolocalization hierarchically. First, we sample multiple times (e.g., 16) and get a rough distribution of candidate locations, i.e. they indicate where the true answer is highly likely to reside. Then, we restrict the retrieval of GeoCLIP to the neighborhoods (200 km radius) of these candidate locations. We find that this simple combination yields better results than both GeoCLIP and LocDiffusion used alone. This approach is to recommendation systems' retrieve and rerank approach. This combination points to interesting future work, too, and we wish to demonstrate that our generative approach is not meant to beat and replace retrieval-based models; instead, they have their respective advantages and the best strategy is to fuse them into one system.

The submission proposes LocDiffusion, a novel latent diffusion model for image geolocalization, leveraging a spherical positional encoding-decoding scheme (SHDD) to operate in a Hilbert space while preserving Earth's spherical geometry. While the paper presents innovative ideas and introduces a new generative framework for geolocalization, its contributions are undermined by limited experimental improvements, incomplete comparisons to related work, and unresolved concerns raised by reviewers during the discussion period.

Reviewers GNES, Be4r, and X6VD expressed concerns about whether the proposed approach offers tangible benefits over existing methods in terms of accuracy and practical usability. Missing or insufficiently analyzed baselines include well-established methods like PIGEON (CVPR 2024) and other retrieval-based geolocalization approaches. Reviewer X6VD highlighted these omissions as critical weaknesses, questioning the significance of the proposed contributions. The authors’ response addressed some writing concerns but failed to adequately clarify key methodological aspects, as noted by Reviewer S6YX.

Reject