LocDiff: Identifying Locations on Earth by Diffusing in the Hilbert Space

Dear reviewer,

Thank you for your comments and suggestions! We will address your concerns in the following responses, respectively.

Rebuttal Table 1: SHDD Encoding/Decoding Time Efficiency

R1. Comparison of Inference Time: Thank you for the question! We will discuss the computational costs of our method with regard to the SHDD encoding/decoding and the diffusion process, respectively.

First of all, we wish to clarify that SHDD encoding/decoding is NOT a computational bottleneck. While the spherical harmonics has very complex formulas and some definitions use recursive equations, we adopt the closed-form precomputation code implemented in [6], which effectively reduces the SHDD encoding into a lookup table. For example, the $d$-dimensional ($d=256, 576, 1024$) SHDD encoding of a point is just a parallel evaluation of a list of two-variable functions, whose time complexity is $O(1)$ and space complexity is $O(d)$. In practice, constructing such a lookup table of 4 million training samples sequentially on CPU only takes 18 minutes in the dataloading stage, and there is no further computational overhead afterwards. SHDD decoding is implemented as a matrix multiplication followed by an argmax operation, which is also very efficient under PyTorch. Please see Rebuttal Table 1.

Secondly, we wish to clarify the major factors that affect the inference speed. In Appendix A.7., we reported the inference time of our method and GeoCLIP: 0.056s (ours) v.s. 0.024s (GeoCLIP). In practice, both our method and GeoCLIP used pretrained image embeddings of CLIP, i.e., we are not actively running a CLIP model to encode an image into an embedding. This step is the true bottleneck. On our 24GB GPU machine, processing one 1024x1024 image through CLIP takes 0.15s, way slower than the diffusion inference process. It is a similar situation with other baselines that uses pretrained image encoders.

Other than that, the current inference setting is to run a 200-step DDPM sampling 16 times and get the ensemble prediction. It is possible to reduce the total inference time by (1) using DDIM to reduce the sampling steps, (2) incorporating the prediction collapsing strategy in [7] to reduce the number of ensembles. We will explore these possibilities in future work.

R2. Clarification of LocDiff-H: Thank you for your suggestion! We will modify the abstract and the introduction to explicitly acknowledge in what scenarios and to what extent our SOTA results rely on GeoCLIP in the camera-ready version.

[6] Geographic Location Encoding with Spherical Harmonics and Sinusoidal Representation Networks. Rußwurm et al. ICLR 2025. [7] Around the World in 80 Timesteps: A Generative Approach to Global Visual Geolocation, Dufour et al, CVPR 2025

Dear reviewer,

Thank you for your comments and suggestions! We will address your concerns in the following responses, respectively.

R1. Responses to Weaknesses:

• Quality. Most hyperparameters are chosen by grid search. As to the Legendre polynomial degree $L$, it is chosen by systematically analyzing the trade-off between spatial resolution and computational stability. Please see Appendix A.4, especially Figure 5(b), for a visual explanation. $L=47$ is the maximum degree where we can still stably train the model.

• Clarity. Thank you for this suggestion. Per the rebuttal rules this year, we can not upload any PDF or links to visual examples, but we will include more illustrative examples and pseudo codes to help readers follow our logic in our camera-ready version.

• Significance. Thank you for this suggestion. We will add more discussions on the societal impacts of geo-localization tasks in the camera-ready version.

• Originality. The hybrid approach, which is effectively a hierarchical combination of our novel method and existing retrieval-based methods, is mostly chosen to reduce the computational complexity of the solution. It does not hurt the originality of our approach. In fact, Table 3 clearly shows that our method bridges the generalizability gaps of existing methods.

R2. Rationale for L=47: The rationale for choosing the maximum degree to be $L=47$ is mostly based on computational feasibility. We find that, while increasing $L$ does increase the spatial resolution of SHDD representations (Appendix A.4 Figure 5(a)), the computational instability also increases. Appendix A.4, especially Figure 5(b) gives a clear explanation for the choice of $L=47$: when $L$ surpasses this threshold, the absolute values of the representation surge to over $10^5$, which makes computing KL-divergence (involving exponential operations) impossible.

R3. Generalizability: In fact, the improved generalizability is one of our key contributions. Table 3 carefully discusses how the data inductive bias of MP16 affects the performance of existing methods such as GeoCLIP, whereas our method remains highly generalizable. Experiments in Table 2 on other datasets such as OSV-5M and YFCC also prove that our method can outperform baselines in more geographically diverse scenarios.

R4. Scalability: Appendix A.4 and A.7 present a careful analysis of the trade-off between spatial resolution and computational efficiency. In fact, the improvement of spatial resolution as $L$ goes up is gradually offset by the side effect of computational instability, which we already discussed in R1. Our strategy is to constrain $L$ below 47, and to improve the finer-scale performance by hierarchically incorporating GeoCLIP, as is discussed in Appendix A.5. In this way, we can avoid high computational costs.

R5. Inductive Bias of Gallery: This effect is discussed in detail in Section 5.2 and Appendix A.6. In summary, the hybrid approach is sensitive to the mismatch between the candidate gallery and the ground-truth. Thus we recommend to use the hybrid approach (LocDiff-H) only if we know that the gallery is a good fit, and to use the base method (LocDiff) for general cases.

R6. Comparison to Baselines: Please see Table 1, where PIGEON is a baseline that we compared to.

Dear reviewer,

Thank you for your comments and suggestions! We will address your concerns in the following responses, respectively.

Rebuttal Table 1: SHDD Encoding/Decoding Time Efficiency

R1. Potential computational overhead: The SHDD encoding/decoding are both deterministic, closed-form operators and do not incur huge computational cost. For example, the $d$-dimensional ($d=256, 576, 1024$) SHDD encoding of a point is just a parallel evaluation of a list of two-variable functions, whose time complexity is $O(1)$ and space complexity is $O(d)$. Not to mention that during training, the SHDD encodings can be precomputed and used as an embedding lookup table. In practice, constructing such a lookup table of 4 million training samples sequentially on CPU only takes 18 minutes in the dataloading stage, and there is no further computational overhead afterwards. SHDD decoding is implemented as a matrix multiplication followed by an argmax operation, which is also very efficient under PyTorch. Please see Rebuttal Table 1.

As for the diffusion process, we used a standard latent diffusion framework and a tailored CS-UNet model, which is much smaller than common CNN-based UNets and runs way faster. The training/inference time on a 600,000 subset of MP16 is reported in Appendix A.7. As an example of the baseline efficiency, the GeoCLIP model takes 227s per training epoch (v.s. Ours 388s) and 0.024s per image inference (v.s. Ours 0.056s). Due to the time constraints of the rebuttal, we can not reproduce all the baselines cited in our paper and report their time efficiency, but we will add these results to our camera-ready version.

R2. Other representations: We wish to clarify that the location encoding methods introduced in A.2 and the implementations of RBF and Sphere2Vec are both based on [4], the most recent and commonly used location encoding benchmark. While not fully reported in our Appendix, RBF and Sphere2Vec are the best-performing location encoders in their respective genres (RBF is a 2-D location encoder, while Sphere2Vec is a 3-D location encoder; see Table 1 in [4]). We will extend Appendix A.8.1 Table 6 and Table 7 to include the results of all representation methods in our camera-ready version.

Appendix A.8.1 Table 6 and Table 7 shows that the collapse in performance is a result of large perturbation drifts during decoding – traditional location encoding methods use non-linear layers to up-project coordinates and make decoding very inaccurate. As we clarified, the RBF and Sphere2Vec implementations both come from the official codebase of TorchSpatial, which is published in NeurIPS 2024 and widely used in other research projects. It is most likely properly implemented.

R3. Similarity of Dirac delta functions: Spherical Dirac delta functions can be seen as probability distributions on the sphere. Thus the natural similarity measurement is the statistical distance between distributions, such as the KL-divergence that we adopt in our paper. Please refer to Eq. 9 where we give the closed-form expression the reviewer requested for.

R4. Explaining notations: Please allow us to clarify the notations.

• (1) Spherical function $F$ and $\delta$ in Line 214. As we introduced in Section 3, any functions defined on a sphere can be uniquely represented as spherical harmonics coefficients. We denote such an arbitrary function as $F$. Among these functions, there is a special type of functions called spherical Dirac delta functions, which we denote as $\delta$, and they uniquely correspond to points on the sphere. The two notations tightly connect to image geo-location in the following way: to identify the geo-location (i.e., a point on the sphere) can be seen as finding the spherical Dirac delta function $\delta$ that corresponds to the point. A diffusion process is effectively altering a random guess (i.e., an arbitrary function) $F$ step by step to approximate the spherical Dirac delta function $\delta$. In this sense, both $F$ and $\delta$ are location embeddings.

• (2) We propose to use KL-divergence to measure the similarity between $F$ (our diffusion prediction) and $\delta$ (the ground truth), which requires both $F$ and $\delta$ to be probability distributions. Therefore, we normalize $F$ into $q_e$ and $\delta$ into $p(\theta, \phi)$, so that we can compute their KL-divergence and use it as the loss for back-propogation. In the context of geo-localization, $p(\theta, \phi)$ is the ground-truth probability of observing a certain image, while $q_e$ is the guess from the diffusion model.

R5. Metrics in Table 1 and Table 2: The metrics are the threshold-based accuracy of geo-localization, i.e., the percentage of model predictions that are less than $x$ km away from the ground-truth, $x=1, 25, 200, 750, 2500$, respectively. This is the conventional metric for geo-localization [5]. Thank you for the suggestion! We will update the captions in our camera-ready version.

[4] TorchSpatial: A Location Encoding Framework and Benchmark for Spatial Representation Learning. Wu et al., NeurIPS 2024.

[5] GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization. Vivanco Cepeda et al. NeurIPS 2023.

Dear reviewer,

Thank you for your comments and suggestions! We will address your concerns in the following responses, respectively.

R1. Prediction Averaging Strategy: We wish to first clarify the way we average samples for prediction, which may be a bit unclear in the original manuscript (Section 5.1). We do NOT average multiple locations from one DDPM sampling; instead, we run multiple DDPM samplings given different augmentations of the test image, use the highest mode in each distribution as the prediction, and average the predictions. This is a successful strategy proven by GeoCLIP[2] and SimCLR[3]. For example, in our experiments, without this averaging step, the 1km accuracy of GeoCLIP on Im2GPS3K may drop from 14% to below 10%. We will improve this part of writing in our camera-ready version.

Then we wish to recognize that the strategy of strengthening guidance to collapse predictions sounds very promising – especially considering it may potentially reduce the Legendre polynomial degree needed to achieve the desired spatial resolution by forcing the probability mass to concentrate (of course, trading off the global coverage). However, as the reviewer has pointed out, the training framework of our method is very different from [1] (e.g., we do not have a hyperparameter to control the guidance strength). To adopt this strategy requires re-designing the pipeline and training a diffusion model from scratch, which is unfortunately not possible within the rebuttal period. We will implement this and report the corresponding results in our camera-ready version.

R2. Benchmark Contamination and Data Source: Sorry for the confusion. The current results reported in Table 2 are based on models trained and evaluated on OSV-5M/YFCC, respectively, following the experimental setup in [1]. Thus our results should be directly comparable to [1].

R3. Proof of SHDD Space Efficacy: This is a very helpful suggestion. In fact, when we designed our framework, we were aware that the generative process can be latent diffusion, flow-matching, or even autoregressive models, as long as it operates in the SHDD space. As is stated in our response R1, due to time constraints, we will add this part of ablation studies in our camera-ready version.

R4. Evaluation of Generative Aspect: As is shown in our Eq. 9 (Line 219), the training objective of our method is actually one kind of probabilistic metrics, i.e., the KL-divergence from the predicted distribution to the ground-truth. It is evidence that our method is not simply collapsing the probabilistic space to improve geolocalization performance. Since we do not have the results for iNat21, we only report the NLL metrics on OSV-5M and YFCC, following the setup of Table 2 in [1].

Rebuttal Table 0: Probabilistic Metrics

We thank the reviewer for pointing us to the generative metrics proposed in [1], and we will include a complete table of generative metrics in our camera-ready version.

R5. Lack of Samples: Thank you for this suggestion. Per the rebuttal rules this year, we can not upload any PDF or links to visual examples, but we will include more illustrative examples in our camera-ready version.

R6. Inference Efficiency Claim: First, we wish to recognize that the statement about the inference efficiency in our introduction (Line 58-60) is inaccurate. What we intended to say is that the re-projection in the non-Riemannian framework introduces extra operations in each noise-adding/removing step and it is more computationally expensive than conventional diffusion/flow-matching in the Euclidean space. We will correct this error in the camera-ready version.

Rebuttal Table 1: SHDD Encoding/Decoding Time Efficiency

Rebuttal Table 1 gives an intuitive sense of our time efficiency. Many of the baselines have not reported their inference time, but as an example, we know that the inference time per image of GeoCLIP is 0.024s. We wish to briefly discuss the factors that truly affect our training/inference efficiency.

First, we wish to clarify that the 200-step sampling reported in our paper is DDPM to match our training setup, while the 16-step sampling in [1] is DDIM. It is possible to reduce our sampling steps to 16 using DDIM.

Then, we wish to clarify the misunderstanding about dimensionality. Both [1] and our method use 3-$d$ (Euclidean xyz) or 2-$d$ (spherical longitude and latitude) as model input. The difference is: for [1], the width of the network is 4x512=2048 (see Figure B. in [1]); for our method, the width of the network is 2x$d$ (see Figure 2 in our paper), where $d$ is the deterministic SHDD encoder dimension, ranging from 256, 576 to 1024, i.e., the maximum network width is the same (2x1024=2048) as [1]. We do not scale up to 4096 dimensions, as is discussed in Appendix A.4.

Finally, we wish to emphasize that using an SHDD encoded multi-scale representation helps the diffusion process to converge faster, because each dimension encodes information at the respective spatial scales. For example, if at a certain step, the generated SHDD representation is 500km within the ground-truth, then the dimensions with larger than 500km scales will remain mostly intact, and only the finer dimensions need to be altered. In practice, our model takes about 2 million steps to converge on YFCC, while the best results reported in [1] take 10 million steps. However, as [1] indicates that FM $\mathbb{R}^3$ outperforms Diff $\mathbb{R}^3$ at some scales, it is worth combining the flow-matching framework with our SHDD multi-scale representation, which may result in even faster convergence and better performance.

R7. Pseudo Code for Readers: This is a very important suggestion. We will add a pseudo code block in the camera-ready version.

[1] Around the World in 80 Timesteps: A Generative Approach to Global Visual Geolocation, Dufour et al, CVPR 2025

[2] GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization. Vivanco Cepeda et al. NeurIPS 2023.

[3] A Simple Framework for Contrastive Learning of Visual Representations. Chen et al.. ICML 2020.