GeoLLM: Extracting Geospatial Knowledge from Large Language Models

Thank you for acknowledging the novelty of GeoLLM, the clear use cases and motivation, and constructive criticism. Your suggestions have provided valuable directions for further strengthening our work. Please find our responses to each of your points below:

Q1: Use of Stronger Baselines

A1: Please see “Stronger baselines” in the overall response. We think that “Full set of ablations” in the overall response may also be helpful.

Q2: Zero-shot Capabilities of LLMs

A2: Please see “Zero-shot and few-shot experiments” in the overall response.

Q3: Complementarity with Available Covariates

A3: We added an additional baseline “Nighlight” in Table 1 and Table 3. Please refer to “Stronger baselines” in our general response for the description and discussion of this baseline. The Nightlight baseline provides insights into the baseline performance attainable using a well-established satellite image based method. Notably, LLMs seem to be far more robust on a wide variety of tasks when compared to this baseline. Somewhat predictably, Nightlight is only particularly strong on the population tasks. This supports the idea that LLMs are a richer source of knowledge and a better covariate compared to the well-established nightlight satellite imagery. In practice, GeoLLM, satellite imagery, and other covariates would be used together for the best predictions.

Q4: Finetuning Other Geo-spatial LLM Models

A4: The main contribution of the paper is to present a novel method to extract geospatial knowledge from LLMs. It would be very interesting to determine whether or not the additional domain specific pre-training and finetuning of geospatial LLMs such as K-2 embeds additional knowledge that can be leveraged by GeoLLM. However, we believe that this investigation would be orthogonal from the main purpose of GeoLLM, which was simply to show that LLMs are new covariates and to present a method to use them as such. We leave this investigation to future work.

Dear reviewer,

As we are approaching the discussion deadline, we hope that your concerns are resolved and that you can reconsider our scores. Thank you very much for your time.

Thank you for your insightful review and the recognition of the novel approach and significance of GeoLLM in extracting geographical knowledge from large language models using OpenStreetMap data. We appreciate your constructive feedback on the areas needing clarification and improvement. Below, we address your concerns and questions:

Q1: Lack of Clear Prompt Templates and Answer Pairs Design

A1: We added Appendix A.1 to provide details on the prompt template that we used across all tasks as well as an additional prompt example that we use in the asset wealth task. You can find details for the source of the auxiliary data used to construct the prompts in section 3.2. We hope this aids in the replicability of our results.

Q2: Lack of Detailed Experimental Design

A2: We added Appendix A.2 to describe the finetuning process and the exact hyperparameters used. Details on the preprocessing of the data for prompts are provided in the last paragraph of section 3.1. Details on the ground truth data for tasks, such as their sources or any task-specific preprocessing that we do, are provided in section 4.1.

Q3: Absence of Comparison with Ground-Truth Data

A3: Pearson’s $r^2$, which is the correlation squared, is shown for every model across all tasks in table 1. These are calculated using the ground truth and the predictions. We also show the mean absolute errors in an identically structured table in Appendix A.3. We clarify the fine-tuning process of the LLMs in Appendix A.2 where we describe how we use unsupervised learning on the prompts and labels (ground truth) concatenated. We also try to use the word “labels” more to be more clear about the fine-tuning process and the fact that we use ground truth. Please let us know if this is a sufficient clarification or if we are misunderstanding your point.

Q4: Clarification on the Statement Regarding Overfitting

A4: We clarify in section 3.3 and Appendix A.2 that we are using unsupervised learning with the prompts and labels concatenated. While the model may overfit (increase in loss on test set) with this unsupervised learning at the last few epochs, it does not seem to be overfitting on the actual “labels” or ground truth that were concatenated which is indicated by the continued improvement in performance on the test sets.

We encourage you to see our overall response to see how we have strengthened our paper further or to answer any more questions you have.

Dear reviewer,

As we are approaching the discussion deadline, we hope that your concerns are resolved and that you can reconsider our scores. Thank you very much for your time.

Thank you for your positive evaluation of our paper, recognition of the clear presentation, and acknowledgment of the potential impact of our work in geospatial applications. We appreciate your constructive feedback and have addressed your concerns and questions as follows:

Q1: Incremental nature of methodology

A1: In addition to showing the importance of comprehensive prompts constructed from auxiliary map data, we revise the paper to show the importance of fine-tuning for overall performance. In particular we find that unsupervised learning (used for the pretraining of the LLMs) for fine-tuning works very well. The details of this finetuning for LLMs are included in section 3.3. Experiments comparing few-shot and finetuning are included in Appendix A.4.

Q2: Expansion of the scope of experiments

A2: With the revision, we add a sentence embeddings baseline, satellite imagery baseline, results for GPT-2 with GeoLLM, 3 new ablations, few-shot performance with GPT-3.5 and GPT-4, and a preliminary evaluation of the biases of LLMs. Please see “Zero-shot and few-shot experiments”, “Full set of ablations”, and “Preliminary investigation of bias” in the overall response.

Q3: Additional evaluation metrics (MAE, MSE)

A3: As a potentially more interpretable metric, we present the mean absolute error (MAE) for all models across all tasks and training sample sizes in Table 3 in Appendix A.3. One can observe that the same conclusions that are made with Pearson's $r^2$ can be made when comparing the MAE of the various models. In particular, the relative ranking in performance of the models within tasks is consistent with Pearson's $r^2$.

Q4: Performance variation in different geographical contexts

A4: While there does not appear to be any obvious signs of performance bias across countries or continents as seen in fig. 2, the existence of biases in performance is inevitable as the internet training corpora of LLMs are inherently biased towards developed and densely populated areas. We show preliminary signs of this bias in table 5 in Appendix A.5. We find that there is a moderate increase in mean absolute error between densely populated and sparsely populated areas and another moderate increase between the areas with above median asset wealth and below median asset wealth. As we discussed in section 5, this demonstrates that GeoLLM has the potential to be used as a tool to reveal LLM biases on a geographical scale. However, substantial additional research is required for a comprehensive analysis of the geospatial biases in LLMs and their training corpora.

Q5: Comparison with "weaker" text embedding models and a wider variety of baselines

A5: Please see “Stronger baselines” in the overall response.

Q6: Performance of more powerful LLMs

A6: Please see “Zero-shot and few-shot experiments” in the overall response.

Q7: Impact of fine-tuning on performance

A7: Please see “Zero-shot and few-shot experiments” in the overall response.

Q8: Few-shot learning capabilities

A8: Please see “Zero-shot and few-shot experiments” in the overall response.

Q9: Independent contribution of coordinates

A9: Please see “Full set of ablations” in the overall response.

Dear reviewer,

As we are approaching the discussion deadline, we hope that your concerns are resolved and that you can confirm that this is the case. Thank you very much for your time.

Thank you for your thoughtful feedback, recognition of GeoLLM’s novelty and of our broad analysis across multiple relevant benchmark datasets with a comprehensive list of baselines and ablations. We appreciate your suggestions on investigating potential biases, few-shot or zero-shot performance, and on comparing with satellite images. You may find responses to your questions below:

Q1: Lack of a detailed analysis on biases of LLMs and training datasets

A1: While there does not appear to be any obvious signs of performance bias across countries or continents as seen in fig. 2, the existence of biases in performance is inevitable as the internet training corpora of LLMs are inherently biased towards developed and densely populated areas. We show preliminary signs of this bias in Table 5 in Appendix A.5. We find that there is a moderate increase in mean absolute error between densely populated and sparsely populated areas and another moderate increase between the areas with above median asset wealth and below median asset wealth. As we discussed in section 5, this demonstrates that GeoLLM has the potential to be used as a tool to reveal LLM biases on a geographical scale. However, substantial additional research is required for a comprehensive analysis of the geospatial biases in LLMs and their training corpora.

Q2: Comparison with methods that use satellite imagery

A2: Please see the Nightlight baseline in the “Stronger baselines” section of the overall response.

Q3: Including results from zero-shot or few-shot prompting

A3: Please see “Zero-shot and few-shot experiments” in the overall response.

Dear reviewer,

As we are approaching the discussion deadline, we hope that your concerns are resolved and that you can reconsider our scores. Thank you very much for your time.

References:

[1] Mai, Gengchen, Krzysztof Janowicz, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao. "Multi-Scale Representation Learning for Spatial Feature Distributions using Grid Cells." In International Conference on Learning Representations. 2020.

Thank you for your thoughtful feedback, recognition of GeoLLM’s evaluation across a broad set of relevant geospatial datasets and tasks, and suggestions for clarifying our design choices. You may find responses to your concerns below:

Q1: GeoLLM performs a specific task/ does it answer general questions?

A1: GeoLLM is not intended to retain natural language or general-purpose response capabilities. It is a method to introduce a novel set of covariates that can enhance various downstream geospatial ML tasks that currently depend on traditional covariates such as satellite images. These tasks include poverty estimation, food security, biodiversity preservation, and environmental conservation. Thus, we can quickly adapt powerful LLMs to a variety of geospatial tasks using our novel prompting strategy. Moreover, we demonstrate that a high level of performance is achievable with relatively cheap finetuning using methods such as QLoRA.

Q2: Can GeoLLM handle different types of spatial/spatio-temporal data

A2: As LLMs have not yet been shown to be effective covariates for geospatial ML tasks, we focus on providing a method that enables this by efficiently extracting their spatial knowledge. In fact, there are two general settings for spatial prediction: location modeling and spatial contextual modeling [1]. The former aims at predicting the attributes/characteristics of a place given its current location. The latter focuses on predicting the attributes/characteristics of a place given its location as well as its spatial context. Our GeoLLM is able to handle both types of tasks. The spatial context modeling corresponds to our current GeoLLM setting since we give the location and description of the current location as well as its nearby places extracted from OpenStreetMap (See Figure 1 (b) bottom). The location modeling corresponds to the setting in which we delete the “Nearby place” in our prompt. We believe we are the first one that shows LLMs’ abilities on a wide range of geo-tasks. We acknowledge that there are other geo-tasks such as spatial relation prediction between two places, or questions requiring spatial footprints more than points (e.g., polylines and polygons). We treat them as future work.

Q3: Simplicity of baselines

A3: Please see “Stronger baselines” in the overall response.

Q4: Limited experiments

A4: Please see “Stronger baselines”, “Zero-shot and few-shot experiments”, and “Full set of ablations” in the overall response.**

Q5: Tokenizer for geospatial coordinates

A5: We recognize the tokenization problem of the current LLM for geo-coordinates. Current LLMs break up the GPS coordinates into multiple tokens and makes understanding them more difficult, especially for smaller LLMs like GPT-2. While larger models such as GPT-3.5 are better able to utilize the numeric value of the coordinates, as shown in the “only coordinates'' ablation in Table 2, the coordinates do not affect the final performance of GPT-3.5 significantly (bump from 0.72 to 0.73 Pearson’s $r^2$) as shown in the “no coordinates'' ablation in table 2. In fact, GeoLLM is motivated by this observation and is developed to overcome this limitation. How to develop a tokenizer that makes LLM better understand geo-coordinate is a very interesting research direction but is orthogonal to our GeoLLM framework. We leave it as one of our future work.

Q6: Relevance of framing as a classification problem

A6: Making geospatial predictions via the generation of tokens is significantly more difficult than regression. As one would expect, GPT-2 struggles to perform well within this paradigm, even doing much worse than MLP-BERT that uses regression. For this very reason, we deliberately introduced decimals in the labels (0.0 to 9.9 as opposed to 0 to 99) to add consistency to the predictions as they always require exactly 3 tokens to generate (e.g. “5”, “.”, “3”). However, to our surprise, the autoregressive nature of LLMs not seem to hinder GPT-3.5 and Llama 2 significantly, with GPT-3.5 exceeding the performance of all models that use regression, including RoBERTa and the newly introduced MLP-BERT and Nightlight baselines. In addition, as GPT-3.5 only allows for fine-tuning through OpenAI’s API, we cannot make any modifications to the tokenization or architecture of the best performing model by far.

Q7: Fine-tuning and inference costs

A7: We present the costs of the finetuning of the LLMs we use in Appendix A.2. In short, fine-tuning GPT-3.5 with 10,000 samples costs 33 dollars. Thanks to its sample efficiency, one can finetune it with 1,000 samples for 5 dollars without much sacrifice to the overall performance. As a set of predictions only needs to be made once or a small number of times for a specific application, the deployment cost is not as significant or as much of a burden as with general-purpose LLMs.

Dear reviewer,

As we are approaching the discussion deadline, we hope that your concerns are resolved and that you can reconsider our scores. Thank you very much for your time.