Can Large Language Models Integrate Spatial Data? Empirical Insights into Reasoning Strengths and Computational Weaknesses

We appreciate the reviews’ thoughtful questions. Here are our responses:

Include the best and worst exact heuristic configurations.

Here are the configurations and we will include them in the appendix.

• Spatial join – best single (p=10); best duo (p=10, o=0.2); best trio (p=20, o=0.3, c=2); worst single (o=0.5); worst duo (p=1, o=0.5); worst trio (p=1, o=0.5, c=5).
• Spatial union: – best single (o=0.8); best duo (p=3, o=0.5); worst single (p=4); worst duo (p=1, o=0.9).

Drop the "plain" settings. Use "hints" as the simplest baseline.

We agree that dropping the “plain” setting makes sense given its poor performance. Even with hints, models struggle with the tasks. We will use the “hints” as our simplest baseline.

Figure 3: where does "avg heuristics" come from?

The “avg heuristics” is the average performance across all heuristic combinations (single, duo, trio) and remains constant across all LLM prompting settings for comparison. We will update the visualization to clarify this.

Figure 4: there is the same title for both charts.

Thank you for the careful reading. We will fix the title.

Why review & refine is better compared to the heuristic prompting

1. review-and-refine introduces additional reasoning steps that enable deeper analysis and better decision-making.

2. Initial answers affect results; for instance, qwen-plus’s refined outcomes range from 94.9% for weaker priors up to 98.4% for stronger ones. This shows review-and-refine benefits from iterative reasoning and good initial guidance.

The task is too simple and narrow, making it unsuitable for the current venue.

We appreciate these concerns, but hope to convince you that our work is well-motivated and is a good fit for COLM:

1. Our work expands the existing understanding of LLM capabilities to encompass spatial data with real-world geometry objects. While the two tasks may appear simple, they are foundational to more complex spatial analysis. Our results show that even these simple tasks challenge LLMs, revealing computational limitations.

2. We examined the use of LLMs in urban spatial data integration — an underexplored area with meaningful real-world impact, especially in accessibility. We see our work as a step toward enabling LLM-driven systems to unify fragmented spatial data, supporting more accurate representations of infrastructure like sidewalks and intersections for accessible urban navigation.

3. We certainly agree that evaluating a wider range of spatial tasks can provide deeper and more general insight into LLMs’ capabilities in this domain. In response, we conducted additional experiments on four more spatial analysis tasks involving spatial reasoning and geometric computation. Due to space constraints, we have included these results in our response to Reviewer VhgA's second question, and hope that you’ll take the time to examine them.

Simple heuristics with minimal tuning are more efficient than LLMs that require prompt engineering and inference cost.

Feature selection and threshold tuning are not one-time efforts; rather, they vary across locations (e.g., cities with distinct physical layouts) and temporal changes (e.g., road expansions). This limits the generalizability of heuristic-based methods. While LLM performance is also context-sensitive, LLMs offer the advantage of integrating external information—such as city-specific context or visual data—and drawing on broad world knowledge, potentially enabling better adaptation across diverse scenarios. Though LLMs can be costly, they may ultimately reduce the manual effort required to tailor spatial analysis tools to new environments.

Data split and leakage concern.

The data was randomly split using Python’s train_test_split() function, ensuring no overlap between the train, validation, and test sets. Class distribution is balanced (60% positive, 40% negative). Train and validation sets were not used for fine-tuning, as we focus on evaluating integration without labeled training data. Therefore, evaluation was done only on the test set, avoiding data leakage.

T2 results are expected, as LLMs struggle with spatial reasoning and heuristics naturally improve their performance.

While heuristics perform well enough on their own (conditioned on tedious threshold search), we had little indication prior to this research of how effectively LLMs would use those heuristics. Our findings show that LLMs do not rely on fixed thresholds but rather dynamically adjust thresholds per instance, a novel insight into how LLMs integrate heuristic information for spatial reasoning tasks. We also note that, as described below, our results with multimodal models indicate that heuristic-based prompting outperforms not only text-based prompting but also prompting with VLM models, further indicating the efficacy of this approach.

TBC...

A:                       B:                           C:
O---O                 O---O                            O
                       \                              /
                        \                            /
                         O                          O
                                                    |
                                                    |
                                                    O

We appreciate the reviews’ thoughtful questions. Here are our responses:

The term "topological" is used in a non-standard way.

Thank you for the clarification and suggestion. Features like distance, angle, and area are better described as geometrical rather than topological in the mathematical sense. We’ll revise accordingly throughout the paper.

Figure 1 is unclear without feature definitions and has typos

We’ll make the following improvements to the figure: (1) move it to the end of Section 3, where the geometric features are defined, and (2) correct the typo in the road coordinates. To improve clarity, we’ll also add a visual example to make the feature definitions more intuitive.

The background colors in Fig 2 are very faint.

Thanks for this feedback, we will make the background clearer.

Clarify that roads are modeled as one-dimensional and explain the use of buffer widths.

Thanks for this suggestion, we will add some text to clarify this. The roads and sidewalks in our dataset are represented as centerlines, which as you observe, makes them one-dimensional. It is standard in spatial data sources like OpenStreetMap. You are also correct that the 10-meter buffer used in preprocessing helps approximate real-world road widths.

Highlight temporal changes and bike paths examples as extra challenges for the tasks

We agree that changes over time, like road expansions, can alter widths and buffers, causing heuristics to fail. The presence of bike paths between roads and sidewalks also adds complexity, especially for blind pedestrians. We will incorporate these points in our discussion.

Mention that Table 8 is in the appendix, and fix the typo "hresholds" on line 323.

We will add the clarification and fix the typo. Thank you.

Testing on just two tasks is a weak reason to reject.

We agree that the tasks are representative of the spatial domain and likely to generalize. However, as the question of whether two tasks is enough came up in several other reviews, we have conducted additional experiments on four more spatial analysis tasks involving both spatial reasoning and geometric computation.

Due to space constraints, we have included these results in the response to Reviewer VhgA. Depending on length considerations, we will include these limited validations either in the discussion of the paper or in an appendix.

We appreciate the reviews’ thoughtful questions. Here are our responses:

What is the relationship of this study with works about LLMs as world models?

Thank you for this perspective. While our study does not directly engage with the literature on LLMs as world models, there may be conceptual similarity, especially in examining whether LLMs can reason about spatial relationships grounded in real-world structures:

1. While existing studies on LLMs as world models often emphasize physical dynamics or simulation-based understanding, we focus specifically on spatial reasoning over structured geographic data, such as roads, sidewalks, and polygons.

2. Rather than simulating general physical environments, we assess whether LLMs possess sufficient latent world knowledge to perform basic spatial integration tasks without explicit training or labeled data. In that sense, we could view this work as an early step toward expanding an LLM’s world model to encompass geospatial contexts and structured environmental data, which are important for urban applications.

The scope is narrow to specific spatial reasoning tasks.

Our study is application-driven and aims to provide a fundamental empirical evaluation of LLMs on two core spatial integration tasks that serve as a foundation for many more complex spatial tasks. Our work is not intended to serve as a comprehensive investigation into LLMs’ spatial reasoning abilities; rather, we know from much prior work in urban spatial analysis that capability in these tasks underlies much of what practitioners hope to accomplish in more challenging scenarios.

We do agree that evaluating a broader range of spatial tasks would yield more general insights into LLMs’ spatial capabilities. Thus, we conducted additional experiments on four spatial analysis tasks that vary in geometry types, computational demands, and reasoning logic. These new experiments broaden the scope of our evaluation while maintaining coherence within the spatial analysis domain. We will add the results as supplementary/ablation study.

• Point-to-Segment Distance (P2S): Compute the shortest distance between a point and a line segment.
• Spatial Containment (SC): Determine whether a point lies inside a polygon (triangle).
• Spatial Intersection (SI): Check if two line segments intersect.
• Convex Hull (CH): Identify the minimal set of points forming the convex hull of a point set.

Each task was evaluated on 50 synthetic samples with geometries within a unit square. We use simple settings for preliminary experiments. We evaluate qwen-plus, 4o-mini, and 4o, and report accuracy. For P2S, a tolerance of 1e-3 was used for correctness. Results are below:

Models performed well on simpler tasks (P2S: 90%, SC: 81%) but struggled with more complex ones (SI: 70%, CH: 57%). A brief qualitative analysis of GPT-4o’s outputs revealed encouraging signs of task understanding and spatial reasoning ability. For instance, it consistently applied projection logic for P2S, ray casting for SC, orientation-based reasoning for SI, and referenced Graham’s scan or Jarvis march in CH. However, computations errors often led to incorrect answers.

These experiments offer preliminary but valuable insight suggesting that LLMs’ spatial reasoning capabilities extend beyond basic integration tasks. We believe that our results on spatial join and union, along with these four supplementary tasks, provides meaningful and a broader scope of insight into the fundamental capabilities and limitations of LLMs in applications involving spatial data analysis. They motivate further evaluation to include real-world geometries and more complex task variants in ongoing work.

The method relies on well-established techniques offering little novelty.

We do not view the primary contribution of our work as a novel methodology. Our main contribution lies in conducting a comprehensive empirical evaluation of LLMs on spatial integration tasks, an area that, to the best of our knowledge, remains unexplored. This study serves as an initial step toward understanding how LLMs can be effectively applied to applications involving spatial data.

In addition, we note that our analysis of spatial reasoning and geometric computation differs from prior work in both domains. Existing spatial reasoning studies often center on visual data and tasks like relative positioning or navigation, while geometric reasoning research tends to focus on abstract shapes and textbook math problems (like analyzing triangles and circles). In contrast, we analyze LLMs’ performance on tasks involving real-world geometries, such as linestrings and polygons, which better reflect practical challenges and offer new opportunities for application-driven research, including in areas like urban accessibility.

We appreciate the reviewer’s thoughtful questions. Here are our responses:

Is the notation “GPT-4o, 2024-08-06 (4o)” used to indicate the model version?

Yes. “GPT-4o, 2024-08-06” refers to the specific model version gpt-4o-2024-08-06. The shorthand “(4o)” is used for brevity in presenting results, as with “(4o-mini)” and “(llama3).”

Does Table 1 show results for a specific duo heuristic or an average of all three?

It is the average over all three duo combinations. We provide Individual results below, showing consistent performance and close alignment with the overall average.

Is using a fixed 10m buffer suitable for all road types given the variation in road widths?

The 10m buffer reflects the origins of our dataset in accessibility research, which is typically concerned only with roads that have sidewalks. Thus, while there is substantial variation in real-world road widths, our dataset includes only road types likely to have adjacent sidewalks – {primary, secondary, tertiary, residential, living_street}, and excludes highways. The selection of the 10-meter buffer is based on typical road widths in Seattle and Bellevue, and applied across all types.

Have you considered approaches like ViperGPT, which use code generation to compose modular visual and spatial reasoning tools via Python APIs?

We haven’t tested ViperGPT or similar approaches, though we agree that these frameworks could be beneficial for addressing some of the computational limitations our work identifies. The computation of the geometric features used in this paper requires very light human programming, rendering feasible a purely natural language approach with geometric features supplied in the prompt. However, where we want to perform spatial tasks that require deriving complex features, or combining information from several data sources, we expect that the benefits of a framework that integrates programming and natural language instructions will become more clear. We see this as a promising direction for future work.

The paper lacks a dedicated method section, which weakens the theoretical grounding of the work and reflects a lack of methodological novelty.

1. We combined methods and experimental setup to conserve space; we will distinguish them in the revision and emphasize the theoretical grounding related to limitations of spatial and geometric reasoning as distinct from other tasks.

2. Our main contribution lies in conducting a comprehensive empirical evaluation of LLMs on spatial integration tasks, an area that, to the best of our knowledge, remains largely underexplored. This study serves as an initial step toward understanding how LLMs can be effectively applied to applications involving spatial data. We believe that our findings provide valuable insights and establish a foundation for future research that may introduce more methodologically innovative solutions.

All experiments focus on only two relatively simple tasks (spatial join and union), which raises concerns about whether the model truly understands spatial relationships.

1. While the two tasks may appear simple, they are foundational to more complex spatial analysis, and focusing on just these ubiquitous, fundamental tasks provides us with a highly interpretable view of LLMs’ basic capabilities in handling spatial tasks. In fact, our results show that even these foundational tasks pose a notable challenge for LLMs, in that they reveal the models’ computational limitations.

2. Nonetheless, we agree that evaluating a wider range of tasks can provide deeper and more general insight into LLMs’ capabilities. In response, we conducted additional experiments on four more spatial analysis tasks involving both spatial reasoning and geometric computation. Due to space constraints, we refer the reviewer to our response to Reviewer VhgA's second question for details on these supplementary results.

The paper lacks a detailed analysis of the large performance gap (hints v.s. features)

We address this in Section 5.3 with a detailed qualitative analysis of how LLMs approach both tasks with and without geometric features.

1. Without features, LLMs attempt to infer spatial relationships by assessing proximity and alignment between pairs. While this shows some spatial understanding, they often fail due to poor execution of geometric calculation, including incorrect logic and flawed calculations.

2. In contrast, providing heuristic features fundamentally shifts the nature of the task. Rather than performing geometric calculation, models simply evaluate the provided feature values. This reduces the task to a rule-checking problem, resulting in better accuracy.