Evaluating VLMs' General Ability on Next Location Prediction

Thank you for your valuable comments. We appreciate the time and effort you have devoted to reviewing our work. Below, we address your concerns in detail.

How much value does this new benchmark provide?

1. From the perspective of large vision-language models (VLMs): The proposed benchmark is fundamentally designed to evaluate the reasoning capabilities of VLMs. Consider a simple example: suppose the vertices of a square are labeled A, B, C, and D in order. If a driver travels from A to B to C, and all roads are accessible without complex traffic rules, it is highly unlikely that the next point will be D—since a direct route from A to D would have been more efficient. This suggests that the prediction of the next location relies on understanding the topological structure of the road network. In more complex real-world maps, this task requires even stronger reasoning skills. Our benchmark provides a framework to quantitatively assess whether VLMs possess such abilities. As shown in our results, current VLMs still fall short of human-level map reasoning and spatial understanding.
2. From the perspective of next-location prediction: Most existing approaches rely on learning city-specific trajectory patterns, often limiting their generalizability. In contrast, our work explores the potential of VLMs to leverage visual understanding of road networks to achieve generalizable next-location prediction. Because the visual representation of road networks tends to be domain-agnostic, VLM-based approaches offer the potential to build universal models for trajectory prediction. Our benchmark thus serves as a standardized evaluation framework for such vision-based next-location prediction, not limited to VLMs alone.

What types of new research can benefit from this benchmark?

1. Spatial computing: One promising application area is spatial computing. Real-world navigation often requires path planning over maps. Using VLMs to understand spatial layouts is an emerging research direction. Our benchmark allows researchers to quantitatively assess a model’s ability to reason about spatial structures and path planning, helping to avoid potential pitfalls in real-world deployments.
2. VLM-based reasoning: As discussed above, the benchmark evaluates the reasoning abilities of VLMs. Since road networks form complex topological graphs, predicting the next step in a trajectory requires non-trivial reasoning over these structures. Our benchmark provides a testbed to evaluate whether VLMs can exhibit human-like map-based reasoning.
3. General-purpose next-location prediction: By leveraging the general visual understanding capabilities of VLMs, there is strong potential to build general-purpose next-location predictors that are not bound to a specific city or dataset. As one of the first works to explore vision-based next-location prediction, we hope this study inspires further research into broadly applicable, vision-driven trajectory modeling.

Once again, thank you for your thoughtful feedback and the time you’ve dedicated to our work. If you have any further questions or if any concerns remain unaddressed, please feel free to reach out, we would be happy to continue the discussion.

Sincerely, The authors of Paper 740

Thank you for your comments. Below, we respond to your concerns. Due to space constraints, some points may be addressed briefly; please feel free to raise any questions.

On the Superior Performance of Claude Models

The Claude series consistently achieves SOTA results on spatial reasoning tasks [1]. Researchers speculate that Claude models may possess a well-developed world model, likely due to optimization for screen control tasks, which require fine-grained spatial understanding of interface layouts, highlighting its spatial reasoning capabilities.

On Novelty

The novelty of this work lies in introducing the first benchmark for next-location prediction based on vision-language models. Since road network structures vary across cities, training city-specific models limits generalization. In contrast, vision encoders provide universal map representations, enabling cross-city generalization. Promising future path include: Reinforcement learning with reward signals, Visual fine-tuning via LoRA. We leave these directions to future work and hope this benchmark inspires broader research in vision-based location prediction.

On the Formal Convergence of Our Method

Our method assumes that VLMs inherently possess an internal capability to predict the next location, but this capability may not be directly shown because of inability to paint on the image. We therefore reformulate the generative task as a discriminative one. For an image of size HxW, the distance between the model’s selected location and its internal prediction is bounded by $\sqrt {H^2+W^2}/{2^i}$ after i steps. This distance converges to 0 as the number of steps increases.

Insights from Failure Cases

We categorize the identifiable errors into two types: Visual hallucinations: In some cases, the region of interest is too small for the model to differentiate, potentially due to the patching mechanism in the vision encoder. Speed-related biases: As shown in Porto Case 41, an abnormally high-speed segment caused the model to overemphasize that portion, leading to an incorrect prediction. This may reflect a limitation in the model’s attention mechanism.

On the Influence of Other Trajectory Factors

In Appendix B.1, we analyze several additional factors: Image scaling. When a unit length represents a shorter real-world distance, performance improves. Angular change. Larger angular shifts tend to worsen accuracy. Path length. Longer trajectories correlate with higher errors. Experiments related to transparency, background color, and trajectory color will be included in the appendix if the paper is accepted.

Regarding the Statement on Insufficient Data for Training RNNs

A simple motivating example is the prediction of next-location trajectories for elderly individuals. GPS data for this demographic is extremely sparse due to the need for specialized data collection equipment. As a result, it is difficult to train RNN-based models effectively in such cases. Leveraging the generalization ability of VLM offers a way to bypass the need for domain-specific, large-scale trajectory data.

On the Transferability Limitations of Other Models

As mentioned in prior works, models such as LLM-Mob and Agent-Move require training a dedicated token for each region within a specific city. These tokens are tightly coupled with the city they are trained on, significantly limiting the transferability of these models across cities.

On Consistent Instructions for Human Participants

We have stated in Line 167 (right column) of our paper, “Users are presented with the same input prompts” This ensures fair comparison.

On the Choice of Distance Thresholds

100m roughly corresponds to the size of a football field. 500m typically covers the space between traffic signals. 2000m approximates the span of a city block. Each threshold offers meaningful granularity for evaluating real-world spatial accuracy.

On the Number of Iteration Steps

As mentioned previously, increasing the number of iterations reduces prediction error, akin to inference-time scaling in large models. We chose to terminate after 10 rounds, at which point the average distance between the predicted region center and the model's internal prediction was under 32 meters, and we consider it acceptable. The final prediction is defined as the center of the region selected in the 10th round.

On Comparison with More Sophisticated Trajectory Models

Following your suggestion, we now include results for four SOTA models. Due to space constraints, we only report the average MAE across all datasets here: Transformer: 216.73, DeepMove: 215.18, GETNext: 214.33, LLM-Mob: 211.12. We will include full experimental results in the final version of the paper if accepted.

Sincerely, The authors of Paper 740

[1] https://mcbench.ai/leaderboard

[2] https://www.anthropic.com/news/visible-extended-thinking

Thank you for your valuable comments. We appreciate the time and effort you have devoted to reviewing our work. Below, we respond to your concerns in detail.

Regarding Experimental Design

1. While it is true that taxi drivers' trajectories are influenced by intent, prior work suggests that human mobility remains highly predictable despite this. Notably, the Science paper "Limits of Predictability in Human Mobility" (2010), which has been cited over 4,000 times [1], showed that for hourly human mobility data, the theoretical upper bound of next-location prediction accuracy can reach as high as 93%. This indicates that, although behavior is goal-driven, there are still strong patterns and structural constraints that make the task meaningful and predictable.
2. In addition, the topological structure of road networks imposes natural constraints on plausible movements. Consider a simple example: suppose the vertices of a square are labeled A, B, C, and D in order. If a driver travels from A to B to C, and all roads are open and unconstrained, it is unlikely that the next step will be D—since a more efficient plan would have been a direct route from A to D. This highlights that predicting the next location depends on a model’s ability to reason about spatial efficiency and road network topology.
3. Lastly, as larger models consistently achieve better results, which is consistent with scaling laws in deep learning, and no model has yet reached human-level performance, it demonstrates that the behavior of taxi driver can be in-deed inferred.

For these reasons, we respectfully ask the reviewer to reconsider their concerns about the experimental design.

Regarding Research Challenges

1. We agree that experiments like color choice are relatively simple, which is why they were included only in the appendix. However, they serve to illustrate that our benchmark design is deliberate and well-grounded. Moreover, in our ablation studies, we show that carefully designed prompts reduce model error by over 22%, indicating a meaningful impact.
2. As for the scientific value of the research challenge: as mentioned earlier, our benchmark directly targets the reasoning capabilities of VLMs. Road networks are complex topological graphs, and predicting trajectory continuations requires non-trivial structural reasoning. Our benchmark provides a first step in evaluating whether models can perform human-like spatial reasoning over such structures.

While we do not focus on improving model accuracy in this work, we see this as a rich direction for future research, and suggest a few promising directions:

• Reinforcement learning with reward signals: Since selecting the correct region (e.g., color) is straightforward to verify, one could define reward functions and train models to improve their reasoning through interaction.
• Visual fine-tuning with LoRA: For example, training a vision-language generation model to take a map as input and output a predicted next-point image. These approaches could significantly improve VLM performance on our benchmark, and we leave their exploration to future work.

We hope this clarifies the scientific challenges and potential of the benchmark.

Regarding In-Depth Summary of the Scientific Insights

1. Vision-based Next-Location Prediction: Because the visual representation of road networks is domain-agnostic, VLMs offer the potential to build generalizable next-location predictors. Our benchmark establishes a unified framework for evaluating such models, not limited to VLMs alone.
2. Reasoning over Complex Road Networks: As emphasized, our benchmark is designed to test whether VLMs can reason over complex topological structures, which is central to understanding spatial intelligence and planning. From the experimental results, the large models exhibit the complex reasoning ability to some extent.

Once again, thank you for your thoughtful feedback and the time you’ve dedicated to our work. If you have any further questions or if any concerns remain unaddressed, please feel free to reach out—we would be happy to continue the discussion.

Sincerely, The authors of Paper 740

[1] Song, Chaoming, et al. "Limits of predictability in human mobility." Science 327.5968 (2010): 1018-1021.