MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models

W2: The benchmark framework is unable to evaluate multi-modal large language models that combine both text and visual information.

We do not understand this point. As discussed in Fig 1, Section 3.2 and so on, our inputs in MapEval-vision task are both a textual query and a map snapshot image which you also have mentioned in your summary. Note that each MapEval-vision query includes a textual context (within ~100 tokens). In Appendix F, we depict full examples of several instances also (e.g, listing 12 on page 33). We have a complete result Section 4.2.3 evaluating multi-modal VLMs in Table 5. In addition, a subcategory of MapEval-vision queries called 'Extra Information Processing' involves map snapshot images containing OCR texts such as user ratings or detailed routes in trip history (mentioned in updated paper line 186) that must be processed to generate answers (see listing 13 in Appendix F). How then it can not evaluate multi-modal models taking both text and image input.

W3: The analysis is not in-depth enough. It would be beneficial to point out the root causes of the performance (rank), which could inspire the future development of map-related foundation models and the gradual specialization of general-purpose models.

Thank you for your valuable feedback. As a dataset and benchmark work, we have already provided comprehensive evaluation and analysis, detailing fine-grained model performances and identifying root causes of errors across various foundation models (discussed in Section 4.3). However, to address your concern for deeper analysis, we have expanded our study in response to your comments. We have now included a performance ranking across different reasoning sub-tasks (e.g., measuring straight-line distances, calculating cardinal directions, and solving counting problems) to better understand model strengths and weaknesses. Additionally, we have added a new section (Section 5 in the updated paper) with extended analyses. We also present complementary enhancement strategies that address identified model limitations, particularly in reasoning aspects where models lack explicit training. These findings can inform the development of more capable next-generation foundation models.

Q1: The last paragraph of the introduction mentions ablation experiments, but I could not find the experimental content or results in the main text.

Thanks for pointing out this. This is a typo, we meant to say “our in-depth analysis” (updated in the revised paper).

Thank you for your recognition of the applicability, effectiveness of our proposed benchmarking task, and comprehensive experiments in our work!

W1: The number of test cases available for evaluating foundation models is relatively small, with only around 550 cases.

(Also discussed in Response to Reviewer bA3F W2)

Due to the high cost of both foundation models as well as tools/APIs, recent language agents often tend to evaluate on a small number of sub-sampled dataset. For example ReACT [1] uses only 500 random samples from AlfWorld dataset, similarly Reflexion often tends to use [2] only 100 examples from HotpotQA). Therefore, recently proposed tool oriented or intense reasoning benchmark datasets are found to reasonable in size in order to be cost-effective: API-Bank: [3] (400 instances) Logical-reasoning benchmark LogiQA: [4] (641 examples), the most popular problem solving (code generation benchmarks) HumanEval: [5] (164 instances only), CodeContests [6] (156 problems), Tao-bench [7] (165 problems), OS World [8] (369 problems), App world [9] (750 problems), TravelPlanner [10] (1.2K problems). Consequently, we carefully construct our problem instances balanced in size and covering different challenges.

However, we thank you for the feedback and leveraging the rebuttal time-periods of ICLR, we strive to increase our problem set to 700 and ensure that each of the 54 countries is now represented by a minimum of 6 questions while maintaining the same quality control and pipeline. This improvement addresses the concern regarding insufficient representation of certain countries, making the evaluation more robust and geographically diverse. In our observation, further attempts in incorporating additional questions from different countries revealed no significant change in model performance. To maintain a cost-effective testbed, we opted not to include more instances, suggesting that our evaluation problem set already offers sufficient diversity for robust and reliable assessment. This improvement addresses the concern regarding insufficient representation of certain countries, making the evaluation more robust and geographically diverse.

In addition, as outlined in Section 3.1, our goal is to benchmark on high standard real-user queries aiming to solve day-to-day actual Map application tasks and hence we do not scale up by synthesizing.

References:

[1] Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." The Eleventh International Conference on Learning Representations.
[2] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).
[3] Li, Minghao, et al. "API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
[4] Liu, Jian, et al. "LogiQA: a challenge dataset for machine reading comprehension with logical reasoning." Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 2021.
[5] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).
[6] Li, Yujia, et al. "Competition-level code generation with alphacode." Science 378.6624 (2022): 1092-1097.
[7] Yao, Shunyu, et al. "$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv preprint arXiv:2406.12045 (2024).
[8] Xie, Tianbao, et al. "Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments." arXiv preprint arXiv:2404.07972 (2024).
[9] Trivedi, Harsh, et al. "AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
[10] Xie, Jian, et al. "TravelPlanner: A Benchmark for Real-World Planning with Language Agents". Poster presented at the Forty-First International Conference on Machine Learning (ICML), 2024.

Q2: This paper compares the evaluation results of several foundation models, but it should further clarify whether any specific geographic specialized foundation models were used to highlight the specialization in geo-spatial reasoning tasks. If none were used, please explain the reasons. Example models are those developed with geographic knowledge like K2, and remote sensing models (the number is increasing).

Clarification on the Exclusion of K2 from Our Evaluations

Thank you for your comment. We would like to clarify the distinction between the models we used and those mentioned, particularly K2 and remote sensing models. K2, as outlined in the full title "K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization," [1] is designed specifically for geoscience-related tasks, not geographic or geospatial reasoning tasks. It focuses on questions related to geoscience knowledge, such as “What kind of rock belongs to dynamic metamorphic rocks?”, which is distinct from the map-based user queries we address in this work.

Given this difference in domain focus, we anticipated that K2 would not perform well for map-based questions. In response to your feedback, we evaluated K2, and the evaluation confirmed that its performance was very poor, with an accuracy of 20.33%. Given that K2 performed no better than random guessing, we decided not to include it in our evaluations. Besides, at the time of our evaluation, there were no geospatial specialized foundation models that met the specific requirements of our task.

---

Table 1: Performance of K2 in MapEval-Textual

---

Explanation on Remote Sensing Models in Our Study

Regarding remote sensing models, it is important to note that our study does not involve remote sensing images. Instead, we provide digital map view snapshots (See Listing 9 - 13 in Appendix F for example snapshots) to Vision-Language Models (VLMs), which focus on interpreting map data rather than satellite or remote sensing images. As such, the task we are addressing is fundamentally different from remote sensing tasks, which is why we did not include remote sensing models in our evaluations. A more detailed comparison is added in Appendix A.3.

We hope this explanation clarifies the rationale behind our model selection. Thank you again for your valuable feedback.

References:

[1] Deng, Cheng, et al. "K2: A foundation language model for geoscience knowledge understanding and utilization." Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 2024.

Dear reviewer,

It is a gentle reminder to acknowledge our rebuttal and make necessary reassessment of the score. Again, we thank you for your time and effort in reviewing our work.

W6: In reality, I feel there's also a lack of discussion on the practical applications of large language models (LLMs) and visual language models (VLMs), as they tend to serve different purposes. Specifically, can the conclusions from this paper guide users on designing a comprehensive framework to solve real-world problems? I think further discussion in these areas would be beneficial.

In the introduction (Section 1), we discuss the motivation behind our work, which is grounded in real users' daily interactions with map services. Foundation models serve as a bridge between map services (APIs) and recommendation systems, enhancing user experience. Potential real-life applications could include copilots map service, AI assistants in customer service, or location-based web applications. We believe our benchmarking dataset will play a pioneering role in advancing research in this area.

Q1: I believe this research could be further improved by discussing why different models perform differently on the collected dataset, and exploring the relationship between the existing results and model designs to provide more insights.

Thanks for the valuable feedback. In this “dataset and benchmark” work, with comprehensive evaluation and analysis, we present fine-grained model performances and identify various root causes of their mistakes and study each foundation model performs in fine-grained categories. We also present complementary ways to enhance (such as in Section 4 particularly in 4.3 and Section 5 (newly added)) which can be inferred as identified limitations or weaknesses of models i.e., potentially on which reasoning aspects the models are not explicitly trained for. We believe these findings can help the development of more capable next generation foundation models. However, even for open-source foundation models, they rarely release full training data and procedure (i.e., in fact they are open-weight models) which limits us from further causal analysis studies. Even so, we believe such further studies are beyond our scope of this paper which we leave as future work.

W4: I think the paper might be lacking in terms of analyzing the distinct capabilities demonstrated by different models based on the evaluation results. The authors should draw conclusions on how to design a model that performs well across all aspects. This would greatly interest readers and significantly enhance the research value of the paper.

W5: If the authors could attempt and verify the suggestions from point four, I think this paper would be very strong.

Thank you for your valuable feedback. In response to your suggestion for a deeper analysis of the distinct capabilities demonstrated by different models, we have expanded our study to highlight their comparative strengths and weaknesses across various tasks in the benchmark. Our analyses cover a wide spectrum of possible enhancements for various aspects.

For instance:

• Claude-3.5-Sonnet excels in identifying cardinal directions, achieving an accuracy of 91%, while Gemma-2.0-27B has the lowest accuracy at 16.67% on this task.
• All LLMs struggled with measuring straight-line distances, with the highest accuracy being only 51.06%.
• In counting-related questions, the open-source Gemma-2.0-27B performed the best, achieving 60.87% accuracy, which underscores areas where other high-performing models, like Claude-3.5-Sonnet, could improve.

Therefore, ensembling different LLMs and their dynamic deployment based on user queries can be a potential solution. Furthermore, we have investigated specific failure cases to identify distinct behavioral patterns. For example, GPT-3.5-Turbo encountered a deadlock 16 times during MapEval-API evaluations, while Claude-3.5-Sonnet did not face such issues.

To address these issues, we extended model capabilities by providing access to external tools (e.g., calculator) specifically for calculating straight-line distances and cardinal directions. This resulted in a dramatic improvement, with accuracies increasing by over 50% in certain cases (see Table 6).

For instance:

• The accuracy of Claude-3.5-Sonnet in calculating straight-line distances increased from 51.06% to 85.11%, demonstrating the utility of integrating external tools.

• GPT-4o-mini, which initially struggled with cardinal direction tasks, saw its performance increase from 29.17% to 91.67%, showcasing a remarkable transformation with tool support.

• Even open-source models like Gemma-2.0-9B benefited, achieving an accuracy boost in straight-line distance tasks from 29.79% to 68.90%.

These improvements highlight the challenges LLMs face when reasoning spatially without external support. By leveraging tools, models can offload computationally intensive or context-specific reasoning tasks, enabling more precise and reliable results.

We also discuss challenges in API calling agents such as infinite loops (in our error analysis in Section 4.3) or further strategies for building enhanced agents in Section 5 such as using Chameleon [1].

We have added the new experimental results and in-depth analysis in the revised paper. Refer to section 4.3 and Section 5 for details.

Reference:

[1] Lu, Pan, et al. "Chameleon: Plug-and-play compositional reasoning with large language models." Advances in Neural Information Processing Systems 36 (2024).

Thank you for your feedback. We understand that there may have been some confusion, and it seems that you might have interpreted our paper as an interpretability or methodology paper. As a result, some of your comments, such as "this paper should be a paper in the data track," were a bit surprising. We would like to clarify that this paper is indeed a submission under the primary area of "datasets and benchmarks" for ICLR 2025. Nonetheless, we appreciate your insights and address your comments below.

W1: This paper introduces a dataset to evaluate the capabilities of existing models, but I didn't gain much insight from it. Specifically, I learned about the capabilities of different models on this dataset, but then what? It feels like an incomplete study. Perhaps this paper would be more suitable for a data track.

W2: Since the authors only collected a dataset and evaluated different models, the research lacks sufficient innovation. It appears more like an experimental report than a research paper.

W3: What I was hoping to see was some form of inspiration or insight from this paper, but unfortunately, I couldn’t find that. All I can gather is that different models may have advantages and disadvantages in different areas.

Given that this paper is submitted under the "Dataset and Benchmark" track, its primary focus is on introducing a benchmark dataset to evaluate the capabilities of existing large language models (LLMs) and vision-language models (VLMs). The key contribution lies in providing a standardized resource for assessing model performance in geospatial reasoning and location-based tasks, an area currently lacking robust benchmarks.

The insights gained from evaluating models on this dataset are intended to (1) highlight existing gaps in model capabilities (discussed in Section 4.2), (2) identify areas for improvement in geospatial reasoning (discussed in Section 4.2, 4.3, Appendix E) and (3) serve as a foundation for future research and model development (Section 4.2, 4.3, Appendix E and 5).

While the evaluation itself demonstrates the utility of the dataset, the broader implications include facilitating advancements in multimodal, tool-oriented and geospatial AI through the availability of a comprehensive benchmark. We believe this aligns well with the objectives of the "Dataset and Benchmark" category, where the focus is on providing resources that can drive progress in AI research.

We hope this clarification addresses your concerns and underscores the completeness of our study within the context of this track.

Dear reviewer,

It is a gentle reminder to acknowledge our rebuttal and make necessary reassessment of the score. Again, we thank you for your time and effort in reviewing our work.

W3: It would be more beneficial to test LLMs with different architectures, rather than focusing on different versions of the same model.

We have made a concerted effort to include a comprehensive range of state-of-the-art (SOTA) foundation models that are related spanning numerous widely adopted model families/series such as GPTs, Geminis, Gemma, Claudes, Llamas, Mistrals, Qwens, Phi and so on (28 foundation models) as discussed in Section 4.1. However, as open-source models are often evaluated with their smaller versions due to the resource constraints, we also needed to evaluate these variations. During this rebuttal, as per the suggestions from other reviewers we also have conducted additional experiments using 7 different models such as the newest Qwen2.5. Therefore, we believe our evaluation is fair, comprehensive, not biased to any LLM family and already addresses this feedback. We would be happy to explore any particular model if you recommend one.

Q1: There seem to be some problems with the API calling agent, can you explain more how you train the agent and possibly fix the problems of the agent?

It is unclear what you meant by “There seem to be some problems with the API calling agent”. In our experimental protocol and setup (Section 4.1), we mention that for a fair evaluation of the foundation models for MapEval-API task we evaluate all models uniformly through the widely adopted tool usage agent ReAct. We adopt the Zero Shot ReAct framework implemented by langchain (https://python.langchain.com/v0.1/docs/modules/agents/agent_types/react/). We clarify that this is an inference-only method that simply prompts the corresponding LLMs under consideration (e.g., instruct/chat versions of LLMs) for which no additional agent training is required. We also discuss challenges in API calling agents such as infinite loops (in our error analysis in Section 4.3) or further strategies for building enhanced agents in Section 5 such as using Chameleon [1]. Please note that there is a huge scope in improving the API calling agent that is left as future learning approaches.

References:

[1] Lu, Pan, et al. "Chameleon: Plug-and-play compositional reasoning with large language models." Advances in Neural Information Processing Systems 36 (2024).

W2: The sample size for each task is limited, you may consider to use some methods to scale up your benchmark with great quality.

(Also discussed in Response to Reviewer DM4A W2-Part2 and W3)

Due to the high cost of both foundation models as well as tools/APIs, recent language agents often tend to evaluate on a small number of sub-sampled dataset. For example ReACT [1] uses only 500 random samples from AlfWorld dataset, similarly Reflexion often tends to use [2] only 100 examples from HotpotQA). Therefore, recently proposed tool oriented or intense reasoning benchmark datasets are found to reasonable in size in order to be cost-effective: API-Bank: [3] (400 instances), Logical-reasoning benchmark LogiQA: [4] (641 examples), the most popular problem solving (code generation benchmarks) HumanEval: [5] (164 instances only), CodeContests [6] (156 problems), Tao-bench [7] (165 problems), OS World [8] (369 problems), App world [9] (750 problems), TravelPlanner [10] (1.2K problems). Consequently, we carefully construct our problem instances balanced in size and covering different challenges.

However, we thank you for the feedback and leveraging the rebuttal time-periods of ICLR, we strive to increase our problem set to 700 and ensure that each of the 54 countries is now represented by a minimum of 6 questions while maintaining the same quality control and pipeline. This improvement addresses the concern regarding insufficient representation of certain countries, making the evaluation more robust and geographically diverse. In our observation, further attempts in incorporating additional questions from different countries revealed no significant change in model performance. To maintain a cost-effective testbed, we opted not to include more instances, suggesting that our evaluation problem set already offers sufficient diversity for robust and reliable assessment. This improvement addresses the concern regarding insufficient representation of certain countries, making the evaluation more robust and geographically diverse.

In addition, as outlined in Section 3.1, our goal is to benchmark on high standard real-user queries aiming to solve day-to-day actual Map application tasks and hence we do not scale up by synthesizing.

References:

[1] Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." The Eleventh International Conference on Learning Representations.
[2] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).
[3] Li, Minghao, et al. "API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
[4] Liu, Jian, et al. "LogiQA: a challenge dataset for machine reading comprehension with logical reasoning." Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 2021.
[5] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).
[6] Li, Yujia, et al. "Competition-level code generation with alphacode." Science 378.6624 (2022): 1092-1097.
[7] Yao, Shunyu, et al. "$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv preprint arXiv:2406.12045 (2024).
[8] Xie, Tianbao, et al. "Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments." arXiv preprint arXiv:2404.07972 (2024).
[9] Trivedi, Harsh, et al. "AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
[10] Xie, Jian, et al. "TravelPlanner: A Benchmark for Real-World Planning with Language Agents". Poster presented at the Forty-First International Conference on Machine Learning (ICML), 2024.\

Thank you for your recognition of the applicability, effectiveness of our proposed benchmarking task, and comprehensive experiments in our work!

W1: Only multiple choice questions are adopted in MapEval. More diversity question type should be added to further explore the ability of LLMs, for real world problems are mostly open-ended.

Thank you for your suggestion. In Section 3.1 (line 147 onwards) we discuss the motivations for MCQ based evaluation choice over open-ended ones. As discussed while open-ended are real-life but their accurate and reliable evaluation is challenging. Therefore, we follow the widely adopted MCQ forms such as in MMLU [1], We note that while designing our queries, we deliberately choose MCQ for the evaluation purposes while keeping the queries flexible so that open-ended forms can also be made using our queries (simply without the options). Examples can be easily visible from Table 1, Table 12, and the qualitative examples in Appendix F. Regarding diversity on more question types, please note that each of our question types can further be decompose into multiple sub-categories such as commonsense reasoning, distance computation, cardinal directional etc., (as discussed in line 243, Section 4.3, Section 5). However for readability we cluster them into Table 1 categories. However, we welcome suggestions for additional real-life map service scenarios from daily activities.

References:

[1] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).

Dear reviewer,

It is a gentle reminder to acknowledge our rebuttal and make necessary reassessment of the score. Again, we thank you for your time and effort in reviewing our work.

W3: As shown in Figure 3, although the questions span 54 countries, many nations are represented by only a single question, which is insufficient to ensure a robust evaluation.

We appreciate the reviewer's observation regarding the spatial coverage of our dataset. In response, leveraging the rebuttal period, we have further annotated more data and increased the dataset size to 700 and ensured that each of the 54 countries is now represented by a minimum of 6 questions (see updated table 13) while maintaining the same quality control and pipeline. This improvement addresses the concern regarding insufficient representation of certain countries, making the evaluation more robust and geographically diverse. In our observation, further attempts in incorporating additional questions from different countries revealed no significant change in model performance. To maintain a cost-effective testbed, we opted not to include more instances, suggesting that our evaluation problem set already offers sufficient diversity for robust and reliable assessment.

W4 and Q2: The set of large language models and vision-language models assessed in the experiments is limited. Some important models, such as the Qwen2.5 series, Qwen2VL series, VILA series, and MiniCPM2.5, should be included to provide a more comprehensive evaluation.

We have evaluated Qwen-2.5 (7B, 14B, 72B), Qwen2-VL-7B, MiniCPM-Llama3-V-2_5 and Llama-3-VILA1.5-8B. All the results and analyses have been updated in the paper. Thanks for your valuable suggestion, as Qwen2-VL becomes the best performing open source VLM and Qwen-2.5-72B becomes the top open source LLM in the hardest textual task ‘Trip’.

W2-part-2: The tasks outlined in Table 1 and Figure 2 are overly simplistic, requiring more expert-level geospatial reasoning knowledge to enhance and extend the design of the evaluation framework.

We thank you for giving this feedback which enables us to further clarify the scope and complexity of our tasks and bring in a new set of analyses. While designing MapEval, our goal was not to synthesize linguistically complex questions, rather we ensure the data properties outlined in Section 3.1 such as real-user queries aiming to solve day-to-day actual Map application tasks. We argue that while some of these real-life queries (e.g., in PlaceInfo, and Routing category in Textual/API tasks) may appear simplistic, they often require deeper spatio-temporal reasoning to answer and domain know-how (i.e., beyond memorization). As mentioned in line 30, our benchmarking reveals that all models still fall short of human performance significantly (e.g., by more than 20% on average). In section 4.3, we identify a number of reasoning challenges in our tasks and refer to the full details of examples (both queries, contexts, options, and model outputs) in Appendix F listings to understand fully. In addition, we updated Table 1 (in the main paper), and presented Table 12 in Appendix to show additional complex queries. However, for your convenience we highlight some examples below.

• What is the direction of the Seurasaari Open-Air Museum from Helsinki Central Station? Options: East, West, North, South (Listing 2, Page 23, Appendix F).

Although it seems a simple question but to answer this question, the model (i) needs to identify the latitude and longitude (co-ordinates) of both Seurasaari and Helsinki Central Station (ii) Resolve two directions in terms of both latitude and longitude from the co-ordinates (iii) Determine the single major direction from them and respond.

Similar other linguistically simple yet challenging examples include:

• In the Nearby category, there are more intricate queries, such as those that incorporate the buffer concept to determine Points of Interests (POIs) within a specified distance of a given location. Example of some complex questions: “How many shopping malls are there within a 500 m radius of Berlin Cathedral?” “I am at Toronto Zoo. Today is Sunday and it's currently 8:30 PM. How many nearby ATMs are open now?”

• In the Routing category, finding the desired route from the step-by-step navigation is challenging due to long context and calculation. Example of some complex questions: “I want to walk from D03 Flame Tree Ridge to Aster Cedars Hospital, Jebel Ali. Which walking route involves taking the pedestrian overpass?” “On the driving route from Hassan II Mosque to Koutoubia via A3, how many roundabouts I will encounter in total?”

W2-part-1: The evaluation is relatively small in scale and lacks a systematic approach.

Due to the high cost of both foundation models as well as tools/APIs, recent language agents often tend to evaluate on a small number of sub-sampled dataset. For example ReACT [1] uses only 500 random samples from AlfWorld dataset, similarly Reflexion often tends to use [2] only 100 examples from HotpotQA. Therefore, recently proposed tool oriented or intense reasoning benchmark datasets are found to be reasonable in size in order to be cost-effective: API-Bank: [3] (400 instances), Logical-reasoning benchmark LogiQA: [4] (641 examples), the most popular problem solving (code generation benchmarks) HumanEval: [5] (164 instances only), CodeContests [6] (156 problems), Tau-bench [7] (165 problems), OS World [8] (369 problems), App world [9] (750 problems), TravelPlanner [10] (1.2K problems). Consequently, we carefully construct our problem instances balanced in size and covering different challenges.

However, we thank you for the feedback and leveraging the rebuttal time-periods of ICLR, we strive to increase our problem set to 700 while maintaining the same high quality. In addition, we believe that our contributions in evaluation strategies (such as three tasks with different challenges, utilizing multiple-choice question (MCQ) setups that enable reliable accuracy-based assessments, while also allowing for expansion into open-ended formats, evaluating all foundation models uniformly with the same ReACT agent for API task, etc), detailed performance studies, qualitative examples, fine-grained error analysis and newly added strategies for improvements, together constitute a complete benchmarking study.

All the statistics, results, error analysis, and strategies for model improvements are updated in the paper (in blue color).
In case you have any additional evaluation criteria in mind to further enhance MapEval, please feel free to specify so that we can address it.

References:

[1] Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." The Eleventh International Conference on Learning Representations.
[2] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).
[3] Li, Minghao, et al. "API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
[4] Liu, Jian, et al. "LogiQA: a challenge dataset for machine reading comprehension with logical reasoning." Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 2021.
[5] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).
[6] Li, Yujia, et al. "Competition-level code generation with alphacode." Science 378.6624 (2022): 1092-1097.
[7] Yao, Shunyu, et al. "$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv preprint arXiv:2406.12045 (2024).
[8] Xie, Tianbao, et al. "Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments." arXiv preprint arXiv:2404.07972 (2024).
[9] Trivedi, Harsh, et al. "AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
[10] Xie, Jian, et al. "TravelPlanner: A Benchmark for Real-World Planning with Language Agents". Poster presented at the Forty-First International Conference on Machine Learning (ICML), 2024.

Thank you for your recognition of the applicability, effectiveness of our proposed benchmarking task, and comprehensive experiments in our work!

W1, Q1: The discussion of related work is insufficient. For example, the paper does not discuss the differences and relationships between this study and prior research, such as GeoQA and GPT4GEO.

We would like to clarify that GeoQA [12] was indeed discussed in our paper, specifically in Section 2. However, we acknowledge that the discussion may have been brief and could have benefited from additional detail. Based on your feedback, we now provide a more detailed comparison.

Comparison with GeoQA Approaches:

Template-based GeoQA models ([2-6]) have predominantly followed a two-step strategy for answering geographic questions: (1) given a natural language query, selecting a template form (e.g., sql template) from a predefined list and (2) using these templates to query structured geographic knowledge sources (e.g., DBpedia [9], YAGO [11], Freebase [10], and OpenStreetMap). Overall, these approaches focus primarily on converting natural language questions into structured query language (sql) scripts. For instance, GeoQuestions1089 [6] contains 1,089 questions paired with their corresponding templated database queries (GeoSPARQL [7]) and answers derived from the geospatial knowledge graph YAGO2geo [8].

Our MapEval framework takes a fundamentally different approach. In MapEval-Textual, we shift from database querying or sql generation to evaluating geospatial reasoning in Large Language Models (LLMs). Annotators collect factual map services data (e.g., from GoogleMap), which serves as context for LLMs, allowing us to isolate and evaluate their reasoning capabilities rather than their knowledge retrieval. MapEval-API adopts a more practical approach by enabling direct interaction with map APIs to answer questions. Thus, in MapEval, LLMs are responsible for answering questions directly, unlike previous works where models generated database queries (e.g., Geoquery, GeoSPARQL) to retrieve answers from external knowledge bases. In addition, MapEval-vision employs multi-modal challenges.

Comparison with GPT4GEO:

GPT4GEO [1] examined GPT-4's geographic knowledge by probing it through template-based queries without API (plugins) or Internet access. Their evaluation focused on analyzing a single model using templated queries about generic location and direction-oriented facts, such as routing, navigation, and planning for well-known cities and places (e.g., renowned cities, shipping ports, airports, and coarse travel times like "8-hour flights to west coast" or plans like Frankfurt to Brussels: From Frankfurt, you'll need to take a train to Brussels, Belgium. This is a 3-hour train ride). While the findings suggest that GPT-4 shows promising geo-spatial knowledge, this approach neither establishes a benchmark for geo-spatial reasoning nor incorporates real-life user queries or map services (e.g., Google Maps) as a geospatial information base.

Our approach employs fundamentally different evaluation and design principles (outlined in Section 3.1). We establish a benchmarking of deeper geo-spatio-temporal reasoning capabilities across multiple foundation models using real user queries rather than templates. Uniquely, our evaluation encompasses multimodal understanding, tool interactions, and answerability determination. Additionally, we provide foundation models with fine-grained map services data through both context and API access, enabling a more comprehensive benchmarking of their geospatial question-answering abilities.

For more in-depth details, a comprehensive related work discussion has also been added in the Appendix A of the revised paper. Additionally, the related work section in the main text has been updated to address the feedback.

References:

[1] Roberts, J., et al. "GPT4GEO: How a Language Model Sees the World's Geography." arXiv, 2023.
[2] Zelle, J. M., and Mooney, R. J. "Learning to parse database queries using inductive logic programming." AAAI, 1996.
[3] Chen, W., et al. "A synergistic framework for geographic QA." IEEE ICSC, 2013.
[4] Chen, W. "Parameterized Spatial SQL Translation for GeoQA." IEEE ICSC, 2014.
[5] Punjani, D., et al. "Template-based QA over linked geospatial data." GIR, 2018.
[6] Kefalidis, S.-A., et al. "Benchmarking GeoQA using GeoQuestions1089." ISWC, 2023.
[7] OGC. "GeoSPARQL: A Geographic Query Language for RDF Data." Open Geospatial Consortium, 2011.
[8] Karalis, N., et al. "Extending YAGO2 with geospatial knowledge." ISWC, 2019.
[9] Auer, S., et al. "Dbpedia: A nucleus for a web of open data." ISWC, 2007.
[10] Bollacker, K., et al. "Freebase: A shared database of structured knowledge." AAAI, 2007.
[11] Suchanek, F. M., et al. "Yago: a core of semantic knowledge." WWW, 2007.
[12] Mai, G., et al. "Geographic question answering: Challenges and future directions." AGILE GIScience, 2021.

Dear reviewer,

It is a gentle reminder to acknowledge our rebuttal and make necessary reassessment of the score. Again, we thank you for your time and effort in reviewing our work.

Dear Reviewer,

Thank you very much for taking the time to give our paper a second look and for reviewing our responses and the additional experimental results. We greatly appreciate your thoughtful consideration and for acknowledging that we have addressed some of your concerns.

However, it is possible that some points may need more discussion. We would be grateful if you could kindly clarify the concerns that remain, so that we can continue to refine and improve our work.

Thank you once again for your valuable feedback.

Dear Reviewers,

We sincerely thank you for your hard work during the review process. We have carefully read your comments, provided additional clarifications, and made revisions to the paper. We are more than happy to engage in discussions with you, as it plays a crucial role in resolving your concerns and improving our paper!

However, as the rebuttal period will conclude in two days, we would be grateful if you could let us know whether we have addressed your concerns at your earliest convenience. If you still have any concerns, we would be delighted to address them promptly. If your concerns have been addressed, we would appreciate it if you could kindly reassess our work and increase the rating.