MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering

1. Gold standard for test set

Clarification on Ground Truth/Gold Standard Creation (Section 3): The term "answers" in Section 3.3 refers to ground truth labels, not model outputs. These were programmatically extracted from geospatial metadata (geojson files) using Python scripts—no visual map inspection was involved. Given that we had access to the original metadata and our questions were designed to be objective and factual (e.g., "What is the attribute class of AL?"), programmatic extraction was not only feasible but also more accurate and reliable than manual annotation.

Example process (Question Type: MCQ, Task Type: Retrieve Value):

1. Parse geojson file and extract the question and the map topic.
2. Identify target state (e.g., Alabama from "What is the attribute class of AL?")
3. Look up ground truth value (e.g., 11% population mobility).
4. Match to correct option (e.g., "d. 10.0–11.2").
5. Return ground truth answer (e.g., ['d']).

Scripts were developed for each Task Type–Question Type combination. We validated representative samples and iteratively refined scripts based on consistent error patterns, ensuring the programmatic approach provided us the gold standard for the test set. The ground truth responses for the test set are provided in the Supplementary Materials (OSF: Folder Name- Ground Truth Responses).

Clarification on Section 4 (Validation and Human Baseline Evaluation): Section 4 contains two distinct processes, both separate from ground truth creation in Section 3:

4.1 Validation (MLLM Response Cleaning)

This was the manual postprocessing performed across the entire test dataset for all model responses to ensure consistent evaluation.

Process Details:

a. Despite explicit formatting instructions (e.g., "Format your response as: 'My answer is [Your Option]'"), models occasionally included extraneous content such as reasoning steps, explanations, or hallucinated information alongside their answers.

b. Example (Molmo response):

• Question: What is the attribute class of WA?
• Raw Response: "To answer this question, I need to: [reasoning steps...] After examining the map and legend, Washington's circle falls into the second largest category, which corresponds to the 99.9–163 range. My answer is 99.9–163"
• Cleaned Response: [99.9, 163]

c. Example (Idefics response):

• Question: True or False: In the Midwest zone, the range of attribute value is between 26.5 to 30.4
• Raw Response: "No."
• Cleaned Response: FALSE

d. The manual validation ensured that:
(1) actual answers were correctly extracted from noisy responses,
(2) responses were formatted consistently for programmatic evaluation, and
(3) formatting variations didn't artificially penalize model performance.

e. Data Availability: All raw model responses and their cleaned versions are provided in the Supplementary Materials (OSF: "MLLM Responses > Baseline").

4.2 Human Baseline Evaluation

This describes a completely separate process: collecting human performance data by presenting 450 questions (~9% of the test set) to human evaluators under identical visual conditions as the MLLMs—meaning they saw the same maps and prompts without access to underlying metadata. This 9% sample establishes human performance benchmarks for comparison with MLLM results.

Important Distinctions:

• Different personnel: The ground truth and MLLM response validation were carried out by one human, which was distinct from the two humans involved in the human baseline evaluation.
• Different purposes: Validation ensures fair evaluation; human baseline establishes performance benchmarks.
• Different scope: Validation covered 100% of the test set; human baseline covered 9% for benchmarking.

Revision plan: We can clarify these distinctions in the manuscript and provide additional details where needed.

---

2. What are the five class labels?

The five class labels used in the Fisher–Jenks classification are:

1. Very Low
2. Low
3. Medium
4. High
5. Very High

Each of the 258 datasets contained continuous data that we discretized using the Fisher–Jenks classification scheme, which minimizes within-class variance while maximizing between-class variance. We chose five classes following standard cartographic practice and to ensure visual encodings remained interpretable for the MLLMs without excessive color or size variations.

---

3. Why do you assign scores of 0 or 100 to calculate accuracy?

We used the standard formula for accuracy—the number of correct answers divided by the total number of questions—which produces a value between 0 and 1. For consistency in reporting, we multiplied this value by 100 to express accuracy as a percentage. For example, an accuracy of 0.45 was reported as 45.

1. Qualitative Analysis and Useful Insights

We agree that qualitative error analysis is crucial for understanding model limitations. While we provide a few quantitative insights for each experimental variable (Section 5), along with ablation study results and analysis (Section 6 and Appendix A.3.2), given below are additional insights that can be extracted:

Failure Mode Analysis: From the results of the ablation study of Qwen2-VL (Section 6), we observed that altering legend font size led to a significant drop in accuracy, suggesting difficulty in interpreting complex or non-standard legends. Similarly, altering color encodings and scales (e.g., flipped, divergent, or spectral palettes) resulted in performance degradation on map-based questions, indicating insufficient visual grounding and sensitivity to design variations.

Map Type and Task Type Interaction Analysis: As a case study, we analyze Claude 3.5 Sonnet’s performance variation across interacting variables, specifically Map Type and Task Type. The model performs notably better on cartograms than on choropleth maps for Spatial Clusters (+5.76) and Retrieve Value (+3.63), likely due to the simplified spatial layout of uniform hexagonal bins, which reduces confounding visual variance. In contrast, symbol maps prove most challenging, especially for Pairwise Point Comparison and Retrieve Value, revealing difficulty in fine-grained size discrimination. Interestingly, for Spatial Clusters, performance on symbol maps exceeds that on choropleth maps (+0.60), suggesting the model can more reliably group similarly sized symbols than interpret similarly colored, variably sized regions.

Together, these findings highlight that MLLMs struggle not only with legend parsing but also exhibit modality-specific weaknesses in visual grounding depending on the map type and visual encoding strategy.

Revision Plan: We can expand this analysis in the appendix and provide specific examples (like the interaction analysis case study) to better understand failure modes and guide targeted improvements.

---

2. Use of Different Domain

Our study targeted low-level visual analytical tasks that relied purely on visual encoding interpretation, deliberately excluding domain knowledge requirements. The questions remained consistent across different map themes, which allowed us to isolate performance variations that may have stemmed from the models’ internal priors rather than differences in question content (see Section 5, Figure 3, variable "Theme"). Domain-specific questions would have shifted the focus toward high-level reasoning, fundamentally changing the research question from "Can MLLMs read visual map information?" to "Can MLLMs perform domain-expert analysis?" To maintain this focus and investigate topic-based bias, we deliberately chose real-world datasets instead of synthetic data. We aimed to explore whether MLLMs performed differently based on their prior knowledge of thematic domains (e.g., demographics vs. environmental data), given that the questions remained unchanged. Using realistic contexts enabled systematic probing of this bias, which would not have been possible with random synthetic data lacking meaningful semantic content.

---

3. Post-processing Details

Yes, postprocessing was manually performed across the entire test dataset for all model responses to ensure consistent evaluation. Despite explicit formatting instructions, models occasionally included extraneous reasoning, explanations, or hallucinations.

Examples:

• Question: What is the attribute class of WA? → Raw Response : "To answer this question, I need to: [reasoning steps...] After examining the map and legend, Washington's circle falls into the second largest category, which corresponds to the 99.9–163 range. My answer is 99.9–163" → Cleaned Response: [99.9, 163]
• Question: True or False: In the Midwest zone, the range of attribute value is between 26.5 to 30.4 → Raw Response : "No" → Cleaned Response: FALSE

Manual validation ensured: (1) correct answer extraction, (2) consistent formatting, (3) fair evaluation without penalizing formatting variations.

Data Availability: Raw and cleaned responses are available in Supplementary Materials (OSF: "MLLM Responses > Baseline").

---

4. Consistent Tense

We will review the text to ensure tense consistency throughout.

---

5. Gemini 1.5 Pro Performance

Gemini's below-random performance likely stems from response bias mismatch: ground truth skews toward False (1221 vs. 779 True), but Gemini systematically favors True predictions (981 vs. 639 False). Additionally, the model may struggle with uncertainty and negation—common zero-shot binary classification challenges.

1. Standardized Question Templates

Our experimental setup was designed to reflect a zero-shot setting, where all models are evaluated on the same set of standardized, task-specific question templates. This ensures consistency and fairness across models, eliminating any confounding effects from prompt engineering or fine-tuning. If by “question paraphrases” you are referring to variations in wording or alternative prompt formulations (e.g., rephrased instructions or stylistic changes), we intentionally did not explore such variants. Incorporating paraphrases would shift the focus toward prompt optimization strategies. Our goal was to assess inherent model capabilities under controlled conditions, not their performance under varying prompt formulations. That said, we agree that evaluating model robustness to paraphrasing is a valuable direction, and our dataset and framework can readily support such future extensions. We will update the future work section to recognize the need for this.

---

2. More Details on Human Validation and Human Baseline Evaluation

In Section 3.3, a single human expert was involved in validating the consistency and accuracy of the ground truth responses extracted programmatically using Python scripts. Since this step focused on verifying deterministic outputs from scripted logic, inter-annotator agreement was not applicable. For Section 4 (Human Baseline Evaluation), two expert map readers independently answered the map-based questions. Given the small annotator pool, we did not compute inter-rater agreement. Instead, we averaged their performance metrics across task types, following the same aggregation strategy used for reporting MLLM performance in Figure 2. This provided a reliable and practical estimate of human performance without requiring additional annotators. We will update Sections 3.3 and 4 in the manuscript to reflect these clarifications.

Important Distinctions:

• Different personnel: The ground truth response validation were carried out by one human, which was distinct from the two humans involved in the human baseline evaluation.
• Different purposes: Validation ensures fair evaluation; human baseline establishes performance benchmarks.
• Different scope: Validation covered 100% of the test set; human baseline covered 9% for benchmarking.

1. The benchmark doesn’t evaluate more complex or realistic tasks (e.g., navigation, multi-hop reasoning).

We appreciate this suggestion, but our study targets low-level visual analytical tasks (Section 2, Appendix A1.2) that rely on visual encoding interpretation, deliberately excluding higher-order reasoning. This design isolates models’ core visual comprehension skills without domain knowledge.

Tasks like navigation or causal inference involve high-level analytical reasoning, requiring complex cognitive processes and domain-specific knowledge. Evaluating those would shift the research question from "Can MLLMs read visual map information?" to "Can MLLMs perform advanced spatial reasoning?" Our current approach builds a baseline for visual understanding, which is essential before assessing more complex reasoning capabilities.

---

2. The benchmark lacks geographic diversity—it only uses U.S. state-level maps.

We acknowledge this and chose U.S. maps as a deliberate starting point:

• Exposure: U.S. data likely dominates MLLM pretraining. If models struggle with familiar maps, greater challenges can be expected with underrepresented regions. This setup provides a conservative baseline.
• Extensibility: Our framework easily scales to city, county, or international levels by substituting underlying data while maintaining task types and evaluation strategy.
• Future Work: We noted this limitation in the paper and believe expanding to international and finer-resolution data is a valuable next step.

Even with the most familiar geography, models exhibit notable performance gaps, validating our benchmark’s challenge level.

---

3. The analysis of model failure modes is somewhat shallow.

We agree and recognize the need for deeper failure analysis. However, several challenges shaped our current scope:

• Methodological Constraints: MLLMs—especially closed-source models like GPT-4V or Gemini—offer limited transparency. We cannot inspect internal representations, making in-depth diagnosis difficult.
• Current Analysis: We conducted ablation studies for Claude 3.5 Sonnet and Qwen2VL (Section 6, Appendix A.3.2), modifying visual elements like legends and color schemes. Results showed performance degradation when legends were small or color schemes were flipped/divergent—indicating weak visual grounding and limitations in legend parsing.
• Next Steps: We plan to expand this section with more insights and clearly tie them to MLLM limitations. Deeper per-model studies could offer further improvements.

---

4. Some question types (e.g., single-value answers) trip up models, but the authors don’t explore fixes.

We agree that improving performance on harder question types is important, but this falls outside our zero-shot benchmarking objective.

• Design Intent: We deliberately avoided question-specific tuning to assess core capabilities across models.
• Optimization Strategies: Techniques like few-shot learning or chain-of-thought prompting could help but would shift our goal from benchmarking to optimization.
• Future Work: We acknowledge this as a limitation and plan to support future work exploring such strategies. Our benchmark offers a standard evaluation foundation for doing so.

---

5. Why were newer models (e.g., Qwen2.5VL, recent Gemini) not included; zero-shot justification?

We selected the most representative models available when the study began. The rapid pace of MLLM updates makes continuous inclusion impractical without delaying research.

• Model Set: We included a diverse mix of open/closed models with different architectures and performance tiers for broad coverage.
• Benchmark Extensibility: The dataset and protocol are reusable, so future models can be evaluated comparably.
• Zero-Shot Rationale: We emphasized zero-shot settings to reflect realistic use without fine-tuning and avoid introducing confounding factors.