Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study

We want to thank the reviewer for your efforts in handling this submission and we appreciate you acknowledging novelty and utility of our proposed SIG and the rich appendix with detailed description. We will address your questions one by one in following.

W-1 Optimality is overstated. We agree that “optimality” is stronger than what our current evidence supports. We will refine the wording to: "derive a set of SIG-informed evaluation...". This phrasing better reflects how our metrics leverage the SIG to assess spatial reasoning without overclaiming optimality.

W-2 Choice of 3 context samples have impacts. To answer this question, we conduct extensive experiments investigating the effect brought by the choice of training samples for ICL on GPT-4o. The results are shown in the following table. As you can see, by more carefully selecting the training samples, the performance of ICL is much higher compared to random data sample selection we mentioned in our paper. These results further demonstrate the potential of SIG as a new data schema for improving visual-spatial intelligence of MLLM.

Table. Quantitative comparison of 3-shot ICL for grid-based VSI tasks on SIGBench-tiny with data selection based on number of objects on GPT-4o. Partial-empty means the number of objects in one or two of [vehicles, traffic signs, traffic lights] equals 0, while non-empty means the number of objects in vehicles, traffic signs and traffic lights in each data sample $\geq 1$. Results in Random is copied from the line GPT-4o ICL-SIG in Tab. 3 of our submission.

W-3 Human-like VSI offers little insights. We apologize for omitting certain implementation details due to space constraints.
Our primary reason for including human‑like VSI is to capture selective attention in driving: human drivers naturally focus on a small subset of scene elements—such as a red traffic light or a pedestrian stepping off the curb, when making safety‑critical decisions, rather than processing every object equally. Incorporating gaze‑derived saliency into SIG allows us to weight model errors according to human importance, offering a more systematic evaluation of decision‑relevant understanding, a feature currently absent in existing AD benchmarks. Our attention-based metrics (SRGS, SRD) not only assess whether a model can reconstruct spatial layouts, but also whether it can prioritize the right elements, mirroring the attention mechnism of a safe driver. This human‑centered weighting transforms SIGBench from a general VSI benchmark into one that evaluates the model’s spatial understanding in real‑world driving contexts.
We will provide detailed homographic transformation in Appendix.

Q-1 Values of gamma and beta. We set $\gamma = \frac{2}{|\mathcal{V}|}, \quad \beta = \frac{1}{|\mathcal{E}|}$ where $|\mathcal{V}|$ and $|\mathcal{E}|$ are the numbers of nodes and edges in the ground-truch SIG, respectively. This 1) bounds and scales insertion/deletion costs across graphs, and 2) weights edge operations more heavily (2:1) than node operations to reflect their inherently greater impact on GED.

Q-2 Why not attention-based MLSM? In designing our human‑like VSI metrics, we choose to apply gaze weighting to SRGS rather than MLSM for two key reasons:

• Metric focus: 1) MLSM evaluates the absolute placement of each object on the grid, i.e. how closely a predicted cell coordinate matches its GT; 2) SRGS assesses the relative structure of the scene graph. Human gaze naturally highlights relational importance ("Which objects matter most in context?") rather than precise metric distances. By weighting SRGS, we ensure that errors among highly attended object relationships incur larger penalties, while preserving MLSM’s role as an unbiased localization check.
• Artistic analogy: Consider an artist sketching a scene: they first lay out all object contours (the "big picture") before adding relational details where attention is focused. In this process, they focus on all objects inside the attention range instead of pixel-level or object-level information. Thus, we think attention-based SRGS is more aligned to the evaluation of human-like VSI.

Q-3 Why does relative ranking of MLLMs doesn't change significantly point to MLLMs are still unable to prioritize objects? Intuitively, if a model truly understood which objects garner human attention, penalizing errors on those high‑saliency objects more heavily would cause its performance to improve relative to others, and thus alter the ranking of models under gaze‑weighted metrics. In other words, a model that prioritizes the "right" objects would gain a bigger boost when those objects are weighted more. Because we observe that the order of models remains essentially unchanged after applying gaze‑based weights, this implies that none of the evaluated MLLMs consistently emphasize high‑attention elements over low‑attention ones. In turn, this stability in ranking indicates that current MLLMs have yet to internalize human‑like prioritization of scene objects.

Q-4 Insights on results of ICL on SIGBench-tiny and ICL using open-source models. The zero‑shot rankings on SIGBench and SIGBench‑Tiny are consistent: Gemini‑2.5‑Pro leads on MLSM and GPT‑4o on SRGS. Their ICL performance gains diverges significantly on SIGBench‑Tiny split. Our insight for this occurrence is that if a MLLM can determine the absolute spatial position of object correctly (revealed from metrics in MLSM), it's easy for it to improve the understanding of relative spatial relationship between objects. However, to improve the localization of objects in 3D space requires further training and fine-tuning. For open-source models, we conduct the ICL using Qwen-VL-7B and Qwen-VL-32B on SIGBench-Tiny. The following results demonstrates both models are acquire significantly improvements on MLSM and SRGS metrics, highlighting SIG's strong potential for improving VSI. Our SIG representation can also be generalized to other domains besides AD, please refer to reviewer wMQv W-1&Q-2.

W-2 Choice of 3 context samples have impacts. We further investigate ICL with adaptive SIG samples. Different from heterogeneous SIG samples, which present each shot with a distinct image and its GT SIG, adaptive SIG samples uses the same image across multiple shots, pairing each predicted SIG with the GT SIG for direct comparison. Specifically, we select and rank several candidate SIGs for each image by their precision scores under the MLSM metric, then present these ranked SIGs as few‐shot examples. Intuitively, exposing the model to both high‐ and low‐quality SIG should sharpen its ability to discern and leverage the best spatial descriptions of the image and convey it using SIG. This strategy mirrors human learning: comparing several imperfect attempts against a correct one sharpens the ability to discern and internalize the optimal solution. The results are shown in the following Table. For MLSM and SRGS, ICL using adaptive SIG samples delivers substantial gains over using heterogeneous SIG samples from 3- to 6-shot ICL. This means when the number of data samples is relatively less, adaptive SIG samples can bring better improvements in model's VSI. However, for 9-shot ICL, heterogeneous SIG samples continue to yield further improvements, whereas adaptive SIG samples plateau over 6-shot, which indicates adaptive SIG samples' limited scalability compared to heterogeneous SIG samples. These occurrences underscore the principle that, while in-depth exploration of a single SIG sample is valuable, achieving optimal VSI also requires learning from a broad diversity of instances. For SRD, adaptive SIG samples consistently achieve better performance than using heterogeneous SIG samples. Taken together, these findings indicate that neither strategy alone is sufficient. To maximize the performance of SIG‐based ICL, we should combine adaptive and heterogeneous SIG samples.

Table. Quantitative comparison of few shot ICL for grid-based VSI tasks on SIGBench-tiny with data selection based on adaptive SIG on GPT-4o. Different means we use different image and its corresponding GT SIG (e.g. 3-shot uses 3 different images), while adaptive means using the same image with different predicted SIG and GT SIG (e.g. 3-shot uses the same image with 3 different predicted SIG).

Please refer to our new comment to reviewer wMQv W-1&Q-2 for the udpated results on SIG-COCO and SIG-ARKitScenes, which demonstrates the strong generalizability of our proposed SIG to domains other than autonomous driving scenario.

Models' ability to prioritize salient objects. Thanks for your helpful feedback. We agree that our current results cannot completely prove that MLLMs do not have the ability to prioritize the object in a scene. We also implement the attention-based MLSM (Human-like MLSM) and the results are shown in the following table. The ranking of MLLMs in MLSM and H-MLSM are still similar, we hope this results can bring some more insights.

Table. Quantitative comparison of different MLLMs on SIGBench for MLSM and Human-like MLSM (H-MLSM). We incorporate human attention into MLSM to create H-MLSM. The MLSM results are borrowed from Tab. 2 in our manuscript for comparison.

Please let us know if you have any further technical questions or concerns about our method and results and we would be happy to discuss them! Thanks again!

Table. Error bar statistics of ICL on Gemini-2.5-Pro using random data selection with 5 run.

Error Bar Statistics of Multi-run ICL. Thank you for your expertise—your review and comments has further strengthened our work. We re-run ICL with random data sample selection on SIGBench-tiny for 5 times using GPT-4o and Gemini-2.5-Pro. The error bar statistics are listed in the following tables. Due to space limitation, we separate two tables to two comments.

Table. Error bar statistics of ICL on GPT-4o using random data selection with 5 run on SIGBench-tiny. ICL-MC meaning ICL using multiple-choice VQA and ICL-SIG meaning ICL using SIG. Std means standard deviation and Sem is standard error of mean.

I think comparing MLSM with and without gaze weighting can indeed paint a picture on how well the current models prioritize objects that are more meaningful to humans. And from the results you show, it does indeed look like most models do not work better on more salient objects (with the exception of, for instance Gemini-2.5-Pro). I am just not sure why the conclusions are being made based on how well the model ranking is preserved, as opposed to just looking at individual models, comparing their results in, for instance, MLSM and HMLSM, and pointing out which change and which don't (since the ranking can in principle also be preserved in the case where all the models are equally good at prioritizing important objects). Other than that, I think this is indeed an interesting and important finding, I am just not sure about that piece of reasoning.

We want to thank the reviewer for your efforts in handling this submission and we appreciate you acknowledging the importance of our work in improving spatial scene understanding of MLLMs in autonomous driving (AD) scenario, along with extensive evaluation. We will address your questions one by one in the following.

W-1 No human baseline. We agree that a human baseline is essential for contextualizing model performance. Accordingly, in Tab. 2 (first row) we report the average scores of four human subjects on our general VSI tasks. We will emphasize these results more prominently in the revised experiments section to clarify how current SOTA models compare against human-level spatial reasoning.

W-2&L-2 Omitting lanes from the evaluation. We agree with the importance of lane and drivable‐space understanding in AD scenarios. You are absolutely right that lane assignments (e.g. “Car 1 is in the left lane”) is critical for maneuver planning. Our SIGBench dataset does include human‐annotated lane masks for images where lanes are visible. Currently, there are several mature methodologies specifically designed for traffic lane detection such as [1], [2], [3]. However, in this work we mainly focus on the spatial relationship among objects rather than spatial traffic rules. Moreover, SIGBench encompasses a diverse range of driving scenarios: some featuring faded or blurred lane markings and others (e.g., intersections or rural roads) where lane demarcations are entirely absent. In the revised manuscript, we will select a subset of images with clearly visible lane annotations and extend our evaluation metrics to incorporate lane‑based spatial relationships for this subset.

W-3 Dynamic spatial relationships between objects. Thank you for this insightful suggestion. We agree that modeling dynamic spatial relationships, such as vehicles approaching a pedestrian crossing or accelerating toward the ego vehicle, is a critical aspect of visual‑spatial intelligence. As SIGBench currently focuses on single‑image scenarios, inferring motion or intent from a static frame can be challenging even for human observers. However, because our source data consists of video sequences, we intend to revisit these clips and introduce temporal annotations in a future extension of the benchmark. This enhancement will allow evaluation of models’ ability to reason about object trajectories and evolving spatial configurations.

W-4&Q-3 The 10x10 grid resolution is coarse. Choosing 10×10 discretization for our Spatial Intelligence Grid (SIG) was motivated by typical U.S. highway layouts: mostly up to four lanes per direction (eight total), plus one extra column on each side for signs and lights, yielding 1+8+1=10 horizontal cells. To form a square grid, we also use 10 rows vertically.
To give an intuition for how this maps to real‑world dimensions, consider a reference plane at 27m in front of the camera. The camera used for recording has focal length 3.6mm and active area =5.4mmx4.0mm. Since our image size is $1920\times1080$, we can get the focal length in pixels as: $f_x = \frac{3.6mm}{5.4mm} \times 1920 = 1280\text{px}, \quad f_y = \frac{3.6mm}{4.0mm} \times 1080 = 972\text{px}$ If we divide the image into 10x10 grids, each image cell is 192px wide ($\Delta u$) and 108px tall ($\Delta v$). If we have $Z = 26$m, we can solve the width ($\Delta X_{grid}$) and height ($\Delta Y_{grid}$) of each cell in our SIG by:

which means each grid in our SIG represents approximately a 4mx3m rectangle plane in real world. We also provide more comprehensive results for SIG-based ICL, please refer to reviewer BBAE W-2 and Q-4. Our SIG representation can also be generalized to other domains besides AD, please refer to reviewer wMQv W-1&Q-2.

W-5 Whether asking MLLM to generate python code for saliency prediction affects the outcome. We choose a Python‑code prompt for saliency prediction generation using MLLMs for two main reasons:

• Token‑limit constraints: A full 1080x1920 attention map contains 2073600 values, far exceeding the maximum output capacity of our evaluated models (e.g., 64000 tokens for GPT‑4o, which is already much larger than that of most proprietary and open-source models). If we want the GPT-4o to output the full attention map of one image, we need 2073600/64000 = 32.4 iterations. Thus, asking the model to emit code that reconstructs the map circumvents this limit by having the model produce a compact algorithm rather than raw data, which can then be executed to yield the full attention tensor.
• Execution robustness: By prescribing a code template that references only our approved libraries and data structures, we ensure that the generated routines run correctly and consistently in our environment. This approach also allows us to validate and debug any edge cases in advance, increasing overall reliability.

Through extensive prompt engineering and manually checked over 100 cases, we found that this code‑templating strategy does not bias the resulting attention maps compared to raw‑value generation. Instead, it simply provides a practical mechanism to deliver high‑resolution saliency outputs within existing MLLM constraints.

Q-1 Does the SIGBench dataset also preserve object IDs that supports tracking? While SIG provides a natural schema for object identity and could facilitate cross‑frame tracking, SIGBench in its current form is limited to single‑image annotations and does not include persistent object IDs. Establishing reliable object tracks would require extensive video‑based labeling, which is beyond the scope of this initial release. We agree that integrating object IDs and temporal relationships would greatly enhance the benchmark, and we plan to explore this extension along with the dynamic object relationship in a future SIG video dataset.

Q-2 Scale-invariant matching. We agree with you that relying on a single camera and frame can introduce scale ambiguity in depth estimation. Our choice of this setting was deliberate: by demonstrating that SIG‑based ICL improves visual‑spatial reasoning under these challenging constraints, we establish a strong foundation. Importantly, our SIG representation is agnostic to sensor modality or scale: it can be directly extended to multi‑camera rigs or LiDAR–camera fusion systems that yield metrically accurate depth.
Although we have not yet performed explicit scale‑invariant matching against BEV ground truth, our grid‑based metrics: Multi-level Spatial Matching (MLSM) and Spatial Relation Graph Similarity (SRGS), operate on normalized cell indices rather than absolute distances, making them naturally robust to uniform scale shifts. More specifically:

• MLSM builds a cost matrix based on the cell‑coordinate differences between predicted and ground‑truth objects; since both are expressed in the same 10×10 index space, any uniform scaling of the scene (e.g., due to camera zoom or depth ambiguity) cancels out.
• SRGS compares the structure of scene graphs via node and edge matches. Its graph‑edit operations count mismatches in graph topology and relation labels rather than measuring literal distances in meters.

As a result, both metrics remain unaffected by global scale shifts: they focus purely on relative spatial configuration and object relationships, ensuring consistent evaluation across different calibration settings.

L-1 Predicted human visual saliency will result in systematic prediction error. We apologize for any misunderstanding. In our experiments, ground‑truth human visual saliency was captured using the SmartEye Pro 5 eye‑tracking system. According to the device specifications and validation studies conducted by SmartEye, the gaze‑direction measurements achieve a precision of approximately $\pm0.5^{\circ}$ [4]. This level of accuracy is sufficient for our purposes, and we therefore consider these recordings to represent reliable ground‑truth saliency.

[1] Xingang Pan, Jianping Shi, ..., Xiaoou Tang. SCNN: Spatial As Deep: A Convolutional Neural Network for Scene Segmentation of Lane Markings. ECCV 2018
[2] Lucas Tabelini, Rodrigo Berriel1, ..., Thiago Oliveira-Santos. Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection. CVPR 2021
[3] Lizhe Liu, Xiaohao Chen, Siyu Zhu, Ping Tan. CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution. ICCV 2021
[4] Smart Eye AB. SmartEye Pro 5 User Manual (Revision 249)

Please refer to our new comment to reviewer wMQv W-1&Q-2 for the udpated results on SIG-COCO and SIG-ARKitScenes, which demonstrates the strong generalizability of our proposed SIG to domains other than autonomous driving scenario.

Please refer to our new comment to reviewer BBAE W-2 for the new results on SIG-based ICL with another data sample selection strategy, which highlights the potential of SIG as a new data schema for improving VSI of MLLMs.

Thank you for your appreciation of our rebuttal. Your thoughtful feedback has clearly strengthen our paper.

Your ideas about dynamic spatial relationships between objects, scale-invariant matching and object id tracking are insightful and represent promising topics for our future work. We will explicitly discuss these aspects as limitations of our current work and directions for future work in the final version of our paper.

Regarding the Python code generation for gaze prediction, our key idea is ensuring the executability of the generated code. For instance, if the model outputs code that imports libraries not available in our conda environment, it fails to produce the predicted gaze attention and instead raises runtime errors. To mitigate this, we explicitly constrain the allowed libraries within the prompt and will automatically rerun the inference until the code is executable. Additionally, during manual inspection, we focus on whether the MLLMs are truly performing gaze prediction based on the attention from previous five frames, rather than simply averaging them. We have reported such occurrences in the manuscript (Lines 284–285) and provided further explanation in Appendix C.3. We will include more detailed clarification on this in the final version.

Thank you once again for your valuable efforts in reviewing our work. Should you have any further technical questions or suggestions, we would be glad to address them and continue the discussion.

We want to thank the reviewer for your efforts in handling this submission and we appreciate you acknowledging our proposed evaluation metrics are well-motivated and our extensive experiments provides strong empirical evidence for Spatial Intelligence Grid (SIG)'s potential in improving visual-spatial intelligence (VSI) of MLLMs. We will address your questions one by one in the following.

W-1&Q-1 Can SIG generalize to other domains than AD. Please refer to our response to reviewer wMQv W-1&Q-2

W-2&Q-2 Analysis of runtime. Runtime efficiency is definitely important, especially for real‑time applications in autonomous driving (AD). Below we summarize both the theoretical complexity of our metrics and empirical timing results on SIGBench.
Theoretical time complexity: Let $n$ denotes the total number of objects (vehicles, traffic signs and traffic lights) per frame.

• For Multi-level Spatial Matching (MLSM), the time complexity for building cost matrix for vehicles, traffic signs and traffic lights is $O(n^2)$. The time complexity for bipartite matching is $O(n^3)$ [1]. For multi-threshold matching, the time complexity is $O(n)$. The overall time complexity for MLSM is $T_{MLSM}(n) = O(n^2) + O(n^3) + O(n) \Rightarrow O(n^3)$.
• Similarly, for Spatial Relation Graph Similarity (SRGS), during the fully connected graph construction, we require $O(n)$ for all nodes and $O(n^2)$ for all edges. Next, the bipartite matching requires $O(n^3)$ similarly. During the graph edit distance (GED), we are performing node-to-node and edge-to-edge comparison, which requires $O(n) + O(n^2)$. Thus, the time complexity for SRGS is $T_{SRGS} = 2(O(n) + O(n^2)) + O(n^3) \Rightarrow O(n^3)$.
• For Semantic Relational Distance (SRD), we are computing pointwise distance. For SRD (Directional) and SRD (Proximal) there are 8 and 5 possible relations, respectively. Thus, the time complexity for SRD is $T_{SRD} = O(8n) + O(5n) \Rightarrow O(n)$.

Empirical timing: The dataset statistics of SIGBench is:

The average per-frame runtimes on CPU (AMD Ryzen 5900X) are:

Estimated real-time performance: Consider a typical AD vehicle with six cameras at 30 FPS and an average of 11 detected objects per frame. The total estimated time cost for our metrics per second (without parallelization) is: $6 \times 30 \times (0.0002 + 0.0004 + 0.0001 ) = 0.126$s, which definitely meets the real‑time demands. We will include these discussion in the Appendix.

W-3&Q-3 Qualitative and quantitative analysis of typical failure modes. To explore common failure modes, we examine zero‑shot performance of two state‑of‑the‑art MLLMs on SIGBench across scenes containing varying numbers of objects. Results are summarized in the following table.

GPT‑4o’s MLSM and SRGS scores steadily decline as object count rises, while its SRD performance fluctuates, demonstrating that graph‑based metrics are more sensitive to scene density and that raw text judgments alone can mask these trends. Gemini‑2.5‑Pro shows similar SRD variability but even improves on some graph metrics (MLSM‑P, SRGS) with more objects, underscoring the complementarity of SIG’s structured representation. Qualitatively, we recognize two key failure scenarios:

• Objects that are either very small or positioned at the periphery of the scene are frequently missed or mislocalized due to their low visual salience.
• High intersection over union (IoU) between objects often cause the models to conflate object identities or assign them to incorrect grid cells, resulting in swapped or merged scene‑graph nodes. We will illustrate each of these failure modes with representative examples with visualization in the Appendix.

W-4 Full fine-tuning or RLHF based on SIG. Thank you for this insightful suggestion. We agree that exploring end‑to‑end integration of SIG into training pipelines could substantially advance model capabilities. We provide more comprehensive results for SIG-based ICL, please refer to reviewer BBAE W-2 and Q-4. Based on our positive ICL results, we think two particularly promising directions are:

• SIG‑guided supervised fine‑tuning. Rather than relying solely on in‑context examples, we could curate a labeled SIG dataset and fine‑tune an MLLM to predict grids and scene graphs directly from raw images. This supervised signal would reinforce precise spatial encoding, potentially yielding improvements not only in SIG reconstruction but also in downstream tasks like VQA or navigation with an improved visual-spatial intelligence with large generalizability.
• RLHF via SIG. SIG outputs offer a structured, graph‑based explanation of model reasoning. By treating human judgments of SIG quality as reward feedback, e.g., ranking or correcting scene graphs, an RLHF loop could encourage models to align their spatial representations with human preferences. Such a protocol may cultivate nuanced attention patterns and more robust relational reasoning tailored to diverse driving contexts. Both avenues harness the rich structure of SIG to move beyond one‑shot prompting toward production‑ready spatial intelligence, and we will investigate these two interesting directions in future work.

Typo at line 120; Line 254 $\rightarrow$ Pearson. We are deeply apologize for these typos and we will correct them in the final version of this paper.

[1] T. H. Cormen, C. E. Leiserson, R. L. Rivest, C. Stein, Introduction to Algorithms, 3rd ed., MIT Press, 2009, Chapter 16.

W-1&Q-1 Can SIG generalize to other domains than AD. Please refer to our new comment to reviewer wMQv W-1&Q-2 for the udpated results on SIG-COCO and SIG-ARKitScenes.

Please refer to our new comment to reviewer BBAE W-2 for the new results on SIG-based ICL with another data sample selection strategy, which highlights the potential of SIG as a new data schema for improving VSI of MLLMs.

We want to thank the reviewer for your efforts in handling this submission and we appreciate you acknowledging the novelty of our proposed data representation Spatial Intelligence Grid (SIG) and evaluation metrics: Multi-level Spatial Matching (MLSM), Spatial Relation Graph Similarity (SRGS) and Semantic Relational Distance (SRD). We will address your questions one by one in the following.

W-1&Q-2 This is likely applicable only for AD scenarios. Although our initial motivation arise from autonomous‑driving (AD) applications, the SIG framework is fundamentally domain‑agnostic and can be applied wherever a fixed ontology of object types exists. To demonstrate this, we have built two proof‑of‑concept benchmarks based on subsets from MS COCO [1] and ARKitScenes [2], named as SIG-COCO and SIG-ARKitScenes, respectively. We run both zero-shot inference and ICL using GPT-4o on these two benchmarks. The results are shown in the following table. Due to time limitations and workload of dataset annotation, we only provide partial results of MLSM and SRGS here, we will update the results of SRD related metrics in 3-4 days.

W-2 Position of this work with other related works. Thanks for drawing our attention to the related literature. In particular, the paper [3] you suggested offers valuable insights closely aligned with our work. In the revised manuscript, we will integrate a discussion of [3] alongside citations to [4], [5], [6], and [7], clearly positioning our contributions in the context of these existing studies. If you are aware of any additional papers that would further enrich our comparison, we would be grateful for your recommendations and are happy to include them.

W-3 Similar as map-based method. While the map‑based approach in [3] bears a conceptual resemblance to our SIG, our work departs in several important ways:

• Scenarios from real‑world: We annotate SIG directly on real autonomous‑driving scenes rather than synthetic or abstract maps, ensuring our benchmarks measure performance under the complex visual conditions encountered “in the wild.”
• Richer object classes: Beyond a generic “object” class, our SIG distinguishes multiple vehicle types (cars, trucks, vans, buses), traffic signs, and lights, and further encodes intra‑class attributes such as color and order.
• Instance‑level complexity: By including scenes with several instances of the same category—each with its own spatial relation and attribute differences—we stress‑test a model’s ability to generalize spatial understanding to densely populated environments.
• One‑shot relational output: Our evaluation framework invites MLLMs to output a complete SIG in a single pass, mirroring how humans naturally describe spatial relations in a cohesive narrative, rather than iterative queries of VQA in [3], which has been studied by enormous of existing papers.

These distinctions underscore how SIG extends and challenges map‑based representations to better capture the visual‑spatial intelligence needed for real‑world driving applications.

W-4 Better motivation for proposed evaluation metrics. When a driver is driving, he/she typically focus on a range containing selected objects and their spatial correlation instead of pixel-level or object-level information. To build a representation that preserve this, inspired from classical grid‑based visual schemes (e.g., Renaissance “drawing machines”) that artists used to decompose complex scenes into structured partitions, we introduce SIG that preserves fine-grained geometric and topological information in a 10×10 grid, directly encoding object positions and their spatial relations. Based on SIG, we design three complementary evaluation metrics:

• MLSM: A thresholded, bipartite‑matching protocol that tolerates minor grounding noise while measuring localization precision at increasing levels of attribute matching (type, order, color).
• SRGS: Based on graph edit distance (GED), this metric rigorously quantifies structural differences between predicted and ground‑truth scene graphs, penalizing missing or mis‐classified relations.
• SRD: By mapping prepositions onto discrete scales (cyclical for direction, linear for proximity), SRD captures the degree of semantic error, rather than a binary correct/incorrect judgment.

W-5 Connection to human gaze evaluation is weak. While the core contribution of this work lies in introducing SIG and SIG‑based evaluation metrics, modeling human‑like VSI via gaze serves two essential roles:

• Bridge to real‑world decision making: In AD scenario, human drivers do not process every object equally. They deploy selective attention, focusing on a few critical elements such as traffic lights at an intersection—while deemphasizing others (e.g., distant vehicles). By integrating gaze‑derived saliency into SIG (Eq. 6), we weight our graph‑ and grid‑based metrics so that high‑importance objects incur greater penalty when mispredicted.
• Human‑aligned relational evaluation: Traditional VQA or text‑only metrics treat each relation equally, ignoring that some spatial errors are more consequential than others. For instance, confusing “at the front of” versus “at the back of” a high‑attention object (e.g., a braking light) can have drastically different safety implications. Our Human‑Like SRGS and SRD scale node‑ and edge‑edit costs by gaze weights, ensuring that mistakes on visually salient entities are penalized more heavily.

Together, these gaze‑weighted evaluations not only emulate how humans allocate attention in dynamic driving scenes but also align model assessment with the situational priorities of real‑world AD systems—thereby strengthening the practical relevance of our human‑like VSI benchmark.

Q-1 Examples of queries and difference from SpatialVLM and VSR. For examples of queries, please refer to Appendix C. Below we illustrate how query formats in SpatialVLM, VSR, and SIGBench differ—both in their structure and in how they are obtained:

This direct, structured prompting—together with our ground‑truth SIG annotations—allows SIGBench to evaluate complete spatial understanding in one shot, rather than separate VQA pairs. We also provide more comprehensive results for SIG-based ICL, please refer to reviewer BBAE W-2 and Q-4.

[1] Tsung-Yi Lin, Micheal Maire, ..., Piotr Dollar. Microsoft COCO: Common Objects in Context.
[2] Gilad Baruch, Zhuoyuan Chen, ..., Elad Schulman. ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data.
[3] Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Krahenbuhl, Vladlen Koltun. DOES SPATIAL COGNITION EMERGE IN FRONTIER MODELS?
[4] Santhosh Kumar Ramakrishnan. Predictive scene representations for embodied visual search.
[5] Runsen Xu, Weiyao Wang, ..., Kevin J. Liang. Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models.
[6] Sihan Yang, Runsen Xu, ..., Jiangmiao Pang. MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence.
[7] Linjie Li, Mahtab Bigverdi, ..., Ranjay Krishna. Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations.

W-1&Q-2 This is likely applicable only for AD scenarios. We update the complete results of zero-shot and 3-shot ICL on SIG-COCO and SIG-ARKitScenes for general VSI tasks using GPT-4o as follows:

Please refer to our new comment to reviewer BBAE W-2 for the new results on SIG-based ICL with another data sample selection strategy, which highlights the potential of SIG as a new data schema for improving VSI of MLLMs.