REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories

If you have time, please read the general comment before reading your specific response, as we feel it builds on it.

Thank you for your comprehensive analysis. Below we address your specific concerns:

1. Evaluation methodology.
   Your “left and right” example is important—our verification correctly marks such answers as incorrect. We use token‑subset checking: if the ground truth is “left,” the model’s output must contain “left” and not “right.” Any ambiguous response fails (rare with our prompting). For counting, we extract a single integer via regex; multiple or missing integers fail. This strict evaluation means only unambiguous correct answers pass. Full verification code will be provided, and the procedure will be documented in the appendix.

2. Frame sampling and scene coverage.
   Frames are not randomly sampled; trajectories are. Each follows deterministic movements (15° rotations, 1 m forward steps). While coverage varies, questions are generated solely from what the camera sees and only when unambiguous. Details of the QA‑generation and verification pipeline will be added to the appendix.

3. MLLM usage and prompting.
   Models receive (1) the full image sequence, (2) explicit actions between frames, and (3) clear instructions about the Blender environment, the task, and the required answer format. We use zero‑shot prompting with frame indexing (e.g., “Frame 0: [image], Action: rotate left 15°, Frame 1: [image] …”). Exact prompts and rationale will be documented in the appendix.

4. Value of diagnostic benchmarks.
   Like GLUE in NLP, targeted evaluations drive progress by exposing weaknesses. REM’s systematic ablations show spatial‑reasoning failures scale predictably with scene factors (Figures 4‑7); the 80–89 % failure rate on Full Rotation highlights a fundamental MLLM limitation in object permanence.

We appreciate your thorough review—your feedback strengthens REM’s contribution as a diagnostic tool.

Thank you for your rebuttal feedback! We’re excited that CoLM has introduced this discussion week so we can respond to any specific concerns you still have about the methodology. If you have time, please feel free to include specific aspects you’d like clarified in your reply, and we’ll respond as quickly as possible.

Regarding the unrealism concern: much like CLEVR’s “unrealistic” blocks proved key to advances in visual reasoning, we hope our controlled environments can do the same for embodied spatial reasoning by demonstrating fundamental limitations of LLM spatial memory.

We look forward to your specific feedback on what methodological insights would be most valuable to include!

If you have time, please read the general author comment as we feel your specific response builds on it.

Thank you for recognizing REM's value in revealing MLLMs' lack of internal world models. We appreciate your specific concerns:

1. Ground‑truth ambiguity.
   Your example perfectly illustrates our robustness checks. If the black sphere ever appears right of the green sphere, that pair is filtered out—no question is generated. Concretely:

   1. We project 3D positions to camera coordinates for every frame.
   2. We track left/right consistency across the trajectory; if black is left of green in frames 1, 5, 8 but right in frame 12, the pair is rejected.
   3. A render‑pass pixel count yields exact visibility percentages; only objects with > 0 % visibility are included.
      Thus every generated question has exactly one correct answer. We will add visual diagrams of this process. Ground truth is derived from Blender’s scene graph, mirroring CLEVR’s methodology.

2. Human performance without natural backgrounds.
   This setup is why we expect humans to excel—the task isolates spatial reasoning. Models are explicitly told (via prompts in the appendix) that objects are distinct Blender shapes on a ground plane and must be tracked through the trajectory. The Full‑Rotation experiment—where GPT‑4o fails 80–89 % of the time—would be trivial for humans who recognize that views at 0° and 180° contain different objects. A human baseline is forthcoming.

3. Symbolic objects.
   Using maximally discriminable shapes is intentional and diagnostic. When models fail catastrophically even under ideal perceptual conditions, the deficit can only stem from spatial reasoning, revealing fundamental architectural limitations.

Your insightful questions help us clarify why REM's controlled approach is essential for uncovering these core deficits.

Thank you for recognizing REM's value in revealing MLLM limitations and motivating trajectory‑consistent representations.

Object movement.
Our focus on camera movement only is a deliberate methodological choice. By keeping objects static, we isolate pure spatial reasoning and viewpoint integration from motion‑tracking complexity—establishing a critical baseline before adding object dynamics. We will clarify this design rationale and note dynamic objects as a valuable future extension.

2.5D scenes.
You correctly identify our scenes as 2.5D. This design enables systematic study of spatial‑reasoning limitations relevant to many real scenarios (e.g., street navigation, indoor robotics). While we will acknowledge this more explicitly and note full 3D as a future direction, the current design already reveals severe model limitations even in simplified settings.

Frame‑count scaling.
Your intuition about frame limits is validated by our results—performance significantly degrades with trajectory length (Figures 4‑7). This finding is a key contribution, quantifying exactly how current models fail to scale to longer visual sequences and motivating architectural improvements.

We appreciate how your feedback helps us better articulate these methodological choices and their diagnostic value. If you have time, please also read the general comment for more context.

If possible, please read the general summarized comment before your specific response!

Thank you for recognizing REM’s value as a controlled 3D environment for systematic evaluation. Addressing your concerns:

Ground truth generation at scale.
Our template system generates thousands of valid questions because each relies on perfect knowledge, deterministic scene data from Blender.

• Counting: we sum unique object IDs (e.g., red_sphere_1, red_sphere_2) across all frames where those objects are unambiguously visible.
• Temporal ordering: we compare first‑appearance frame indices (e.g., if blue_cone_3 appears at frame 10 and green_cube_7 at frame 25, the answer is “before”).
• Comparisons: we use these exact counts.
• Left/right: we project 3D positions to camera coordinates for every frame—questions are generated only when ordering is consistent across all co‑visible frames.

This scales perfectly because ground truth comes from Blender’s scene graph, not human annotation. We will document this pipeline with examples in the appendix. We consider this a strength of REM, inspired by CLEVR’s usage of Blender scene graphs.

Spatial reasoning breadth.
REM tests four distinct aspects beyond left/right:

1. Counting across viewpoints probes object permanence and quantity representation.
2. Numerical comparison evaluates relative quantity representation.
3. Temporal ordering assesses trajectory‑based memory.
4. The “left‑of” / “right‑of” directional questions evaluate spatial relations.

These tasks collectively probe different facets of embodied spatial cognition identified in cognitive‑science literature. We also show in our “Rotation” experiment that models fundamentally fail at object permanence in a simplified scenario.

Human evaluation.
As noted in our general response, we agree and will add human baselines showing we expect near‑perfect performance given our controlled setup, validating that the tasks are solvable with proper spatial reasoning. Note that humans will need to navigate freely between images (like MLLMs accessing their context) for larger object counts.

Your feedback helps us clarify REM’s comprehensive and rigorous approach to evaluating spatial reasoning!