SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models

Thank you for your positive and helpful review! We are glad you find this work valuable, scalable, and that the results show benefits on real-image benchmarks.

Yes, this pipeline can also be used to generate structured annotations (that benefit more detailed reasoning) or captions. We completely agree and will clarify that in this work, we are focusing on QAs for instruction tuning.

Thank you again for catching the typos. We will update the citations and the formatting issues. The batch size of 8 is after gradient accumulation—so, a batch of 1 with 8 gradient accumulation steps leading to an effective batch size of 8. We will clarify this.

Thank you for the positive and helpful review! We are glad you find this work crucial for many applications and that the results are impressive and reveal interesting insights.

Here are some answers to address your concerns:

1. A deeper discussion of failure cases or where the approach breaks down beyond camera movements and discussions on out-of-distribution scenarios.

Thank you for this great suggestion! Some failure cases are shown in the appendix in Figure 6. First, we see that abstract relationships like “looking away” are hard to capture in simulation, and hence models struggle with them (Appendix Figure 6, bottom row, 2nd from the left). Please see reply to 2SN7 point 2 for more details.

To provide a more quantitative analysis, we compute the top 10 question/scene words that are seen in incorrect answers vs correct answers and present them below.

First, we see that models continue to struggle with counting, as denoted by the number words in the “incorrect words” column. This is a well-known limitation in the community, and further research is needed to improve counting. Since our focus is on spatial reasoning, if we exclude counting words to analyze the top spatial failure categories (third column), we see that models fail on smaller objects in the scene, like donuts and cookies. Extending our pipeline with diverse assets and scenes is an exciting future work avenue.

We will add this discussion to the camera-ready paper.

2. Transfer to real-world spatial scenarios that are different from what the simulation can cover.

Great point. We do see that the spatial knowledge transfers to more complicated real-world scenes and scenarios, as demonstrated by results on VSI-Bench, which contain more complicated spatial scenarios in videos like “route planning” than the reasoning in SAT.

Following your suggestion, as another challenging evaluation, we evaluate our model on MME-RealWorld-Lite since it contains outdoor environments, and dynamic human and vehicle interactions, charts and graphs. We see improvements there too, notably across some spatial categories shown below. Please see the reply to 2SN7 point 2 for more details.

A great avenue for further study is on such complicated dynamic scenarios, since significant room for improvement remains. We will include these results and a discussion on this in the camera-ready

3. Transfer of improvements to embodied AI applications like navigation in the real world.

We tested on route-planning on VSI-Bench (see Table 4, column Rt. Plan), which deals with planning a route for navigating in real-world scenes. Our spatial training helps in that regard, showing promise that such training might benefit real-world navigation. As another preliminary analysis, we also saw an increase in the accuracy of next-action prediction given a scene in the SPOC ObjectNav benchmark (see Appendix A1 lines 666-667). However, we agree that a deeper analysis on real robots and complex scenarios, such as self-driving, is a great avenue for future work that uses our models as foundational planning models. We will highlight this discussion in the camera-ready.

4. Sometimes, having lower synthetic test set accuracies than the real test set.

We posit that this may be due to the domain gap between real images and simulated images. In a zero-shot setting, MLMs pretrained primarily on real images may perform worse on synthetic images. The synthetic test set is simply a diagnostic test set to see where improvements may be difficult/easy—i.e., the delta gains after tuning on SAT-train are reflective of real-world improvements (lines 270-275 in main paper). Please see reply to reviewer 2SN7, point 1 for more details.

This is an insightful point, and we will add this clarification to the camera-ready.

Thank you for the positive and helpful review! We are glad you highlighted the paper's direction in extending spatial reasoning to dynamic settings, the well-designed data generation pipeline, and the experiments that justify the effectiveness of the data, along with the insights from ablation studies.

Below, we answer your concerns:

1. The amount of real data and differences in evaluation values between synthetic and real

All our main evaluations are on real image benchmarks with over 9000 image-QA examples from SAT-Real (150 im-qa), BLINK (400), CVBench (2638), VSI-Bench (2900), VSR (3000). Thank you for the feedback and we will add this detail to the camera-ready paper.

SAT-Real and SAT-Synthetic may have different zero-shot metric values since the visual domain in real and synthetic is different. The synthetic test set is simply a diagnostic test set to see where improvements may be difficult for dynamic splits—i.e., the delta gains are reflective of real-world improvements. This is discussed in lines 270-275 in the main paper. For example, higher gain on SAT-Synthetic-EgoM is also reflected by a higher gain on BLINK-MV (related capability of detecting camera movements)—LLaVA-video has a higher gain than LLaVA-1.5. Same as GoalAim to CV-Bench-2Drel (since pointing to the goal requires reasoning about left/right relations). More gain on SAT-Synthetic avg also leads to more gain on SAT-Real (LLaVA vs LLaVA-Video).

Thank you for raising this insightful point! We will add this extended clarification to the camera-ready.

2. Generalization to a wider range of real-world scenarios, including outdoor environments or dynamic human interactions

Great point, our evaluation sets—CVBench, BLINK, and VSR—contain outdoor scenes and human relationships. We see that simulation training transfers well to these datasets, and qualitative results suggest (as shown in Appendix Figure 6) that the spatial knowledge transfers well to outdoor scenes as well.

Following your suggestion, we evaluate our model on MME-RealWorld-Lite as another challenging benchmark containing outdoor environments, and dynamic human and vehicle interactions, charts, and graphs. Overall, we see a 4% gain in performance compared to the baseline LLaVA-Video-7B when trained on SAT. Notably, if we select some spatial categories of interest, we see that understanding interactions improve by 7%, motion by 6.4%, and positions of objects by 8%. We will add all these results to the camera-ready.

Some failure cases can be seen in Figure 6 in appendix. Abstract relationships including humans like “looking away” are hard to capture in simulations and, hence, models struggle with them (Appendix Figure 6, bottom row, 2nd from the left). Similarly, natural language noise ambiguities are difficult. E.g., in Figure 6, bottom row right-most image, the GT is “yes” for is the “bicycle on the bus”. But, technically, it is not “on” but “in front of.”

3. Comparison to a base model fine-tuned on a similar amount of spatial visual questions to reflect the uniqueness and superority of SAT over general VQA generation pipelines

Yes, this is a great point. We have this in Table 6 and described in Section 4.2. In summary, we trained pseudo-annotated spatial questions on the GQA dataset following other works that use pseudo-annotation as a spatial VQA generation pipeline (row b). We also trained on manual spatial annotations in datasets like VSR and 2.5VRD (row c). We see that SAT training performs better overall (row d and e) (+10% on CVB and BLINK). Details are in lines 305 to 316. These general VQA generation pipelines mostly have static data since automatically inferring the effect of dynamic actions on images is difficult. However, SAT is simulated and hence we can easily predict how the image changes with actions/motions. Further, even static spatial relations in SAT are highly accurate because the data is simulated—with perfect GT annotations. That said, more sophisticated ways to pseudo-annotate spatial QAs by lifting scenes to 3D and leveraging videos might be a great avenue of future research. We will make this discussion clearer in the camera-ready.

Thank you for the positive and helpful review! We are glad you highlighted our dynamic questions as new to the community and found our results to be promising.

Below, we answer your concerns:

1. Dataset created from CVBench, BLINK, and VSI-Bench with pseudo-annotated Q&As.

The training set doesn’t include any examples from CVBench, BLINK, or VSI-Bench. These are used for testing only. The training set is our synthetic SAT dataset. (Please see section 4, lines 224, 225), We will clarify this detail in the camera-ready.

2. A more detailed description of the dataset and its comparison to other similar datasets.

Thank you for the feedback, and we will make the comparisons more explicit in the paper.

Dataset details are in Section 3. Briefly, the database is a Visual Question-Answering dataset. It contains both static (127K) and dynamic (48K) portions, for a total of 175K QAs. The dataset is generated synthetically as described in Figure 2 using 175K scenes composed of 1K object class assets in ProcTHOR. The details on the individual splits are in Section 3. Please let us know if any other detail is needed. We will also include a word cloud and object class histogram in the camera-ready.

Comparison to other datasets is in Table 1 and Related Works lines 91-105. The closest related datasets are SpatialVLM (not publicly available) and SpatialRGPT, both of which cover static relations and not dynamic scenes with multi-step spatial-reasoning pseudo-annotated on real images. Our focus on dynamic scenes and perfect 3D information from simulation distinguishes SAT from these pseudo-annotated real-image static datasets. Compared to other 3D point-cloud-based datasets, we have 2D images or multiple images since most state-of-the-art MLMs are tuned to take image inputs. Images are also cheaper for our use case, since recomputing 3D scans for dynamic scenes with moving objects is expensive.

We will clarify these details more explicitly in the camera-ready.

3. Model and algorithm novelty.

We saw that re-using the large-scale pretrained model without changing the architecture led to strong generalization across multiple benchmarks since it preserves the pre-trained general vision-language commonsense (Table 5, main paper). The scope of the paper and its novelty lie in the generation pipeline of the simulated dynamic spatial reasoning dataset. We present fine-tuning on the dataset as a strong baseline. We completely agree that research on better ways to use our rich interactions in simulation to improve beyond simple fine-tuning is a great avenue for future work.