SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning

Thanks for your careful and valuable comments. We will explain your concerns point by point.

Q1: The latent tokens were aligned into semantic space of LLM through training stage 1, thus VLM based on LLM as backbone can understand the meaning of token to a certain extent.

Q2: Limited by time and computational resource constraints, we provide supplementary experiments for the following general VQA tasks based on the SSR-3B model:

Our proposed SSR model outperform Qwen2.5-VL on multiple general benchmarks, this demonstrates that the pre-training knowledge of the Qwen2.5-VL will not be forgotten, but more common knowledge will be learned during training.

Q3: We evaluate the Qwen2.5-VL after fine-tuning on SSR-CoT dataset, overly protracted and convoluted textual intermediate reasoning chains not only increase the risk of erroneous conclusions but also impose prohibitive computational costs that undermine inference efficiency:

Q4: We thank the reviewer for this perceptive observation. We attribute SSR-3B's occasional superiority over SSR-7B on SpatialBench metrics primarily to two factors: (1) training both models on the identical dataset may inadequately support the larger model's capacity, and (2) SpatialBench's limited size increases susceptibility to statistical fluctuations. Finally, owing to computational constraints we could not perform multiple random restarts to obtain a reliable variance estimate. These factors jointly confound any firm conclusion about a occasional drop in SSR’s effectiveness as parameters scale. We will explicitly discuss these limitations in the final version and investigate them more thoroughly in future work.

Q5: The erroneous example depicted in Figure 1 will be amended, with the corrected version also provided in Figure 5.

Thanks for your careful and valuable comments. We will explain your concerns point by point.

W1: We will revise the paper per your suggestions and carefully review all expressions in the paper.

W2 i): Consistent with the prevailing paradigm in modern VLMs (e.g., LLaVA), we adopt CLIP as the RGB encoder. For depth modalities, we follow the SpatialBot design and employ SigLIP.

W2 ii): As shown in Table 3(a) of the original Meteor paper, the Mamba model has inherent computational efficiency based on linear complexity and strong long sequence modeling capability. With the same parameter count, BERT, XLNet, and GPT all require roughly twice the computation of Mamba, highlighting its clear advantage in throughput.

W2 iii): We align the latent tokens with the LLM rather than with the feature space of a particular VLM, because our objective is to situate the tokens within a universal semantic space, like CLIP, thereby maximizing cross-model generalizability. Consequently, after Stage 1 training, MIDI can be seamlessly transferred to diverse VLMs in a plug-and-play fashion:

As illustrated in the table below, when MIDI is combined with InternVL3-8B without any task-specific adaptation, it achieves markedly superior performance, underscoring both the generalization capacity and the broad applicability of our approach.

W2 iv): The majority of experiments in this paper employ models from the Qwen series, which have emerged as one of the most widely adopted open-source large language models in the research community. Their extensive usage, like InternVL series, offers a well-established and readily available foundation, thereby facilitating the seamless integration and evaluation of our proposed approach.

W3 a): Thanks for your suggestions, we supplement these contrastive experiments to provide a clearer understanding of the role of depth information:

We also provide the performance indicators of the qwen2.5-vl base model after adding depth information. Please refer to the response for reviewer 5EXr's W3, where the analysis description will also be provided.

W3 b): Both Visual Spatial Reasoning and What’s Up were constructed prior to the era of LLM; owing to time and resource constraints, we report zero-shot performance instead of task-specific fine-tuning on these datasets in the table below:

As for MME benchmark, our primary objective is to demonstrate that the proposed method directionally strengthens the VLM’s spatial-reasoning capabilities without degrading its performance on general vision–language tasks, please refer to the supplementary experiments we provided to Reviewer aoD6's Q2 for more results.

W3 c): As mentioned in lines 231–236, the LLM-Assistant evaluation only requires a straightforward textual comparison between the model’s response and the ground-truth answer, which is a easy operation for contemporary, highly-capable LLMs. To mitigate potential biases arising from employing models of the same family as judges, we re-assessed the SSRBench results with GPT-4o-mini, the outcomes are reported in the table below (For each result, the left column presents the original Qwen-based evaluation score, while the right column reports the corresponding GPT judgment):

The close agreement between the evaluation scores assigned by different LLM judges in the table indicates that simple answer-comparison tasks are largely immune to bias arising from the use of models within the same series.

Q1: RGB images alone provide insufficient spatial information for robust spatial understanding, as evidenced by recent studies (e.g., SpatialVLM, SpatialBot, SpatialRGPT). To address this limitation, existing work focuses on enriching VLMs with explicit spatial cues. While depth information is typically derived from RGB data, it offers a more effective spatial representation by distilling critical distance relationships. Conversely, raw RGB data may introduce ancillary visual noise that obscures spatial reasoning. This principle parallels the role of optical flow in video understanding field: though computationally extracted from RGB sequences, optical flow provides essential motion semantics that significantly enhances neural network performance, which demonstrating how processed geometric representations can surpass raw pixel data in conveying structural information.

Q2: We consider that what existing VLMs lack is the ability to perceive spatial information and reasoning. Spatial perception indicates identify the position relationship, real distance between different objects and so on, which are ambiguous to obtain from rgb image. The reasoning ability denotes the integrate and extract the portion of this spatial information that is beneficial to answering the question. The sentence means input depth information directly into VLMs without any adaptation like SpatialBot, we argue that this compromises the efficient use of depth information. We will correct the misquote of this sentence in the final version, thank you for your careful reading.

Q3: Thanks for your insightful suggestion. In the final version, we will expand the corresponding paragraph to elucidate why and how SSR converts raw depth maps into a structured latent representation within in semantic space. In addition, we will explicitly highlight the SSR’s superior generalization capacity and computational efficiency supported by supplementary experiments (Response to W2 iii and Response to Reviewer KarK’s Q2).

Q4 - Q6: We sincerely appreciate your meticulous suggestions; all typos will be corrected in the final version.

Q7: The two equations are indeed the same mathematical loss (causal, next-token cross-entropy), but they are applied to different model inputs and targets, and this difference is the key point. Equation 1 is computed on the reasoning chain. Input: $Z_{R}, Y_{R,<i}$, Target: $Y_{R,i}$, Goal: train the lightweight LLM to (i) perform implicit reasoning correctly and (ii) keep its latent representations aligned with the frozen LLM’s semantic space. Equation 2 is computed on the final answer tokens. Input: $Y_{A,<j}$, Target: $Y_{A,j}$, Goal: train the VLM to interpret the latent tokens emitted by the LLM and generate the correct textual answer. Thus, the loss are identical in form but operate on disjoint data streams and serve different training objectives in the two-stage pipeline.

Q8: Similar to the quality-assessment protocol shown in Table 2, we randomly draw 10 % of the cached samples and evaluate VQA accuracy both with and without their generated rationales. Rationales that improve accuracy are retained and incorporated into the final SSR-CoT dataset; those that degrade performance are discarded, and the all samples in cache are re-submitted for re-annotation.

Q9: Owing to the high computational costs inherent in training LLMs, hyper-parameters were configured based on a comprehensive synthesis of prior empirical experience and existing papers.

Q10: The MME score shown in Table 4 is indeed an aggregate (sum) of multiple sub-task accuracies. To verify that the targeted spatial-enhancement does not harm general capability, we evaluate the SSR model on more general visual-language tasks, please see our detailed response to Reviewer aoD6's Q2.

Q11: PAP indicates the MIDI module is directly applied to the pre-trained VLM in a plug-and-play manner after the training stage 1, without co-training. We will add a clearer explanation to the final version.

Q12: Figure 6 visualizes the cosine similarities between the latent tokens produced in two different paradigms: x-axis: latent tokens inserted inside the rationale, y-axis: latent tokens inserted immediately after the question and used to start the answer generation. Diagonal cells represent these two states of the same sample. High values on the diagonal indicate that the model has learned to map the rationale to the latent representation, confirming that it successfully distills the spatial knowledge embedded in the rationale. Low off-diagonal values indicate that the latent tokens remain sample-specific and do not collapse to a generic representation. For improved clarity in the final version, axis labels and enhanced descriptions will be added.

Q10: we report all tasks separately following LLaVA [1] to provide a comprehensive understanding of the performance metrics for Reviewer aoD6. We will present the average values of these tasks in Table 4 for enhanced readability, while detailed metrics for each individual task will be provided in the appendix for thorough analysis.

We sincerely appreciate your thorough review and constructive feedback, which has significantly enhanced the quality and completeness of our work. We will incorporate all your valuable suggestions into our paper in an appropriate manner.

[1] Haotian Liu, Chunyuan Li, Yuheng Li and Yong Jae Lee. "Improved Baselines with Visual Instruction Tuning". In Proc. of CVPR, pp. 26286–26296, 2024.

Thanks for your careful and valuable comments. We will explain your concerns point by point.

W1 & Q1: Due to conference policy, we are unable to provide full image cases. However, we have analyzed error cases and can describe key phenomena verbally. For counting tasks, most failures stem from ambiguous, unclear, or boundary-indistinct target objects (e.g., two overlapping giraffes being misidentified as one), indicating the requirement for more fine-grained, object-oriented visual understanding. Similarly, failure in other tasks arise from issues like small, unclear objects (e.g., a child’s shoe in the image being misrecognized as a sandal due to its pattern), highlighting the need for VLMs to receive and comprehend higher-resolution images.

W2 & Q2: We added inference time and memory usage statistics when supplementing general task experiments for Reviewer aoD6's Q2. Across all evaluated tasks, the computational consumption remains highly consistent; a representative result for POPE benchmark is summarised in the table below:

Moreover, we counted the relevant resource consumption data when fine-tuning Qwen2.5-VL with SSR-CoT for Reviewer JmB8's W3, which can better reflect the importance of Latent Reasoning method in CoT application:

The results demonstrate that, although SSR introduces a modest absolute latency per generated token, its latent-reasoning paradigm dramatically curtails the number of CoT tokens needed to reach a final response. Consequently, under the CoT-based evaluation framework, the overall end-to-end inference speed is substantially improved.

We appreciate your careful review and insightful suggestions. Accordingly, the aforementioned experimental results will be incorporated into Section 5.

Thanks for your careful and valuable comments. We will explain your concerns point by point.

W1: A key differentiator between our proposed SSR framework and existing approaches (SpatialVLM, SpatialBot, SpatialRGPT) lies in the portability of spatial understanding capabilities. Specifically, our MIDI module enables zero-shot transfer of spatial reasoning abilities to diverse VLMs without requiring retraining. This is achieved by translating spatial information into latent tokens within the semantic space, allowing seamless integration. Notably, as demonstrated in our response to Reviewer JmB8's W2 iii, when integrated with latest powerful VLMs like InternVL3, the MIDI module enables performance exceeding Qwen2.5-VL. In contrast, adapting the existing methods to newer, more capable VLMs necessitates complete re-adaptation and retraining, which is a process demanding significant computational resources.

W2: Thank you for suggesting these spatial understanding baselines. SpatialVLM has not released official code or checkpoints; we therefore evaluated the community-recommended third-party re-implementation version:

We will incorporate this results into Table 3 in the final version. Regarding SpatialRGPT, we note that it is an object-centric approach that expects one or more target objects to be supplied with bounding boxes or segmentation masks in every query. Designing a protocol that fairly aligns this requirement with our setting is non-trivial, and we were unable to complete such an adaptation within the current submission deadline. We will explore appropriate evaluation methods and update the evaluation results in time.

W3: Thanks for your suggestions, to comprehensively evaluate our SSR method, we conducted supplementary ablation studies focused on depth information. Specifically, we established two baselines: (1) directly integrating depth data into the backbone model, and (2) removing depth information from the plug-and-play setting. We then rigorously compared the performance of SSR against these baselines:

As shown in above table, directly integrating depth information into the backbone model impairs its understanding capability, leading to performance degradation. However, spatial-related CoT learning systematically enhances the model's spatial reasoning capacity. Subsequent incorporation of depth information further amplifies this spatial understanding, demonstrating the complementary effect of depth data when processed through the CoT framework.