[Weakness 1]: Struggle with ambiguous human instructions, such as probabilistic preference and spatial compatibility.

A: Thank you for the suggestion. We have discussed limitations such as probabilistic preference and spatial compatibility in Appx. F, along with potential solutions including procedural synthesis of intent-aware data and co-training with intent-focused datasets. As our work mainly focuses on multi-step spatial referring with explicit instructions, handling ambiguous human intents is beyond the current scope and will be explored in future work.

[Question 1]: Segment each subtask within a video stream.

A: In fact, we do not divide the video stream into subtasks. Instead, we utilize the CA-1M dataset at frame level rather than clip level (see L215 and Appx. A.2). With (1) dense per-frame 2D/3D annotations and (2) complete camera and depth information, we construct fine-grained, spatially-aware template and reasoning QA pairs. Moreover, the videos in CA-1M were collected by individuals walking indoors without specific goals, making it difficult to segment the video stream into subtasks.

[Question 2]: Is RoboRefer integrated by incorporating its results as additional training data during the training process?

A: Yes, RoboRefer can serve as a perception tool to localize and place interactive objects (see L284). Integrated with open-loop control, it generates action data with spatially constrained instructions, enriching VLA training. In Rebuttal Table 1 below, fine-tuning OpenVLA with RoboRefer-collected data on Open6DOR-v2 enhances spatial ability, enabling the successful completion of previously unsolvable Level-3 tasks involving complex spatial relations.

Rebuttal Table 1: Results on Open6DOR-v2 benchmarks (simulation). Numbers represent success rates ( $\uparrow$ ).

[Question 3]: Is RoboRefer used as a planner tool to segment tasks into subtasks?

A: No, RoboRefer isn't used to decompose tasks in our experiments, but it can provide predicted points as visual prompts to guide VLA in generating actions if combined with the VLA model, improving success under spatially constrained instructions.

[Question 4]: Given the limited gains observed in subtask segmentation for small tasks due to poor prompt following by existing VLA models, how effective is this integration strategy?

A: We do not integrate RoboRefer as a planner with VLA due to marginal benefits in our experiments. Instead, RoboRefer’s predicted points can serve as effective visual guidance to enhance VLA’s instruction-following compared to directly following textual instructions, as shown in previous VLA works [1][2].

[Question 5]: How does the system determine when a subtask is completed by the VLA model to switch to the next task?

A: With 0.4s inference (2.5Hz; see L295), RoboRefer can efficiently serve as a checker to monitor sub-goal completion under spatial constraints (e.g., verifying if the hamburger is placed in front of the teddy bear) using spatial knowledge acquired during SFT if combined with the VLA model.

[Question 6]: How does the presence of depth noise impact the performance of the model, and what steps are being taken to mitigate this issue?

A: In real-world experiments, we use a relative depth estimation model, DepthAnything v2, to obtain relative depth as the model's depth input (see L236), which effectively reduces depth noise from a real camera.

[Question 7]: Have the authors conducted experiments to quantify the effect of depth noise on the model's accuracy and robustness?

A: We evaluate success rates under depth noise in real-world settings (see Rebuttal Table 2 below). Depth maps generated from the strong monocular relative depth estimation model (i.e., DepthAnything V2) offer the highest robustness and success. Despite depth noise from a real camera, RoboRefer maintains great performance by leveraging RGB priors due to mixed RGB and RGB-D training during SFT stage.

Rebuttal Table 2: Results on real-world experiments. Numbers represent success rates ( $\uparrow$ ).

References:

[1] Improving Vision-Language-Action Models via Chain-of-Affordance, ICCV'2025

[2] DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping, Arxiv'2025

We sincerely appreciate your recognition of our work and will do our best to incorporate your suggestions to further strengthen the paper.

[Suggestion]: Add a simple manipulation evaluation of OpenVLA (finetuned) on one manipulation benchmark to verify the effectiveness in manipulation.

A: Since the Open6DOR-v2 manipulation benchmark is built upon the LIBERO manipulation benchmark and uses the openvla/openvla-7b-finetuned-libero-spatial checkpoint to represent OpenVLA performance, we directly evaluate our finetuned OpenVLA (further trained with RoboRefer-generated data on this OpenVLA checkpoint) on the LIBERO-Spatial benchmark. The results are presented in Rebuttal Table 3. Fine-tuning OpenVLA with RoboRefer data enhances spatial ability, yielding higher average success rates and reduced performance variance compared to the original OpenVLA.

Rebuttal Table 3: Results on LIBERO-Spatial benchmark. Numbers represent success rates ( $\uparrow$ ).

We hope the above response and additional results meet your expectations, and we welcome further discussion if you're interested in more aspects.

Dear Reviewer 2QkV,

As the discussion phase progresses, we would like to confirm whether our response has addressed your concerns or met your expectations. If you have any remaining questions, we would be happy to discuss and address them. Thank you once again for your valuable feedback.

The Authors of Paper 72

[Question 2]: Is there any specific reason why NVILA is chosen as the backbone?

A: Please check Table 1 in the main paper. NVILA outperforms other open-source VLMs under comparable model scales, such as Qwen 2.5-VL (even 72B), in spatial understanding. Enhancing a strong baseline with our dataset and training strategy further validates their effectiveness. Notably, our dataset is model-agnostic and transferable to other backbones. Despite partial training on the RefSpatial dataset, Qwen2.5-VL-7B still shows notable improvements on spatial understanding benchmarks in the Rebuttal Table 1 below.

Rebuttal Table 1: Results on spatial understanding benchmarks. Numbers represent success rates ( $\uparrow$ ).

[Limitation 1]: The method and training pipeline are not entirely new and fall within expectations.

A: RoboRefer introduces a novel process-based RFT pipeline tailored for multi-step spatial referring. Unlike prior RFT works that rely solely on outcome-based rewards or struggle with process rewards in sequential decision-making tasks (see Appx. L1083–L1089), we identify a key insight: the metric sensitivity and order invariance of intermediate outputs in spatial referring (see Appx. L1090–L1100), which are uniquely suited to this task rather than general sequential decision-making tasks. Based on this, we design significantly new metric-sensitive process rewards (see L164) and special annotations of reasoning process in RefSpatial dataset, leading to good performance gains (see L326).

[Limitation 2]: By predicting a single target point, it mitigates 2D detection ambiguity under occlusion”. Why does it help with occlusion?

A: We apologize for the confusion. We clarify that masks generated from a single target point are more accurate than those from 2D boxes under occlusion. In cluttered scenes, occlusions often cause 2D boxes to include irrelevant objects, resulting in ambiguous visual prompts for SAM2 and imprecise masks. This degrades grasp accuracy and success rates in the open-loop AnyGrasp pipeline, a known limitation of 2D boxes also noted in prior work [1].

[Question 1 & Limitation 3]: The depth-to-3D mapping assumption is central but under-specified. How is 3D accuracy guaranteed by predicting a 2D point, especially in real-world robotic use where depth noise or partial views are common?

A: Thank you for the suggestion. We will clarify this assumption in the revision. While depth noise and partial observations are important real-world challenges, our setting follows prior work [1][2][3], which assumes that accurate depth-to-3D mapping is feasible given known camera intrinsics and extrinsics—sufficient for common manipulation and navigation tasks. Moreover, these challenges can be effectively mitigated via the following strategies:

• Depth noise can be mitigated by recent advances in monocular depth estimation [4], monocular geometry prediction [5], and stereo methods [6]. In cases of severe noise, we employ [6] in real-world settings to mitigate this issue.

• Partial views are mitigated in our method by leveraging pixel-level target points. Further improvement is possible by incorporating RoboRefer as a spatially-aware planner for active perception, which we leave for future work.

References:

[1] RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics, CoRL'2024

[2] RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete, CVPR'2025

[3] A0: An Affordance-Aware Hierarchical Model for General Robotic Manipulation, ICCV'2025

[4] UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler, CVPR'2024

[5] MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details, Arxiv'2025

[6] FoundationStereo: Zero-Shot Stereo Matching, CVPR'2025

We greatly appreciate your recognition of our work! We will include additional experiments to explain the reason why NVILA is chosen as the backbone, and further clarify previously unclear points, especially the depth-to-3D mapping assumption, in the final version. Thank you once again for your valuable time and feedback!

[Weakness 1]: Distinction from SpatialRGPT.

A: Our approach differs from SpatialRGPT in 4 key aspects:

• [Task Setting]: Our model addresses a more challenging spatial referring task, where take a spatially constrained textual instruction as input. It requires multi-step spatial reasoning with learned spatial knowledge to precisely localize the referred object as a 2D point step-by-step. In contrast, SpatialRGPT addresses a simpler VQA task and relies on externally provided region information as input for specific object referring.

• [Model Usage]: Unlike SpatialRGPT, which needs additional masks or detection tools to generate masks or 2D boxes as inputs for object reference and simplify referring tasks, our model can use textual descriptions for object referencing (see L283), which better aligns with real-world robotic applications.

• [Data Pipeline]: Our data pipeline adopts a more structured, progressive design than SpatialRGPT. It first uses 2D image data to teach core spatial concepts and general depth perception across diverse indoor and outdoor scenes. Next, accurate 3D data enhances fine-grained spatial understanding in indoor settings for robotics. Finally, simulation data introduces multi-step spatial referring with reasoning. This staged approach yields stronger spatial understanding and reasoning than SpatialRGPT, which relies solely on data generated from 2D images and lacks precise spatial perception for more complex spatial referring tasks.

• [Training Pipeline]: Our training pipeline includes process-based RFT after SFT to further improve multi-step reasoning and generalization for spatial referring tasks, whereas SpatialRGPT is trained with SFT only.

[Weakness 1]: Lack the details necessary to reproduce simulation data.

A: Please see Appx. A.3 for details on simulation scene generation, asset cleaning, QA generation, and reasoning process labeling. We will release all cleaned assets, scenes, and the full pipeline to help reproduce it.

[Weakness 2]: Not include a direct comparison with a VLM trained on SpatialRGPT data.

A: Following the reviewer's suggestion, we add SpatialRGPT’s performance on spatial understanding benchmarks in Rebuttal Table 1 below. RoboRefer outperforms SpatialRGPT, benefiting from our RefSpatial dataset, which is large-scale and offers more precise and diverse spatial relations (31 vs. 17). Moreover, SpatialRGPT’s reliance on masks or boxes oversimplifies object references, limiting performance.

Rebuttal Table 1: Comparison with SpatialRGPT on spatial understanding benchmarks. Numbers represent success rates ( $\uparrow$ ).

[Weakness 3 & Question 3]: Justification for RFT.

A: RFT brings two main benefits:

• [Generalization to Unseen Cases]: In Table 2 in the main paper (RefSpatial-Bench Unseen raw), which features novel combinations of spatial relations absent from RefSpatial dataset, our 2B-RFT model surpasses 2B-SFT by 9.1% in accuracy, showing the strong generalization enabled by the RFT stage (see L264).
• [Enhance Multi-Step Reasoning Ability]: In the Rebuttal Table 2 below, the RFT-based model consistently outperforms the SFT-based model across varying reasoning steps, especially at larger steps, showing the RFT stage's effectiveness in enhancing multi-step reasoning (see L258).

Rebuttal Table 2: Results on RefSpatial-Bench of different reasoning steps. Numbers represent success rates ( $\uparrow$ ).

[Question 1]: Is depth information included when evaluating RoboRefer

A: Yes. RoboRefer is evaluated with RGB-D inputs by default, with depth maps generated from RGB images via DepthAnything V2.

[Question 2]: For the 2D and 3D data, do the answers require explicit reasoning steps, or are they primarily spatial recognition tasks?

A: Yes, the 2D and 3D datasets are primarily spatial recognition tasks. They are not annotated with explicit reasoning steps (see L190) and are mainly used to learn diverse spatial concepts and improve spatial understanding.

[Suggestion 1]: A full-page diagram illustrating each pipeline would be helpful.

A: Thanks for your advice. We will polish the pipeline part to make it easier to understand.

[Suggestion 2]: Adding additional spacing between Tables 3 and 4.

A: Thank you for the suggestion. We will revise the paper accordingly..

Thank you for your interest in the details of how we construct QA templates in simulation. We provide a more detailed explanation of the "Locate Empty Space" QA type, including representative templates and examples (in simulation data or real-world evaluation).

This QA type includes four categories of templates, and herein we only present a subset of templates here; additional augmented versions follow the same variable structure. Four categories are detailed as follows:

Orientation and Distance-Based Prompts

• Templates:
  • Find a location in blank space located {distance} {direction} of {object}.
  • Locate a point within vacant space located {distance} {direction} of {object}.
• Examples:
  • Find a location in blank space located 0.2 meters to the left of the table. (See Appx. Fig.41, Row 1, Col 1)
  • Locate a point within vacant space located 0.05 meters to the front-facing side of the teddy bear. (See Appx. Fig.41, Row 2, Col 2)

Edge and Corner-Based Prompts

• Templates:
  • Select a point that is positioned in the free space at {position} on the desktop.
  • Select a point located in the vacant area at {position} of the desktop.
• Examples:
  • Select a point that is positioned in the free space at the right on the desktop. (See Appx. Fig.26, Corner & Edge)
  • Select a point located in the vacant area at the far-left corner of the desktop. (See Appx. Fig.42, Row 1, Col 3)

Between-Objects-Based Prompts

• Templates:
  • Please choose a spot located in the free space between {object1} and {object2}.
  • Please select a point in the vacant area between {object1} and {object2}.
• Examples:
  • Please choose a spot located in the free space between the red camera and the right camera. (See Appx. Fig.26, Free Space)
  • Please select a point in the vacant area between the cup and the car. (See Appx. Fig.42, Row 1, Col 1)

Two-Constraints-Based Prompts

• Template:
  • Please provide a point in the vacant area on the desktop that simultaneously satisfies the following two spatial conditions: 1. {condition1}; 2. {condition2}.
• Example:
  • Please provide a point in the vacant area on the desktop that simultaneously satisfies the following two spatial conditions: 1. On the left side of the car; 2. In front of the brown toy. (See Appx. Fig.42, Row 2, Col 3)

We will incorporate more details on QA template construction of simulation into the revised version. We hope the above response meets your expectations, and we welcome further discussion if you're interested in more aspects.

Thank you for acknowledging our work and rebuttal! We greatly appreciate your decision to upgrade the rating of our paper. We will incorporate all of your valuable suggestions into the revised version. Additionally, we will include the additional experiments in Rebuttal Tables 1 and 2 in the main text or appendix of our paper to further strengthen it!

[Weakness 1 & Question 2]: Benchmark scale & diversity

A: Thank you for the suggestion. Actually, spatial referring tasks demand precise mask annotations and complex textual instructions with spatial constraints, which are costly and challenging to obtain. Therefore, existing benchmarks are typically on the order of hundreds of images (e.g., Where2Place: 100; RoboSpatial: 122). Our RefSpatial-Bench (200 images) follows this common scale. Nevertheless, to further validate our approach, we expand it to RefSpatial-Bench-Large (2,000 images), covering diverse indoor and outdoor scenes, annotated by external annotators. Data sources and statistics are summarized in Rebuttal Table 1 below. Results in Rebuttal Table 3 below on this extended benchmark consistently support our claims: the RFT stage improves reasoning (see L258) and generalization (see L264), enabling generalized multi-step spatial referring with reasoning (see L8). We will release the extended RefSpatial-Bench publicly.

Rebuttal Table 1: Data sources and statistics of RefSpatial-Bench-Large. 'Old' refers to the original scene categories in RefSpatial-Bench, while 'New' indicates the newly introduced ones. Each part contains 600 images from the OpenImages test split, 200 from nuScenes, 100 from SUNRGB-D, and 100 from the original RefSpatial-Bench. Numbers indicate the number of benchmark samples for each scene category.

[Weakness 2]: Depth-fusion idea mirrors MM-Spatial and Spatial RGPT, novelty is incremental and not rigorously compared.

A:

• We have discussed the limitations of MM-Spatial and SpatialRGPT (see L43, L136) and highlighted key differences (see L315). Our method introduces an additional depth branch that enables effective co-training on limited RGB-only data with minimal performance drop in general VQA (see Table 4 in the main paper), while significantly reducing training costs (only requires 1/20 QA pairs of RefSpatial). In contrast, MM-Spatial and SpatialRGPT require over twice as much RGB-only data relative to spatial data to maintain general VQA performance. Moreover, SpatialRGPT relies on the region-level depth of given masks or bounding boxes, which oversimplifies the depth perception of the specific object compared to our full-image depth approach.

• We report SpatialRGPT’s results on spatial understanding benchmarks in the Rebuttal Table 2 below. We find that its RGB images and Depth maps as inputs rely on the same shared image encoder and imprecise data construction from 2D images, leading to inferior performance compared to RoboRefer. MM-Spatial is not open-sourced, precluding quantitative comparisons across these benchmarks.

Rebuttal Table 2: Comparison with SpatialRGPT on spatial understanding benchmarks. Numbers represent success rates ( $\uparrow$ ).

[Weakness 3]: RFT is a straightforward RLHF method (GRPO) without major modification.

A: Our RFT introduces novel metric-sensitive, process-based rewards tailored for spatial referring tasks (see L164) and is significantly different from vanilla GRPO, which relies solely on outcome-based rewards. This design is motivated by the nature of spatial referring tasks, whose intermediate outputs are not inherently sequential but instead require metric sensitivity and order invariance (see Appx. L1090–L1100)—key properties that guided our modification of vanilla GRPO into process-based RFT.

[Weakness 4 & Question 1]: Demonstrate RFT at scale.

A: Thank you for the suggestion.

• As noted in L231, we excluded 8B-RFT due to its high computational cost. Our TRL (Transformer Reinforcement Learning)-based training framework lacks multi-node support, and vLLM acceleration is not applicable since our model requires RGB-D input. Consequently, training 8B-RFT would take over 10 days on 8×A100 GPUs, which exceeds the rebuttal timeframe. We will include 8B-RFT results in the next version.

• Section 4.2 and Table 2 in the main paper have already provided detailed support for our claims about RFT. Results in Rebuttal Table 3 below on extended RefSpatial-Bench-Large remain consistent: the RFT stage improves reasoning (see L258) and generalization (see L264), enabling generalized multi-step spatial referring with reasoning (see L8). Recent work [1][2] has also confirmed that 2B-RFT is effective, and 7B/8B-RFT will also be effective as expected.

Rebuttal Table 3: Results on RefSpatial-Bench and RefSpatial-Bench-Large. L. and P. denote the benchmark’s Location and Placement parts. Numbers represent success rates ( $\uparrow$ ).

[Weakness 5]: Even after “speed-ups”, the model needs ~29 s per command on robot hardware, far from real-time.

A: We apologize for the confusion. The reported execution time in the Open6DOR v2 simulation benchmark covers the entire pick-and-place pipeline: (1) perception (e.g., object localization/placement prediction, mask prediction, object point cloud extraction), (2) motion planning (e.g., IK solving), and (3) robotic execution. RoboRefer is responsible only for object localization/placement of step (1) perception and requires just 0.4s per inference on RTX4090. A detailed runtime breakdown is provided in Rebuttal Table 4 below. Notably, we do not claim real-time performance. Section 4.4 merely demonstrates RoboRefer's robotic application in both simulated and real-world environments.

Rebuttal Table 4: Detailed runtime on Open6DOR-v2 benchmarks (simulation). Numbers represent time cost ( $\downarrow$ ).

References:

[1] Visual-RFT: Visual Reinforcement Fine-Tuning, ICCV'2025

[2] Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning, Arxiv'2025

[Weakness 4 & Question 1]: Demonstrate RFT at scale.

A: Fortunately, we successfully train the 8B-RFT model within the discussion period, to the extent permitted by our computational resources. Although training logs cannot be included due to multimedia constraints, we report its performance on RefSpatial-Bench and RefSpatial-Bench-Large in Rebuttal Table 5 below. 8B-RFT substantially outperforms 8B-SFT, consistent with trends observed in our 2B models and prior work [1][2], further validating the effectiveness of RFT at scale.

Rebuttal Table 5: Results on RefSpatial-Bench and RefSpatial-Bench-Large. L. and P. denote the benchmark’s Location and Placement parts. Numbers represent success rates ( $\uparrow$ ).

We hope the above response addresses your concerns, and we welcome further discussion if you have any additional questions or are interested in more aspects. Thank you for your valuable time and feedback!

References:

[1] Visual-RFT: Visual Reinforcement Fine-Tuning, ICCV'2025

[2] Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning, Arxiv'2025

Dear Reviewer ksPZ,

We are truly grateful for the time and effort you put into reviewing our work. To address your concerns regarding the Benchmark scale & diversity and Demonstrate RFT at scale, we have made our best effort to extend the benchmark to 2,000 images, covering diverse indoor and outdoor scenarios to enable a more comprehensive evaluation. Furthermore, we have included results of 8B-RFT, which more effectively demonstrate the validity and contribution of RFT. Meanwhile, we have also provided further clarification on parts of the paper that were previously unclear.

Therefore, we hope these responses have adequately addressed our comments, and we remain available for any further discussion. If our rebuttal has resolved your concerns, we would greatly appreciate your consideration of a higher rating, as it would be a strong encouragement for our work.

The Authors of Paper 72