SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation

We thank the reviewer for the constructive feedback and positive assessment of our contributions (Semantic Orientation, OrienText300K, PointSO, and the SoFar system). Below we address each concern point‑by‑point and provide additional results obtained during the rebuttal window. We also clarify measurement protocols and planned revisions to the paper.

R1. Robustness to RGB‑D Noise and Occlusion

The quality of depth data indeed impacts model performance, particularly in real-world scenarios. To address this challenge and enhance the robustness of our model to point cloud variations, we adopt the following strategies:

1. Domain randomization and perturbation are applied during training. Specifically, we inject Gaussian noise and perform data augmentations on the point clouds in OrienText300K to better simulate real-world depth variations.

2. To further denoise and smooth the point clouds generated by the D415 structured-light camera, we employ the point cloud smoothing and reconstruction algorithm from AsGrasp, which has been used in object grasping tasks to improve data quality.

3. We incorporate pointcloud voting strategy to reduce prediction variance and enhance the accuracy and robustness of the PointSO network.

Experiments. Section 4.2 reports a robustness study for PointSO using the corrupted validation split OrienText300K‑C (single‑view, Gaussian jitter, random SO(3) rotation, and “All” combined). The largest variant PointSO‑L retains 76.56% / 81.25% / 77.34% / 74.22% accuracy on Single‑View / Jitter / Rotate / All, respectively, indicating resilience to missing views and geometric noise. These controlled corruptions are designed to emulate partial observations and sensor artifacts.

R2. Task Diversity (Articulated Objects and Continuous Rotation)

We agree that broader coverage beyond pick‑and‑place is important. Our real‑world benchmark already includes 60 tasks across position‑only, orientation‑only, and full 6‑DoF tracks with diverse objects and embodiments. To specifically address the reviewer’s request, we added a targeted subset emphasizing articulated manipulation (drawers/doors) and continuous‑rotation orientation control. The table below summarizes the per‑task success across baselines and our methods (5 trials per task, success counted per trial).

New real-world experiments (articulated and continuous‑rotation tasks):

Takeaways. SoFar improves the total success rate to 51.7%, outperforming CoPa (+13.4 pts), ReKep‑Auto (+18.4 pts), and SoFar‑LLaVA (+8.4 pts). Gains are most pronounced on orientation‑intensive operations (e.g., handle alignment and facing direction), consistent with our core claim that semantic orientation complements spatial reasoning for 6‑DoF control.

R3. Latency and Suitability for Real‑Time Use

What our “Time Cost” measures.

In Table 1, Time Cost (s) refers to planning latency per trial—i.e., the end‑to‑end perception → orientation estimation → scene‑graph construction → VLM reasoning → grasp proposal → motion plan generation—excluding physical execution (e.g., controller rollout). This is consistent across all compared methods. We will make this definition explicit in the paper.

What contributes to the planning time.

• SoFar: Florence‑2/GroundedSAM for segmentation, PointSO for semantic orientation, and a VLM agent for spatial CoT reasoning (2 VLM calls).

• ReKep: GroundedSAM for segmentation, DINOv2 for keypoints/affordances, and a VLM agent (typically 3–4 calls).

• CoPa: multi‑round Set‑of‑Marks prompting, GroundedSAM for segmentation, VLM agent, and constraint generation (5 API calls), which dominates its runtime.

The API call is the most time-consuming part (2~3s per request), so CoPa exhibits the highest time cost.

Inference footprint (measured under the same resolution, batch=1, and single RTX 4080 16G GPU):

These measurements reflect perception and planning workloads (segmentation, orientation, VLM reasoning, grasp proposal, and motion planning). We will add a per‑stage latency breakdown in the appendix (e.g., segmentation/orientation/VLM/motion plan) to make the contributions transparent, and we will report the exact hardware/software configuration (GPU model and memory, CUDA/Driver, PyTorch).

I would like to thank the author for the rebuttal points, i will increase my rating.

We sincerely thank the reviewer for the constructive feedback. We address the two main concerns below and will incorporate the corresponding clarifications and additional results in the revised manuscript.

R1. CUDA memory and time costs (training and inference) for methods in Table 1

What our “Time Cost” measures.

In Table 1, Time Cost (s) refers to planning latency per trial—i.e., the end‑to‑end perception → orientation estimation → scene‑graph construction → VLM reasoning → grasp proposal → motion plan generation—excluding physical execution (e.g., controller rollout). This is consistent across all compared methods. We will make this definition explicit in the paper.

What contributes to the planning time.

• SoFar: Florence‑2/GroundedSAM for segmentation, PointSO for semantic orientation, and a VLM agent for spatial CoT reasoning (2 VLM calls).

• ReKep: GroundedSAM for segmentation, DINOv2 for keypoints/affordances, and a VLM agent (typically 3–4 calls).

• CoPa: multi‑round Set‑of‑Marks prompting, GroundedSAM for segmentation, VLM agent, and constraint generation (5 API calls), which dominates its runtime.

The API call is the most time-consuming part (2~3s per request), so CoPa exhibits the highest time cost.

Inference footprint (measured under the same resolution, batch=1, and single RTX 4080 16G GPU):

These measurements reflect perception and planning workloads (segmentation, orientation, VLM reasoning, grasp proposal, and motion planning). We will add a per‑stage latency breakdown in the appendix (e.g., segmentation/orientation/VLM/motion plan) to make the contributions transparent, and we will report the exact hardware/software configuration (GPU model and memory, CUDA/Driver, PyTorch).

R2. On the performance gap for drawer opening (Table 4)

Diagnosis.

Drawer opening in SimplerEnv primarily stresses execution‑side grasping rather than orientation reasoning: the handle is small, horizontally oriented, and close to contact surfaces. In our pipeline, grasp proposals are produced by a generic grasp generator (e.g., AnyGrasp or GraspNet); on small horizontal handles these generators often (i) fail to propose high‑quality, collision‑free grasps that also satisfy wrist approach constraints, or (ii) produce grasps that are feasible in isolation but degrade after trajectory optimization due to clearance limits. In contrast, SoFar’s semantic orientation module reliably predicts the target facing/rotation (as confirmed by high performance on orientation‑focused tracks), but the pipeline can still fail when grasp feasibility is the bottleneck. We will add qualitative failure cases and module‑level diagnostics in the appendix to make this explicit.

Affordance‑aware grasping improves drawer opening.

To reduce handle‑specific failures, we tested an affordance‑guided grasping module (GraspMolmo) in place of the default generator. The affordance prior biases grasp candidates to handle‑like affordance regions and compatible wrist approaches. On challenging real‑world tasks, we observe consistent improvements for fine-grained manipulation, including drawer opening:

We thank the reviewer for the thoughtful and constructive feedback, and for the positive assessment of our paper’s clarity, novelty, and empirical support. We respond to each question in turn and commit specific revisions accordingly.

Q1. 6‑DoF scene‑graph “edges” are unclear; please provide more detail and a JSON example.

Detailed Definition. In Sec. 3.1, we represent the environment with a 6‑DoF scene graph $𝐺=(𝑉, 𝐸)$. Each node encodes object phrase, instance ID, 3D centroid, bounding box, and a set of semantic orientations with their language descriptions. Each edge $𝑒_𝑖𝑗 ∈ 𝐸$ encodes inter‑object spatial relations, specifically the relative translation and size ratio between objects $𝑜_i$ and $𝑜_𝑗$. We will clarify this in the main text and highlight that edges serve downstream spatial reasoning and planning.

To address your request for a concrete example, we will add to the appendix a complete JSON‑formatted scene graph corresponding to Fig. 5, including nodes and edges with the above attributes (object phrases/IDs, centroids, bounding boxes, semantic orientations and their language labels; edges with relative translation and size ratio). We will also cross‑reference Fig. 5 where the “JSON‑format Scene Graph” is introduced.

{
  "nodes": [
    {
      "id": "obj_1",
      "phrase": "flashlight",
      "centroid": [0.32, 0.11, 0.85],
      "bbox_xyz": [0.28, 0.08, 0.78, 0.36, 0.14, 0.92],
      "semantic_orientations": [
        {"illuminate": [0.93, 0.00, 0.36]}
      ]
    },
    {
      "id": "obj_2",
      "phrase": "Loopy",
      "centroid": [0.80, 0.10, 0.85],
      "bbox_xyz": [0.74, 0.06, 0.78, 0.86, 0.14, 0.92],
      "semantic_orientations": []
    }
  ],
  "edges": [
    {
      "source": "obj_1",
      "target": "obj_2",
      "relation": "points_to",
      "delta_position": [0.48, -0.01, 0.00],
      "delta_rotation_quat_wxyz": [0.98, 0.00, 0.20, 0.00]
    }
  ],
  "units": {"length": "m", "rotation": "quat(wxyz)", "frame": "world"}
}

Q2. Without an affordance map, how does SO / SOFAR decide which part to grasp (e.g., the handle of a knife rather than the blade)?

Following CoPa, we leverage vision-language models and vision foundation models to identify the target grasping part. After parsing the desired object or part name using a VLM, we employ an open-world grounding module (e.g., Florence-2 or GroundedSAM) to localize the corresponding region and point cloud. Subsequently, the use of SoM, coarse-to-fine grounding, or affordance-based grasping models can further enhance manipulation performance.

Since our primary focus lies in the integration of Semantic Orientation with various robotic tasks, we did not place much emphasis on precise position localization. This is because part-level grounding is sufficient to achieve satisfactory performance for most tasks. Furthermore, we explore using GraspMolmo as an alternative affordance generator to replace the Florence-2 + SAM pipeline. This substitution is evaluated across a series of challenging tasks, where it led to performance improvements:

Q3. Coarse six directions limit fine‑grained manipulation (e.g., “pour tea into a cup,” align kettle spout).

We acknowledge that the current data construction pipeline is limited by the use of basic canonical orientations and four oblique views. However, the definition of Semantic Orientation itself is not restricted to these base directions, and the predictions produced by PointSO are continuous 3D vectors rather than discrete labels. We anticipate that with more efficient and accurate data construction methods—such as using more views as labels, leveraging world models, using robotic data, or human annotations—it will be possible to handle more complex object orientation estimation tasks.

Compared to keypoint‑only methods such as ReKep. Keypoints alone can ensure spatial proximity but may fail to capture orientation. ReKep relies solely on affordance keypoints generated by DINOv2 for planning and tracking, which can result in the spout of a teapot being oriented in irrelevant directions, such as to the left or right. Building upon ReKep’s keypoint constraints, semantic orientation introduces additional constraints that explicitly account for object orientation, so it will be better than the baseline.

We thank the reviewer for the careful reading and constructive feedback. Below, we respond to the main concerns regarding motivation, novelty, fairness of evaluation, and missing details, and we clarify the questions (Q1–Q4).

W1 -- Does semantic orientation truly benefit manipulation?

Motivation. In real-world robotic tasks, many instructions require a nuanced understanding of orientation, particularly in manipulations involving object-object interactions and rich semantic context—for example, inserting a pen into a holder, cutting bread with a knife, or using a flashlight to illuminate. Therefore, SoFar aims to enhance object manipulation and navigation performance by improving the model’s comprehension of both semantics and orientation.

Evidence. Empirically, SoFar achieves SOTA performance on the Open6DOR object rearrangement benchmark—substantially outperforming VLM/VLA baselines—while also showing SOTA performance on SIMPLER (both Google Robot and Widow‑X settings). In SIMPLER, SoFar’s zero-shot performance surpasses methods trained on a large set of robot trajectories. These results indicate that explicit, language-grounded semantic orientation improves not only spatial reasoning but also downstream manipulation.

W2 -- Novelty of semantic orientation vs. prior art ([A] Po3D‑VQA; [B] ImageNet3D; [C] Orient Anything)

Our Semantic Orientation maps free‑text descriptions to precise 3D unit vectors in the object’s point‑cloud frame, which planners and controllers can consume directly. It is designed for manipulation because VLA systems must translate ambiguous natural language into exact metric actions.

Key distinctions between semantic orientation and prior work:

Po3D‑VQA ([A]): This work improves VLM's understanding of pose via VQA (e.g., “Which direction is the bus facing?—Left”), i.e., text&vision-in → text-out. Such question-answerings are not sufficient to execute robotic manipulation. In contrast, SoFar is text&vision-in → continuous 3D orientation-out, producing vectors that integrate seamlessly with planning and control. Moreover, [A] only trains with 21 categories of objects using hand-crafted templates, which can't be seen as open-vocabulary generalization.

ImageNet3D & OrientAnything ([B], [C]): These works focus on canonical axes estimation from images. In contrast, our semantic orientation supports open‑world, free‑text queries spanning relative directions, part‑level targets, and inter‑object interactions (e.g., “USB's plug‑in direction”, “cutting direction of the knife”, “aim the flashlight at Loopy”). This functional, instruction‑conditioned grounding is what enables robotic action generation from language.

Moreover, predicting the semantic meaning of axes from 6D orientation using a language model is non-trivial, as the language model cannot access multi-view information of the object during inference. Inferring the correspondence between axes and semantics (e.g., X-axis <--> pick up, Y-axis <--> cutting) from a single, potentially occluded image is inherently challenging. In contrast, PointSO is trained on multi-view data and thus achieves superior performance, as further evidenced by the results in W3.

More importantly, the definition of semantic orientation is not limited to the six canonical directions. The values predicted by PointSO are continuous 3D vectors. During training, we used six canonical views and four oblique views for supervision. Incorporating more views as labels will further enhance directional fitting capability. In contrast, injecting semantics into standard 6D orientation axes will reduce the problem to constrained six-way classification. Semantic orientation is not only end-to-end trainable but also exhibits a higher grain and better scalability.

The limitations of prior works motivate the introduction of Semantic Orientation:

1. Takes open-world free text as input (e.g., “plug-in direction of a USB”, “cutting direction of a knife”, “point the flashlight at Loopy”), thus jointly generalizing over object categories, parts, and interaction verbs.
2. Outputs continuous, metric 3D unit vectors in the object’s point-cloud frame, which can be directly consumed by robotic planners and controllers.
3. Is template-free and reference-frame-free, enabling zero-shot generalization without category-specific CAD templates or canonical pose assumptions.

W3 -- Is semantic orientation necessarily better than standard 6D orientation?

Standard 6D orientation doesn't encode task semantics (e.g., “cut”, “plug-in”, “pour out”) and typically relies on category/template priors that harm zero-shot generalization. In contrast, semantic orientation is (i) instruction-conditioned & function-aware and (ii) template/frame free. This is crucial when the same object must be manipulated along different functionally meaningful directions (e.g., hold the handle of the mug, pour out the water).

Furthermore, we conduct ablation studies to compare with standard 6-DoF pose estimation. Specifically, we employ FoundationPose & OrientAnything to predict the pose of the manipulated object. Due to the absence of CAD models in FoundationPose, we follow the original protocol by capturing 16 images to construct an implicit representation. After obtaining the predicted 6-DoF pose, we mark it in the image and utilize an additional VLM (GPT-4o) to select the desired orientation from the set of standard 6D directions. The real-world manipulation results are as follows:

It can be seen that our proposed semantic orientation achieves superior performance with fewer VLM calls (less time costs). Furthermore, we would like to clarify that SoFar is an integrated system. In addition to semantic orientation, our contributions also include the OrienText300K dataset, the Open6DOR V2 benchmark, and a broad suite of downstream tasks spanning cross-embodiment, cross-view, and cross-task generalization.

Besides, regarding your mention of finetuning standard 6D orientation models on our language-orientation paired dataset, I agree that this approach would be effective, as it essentially aligns 6D orientation with semantics—representing a simplified and discretized form of semantic orientation.

W4 -- Spatial VQA seems unfair; SoFar is the only model finetuned with extra data and also uses extra experts

First, we divide VQA baselines into “General VLMs” and “VLMs with Spatial Awareness” following their own papers’ claims and training regimes. Many of these already incorporate extra data and off-the-shelf modules (e.g., SpatialBot uses depth API); we report them as-is to respect each method’s intended recipe. We also compare our method with proprietary models such as GPT-4o, which benefit from larger model capacities and broader pretraining corpora.

Besides, we want to clarify that VQA is not the main contribution of our work. SoFar is a robotic manipulation system, and the VQA experiments are conducted to validate the effectiveness of semantic orientation. Our goal is to demonstrate that incorporating semantic orientation indeed enhances the spatial understanding capabilities of VLMs, which our results confirm.

Answers to Questions

Q1. Is PointSO trained on the same set of “semantic orientations” as used downstream? Does it generalize to unseen ones?

All objects used in our real-world experiments were not deliberately selected or purchased from the Objaverse dataset, aligning with the settings of novel objects and in-the-wild generalization. Our OrienText300K dataset contains over 350K diverse objects, with category coverage that matches and exceeds that of LVIS, thus supporting strong category-level generalization. Furthermore, for the evaluations reported in Tables 2 and 3, we ensured fair and accurate testing by separating the training and test sets within the OrienText300K dataset.

Q2. How is the “time cost” in Table 6 estimated?

The time costs reported in Table 1 and Figure 7 reflect the complete planning duration. For the SoFar model, this includes the processing time of components such as SAM, PointSO, and the VLM. For ReKep, the time costs encompass SAM, DINOv2, and the VLM. For CoPa, they include multiple rounds of SoM, VLM calls, and constraint generation. SoFar demonstrates a clear advantage in time efficiency compared to baselines such as CoPa and ReKep. CoPa exhibits the highest time cost, primarily due to up to 5 times VLM API calls, whereas SoFar requires only 2 times. The table below presents the detailed inference breakdown for each method, using a single RTX 4080 16G GPU:

Q3. Frequency / FPS & running time of modules in Fig. 5?

All perception & reasoning modules (SAM, PointSO, VLM) are invoked once per planning step. The subsequent low-level execution runs at the robot’s native control frequency (e.g., Franka 1 kHz, UR5e 500 Hz, Unitree Go1 1 kHz), using the planned 6‑DoF trajectory. We will include a table in the appendix with per-module latency and overall FPS for clarity. In brief, the two VLM calls take approximately 5 seconds, while the SAM and PointSO modules require about 1 second, and the Motion Planning module takes around 2 seconds.

Q4. What VLM is used in Table 1?

SoFar uses GPT‑4o as the agent VLM; SoFar‑LLaVA uses LLaVA-7B as the agent VLM.