Thanks for your insight review and positive feedback! We have incorporated discussion on SGRv2 in our updated paper (Appendix C) following your suggestion.

Similarity to SGRv2: I noticed that the waypoint policy in this paper is quite similar to the one proposed in SGRv2 ...

Thanks for pointing out missing related work. SGRv2 is a very recent work from CoRL2024 and we were unaware of it at the time of writing. We can definitely extend related work to cover it. Here we briefly list out the differences between SGRv2 and our work, as well as some thoughts on how the two works would compare in terms of performance:

• SGRv2 operates on a mixture of point cloud and image data. It predicts actions as per-point offset for all points, and uses a weighted average to get the final action output.
• Although both works represent the actions as offsets w.r.t. points, Sphinx utilizes “salient point” (learned from human labels) as the anchor of the action, and casts part of the action prediction, a typical regression only problem, to classification + regression to make it more precise and easy-to-learn. The ablation in the right panel of Figure 4 shows that the classification loss is helpful, that even just having it as an auxiliary task significantly improves performance of the “vanilla waypoint”.
• Due to the existence of salient points, Sphinx applies the offset loss only to the proximal points of the “demonstrator-specified salient point”, i.e. the red points in Figure 3 and Eq (3). This makes it easier for the neural network to learn since it does not need to allocate capacity to predict offset for points far from the salient points. It also allows the network to have different position targets for different parts of the points, e.g. the offsets near the top drawer correspond to actions for opening the top drawer while the offsets near the bottom drawer correspond to actions for opening the bottom drawer. In early development, we found applying the offset loss to all points noticeably hurt the performance as it lowers the precision for the most critical points that are used during inference.
• On a minor note, Sphinx uses a GPT-2 style Transformer while SGRv2 uses PointNeXt as backbone. This is an implementation detail and both methods should work well with either architecture choices but we think it is interesting to see a generic architecture not designed for point cloud to perform quite well in these tasks.

We notice that SGRv2 also evaluates on the Square task from the MimicGen, which is the same as the one in Robomimic. With 50 demonstrations, SGRv2 only achieves a 2.8% success rate, which is significantly lower than Sphinx (86.5%) and the popular diffusion policy (44%) baseline. We have trained the diffusion policy using 50 demonstrations from the original Robomimic dataset and it achieves comparable performance (45%) as the one trained on our data, indicating that the data quality is similar. Therefore, from the published result it seems that there would be a big gap between SGRv2 and Sphinx on Square.

Insufficient Visual Generalization Experiments

Although we demonstrate SPHINX across perturbed viewpoints and in the presence of distractor objects, we agree that we have yet to explore more drastic visual variations. One promising avenue for future work is leveraging recent foundation models like MOLMO [1] which have demonstrated impressive performance for zero-shot semantic keypoint detection (i.e. “point to the ___” in an image, as shown here). Instead of directly predicting salient points and offsets jointly, one possible extension is to first infer salient points via models like MOLMO, and condition the waypoint actions on these. We could still collect data for and train SPHINX largely in the same way. However, this would open up SPHINX’s visual generalization capabilities at test-time, since these foundation models could generalize better for salient point detection on completely unseen object instances, with which we can condition the policy.

[1] Deitke, Matt, et al. "Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models." arXiv preprint arXiv:2409.17146 (2024).

Limited Exploration of Dynamic Tasks

We acknowledge that in our current work SPHINX uses a linear controller for waypoint reaching, limiting its capacity for dynamic tasks. To relax this, future work could predict spline-based trajectories to reach a waypoint or replace SPHINX’s linear controller with a learned [1] or engineered motion planner for dynamic, collision-aware reaching. SPHINX’s dense policy can in theory be able to handle dynamic motions as long as the hardware permits, so the truly dynamic parts of a task (i.e. wiping, sweeping, higher-velocity actions) can also be learned by the dense policy in the future.

Re. Novelty:

The primary weakness of this paper lies in its lack of novelty. Guiding fine-grained manipulation policies with coarse waypoints is not a new approach. Similarly, using saliency prediction to assist manipulation has been widely addressed in prior works, often categorized as affordance-based approach.

We appreciate your feedback, though we don’t agree on the argument for lack of novelty. Our novelty primarily lies in these two aspects:

Waypoint Action Decomposition

1. We decompose waypoint action prediction (often a global regression task which can be difficult to learn precisely for far-away targets) into a salient point classification (softmax over points in the point cloud) and offset relative to the salient point (an easier, more local regression task. In comparison:

• PerAct[1] omits offsets and instead casts action prediction as a classification problem over voxels, and the precision of prediction depends on the granularity of the voxel grid. Increasing this granularity not only significantly increases the training cost, but also makes the prediction task much harder as the number of action candidates grows cubically in the number of voxels.
• Act3D[2] performs coarse-to-fine scoring for “ghost points”, which addresses the granularity issue of PerAct, but still lacks any kind of explicit intermediate representation such as salient points.
• Affordance learning can refer to classifying the nature of interaction for certain object points (e.g., a tool handle is "graspable,” a button is "pressable") [3], or more broad action candidates or success possibilities associated with objects [4]. Salient points in SPHINX can be thought of as a form of per-point affordance, but must be combined with offsets to specify how the end-effector should interact with a given point. Critically, we demonstrate that the per-point affordances we learn in SPHINX allow the policy to focus on task-relevant features and avoid paying attention to arbitrary objects, leading to robust execution in the presence of visual distractors.

Hybrid Input AND Output Representations

2. To our knowledge, SPHINX is the first IL agent to be able to interchange both its input/output representations on the fly depending on the task phase, which is crucial for success in precise, long-horizon tasks.

• Prior works such as Hydra have proposed using a hybrid action space of waypoint/dense actions for IL, but only consider a fixed input space of images. SPHINX performs 5.4 times better than Hydra on complex real-world tasks, demonstrating the benefits of using point cloud observations in combination with our novel waypoint action representation
• Several prior works propose waypoint-only policies for manipulation with 3D representations as input; although such representations provide rich spatial information, they do not capture object details at a sufficient resolution necessary for accomplishing any one of the real world tasks we consider in this paper. For example, in the train track task (better viewed through the video from website https://sphinx-il.github.io/), the policy needs to identify the “jigsaw puzzle”-like structure on the bridge and track to properly connect them. Point clouds from consumer-grade depth cameras do not provide a sufficiently high resolution to capture such information and there is also the occlusion issue that prevents the 3rd person camera from seeing the gripper-object interaction clearly. Any policy solely based on point clouds will struggle with tasks that require close-up observation, motivating SPHINX’s dense policy.

Given these clarifications, we respectfully ask the reviewer to reevaluate our novelty contribution and kindly point out missing prior works if either one of the contributions mentioned above has been well-studied previously. We also plan to add these discussions to the paper to better emphasize our contribution.

[1] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. "Perceiver-actor: A multi-task transformer for robotic manipulation." Conference on Robot Learning. PMLR, 2023.

[2] Gervet, Theophile, et al. "Act3d: Infinite resolution action detection transformer for robotic manipulation." arXiv preprint arXiv:2306.17817 (2023).

[3] Borja-Diaz, Jessica, et al. "Affordance learning from play for sample-efficient policy learning." 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022.

[4] Ahn, Michael, et al. "Do as i can, not as i say: Grounding language in robotic affordances." arXiv preprint arXiv:2204.01691 (2022).

I appreciate your clarifications and the additional experiments. I have raised my scores as the following major concerns have been addressed:

1. Benefit of hybrid approach over existing keypose-based approaches. The authors conducted several additional experiments including a comparison with PerAct and an ablation study on a vanilla waypoint model without saliency prediction. These experiments demonstrate the benefits of the saliency-based approach. Furthermore, the authors added an extended related works section in Appendix C. This clarifies the motivation behind SPHINX over existing methods. Lastly, the authors made a fair argument that classification is easier than regression, postulating that SPHINX's generalization capability might not be solely attributable to the 3D representation but also to saliency modeling.

2. Lack of Novelty. I agree with the authors that switching between fine-grained image-based policy and coarse-grained keypose/waypoint models in a real-time closed loop is underexplored. I had underestimated this novelty. It provides a more natural solution to precision tasks than the multiresolution modeling found in recent keypose-based approaches.

3. Limitation of saliency-based waypoint approach. This concern was mainly due to my misunderstanding that the waypoints were 3D points rather than 6-DoF (or 6+action_dim) poses. The authors have promised to address this potential confusion by using the term "keyframes" instead of "waypoints":

... though we used the term “waypoint” for consistency with Hydra, we can instead refer to these as “keyframes,” etc. in a camera-ready version to convey they capture the 6D pose.

I would, however, suggest using "7-DoF waypoint" or "(6+1)-DoF waypoint," as "keyframe" could also be confused as a video frame. That said, the decision is entirely up to the authors.

[Remaining Concerns]: I am still not fully convinced of the necessity of explicit saliency modeling. The authors claim that explicitly modeling saliency in the (6+1)-DoF waypoint prediction provides three benefits over existing keypose prediction/generation methods:

1. Classification is easier than regression.
2. Saliency points encourage recognition of task-relevant local points, improving robustness to visual distractors.
3. Saliency prediction serves as an auxiliary learning objective.

While I agree with these claims, I am still not sure whether explicit saliency modeling is truly necessary. The authors benchmarked PerAct and attributed its low success rate to its low spatial resolution. However, this has already been addressed in more recent works like RVT-2, Act3D, and 3D Diffuser-Actor, which leverages multiresolution approaches and/or image-3D hybrid modeling. It remains unclear whether saliency modeling would still provide significant benefits over these more advanced models in which spatial precision is not a major concern.

In addition, unlike PerAct, these recent models are more sensitive to locality. This can also "encourage the policy to recognize task-relevant features of the point cloud,", achieving similar effects to what explicit saliency modeling aims to accomplish but in a much flexible way. Given that these SOTA keypose models are far more competitive than PerAct, there remains a strong possibility that they could outperform the simple waypoint transformer in SPHINX, even without explicit saliency modeling.

[Conclusion]: I am more inclined to accept the paper after the rebuttal, as several important concerns have been addressed by the authors. Accordingly, I am revising my recommendation. That said, the concept of switching between waypoint policy and dense policy seems orthogonal to saliency modeling, and I am uncertain about the necessity of the latter when more advanced keypose methods are used for waypoint inference. The authors conducted additional experiments with PerAct, but PerAct is significantly behind recent SOTA methods.

HYDRA Performance Insights: It would be interesting to understand why HYDRA, despite using a similar waypoint and dense manipulation modes, performs way less effectively. Does it struggle more in the waypoint phase or the dense manipulation phase?

As discussed above, the waypoint head in Hydra is similar to the vanilla waypoint in ablation, both trained to directly regress a target pose without considering the salient point. Vanilla waypoint struggles in the Square task, suggesting that just doing the waypoint-dense decomposition alone is not helpful compared to modern imitation algorithms like diffusion policy. This highlights the value of our novel action representation (salient point classification + offset regression). Hydra struggles a lot in the waypoint phase as its waypoint prediction is much worse than Sphinx. The error of Hydra’s waypoint predictions is often a few centimeters while the error of Sphinx’s waypoint predictions is less than 1cm.

Real-World Experiment Details: Could the authors share more on how they measure success rates? Do you count broken down points as OpenVLA, or does it only reflect the final task outcome? A failure analysis would also be insightful.

We measure success rate as follows:, if the robot successfully finishes all manipulation steps for a task then it is 1, otherwise it is 0. We do not follow OpenVLA’s way of computing scores via partial progress. We have video recordings for all the evaluations, best viewed at https://sphinx-il.github.io/ (you can click on any bar in the “Hybrid Task Results” section to view rollouts for any policy). Different methods fail for all sorts of reasons, failing to pick up the object, cannot navigate to the correct location etc.

Point Cloud Format: What specific format of point cloud data does the waypoint transformer use? Are both RGB and XYZ coordinates involved, or only XYZ? As I am curious how the salient point prediction can distinguish the pink and yellow cup.

As mentioned in the paper, we use colorized point clouds, i.e. each point represented by x,y,z,r,g,b, a tuple of 6 floating points.

Camera Viewpoint Calibration In Novel Viewpoint: Were novel camera viewpoints recalibrated during inference? Additionally, are the training and inference data in the same coordinate system, like the robot’s base?

Yes, the novel camera viewpoints are calibrated. Despite calibration, the novel camera view leads to quite different images (Figure 7, middle panel) as the robot body is mostly out of the view for the training camera while occupying half of the image (and thus accounting for a significant portion of the point cloud input) in the novel inference camera. Yes the coordination system is the same throughout our paper.

Waypoint Transformer: It’s interesting that the waypoint transformer model can effectively handle unordered points with relatively small data. Could you provide more details on the network structure and training? What's more, were any data augmentations applied?

Yes it is indeed interesting that a GPT2 style transformer with no point cloud specific structure can handle points. During early development we also found it worked better than PointNet++. The network structure is similar to GPT2, using the nanoGPT (https://github.com/karpathy/nanoGPT) as reference implementation. As described in Appendix B1, the waypoint transformer has 6 transformer blocks, has no positional embedding for the points, and uses dropout=0.1 for all blocks. We train it with a cosine learning rate schedule and return the exponential running average as the final policy.

As described in the last paragraph of Section 4.2 (right above Section 4.3 on page 6), we use “temporal data augmentation”. At time step t, to reach a waypoint (wt+k), we use a controller that moves the arm over the interval [t, t+1, …, t+k]. Observations at intermediate steps share the same waypoint label (wt+k), allowing us to use these steps for data augmentation. In all experiments, we use the first 20% of interpolated steps (t, …, t + 0.2*k) as additional training data for the waypoint policy. For example, in the Square task, this technique augments 300 demonstrations into a dataset of approximately 1,800 data points. On top of the “temporal data augmentation”, we apply random translations to the input and target, a typical data augmentation technique for point cloud based neural networks. We have since added these clarifications to Appendix B (Implementation Details of SPHINX).

Thanks for your insightful feedback! We are happy to add more details and clarify some misunderstandings. We are trying to address all of your concerns below but changed the orders of some questions to make the explanation more coherent.

Handling Distant Salient Points: When a salient point isn’t near a physical point—like opening an oven or a door—how does the method handle this? If there’s a substantial distance to any nearby object, does the model has to use the mode of dense manipulation?

It seems that there is a misunderstanding here which we will address in the revised version through a clearer explanation of salient points versus waypoints. In our design, the salient point must be a physical point that exists in the point cloud. For example, when opening an oven/door, the demonstrator clicks on the handle to specify the salient point. The eventual effector pose can be anywhere, like 3 cm or 10 cm away from the salient point along a certain direction. We record the waypoint action as a combination of salient points (a point in the point cloud that physically exists) and offset w.r.t. that point. It is a core part of Sphinx that the salient points are physical points in the point cloud and that is what allows us to cast part of the action prediction problem into classification over a set of points. Classification is generally more robust as it encourages competition among the candidates, and it also makes residual offset prediction simpler. It is one of the main insights that motivates the Sphinx method.

Reliance on Salient Points

It is true that Sphinx relies on the annotated salient point for training. But the salient point annotation is organically integrated into the data collection process (better illustrated through videos in the “SPHINX Data Collection Interface” section on the website https://sphinx-il.github.io/). With our GUI, it is not necessarily an overhead to add the salient point. Instead, it may simplify the data collection process compared to the very common “dense” teleoperation interfaces like a spacemouse or VR controller that require constant attention to control the robot precisely. For example, opening a drawer is just 3 mouse clicks, each followed by some click-and-drag to adjust the offset and angle, which is considerably simpler than commanding the robot using a VR controller.

Thank you for your thoughtful review as it truly helps up to improve the paper. Hopefully we can address all of your concerns.

The novelty of the paper could be further clarified.

Our novelty primarily lies in these two aspects:

1. The novelty of our waypoint policy lies in decomposing action prediction into salient point classification (softmax over points in the point cloud) and offset regression (predicting a local offset relative to the salient point).

• The salient point classification encourages the policy to focus on task-relevant objects and avoid paying attention to visual distractors.

• Classifying salient points in a point cloud makes the learning objective easier. Classification is generally an easier task for neural networks to learn compared to regression. Additionally, we now only need to regress a small offset within a local neighborhood to recover the positional component of the waypoint, which is easier for objectives like MSE compared to directly trying to regress potentially far away targets. PerAct [1] also casts waypoint prediction as classification, but over voxels which is notably harder; the precision of prediction depends on the granularity of the voxel, and the number of action candidates grows cubically in the number of voxels.

• The benefit of this action decomposition for waypoint policy is clearly demonstrated in ablations (Figure 4, right panel) where we show that SPHINX performs significantly better than the vanilla way of predicting waypoints through regression (vanilla waypoint), even when the baseline has the same amount of supervision signal (vanilla waypoint + aux prediction).

2. To our knowledge, SPHINX is the first IL agent to be able to interchange both its input/output representations on the fly depending on the task phase, which is crucial for success in precise, long-horizon tasks.

• Prior works such as Hydra have proposed using a hybrid action space of waypoint/dense actions for IL, but only consider a fixed input space of images. SPHINX performs $5.4\times$ better than Hydra on complex real-world tasks, demonstrating the benefits of using point cloud observations in combination with our novel waypoint action representation

• Several prior works propose waypoint-only policies for manipulation with 3D representations as input; although such representations provide rich spatial information, they do not capture close-up details at a sufficient resolution necessary for accomplishing any one of the real world tasks we consider in this paper. For example, in the train track task (better viewed through the video from website https://sphinx-il.github.io/), the policy needs to identify the “jigsaw puzzle”-like structure on the bridge and track to properly connect them. Point clouds from consumer-grade depth cameras do not provide a sufficiently high resolution to capture such information and there is also the occlusion issue that prevents the 3rd person camera from seeing the gripper-object interaction clearly. Any policy solely based on point clouds will struggle with tasks that require close-up observation, motivating SPHINX’s dense policy.

We also plan to further clarify these two points in the introduction section in a camera-ready version if accepted.

[1] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. "Perceiver-actor: A multi-task transformer for robotic manipulation." Conference on Robot Learning. PMLR, 2023.

The first stage’s visual module appears to use a transformer for salient point extraction, which resembles affordance learning. It would be beneficial to include a discussion on affordance learning.

The connection between "affordances" and "salient points" is a valid question. In robotics, "affordances" can characterize the nature of interaction for certain object points (e.g., a tool handle is "graspable,” a button is "pressable"), or more broader action candidates or success possibilities associated with object points [3]. Salient points in SPHINX can be seen as a form of per-point affordance, but must be combined with offsets to specify how the end-effector should interact with a given point.

For instance, in the coffee-making task, after picking up the mug, the next salient point might lie on the center of the platform of the machine. This provides a meaningful spatial prior over where the policy should interact to place the mug, but does not specify how the robot should orient or position its arm to do so (which a waypoint action specifies). Thus we learn to predict waypoints where the positional component is parameterized as an offset to a salient point.

[2] Borja-Diaz, Jessica, et al. "Affordance learning from play for sample-efficient policy learning." 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022. [3] Ahn, Michael, et al. "Do as i can, not as i say: Grounding language in robotic affordances." CoRL 2022.

We are glad to hear that you appreciate our hybrid policy architecture and the real-world experimental results!

Re. flexibility of mode switching

Mode switching between long-distance free space motion, and short horizon fine-grained manipulation imposes a major constraint on the kinds of problems it can represent effectively; hard attention switching across the inputs limits flexibility.

We actually intended for our hybrid action space to be able to accommodate a broad range of manipulation tasks that waypoint-only or dense-only policies traditionally struggle with. Waypoint-only policies predict movements too sparsely to handle parts of a task requiring reactive, precise manipulation, like inserting a coffee pod into a slot or assembling puzzle pieces like the train track and bridge together. This limitation also applies to the class of waypoint policies mentioned (3DDA, RPDiff, NDF, TAX-Pose). Conversely, dense-only policies, while flexible, often fail to handle longer-horizon tasks with significant visuospatial variations. For example, in our evaluations, dense policies achieved only 5% and 35% success rates on the sequential train track and coffee-making tasks, respectively, compared to 80% and 70% for SPHINX, demonstrating the effectiveness of hybrid actions. Having access to these two distinct modes actually provides more flexibility, as SPHINX can arbitrarily switch back and forth between dense and waypoint modes at any time.

That said, we acknowledge that in our current work SPHINX uses a linear controller for waypoint reaching, limiting its capacity for dynamic tasks. To relax this, future work could predict spline-based trajectories to reach a waypoint or replace SPHINX’s linear controller with a learned [1] or engineered motion planner for dynamic, collision-aware reaching.

[1] Dalal, Murtaza, et al. "Neural mp: A generalist neural motion planner." arXiv preprint arXiv:2409.05864 (2024).

Re. Data Collection Assumptions

Adding a different label for mode is a major annotation requirement, and can be quite noisy/arbitrary

We would like to clarify that demonstrators do not need to manually annotate mode labels. Modes are automatically determined based on input: dense mode is triggered by joystick movements on the Spacemouse, and waypoint mode by mouse movements in the web UI.

Generally speaking they are not a consistent/principled thing to expect a human to label - who is to say which specific points should be annotated? Seems quite noisy, and a major source of error.

Re. noise in salient point annotations: The intent is for the demonstrator to label salient points that are meaningful to the task (e.g., the drawer handle for opening, mug handle for coffee making, etc.). This is often a more clear and direct annotation as opposed to more free-form human annotations such as providing language instructions or narrations. In our case, as our datasets are collected by a single demonstrator, we did not encounter label inconsistency. Additionally, since SPHINX predicts actions as offsets relative to salient points, it can theoretically compensate for somewhat noisy salient points by adjusting the offset predictions accordingly.

That said, we agree that consistent salient point labeling is crucial for scaling up SPHINX data collection. A promising direction for future work is offloading salient point detection to foundation models. For instance, recent models like MOLMO [2] excel at zero-shot semantic keypoint detection (i.e. “point to the ___” in an image, as shown here). This could be used to annotate salient points automatically and only have the demonstrator specify the corresponding waypoint action (offset + orientation + gripper state).

Re. noise in offset specification: Multimodality in teleoperated data is in general an open challenge works that do crowd-sourcing such as [3-5], regardless of the interface used (VR, puppeteering, Spacemouse), but we believe our data collection interface has several features in place to promote more consistent action labels. Our UI allows the demonstrator to adjust a waypoint action (offset/orientation) as much as they would like via simple clicks and drags on a gripper CAD model in the scene, before sending the controller to reach the final specified waypoint. We see this as a strength of our approach, as it allows the demonstrator to be precise with specification before any waypoint actions are executed (as opposed to retrying after being imprecise). The resulting trajectories are also relatively clean due to the linear controller.