We thank the reviewer for their feedback, and appreciate the reviewer’s recognition of the novelty of using multilateration for reasoning and the clarity of presentation.

Responding to individual points raised by the reviewer:

The problem statement section as well as some figures are directly borrowed from TAX-Pose. I think it severely damages the presentation of this paper.

We did reproduce some elements of the problem statement (including one figure) with permission from TAX-Pose authors. We did this to emphasize that the problem setting is nearly identical to theirs; however, we will reword certain sections to differentiate a bit. And we will create an original figure by the end of the rebuttal period to incorporate into the revised manuscript.

The result in Table 1 is margianl improvement compared with PAX-Pose, though the proposed method does exhibit some advantages in higher-precision settings.

We agree that the “overall” improvements appear somewhat small compared to TAX-Pose; however, the original evaluation procedure proposed by NDF and utilized by TAX-Pose is not particularly robust to geometrically-infeasible predictions, which means that predictions that look poor may still be counted as a success because of peculiarities of simulation modeling. That is why we introduced the “precision” metric, to quantitatively demonstrate that our predictions result in higher-quality predictions than TAX-Pose that respect geometric constraints (and are thereby more precise) more frequently.

This method is evaluated with offline real-world trajectories collected by TAX-Pose. I won't accept this as real-world experiments.

This is a reasonable criticism. We included this experiment to demonstrate that our method could be trained on real-world data and produce reasonable predictions - however we stopped short of executing these predictions on a real robot system for specific implementation issues mentioned below.

In figure, the rotational error is 35 degrees. While the authors have made some explanations, I won't regard it as a satisfactory result.

We agree that this rotational error appears unsatisfactory. The metric we chose to present, rotational error w.r.t. A held-out set of demonstrations, does not inherently capture the multimodality of the demonstration set. There are many viable predictions that result in task success, which may differ dramatically in rotation from those seen in demonstrations. For instance, hanging the mug to the left side and to the right side are both valid (and appear in the demonstration set). A better evaluation would be task success rate, where we measure how frequently the predictions lead to a good placement on a real robot system.

Will there be genuine real-world experiments conducted in the future?

We do have a robot system in our lab with 4 cameras and a robot arm, and before submission did attempt to build a system which would execute the predictions made by our method. We were able to get predictions, however unfortunately we struggled to calibrate the robot arm’s motion to the cameras to a degree of precision which is necessary to execute the predictions open-loop with motion planning. We were able to get 2-3cm end-effector precision in our system, which is not sufficient to place mugs on racks effectively. This was independent of our RelDist prediction quality; it was purely an engineering issue.

Because of the complexity of increasing end-effector precision in a calibrated 4-camera setup, we don’t believe we will be able to produce robot experiments by the end of the rebuttal period (in 12 days). However it is definitely on our roadmap to build a robust real-world evaluation setup eventually which has the hand-eye calibration precision required.

Question for the reviewer: Are there any real-world experiments you might accept that don’t require executing the RelDist predictions on a real robot?

We have completed new experiments on 5 high-precision tasks from the RLBench suite - see our Official Comment posted at the top. To summarize, our method achieves very high levels of precision (<1.25 degrees of rotational error, <2mm of translational error on average) on every task, which is within the tolerance margin for successful predictions for each task. The baseline we compare against, TAX-Pose, does not achieve this level of precision at convergence.

Given that our method achieves a high degree of precision on these 5 new high-precision tasks, would you consider increasing your score?

We are also in the process of conducting additional experiments + writing to address your other concerns, and will post again tomorrow.

We thank the reviewer for their feedback, and appreciate the reviewer’s recognition of our novel SE(3)-invariant representation and clear presentation.

Addressing the reveiwer’s questions:

As is mentioned in Weakness, my main comment is about the experiment section. The paper would be much stronger if there is an environment that actually requires high place precision (e.g., gear/kit assembly) rather than merely adjusting the precision requirement in the mug-hang task.

We can certainly attempt to prepare an additional set of high-precision tasks to benchmark the performance of our method against baselines. Specifically, we are considering the following additional experiments:

• 5 relative placement tasks from the RLBench suite of tasks. https://github.com/stepjam/RLBench
  • Insert Peg (precise placement of disc on peg)
  • Insert USB (precise placement of USB in computer)
  • Toilet Roll on Stand
  • Phone on Base
  • Wine Bottle on Rack

There are no specific gear/kit assembly tasks in RLBench, but “Insert Peg” is quite close in spirit (and has existing demonstrations we can leverage).

Additionally, we are attempting to rectify the issues we faced with the TAX-Pose symmetry-breaking techniques that suppressed performance. Hopefully that will allow us to present complete results on Bottle and Bowl placement.

Question for the reviewer: Would including results on these additional tasks satisfy the reviewer’s desire for more high-precision tasks? If not, are there additional specific existing benchmarks that the reviewer would like to see?

Moreover, incorporating the performance of NDF under the 1cm and 3cm precision requirements in Tables 1 and 3 would likely strengthen the paper as well.

We agree that this would be a good and interesting evaluation. Time permitting, we will attempt to extract these values by the end of the evaluation period.

Does the proposed method have a faster runtime compared with the baselines?

Compared to TAX-Pose, we require slightly more compute (due to the additional reasoning layers), but the sample complexity is only a constant factor difference. Compared to NDF, which uses an iterative optimization, we execute significantly faster since we only require a single forward pass of the network to regress.

Does the proposed method have a higher sample efficiency?

Because the network is SE(3) equivariant, we can learn per-object relationships from very few samples (potentially even from a single example). As for class-level relationships (i.e. generalizing to all mugs instead of a single mug), we will attempt an experiment which is similar to that presented in Table 2 of TAX-Pose, i.e. seeing how success rate changes with number in-class demonstrations.

We will provide another update with experiments as they are ready during the rebuttal period.

We have completed new experiments on 5 high-precision tasks from the RLBench suite - see our Official Comment posted at the top. To summarize, our method achieves very high levels of precision (<1.25 degrees of rotational error, <2mm of translational error on average) on every task, which is within the tolerance margin for successful predictions for each task. The baseline we compare against, TAX-Pose, does not achieve this level of precision at convergence.

We switched out the “Insert USB” task because we found that the demo-generating code for this task was broken in the RLBench code release, yielding no demos on which to train. Instead, we have chosen a different high-precision task, “Place Hanger on Rack”, where there is only ~1cm of clearance.

Given that our method achieves a high degree of precision on these 5 new high-precision tasks, would you consider increasing your score?

We are also in the process of conducting additional experiments + writing to address your other concerns, and will post again tomorrow.

We thank the reviewer for their feedback, and for recognizing the novelty of our formulation, RelDist, in solving relative placement tasks in a provably SE(3)-Equivariant way. We also appreciate that the reviewer felt that the work was clear and of high-quality.

To address the questions specifically:

Could experimental results be further augmented with a more diverse set of tasks or scenarios? What could be the choices?

We can certainly attempt to prepare an additional set of tasks to benchmark the performance of our method against baselines. Specifically, we are considering the following additional experiments:

• 5 relative placement tasks from the RLBench suite of tasks. https://github.com/stepjam/RLBench
  • Insert Peg (precise placement of disc on peg)
  • Insert USB (precise placement of USB in computer)
  • Toilet Roll on Stand
  • Phone on Base
  • Wine Bottle on Rack

In each of these tasks, we would only evaluate the placement of an object relative to another object (and therefore eliminate the need to do robot control for grasping), which should demonstrate applicability across a wider range of tasks.

Additionally, we are attempting to rectify the issues we faced with the TAX-Pose symmetry-breaking techniques that suppressed performance. Hopefully that will allow us to present accurate results on Bottle and Bowl placement.

Question for the reviewer: Would including these additional experiments strengthen the paper to your satisfaction?

If so, we will update the rebuttal later in the rebuttal period with results.

If not, what additional simulated experiments would you like to see?

How does the system handle noisy data or incomplete point clouds, common in real-world scenarios?

Our method can be trained on partial/occluded point clouds - for instance, in our Real World experiment (Section 6.2), we do use four cameras, but they do not see the entire mug. There is no fundamental reason why the method shouldn’t work under noise/occlusion - the RelDist representation is still valid under those circumstances. However, the training procedure would need to incorporate that distribution (we already have a synthetic occlusion data augmentation, and could feasibly incorporate a viewpoint dropout or noise augmentation).

Could the authors elaborate on potential strategies for overcoming the limitations related to symmetric objects or multimodal placements?

There are two potential approaches we have considered. The first is empirical symmetry-breaking, similar to what we use in the Bottle/Bowl experiments. You could “paint” each object with a feature field which breaks the symmetry - this painting could be learned generatively to flexibly represent the set - and could be consistent across the class. In this way, the learned kernel function would be able to break the symmetry unambiguously. Precisely formulating this learning process is on our roadmap for future work.

The second is analytical, similar to https://openreview.net/forum?id=psyvs5wdAV which enumerates the symmetry group for each object category. With this privileged information, you could assign the supervision target to be a specific canonical element of that symmetry group (i.e. the RelDist for a particular rotation of a symmetric cube), even if the demonstration is a different element of that group, and transform the predicted canonical representation into the group element seen at training time (to make the losses consistent). However, this requires a lot of assumptions, including the existence of a consistent symmetry group across a class (may not be true, i.e. some bottles are symmetric, some are not).

We have completed new experiments on 5 high-precision tasks from the RLBench suite - see our Official Comment posted at the top. To summarize, our method achieves very high levels of precision (<1.25 degrees of rotational error, <2mm of translational error on average) on every task, which is within the tolerance margin for successful predictions for each task. The baseline we compare against, TAX-Pose, does not achieve this level of precision at convergence.

Given that our method achieves a high degree of precision on these 5 new high-precision tasks (and provides a more complete evaluation of the method), would you consider increasing your score?

We thank the reviewer again for their feedback. We wanted to provide a final update, including a new set of requested experiments and updates to our manuscript.

New Experiments

During the rebuttal period, we performed the following experiments:

• We evaluated our method on a new set of 5 precise placement tasks from the RLBench task suite. We found that our method achieves substantially higher precision than the closest baseline, TAX-Pose. We believe that this is strong evidence of our method’s suitability for precise placement tasks.

• We have performed an additional sample efficiency ablation. See our second post above, entitled “New experiment: Strong sample efficiency for learning precise placement”. We find that our method is able to learn a highly-precise relationship from only a single demonstration, and uniformly achieves higher precision-per-sample than TAX-Pose.

Manuscript Revisions

We have made the following changes to the manuscript:

• Added a new experiment subsection, which describes the RLBench suite of tasks and presents our main results and includes some visualizations of the task.
• Added the new sample efficiency ablation
• Created a new original figure illustrating relative placement for the Problem Statement section.

Overall, we hope we have addressed the majority of the reviewers’ concerns about our experiments section. We believe that our revised manuscript is substantially stronger with the addition of RLBench experiments and analysis.