3D Diffuser Actor: Multi-task 3D Robot Manipulation with Iterative Error Feedback

I also found that this paper is very similar to Act3D (it seems a lot of contents are borrowed). In addition, this work [1] seems to be also very related and in a very close style.

I personally think this problem might be much worse than just a strong reject, if without any descent explanation from the authors.

[1] ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation, https://chained-diffuser.github.io/

1. Further analysis on the importance of diffusion

No deeper analysis about why diffusion models could help. The ablation results only show two factors matter, but all these factors seem to be not novel and not surprising, thus deeper analysis might be necessary.

Motivated by your comment and the comment of Reviewer Ws6y, and in an attempt to validate the argument regarding utility for diffusion, we conducted an additional experiment where we use a close to identical architecture to 3D Diffuser Actor for regressing the 3D location and 3D rotation of the target keypose, as opposed to the noise and the use of iterative denoising (no denoising iteration index input for regression). We show the comparison for 3D location prediction on train and val sets (here) and the comparison for 3D rotation prediction on train and val sets (here).

As you can see, 3D diffuser actor generalizes much better than the regression model, especially for 3D location. Both models use the same scene encoder, language encoder, and 3D relative transformer decoder architecture, but one predicts the end-effector through iterative denoising, while the other(regression) through a single feedforward pass. Our contribution: We propose the first 3D keypose diffusion model denoised from 3D scene representations for learning 3D manipulation from demonstrations. We achieve superior performance over many prior methods in apples-to-apples comparisons in an established experimental setup on RLBench(+12% compared to state-of-the-art methods). Our model, 3D Diffuser Actor, marries 3D representations with diffusion policies for robot manipulation, as we describe clearly in the abstract ("a framework that marries diffusion policies and 3D scene representations for robot manipulation") and in the intro. In both abstract and intro we first describe diffusion policies (in abstract: "Diffusion policies capture the action distribution conditioned on the robot and environment state using conditional diffusion models. They have recently shown to outperform both deterministic and alternative generative policy formulations in learning from demonstrations."), then 3D policies (in abstract: "3D robot policies use 3D scene feature representations aggregated from single or multiple 2D image views using sensed depth. They typically generalize better than their 2D counterparts in novel viewpoints.") and we say we introduce 3D diffusion policies that marry the above two: (in abstract: "We unify these two lines of work and present a neural policy architecture that uses 3D scene representations to iteratively denoise robot 3D rotations and translations for a given language task description.") The specific way this marriage is accomplished is important: we represent the current noisy end-effector pose estimate as a 3D token in the 3D feature cloud and featurize it with relative 3D attentions. Since the ICLR submission deadline, we ran the exact same model with absolute position attention (closer to what 2D diffusion policies do) on the whole PerAct benchmark and obtained an 8% worse performance. We consider the above 3D diffusion keypose policy model to be our contribution.

Please, also see the general response in the beginning of the rebuttal.

I have also checked the supplementary files and the presented Figure 7 looks interesting, but why scaled linear is worse than square cosine when the former one seems to cover the original distribution better?

The objective of the noising process is to cover the whole space densely, as during inference we start from pure Gaussian noise and gradually denoise, following the DDPM method. As Figure 7 shows, the scaled linear schedule captures the distribution instead of covering the whole space of possible rotations. As a result, when randomly initializing from Gaussian noise, our model gets out of distribution and cannot denoise to a valid solution.

Thank the authors for detailed explanations. I have the following further questions.

4. Real-robot experiments.

This is hard to do as it requires resetting the real environment to identical state across multiple models. PerAct, Act3D, RVT, InstructRL and most other works do not consider baselines in the real world experiments for that reason. You are absolutely correct, we did not have the time to do that, it is within the capabilities of our model. We leave this for future work.

Considering the time given in the discussion period (~12 days), I do not think any initial real robot experiments (e.g., one task and 10 trials) could not be conducted. Training one baseline agent might take mostly 1-2 days.

And yes, you are right that most of the papers you mention do not compare other baselines in the real world. However, as the authors describe, the setting is closely following PerAct [1]. PerAct is not comparing with other baselines mainly because there are no competitive baselines. In addition, there are also papers in a close setting that actually do, e.g., GNFactor [2].

Overall, I do not think the excuse from the authors holds.

5 Comparision to GNFactor

Following your suggestion, we will report our performance with one camera and 20 demonstrations on these 10 tasks to be able to compare with GNFactor. Thank you for your suggestion.

It seems the updated paper does not include any new discussion or new experiments regarding this question.

Due to these discrepancies, we did not know how to judge their setup.

Again, this discrepancy can not hold as an excuse for no discussion and no experiments, since the code of GNFactor [2] is available online, and the small discrepancy (only the task and the number of demonstrations) is very common for imitation learning papers. Comparing them under the same setting is not an impossible thing, let alone with enough time for the discussion period.

I would maintain my score, but be willing to raise my score if the problems are addressed better.

[1] Shridhar, Mohit, et al. "Perceiver-actor: A multi-task transformer for robotic manipulation." CoRL, 2022.

[2] Ze, Yanjie, et al. "Gnfactor: Multi-task real robot learning with generalizable neural feature fields." CoRL, 2023.

1. Relationship to Act3D

My primary concern about this work is that a portion of the technical method and writting appears to be directly borrowed from [1]. However, this relationship is not transparently acknowledged.

Please, see the general response in the beginning of the rebuttal.

The proposed framework adopts the 3D scene representation (in a simplified form), 3D relational transformer, and training/evaluation setups from [1], which is never explicitly mentioned in the paper.

Please, see the general response in the beginning of the rebuttal.

2. Discussion on multimodality and contribution

The main contribution of this work lies in the use of a diffusion policy to capture multimodal action distributions. However, this aspect is not extensively discussed or thoroughly evaluated.

Motivated by your comment and the comment of Reviewer sG3K, and in an attempt to validate the argument regarding utility for diffusion, we conducted an additional experiment where we use a close to identical architecture to 3D Diffuser Actor for regressing the 3D location and 3D rotation of the target keypose, as opposed to the noise and the use of iterative denoising (no denoising iteration index input for regression). We show the comparison for 3D location prediction on train and val sets (here) and the comparison for 3D rotation prediction on train and val sets (here).

As you can see, 3D diffuser actor generalizes much better than the regression model, especially for 3D location. Both models use the same scene encoder, language encoder, and 3D relative transformer decoder architecture, but one predicts the end-effector through iterative denoising, while the other(regression) through a single feedforward pass.

Our contribution: We propose the first 3D keypose diffusion model denoised from 3D scene representations for learning 3D manipulation from demonstrations. We achieve superior performance over many prior methods in apples-to-apples comparisons in an established experimental setup on RLBench(+12% compared to state-of-the-art methods). Our model, 3D Diffuser Actor, marries 3D representations with diffusion policies for robot manipulation, as we describe clearly in the abstract ("a framework that marries diffusion policies and 3D scene representations for robot manipulation") and in the intro. In both abstract and intro we first describe diffusion policies (in abstract: "Diffusion policies capture the action distribution conditioned on the robot and environment state using conditional diffusion models. They have recently shown to outperform both deterministic and alternative generative policy formulations in learning from demonstrations."), then 3D policies (in abstract: "3D robot policies use 3D scene feature representations aggregated from single or multiple 2D image views using sensed depth. They typically generalize better than their 2D counterparts in novel viewpoints.") and we say we introduce 3D diffusion policies that marry the above two: (in abstract: "We unify these two lines of work and present a neural policy architecture that uses 3D scene representations to iteratively denoise robot 3D rotations and translations for a given language task description.") The specific way this marriage is accomplished is important: we represent the current noisy end-effector pose estimate as a 3D token in the 3D feature cloud and featurize it with relative 3D attentions. Since the ICLR submission deadline, we ran the exact same model with absolute position attention (closer to what 2D diffusion policies do) on the whole PerAct benchmark and obtained an 8% worse performance. We consider the above 3D diffusion keypose policy model to be our contribution.

Please, also see the general response in the beginning of the rebuttal.

1. Same backbone as Act3D

The backbone of the proposed network is extremely similar to Act3D, including using multi-view image input, using the pyramid network for feature exaction, the generation of the 3D feature cloud, and the 3D relative transformer. I understand that Act3D is a very recent work, however, as the authors are already aware of Act3D, proper discussion regarding the relationship between this work and Act3D should be addressed.

Please, see the general response in the beginning of the rebuttal regarding the relationship between this work and Act3D.

2. Explanation of 3D feature cloud

When building the 3D scene feature cloud, the authors claim, We associate every 2D feature grid location in the 2D feature maps with a depth value, by averaging the depth values of the image pixels that correspond to it. What is a grid here? Is each 2D feature map from the feature pyramid network separated into an NxN grid akin to ViT?

An image encoder extracts a $H \times W \times C$ feature map from each camera view, where $H$, $W$, and $C$ denotes the height, width and channel dimension of the feature map. Each $1 \times 1 \times C$ feature vector (we call it token) in the feature map corresponds to a fixed-size patch in the input image. We can thus lift a 2D feature token to 3D by averaging the 3D locations of all pixels within the patch. The 3D locations of pixels are obtained by combining the 2D pixel locations and depth values (x, y, d) using the camera intrinsics and pinhole camera equation. We updated the manuscript and Figure 1 to explain the feature extraction and 2D-3D lifting procedure more clearly.

When generating the 3D feature cloud, how does the method handle cases where multiple points (pixels) from different views correspond to the same 3D location? Are the features averaged in such instances?

We found that having two points with identical 3D locations rarely happens, so we did not explicitly handle this case. Although some merging could be applied, such as averaging, we adopted a more general solution and simply let attention focus on the important features.

What constitutes a visual token in this context? Are individual 3D points considered tokens?

Each feature vector from the 2D feature maps lifted to 3D is a 3D token.

3. Predicting further in the future

Given that diffusion policies suggest the utility of diffusing multiple action steps into the future, has the method been evaluated with multiple keypoint steps in the diffusion process?

Thank you for your suggestion. Although in this version we only predict the next keypose, our model could indeed support the prediction of multiple keyposes and it would be interesting to explore that in the future as a way to plan. Nonetheless, we would like to emphasize that our diffusion model predicts the next subgoal to reach, not low-level actions directly. Instead, once a subgoal keypose has been predicted, then a low-level planner is assigned with predicting a sequence of actions that connect the current pose and the predicted keypose. This decomposition has been followed by previous approaches as well (e.g. C2F-QA, PerAct, Act3D). This is different from what related works on diffusion policies do, which is to predict low-level actions directly.

1. Contribution

The scientific contribution of this paper is unclear. Diffusion model could not be the contribution of this paper, and adopting a diffusion model for trajectory generation is not a new idea (see [a]).

Our contribution: We propose the first 3D keypose diffusion model denoised from 3D scene representations for learning 3D manipulation from demonstrations. We achieve superior performance over many prior methods in apples-to-apples comparisons in an established experimental setup on RLBench(+12% compared to state-of-the-art methods). Our model, 3D Diffuser Actor, marries 3D representations with diffusion policies for robot manipulation, as we describe clearly in the abstract ("a framework that marries diffusion policies and 3D scene representations for robot manipulation") and in the intro. In both abstract and intro we first describe diffusion policies (in abstract: "Diffusion policies capture the action distribution conditioned on the robot and environment state using conditional diffusion models. They have recently shown to outperform both deterministic and alternative generative policy formulations in learning from demonstrations."), then 3D policies (in abstract: "3D robot policies use 3D scene feature representations aggregated from single or multiple 2D image views using sensed depth. They typically generalize better than their 2D counterparts in novel viewpoints.") and we say we introduce 3D diffusion policies that marry the above two: (in abstract: "We unify these two lines of work and present a neural policy architecture that uses 3D scene representations to iteratively denoise robot 3D rotations and translations for a given language task description.") The specific way this marriage is accomplished is important: we represent the current noisy end-effector pose estimate as a 3D token in the 3D feature cloud and featurize it with relative 3D attentions. Since the ICLR submission deadline, we ran the exact same model with absolute position attention (closer to what 2D diffusion policies do) on the whole PerAct benchmark and obtained an 8% worse performance. We consider the above 3D diffusion keypose policy model to be our contribution. Please, also see the general response in the beginning of the rebuttal.

2. Explanation of 3D feature cloud

The figures in this paper lack sufficient illustrative value and information. For instance, Figure 1 appears to show the model taking a multi-view image as input, yet the caption indicates it uses a 3D scene feature cloud. A clear connection between these two elements would greatly improve understanding.

An image encoder extracts a $H \times W \times C$ feature map from each camera view, where $H$, $W$, and $C$ denotes the height, width and channel dimension of the feature map. Each $1 \times 1 \times C$ feature token in the feature map corresponds to a fixed-size patch in the input image. We can thus lift a 2D feature token to 3D by averaging the 3D locations of all pixels within the patch. The 3D locations of pixels are obtained by combining the 2D pixel locations and depth values (x, y, d) using the camera intrinsics and pinhole camera equation. We updated the manuscript to explain the feature extraction and 2D-3D lifting procedure more clearly. We also updated Figure 1 to illustrate the lifting process.

3. Frequency of keypose prediction

Your diffusion model appears to produce only the target pose of the action, after which a trajectory is generated using the MoveIt planner given the initial joint state and target joint state. Does the robot execute the entire trajectory directly until the end, or does it update the target pose using the diffusion model and regenerate the trajectory after each forward step?

Once a keypose is predicted, a low-level planner is tasked to predict a trajectory between the current and the predicted pose. This trajectory is executed open-loop until the end. We found that this was sufficient for all 18 tasks we focus on. At the same time, it’s more efficient computationally, since the denoising process is sparsely called.

I would like to thank the authors for their comprehensive responses, touching up of the figures.

The overall writing of this paper is more clear. However, after going through all the comments (from the authors and other reviewers), I am still not convinced by the claimed contributions. The proposed method is more of an assemble of existing techniques without providing adequate sharp insight the of manipulation problem.

I will downgrade my rating as the authors does not provide adequate strong argument for their contributions and evidence for the advantages of the proposed method in real world.

Plus, I also change my rating of the presentation of this paper from 2 to 3.

The models of Act3D and 3D Diffuser Actor are thus different in that:

1. One is a diffusion model the other is a deterministic model
2. They have different inputs and outputs as noted above. One maps x to y directly through coarse to fine 3D point sampling for 3D location and regression for 3D rotation, the other is a denoising diffusion model that takes x and the current y estimate (3D location and 3D rotation) and iteration index and predicts the noise in the current estimate.
3. There is no coarse-to-fine sampling or any form of point sampling in 3D Diffuser Actor. 3D Diffuser Actor does not build a 3D map (a map of 3D locations where each one is scored whether it contains the next end-effector 3D keypose or not).
4. The 3D relative transformers differ as noted above: 1) the token sets are very different as explained above, 2) one is a cross-attention and the other a self-attention.

The models of Act3D and 3D Diffuser Actor are similar in that:

1. They both predict 3D keyposes for the end-effector and are trained from demonstations. Many more methods live in this setting such as: PerAct, RVT, InstructRl, HiveFormer, C2F-QA.
2. They lift RGB-D images into 3D feature clouds and use 3D transformers with relative attentions (a similarity we acknowledge at the top of page 6 and clearly explain in the related work).

We did not acknowledge the 3D feature building because we believe it is a standard way of obtaining a 3D feature point cloud. For example, it has been used before in:

1. OpenScene: 3D Scene Understanding with Open Vocabularies, https://openaccess.thecvf.com/content/CVPR2023/papers/Peng_OpenScene_3D_Scene_Understanding_With_Open_Vocabularies_CVPR_2023_paper.pdf
2. Virtual Multi-view Fusion for 3D Semantic Segmentation https://arxiv.org/abs/2007.13138.
3. ChainedDiffuser(discussed later)

We cited Act3D 10 times in our submitted paper, and it is our most competitive baseline. We clearly do not claim any of its contributions. In the submitted paper we do not claim 3D representations to be our contribution.

Our contribution

We propose the first 3D keypose diffusion model denoised from 3D scene representations for learning 3D manipulation from demonstrations. We achieve superior performance over many prior methods in apples-to-apples comparisons in an established experimental setup on RLBench(+12% compared to state-of-the-art methods). Our model, 3D Diffuser Actor, marries 3D representations with diffusion policies for robot manipulation, as we describe clearly in the abstract ("a framework that marries diffusion policies and 3D scene representations for robot manipulation") and in the intro. In both abstract and intro we first describe diffusion policies (in abstract: "Diffusion policies capture the action distribution conditioned on the robot and environment state using conditional diffusion models. They have recently shown to outperform both deterministic and alternative generative policy formulations in learning from demonstrations."), then 3D policies (in abstract: "3D robot policies use 3D scene feature representations aggregated from single or multiple 2D image views using sensed depth. They typically generalize better than their 2D counterparts in novel viewpoints.") and we say we introduce 3D diffusion policies that marry the above two: (in abstract: "We unify these two lines of work and present a neural policy architecture that uses 3D scene representations to iteratively denoise robot 3D rotations and translations for a given language task description.") The specific way this marriage is accomplished is important: we represent the current noisy end-effector pose estimate as a 3D token in the 3D feature cloud and featurize it with relative 3D attentions. Since the ICLR submission deadline, we ran the exact same model with absolute position attention (closer to what 2D diffusion policies do) on the whole PerAct benchmark and obtained an 8% worse performance. We consider the above 3D diffusion keypose policy model to be our contribution.

Multi-task real-world experiments, baselines for the real-world experiments

Thank you for your feedback. To satisfy your concern we conducted multi-task real-world experiments and we compare our 3D Diffuser Actor with Act3D under such a multi-task setup. We consider the folowing five tasks: 1. pick up bowls, 2. put grapes in the bowl, 3. flip bottle, 4. fold towel, 5. open oven. We use 20 demonstations per each task to train, and 10 for testing. We show results in the Table below, which we will add in a section in our final version.

\begin{array}{l|cccc} & \text{Flip bottle} & \text{Pick-up bowls} & \text{Put grapes} & \text{Fold towel} & \text{Open oven} \newline \hline \text{3D Diffuser Actor} & 100 & 100 & 50 & 60 & 20 \newline \text{Act3D} & 100 & 70 & 60 & 50 & 10 \newline \end{array}

We see 3D Diffuser Actor outperforms Act3D also in the real world. This is not surprising as real world demonstations are even more multimodal than the ones in the RLbench simulator. We are working on collecting more real world tasks for our model, to match the task diversity of Per-Act's real world setup.

Comparison with GNFactor

Reviewer sG3K asked us to compare with GNFactor (Ze et al. CoRL 2023). The contribution of GNFactor is to train a neural network to complete a 3D feature volume given a single view. Given the complete 3D feature volume, they use an action prediction head identical to Per-Act. Thus, GNFactor's contribution is orthogonal and complementary to 3D Diffuser Actor: we can use such 3D feature cross-view completion with our own 3D Diffuser Actor action prediction head and get similar performance boosts. Thus comparing an action prediction head with a 3D feature completion method may not be an apples-to-apples comparison. However, we believe it is very valuable to still compare between the two, to also satisfy the reviewer's concern. To do that, we are training a version of 3D Diffuser Actor that uses:

• a single camera (instead of 4 cameras)
• 10 task selected by GNFactor (instead of 18)
• 20 demos per task (instead of 100)

The Table below shows our manipulation results, we also include GNFactors for comparison. We believe the setup is identical to the one GNFactor considers.

\begin{array}{l|ccccccccccc} & \small\text{Avg.} & \small\text{close} & \small\text{open} & \small\text{sweep to} & \small\text{meat off} & \small\text{turn} & \small\text{slide} & \small\text{put in} & \small\text{drag} & \small\text{push} & \small\text{stack} \newline & \small\text{Success.} & \small\text{jar} & \small\text{drawer} & \small\text{dustpan} & \small\text{grill} & \small\text{tap} & \small\text{block} & \small\text{drawer} & \small\text{stick} & \small\text{buttons} & \small\text{blocks} \newline \hline \text{3D Diffuser Actor} & \small 72 & \small 46 & \small 90 & \small 100 & \small 75 & \small 69 & \small 86 & \small 88 & \small 99 & \small 71 & \small 0 \newline \text{GNFactor} & \small 31.7 & \small 25.3 & \small 76.0 & \small 28.0 & \small 57.3 & \small 50.7 & \small 20.0 & \small 0.0 & \small 37.3 & \small 18.7 & \small 4.0 \newline \end{array}

As you can see, 3D diffuser Actor without any 3D feature completion dramatically outperforms GNFactor that uses a Per-Act action prediction head. The results show that 3D Diffuser Actor is robust to the number of views used. Indeed, in the real world, 3D Diffuser Actor considers a single viewpoint. We would like to thank Reviewer sG3K for urging us in this comparison. We will include these results in our manuscript.

Low-performance on the close_jar task

Thank you for pointing this out. After looking into this issue, we found that the success conditions of the task might not be correct. We measured the success rate of the expert demonstrations on the validation set, where demonstrations and RLBench library are released by PerAct. Surprisingly, we found that all ground-truth demonstrations fail: the overall success rate is 0% by replaying demonstrations. We contacted Act3D authors, who confirmed our findings and their modification of the success condition. We will report this issue to the PerAct authors so the benchmark can be corrected. Using the fixed success conditions from Act3D, 3D Diffuser Actor achieves 97% on close_jar. Thank you for pushing us in this investigation as we believe this will correct a benchmark that will be used by many researchers.

Additional experiments: Complete ablations on 18 RLBench tasks

Since submission, we were able to expanded our ablations to the full PerAct 18-task benchmark (in the submitted paper we only show ablations across indivisual tasks due to lack of resources). Specifically, we consider the following ablative versions of our model which we introduced in our submission and we present again here for completeness:

• 2D Diffuser Actor, is a method identical to 3D Diffuser Actor but uses 2D scene representations similar to (Chi et al., 2023), instead of 3D. We remove the 3D scene encoding from 3D Diffuser Actor, and instead generate per-image 2D representations by average-pooling features within each view, and add learnable embeddings to distinguish different views, following (Chi et al., 2023). We use attention layers for jointly encoding the action estimate and 2D image features, following (Chi et al., 2023)

• 3D Diffuser Actor w/o RelAtt, an ablative version of our model that uses standard non-relative attentions to featurize the current rotation and translation estimate with the 3D scene feature cloud, and does not ground the current gripper estimate in the scene.

As shown in the following table, our 3D Diffuser Actor consistently outperforms these two ablative versions among most tasks. The results verifies the two main claims of this work:

• explicitly using 3D scene representations over 2D ones is crucial for learning to manipulate and
• representing the gripper as a scene token and contextualizing it with relative attentions is important for generalization. We will include the new results in the final version of the paper.

\begin{array}{l|c|c|c|c|c|c|c|c|c|c} & \small\text{Avg.} & \small\text{open} & \small\text{slide} & \small\text{sweep to} & \small\text{meat off} & \small\text{turn} & \small\text{put in} & \small\text{drag} & \small\text{place}& \small\text{close} \newline & \small\text{Success} & \small\text{drawer} & \small\text{block} & \small\text{dustpan} & \small\text{grill} & \small\text{tap} & \small\text{drawer} & \small\text{stick} & \small\text{cups} & \small\text{jar} \newline \hline \smallPerAct & \small43 & \small80 & \small72 & \small56 & \small84 & \small80 & \small68 & \small68 & \small0 & \small60 \newline \smallRVT & \small63 & \small71 & \small82 & \small72 & \small88 & \small94 & \small88 & \small99 & \small4 & \small52 \newline \smallAct3D & \small65 & \small93 & \small93 & \small92 & \small94 & \small94 & \small90 & \small92 & \small3 & \small92 \newline \hline \small2D Diffuser Actor & \small44 & \small68 & \small65 & \small90 & \small75 & \small72 & \small55 & \small79 & \small0 & \small94 \newline \small3D Diffuser Actor without Rel. Attn. & \small68 & \small96 & \small99 & \small87 & \small98 & \small97 & \small91 &\small 98 & \small0 & \small94 \newline \small3D Diffuser Actor (ours) & \small79 \small\textbf{} & \small87 & \small97 & \small99 & \small95 & \small97 & \small92 & \small100 & \small32 & \small97 \newline \hline & \small\text{stack} & \small\text{screw} & \small\text{put in} & \small\text{place} & \small\text{put in} & \small\text{sort} & \small\text{push} & \small\text{insert} & \small\text{stack} \newline & \small\text{blocks} & \small\text{bulb} & \small\text{safe} & \small\text{wine} & \small\text{cupboard} & \small\text{shape} & \small\text{buttons} & \small\text{peg} & \small\text{cups} \newline \hline \hline \smallPerAct & \small36 & \small24 & \small44 & \small12 & \small16 & \small20 & \small48 & \small0 & \small0 \newline \smallRVT & \small29 & \small48 & \small91 & \small91 & \small50 & \small36 & \small100 & \small11 & \small26 \newline \smallAct3D & \small12 & \small47 & \small95 & \small80 & \small51 & \small8 & \small99 & \small27 & \small9 \newline \hline \small2D Diffuser Actor & \small1 & \small48 & \small57 & \small68 & \small3 & \small0 & \small9 & \small6 & \small0 \newline \small3D Diffuser Actor without Rel. Attn. & \small35 & \small48 & \small97 & \small94 & \small64 & \small7 & \small98 & \small6 & \small12 \newline \small3D Diffuser Actor (ours) & \small68 & \small91 & \small99 & \small90 & \small74 & \small1 & \small99 & \small79 & \small34 \newline \end{array}

RWs6y: explicitly demonstrate the multi-modality robot behavior qualitatively or quantitatively

We believe the curves comparing 3D Diffuser with regression show quantitatively that diffusion helps over regression. For qualitative results you can see our Figure 1 and our supplementary videos. In Figure 1, you can see that the model has learnt 3 different modes for the same task, two of which grasp the same object in two different ways and one that grasps a different object first. Our method predicts keyposes, not continuous trajectories for the robot's end-effector, that is why you see not continuous trajectories visualized in the figure, but rather, next possible keyframes.

RWs6y: Incremental contribution, trivial extension to Act3D

Reviewer Ws6y mentions: "However, as the scene representation it uses can be trivially adapted from Act3D, replacing the regression policy head in Act3D with a diffusion model is an incremental contribution." The rebuttal we submitted, as well as the additional experiments and ablations we showed above suggest this is not the case.

Summary

We have presented a model that uses 3D representations and rendering of the estimates of a 3D gripper in the scene in a 3D denoising transformer model for learning multi-task manipulation policies from demonstations. The model works well with varying number of input views. It has state of the art performamce with a wide margin against the previous sota (12%) in RLbench, and it also works very well in the real world. It is general and principled. We thank the reviewers for pushing us to improve our work by asking for clearer writing, additional experiments, qualitative results and comparisons with latest methods.