RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches

“The ordering of Fig. 2 seems rather arbitrary.”

Thank you for the feedback. We agree that the figure can be subjective. We will remove it in the revision.

“The approach makes strong assumptions that contradict the claim on scalability.”

Thank you for pointing out our assumptions that might affect scalability.

1. Static and calibrated camera during training and inference: When creating hindsight trajectory labels during training, we extracted segments from demonstrations which only contained stationary manipulation. During inference, one potential way to extend to mobile camera scenarios is to redraw trajectories if the robot moves during the rollout. As a future work, we are exploring automatically re-generating new trajectories on-the-fly during inference. In addition, we would like to emphasize that both our training data and evaluation are collected and conducted across different robots (distributing collection and evaluation on more than 10 robots). Thus, it is not required to have a highly precise camera calibration.

2. “The end effector should always be visible in the camera”: The end-effector is not required to always be visible within one episode. We will snap the projected trajectory to the edge of the image if the end-effector is out of view. It is also reasonable to assume that the end-effector is within the view for most time for a visual servoing system.

“what "unrelated semantic categories" mean for a low-level skill.”

Thank you for the question. We have refined Fig. 8 and removed this description in the revision. For a learning-based policy, the task specification (e.g., semantics) can be highly related to the low-level motion. For example, for a language-conditioned policy, even if the low-level motion is similar, due to large gaps of language conditioning between training and test, the policy can still fail to generalize. We intend to emphasize that our approach can generalize to new tasks with unseen semantics because it is designed to generalize based on motion.

Thank you for the appreciation of our work and the helpful comments. We have carefully incorporated improvements to our work based on your feedback. We hope we have addressed your concerns, and are happy to follow up with any additional questions!

Lack of strong enough connection between the traditional idea of trajectory tracking and the proposed approach

We thank the reviewer for pointing out the missing discussion on traditional trajectory tracking for robotic control. We have included a dedicated subsection in the related work to situate our method against the relevant literature on trajectory tracking in control theory and robotics. Namely, we would like to highlight the body of work on adaptive and learning-based control which may be applied to closed-loop control for tracking reference trajectories, which may be provided after an initial trajectory planning stage or be updated dynamically in an online fashion. Most importantly, we emphasize that these model-based trajectory tracking algorithms are generally sensitive to accurate trajectory plans (which must be overspecified in ground truth state space) and are not robust to tracking errors in presence of dynamic or uncertain situations. Our proposed method makes fewer assumptions on the fidelity and accuracy of the coarse trajectory sketch, and aims to leverage the benefits of end-to-end learning to generalize to uncertain or complex scenarios. Nonetheless, we view our method and similar approaches as a hybrid abstraction that aims to leverage some of the benefits showcased in model-based trajectory tracking systems while also exploiting the benefits of end-to-end data-driven policy learning.

“The comparison against baselines in Sec 4.2 is not apples to apples.”

We acknowledge that the comparison in Sec 4.2 is not apples to apples. The objective of this section is to demonstrate that trajectory-conditioned policies outperform baselines with language or goal image conditioning. It indicates a policy conditioning representation showing how to complete a task can be more amenable to task generalization than common baseline approaches.

“The approach is, on average, worse than generating a trajectory with a VLM and tracking it with an IK, which somehow is against the main paper's contribution.”

Our approach performs worse on the task, Open Drawer, since the waypoints (always grasping the drawer handle) generated by LLM on this task are quite different from the motions in our training data (which mainly grasps the drawer edge). As shown in the Appendix (retry behaviors), RT-Trajectory can follow the trajectory in order to grasp the handle first, while finally opening the drawer by grasping the edge. Such retry behaviors are inherently impossible for the IK planner. Besides, we would like to point out that the IK planner requires relatively accurate trajectories of 6D poses while our approach only takes a 2D sketch as input.

“Each subsection seems to evaluate on different "tasks", and it is unclear how the task selection is done.”

Thank you for pointing out this question about task selection. We have done additional experiments to compare RT-Trajectory against the IK planner given trajectories generated from human demonstrations videos on the Open Drawer skill and prompting LLMs on the Fold Towel skill, in order to ensure the same set of skills are evaluated in Table 1. Notably, the IK planner consistently fails to grasp the drawer edge (resulting in a success rate of 0), likely caused by the noisy trajectories generated from human demonstration videos for Open Drawer.

The reason why we only evaluate some of the seen or unseen skills in Sec 4.3 rather than all 7 unseen skills in Sec 4.2 is that the proposed unseen skills might be challenging for certain trajectory generation methods. For example, prompting LLMs to achieve swiveling a chair or folding a towel is empirically very difficult and out of the scope.

“Still, its main experiments on generalization over scenes and tasks only occupy a small part of the paper (Sec. 4.5).”

In this work, we emphasize task-level generalization, and mainly present quantitative results on 7 unseen skills in Sec 4.2, as well as motion analysis in Sec. 4.4. Sec. 4.5. mainly shows some emergent abilities and case studies qualitatively.

With the rebuttal period coming to a close, we hope we've addressed your concerns. We kindly ask that you consider raising your rating based on the above response. Feel free to let us know if you'd like us to address anything further. Thanks again for your thoughtful feedback!

Interestingly, the trajectory generated by LLM prompting outperforms the proposed approach. Why does this happen? I would evaluate this more broadly and on all tasks of Fig. 4. If that works best overall, this might be even more scalable (no need for humans in the loop!).

While it is true that the LLM-generated trajectories perform well for the "Pick" skill when paired with either the IK Planner Policy or RT-Trajectory Policy, we find that when using trajectories generated by LLMs, the IK Planner is roughly on par with our method (2.5D version) on average (IK Planner average success rate is 60%, while ours (2.5D) average success rate is 58%). In addition, executing the IK Planner with trajectories generated by prompting LLMs is not always the best across the board. In Table 1, we find that executing RT-Trajectory (2.5D) with trajectories generated by human demonstration videos shows the highest average success rate (88%).

Additionally, we would like to note that our proposed approach is mainly focused on training the policy with hindsight-extracted trajectories; at inference time, we believe the flexibility to interact with different types of approaches to generate trajectories (via prompted LLMs, drawings, VLMs, human videos) are all equally viable approaches. In future work, we hope that stronger statements about the "most optimal" trajectory specification method can be made.

Regarding extending LLM-generated Trajectories to all tasks of Fig. 4, we agree that this would be a very interesting experiment. Prior work leveraging such an approach has found significant variance with the types of tasks that LLM-generated Trajectories can accomplish (varying with scene complexity, precision required, amount of contact, etc.) [1] [2]. We empirically also experience this, where we find near 0 success rate of IK Planners using LLM-generated Trajectories on the tasks in Fig. 4, since the quality of the proposed trajectories were so low. Thus, we did not further explore LLM-generated Trajectories with RT-Trajectory on those tasks, since the baseline was so poor, and the other trajectory generation methods seemed to work well. Nonetheless, we are optimistic that improved LLMs in the future may be able to produce better trajectories (in both 2D image space [3] as well as in 3D waypoint space), and would be excited to explore these directions in future work.

References:

• [1] "Code as Policies: Language Model Programs for Embodied Control", J. Liang et al. ICRA 2023, https://arxiv.org/abs/2209.07753
• [2] "How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies", M.G. Arenas et al. CoRL 2023 TGR Workshop, https://openreview.net/forum?id=T8AiZj1QdN
• [3] "The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)", Z. Yang et al. Arxiv Preprint 2023, https://arxiv.org/abs/2309.17421

Finally, I see the point the paper makes a generalization, but I don't understand why one cannot do a more in-depth quantitative evaluation, possibly on the same tasks as before but with a different background/scene/drawer, etc.

We acknowledge that additional experiments on the same tasks but with different background/scene/drawer would have provided more in-depth quantitative evaluation of generalization. Nonetheless, we do believe that generalization is an overly broad term, and can mean robust performance under many types of distribution shifts (and combinations of distribution shifts). The main focus of this work is to demonstrate RT-Trajectory can perform tasks that language or goal conditioned end-to-end learning methods find difficult to perform. We ran more than 300 episodes of evaluation on real robots, just for section 4.2 alone, on 7 new tasks and demonstrated conclusive evidence that RT-Trajectory outperforms strong language and goal conditioned baselines on these tasks.

We fully agree that studying generalization on the same set of tasks but with different background/scene/drawers are interesting areas of future work, especially in tandem with building trajectory-conditioned methods on top of high-capability models such as RT-2 [4]. We are very excited to try these additional experiments for the paper, but due to the limited time of the rebuttal period, we haven’t had the chance to run these experiments yet.

References:

• [4] "Rt-2: Vision-language-action models transfer web knowledge to robotic control.", Brohan, Anthony, et al., arXiv preprint arXiv:2307.15818 (2023).

Thank you for the appreciation of our work and the helpful comments. We look forward to your feedback on our response.

Generating the trajectory might be non-trivial and can become a bottleneck.

Thank you for the valuable feedback. We acknowledge that trajectory sketches are not as easy to acquire as language instructions, but also note that we show that trajectory-conditioned policies are easier to prompt for generalizing to new robotic tasks compared to language-conditioning. We believe that as foundation models (e.g., image-generating VLMs) rapidly improve, we expect that their trajectory sketch generating capabilities will improve naturally in the future and be usable by RT-Trajectory.

“More analysis of the robustness of the trajectory specification could strengthen the paper.”

Thank you for the valuable suggestion. We have conducted additional experiments to test the robustness of our trajectory specification by adding different types of noise to human-drawn trajectory sketches: 1) adding a global offset (5 pixels) to the entire 2D trajectory, 2) adding independently sampled local offsets (5 pixels) to individual interaction markers. For each noise type, we ran a total of 22 trials on 3 skills: Place Fruit, Fold Towel, Move within Drawer. The results are shown in the table below, where we find that RT-Trajectory is robust to mild noise.

“Limited to a fixed-view camera with calibration”

We agree with this limitation of our current implementation. Nonetheless, we are optimistic that this limitation can potentially be addressed by training on cross-embodiment data with different camera configurations, which we leave to future work.

“Fig 8 is confusing”

Thank you for pointing out the unclarity of Fig. 8. We have updated Fig. 8 in the revision, by separating it into multiple figures and improving captions for more clarity.

The left side of Fig. 8 shows the 10 most similar trajectories in the training data. Note that we also show additional visualizations from other viewpoints in Fig. 13 (new), where the most similar training trajectories might look more different from query ones.

We find that the trajectories of some unseen tasks can be surprisingly similar to those of seen tasks. Thus, we analyze statistics of the most similar training samples over a set of queries for unseen skills on the right side of Fig. 8. In Fig. 12 (new), query trajectories of unseen skills in general see larger Frechet distances to the most similar training trajectories, compared to query trajectories from training skills. It indicates that many proposed unseen tasks show significant differences from seen tasks w.r.t. motion.

Besides, in Fig. 10 (new), we show the distribution of semantics skills of the most similar training trajectories to rollout trajectories. The unseen task placing fruit into bowl is similar to a seen task move X near Y w.r.t. motion; however, their semantics are quite different, which explains why language-conditioned policy struggles to generalize in this case. Moreover, instead of “extrapolating”, we expect RT-Trajectory to stitch seen motion segments to tackle new tasks. As a good example, RT-Trajectory manages to combine motions from Open top drawer and Place object into top drawer to tackle Move within Drawer.

Furthermore, we would like to point out that some criteria are missing in the current version of our proposed motion similarity metric, e.g., the end-effector orientation and when the gripper is closed or opened. For example, RT-Trajectory can respect the trajectory conditioning for Swivel Chair and does not close the gripper during execution, while its most similar training trajectories all require at least one gripper interaction. Such a difference is not captured yet. And we leave it to future work.

Thank you for the appreciation of our novelty and the helpful comments. We are glad to hear your feedback on our response.

“Does the smoothness or other properties of the trajectory affect the quality of the task conditioning?”

In this work, we experimented with trajectory sketches from 4 different sources: human drawings (Fig. 4 and 13), human demonstration videos (Fig. 6), prompting LLMs with Code as Policies (Fig. 16 and 17), and text-guided image generation models (Fig. 7 and 14). These sources naturally result in trajectories which show different levels of smoothness. Trajectories from human drawings tend to be smooth, while ones from human demonstration videos are usually rough due to noise from the hand pose estimator. Trajectories from prompting LLMs are usually stiff since they are created by connecting sparse waypoints. Trajectories predicted by the text-guided image generation model are the most noisy and fuzzy. A qualitative comparison between them is updated in new Fig. 21. RT-Trajectory is able to condition on trajectories generated by distinct methods with different levels of smoothness.

We compare the average curvatures of the trajectories generated by different approaches for the skill Pick, shown in the table below. To compute the curvatures of a (2D) trajectory, we first reparameterize it by an arc length of 5 pixels, compute the tangent vector at each point after arc-length reparameterization, and derive the curvature as the norm of the derivative of the tangent vector. We average curvatures over 18 trajectories generated by human drawings, human demo videos and prompting LLMs as well as 36 trajectory labels sampled from training data. We do not compute curvatures for the trajectories generated by image generation models, since their outputs are images directly rather than parameterized curves.

Related work: diagrammatic teaching

Thank you for pointing out this related work. We have updated the reference in the revision. Diagrammatic teaching features reconstructing 3D motion trajectories from multi-view user sketches while RT-Trajectory focuses on learning to condition on 2D trajectory sketches. We see many interesting future directions to combine diagrammatic teaching and RT-Trajectory, e.g., extending Ray-tracing Probabilistic Trajectory Learning (RPTL) in diagrammatic teaching to handle user-specified interaction markers, and conditioning RT-Trajectory on trajectory sketches generated by RPTL.