Sketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketches

Clarifications

R5: ” Sec 4.1 - is there any quantitative evaluation as well, or is it all qualitative?” “What if the two human sketches are inconsistent? It might be tricky to ensure that they are consistent.”

While we don't have quantitative comparisons of generator performance under different sketch variations, our design focuses on building in robustness through two complementary approaches. During training, we apply specific augmentation strategies:

• First, we employ image augmentations that only update the VAE component, including random rotation (±25°), random translation (±20% of image size), scaling variations (0.8-1.5x), and Gaussian noise (σ = 0.01). This helps the model learn robust sketch representations.
• Second, we apply controlled elastic deformations to both sketches and trajectories when training the full generator. The sketches undergo elastic deformation with alpha=50.0 and sigma=5.0, while trajectories are perturbed with Gaussian noise (σ = 0.015) before refitting and resampling to ensure smoothness. This second strategy specifically addresses potential inconsistencies in hand-drawn sketches.

These carefully designed augmentations simulate the natural variations in human-drawn sketches. The effectiveness of this approach is demonstrated qualitatively through our latent space interpolation results (Figure 12), where we show the generator produces smooth, coherent trajectories even between quite different sketch pairs.

Minor Points

We have addressed the typographical error on line 71 and other minor issues throughout the document.

[1] Seo, Y., Hafner, D., Liu, H., Liu, F., James, S., Lee, K., & Abbeel, P. (2023, March). Masked world models for visual control. In Conference on Robot Learning (pp. 1332-1344). PMLR.

[2] Mandlekar, Ajay, et al. "What matters in learning from offline human demonstrations for robot manipulation." arXiv preprint arXiv:2108.03298 (2021).

Generator Component Analysis and Design Choice

R5: "how well does the sketch-to-3D generator generalize to new settings? Can it be applied to new tasks or new domains outside of the training data? Is the architecture choice critical?”

The generalization capability of our sketch-to-3D generator is demonstrated through several experiments. In the latent space interpolation results (Figure 12), we show smooth transitions between different trajectories, indicating the model learns a continuous and meaningful representation that can generalize to unseen trajectories. Our generator is trained on randomly generated trajectories in simulation, and we demonstrate it works effectively across all the diverse tasks presented in the paper.

Regarding architecture choices, our experiments with various models led us to the VAE-based design for several reasons. The VAE's latent space provides natural support for generating multiple plausible trajectories from a single sketch pair by adding controlled noise - a feature that proved valuable for policy learning. We tested alternatives including standard CNNs and MLPs, but found they either struggled with trajectory consistency across views or lacked the ability to generate diverse yet plausible trajectories.

Real-world BC vs RL performance

R5: "in the real-world experiment, if BC already achieves 80%, what is the point of RL finetuning, if it achieves the same performance?"

While Behavior Cloning does indeed show high initial success, reaching up to an 80% success rate in our tests, the real advantage of RL fine-tuning comes into play under varied environmental conditions. RL fine-tuning is designed to enhance adaptability and robustness, qualities that are essential in practical deployments where environments cannot be perfectly controlled. This adaptability ensures that the robot can handle unexpected variations effectively, something BC might not consistently manage.

Efficiency and Justification for Data Use in Training the Generator

R5: “Furthermore, training the sketch generator seemed to require 85 teleoperated trajectories… Training the 3D generator and then running RL seems much more painful …”

Initially, training our sketch-to-3D trajectory generator with 85 teleoperated trajectories might seem resource-intensive. However, this should be seen as a strategic, one-time investment that substantially enriches our training dataset, enabling the generator to produce versatile and adaptive responses without needing new data for each task.

There are several ways to make this initial data collection even more efficient:

• The data can be collected in simulation with matching camera setups, requiring no physical robot time;
• We can utilize existing trajectory datasets by transforming them to match our workspace coordinates;
• The collected data can be augmented to create a richer training set.

Once trained, the generator serves as a powerful tool that translates simple sketches into useful demonstrations, eliminating the need for specialized hardware or expertise in data collection. This approach reduces the long-term burden of demonstration collection, increasing the scalability and efficiency of our deployment.

Application Across Multiple Real-World Experiments: The utility and efficiency of our data use are exemplified by the ability to apply the trained generator to various tasks without retraining. For instance, once our generator was fine-tuned during the initial setup, it was adeptly applied to tasks like Toaster Button Press and Bread Pick Place (as shown in figure 16 and 17) in real-world environments with consistent camera setups. This adaptability underscores the practical value of our initial data investment and highlights the framework’s capability to generalize across different tasks effectively.

We sincerely thank the reviewer for their thoughtful feedback and recognition of the paper's clear writing and successful demonstration on a common benchmark. We address each concern below. (Additional experiments and evaluation are reported in the appendix in the new version of the paper and marked in blue color).

Practicality of Approach

R5: "The approach depends on a trained sketch-to-3D trajectory generator... the burden of collecting this dataset would outweigh the benefits..."

The initial data collection is actually quite efficient. We use "play data" that only needs to be collected once during robot setup. This data can come from:

• Direct hardware interaction in a few hours;
• Simulation, which requires no physical robot time;
• Existing recorded trajectory datasets transformed to match the workspace coordinates;
• A mix of these approaches.

Once trained, the generator works across new tasks without retraining. This one-time investment is much more efficient than collecting new teleoperated demonstrations for every task. Our experiments demonstrate that the pretrained generator, trained on 24k trajectories, is sufficiently robust to support sketch-based learning across a variety of tasks.

R5: "The current method can only be used for position-controlled tasks..."

It's true that our sketches show just the path, but that doesn't mean our method can only be used for position-controlled tasks. These sketch trajectories act as a starting point, a way to guide exploration, while the RL component learns the complete task including orientation and gripper controls. We demonstrate this with additional experiments in RoboMimic’s Can Pick and Place task, our system successfully learns proper grasping angles and gripper actuation despite sketches only showing the reaching and placing paths. This showcases a key strength of our approach - we can bootstrap complex learning from simple 2D sketches while letting the RL discover the full spectrum of required controls. Think of the sketches as giving the robot a rough idea of "where to go," while letting it figure out the details of "how to get there" through interaction.

Method Novelty

R5: "The method itself seems limited in novelty... the discriminator itself seems to be unneeded..."

While we build on existing components like IBRL, our key contribution is the novel integration of sketches into RL. This is fundamentally different from prior work that only used sketches for policy conditioning. We demonstrate that even approximate sketch-based demonstrations can effectively bootstrap RL. Regarding the discriminator - our new experiments on more complex tasks like Assembly (MetaWorld, Figure 18) show clear benefits: the success rate improves from 0.2 to 0.85 with the discriminator. Even in simpler tasks, the discriminator helps stabilize learning.

Expanded Task Complexity

R5: "The tasks presented in the paper appear to be limited in complexity..."

To demonstrate our method's capabilities on more challenging tasks, we conducted extensive experiments on complex manipulation benchmarks:

• MetaWorld Assembly Task: A two-stage task, classified as “hard” [1], where the robot first locates and reaches a peg, then picks it up and assembles it with a cylinder. In this task, our framework achieved a success rate of 93%, demonstrating its capability in handling precise manipulations under complex conditions.
• RoboMimic Can Pick and Place: Another two-stage task, where the robot must first locate and reach a can, then pick it up and place it in a designated bin. Robomimic [2] is well established benchmark with demonstrations collected by human teleportation. The policies require predicting the orientation of the gripper as well and are longer than in MetaWorld. This task tested our policy's ability to predict the orientation of the gripper, with a success rate improving to 98% after refining our reinforcement learning approach.
• Real-World Tasks: We tested our framework on two additional tasks, Toaster Button Press (single-stage) and Bread Pick Place (two-stage). These real-world environments included added clutter to simulate more realistic settings. The success rates here were 80% within 30k interactions steps and 95% within 12k interactions steps, respectively, showcasing the framework's adaptability to diverse and unstructured environments.

We sincerely thank the reviewer for their thoughtful feedback and recognition of the strengths in our work, including its novel approach, logical design, and practical applicability. Below, we address the concerns and provide clarifications, incorporating additional results to strengthen our contributions. (Additional experiments and evaluation are reported in the appendix in the new version of the paper and marked in blue color).

Discriminator Contribution:

R3: "In Fig. 6, performance with and without discriminator seem fairly close..."

We agree the discriminator's impact varies across tasks. While the overall performance appears similar in Figure 6, the discriminator's value becomes clear when examining different task complexities. We conducted additional experiments on more challenging environments to demonstrate the benefit of using discriminator - specifically Assembly (a MetaWorld [1] hard task, Figure 18) and Pick and Place Can (from RoboMimic [2], which offers significantly higher difficulty than MetaWorld, Figure 15). In these complex manipulation tasks, as well as in ButtonPressTopdownWall (Figure 5, 6), the discriminator significantly improves performance by helping the policy better align with sketch-generated trajectories. For instance, in ButtonPressTopdownWall, it improves final success rate from 0.4 to 0.75.

For simpler tasks like ButtonPress and Reach, both versions achieve similar final performance (~0.95 success rate). This makes sense - in straightforward tasks, standard exploration strategies already work well. However, we kept the discriminator as a standard component since it ensures reliable performance across all scenarios, especially those requiring precise control, without harming performance in simpler cases.

Number of Generated Demonstrations per Sketch:

R3: "It is not clear that more demos per sketch is necessarily better... In fact, in Fig. 14, 10 demos + discriminator seems worse than others.”

You raise an excellent point about the varying impact of demonstration numbers across different tasks. Looking at Figures 8 and 14 more closely:

For simple tasks like ButtonPress, Reach and ReachWall, we observe that:

• Even with slightly worse performance using 10 demos, the difference is minimal
• 1-3 demos per sketch are sufficient for these tasks, as they involve straightforward point-to-point movements
• Additional demonstrations provide little benefit since basic exploration can effectively solve these tasks

However, for more complex tasks, like BoxClose and ButtonPressTopdownWall:

• 10 demos show clear benefits in reducing performance variability and improving stability
• These tasks require more precise control and have more failure modes, making the broader coverage from additional demonstrations valuable
• The extra demonstrations help create a more comprehensive experience dataset that better captures task complexity

This task-dependent pattern suggests that while simpler tasks can be effectively learned from fewer demonstrations, more complex manipulation tasks benefit from the richer training signal provided by additional demonstrations generated per sketch. This is one of the strengths of our approach. We can generate more demos but from same sketches, essentially not increasing the effort on the part of the user

We sincerely appreciate the reviewer's thorough assessment and recognition of the novelty in our approach. We are grateful for the insightful questions and suggestions, which have helped us strengthen our work. Below, we address each point and provide additional information to clarify the concerns raised. (Additional experiments are reported in the appendix in the updated paper, marked in blue).

Trajectory Generator Robustness to Spatial Correlation Variations

R4: "It's not quantitatively clear how robust the trajectory generator is to variations in the spatial correlation between these sketches."

While we don't have quantitative comparisons of generator performance under different sketch variations, our design focuses on building in robustness through two complementary approaches. During training, we apply specific augmentation strategies:

• First, we employ image augmentations that only update the VAE component, including random rotation (±25°), random translation (±20% of image size), scaling variations (0.8-1.5x), and Gaussian noise (σ = 0.01). This helps the model learn robust sketch representations.
• Second, we apply controlled elastic deformations to both sketches and trajectories when training the full generator. The sketches undergo elastic deformation with alpha=50.0 and sigma=5.0, while trajectories are perturbed with Gaussian noise (σ = 0.015) before refitting and resampling to ensure smoothness. This second strategy specifically addresses potential inconsistencies in hand-drawn sketches.

These carefully designed augmentations simulate the natural variations in human-drawn sketches. The effectiveness of this approach is demonstrated qualitatively through our latent space interpolation results (Figure 12), where we show the generator produces smooth, coherent trajectories even between quite different sketch pairs.

Number of Hand-Drawn Sketches Collected

R4: "In line 369, how many hand-drawn sketches do you collect? Is it also three (i.e. number of expert tele-operated policies used)? "

Yes, for each task, we collected three hand-drawn sketches, corresponding to the number of expert tele-operated demonstrations used in our experiments. We have clarified this in the main text (line 369) to ensure consistency and avoid confusion.

Reward Weighting in Discriminator Ablation

R4: "The choice of reward weights seems arbitrary. A more logical progression (e.g., 0.1, 0.01, 0.001) would make the analysis clearer."

Thank you for your feedback regarding the choice of reward weights in our ablation study. In response to your comment, we selected reward weights (0.1, 0.5, 0.05, 0.005) for our experiments on the MetaWorld task 'CoffeePush', which are presented in Figure 9 of our paper.

Future Work on Time Parameterization

R4: "The authors briefly mention "time parameterization of the trajectory" in future work (line 529).This raises the question of how they envision solving the increased difficulty in achieving correlation between two sketches when time is factored in."

We envision solving time correlation challenges by incorporating visual cues like color gradients in sketches to represent temporal information.

Relation to Vision-Language Agents (VLAs)

R4: "The paper would benefit from a discussion situating this method within the broader context of recently popular Vision-Language Agents (VLAs) for robotic manipulation."

Thank you for this insightful suggestion about positioning our work in the context of VLAs. We see sketches as complementary to the language-based instruction paradigm of VLAs in several ways:

• While VLAs excel at understanding high-level task descriptions, sketches provide intuitive spatial guidance that can be challenging to convey through language alone;
• Sketches could serve as an additional modality for VLAs, particularly for specifying precise trajectories or motion patterns that are difficult to describe verbally;
• Our sketch-based demonstrations could potentially enhance VLA pre-training by providing additional structure to exploration.

Regarding the choice of RL versus pure imitation learning in this context, RL offers key advantages:

• It allows the policy to discover better solutions than the sketch-generated demonstrations while using them as a starting point;
• The policy can adapt to variations not captured in the sketches, like precise object positions or orientations;
• The discriminator-guided exploration helps maintain consistency with human intent while optimizing for task success.

Minor Points

• We have corrected the typo on line 071.
• To improve accessibility, we have updated the figures to be more color-blind friendly. We have changed the colors of the Figure 5 and 6, and incorporated textures in the bar plots of Figure 7.
• The asterisk after "Replay Buffer" in Fig. 3 indicates that the buffer is initialized with the open-loop servoing demonstrations. We have added a clarifying note in the figure caption.

We sincerely appreciate the reviewer's positive feedback on the clarity of our paper and the thoroughness of our analysis within the chosen manipulation tasks. We also thank the reviewer for raising important concerns about the scalability and complexity of our approach. Below, we address these concerns and provide additional insights and results to strengthen our contributions. (Additional experiments and evaluation are reported in the appendix in the new version of the paper and marked in blue color).

1. Role and Scope of Sketch-Based Demonstrations

Our framework is designed to use sketch-based demonstrations as an accessible and low-cost method to initialize policy learning, particularly in scenarios where teleoperation is infeasible or prohibitively expensive. We do not position sketches as a complete replacement for high-fidelity demonstrations but rather as a practical supplement in data-scarce or constrained environments. This distinction underscores the intended utility of our approach in complementing existing methods.

2. Addressing Task Complexity with Expanded Experiments

R1: "I fear that the sketch-based approach presented here is simply too limited and will quickly become overly cumbersome as the task difficulty increases."

We acknowledge the reviewer's valid concern about the scalability of our method to more complex, real-world manipulation tasks. To address this, we have expanded our experimental evaluation to include more challenging and dynamic scenarios:

• Additional Real-world Experiments: Toaster Button Pressing and Bread Pick-and-Place. These tasks involve interacting with cluttered, real-world environments, requiring precise planning and execution. The results, documented in Figures 15 and 16, illustrate the scalability of our method beyond basic scenarios.
• Benchmarks with Long-Horizon, Multi-Stage Manipulations: We conducted experiments on challenging tasks such as Pick and Place Can (from RoboMimic) and Nut Assembly (from MetaWorld). These tasks require sequential reasoning and adaptation, highlighting our framework’s applicability to longer and more intricate task horizons. Detailed results and comparisons are included in the appendix.

Through these expanded experiments, our approach demonstrates robustness and adaptability, addressing concerns about its scalability to real-world applications.

3. Addressing Challenges of Time-Dependent and Orientation-Specific Tasks

R2: "Even providing a time-dependent orientation for the robot's end effector would likely be non-trivial through the pair of 2d sketches used as input for the system that is proposed in this paper.”

We acknowledge the reviewer’s concerns regarding the difficulty of encoding time-dependent or orientation-specific information in sketches. While these challenges are not fully addressed in the current scope, we view them as natural extensions of our approach. For instance:

• Incorporating visual markers or gradients within sketches to encode temporal and orientation-specific information, as inspired by research like the RT Trajectory paper, is a promising direction.
• In tasks like the RoboMimic Pick and Place Can, the sketch-based demonstrations provide initial position cues, while our framework leverages reinforcement learning (RL) to refine the policies, dynamically adjusting for orientation and time-specific factors during execution. This approach does not require fully detailed trajectory information, as the sketches primarily guide exploration during RL. Notably, our results are comparable to IBRL, which relies on human demonstrations with explicit orientation details (Figure 15). This highlights the effectiveness and robustness of our sketch-based method in handling orientation and other task complexities without needing fully specified inputs.

Thank you for your feedback. We agree with you that there are certain tasks where teleoperation is better suited for giving demonstrations. However, as evidenced by a number of recent papers investigating the use of sketches in robot learning, sketches are useful in practice. [1-5] The goal of sketches is to complement, rather than replace, high-fidelity demonstration techniques such as teleoperation. Sketches make robot learning more accessible and when high precision is not paramount or where teleoperation is costly or not feasible. As shown in prior research, even basic sketches can effectively initiate and guide the learning process for a variety of tasks (refer to [1-5]).

Our approach using sketches is part of an evolving exploration within the robotics community. Sketch-based methods have been previously studied as high-level guidance for initiating robot learning, notably in tasks where exact precision is not the primary requirement. Our work builds on this foundation and extends its application to reinforcement learning (RL), showcasing how sketches can effectively guide policy development even without detailing every aspect of a task's execution.

For example, in the RoboMimic Can Pick and Place task, our framework successfully learned necessary orientation adjustments through RL interactions, despite these details not being explicitly depicted in the sketches.

Looking ahead, there is significant potential to enhance the utility of sketches in robotic instruction. Incorporating elements such as orientation markers or color-coded actions within sketches could provide more detailed control cues, broadening the scope of tasks to which this method can be effectively applied.

[1] Zhi, Weiming, Tianyi Zhang, and Matthew Johnson-Roberson. "Instructing robots by sketching: Learning from demonstration via probabilistic diagrammatic teaching." 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.

[2] Gu, Jiayuan, et al. "RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches." The Twelfth International Conference on Learning Representations.

[3] Sakamoto, Daisuke, et al. "Sketch and run: a stroke-based interface for home robots." Proceedings of the SIGCHI conference on human factors in computing systems. 2009.

[4] Ahmad, Haseeb, et al. "A sketch is worth a thousand navigational instructions." Autonomous Robots 45.2 (2021): 313-333.

[5] Skubic, Marjorie, et al. "Using a hand-drawn sketch to control a team of robots." Autonomous Robots 22 (2007): 399-410.

We sincerely thank the reviewer for their thoughtful feedback and recognition of SKETCH-TO-SKILL's innovative approach and potential for making robot learning more accessible through simple sketches. Below, we address the concerns about practical applicability and experimental validation, incorporating additional details and clarifications to strengthen our contributions. (Additional experiments and evaluation are reported in the appendix in the new version of the paper and marked in blue color).

Real-World Performance & Complex Task Handling:

R1: "The examples shown focus on short, simple trajectories... experiments are carried out in simplified environments without much complexity or occlusion... real-world tasks demonstrated are quite basic..."

We have rigorously evaluated our framework's capability with complex trajectories and realistic environments through extensive additional experiments:

• MetaWorld Assembly Task: A two-stage task, classified as “hard” [1], where the robot first locates and reaches a peg, then picks it up and assembles it with a cylinder. In this task, our framework achieved a success rate of 93%, demonstrating its capability in handling precise manipulations under complex conditions.
• RoboMimic Can Pick and Place: Another two-stage task, where the robot must first locate and reach a can, then pick it up and place it in a designated bin. Robomimic [2] is well established benchmark with demonstrations collected by human teleportation. The policies require predicting the orientation of the gripper as well and are longer than in MetaWorld. This task tested our policy's ability to predict the orientation of the gripper, with a success rate improving to 98% after refining our reinforcement learning approach.
• Real-World Tasks: We tested our framework on two additional tasks, Toaster Button Press (single-stage) and Bread Pick Place (two-stage). These real-world environments included added clutter to simulate more realistic settings. The success rates here were 80% within 30k interactions steps and 95% within 12k interactions steps, respectively, showcasing the framework's adaptability to diverse and unstructured environments.

R1: "How does trajectory complexity affect the learning cost... How feasible is it for users to sketch accurately for more complex, multi-step tasks?"

Sketch-to-Skill naturally extends to multi-step tasks through segmented sketching, following the common practice of decomposing complex tasks into simpler steps. As shown in Figures 15, 16, and 18 (Sections E1, E2, and E3), this approach successfully completes complex manipulation sequences including assembly and pick-and-place tasks, demonstrating the same effectiveness as single-step scenarios. For instance, in the MetaWorld Assembly task, our sketch-based approach achieved a success rate of 93%, closely matching the performance of baseline methods which generally average around 95% in similar tasks.

R1: "How sensitive is the model to changes in the scene, such as additional objects or occlusions?"

To address performance in realistic settings, we conducted thorough testing in cluttered environments. We added various daily objects and occlusions in the Toaster Button Press and Bread Pick Place tasks, as shown in Figures 16 and 17. Results demonstrate consistent performance despite environmental complexity.

These comprehensive experiments demonstrate that our framework effectively handles complex, multi-step tasks and performs robustly in realistic, cluttered environments, supporting its practical applicability in real-world robotics applications.

Discriminator Contribution

R1: "The discriminator doesn't appear to contribute significantly... unclear how essential it is or if it's been optimized effectively for generalization"

We conducted additional experiments to explicitly demonstrate the discriminator's value. In MetaWorld tasks (Figures 5 and 6), we tested 'sketch-to-skill no-BC' scenarios to isolate the discriminator's impact. The results show that 'sketch-to-skill no-BC' consistently performs better than pure RL, demonstrating the discriminator's effectiveness in guiding exploration even without behavior cloning. We further validated this in more challenging scenarios - in Pick and Place Can and Assembly tasks (Figures 15 and 18), where baseline BC scores with sketches were initially low (0.1 and 0.05), incorporating the discriminator loss markedly improved performance. While simpler tasks work well with standard exploration, these results show the discriminator becomes particularly valuable as task complexity increases.

Thank you for the suggestion. We've conducted some additional experiments to address the concerns about overlapping and complex trajectories. (Additional experiments and evaluation are reported in the appendix in the new version of the paper and marked in red color).

R1: “ it remains unclear whether the color gradients are consistently interpretable in more complex trajectories…”

We implemented the notion of color gradients for time-parameterization of the trajectory in the sketches and conducted experiments with overlapping trajectories. These are reported in Appendix E5. This notion of color gradient in sketches was introduced by RT-Trajectory [1]. An example is shown in Figure 19. Here, the color of the sketch changes from green (start) to red (end). This is particularly helpful when the sketch crosses each other or overlaps. We conduct additional experiments with the generator to evaluate the effect of incorporating gradients in the Sketch-To-3D trajectory generator.

Specifically, we use the MetaWorld Assembly task (Figure 18) where the sketch overlaps in the middle as the end-effector picks up the tool and carries it to the goal position. We trained two generators without and with color gradients. The performance of these two generators are reported in Table 3. We observe that incorporating the gradient in such a case results in lower reconstruction and trajectory losses. With lower trajectory losses we can handle more complex trajectories that overlap with color gradients. Note that incorporating gradients also does not require any change to the downstream architecture, and only requires minimal changes to the generator architecture.

R1: ”regarding … partial occlusions...reliance on human inference are reasonable ideas…”

Regarding occlusions: this is part of the reason why we use two camera views. This reduces the risk of both views with significant occlusions, and allows the human to still be able to draw the sketches. In general, we do not need the sketches to be perfect or to include all the finer details. Our key contribution is to show that even with coarse sketches (which may happen with a clutter/occlusions in the environment), we can still better explore in RL with BC bootstrapping and discriminator-based reward shaping. In fact, in some of the real-world experiments, we do observe occlusions in one of the views without affecting performance.

R1: “The role of the discriminator is supported by experimental results, particularly in complex tasks, but its generalization to unseen tasks…”

The discriminator in our framework, inspired by standard imitation learning. It evaluates whether a transition resembles those from expert trajectories. Originally, discriminators[2,3] have been used to evaluate transitions against actual demonstrations from experts. Our approach innovates by applying the discriminator to transitions from sketch-generated trajectories, leveraging simplified inputs while maintaining critical evaluation.

Training a discriminator typically requires extensive data collection. We address this by generating diverse trajectories from each sketch, reducing the need for numerous demonstrations and improving learning efficiency. Our generator remains task-agnostic, showing generalization across tasks without retraining. The experiments presented in the paper involve tasks unseen during the generator's training.

R1: "For tasks involving gripper rotation...Does the framework rely on predefined templates …"

The framework doesn’t rely on predefined templates or rules to predict rotations. This is the key contribution of our work - we do not need the sketches to be perfect or to include all the finer details. The bootstrapping with BC policy and the discriminator reward shaping act as guidance during exploration in RL improving the learning performance.The rotations are learned from RL interactions with the environment; we obtain reward feedback from the environment to ensure the alignment with task objective.

For example, in the RoboMimic’s Can Pick and Place task, our system successfully learns the necessary grasping angles. This is achieved despite the sketches only providing basic paths for reaching and placing. This demonstrates a key advantage of our approach: we can bootstrap complex task learning from simple 2D sketches, allowing the RL to uncover and master the finer details necessary for task completion. Essentially, the sketches give the robot a rough map of "where to go," while the RL determines "how to get there," refining the approach through active interaction and learning.

R1: "While the experiments …they do not address the framework's adaptability to dynamic changes or higher complexity, such as moving objects”

This is a good point. Our current work is focused on static scenarios. Handling dynamic environments is very interesting but beyond the scope of this particular work. Our primary aim is to show that in many cases where we use teleoperated demonstrations, we can use sketches to simplify and expedite the learning process. Addressing dynamic changes involves additional complexities which are interesting but would constitute a separate line of research.

[1] Gu, Jiayuan, et al. "Rt-trajectory: Robotic task generalization via hindsight trajectory sketches." arXiv preprint arXiv:2311.01977 (2023). [2] Zhang, Wenjia, et al. "Discriminator-guided model-based offline imitation learning." Conference on Robot Learning. PMLR, 2023. [3] Xu, Haoran, et al. "Discriminator-weighted offline imitation learning from suboptimal demonstrations." International Conference on Machine Learning. PMLR, 2022.