We thank the reviewer for the constructive feedback. Below, we address each of the key points raised.

Coarseness of Scaffolds

Note that scaffolds are only intended to be used as a reward signal and to guide exploration – the policy is free to learn whatever it wants on top of the reward signal and thus is not strictly “imitating”. We acknowledge that our scaffold-based reward signals are coarser than those derived from human demonstrations or hand-crafted dense rewards. However, this is a deliberate design choice, balancing scalability and supervision quality. Human demonstrations are often costly, time-intensive, and difficult to scale – especially as new tasks require collecting entirely new datasets. Similarly, crafting dense, high-quality reward functions demands significant manual engineering and domain expertise.

In contrast, our scaffolds provide lightweight, task-agnostic, semantically meaningful supervision with minimal human input. While inherently less precise, we show that these signals are sufficiently informative to guide learning across a wide variety of tasks.

We see scaffolds as a practical middle ground: they are significantly more scalable than demonstrations or hand-tuned rewards, while still effective across diverse domains. Future work may explore hybrid approaches – e.g., combining scaffold-based rewards with sparse success signals or contact-based feedback – to enable finer control when needed.

Lack of Explicit Orientation Information

Though each keypoint does not encode orientation, multiple keypoints together do implicitly capture orientation when at least three non-colinear keypoints are annotated – an approach that is feasible for many real-world objects. In this way, motion scaffolds can effectively be used for tasks requiring rotation.

To demonstrate this capability, we modified the original bottle task: the bottle may now start either lying down or standing upright on one side of a sink and must be placed upright on the opposite side. This setup requires the policy to infer the initial pose and adjust its behavior accordingly. We report a success rate of 0.64 ± 0.1 across three seeds. By using multiple spatial keypoints together, the policy learns to rotate and upright a bottle. Unfortunately, due to the limitations of rebuttal this year we are unable to attach images or videos.

We acknowledge that for smaller objects, where one or two keypoints may be insufficient to disambiguate orientation, this becomes more challenging. In such cases, richer scaffold representations – potentially powered by advances in spatially grounded vision-language models (VLMs) – represent a promising direction to improve expressiveness and generality.

Physical Realism and Real-World Complexity

We appreciate the reviewer’s concern regarding the complexity and physical realism of our tasks. To address this, we have extended our real-world evaluation to include a significantly more demanding hammering task, in which the robot must grasp a hammer by the handle and strike a table three times. This task demands substantially more precision and physical realism than pick-and-place or pushing tasks. Due to NeurIPS guidelines, we are unable to provide additional visual materials (e.g., images or videos) at this stage.

Despite the increased difficulty, our system achieves a 60% success rate over 20 rollouts. Importantly, most failure cases arise not from deficiencies in the learning framework or simulation fidelity, but from hardware-related constraints, such as the mechanical limitations of the Allegro Hand [LEAP Hand, Shaw et al RSS 23] and noise in depth sensing [Learning to estimate 3d states of deformable objects from single-frame occluded point clouds, Lv et al 23].

We acknowledge that sim-to-real transfer remains an open and well-recognized challenge, particularly for tasks requiring high-precision contact and control [ObjDex Chen et al. 24, RoboDexVLM Liu et al. 25, Solving a Rubrik’s Cube, OpenAI et al. 19]. As part of ongoing work, we plan to investigate improvements in simulation realism, multi-view visual perception, and sensor fusion – directions we believe are essential for narrowing the sim-to-real gap.

Generalization to Unseen Scenes and Objects

We thank the reviewer for highlighting the importance of generalization. Our approach is designed to promote generalization by leveraging VLM-generated scaffolds and a policy conditioned on keypoint observations and proprioception, rather than raw visual input. This decouples the learning process from pixel-level variability and allows for greater robustness.

Empirically, we observe strong zero-shot generalization in real-world settings. For example, in the bottle pick-and-place task, we successfully replaced the bottle with cups and mugs, maintaining high success rates by modifying only a single word in the prompt. Additionally, the system demonstrates robustness to changes in lighting conditions, background clutter, and instruction phrasing, with no significant degradation in performance across these variations.

That said, we acknowledge that generalization to entirely new task categories or structurally different objects remains a significant challenge. In practice, doing so has required training policies across an extremely broad set of objects [DextrAH, Singh et al 24], or learning object agnostic representations. We see this as an important and exciting direction for future work in combination with our proposed motion scaffolds.

We thank the reviewer for their thoughtful and constructive feedback. Below, we address the key points regarding the novelty of our components, prompt design, and experimental baselines.

Novelty of Individual Components

We respectfully disagree with the assessment that the individual components of our framework lack novelty. While our primary contribution lies in the system as a whole – which successfully enables real-world task execution, a recognized contribution in itself – this system is built on several novel and impactful ideas.

To the best of our knowledge, no prior work leverages VLMs to generate keypoint trajectories for guiding reinforcement learning, nor uses keypoint sequences as a scaffold for both reward shaping and exploration in RL. Prior works have instead only used keypoints for rewards [IKER Patel et al 25] or used demonstrations for residual RL and tracking [ObjDex Chen et al 24]. Our work represents a significant departure from conventional approaches, which typically rely on these more expensive sources of supervision like motion capture, demonstrations, or scripted heuristics.

We instead use structured, semantically meaningful guidance derived directly from natural language prompts to a VLM. This eliminates the need for demonstrations or complex setup procedures, and substantially lowers the cost and complexity of deploying RL in real-world manipulation tasks.

Prompt Specificity

We appreciate the concern about the risk of prompt specificity introducing unintended task knowledge. In response, we revised our approach to use a single, concise natural language instruction per task (e.g., “grasp the apple and move it onto the cutting board” or “grasp the refrigerator handle and fully open the top refrigerator door”).

To do this, we use a standardized preamble to elicit task-relevant keypoints from the VLM:

"You are an expert in robot manipulation. Above is an image of the environment. Your task is to {task_description} with a robot hand. Identify a minimal set of keypoints needed to plan the motion. Avoid small, sharp, or occluded areas like corners or edges. Choose large, stable regions visible from many angles. Use spatially descriptive names (e.g., "left handle", "top surface") and avoid ambiguous labels like "part 1". Only include points that are essential for completing the task. Reply with a JSON list of keypoint names."

The keypoints generated by the VLM using this prompt are employed both for keypoint detection and for motion trajectory planning. We find that the chosen keypoints closely align with human intuitions about salient features necessary for task completion.

To assess the impact of the new prompting strategy, we evaluated performance across three manipulation tasks and report the mean success rate ± standard error over three seeds.

These results indicate that simplifying and standardizing the prompts preserves task performance while reducing the risk of embedding task-specific priors, thereby improving the generality of our approach.

Baselines

To strengthen the empirical evaluation, we have incorporated four additional baselines. We report success rates as the mean +- standard error computed over three seeds for each task.

Iterative Keypoint Rewards (IKER)

We implement an IKER-style baseline in which a vision-language model (VLM) generates code for reward function parameters, using the exact prompts and methodology from the original paper. However, as IKER assumes a fixed set of keypoints, we use the same VLM-identified keypoints as in our method for parity.

RL From Scratch

We introduce an additional baseline in which a policy is trained from scratch using RL. The reward function mirrors that of our method – combining contacts and keypoint-distance-to-target – but relies on oracle keypoints rather than those detected by a VLM. Importantly, this baseline does not use any trajectory guidance or demonstrations. To ensure a fair comparison, we evaluate it alongside our method using the same oracle keypoints (i.e., Ours + keypoint oracle), but retaining VLM-generated trajectories. Policies are trained using PPO for the same number of timesteps as our method.

Imitation Learning (IL)

To evaluate imitation learning, we train Diffusion Policy using successful rollouts of our oracle RL policy as demonstrations. We conduct experiments using 10, 20, and 50 demonstrations. This represents the best-case performance for imitation learning: the demonstrations are perfect, resulting from scripted trajectories and an RL policy, unlike standard human demonstrations.

Comparison to Other Reward Designs

The reviewer asked about comparisons to other, more complicated reward designs. To shed light on the effectiveness of our reward design, we additionally compare RL from scratch using more detailed, task specific, hand-crafted reward functions. This reward includes contact, keypoint-distance-to-target, hand-distance-to-object, and an additional ground truth success signal, assuming access to a success detector. This baseline is intended to reflect the upper bound of performance achievable through extensive manual reward engineering. As with the previous baseline, it does not utilize any trajectory supervision or demonstrations. To ensure a fair comparison, we evaluate it alongside our method using the same oracle keypoints (Ours + keypoint oracle). Policies are trained using PPO.

I thank the authors for their rebuttal comments.

As stated in the original review, I recognize that the overall framework is novel. With this, I am referring to the idea of using a VLM to generate keypoint trajectories and then adopting a RL-based residual policy for achieving such keypoints. However, it still remains unclear what are the novelties compared to the literature, in terms of machine learning techniques advancement. The VLM is a publicly available pre-trained model. The main difference, compared to other approaches, resides in the prompt. The RL policy is trained to track the keypoints provided by the VLM, while keeping contact with objects. In the literature, keypoint tracking is generally used for imitating mocap data [1,2], while contact rewards are more common in manipulation. Residual RL has also been investigated in the past [3].

I hope the above clarifies why my general feeling is that no component of the framework introduces particular technical novelties. Nonetheless, I still believe that the idea proposed by the authors is valuable, interesting from an engineering perspective, and it makes for a flexible and performant approach.

Other than this, I think the authors have fairly addressed my other concerns:

• the more general prompt improves the applicability of the approach to new scenarios, without compromising performance
• the experiments presented in the rebuttal improve the empirical validation of the approach by comparing with several baselines

I will increase my rating of the paper accordingly.

[1] Learning human behaviors from motion capture by adversarial imitation, Merel et al

[2] CoMic: Complementary Task Learning & Mimicry for Reusable Skills, Hasenclever et al

[3] Residual Reinforcement Learning for Robot Control, Johannink et al

We thank the reviewers for their constructive feedback. Below, we respond to the key points raised.

Use of Robot-Specific Vision-Language Models (VLMs)

We appreciate the suggestion to explore robot-specific VLMs. As part of our investigation, we evaluated Molmo as an alternative to general-purpose models. We have tried detecting keypoints and generating scaffolds for 3 tasks. The table below shows the percentage of correct keypoints and feasible trajectories.

While Molmo reliably identified keypoints, it consistently failed to generate coherent plans and exhibited similar failure modes across multiple tasks and prompts – indicating limited capacity for planning. In contrast, Gemini offers more advanced high-level reasoning, which is central to our framework. Additionally, recent versions of Gemini have also demonstrated strong performance in spatial grounding and pointing, making it particularly well-suited for scaffold generation in our setting. One could continue running experiments to determine the best VLM for each component of the pipeline, but we view doing so as further optimization which would only improve performance.

Impact of VLM Prediction Errors

We agree that the effectiveness of our method is influenced by the accuracy of outputs from the VLM. Errors in keypoint detection or trajectory prediction can indeed degrade downstream RL performance.

To quantify this, we have already included results using oracle keypoint and trajectory predictions – providing an upper bound on performance in the presence of perfect VLM outputs. These are shown in Figure 7 (right) as “Keypoint Oracle” and “Trajectory Oracle.” As discussed in the paper, oracle predictions yield varying degrees of performance improvement across different tasks, and in some cases, enable near-perfect success rates.

As requested, we have added results in the reverse direction -- demonstrating how performance degrades with artificially injected gaussian noise. Specifically, we add Gaussian noise N(0, sigma^2) to the waypoints proposed by the VLM, thereby simulating planning errors. We add noise both during training and inference. We report success rates as the mean +- standard error computed over three random seeds for each task.

Limitations of Representation

Note that motion scaffolds are designed to 1) provide dense rewards for the agent and 2) guide exploration with residual actions. This does not limit the information that the RL policy can learn from – it could use force information or contacts if desired to learn to do the task effectively though we did not as such information is more difficult in the real world. However, there is nothing about our framework that prevents the policy from doing so. While our reward function is based on spatial information, the low-level RL agent can explore different amounts of force, torque, and contact behaviors. For example, we found that the RL agent automatically used stable grasps with high contact forces for pick and place tasks with large objects, while choosing small forces for finer tasks such as closing the pair of scissors.

On the other hand, adding force and torque constraints to the reward function is possible, but doing so requires an extensive amount of manual reward design which may not be broadly applicable. The purpose of our work, however, was to show how much scaffolds reduce the burden of reward design across a broad set of tasks. To this end, while adding more specific information could increase performance in specific cases, it comes at the cost of generality. If a task requires additional terms, they could be added on top of the terms for the motion scaffold and used in combination. We leave this exploration to future work.

Furthermore, the spatial information encoded by our reward design can go really far. To demonstrate this, we modified the original bottle task: the bottle may now start either lying down or standing upright on one side of a sink and must be placed upright on the opposite side. This setup requires the policy to infer the initial pose and adjust its behavior accordingly to rotate the bottle upright – all from using multiple keypoints to infer rotation. Unfortunately, due to the limitations of rebuttal this year we are unable to attach images or videos. We report results across three seeds below:

While the discussion above focused primarily on reward design, we would like to note that there is a plethora of existing literature in constrained RL [Responsive Safety in RL by PID Lagrangian Methods, Stooke et al 2020, Constrained RL with Smoothed Log Barrier Function Zhang et al 24], which is orthogonal to our effort which could be used for actually enforcing constraints instead of “encouraging compliance” with reward penalties.

We thank the reviewers for their constructive feedback. Below, we address their stated concerns.

Baseline Comparisons

To strengthen the empirical evaluation, we have incorporated four additional baselines. We report success rates as the mean +- standard error computed over three random seeds for each task.

Iterative Keypoint Rewards (IKER)

We implement an IKER-style baseline in which a vision-language model (VLM) generates code for reward function parameters, using the exact prompts and methodology from the original paper. However, as IKER assumes a fixed set of keypoints, we use the same VLM-identified keypoints as in our method for parity.

RL From Scratch

We introduce an additional baseline in which a policy is trained from scratch using RL. The reward function mirrors that of our method – combining contact and keypoint-distance-to-target – but relies on oracle keypoints rather than those detected by a VLM. Importantly, this baseline does not use any trajectory guidance or demonstrations. To ensure a fair comparison, we evaluate it alongside our method using the same oracle keypoints (i.e., Ours + keypoint oracle), but retaining VLM-generated trajectories. Policies are trained using PPO.

RL with Complex Reward Design

We also introduce a baseline that employs a more detailed, task-specific, hand-crafted reward function. This reward includes contact, keypoint-distance-to-target, hand-distance-to-object, and an additional success signal, assuming access to a success detector. This baseline is intended to reflect the upper bound of performance achievable through extensive manual reward engineering. As with the previous baseline, it does not utilize any trajectory supervision or demonstrations. To ensure a fair comparison, we evaluate it alongside our method using the same oracle keypoints (Ours + keypoint oracle). Policies are trained using PPO.

Imitation Learning (IL)

To evaluate imitation learning, we train Diffusion Policy using successful rollouts of our oracle RL policy as demonstrations. We conduct experiments using 10, 20, and 50 demonstrations. This represents the best-case performance for imitation learning: the demonstrations are perfect, resulting from scripted trajectories and an RL policy, unlike standard human demonstrations.

We do not compare against RoboDexVLM because their method relies on a library of manually crafted primitive skills, while our method tries to learn those skills implicitly with RL. Those primitives are difficult to scale to unstructured motions such as hammering, or precise skills such as closing a pair of scissors. Because of this, RoboDexVLM sticks to simpler grasping tasks. Additionally, the authors of RoboDexVLM have not released their code, making replication difficult.

Real-to-Sim Transfer

We acknowledge that our approach assumes a real-to-sim transfer pipeline. However, we would like to note that this is a common assumption made by works in dexterous manipulation [ObjDex Chen et al. 24, RoboDexVLM Liu et al. 25, Solving a Rubrik’s Cube, OpenAI et al. 19]. To our knowledge there are only a handful of works that train dexterous RL policies entirely in the real world, which comes with its own challenges (resets etc.) [PDDM, Nagabandi et al. 19]. While we do assume access to a simulated model of the robot, such an assumption is commonplace in several renowned works in dexterous manipulation. We will add more details about our specific sim-to-real pipeline in the final version of the manuscript.

Real-World Task Complexity

We appreciate the suggestion to explore more challenging tasks in the real world. In response, we have added a hammering task, where the robot grasps a hammer by the handle and performs three strikes on a table. This task requires substantially more precision than previous ones (bottle-picking or pushing) and our method achieves a 60% success rate out of 20 trials. Due to the limitations of the rebuttal this year, we are unfortunately unable to upload links or images. To our knowledge, this is the first demonstration of such a task using a multi-fingered robot hand trained via RL.

For more precise tasks, we are currently limited by hardware constraints, including depth sensor resolution [Learning to estimate 3d states of deformable objects from single-frame occluded point clouds, Lv et al 23] and the mechanical tolerances of the Allegro Hand [LEAP Hand, Shaw et al RSS 23] which restrict performance on high-precision manipulation.

Kinematic Reachability

For all real-world experiments, we use a 7-DoF robotic arm in both simulation and the physical setup. This ensures that the policy is trained with kinematic constraints similar to those of the real world. The floating-hand configuration is used only for the simulation-only experiments. In both setups, the policy outputs 3D wrist poses, which are converted to joint angles via the pseudo-inverse method. We ensure that the initial hand pose is kinematically feasible and that all objects are placed within the robot’s reachable workspace.

While VLMs are initially unaware of robot kinematics, our few-shot refinement phase conditions the model on successful prior samples, improving its ability to generate feasible trajectories. Additionally, we introduce substantial noise during plan sampling, encouraging diversity of action proposals such that a significant number of the sampled plans during RL training are kinematically viable.