OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation

We thank the reviewer for the valuable comments. We are happy that you found our human-franka experiments impressive. We address each of the concerns (weaknesses) and questions in the following.

Concern 1: Although the experimental results indicate that incorporating the world model loss is beneficial, it remains unclear why this leads to improvements on unseen tasks. Training a model on a small dataset is prone to overfitting, which would be expected to hinder its generalization capabilities on novel tasks. More detailed qualitative and quantitative analyses of the world model loss should be provided.

Response: Training a model on a small dataset can indeed lead to overfitting if not properly regularized. Most existing one-shot visual imitation methods are trained end-to-end to predict actions, often relying heavily on image semantics while underutilizing the expert demonstrations. These methods typically emphasize features crucial for action prediction alone. While regularization techniques like asymmetric demonstration image mixup, as used in AWDA [5], have shown improvements in generalization by disrupting this correlation, we enhance this further by incorporating the world model loss. This loss $\mathcal{L}_{wm}$ on the future latent states predicted by the world model is independent of the ground truth actions (actions being waypoints in our case). This additional supervision helps the model focus on learning robust representations of environment dynamics, which improves generalization to unseen tasks.

Concern 2: The similarity between training scenarios and novel validation scenarios would significantly influence the performance of one-shot imitation learning systems. However, this lacks a comprehensive analysis of this relationship.

Response: Thank you for the suggestion. We agree that the similarity between training scenarios and novel validation scenarios can impact the performance of one-shot imitation learning systems. During training, the model learns two key components: inferring tasks from the expert and acquiring the skills needed to complete these tasks. Generally, the higher the similarity between the skills required in the training tasks and those in the test tasks, the better the generalization. To explore this further, we conducted two new ablation studies:

• Ablation 1: Button Press Tasks Removed from Training. We removed all training tasks related to button pressing (Button-Press-Topdown, Button-Press-Topdown-Wall, Button-Press-Wall) and evaluated the model on the Button-Press task. The model still achieved a high success rate of 97%, nearly identical to when the button-related tasks were included. This is because button pressing is relatively kinematically less complicated and the required skills were already acquired from other training tasks.
• Ablation 2: Window Tasks Removed from Training. We removed the Window-Close task from training and evaluated on Window-Open. In this case, performance dropped from 95% to 80%. The sliding window tasks here involve more complex physical dynamics. So the absence of related training data led to a noticeable drop in generalization.

We will include these results and a corresponding discussion in the updated manuscript to better clarify this relationship.

Question 1: Why does the world model loss bring benefits to the one-shot imitation learning on novel scenarios? For example, the Human-Franka-PP task only contains 3 training manipulation tasks; the world model would overfit to these tasks, so why would it have better generalization ability for novel tasks?

Response: As detailed in the response to your Concern 1, the world model loss provides an additional regularizing training signal, encouraging the model to focus on general features necessary for predicting the next states of the environment, alongside predicting the waypoints. This impact is more evident on larger datasets like Meta-World (as shown in Fig. 7 of the manuscript) compared to smaller datasets like Human-Franka-PP. Increasing the amount of data would certainly help mitigate overfitting. Nonetheless, OSVI-WM achieves high success rates on the novel tasks within Human-Franka-PP, Human-Franka-Push, and Two-Franka-PP (Table 2 of the manuscript), outperforming competing methods.

Question 2: For dynamic world modeling, various methods exist to reconstruct subsequent frames. Could you provide a comparison with other techniques, like VQ-VAEs and diffusion models, to further assess how well the representation captures dynamic scenes?

Response: VQ-VAE and diffusion models have proven effective for modeling dynamic scenes and have been successfully extended to world models (e.g., [A, B]). Pixel-space world models such as those based on VQ-VAE and diffusion techniques are often computationally intensive to train and deploy. Given the scope and constraints of this rebuttal, we provide a qualitative comparison with these methods based on insights from existing literature. We compare OSVI-WM’s world model against VQ-VAE and diffusion-based world models along several key dimensions:

1. World model representation: As discussed in Section 2 of the manuscript, world models can represent the environment’s state either in pixel space or in a latent embedding space. VQ-VAE and diffusion-based world models usually operate in the pixel space, where future frames are generated or reconstructed directly as images. In contrast, OSVI-WM represents the world in a latent embedding space, optimized specifically for downstream prediction tasks such as waypoint planning. Several recent works ([2, 23, 22, 29, 61]) have shown that latent-space representations are more effective for learning meaningful abstractions, as they avoid modeling low-level pixel details that are often irrelevant for downstream tasks.
2. Dynamic world: World models are generally designed to capture how environments evolve in response to agent actions. However, they differ in how they handle other dynamic elements (e.g., moving obstacles, background agents). Diffusion models can implicitly model such non-agent-dependent dynamics due to their probabilistic nature, though this comes with significant computational cost. OSVI-WM, in its current form, does not explicitly model dynamics unrelated to the agent. However, it can be extended to do so by introducing an additional environment-dependent learnable latent token, enabling the representation of external scene changes.
3. Computational efficiency: OSVI-WM is more efficient, as it bypasses costly pixel-level reconstruction. VQ-VAE and especially diffusion models are, typically, more resource-intensive due to decoder use and iterative generation steps.
4. Planning Efficiency: OSVI-WM’s latent embeddings are directly usable for downstream tasks like planning and control. In fact, some recent works (e.g., [61,C,D]) have shown that learned latent states can be effectively combined with model-based controllers without any further training. In contrast, VQ-VAE and diffusion outputs often require an additional encoder to derive planning-compatible representations, adding overhead.
5. Interpretibility and visualization: VQ-VAE and diffusion models offer more intuitive visualization since their outputs are in pixel space. OSVI-WM requires a separate decoder to visualize latent states, making interpretation less direct.
6. State-Action space: VQ-VAE uses a discrete latent space, while OSVI-WM and diffusion models operate in continuous spaces making them more suitable for continuous control.

We will include a brief discussion of the above in the revised manuscript.

Question 3: How do different types of similarity—such as semantic similarity and visual similarity—between the training dataset and validation tasks affect learning outcomes? Furthermore, the impact of various related tasks within the training dataset on overall performance remains underexplored.

Response: We conducted new ablation studies (as detailed in the response to your Concern 2) showing that skill similarity, rather than purely visual or semantic similarity, has a stronger impact on generalization. When core skills needed for the test task are present in the training set, performance remains high even if the tasks differ visually or semantically. We'll include these insights and results in the updated manuscript.

References

[A] Robine, Jan, Tobias Uelwer, and Stefan Harmeling. "Smaller world models for reinforcement learning." Neural Processing Letters 55, no. 8 (2023): 11397-11427.

[B] Ding, Zihan, Amy Zhang, Yuandong Tian, and Qinqing Zheng. "Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning." arXiv preprint arXiv:2402.03570 (2024).

[C] Bar, Amir, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. "Navigation world models." In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15791-15801. 2025.

[D] Bai, Yutong, Danny Tran, Amir Bar, Yann LeCun, Trevor Darrell, and Jitendra Malik. "Whole-Body Conditioned Egocentric Video Prediction." arXiv preprint arXiv:2506.21552 (2025).

We thank the reviewer for the valuable comments. Some of the weaknesses listed, such as Limited Handling of Dynamic Environments, Redundancy in Latent States, and Limited Theoretical Analysis, are indeed active research areas within world models. OSVI-WM beats all previous results on one-shot visual imitation. Additionally, the modularity of our framework will facilitate integration of future advancements in the above three research areas. Below, we address each of the concerns (weaknesses) and questions in detail.

Concern 1: Lack of End-Effector Orientation Control.

Response: As noted in the limitations paragraph in Section 5 of the manuscript, OSVI-WM currently predicts only 3D positions and gripper states, aligning with existing works like [10,5,32]. While our current tasks do not demand precise orientation control, the framework is designed to be flexible. It will be extended in future work to include orientation control by incorporating end-effector orientation data into the waypoints during training. This adaptability ensures that OSVI-WM can be tailored to more complex tasks like screwing, drilling, or side-grasping.

Concern 2: Limited Handling of Dynamic Environments

Response: The trained world model is generally good at handling object movements directly influenced by the agent's actions, such as a button pressed by a robot (Fig. 9). Even in cases where external agents move objects important to the task, the world model can usually dynamically adapt to such changes appropriately. For other dynamic elements that do not directly affect the agent, like moving people or other robots, probabilistic modeling or optimization over environment-dependent latent variables can be employed. However, there is no widely accepted method for modeling such dynamic movements, and it is an active area of research in world models. Existing works in robotic control [32, 5, 10, 57, 61, 8] typically focus on static environments. While our current approach aligns with this trend, future work could explore integrating more sophisticated models to better handle dynamic and uncertain environments.

Concern 3: Redundancy in Latent States

Response: As the reviewer correctly notes, latent states are intended to provide compact representations of the environment that focus on the most relevant features. To reduce the influence of irrelevant factors such as texture or lighting, we train the world model directly in the latent embedding space rather than in the image pixel space. Specifically, the loss function (Section 3.3) is applied on the latent state embeddings instead of the decoded image pixels, where cues like texture and lighting tend to dominate. This approach is consistent with recent advances in self-supervised learning [2] and robotics [61, 8], where training in latent space has been shown to better capture task-relevant features while avoiding overemphasis on pixel-level patterns like brightness or texture.

While quantifying redundancy in latent representations remains an active area of research, we argue that for the purposes of the current work, some degree of redundancy in a larger embedding space is not detrimental to performance. This is because the latent embeddings are subsequently passed through pooling and MLP-based waypoint predictor layers, where the MLP learns to attend to task-relevant features. The primary requirement for the latent representation is to effectively disentangle image features. As long as the latent space captures the underlying structure of the scene, the downstream MLP can leverage this disentanglement to generate accurate waypoint predictions. Most learning-based methods, including competing approaches like [5, 32, 10, 57, 60], utilize encoders such as ResNet to extract features. Although redundancy is possible in all of them, it is not widely recognized as a major factor in reducing performance.

Concern 4: Limited Theoretical Analysis

Response: We acknowledge the importance of theoretical analysis in understanding the properties of any framework. However, providing a rigorous theoretical analysis for large learning-based methods, such as OSVI-WM, is inherently challenging due to their complex and data-driven nature. Consequently, similar to other works in this area, we have focused on empirical validation to demonstrate the effectiveness and robustness of the various design choices in our framework (Section 4.2).

Question 1: Generalization to Unseen Positions and Objects

Response: As outlined in the ablation studies in Section 4.2 of the manuscript, OSVI-WM’s generalization capabilities arise from several key components: (a) world-model-guided trajectory generation with WM loss, (b) re-planning, (c) stop-gradient in WM supervision, (d) spatial pooling after the world model, and (e) multi-step trajectory prediction. Many one-shot visual imitation methods struggle with generalization because the model plans actions based on the objects in the image while underutilizing the expert demonstration. To discourage this, we employ the world model to predict the next latent states alongside the predicted waypoints. The WM loss applied to these predicted latent states is independent of the waypoints, providing a regularizing effect. Additionally, the use of asymmetric demonstration mixup enhances generalization. We will incorporate these clarifications into the updated manuscript for further clarification.

Question 2: Visualization and Analysis of Latent States and Actions

Response: The latent actions in our model do indeed correspond to the movements of the robotic arm. We qualitatively analyze this relationship through the following experiments:

• Effect of Latent Actions on Latent States: We observe that when a zero vector is used as the latent action, the predicted latent state remains unchanged, indicating no movement. In contrast, when a random latent action vector is applied, the visualized latent state exhibits small, unstructured movements of the robotic arm. This qualitatively confirms that latent actions have a direct and controllable influence on latent states.
• Effect of Latent States on Action Generation: To assess how changes in latent states drive the generation of latent actions, we perform an intervention during multi-step prediction in Eq. 1. Specifically, we manually reset a future predicted latent state $r_j$​ (where $1<j<M−1$) to be equal to the initial state $r_1$​, and feed this into the action encoder. We find that the latent actions adapt accordingly, attempting to recover the trajectory and guide the latent state back toward the originally planned sequence. This demonstrates that the action generation mechanism is sensitive to changes in the latent state and aims to maintain trajectory consistency.

While we are unable to include these new visualizations in the rebuttal due to submission guidelines, we will incorporate them into the final version of the paper to enhance interpretability and provide clearer insights into the interaction between latent states and actions.

Question 3: The Significance and Efficiency of Replanning

Response: When task execution fails (detected by the simulation environment configuration), OSVI-WM triggers re-planning by taking a new observation and generating a fresh sequence of waypoints based on the robot's current state. The replanning mechanism in OSVI-WM is, therefore, not equivalent to simple repeated execution. Instead of open-loop retries, replanning is a closed-loop process that takes a new observation of the current robot state and generates fresh waypoints based on that feedback. This closed-loop replanning does incur additional time cost. For the results shown in Figure 8, the time comparison is as follows:

• Without re-planning: Each episode runs for a maximum of 500 steps (~40 seconds on our system with one RTX A4000 GPU, 128 GB RAM, and an Intel i9 CPU).
• With re-planning: New waypoints are generated every 500 steps, for up to 2000 steps (~160 seconds) total.

Despite this overhead, replanning substantially improves success rates. Notably, in our paper, replanning is only applied for the ‘hard’ Meta-World tasks. Even without replanning, OSVI-WM achieves a success rate of ~42% on these tasks (71.5% after re-planning), which is already higher than all existing baselines (Fig. 8). Additionally, it is important to highlight that many baseline methods, such as [10], are also inherently closed-loop but still perform worse than OSVI-WM, further emphasizing the strength of our approach beyond just repeated planning. We will include a discussion on the time cost of replanning in the revised manuscript.

Question 4: Redundancy in Latent States

Response: As discussed in our response to your Concern 3, we explicitly address the issue of unrelated background information, such as texture or lighting, by training the world model in the latent embedding space instead of the image pixel space. This design choice ensures that the model focuses on higher-level, task-relevant representations, rather than low-level visual features. Recent works in self-supervised learning and robotics [2, 8, 61] have similarly adopted this approach to improve generalization and reduce sensitivity to irrelevant visual cues.

Similar to the response to your Concern 3, we claim that, while some redundancy in latent features may exist, the primary requirement for the latent representation is to effectively disentangle image features. As long as the latent space captures the underlying structure of the scene, the downstream MLP can leverage this disentanglement to generate accurate waypoint predictions. Most learning-based methods, including competing approaches like [5, 32, 10, 57, 60], utilize encoders such as ResNets to extract features. Although redundancy is possible in all of them, it is not widely recognized as a major factor in reducing performance.

We thank the reviewer for the detailed review and for the constructive comments. We further appreciate your recognition of the strengths of the paper. In the following, we address each of the concerns (weaknesses) and questions.

Concern 1: Lack of comparison against one-shot imitation learning methods through alignment and open-loop replay, e.g., DINOBot.

Response: DINOBot (and similar pixel alignment and open-loop replay methods) perform one-shot imitation by using a gripper-mounted camera to visually compare the current scene to demonstration frames. It retrieves the closest visual match and estimates pixel-level alignment to grip the object and execute a trajectory. While its use of foundation models for visual retrieval followed by pixel-level alignment is unique and effective without further training, its problem setup differs from ours. DINOBot relies solely on an ego-centric gripper camera, whereas OSVI-WM uses external third-person cameras. Consequently, DINOBot has shortcomings against external camera based methods as discussed below. Moreover, adapting DINOBot’s retrieval and pixel alignment framework for external cameras is not trivial. Therefore, we restricted our comparisons to one-shot visual imitation methods that use external third-person camera views. The main shortcomings we observed when attempting to use DINOBot are detailed below.

1. To conduct retrieval and pixel alignment, DINOBot requires the target object to be visible to the gripper camera at all times. In our experiments, the target is often initially outside the gripper’s field of view, making such assumptions invalid.
2. In multi-object settings like our pick-and-place tasks, which involve four randomly arranged objects and four targets, both the current observation and demonstration images contain multiple objects with different configurations. As a result, there is no single, consistent visual transformation that aligns the current scene’s pixels to that of the demonstration. DINOBot’s reliance on clear visual correspondence for a single object breaks down in such cluttered and compositional environments.

We will include this discussion on DINOBot and similar works in the updated manuscript to clarify the differences.

Concern 2: I am not familiar with world-model-based methods for manipulation. While Human-to-Robot skill transfer is impressive to me, the studied manipulation tasks are generally simple, i.e., top-down grasping with low precision requirements. Given that BC-based or VLA-based policies have enabled very dexterous manipulation behaviors, I believe a discussion is necessary to illustrate the high-level pros and cons of this world-model-based approach when compared with more prevalent and straightforward BC-based methods.

Response: Thank you for your suggestion. We will include a discussion on the pros and cons of our world-model-based approach compared to straightforward BC-based or VLA-based approaches in the updated manuscript. Generally, both world-model-based and BC-based (or VLA-based) approaches ultimately aim to execute tasks by predicting actions. However, the key distinction is that world-model-based approaches also learn the world dynamics independently of the task. This discourages the model from overly fixating on action prediction and encourages learning more generic features through the additional training signal provided by the world model loss. Most existing methods, whether world-model-based or BC-based, typically focus on training and testing within the same set of tasks, as detailed in our manuscript. Many of these methods have demonstrated good results in complex dexterous manipulation tasks, as you mentioned. However, when faced with out-of-distribution tasks, like those in our examples, existing BC-based methods such as [5, 10, 57] often show low success rates, even on top-down grasping tasks. Additionally, our experience indicates that even recent VLA-based robot foundation models like Pi0 [A] and Groot [B] require in-context retraining for new tasks for accurate manipulation, despite being pre-trained on large datasets.

Question 1-A: In the simulated experiments, it seems that how to divide the train and test tasks will dramatically affect the experimental results. For instance, the Button-Press task shares very similar action behaviors with the Button-Press-Topdown task.

Response: How one divides the train-test task will indeed impact results, particularly in the Meta-World simulation environment. In OSVI-WM, to be consistent with prior work, we adopt the train-test split proposed in AWDA [5] for Meta-World. During training, the model learns two key components: inferring tasks from the expert and acquiring the skills needed to complete these tasks. Hence, careful consideration is required when choosing training tasks to ensure meaningful generalization to unseen tasks. To explore this further, we conducted two new ablation studies:

• Ablation 1: Button Press Tasks Removed from Training. We removed all training tasks related to button pressing (Button-Press-Topdown, Button-Press-Topdown-Wall, Button-Press-Wall) and evaluated the model on the Button-Press task. The model still achieved a high success rate of 97%, nearly identical to when the button-related tasks were included. This is because button pressing is relatively kinematically less complicated and the required skills were already acquired from other training tasks.
• Ablation 2: Window Tasks Removed from Training. We removed the Window-Close task from training and evaluated on Window-Open. In this case, performance dropped from 95% to 80%. The sliding window tasks here involve more complex physical dynamics. So the absence of related training data led to a noticeable drop in generalization.

If the training set included a very large variety of tasks with diverse motions and skills, the model would potentially perform well on any task, even if no similar tasks were present in the training set. However, this approach would require large out-of-context datasets, significant training resources, and extended training times.

Question 1-B: So what are the considerations behind this split?

Response: Based on Ablation 1 and Ablation 2, we could, in principle, construct numerous train-test splits and get different results. However, to maintain consistency with prior work, we adopt the split proposed in AWDA [5] for Meta-World.

Question 1-C: How are the similar tasks in the trainset different from the testing tasks?

Response: As mentioned in Section 4.1 of the manuscript and detailed further in Section C of the appendix, the similar tasks in the training set differ from those in the testing set in Meta-World either through the introduction or omission of distractors such as walls (for the ‘easy’ tasks), or by requiring a different set of motions (for the ‘hard’ tasks), which can only be executed correctly by following the expert demonstration.

Question 2: What are the specific settings of Human-Franka tasks? What is the camera setup? Are the objects fully randomized on the tabletop?

Response: In the Human-Franka tasks (Human-Franka-PP and Human-Franka-Push), the Franka agent is positioned on a tabletop, with a camera fixed near the table's edge at a height that provides a clear view of the table and the objects, as depicted in Figs. 5b and 5c. We utilize an Intel RealSense D455 camera with known intrinsic and extrinsic (camera-to-robot transformation) parameters, capturing images at 640x480 resolution. Additionally, an Intel RealSense D435 is mounted on the robot's end-effector to assist with grasping, as detailed in Section 3.2 of the manuscript. Object locations are fully randomized during both training and testing. For testing, to ensure fairness across all methods, we use the same 25 random object configurations across all methods.

References

[A] Black, Kevin, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai et al. "$\pi_0$: A Vision-Language-Action Flow Model for General Robot Control." arXiv preprint arXiv:2410.24164 (2024).

[B] Bjorck, Johan, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang et al. "Gr00t n1: An open foundation model for generalist humanoid robots." arXiv preprint arXiv:2503.14734 (2025).

We thank the reviewer for the detailed review, acknowledging the strengths of the paper, and for the constructive comments. We address each of the concerns (weaknesses) and questions in the following.

Concern 1: The re-planning … application.

Response: OSVI-WM uses re-planning only for Meta-World 'hard' tasks (Section 4.1). Since these are simulation tasks, success is automatically verified by the environment configuration. Although we do not apply re-planning in real-world experiments, enabling autonomous re-planning would require automated task success detection. Some prior works have explored vision-language models [A,B] or reward/value networks [22] for this purpose. We will include a discussion on this in the updated manuscript. Notably, even without re-planning, OSVI-WM surpasses the performance of existing methods (Fig. 8).

Concern 2: The paper's … manipulations.

Response: During training, the model essentially learns two components: to infer tasks from the expert and to acquire the skills necessary to complete these tasks. In completely out-of-domain tasks, the model might still grasp the task context from the expert but may not learn the required skills if the training and testing tasks involve completely different skill sets. For instance, pick-and-place tasks involve skills like reaching, grasping, moving, and placing objects, whereas wiping requires maintaining contact with a surface while moving the end-effector. This skill disparity can lead to failures in testing if the model is trained only on pick-and-place tasks. To further support this, we conducted two additional ablations:

• Ablation 1: We removed all training tasks related to button pressing (Button-Press-Topdown, Button-Press-Topdown-Wall, Button-Press-Wall) and evaluated the model on the Button-Press task. The model still achieved a high success rate of 97%, nearly identical to when the button-related tasks were included. This is because button pressing is relatively kinematically less complicated and the required skills were already acquired from other training tasks.
• Ablation 2: We removed the Window-Close task from training and evaluated on Window-Open. In this case, performance dropped from 95% to 80%. The sliding window tasks here involve more complex physical dynamics. So the absence of related training data led to a noticeable drop in generalization.

However, if the training set included a very large variety of tasks with diverse motions and skills, the model would potentially perform well on any task, even if no similar tasks were present in the training set. Achieving this would require large datasets, high training resources, and training times. We will add a discussion on this in the updated manuscript to clarify how the generalization scales to different unseen tasks.

Question 1-A: The action model is an inverse dynamic model (IDM) conditioned on given tasks, and the world model is a corresponding forward dynamics model (FDM), in the context of latent action learning [1-3].

Response: The world model functions like a forward dynamics model, while the action model resembles an inverse dynamics model, with the distinction that OSVI-WM’s action model is conditioned on expert videos rather than the agent’s future states ($a_i = \pi(r_i, \dots, r_1, e_N, \dots, e_1)$).

Question 1-B: Could you elaborate … approaches?

Response: Directly predicting the next latent state using only ground truth actions can cause the model to overfit to those actions, resulting in the model only learning features necessary for such action prediction. This dependency often results in poorer generalization to unseen test tasks (Fig. 7). In contrast, incorporating a world model loss provides auxiliary signals that help the model learn environment dynamics independently of any tasks. While a seq2seq-style connector directly learns the ground truth states, the factorization between the action and dynamics (world) model offers key advantages. The dynamics model depends only on the latent actions and current agent states, not directly on the goals, thus enhancing the model's interpretability and modularity and encouraging better representation learning. Additionally, if the goal representation changes (e.g., to text), only the actions model needs retraining, leaving the world model intact.

Question 2: The world model … precise

Response: OSVI-WM’s world model is conditioned solely on the latent actions and the agent’s states, rather than directly on the expert videos ($r_{i+1} = \mathcal{W}(r_i, a_i, \dots, r_1, a_1)$). The action model, however, is conditioned on the expert videos ($a_i = \pi(r_i, \dots, r_1, e_N, \dots, e_1)$). By making this separation, we get key advantages as detailed in the response to your Question 1-B. We will make this difference clear in the updated manuscript.

Question 3: The process of … robot?

Response: The generated waypoints represent the end-effector’s 3D positions in the camera’s coordinate frame and binary gripper states (Section 3.2). These positions are transformed into the robot frame using a known camera-to-robot transformation. We denote these 3D positions in the robot’s frame as $p_k = (x_k, y_k, z_k)$, where $k = \{1, 2, \ldots, q\}$. The desired rotation $\mathrm{rot}_k$ of the end-effector (which remains constant in our case) is concatenated to $p_k$ to obtain 6D desired pose $H_k = (x_k, y_k, z_k, \mathrm{rot}_k)$. We use this desired pose with the controller as detailed below:

• Simulation Experiments: Obtaining the joint angles $\theta_k$ involves solving the inverse kinematics problem , $\mathrm{F.K.}(\theta_k) = H_k$. Given that the robots used in our experiments have more than 6 degrees of freedom, the inverse kinematics does not have a unique solution. Therefore, we utilize the following optimization problem: $\min_{\theta_k} \left\| H_k - \mathrm{F.K.}(\theta_k) \right\|$ where $\theta_k$ is the robot's joint angles and $\mathrm{F.K.}$ is the forward kinematics. To avoid jerky motion from large pose jumps between waypoints, we interpolate sub-waypoints between the current and target waypoint poses. The controller tracks these sub-waypoints, while the gripper follows the predicted gripper state.

• Real-World Experiments: We employ a differential inverse kinematics approach, optimizing for joint velocity by solving: $\min_{\dot{\theta}_k} \left\| J \dot{\theta}_k - K_p e \right\|^2 + \lambda \left\|(I - J^\dagger J)(\dot{\theta}_k - \dot{\theta}_b) \right\|^2$ $J$: Jacobian of the end-effector; $\dot{\theta}_k$: joint velocity; $K_p$: proportionality constant; $e$: pose error between the current and target poses. The second term in the equation above ensures that the joint velocities remain close to a nominal joint velocity $\dot{\theta}_b$ (please refer to [C] for more details). The resulting velocities are passed to the Franka robot, running an internal proprietary controller at approximately 4 kHz. As in simulation, sub-waypoints ensure smooth motion, and the gripper executes the predicted open/close action.

Question 4: The current … actions?

Response: In most robotics datasets [58,10,D,E,F,G,H] and real data collected in the lab, access to the robot’s proprioceptive states, typically joint angles and angular velocities, is standard. These states can be converted into end-effector poses and velocities via the robot’s forward kinematics. In OSVI-WM, we utilize these end-effector poses to generate ground-truth waypoints, which serve as target actions for training the model as detailed in the manuscript. However, in the uncommon scenario where proprioceptive states are unavailable, robotic datasets may represent actions as the difference between successive joint angles (or end-effector poses). We can still reconstruct the full joint trajectory by cumulatively summing these deltas starting from a known initial joint configuration (which is feasible with current-day robot hardware, which provides reasonable sensor resolutions; methods to reconstruct estimates of the robot pose from external cameras are also available). Using forward kinematics, we can then convert the reconstructed joint angles at each time step into end-effector poses, which are subsequently used to generate ground-truth waypoints for training OSVI-WM. This reconstruction-based approach allows us to maintain the overall framework and learning objectives of OSVI-WM even in the absence of direct proprioceptive state data.

Limitations: The authors … scope.

Response: Thank you for this feedback. We will include a discussion of this in the limitation section, along the lines of the response to your concern 2.

References

[A] Liu, Haichao, et al. "Robodexvlm: Visual language model-enabled task planning and motion control for dexterous robot manipulation." arXiv preprint arXiv:2503.01616 (2025).

[B] Xiong, Chuyan, et al. "Autonomous interactive correction MLLM for robust robotic manipulation." In 8th Annual Conference on Robot Learning. 2024.

[C] Lynch, Kevin M., and Frank C. Park. Modern robotics. Cambridge University Press, 2017.

[D] Nasiriany, Soroush, et al. "Robocasa: Large-scale simulation of everyday tasks for generalist robots." arXiv preprint arXiv:2406.02523 (2024).

[E] Abhishek Gupta, et al.. “Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning”. arXiv preprint arXiv:1910.11956, 2019. 2, 5, 17

[F] Vikash Kumar. “Manipulators and Manipulation in high dimensional spaces”. PhD thesis, University of Washington, Seattle, 2016.

[G] Kumar, Vikash, et al. "Robohive: A unified framework for robot learning." Advances in Neural Information Processing Systems 36 (2023): 44323-44340.

[H] Khazatsky, Alexander, et al. "Droid: A large-scale in-the-wild robot manipulation dataset." arXiv preprint arXiv:2403.12945 (2024).