Object-centric 3D Motion Field for Robot Learning from Human Videos

Thank you for the thoughtful review! We would like to highlight several key novel contributions of our work:

1. More challenging data setups: Unlike existing systems such as Track2Act, which require in-domain robot data, our method operates solely on human demonstration videos. In contrast to methods like GFlow, which rely on extensive point cloud filtering and heuristic, hand-engineered preprocessing (see their appendix), our approach avoids such manual steps by leveraging a learned motion denoiser. A detailed comparison with other systems can be found in Table 2. We made strictly less assumptions over data in this paper.

2. Scalability: Many prior works, such as ATM and General Flow, use point-based prediction architecture, which makes it difficult to integrate them with standard generative modeling frameworks. Our method uses an image-based motion representation that aligns naturally with image generative models, facilitating greater scalability.

3. More precise tasks: Previous works have mostly focused on coarse manipulation tasks (e.g., mostly pick-and-place motion of large objects) and have not explored fine-grained scenarios involving small objects or tool use. Our paper demonstrates that such challenging tasks can, in fact, be addressed using a purely video-based learning pipeline.

We acknowledge that, at a high level, our method may resemble existing works in predicting "flow"-like representations for control. However, we argue that the specific choice of representation and the way to extract it are critical factors that determine the system’s capability. In summary, our contributions are twofold: 1. We propose a scalable pipeline that relies on significantly fewer data assumptions than existing methods. 2. We show that this pipeline can handle fine-grained manipulation tasks that prior approaches struggle with.

Together, these results provide the first strong evidence that simple, pure video-based policy learning is a viable and promising direction for solving complex, high-precision tasks—and now we are confident in its potential to scale further.

Thanks for the review! Our responses to your questions are as below.

Q1. Thank you for the question. In most of the scenarios, we may want to prompt the robot with language during deployment. There are many existing vision-language models (VLMs) and open-vocabulary object detectors (e.g. Grounding-DINOv2) capable of grounding target objects in human language instructions to a box/point in the current scene automatically. We believe that this will not be an issue in the near future due to rapid advances in VLM and vision research. In our setup, the object only needs to be labeled once at the beginning. Its mask is then robustly tracked throughout execution using SAM2. We have experimented with Grounding-DINOv2 + GPT and it works well in our setups.

Q2. Thank you for the question. The human videos we use are task-relevant—for example, if the task involves picking up a bottle and pouring it, the predicted motion field consistently reflects that intention (e.g., the bottle moves upward and then tilts). We acknowledge that motion in human demonstrations can be multi-modal—for instance, the bottle might be lifted slowly in one video and quickly in another. Our method addresses this by using an image-based generative model that learns to produce plausible motion trajectories observed in the training data. In other words, it can generate either a slow or fast lifting motion based on the current visual context, but the motion itself always remains valid and task-relevant. Furthermore, the predicted 3D object motion field defines how each visible point on the object should move in 3D space, relative to the camera frame. This makes the motion representation unambiguous—unlike 2D motion fields, which can be affected by depth ambiguities or projection artifacts.

Thanks for the review! Our responses to your questions are as below.

Q1. Thank you for the question. Our method directly generates the next target positions for all object points, which are then used to compute the corresponding robot poses. The robot proceeds to the next prediction cycle only after successfully reaching the current target. We can command the robot to the target within the desired time interval.

Q2. We use a control frequency of 10 Hz in our experiments, corresponding to a 0.1-second interval between predictions.

Q3. In the second phase, the depth in the camera input is still very noisy, and our policy can output a cleaner depth map for motion recovery.

Thank you for appreciating our work! Our responses to your questions are as below.

Q1. Thank you for the suggestion. We will release the synthetic data generator, pretrained model, and data labeling pipeline to ensure reproducibility and enable researchers to apply our method in their own task setups.

Q2. Yes, the manipulator is controlled via inverse kinematics after computing the next target pose.

Q3. The image in Figure 7 was recorded from a different viewpoint solely for visualization purposes and does not reflect the policy's actual observation. In practice, the object appears significantly smaller in the policy's visual input.

Q4. Due to the difficulty of the precise insertion task, we opted not to rigidly fix the base structure. This simplification helps avoid damage that can occur when the stiff robot controller applies excessive force. However, it's important to note that this is still very hard as our control policy is purely vision-based (from 40-50cm away) without any force or tactile information, and accurately inferring the insertion angle from vision alone is inherently challenging (and this may result in a slight base movement). For the success trials, the base may be moved by around 1-2mm based on our measurement on the base corners. We hypothesize that in the future, the use of higher-resolution depth cameras or multi-view sensing setups (e.g. fusing the 3D motion flow prediction) could help improve performance in such tasks.