Building 3D Representations and Generating Motions From a Single Image via Video-Generation

We thank the reviewer for their constructive and extremely helpful feedback. We appreciate the positive feedback of our methodology and the clarity and presentation of the manuscript.

Here, we’d like to answer some of the questions presented. Our proposed VGER goes directly from a single RGB image to a scene, and also attempts to provide estimates in regions that are occluded. Generally, the reconstructed surfaces provided by 3D foundation models are of higher quality (less noise) than off-the-shelf RealSense cameras. With a good stationary depth image, I would expect it not to have the frustum-shaped errors that are common when trying to extract structure from monocular depth estimation models (e.g. DepthAnything). However, VGER still provides additional value by enabling the generation of regions that are occluded.

We agree that the resulting point cloud from the 3D foundation models could perhaps be used for collision checking in motion planners such as RRTs. The construction of the implicit model enables a smoothly varying distance to the surface to be learned, enabling the construction of the policy, given as a vector field in 3D space. This will enable trajectories to be efficiently integrated at different initial positions, whereas sampling-based motion planners, which do collision-checking based on the point cloud, generally return a single trajectory/path, rather than a policy. We acknowledge that training the implicit model from the point cloud representation needs to be done offline. However, this is not overbearingly slow, and the per-scene training time is approximately several minutes on consumer-grade GPUs.

We thank the reviewer for their constructive comments and for highlighting the performance and innovation of our approach!

We wish to highlight that the pictures of both the indoor and outdoor scenes were images that we took in the real-world, so the performance seen is indicative of performance on real images. We agree with the suggestions here that the uncertainty from the 3D foundation model can be propagated to the VGER model to guide the motion generation, and potentially achieve active perception by enabling robots to actively perceive these regions. This shall be explored in subsequent work. Additionally, the Riemannian Metric here is constructed such that collision avoidance is achieved. However, you’re completely right that this metric can also be learned to, for example, imitate a set of provided demonstrations. Finally, we definitely plan to deploy the method onto a real robot and have identified table-top scenes to be particularly suitable, as VGER would be able to estimate regions of the scene which are occluded, which would be particularly valuable.

We thank the reviewer for their constructive comments and efforts in helping us improve the manuscript. Thank you for highlighting the methodological contributions and empirical results in our work.

We particularly appreciate the reviewer taking the time to outline the methods, Cat3D, Viewcrafter, and Gen3C. These are contemporary methods which share similarities with VGER. In particular, Viewcrafter similarly leverages the 3D foundation model DUSt3R, Gen3C integrates a video generator, while VGER makes use of both.

The comparison results are tabulated above, we observe that all three of the methods have solid performance, as measured by chamfer distance. VGER generally outperforms the alternative methods, with Gen3C also having strong performance in the garden and bench scenes.

The above table provides information of VRAM usage and runtime, we observe that VGER and ViewCrafter have relatively lower VRAM usage and runtime.

Implicit representations, such as learned distance functions are typically used in trajectory optimization methods for motion generation. Our approach differs in that it is reactive, it does not require an entire trajectory to be found, and instead focuses on generating the immediate action. This enables reactive methods to be a lot faster and produces an entire policy (velocity field) rather than a single trajectory. This is conceptually the most similar to Riemannian Motion Policies (RMPs) (Ratliff, 2018), but unlike RMPs which require an environment representation constructed from primitive shapes, we allow for the learned implicit model to be used.

We also appreciate the point raised about reliability, potential hallucinations in unseen regions, and agree with the opportunity to leverage active perception. We note that we can actually track which regions of the reconstruction are connected to the original input image, and which regions are connected to frames generated via video generation. Additionally, the produced reconstruction from VGER includes pointwise confidence maps, which are inherited from DUSt3R. These confidence values can indeed be used to guide robots to actively perceive and reduce the uncertainties in areas not covered by previous images. However, we believe that producing an estimate of the environment structure even in unseen regions remains a valuable prior. Accomplishing object-level shape completion, for occluded objects, is valuable in robotics, likewise VGER accomplishes the completion of scene level occlusions not captured in a single image in one go. Overall, we definitely agree that active perception would be a practical solution to potential hallucinations described in the limitations section, and we hope to incorporate active perception as a follow-up to this work.

What are the scores in table 1 mean?

We thank the reviewer for the thoughtful comments and for highlighting both the performance and presentation of the manuscript.

We thank the reviewer for pointing out additional baselines for experimental comparisons. Here we report normalized Chamfer distances of VGER as compared to the contemporary methods, ViewCrafter and GEN3C:

The comparison results are tabulated above. We observe that all three of the methods have solid performance, as measured by chamfer distance. VGER generally outperforms the alternative methods, with GEN3C also having strong performance in the garden and bench scenes. Additionally, VGER and ViewCrafter have comparable VRAM and runtime costs while GEN3C requires more VRAM and time.

The introduced method is the first reactive approach, which produces a policy (velocity field) in the 3D environment that leverages an implicit representation constructed from a dense point cloud. Alternative approaches such as motion planning via sampling-based motion planners and trajectory optimization approaches generally produce a single path or trajectory rather than a vector field which can be integrated. In that sense, the proposed approach is reactive, and is conceptually similar to Riemannian Motion Policies (RMPs) (Ratliff 2018) and Geometric Fabrics (Van Wyk, 2024). These approaches typically assume that the environment is constructed of a set of primitive shapes. The training of the implicit method with the multi-scale noise sampling is fairly efficient and can be achieved in minutes on consumer-grade GPUs.

The hyperparameters for the motion policies constructed are in Table A1 in the appendix under “Motion Policy Blow-up”, we will make sure to make this more prominent in the next version of this manuscript. Additionally, the number of points in the representation before implicit representation is constructed differs from scene to scene, depending on the confidence outputted by the 3D foundation model, but is generally in the millions of points. We also thank the reviewer for emphasizing the connection with robotics tasks. In this paper, we focused on relatively larger both indoor and outdoor scenes. An interesting scene that further highlights the benefits of our approach is table-top scenes to do motion generation for manipulators. This is an area where predicting complete geometries can prove to be very important, we will add this scene into the next version of our submission.