RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion

Thank you FXY9 for these great questions, we will modify the paper to reflect our answers below:

Q1. Comparison against OctMAE: Following your suggestion, we trained RaySt3R only on the OctMAE dataset and with only the OctMAE augmentations. The table below shows RaySt3R trained on either dataset outperforms OctMAE (bold is best, italics is second best).

While we did not have the resources to train OctMAE on our dataset, we believe that this experiment nevertheless shows the contribution of our method.

In retrospect, our description of the datasets may have been unclear. Our dataset uses the exact same objects as OctMAE, and only 10% more input images. "1.1 million scenes", refers to the unique RGB-D input images; but the actual scenes are only 251k. Also, "12 M views" includes both the input images and the depth maps used for supervision. See table below.

Q2. Evaluation Metrics: We follow prior work by computing the Chamfer distance with the L2 norm (the Euclidean distance) for both accuracy and completeness. See Equations (1)-(3) in the supplemental material. (In the paper, the reference to "Chamfer-L1" is an error which we will correct in the final version of the paper.)

As requested, to further study the impact of the introduced noise, the tables below provide the CD computed based on RMSE, as well as the accuracy and completeness. These additional metrics show that RaySt3R consistently outperforms the baselines.

Confidence threshold: Fig 4 shows a tradeoff: a lower confidence threshold yields better Chamfer distance but worse accuracy. To avoid unfairly choosing hyperparameters from the test data, we arbitrarily selected a threshold of 5 for all experiments, qualitative and quantiative. The table below shows the Chamfer distance for each scene with different confidence thresholds. Overall, RaySt3R is not sensitive to the choice of confidence threshold and consistently outperforms the baselines under all confidence thresholds.

Q3. View-based vs volumetric approachs: We agree the pros and cons of view-based completion vs 3D representations should be emphasized more. The advantages of view-based completion are: (1) the ability to query an arbitrary number of views to trade off quality and inference speed; (2) the ability to supervise directly on real-world data without GT meshes; and (3) pixel-aligned and more accurate prediction compared to volumetric methods. In terms of limitations, while view-based methods may suffer from multi-view consistency issue such as edge bleeding, our proposed confidence-based thresholding and multi-view fusion strategies effectively enhance the raw prediction from the network. Reconstruction results evaluated under multiple metrics indicate our method consistently outperforms volumetric approaches such as OctMAE. In the original abstract, the limited multi-view consistency refers to the other view-based methods such as ViewCrafter, which has been compared against and outperformed by our method in the supplemental material. However, we do admit there are a few other limitations which we will discuss more in the updated version of the paper, including: (1) deciding where to sample novel views involves more designs than volumetric methods; (2) when a relatively large number of views are queried, it leads to slower inference times; (2) generalizing to objects far beyond the current training dataset, such as transparent objects.

Weaknesses We respond to other points raised below:

• Video animation: We added the first animation to the video only for illustrative purposes, showing only 3 queried views instead of 22. To see the full reconstruction, please skip to 2:21 in the video or Figure 15 in the supplemental material. RaySt3R successfully reconstructs the bottom of the object because it samples from the viewing sphere (even though the bottom is not shown).
• Training without complete geometry: While we train RaySt3R on a synthetic dataset, our method (unlike volumetric methods) does not require ground-truth meshes and can be trained on 2 or more depth images of a scene (as would be available from real-world depth cameras). We will clarify this in the final version of the paper.
• Random sampling: We worded this incorrectly. We did not use random sampling, since we found it is more likely to miss areas of the scene and introduce non-determinism. We place query views on a grid evenly spaced on a cylindrical equal-area projection of a sphere.
• Noisy input depth: We do not use the 3D points from the input depth map. In the real datasets they tend to be noisy, and introduce floaters especially at edges of objects. There are no 3D consistency issues. We conduct an experiment by appending the input depth map to RaySt3R's predictions in the YCB-V dataset, and observe the Chamfer distance increases from 3.56 [mm] to 3.66 [mm].
• Runtime: We appreciate your suggestion to provide a more comprehensive runtime analysis. In the table below, we provide additional information as benchmarked on our own hardware. SceneComplete is more challenging to benchmark, as it relies on an OpenAI API call to use their VLM. We find RaySt3R with flash attention runs faster than vanilla torch attention (568 ms vs 1200 ms).

• Unified set of parameters: Not all code for the listed baselines is available, hence we are unsure whether hyperparameters were varied for each dataset. Nevertheless, we study RaySt3R's performance with a single set of hyperparameters instead of two sets for synthetic and real datasets. This experiment (see table below) suggests RaySt3R outperforms the baselines even with a single set of hyperparameters.

Thank you for responding to our rebuttal. We answer your questions below:

• How to explain the visualizations and metrics? We visualize the results using a colored point cloud, where even a small number of outliers can make the point cloud appear noisy in visualizations. Although this small number of noisy predictions may not have a large effect on the metrics, OctMAE tends to predict overly smooth shapes and even produces holes (e.g., the last row of Figure 3). Overly smooth or partially missing geometry will have a large impact on all metrics considered. These inaccurate predictions are less noticeable on the static figure, but may cause problems in downstream applications.
• How to use the confidence threshold? While previous works such as VGGT and DUSt3R leverage confidence similarly, we are (to the best of our knowledge) the first to study its implications. The confidence threshold enables users to strike a balance between accuracy and completeness. With Figures 4 and 5, as well as the qualitative Figures, we aim to give the reader insight into this tradeoff. Downstream applications relying on visual quality may choose a higher confidence threshold, while 3D scene completion in robotics may prioritize completeness with a lower threshold. Thanks to the experiments you suggested, our results show RaySt3R outperforms the baselines in both accuracy and completeness with a single confidence threshold. The best confidence threshold for the average Chamfer distance (accuracy and completion) depends on the scene and dataset, as the network's absolute confidence values vary. Investigating the confidence threshold even further, or proposing alternative confidence formulations, would be an interesting topic for future research.
• Are there better metrics to use? We follow the evaluation protocol from prior works, such as OctMAE, under commonly considered metrics. By your suggestion, we compute the F1 scores at various thresholds and report them in the tables below. RaySt3R outperforms all baselines consistently under varying thresholds. This confirms that edge bleeding noisy points only form a small fraction of the total prediction, while accurate surface prediction occupied by the majority of points is more critical for the metrics and downstream applications. In practice, we envision that those minor outlier points are relatively easy to remove if desired, by means such as TSDF or statistical outlier removal. Please let us know if there are other metrics you would like us to evaluate.

• Any other methods to reduce noise? The noise we observe with RaySt3R is similar to that produced by a depth camera. In the open-source release, we will include an off-the-shelf TSDF depth map merging algorithm, which will output smoother meshes for downstream applications.
• Why focus on zero-shot performance? We agree we have over-emphasized the zero-shot aspect, and commit to toning down the claims. We meant to highlight encouraging performance on the real-world datasets with real-world sensor noise and unseen objects. We believe this is an encouraging result, which may improve further when combining synthetic and real-world training data.

Thank you G7Ap for these great questions, we will modify the paper to reflect our answers below:

Q1. With RaySt3R, we have shown that casting 3D shape completion as a novel view synthesis task enables leveraging recent insights from DUSt3R for object-centric tasks. With mask prediction and multi-view merging, we believe our method may be applied to other object-centric tasks such as 3D-complete tracking or panoptic segmentation. Besides the results, an important aspect is the ability to train on real-world data without GT meshes.

Q2. In our setting, zero-shot transfer means we train RaySt3R only on synthetic data and evaluate the model on real-world scenes outside the training set, consisting of unseen objects, scene configurations, and noisy depth sensing input.

Q3. The 3D visualizations show dense point clouds (not meshes), where colors are assigned based on each point's 3D location using a Turbo colormap.

Weaknesses

In addition to the weaknessess we addressed in answering the questions above, we note

W1. Justification of learning novel views vs other 3D-completion proxies: Layered ray intersections (LaRI) is an example of an alternative proxy of 3D completion where effectively the back of an object is predicted. Benchmarking against LaRI and other volumetric approaches, the results suggest RaySt3R produces better completion results. RaySt3R can also be trained on any pair of (real-world) RGB-D views, all other baselines require access to the full scene geometry.

Limitations: Thank you for pointing out the limited discussion in the conclusion, we commit to expanding the limitations and future work in the camera ready version. We will discuss the pros and cons of view-based multi-object completion. The advantages we see are the ability to query an arbitrary number of views to trade off quality and inference speed, the ability to supervise directly on real-world data without GT meshes, and empirically more accurate results compared to the volumentric baselines. View-based methods suffer from the common edge bleeding issue, and deciding where to sample novel views can be challenging. Finally, we will highlight RaySt3R's limitation to objects far beyond the current training dataset, such as transparent objects.

Thank you bxUS for these great questions, we will modify the paper to reflect our answers below:

Q1. Noisy masks: Indeed the network may be challenged by noisy depth masks as present in the datasets. We try to mimic this situation in the data augmentations (Sec 5.3), and show it improves performance (Table 2).

Q2. Sampling views: When sampling fewer views there is a higher chance of missing part of the geometry while inference will be faster. This is an application-specific tradeoff, e.g. when reconstructing only a single object fewer views may be sufficient.

Below we reply to weaknesses raised in your review:

W1. Texture: Geometry is the essential modality needed for a variety of robotic tasks such as grasping, 3D occupancy reasoning and motion planning, which is the focus of this work. In those applications, texture is not commonly used. However, we agree it would be an exciting future direction to also predict texture for broader applications such as digital twin creation.

W2. Non-planar predictions: While we did not show these examples in the paper, the HOPE dataset contains several scenes where objects are placed in a non-planar fashion with heavy occlusions. For example, in HOPE's scene 3 objects are placed on the different parts of a chair (seat, handrest, headrest). In HOPE's scene 9, objects are placed on top of a cabinet as well as inside the drawers.

W3. Improving smoothness: The lack of smoothness and occasional noisy points can be limited by setting a higher confidence threshold, sampling fewer views, or by fitting a TSDF to the depth map predictions. We will include an off-the-shelf TSDF fitting method in the open-source release.

Instance segmentation: We agree individual instance segmentation would be a useful future direction, allowing users to export individual objects to a simulation environment. Given a modified dataset, a future version of RaySt3R may leverage ideas from PanSt3R [1] to also achieve instance segmentation.

[1] PanSt3R: Multi-view Consistent Panoptic Segmentation, Lojze Zust, Yohann Cabon, Juliette Marrie, Leonid Antsfeld, Boris Chidlovskii, Jerome Revaud, Gabriela Csurka

Thank you cWX6 for these great questions, we will modify the paper to reflect our answers below:

(W.a) Comparison against OctMAE: Following your suggestion, we trained RaySt3R only on the OctMAE dataset and with only the OctMAE augmentations. The table below shows RaySt3R trained on either dataset outperforms OctMAE (bold is best, italics is second best).

While we did not have the resources to train OctMAE on our dataset, we believe that this experiment nevertheless shows the contribution of our method.

In retrospect, our description of the datasets may have been unclear. Our dataset uses the exact same objects as OctMAE, and only 10% more input images. "1.1 million scenes", refers to the unique RGB-D input images; but the actual scenes are only 251k. Also, "12 M views" includes both the input images and the depth maps used for supervision. See table below.

(W.b) LVSM We agree LVSM is a relevant method with impressive performance, and commit to discussing it in detail in the final version of the paper. We cast 3D shape completion as a novel view synthesis problem which allows leveraging ideas from LVSM, DUSt3R and other recent works. A key difference between LVSM and RaySt3R is indeed the scene encoding, where we encode the query view extrinsics into the input view point map. The advantage of our approach is that we do not need full plucker coordinates in the query tokens, and instead only encode the intrinsics (HxWx2). A key advantage of the LVSM approach would be improved efficiency for many query views, as the encoder has to be run only once. We agree an ablation on this aspect would be useful for the community, but it goes beyond the compute resources available to us during the rebuttal.

(W.c) Comparisons vs TRELLIS and LaRI Thank you for raising concerns about fair evaluation of the TRELLIS and LaRI baselines. First of all, our use case is to complete cluttered scenes based on a RGB-D image, which is a common while challenging setup in robotics. We believe the mentioned baselines are valuable in putting RaySt3R's results into perspective of recent and popular image to 3D pipelines, and help motivate depth conditioning for RaySt3R. We also made our best attempt to improve their results, by aligning the results of the baselines to the GT mesh with a brute force search + ICP, including a hyper parameter ablation (supp 1.2.1). RaySt3R outperforms the baselines, even when they are provided access to the GT mesh for alignment, which is unavailable to our method.

However, following your suggestion, we additionally evaluate RaySt3R and baselines on a single object setting from YCB-V dataset. For each frame, we evaluate on the largest object with no occlusions which is often assumed by single-view RGB-based object reconstruction methods such as TRELLIS.

Clarifications

• 226k GSO in Table 2: This ablation trains RaySt3R on the subset of our dataset curated from Foundation Pose, containing GSO objects only.