Real2Code: Reconstruct Articulated Objects via Code Generation

Dear reviewer#mGLU,

Apologies for the late response, we were still in progress of gathering experimental results to better respond to your comments (e.g. running additional experiments on novel category objects, evaluating the cited method [1], which were time-consuming). We are updating with a better-formatted full response here. We hope you could consider changing back the score.

Weaknesses section

W1 Performance on new or unseen categories

We provide additional experiment results, which are divided into two sets

1. Evaluation on smaller objects with complex shapes. We prepared data for two new object categories from PartNet-Mobility - Eyeglasses and Scissors. Both categories have complex shapes beyond simpler, cuboid-like shapes. We use the new dataset to re-train both our proposed SAM fine-tuning model and shape-completion model as described in the main paper, and test the new models on 8 unseen, held-out instances.

Please see this link: https://sites.google.com/view/real2code-submission/complex-shapes for more details and result visualizations. These objects post additional challenges due to having parts with thinner shapes (e.g. blades) and containing more intricate details (e.g. handlebars), and taking up smaller areas in the rendered RGBD images. Qualitatively, we observe the SAM-based segmentation model is able to propose kinematically correct segmentations, but fails at some instances where the glasses legs or scissor blades are too thin.

2. Generalization to novel object categories.

We additionally added 8 more test objects from Microwave and Door categories of PartNet. These two categories were never seen in our training set. Many Microwave or Door objects have similar structural complexity to the StorageFurniture objects we trained on (e.g. a microwave also has a box-like parent body and a hinge door), but contains novel visual appearances and new geometries unseen from the training set. Therefore, our LLM module can generalize to these objects, but our fine-tuned SAM model and our shape completion model does not handle the OOD geometries well (e.g. the press buttons in a microwave dial panel). We have provided the additional details and results on these generalization experiments at link: https://sites.google.com/view/real2code-submission/generalization-to-novel-category

W2 regarding error propagation

This is indeed a limitation of our method. Overall, the main bottleneck of our system is the part-level segmentation quality, because as long as a valid part gets segmented out into a reasonable point-cloud, 1) the shape completion model is trained with noisy input such that it can complete the input; 2) it will result in a reasonable OBB as input to the LLM, which is essentially copying over one of the input edges as a joint axis. In the updated submission pdf, we have added a Discussion & Limitation section to ensure this limitation is well disclosed.

To further improve generalization, an interesting future direction would be involving SAMv2 to obtain more robust segmentations, or using large-scale pre-trained 3D generative models to complete the object shape. See: https://sites.google.com/view/real2code-submission/dataset-details for an illustrated demo of incorporating SAMv2

Questions section

Q1 regarding comparison with [1]:

Due to limited space, please see the other comment below for more details.

Q2 on cross-category generalization:

Following the discussion above and results here: https://sites.google.com/view/real2code-submission/generalization-to-novel-category We observe that, again, 2D part segmentation is the main bottleneck of our pipeline (our model's mesh completion from GT segmentation has clearly better quality), and completely fails for more extreme OOD instances where the object looks very different from training set (e.g. the glass door ID 9107).

Q3 on training & inference time:

Training time for SAM fine-tuning, shape completion model training, and LLM fine-tuning takes approximately 24hrs, 12hrs, and 10hrs respectively. At inference time: 1) SAM prompting takes up approximately 3min per object because of our view-consistent prompting scheme, but the inference time is still kept reasonable because we cache the image embedding from SAM and only call the light-weight prompt decoder for later prompt points; 2) shape-completion model forwarding is single-pass, which can be batched and takes <1min for all parts; 3) LLM generation takes ~2min per object, because the output is strictly formatted to be concise, it requires only ~200 output token length.

Q4 on extracting oriented bounding box: we first use the multi-view RGBD images or the Dust3r-output point maps to get segmented part-level point clouds, then extract OBB using open3d (open3d.geometry.OrientedBoundingBox).

Thank you for the detailed reviews and feedback. and please let us know if you have further questions.

The authors have not made any response to my concerns and questions as well as those of other reviewers. Considering my concerns and those of other reviewers, I reduce the score to 6.

Thank you for the detailed reviews and feedback. We hope our additional clarifications, experiments, and the updated submission pdf will address your raised concerns and increase your confidence in accepting our submission, and please let us know if you have any further questions.

Assumption on object joint structure

Our pipeline can be divided into two parts: 1) geometry reconstruction (includes 1.1 part segmentation and 1.2 shape completion); 2) articulation prediction. We’d like to clarify that our method for 1) can handle arbitrary geometry shapes (e.g. scissors, faucet handles), if these objects are put into the training set for fine-tuning the part segmentation model and shape completion model. However, for 2), because we use the OBB formulation and select OBB edge as rotation center, our method can handle sliding joints (e.g. a sliding oven rack) but will be inaccurate for hinge joints where the joint is not overlapping with any OBB edge (e.g. scissors). To also handle these objects, one possible extension is to add a regression MLP that takes in the OBB information and the LLM’s selection of rotation direction, then predicts a more precise joint axis. We have updated the submission pdf to better clarify our assumption and the subsequent limitations. Since cuboid-shaped objects (cabinets, doors) are commonly seen, we still believe it’s of much value to reconstruct those objects, especially those with many parts, which our method handles much better.

Evaluation on more object categories

Following the discussion above, we provide additional experiments to further validate (1), i.e. and to test whether Real2Code can handle objects with complex geometries. We collected a new dataset that included two new object categories from PartNet-Mobility - Eyeglasses and Scissors. These two categories have complex shapes beyond simpler, cuboid-like shapes (e.g. cabinets) that we evaluated on in the main paper. To prepare the dataset, the original PartNet object assets require mesh repairing processing, manual cleanup, deduplication (several instances are repeated and got removed to keep the train-test split clean), and render RGBD images and occupancy grids. This results in 52 and 43 training objects for Eyeglasses and Scissors, respectively, a total of 20200 segmentation masks and 14520 part ground-truth meshes.

We use the new data to re-train both our proposed SAM fine-tuning model and shape-completion model as described in the main paper, and test the new models on 4 unseen, held-out object instances from each category. Please see more details and qualitative results here: https://sites.google.com/view/real2code-submission/complex-shapes. These objects are challenging due to having parts with thinner shapes (e.g. blades) and containing more intricate details (e.g. handlebars), and taking up smaller areas in the rendered RGBD images. Qualitatively, we observe the SAM-based segmentation model is able to propose kinematically correct segmentations, but fails at some instances where the glasses legs or scissor blades are too thin.

We have not added more categories for now due to time constraints in data processing and model training (for the Lamp and Globe objects mentioned by the reviewer, the mesh processing for the assets are quite demanding), but hope these results provide sufficient evidence that our method can handle more diverse and complex object geometry.

Thank you for the detailed reviews and feedback. We hope our additional clarification, experiments, and the updated submission pdf will address your raised concerns and increase your confidence in accepting our submission, and please let us know if you have any further questions.

Question 1 & 3: Fragile to failures in specific components in the pipeline, and system bottleneck

• Error propagation is a valid limitation of our method. Overall, the main bottleneck of our system is the part-level segmentation quality, because as long as a valid part gets segmented out into a reasonable point-cloud, 1) the shape completion model is trained with noisy input such that it can complete the input; 2) it will result in a reasonable OBB as input to the LLM which is essentially copying over one of the input edges as a joint axis.

• This can be better illustration by the qualitative results in here: https://sites.google.com/view/real2code-submission/complex-shapes We observe when give ground-truth part-level segmentations, the shape completion model produces high quality mesh reconstructions, and the resulting OBBs are naturally well fitted and oriented. However, when there are sufficient errors in some of the segmented views, we see the segmented point clouds having “leakage” between object parts. See Eyeglasses object#101303: The resulting mesh is inflated because the input point cloud on the glasses part contains inaccurate points on the legs, hence the model receives a bad normalized input pcd (recall that we normalize all the part point clouds using partially-observed OBB during inference).

• In the updated submission pdf, we have added a Discussion & Limitation section and ensure this limitation is well disclosed (see Appendix section A.1.3). And we will update the submission with these additional results to provide more insights. However, there is also several possibilities to increase the robustness of our part segmentation module: 1) provide additional user prompt points, this will allow the model to focus more on the correct parts; 2) incorporate the latest SAMv2 model that segments videos. We can concatenate multi-view inputs into a video, first run our fine-tuned model to propose kinematically-accurate parts, then run SAMv2 to obtain view-consistent masks. We provide a demo of this process in the video here: https://sites.google.com/view/real2code-submission/dataset-details

Q2: Results on more object categories.

• We have added additional experiments to evaluate our method on new object categories, please see qualitative results here: https://sites.google.com/view/real2code-submission/complex-shapes We prepared a new dataset using two new object categories from PartNet-Mobility - Eyeglasses and Scissors, both have complex shapes beyond simpler, cuboid-like shapes (e.g. cabinets) that we evaluated on in the main paper. We use the new data to re-train both our proposed SAM fine-tuning model and shape-completion model as described in the main paper, and test the new models on unseen, held-out object instances. Qualitatively, we observe the SAM-based segmentation model is able to propose kinematically correct segmentations, but fails at some instances where the glasses legs or scissor blades are too thin.

• Regarding questions on LLM training details: 1) yes, LLM trained on all categories together. 2) the objects across categories are normalized to the same scale in size, subsequently CodeLlama inputs are also normalized. See https://sites.google.com/view/real2code-submission/dataset-details for a mesh visualization of a Laptop object and a Box object, which we show are already on a matching scale.

Q4: Generalization to other categories:

• Take the example of Microwave objects from PartNet. Microwave is an unseen category not included in our training set, objects here have similar structural complexity to the StorageFurniture objects we trained on (a box-like parent body and a hinge door), but contains novel visual appearances and new geometries unseen from the training set. Therefore, our LLM module can generalize to these objects, our fine-tuned SAM model sometimes fails on object instances that have OOD visuals, and our shape completion model does not handle the geometries well (e.g. the press buttons in the dial panel).

• Overall, because of our OBB formulation, the LLM can generalize reasonably within similar structures to our training set. In terms of geometry reconstruction, the generalization is bounded by both the visual appearances and diversity in shape completion model training data. To further improve generalization, an interesting future direction would be involving SAMv2 to obtain more robust segmentations, or using large-scale pre-trained 3D generative models to complete the object shape. We have provided the additional details and results on these generalization experiments at link: https://sites.google.com/view/real2code-submission/generalization-to-novel-category

Dear Reviewer 9fCy

Thank you for the detailed reviews and feedback. We hope our additional clarifications and the updated submission pdf will address your raised concerns and increase your confidence in accepting our submission.

Weaknesses section

1. Q: Reason for using permutations during evaluation.

A: This is because our method does not get explicit information about how many parts to predict for a given test object or which specific part to reconstruct, i.e. it must predict all parts together, and it may or may not match the ground truth. Therefore we use permutation as an automatic way to evaluate all possible matches and take the lowest error one. We believe this is a more systematic way of evaluating the predictions, and not inherently an issue with our method or problem formulation.

2. Q: links to visualizations

The animated figure was intended to show that our reconstructed object can be directly imported to a physics simulation (MuJoCo) and have a robot arm manipulating it. We have also since updated the submission website to include more visualizations and additional results requested during the rebuttal phase.

3. Q: confusing content in Tab. 1

A: Apologies for the confusion. The Real2Code+gtSeg result is to validate the need for our shape completion model: we use ground-truth image segmentation to get correct but partially observed point clouds (because the object observations contain occlusions). This subsequent mesh extraction from the aggregated point clouds is better than using SAM segmentation, but is still incomplete and hence leads to larger chamfer-distance errors. We have also updated section 4.2.1 to better clarify the meaning of Real2Code+gtSeg

Questions section:

1. Q: Typo in Table 3.

A: We have updated Table 3, Cls. should indeed be Rel.

2. Q: Reason for larger error in Tab.3 “OBB + Rot”

A: Because our error is measured in degrees, getting one joint wrong will lift the average error from 0.0 to 5.0-degree (because error angle is 90-degree and we evaluate on 9 test objects with 2 joints each.

3. Q: how to determine parent/child

A: The model first predicts a parent body (this is the ‘root_geom’ prediction in Figure 4), and every other new joint prediction is connected to the root geom. Note that this is more challenging for objects like multi-drawer/door cabinets, because the LLM needs to infer which of the input OBBs is the root geom based on its sizes and relative orientation, but matters less for two-part objects like laptops.

Dear Reviewer S2Gc,

Thank you for the detailed reviews and feedback. We hope the following additional clarifications, additional results, and the updated submission pdf will address your raised concerns.

• Q1: Why is PARIS reported performance lower than the original paper. PARIS jointly optimizes object parts and the motion model based on a single rendering objective. As a result, their optimization is unstable and has large performance variances across different trials. While PARIS reports their best results across multiple trials in their paper, for fairness we report their average performance across 5 trials with random initializations.

• Q2: how to handle more complex-shaped objects.
  A: Our pipeline can be divided into two parts: 1) geometry reconstruction (includes 1.1 part segmentation and 1.2 shape completion); 2) articulation prediction. We’d like to clarify that our method for 1) can handle arbitrary geometry shapes (e.g. scissors, faucet handles), if these objects are put into the training set for fine-tuning the part segmentation model and shape completion model. However, for 2), because we use the OBB formulation and select OBB edge as rotation center, our method can handle sliding joints (e.g. a sliding oven rack) but will be inaccurate for hinge joints where the joint is not overlapping with any OBB edge (e.g. scissors). We have updated the submission pdf to better clarify our assumption and the subsequent limitations. Since cuboid-shaped objects (cabinets, doors) are commonly seen, we still believe it’s of much value to reconstruct those objects, especially those with many parts, which our method handles much better.

• Q3: Timing of our method. A: Our inference-time compute requirement is indeed larger than end-to-end methods like CARTO. To be specific: during SAM-prompting phase, we sample in 3D then projecting each 3D point onto 2D points across each single RGB image, hence the number of SAM forward pass scales linearly with the number of input camera views, however, we can cache the image embedding from SAM for every RGB image, and only call the light-weight prompt decoder for additional prompt points. Additionally, the inference compute for LLM code generation is dependent on the number of object parts, and roughly scales linearly with the generation token length. The shape-completion model forwarding is single-pass, which can be batched and takes <1min for all parts. Overall, our system is slower at inference time, but we deem it an affordable price to pay since this formulation can handle arbitrary numbers of object parts, and takes advantage of the strong generalization ability from pre-trained SAM models. CARTO-like model is single-shot on objects with one joint, hence would require either modifying the architecture for model output head, or running multi-round interaction to handle more object parts. We have updated the A.1 Discussion & Limitation section to better discuss these timing constraints of our method (see A.1.2), and also added the missing citation for CenterSnap.

• Q4: Details on pre-training datasets & code availability. Please see Appendix 4.2 + 4.3 for more details on dataset preparation and model training details. We use a subset of PartNet-Mobility object assets and did our own RGB rendering and code conversion from raw URDF to OBB-relative MJCF files. Please let us know if you have further detailed questions. We will open-source code upon paper acceptance.