URDFormer: Constructing interactive Realistic Scenes from Real Images via Simulation and Generative Modeling

Main Weakness Points:

• How do we get the bounding boxes and segmentation results for real-world images? It is straight-forward to use masks in the training data because the training data is procedural. But how do we train the detection and segmentation models for that, especially for the parts? What dataset is used for training these two models and how about the quantitative performance.visualization of these models?
• It is more like using data-augmentation (driven by generative models) to train a model for URDF prediction. However, as mentioned in the paper, we do not want the object hierarchy changes too much. That means the generative model will not do much more than applying new generated textures (from $D_{object}$) and object replacement (from $D_{scene}$). This will make the training data lacking of part-level hierarchy variation, which is the essential part of procedural generation. Therefore, I doubt its generalization capability on real-world images, especially for the objects which has different part-level hierarchy but still belong to the same category.

Summary

Apart from the engineering part (diffusion-based renderings), the core contribution appears to be the conversion of a hierarchical part-based segmentation to an URDF file (parent, location, articulation). I belive this is simply not good enough for a main track paper, especially given the quality and quantity of the comparisons and metrics -- [1*] being the main contender.

Detailed issues

• limited testing data
  • a total of 380 web-captured images from two datasets (300 + 80) is compiled by the authors
    • furthermore, the accuracy of some dataset images is questionable, see Fig 6 lower right image - the pose of the objects is not accurate
  • lots of [large] parts datasets that could be used/compared against (e.g., [1*] -- closest to the scope of the paper, [2*], some from [6*])
• limited method comparisons
  • a pretrained BLIP2 is the only method tested and it is by far not the best one -- see [4*] for a more recent overview, lots of multimodal LLMs that could be prompted in the same way, or even better, use a hierarchical part segmentator such as [7*]
• metric
  • novel metric introduced while a standard [hierarchical] part metric such as PQ / PartPQ [3*] or even just a mAP + mobility metric (see [1*]) can be used
• weak baselines
  • I would argue that it is trivial to assess that just rendering 3D object with single colors or fixed texture would yield a lower quality result compared to the generative-based variety; generative models have been used before to enhance semgentation capabilities, for example in part-based[5*] or more recently few shot image segmentation [6*] -- and arguably this paper is solving the same task
• limited novelty / experiments
  • arguably, it is easy to convert to an URDF format once the part boxes and hierarchy are estimated with an existing hierarchical part-based method, especially for the simiplified scenario considered by the authors (object-centric crop); there are even open vocabulary hierarchical segmetation methods that could be used as a baseline before the URDF conversion (i.e., [7*]); no attempt has been made in this regard.
  • disregarding the URDF ouptut, the problem can be modeled as a part-based domain adaptation from synthetic to real, which is what happens in [8*]
  • C.2 appendix -- a set of methods listed but summarly dismissed

[1*] Mao, Y., Zhang, Y., Jiang, H., Chang, A., & Savva, M. (2022). MultiScan: Scalable RGBD scanning for 3D environments with articulated objects. Advances in Neural Information Processing Systems, 35, 9058-9071. https://github.com/smartscenes/multiscan

[2*] Mo, K., Zhu, S., Chang, A. X., Yi, L., Tripathi, S., Guibas, L. J., & Su, H. (2019). Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 909-918).

[3*]de Geus, D., Meletis, P., Lu, C., Wen, X., & Dubbelman, G. (2021). Part-aware panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5485-5494).

[4*] Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2023). A Survey on Multimodal Large Language Models. arXiv preprint arXiv:2306.13549. https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation

[5*]Eslami, S., & Williams, C. (2012). A generative model for parts-based object segmentation. Advances in Neural Information Processing Systems, 25.

[6*] Tan, W., Chen, S., & Yan, B. (2023). Diffss: Diffusion model for few-shot semantic segmentation. arXiv preprint arXiv:2307.00773.

[7*]Wang, X., Li, S., Kallidromitis, K., Kato, Y., Kozuka, K., & Darrell, T. (2023). Hierarchical open-vocabulary universal image segmentation. arXiv preprint arXiv:2307.00764. https://people.eecs.berkeley.edu/~xdwang/projects/HIPIE/

[8*]Liu, Q., Kortylewski, A., Zhang, Z., Li, Z., Guo, M., Liu, Q., ... & Yuille, A. (2022). Learning part segmentation through unsupervised domain adaptation from synthetic vehicles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 19140-19151).

1. While the proposed method generates the URDF of the scene, it lacks accompanying 3D mesh files. It would enhance the paper's value if the authors provided concrete examples illustrating the practical use of these generated URDF files in the absence of mesh files. Further, elucidating the benefits of augmenting the number of URDFs with concrete applications could strengthen the presentation.

2. The method's current design presupposes easy access to procedurally generated 3D scenes or 3D object datasets with kinematic structures. This assumption may not hold true, as these could represent the genuine bottlenecks in the approach. For instance, curating 3D articulated object datasets demands substantial human intervention. Procedurally generating diverse and plausible 3D scenes isn't a straightforward task either. How the diversity of these scenes and objects impacts the inverse module's performance? Additionally, a clearer exposition regarding how training scenes and objects are obtained in the experiments would be beneficial.

3. Given the challenges in acquiring 3D scenes and articulated objects with URDFs, the method's applicability seems confined to a narrow range of scene or object categories.

4. As per Table 1, employing "selected textures" already produces almost comparable results. This observation somewhat diminishes the perceived necessity and advantage of using 2D diffusion models.

5. The evaluation appears somewhat limited. Incorporating a broader range of baseline methods beyond the currently employed self-designed baselines would offer a more comprehensive assessment. For example, there is a line of work on predicting the kinematic structure of the articulated objects. They should also be discussed and fairly discussed.

From the shown experiments we can see that the predicted object parts are mostly boxes. Since this method is actually predicting the object category instead of reconstructing the geometry, the fidelity of the generated results is relatively low, especially for objects with more variety in shapes, like chairs. The proposed paper may explain and compare the differences between works like [1].

This method does handle the joints in an ad-hoc way. They use the mesh types such as ”left door”, and ”right door” to infer link axis and joint types. Joints should be decoupled from the geometry. Sometimes, right/left is not well-defined when there are multiple parts and the object is not axis-aligned. Moreover, how to distinguish sliding/hinge/ball joints could be a problem. Moreover, the joint axis is also an important property for simulation.

The generated results are seemingly okay but might not be simulatable in some cases. In the examples shown on the website, there are overlapping doors which can result in collisions. It might be better if there were some post-processing steps to improve the prediction quality.

[1] Deep Part Induction from Articulated Object Pairs. Siggraph Asia 2018