MIDGArD: Modular Interpretable Diffusion over Graphs for Articulated Designs

Part-level generation

Part-level generation requires conditioning on the articulation graph, since the appearance of a part depends on its role within the graph. In future work, we aim to test further possibilities to condition the part-generation on already existing parts, e.g. by framing it as a shape completion problem. However, we chose this approach since it also has several benefits:

• As already noted by the reviewer, it allows for part-level control via text or images.

• learning to generate object parts is arguably simpler than learning to generate arbitrary objects. Generating a complex high-resolution object is still an open challenge in 3D generation, but generating basic components of objects such as wheels and boxes is feasible.

Unconditional generation

NAP is already a rather compact and powerful model but the quality measurement metric (Instantiation Distance or ID) has limitations, as it mixes shape geometry with structure and motion evaluations; two properties that should be separately measured. Since points are sampled randomly for the CD, this ignores unrealistic states of small parts, such as the lid of the bottle in Fig 4. This is an issue that was also raised in CAGE. However, for a fair comparison to NAP's results, we used their evaluation protocol. We provide a qualitative comparison between NAP and MIDGArD in the attached PDF, Figure (B). We also provide further quantitative results to show the improvement of our approach in terms of physical plausibility, we compute the distribution of joint types and compare the distribution among our generated objects with the ones from NAP and the real data. As shown in the main rebuttal section, our model learns to match the distribution well. Since we determine the joint ranges based on the type, this directly leads to an improved motion plausibility. Finally, it is worth noting that one of our contribution is the higher level of control achieved with our approach, offering novel ways to guide the generation of articulated objects. Please refer to the attached PDF, specifically Figure (A) in the main rebuttal section.

Simulation ready sounds overclaimed.

By "simulation-ready", we refer to our contribution of providing the possibility to export the result to Mujoco, which is indeed a physically accurate simulation environment. Building a MuJoCo environment with multiple articulated assets will be straightforward with our codebase, which is why “simulation-ready” seemed to be a suitable term. Nevertheless, we do not want to neglect the huge effort involved with building full simulations, so we will weaken “simulation-ready” to “simulatable” in our revised manuscript.

It seems that the parts are not well connected

We believe the reviewer refers to the fact that screws and other connectors are not shown in the simulated videos. The connectors are omitted since they are not part of the ground truth dataset. However, adding those would be straightforward based on the predicted edge features in the graph (joint type & Plucker coordinates). The crucial part of the simulation in our point of view is the motion of the object parts, which is realistic in our videos. Please let us know if we have addressed your concerns or if any further clarifications would be useful.

There is no explicit physics enforcement.

We agree that there is no strict enforcement of physics in the framework; however, this is a rather difficult endeavor as it usually requires the interaction of the learning pipeline with a physics engine. Meanwhile, several components of our framework ensure physical plausibility:

• Our bounding-box constrained generation approach (with subsequent part scaling and orientation) ensures that the parts fit within the object in a physically plausible way.

• We enforce realistic kinematics by specifying the joint ranges based on the joint category. As Fig 4B shows, this works much better than the approach taken in NAP, i.e., predicting the joint range directly.

Frontal-view images

While we use a specific perspective for training and testing in the initial submission, we provide additional results here showing that our model generalizes to image inputs from other perspectives (see Fig. D in the main rebuttal section). The PartnetMobility dataset is already normalized such that all parts are shown from the front (if such asymmetry even exists). However, rendering with a camera angle of [0,0,0] is problematic since some objects are not recognizable from the front; e.g., a wheel may appear as a rectangle from the front. Therefore, our “frontal view” refers to a rendering from an axis angle [\phi/6, \pi / 12, 0] which usually shows the relevant part geometry. We've added this clarification in the manuscript. Due to the variety of parts in our training dataset, we hypothesize that our model should also generalize to images from other views. We tested this by conditioning the model on images from a randomly sampled perspective.

Are the images in Suppl. Fig. 5 of "frontal view"?

Yes, the images in Suppl Fig 5 are rendered with the same angle as the images used for training

Bounding boxes

In the articulation graph, the part-bounding box is represented by three parts: 1) its center, 2) its size 3) its rotation. These three components are generated for each part in the structure generator. The shape generator, in turn, is conditioned on the desired size (2) and generates a centered object of the correct size. The rotation (3) is applied afterwards. Note that this approach allows to generate parts in arbitrary orientation, in contrast to previous approaches (CAGE), that assumes that all parts have axis-aligned bounding boxes when the full object is in resting state.

Thank you for the constructive feedback that help us to improve the clarity and quality of our paper.

Writing (comment: I would limit the contribution bullets to the technical contribution of this work rather than focusing on results and open sourcing):

Thank you for your suggestion. We fully agree and will replace bullet points 4-6 with the following technical contributions:

• An approach for constrained shape generation within oriented bounding boxes that improves alignment of the kinematic links by 17% on average.
• A pipeline to create fully simulatable assets with an interface to Mujoco

Notation and clarity (comment: I would introduce proper notations for all steps discussed in section 3.3 as I feel it is hard to understand the setting just from the textual descriptions, e.g. what is input, what is output. I have a similar concern regarding section 3.4)

Thank you for this suggestion to improve the clarity of the paper. We will rewrite this part to improve the description of the setting. Specifically, we will add that we train the model to learn the distribution of articulated object graphs, by applying a denoising diffusion process:

• Input: Noisy Graph $\boldsymbol{\tilde{G}}_N = \left\lbrace \boldsymbol{\tilde{x}}, \boldsymbol{\tilde{e}} \right\rbrace$ with node and edge features as introduced in section 3.3. For conditional generation, the input is a partially noisy graph, where certain node or edge features are masked.
• Output: Denoised articulation graph $\boldsymbol{G}_N = \left\lbrace \boldsymbol{x}, \boldsymbol{e}\right\rbrace_N$

Similarly, for section 3.4, we will add the following description:

The aim of the structure generator is to learn the conditional probability distribution $P(z_i | a_i, b_i, d_i, r_i, t_i, g_i)$ for generating the latent representation $z_i$ of a part geometry's Signed Distance Function (SDF) using a diffusion model, conditioned on inputs $[a_i, b_i, d_i, r_i, t_i, g_i]$ as explained in section 3.3.

This is achieved 1) by transforming $a_i$ and $b_i$ into a text description and encoding with the BERT model, 2) by transforming the image (decoded $g_i$) via a ResNet and 3) by applying constrained generation with $r_i$ and $t_i$ as explained in section 3.4

Video (comment: I would have expected to also see some video results of the generated and animated objects. While this is not strictly required, it would have made the exposition much more complete.) :

We have provided videos in a repository here: https://anonymous.4open.science/r/MIDGArD-E1DE/README.md (see folder “gifs”). We apologize if the reference to this repository was unclear in the paper. Upon publication, we will provide a proper website with these examples in addition to the open-source code.

Thank you for highlightning the strength of our approach. We are delighted that you find the MuJoCo simulations in the supplementary repository impressive and recognize the significance of our technical improvements, including shape representation, Plücker manifold encoding, and bounding box alignment. These were key motivations in our effort to advance the field of articulated object modeling.

Conditioned generation results: while the part-level multimodal information is used in shape encoding, it would be better to show more conditioned generation results using this information.

We acknowledge your request and have included additional conditioned generation results in the attached PDF, specifically Figure A ("Conditional generation"). This figure showcases our model's capability to generate consistent articulated assets based on supplied part features in form of image and text. In addition to the experiment in Figure 5 of the original submission, this figure shows two settings: Generating an articulated object solely based on image and text input, and generating the articulated object based on image, text and bounding box input. The latter is comparable to the "Part2Motion" setup in NAP. However, in NAP one would have to provide the full geometry for each part, whereas we demonstrate the same capability just based on human-interpretable input features.

Furthermore, we have extended our experiment on image-based conditioning (Figure 3 and 5 of the original manuscript) by another experiment using images from arbitrary viewpoints for conditioning (Figure D in the attached PDF). This shows the generalisation capability of our approach.

We hope these experiments match the reviewer's expectations and we welcome further suggestions to improve our empirical results.

Open source: it's not clear whether the shape part encoding training data will be publicly released or not.

Thank you for your suggestions. We commit to open-source the data (text, bounding box etc) and source code upon the paper's acceptance facilitating reproducibility and further research in this area. This commitment aligns with our goal of promoting transparency and accessibility within the research community.

Weakness 1.1 While our structure generator builds up on NAP \cite{lei2023nap} and also applies a denoising diffusion process on the graph, the representation and generative process are fundamentally different: 1.) Our approach modularizes articulated asset generation. This fundamentally diverges from the approach taken in NAP which aims to learn both shape and structure generation within one model. While both the shape generator and the structure generator build up on previous work, they are substantially modified to enable their interplay for generating articulated assets. Only this modular approach enables fine-grained control on the object level (Table 1 and Fig. A in the attached PDF) and part level (Fig. 3, Fig. 5 appendix). 2.) The graph representation is different to NAP's representation. In the node features, the geometry encoding is removed, the image encoding and categorical variables are added, and bounding boxes are replaced by oriented bounding boxes. Only the part-existence indicator is the same. For edge features, joint category is added and we directly denoise on the Plücker manifold. 3.) These changes enable more control and enhances intuitiveness for the use (see examples for conditional generation, Figure A in the attached PDF).

Weakness 1.2 We respectfully disagree. As outlined in Section 3.4, we do not only supply the graph as another conditioning input, but modify the SDFusion pipeline to generate only the difference to the bounding box (see Figure 2). Thus, the core innovation within the shape generation pipeline is the introduction of a bounding box constraint. To the best of our knowledge, this is novel. This constraint, combined with the use of oriented bounding boxes (OBBs), enables generation of geometries that are more closely aligned with the articulation solutions proposed by the structure generator, as demonstrated in Table 2. The synergy between conditional graph generation, and constrained 3D generation based on OBBs results in realistic articulated assets with appropriate dimensions, as shown in Fig. 4A.

Weakness 2.1 Per the reviewer's suggestions, we've added a side-by-side qualitative comparison with NAP (see general author rebuttal). Figure B in attached PDF shows that our model 1) improves the quality of the geometry (see fan), 2) improves the physical plausibility (see laptop motion), and 3) generates more consistent and realistic shapes (see both globes).

Weakness 2.2 We acknowledge the concern raised regarding the fairness of the comparison, and will clarify this in the revised manuscript. Our aim was to demonstrate the conditioning capabilities of our approach, which are not present in NAP. We do not see a feasible alternative that would ensure a fair comparison and are very open to a fairer comparison that can highlight MIDGArD's capability of conditioning (compared to the lack of any text- or image-based conditioning in NAP) as one of the novelties and contributions of our work.

Regarding the suggestion that NAP should have been given the latent code extracted from the ground truth part geometry: This would create an unfair advantage over our method, which relies solely on single-view images. Additionally, the practical applications where part-geometry is readily available are quite limited, thus the utility of such a capability is uncertain. Leaving this drawback aside, for a fair qualitative comparison one can consider the Part2Motion experiment in NAP (\cite{lei2023nap} Fig 5) as the counterpart to our results for text- and image-based conditional generation provided in the attached PDF Figure A.

Weakness 2.3 See general response for experiments and results.

Question 1 By "consistency across the graph," we refer to the semantic coherence of part relationships within the articulated structure. For instance, a "bottle" is unlikely to contain a "drawer". We found that adding categorical information to the graph improves the consistency. In MIDGArD-generated data, we found that our model generates 100% valid examples (see general author rebuttal).

Qualitatively, this advantage of our approach is seen in the failure cases (attached PDF Fig. C). In NAP, the lack of categorical information leads to objects of mixed parts that do not fit together. In our approach, there may be issues with the geometry or kinematics, but the object parts remain consistent.

Question 2 A fair comparison with CAGE is not possible, because CAGE was designed for a very limited setting, i.e., (1) providing shape geometry via part retrieval instead of part generation. This method does not generalize and leads to failure cases where parts overlap and don't fit (see \cite{liu2023cage} Fig 10), (2) assuming canonical poses of the objects and axis-aligned bounding boxes. Thus, the experiments in CAGE are restricted to 8 categories from PartnetMobility (3) requiring actionable parts, e.g., doorknobs and handles, which were added by the authors of CAGE and only for 8 categories.

Question 3 We have corrected and clarified the mentioned line. Indeed, for NAP we refer to the latent encoding the geometry. For CAGE, it is rather the part-retrieval procedure that prevents control on this level. Our framework allows users to specify part-level details through images and text, e.g., requiring a lamp to consist of a "lamp stand" and a "lamp shade" and guiding the style of the lamp shade with an image. This control mechanism is not available in prior work.

Question 4 It generalises. See general response for results.

Limitation 1 One could leverage existing image segmentation pipelines to automatically extract relevant images from a single picture. Also, one image oftentimes suffices (e.g. sunglasses), 3) we are planning to provide a library of part images to assist guidance.

Limitation 2 As shown in the attached PDF Figure A, the user can directly input images and text instead of post-editing.