SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects

Thank you for appreciating the importance of our work and providing insightful feedback.

W1 & W2 & Q1: Please refer to the general response.

Thank you for appreciating the importance of our work and providing insightful feedback.

W1 "Results on real pictures?": Please see the general response and refer to Fig. 10 for qualitative results.

W2 "The retrieval won't work if some shape in the image are not in the model library": Our primary focus is not on detailed geometry reconstruction but on addressing the challenge of recovering the geometric and kinematic structure of articulated objects from a single input image. We agree that improvements in detailed geometric reconstruction are a promising direction to investigate, but this is beyond the scope of our paper.

W3 "Cannot correct the wrong graph": We design the graph extraction as an intermediate step so that the user can easily correct the graph and control/edit the output object in an interactive generation fashion. In contrast, prior work addresses the task by producing the graph and other shape/motion attributes altogether. Their end-to-end design limits user intervention, making it difficult to re-generate the output by incorporating changes. Please also refer to the general response for further discussion.

Q1 "What will the attention map look like when removing $L_\text{fg}$": The visualization of the attention map is shown in Fig. 11 in the appendix. We observed that without this loss, the attention tends to be scattered and occasionally distracted by the noisy feature in the background.

Q2: See response for W1.

Q3 "How to switch between different outputs due to the ambiguity? How to change the generation results? ": The ambiguity in geometry and articulation (as shown in Fig. 1) is reflected in a distribution over the output objects. To get multiple outputs with variations, different random noise initializations can be used to produce multiple outputs, from which the user can select the desired ones. Or the user can specify the attribute (e.g., semantic label) for certain parts to control the generation, as shown in Fig. 12.

Q4 "Are the ambiguities taken into account during evaluation?": For quantitative evaluation, each test image is associated with only one ground-truth object in the dataset, so our metric cannot evaluate whether a different output constitutes another plausible solution or not. To compensate for this challenge in quantitative evaluation, we conducted a user study to evaluate the overall realism based on human intuition (see line 458 in paper). We agree that it is valuable to take this ambiguity into account for future metrics.

Thank you for recognizing the value of our paper and providing insightful feedback.

W1 "Coarse-to-fine is not uncommon": Prior work produces the graph and other shape/motion attributes at once. This design limits user controllability/editability, and makes it difficult to correct errors or incorporate changes and re-generate the output. The general concept of a coarse-to-fine strategy is not new, but to the best of our knowledge, we are the first to introduce such a pipeline for image-conditioned articulated object generation.

W2 & W3: Please refer to the general post above.

W4 "Can the module retrieve complex and fine-grained parts?": Our focus is not on detailed geometry reconstruction but on recovering the geometric and kinematic structure of articulated objects from a single image. As part geometries in household articulated objects are often structurally aligned and semantically shared, we adapted a retrieval approach based on structural and semantic matching of parts that is similar to prior work. This approach works fairly well with few cases where the output is unrealistic due to flaws in the part geometry. We agree with the reviewer that further improvements in part geometry reconstruction is a promising direction to investigate.

W5 "Minimal gains on cCF and $L_\text{fg}$": We add cCF so that our model avoids taking the object category as a mandatory condition. Having this category-agnostic training helps learn common features across object categories, which contributes to better generalization on unseen data, as shown in Table 4. We add the $L_\text{fg}$ loss to minimize the distraction of the noisy DINOv2 features in the background. We visualize the attention map in Fig. 11. Its contribution to the overall loss in our ablation is less prominent because our testing images have pure background and the noise is not significant.

Q1 "What does r_i represent?": r_i is the motion range for each part. Thank you for pointing this out. We have fixed the typo in line 202.

Q2 "Explain the phrase in more detail": What the diffusion model outputs is essentially a list of part attributes, where each part corresponds to a node in the graph. The ordering of parts in the list is in accordance with the conditional graph represented as an adjacency matrix. The model generates parts in the order indicated by the graph, with relationships between the generated parts accordingly respected. For example, a root node in the graph should not contain the attributes of a handle.

Q3 "Why graph accuracy of NAP-ICA-GTGraph is not 100%?": The difficulty of controlling the output using a graph is a known weakness of NAP that has been discussed by Liu et al. [1]. This happens because the graph topology produced by NAP is post-processed by a minimum spanning tree algorithm based on the size of generated parts, which can ignore the node-edge existence specified by the user input. In our experiments, we observe that specifying the graph for NAP as a condition can even worsen its performance.

Q4 "Unclear about the Acc% metric": We evaluate the correctness of the graph by checking whether its topology and edge directions exactly match that of the ground-truth graph. Then we report the percentage of the correct graph across the test data.

Q5 "Do you use one token for each part?": Each part attribute is a separate token, following our base model. The tokens have positional encodings so that their association with the part is captured.

Q6 "What is the difference between iCF and ICA?": ICA is the cross-attention module that injects the image condition into the diffusion model. We apply the classifier-free training for images iCF by dropping the image condition (by replacing the input patch feature with a dummy one), at a particular ratio. In this way, we can enforce stronger image guidance during inference by giving it higher weights as mentioned in line 303.

[1] Liu, J., Tam, H. I. I., Mahdavi-Amiri, A., & Savva, M. (2024). CAGE: Controllable Articulation GEneration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17880-17889).

Thank you for appreciating the importance of our work and providing insightful feedback.

W1 "No graph correction? The graphs depend on the prior distribution of the object collection?": Please refer to the last point in the general response for the graph correction mechanism. It’s unknown how the prior knowledge is used by GPT-4o for graph prediction, but we prompt it to recognize all the parts from the image first, and then connect them (see Appendix A.2). We observe that errors are mainly due to over- or under-recognition of parts. “Invalid part connection” errors such as connecting two drawers together are uncommon.

W2 "Would the 14x14 image patches be too coarse for small parts?": We observed empirically that 14x14 patches work well when parts are not too cluttered to be distinguished. We attribute this to the use of DINOv2 for the detailed feature extraction. We agree that the patch resolution could be a bottleneck when generalizing to more complex objects (e.g., with a much higher number of parts and highly compact part arrangement), especially when small and thin parts considered.

Q1 "Clarify the rendering state in the dataset. What if the object is under different states?": Thank you for the suggestion. We have clarified in our revision (in line 339) that we render the resting state of the objects from the dataset to train our model. As suggested, we tested our model on images of objects in arbitrary articulated states in a zero-shot manner. Please see Fig. 13 in the appendix. Overall, we generate plausible objects when the parts exhibit a reasonable amount of articulation and part visibility. However, our approach struggles to produce consistent results with the input image when the parts are in extreme states (e.g., with doors widely opened).