DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Data

Thank you for your thorough review and constructive feedback on our manuscript. We appreciate the time and effort you have invested in evaluating our work. Your detailed comments and suggestions have been invaluable in helping us improve the paper. Below, we provide point-by-point responses to address your concerns and questions.

Q1 Details of Graph Reasoner

Details of CoT

We illustrate the prompt of Graph Reasoner in supplementary materials. We will add more details of step-by-step CoT in revision version.

Add Graph into Diffusion Blocks

We convert the predicted articulation graph into a binary adjacency matrix $\mathbf{A} \in \{0,1\}^{N \times N}$, where $N$ is the number of predicted parts. Each entry $\mathbf{A}_{ij} = 1$ indicates the existence of a valid articulated connection from part $i$ to part $j$.

This matrix is used as a structural prior to guide the attention computation in the diffusion model. Specifically, given the latent feature matrix $\mathbf{X} \in \mathbb{R}^{N \times d}$, the attention is computed as:

$\text{Attn}(\mathbf{X}) = \text{Softmax}\left(\frac{(\mathbf{XW}_Q)(\mathbf{XW}_K)^\top}{\sqrt{d}} + \log(\mathbf{A} + \epsilon)\right) \cdot (\mathbf{XW}_V)$

where $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V \in \mathbb{R}^{d \times d}$ are learnable projection matrices, and $\epsilon$ is a small constant (e.g., $1\text{e}^{-6}$) to avoid taking the logarithm of zero. The term $\log(\mathbf{A} + \epsilon)$ serves as a structural attention bias, encouraging the model to focus on physically valid connections during generation.

Details of LLM

We use the API of GPT-4o in Graph Reasoner. The parameters of GPT-4o is about 200B acoording to the official report.

Q2 Generalization Ability of DIPO

Experimental Settings

Our experiments include totaling seven diverse categories from the PartNet-Mobility dataset, which are Storage Furniture, Table, Refrigerator, Dishwasher, Oven, Washer and Microwave. We select these categories to follow the experimental setting of SINGAPO [?] for a fair comparison. There are no strong category-specific assumptions on our diffusion model and graph reasoner.

In addition, the ACD dataset provides a finer-grained categorization of object classes. The detailed mapping between PM and ACD categories used in our evaluation is shown below.

Complex Objects

However, our method may fail in some extremely complex objects like robots. Moreover, assets such as keyboards with a large number of visually similar and spatially redundant parts pose challenges for feature extraction using DINOv2, as the part-level features may lack sufficient discriminability. To address this, a promising direction is to incorporate segmentation-aware priors. In particular, we plan to explore integrating SAM-derived segmentation masks to guide part proposal and filtering, enabling more robust articulation reasoning in such densely repetitive structures.

Occlusion and Unchanging Appearance in Dual-State Images

In cases where the articulated-state image provides limited additional information due to occlusion (e.g., a dishwasher tray) or minimal appearance change (e.g., a globe with uniform texture), the dual-state input may not fully reveal articulation differences. However, our input setting inherently contains at least as much geometric and structural information as single-image methods like SINGAPO, ensuring a comparable performance lower bound.

To better handle occlusion, we consider incorporating sparse multi-view images (e.g., 2–3 views) as a future extension, which can alleviate visibility issues without incurring significant data collection cost.

In addition, nearly identical dual-state images can still provide useful priors. For example, spherical objects like globes often imply rotation around a central axis, which can guide plausible articulation inference.

Q3 Impact of Partially Opened Articulated Images on Joint Range

Our method remains robust to partially opened articulated-state inputs — that is, the predicted joint type and axis remain stable. However, incomplete articulation may influence the estimated joint range, since the model learns this from the observed displacement between states.

For practical use, we recommend using fully opened articulated images to better reveal the maximum joint extent and improve articulation estimation.

To assess the effect of articulation coverage, we additionally render a variant of the test set where the articulated-state poses are randomly sampled (not fully opened), and evaluate our model on this setting. The results are illustrated in the table below.

Q4 Training and Inference Time

Training our full pipeline on eight NVIDIA RTX 4090 GPUs takes approximately 23 hours.

During inference, we use a pair of $224\times224$ images as input and predict an articulated object with 5 parts. The time consumption of each stage in the inference pipeline is listed in the table below.

I appreciate the effort and detailed response provided by the authors in the rebuttal. I believe that my concerns have been well addressed. I therefore maintain my "Accept" rating.

We sincerely thank Reviewer kNAa for their constructive and detailed feedback. We greatly appreciate the reviewer’s recognition of the technical novelty of our proposed framework, the introduction of the PM-X dataset, and the use of Chain-of-Thought (CoT) reasoning. We also value the reviewer’s suggestions regarding reproducibility and generalization. Below, we provide clarifications and additional evidence to address the concerns raised.

Q1 Reproducibility Concerns

We appreciate the reviewer’s concern regarding reproducibility. To facilitate understanding and verification, we provide demo code and pretrained model checkpoints in the supplementary material.

Upon paper acceptance, we will release the full codebase, including all modules (diffusion, CoT-based graph reasoner, layout predictor), as well as the PM-X dataset, trained model weights, and detailed documentation covering implementation, prompt formats, training settings, and data preprocessing steps.

We are committed to ensuring the method is easy to reproduce and build upon.

Q2 Generalization Beyond Cabinet Category

Although the paper primarily displays visual results on cabinet-like objects, our experiments are conducted on seven object categories from the PartNet-Mobility dataset: Storage Furniture, Table, Refrigerator, Dishwasher, Oven, Washer, and Microwave. All reported quantitative results are evaluated across these seven categories. In addition, the ACD dataset provides a finer-grained categorization of object classes. The detailed mapping between PM and ACD categories used in our evaluation is shown below.

Correspondence Between PM Categories and ACD Categories

We chose to emphasize cabinet-type objects in the qualitative figures because they typically exhibit the most structural complexity and better demonstrate the advantages of our approach. However, we agree that including more visualizations from other categories would help illustrate the model’s generalization ability. We will add such examples in the revised version to provide a more complete comparison.

Moreover, we report quantitative results across all seven categories in the ACD dataset, as shown in the table below.

Comparison of DIPO and SINGAPO Across Categories and Metrics (↓ means lower is better)

We sincerely thank Reviewer r4pp for the thorough review and insightful suggestions. We appreciate your recognition of the framework’s scalability and the clarity of our methodology and writing. Below we address the concerns regarding evaluation scope, VLM-related robustness, and the design choice behind LEGO-Art in detail.

Q1 Successful Rate of Filtering

We do not perform any manual filtering for the PM-X objects generated by LEGO-Art, since they are used exclusively for training data augmentation, where a certain degree of noise is tolerable. While some noisy samples may persist, we empirically find that the large-scale diversity introduced by LEGO-Art still benefits the training of DIPO, and the network learns to ignore occasional outliers.

The LEGO-Art pipeline may fail in two stages:

1. Grid Generation Stage: When converting VLM-generated descriptions into grid-level 3D layouts, hallucinated may lead to invalid or unparseable object layouts. The success rate at this stage is approximately 68.24%.
2. VLM-based Filtering Stage: We apply multi-stage visual filtering using CLIP and BLIP to ensure geometric plausibility and articulation consistency. The pass rate of this filtering stage is around 85.11%.

Q2 Categories Limitation

Although most visualizations in the main paper focus on cabinet-like categories for clarity, our method is not restricted to cabinet objects. In fact, our experiments include totaling seven diverse categories from the PartNet-Mobility dataset, which are Storage Furniture, Table, Refrigerator, Dishwasher, Oven, Washer and Microwave. In addition, the ACD dataset provides a finer-grained categorization of object classes. The detailed mapping between PM and ACD categories used in our evaluation is shown below.

Correspondence Between PM Categories and ACD Categories

We select these categories to follow the experimental setting of SINGAPO for a fair comparison. There are no strong category-specific assumptions on our diffusion model and graph reasoner.

Moreover, most of the previous work like SINGAPO, URDFormer, CAGE and ACD dataset are focus on openable containers, as these are the prevalent in indoor scenes. However, there are still many issues to be resolved in the construction of articulated openable containers. We believe that the progress in these categories we achieved is meaningful and will be beneficial to the community. We further provide the performance on ACD datset across all categories to demonstrate the generalization ability of DIPO.

Comparison of DIPO and SINGAPO Across Categories and Metrics (↓ means lower is better)

Q3 Evaluation of Graph Reasoner

Some of our statements may have caused you a misunderstanding. The successful rate of Graph Reasoner is illustrated in the Graph Accuracy item in Tables 2 and Table 3. We list them separately in the table below. Moreover, during extensive experiments, we did not encounter any malformed or structurally invalid graphs that would cause downstream inference failures.

Q4 Concerns about LEGO-Art

The LEGO-Art pipeline is proposed to build complex articulated object. In our daily life and experimental settings, we observe that most structurally complex objects that contain many moving parts are in boxy shape, e.g, large storage cabinet. This boxy shape makes them naturally compatible with a grid-based construction scheme, which enables structured, interpretable, and scalable generation.

Motivated by the need to provide supplementary training data featuring such complex structures, we introduce LEGO-Art as an automatic pipeline to synthesize articulated objects with diverse part layouts and articulation semantics.

Considering the hallucination issue of VLMs, we do not let the model predict precise continuous coordinates directly, but instead constrain the prediction within discrete grid coordinates.

Thank you for your detailed review and the time you invested in evaluating our work. We appreciate your constructive feedback, which has helped us identify important areas for improvement. We have carefully addressed each of your concerns and believe our responses demonstrate the value and rigor of our approach. Below, we provide point-by-point responses to your questions and comments.

Q1 Clarification on RS-/AS- Metrics

In all evaluation, our model are consistently conditioned on both resting-state and articulated-state images as input. The RS- and AS- metrics instead correspond to the evaluation articulation state, not the input modality. Specifically:

• RS-: We evaluate the predicted 3D object in the resting state, and compare it against the ground-truth shape in the same resting state.
• AS-: We evaluate the predicted 3D object in the articulated state, and compare it against the ground-truth shape in that articulated configuration.

We will revise the manuscript to make this clearer in both the table captions and main text.

Q2 Clarification on experiment categories

Our experiments are not limited to only Storage Furniture and Table. We also include other articulated object categories from the PartNet-Mobility dataset, including Storage Furniture, Table, Refrigerator, Dishwasher, Oven, Washer and Microwave, totaling seven diverse categories. In addition, the ACD dataset provides a finer-grained categorization of object classes. The detailed mapping between PM and ACD categories used in our evaluation is shown below.

Correspondence Between PM Categories and ACD Categories

This seven categories are selected to follow the experimental setting of SINGAPO [Liu et al., ICLR 2025], ensuring a fair and direct comparison. We also mentioned this point in supplementary materials.

We further provide the performance on ACD datset across all categories to demonstrate the generalization ability of DIPO. We will highlight our experimental settings in the revised version prevent readers from misunderstanding the generalizability of our method.

Comparison of DIPO and SINGAPO Across Categories and Metrics (↓ means lower is better)

Q3 Clarity of LEGO-Art Pipeline

We would like to clarify the role of the AI Agent Designer (Lines 179–182). The AI Agent Designer is not part of the runtime inference pipeline. Instead, it is responsible for crafting high-quality system prompts for the three core agents in the LEGO-Art synthesis pipeline (e.g., the Description Roller, Layout Generator, and Visual Filter). This design phase ensures coherent collaboration among agents, but does not participate in the actual generation process.

For the description of each step, we provide the full system prompts and workflow logic for each agent in the supplementary material. In the revised version, we will incorporate key details into the main paper and expand the descriptions of each module, including their responsibilities and interaction patterns, to enhance clarity and self-containment.

Moreover, we will detail how we design the prompts of each agent with the help of LLMs in a chatting manner in the supplementary materials.

Q4 Contribution of PM-X dataset

1. In Sec. I Introduction, we clarify that the motivation of proposing the PM-X dataset is to provide redundant training data of complex objects. Meanwhile, we only use the PM-X for model training.
2. Although PM-X currently includes only two categories (StorageFurniture and Table), this was a deliberate choice rather than a limitation of the synthesis method. In seven categories we studied, we found that these two categories exhibit the most complex and diverse articulated structures — such as cabinets with nested arrangements, multi-panel doors, and densely connected subcomponents.
3. The role of the PM-X dataset can be demonstrated in Tab.5 and Fig.6. The PM-X improves the performance for each variant in the ACD testing set. Moreover, performance increases when we add more training data from PM-X.
4. We further build 100 objects by LEGO-Art pipeline for a standalone evaluation. We compare the performance of DIPO trained with and without PM-X on the validation set. The results show that incorporating PM-X significantly improves the performance on complex objects. The results are illustrated as follow.

Performance comparison of DIPO trained with and without PM-X on the validation set

Q5 Novelty of LEGO-Art

While we leverage LLMs and VLMs as foundational tools, our method goes beyond simple integration. A central contribution of our work is a grid-based spatial reasoning framework that overcomes the limitations of direct coordinate prediction from LLMs.

A key innovation of our framework lies in the design of a grid-based layout prediction mechanism. Instead of asking the LLM to generate precise 3D coordinates, which often leads to hallucinations, we constrain the placement of parts to a discrete 3D grid. We then convert the predicted grid layout into accurate object geometry using a custom mapping script. This design enables reliable structure generation while retaining the controllability and flexibility of language-based input.

We sincerely appreciate the reviewer’s positive feedback and are thrilled that the response has addressed the concerns raised. We are also grateful for the reviewer’s decision to raise the score to 4, acknowledging the contribution of our work.

Regarding the concerns about the PM-X dataset, we would like to provide additional clarifications and updates:

1. Sufficiency of Two Categories for Training

We further train our DIPO with only PM-X and evaluate the performance on the test set of PM-X.

The results demonstrate that it is enough to achieve a favorable performance with only two categories of PM-X for training. Moreover, the performance will increase when we add the PartNet-Mobility (PM) dataset for training, as simple data is beneficial to model convergence.

However, our LEGO-Art is capable of generating simpler structures, as it does not solely rely on complex data but is flexible enough to handle a wide range of articulated objects, including those with simpler configurations.

2. Future Expansion of PM-X Dataset

In line with the reviewer’s suggestion, we acknowledge the potential of expanding the PM-X dataset to serve as an independent benchmark for articulated object generation. We have already added a test set for evaluation.

In future work, we plan to enhance the PM-X dataset by incorporating a broader range of complexity levels, including both simpler and more intricate articulated structures. Additionally, we aim to increase the dataset's size to achieve better generalization and more comprehensive evaluation. To further improve its quality, we will apply detailed manual curation to select high-quality and diverse examples.

These efforts will transform PM-X into a complete and robust benchmark that can be used by the community to evaluate and compare different approaches to articulated object generation. We believe this expansion will significantly contribute to the advancement of the field.

We are excited that our response has been well-received, and we look forward to implementing the suggested improvements. Thank you once again for the thoughtful feedback and your support of our work.