URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model

We sincerely thank the reviewer for their encouraging and positive assessment. We are especially grateful for your acknowledgment of the paper's clarity, the novelty of our technical approach in using a 3D MLLM for joint prediction, and the elegance of the integration. Furthermore, we appreciate your recognition of our strong quantitative results, particularly on OOD data, and the practical value of our ready-to-simulate URDF outputs for the robotics community.

W1: Lack of Error Bars

We sincerely thank the reviewer for this constructive feedback. We agree that providing a thorough analysis of result variance is crucial for demonstrating the robustness of our method.

We have now calculated the standard deviations for our key metrics, which we are happy to provide for clarification.

Table 1: Quantitative Results for Part-Level Link Segmentation.

Table 3: Physical Executability Rate (%↑).

Table 4: Ablation Study on Input Modality for Joint Parameter Prediction.

We will incorporate these updated tables with full variance reporting into the revision of our paper. Thank you again for your valuable suggestion, which has helped improve the rigor of our work.

W2: Post-processing Step

Thank you for highlighting this important evaluation aspect. To address your point, we have conducted a new experiment to quantitatively evaluate the final mesh output from our pipeline, including the point-to-mesh post-processing stage. We measured the geometric fidelity using the standard Chamfer Distance (CD) metric. Results are as follows. We will add this evaluation to the revision.

W3: Unreported Training Resources

Thank you for pointing out this omission. Our model was fine-tuned in 2.5 hours on a single NVIDIA A800 (80GB) GPU. We will add these details to the revision of the paper and make the code open source.

W4: Lack of Dynamic Parameters

We thank the reviewer for this insightful question, which allows us to clarify the well-defined scope of our work. Our primary focus is on a central challenge in robotic manipulation: reconstructing an object's geometry and kinematic structure from visual input.

This focus is consistent with the established scope of other state-of-the-art methods in this domain. Prominent works, including Articulate-Anything, Real2Code, and URDFormer, also concentrate exclusively on geometric and kinematic parameters. The estimation of dynamic properties (mass, inertia, friction, etc.) from vision alone is a distinct and highly challenging research problem, which the community typically treats separately.

We argue that providing a kinematically accurate model is, in itself, a highly valuable contribution. An accurate kinematic model alone is sufficient to meet the requirements of a vast range of robotic applications, including manipulation planning, grasping, and task simulation. In many real-world scenarios, a robot's own controllers (e.g., through impedance or admittance control) can adapt to an object's unknown dynamics during interaction, making the kinematic model the most critical component for planning and execution.

Therefore, our contribution is a robust and complete solution to the core kinematic reconstruction problem. While incorporating dynamics is an exciting future direction, our work establishes the essential structural foundation upon which such dynamic models could eventually be built.

Limitations: Failure Case Analysis

Thank you for this valuable suggestion. We will add a dedicated section to the appendix in our revision to include and discuss visualizations of typical failure cases.

We sincerely thank the reviewer for their positive feedback. We are particularly encouraged by your recognition of the novelty of our core idea, the valuable insights from our ablation study, and the strong experimental gains across segmentation, joint prediction, and simulation executability.

W1: Related Work Section

Thank for pointing out the imprecisions in our Related Work section. We will make the following corrections:

Rewrite "Research on [...] is a related but distinct field." to "The automated modeling of articulated objects for robotic manipulation is an active field of research, with several distinct methodological approaches."

Rewrite "[29] focuses on generating articulated simulation scenes from images, which is related but distinct from accurate individual object" to "[29] is fundamentally limited by its reliance on a hard-coded system to assign kinematic parameters and retrieve meshes based on the network's discrete part classification output, a method which compromises final geometric and kinematic fidelity."

W2: Input Pipeline

Thanks for in-depth thinking on our input pipeline design. We fully agree that the design should be principled rather than based on convenience.

Our core principle is to use the most suitable, highest-fidelity method available for each type and quality of input source. This ensures that the geometric information received by our core model is of the highest possible quality. The two input scenarios (real multi-view vs. single-view) are fundamentally different problems in nature and difficulty:

For multi-view input (using DUSt3R): When real, high-quality multi-view images are available, they contain the most reliable geometric information. In this case, the most direct and high-fidelity approach is to use a specialized multi-view 3D reconstruction algorithm. We chose DUSt3R because it is a classical method in this specific domain (reconstructing geometry from real multi-view images). Forcing the use of a generative method like LGM in this scenario would actually degrade the quality of the point cloud fed to our core model by introducing artifacts and uncertainties. It is unnecessary to "downgrade" real data to generated data before processing.

For single-view input (using LGM): When only a single-view image is available, the problem fundamentally changes from 'reconstruction' to 'conditional generation' or 'hallucination'. We must leverage a generative model to complete the missing 3D information. We chose LGM because it is a classical solution for this specific problem (generating 3D from a single view).

Regarding your suggestions for a unified pipeline:

On using "ImageDream + DUSt3R": This insight actually points to the internal workings of models like LGM. Advanced single-view generative models such as LGM already integrate modules similar to MVDream/ImageDream for synthesizing multiple views from a single one. Therefore, using the highly integrated and optimized LGM is a more rational and efficient choice than manually assembling a similar pipeline. On using "LGM in the multi-view case as well": As mentioned before, this would mean applying a tool designed for "generation" to a "reconstruction" problem, which is not the optimal choice.

W3

W3.1 & W3.2: Uni3D Baselines / Uni3D

Here we clarify the implementation and purpose of our two key baselines: ‘Uni3D w/ text’ simulates text-guided segmentation. Following the Uni3D paper, we freeze the backbone and train a feature propagation layer to align local point features with text features from a provided list of part names (e.g., ['furniture body', 'laptop base']). At inference, each point is assigned the label of the text prompt with the highest feature similarity.

‘Uni3D w/o text’ is a supervised closed-set method designed to test the raw quality of the pre-trained geometric features. Inspired by classic methods like PointNet, we add an MLP head to the frozen Uni3D backbone and train only this head to classify points into a predefined set of all part categories from complete dataset(including both ID and OOD objects). This process requires no text input.

The purpose of these baselines is twofold: The poor performance of ‘Uni3D w/o text’ demonstrates that relying on purely geometric features is insufficient and proves the necessity of semantic guidance for this task. Its weakness stems from a paradigm mismatch, as it discards the rich, cross-modal semantic alignment capability Uni3D was pre-trained for.

By outperforming ‘Uni3D w/ text’, we show that our [SEG] token mechanism is a superior guidance strategy. Unlike static text prompting, our method dynamically and end-to-end couples the segmentation signal with the autoregressive generation of the kinematic structure, leading to more accurate and context-aware results.

W3.3: Justification for [SEG]'s Superiority

The superiority of our [SEG] token mechanism over the baseline stems from two fundamental differences: First, our [SEG] token is a dynamic and context-rich signal, unlike a static word used in the ‘Uni3D w/ text’ baseline. It is generated by the LLM after the model has already processed both the complete point cloud geometry and the preceding kinematic parameters(e.g., parent-child links, joint types) within the autoregressive sequence. Therefore, the token’s embedding is inherently informed by both geometric and structural context, making its guidance for segmentation far more precise than a simple class label. Second, our training paradigm learns a joint probability distribution over both kinematic structure and geometry. Through end-to-end optimization, the generation of a plausible structure and the successful segmentation of its corresponding geometry are tightly coupled. This is fundamentally different from the baseline's decoupled approach, where segmentation is merely a post-hoc classification task. In essence, our method transforms segmentation from simple matching into a context-aware grounding problem, leading to superior accuracy. To prove the crucial role of our [SEG] token mechanism, we have conducted new ablation studies. We summarize the key experiments and findings below:

1. Segmentation-Only: This model only outputs [SEG] tokens, trained solely for part segmentation and does not generate any kinematic structure. This experiment validates the necessity of "context" and "joint training".
2. Geometry-Blind: In this model, the LLM is given ground-truth kinematic structure but has no access to the input point cloud when attempting to output segmentation commands. This experiment validates the necessity of "geometric features" for the [SEG] token.

Results are as follows:

The performance of the "Segmentation-Only" model is slightly lower than that of the full model, indicating that kinematic structural context and joint structure-geometry training are beneficial for achieving high-precision segmentation. The "Geometry-Blind" model completely fails the segmentation task, decisively proving that the LLM's direct perception of geometric features is the cornerstone of the [SEG] mechanism's functionality.

W3.4: Omission of Standard "Joint Success Rate" Metric

Thanks for pointing this out. Due to the length constraints of rebuttal, please refer to Table 1 in our response to reviewer KtPh, for a complete breakdown of the joint success rate.

W3.5: Omission of Real2Code in Table 3 / Table

The reason for its initial absence is that the official LLM model checkpoint and training code for Real2Code have not been released. To provide this missing result, we have now completed the evaluation by reproducing the Real2Code pipeline ourselves. We have updated Table 3 with the physical executability result from this implementation.

Q1: Determination of Object Properties at Inference

Thank you for the question. Our model infers both the object category and the number of parts directly from the input point cloud, without requiring annotations. For the object category: As shown in Section 3.3 and our Appendix, our model's output directly includes semantic part categories (e.g., "link_0": "base_cabinet[SEG]"). This demonstrates that the model implicitly understands the overall object category by deconstructing it into its constituent parts during the generation process. Separately, one could also directly query our ShapeLLM backbone for the object's category, as this is an inherent capability from its pre-training. For the number of parts: This is not an input but an outcome of the autoregressive process. The model generates the structure joint-by-joint and implicitly learns when to stop, determining the final number of parts by the length of the predicted joint list.

Q2: Performance Breakdown and Failure Analysis

Thank you for the question. To provide a more detailed error analysis for physical executability, we have conducted a deeper failure analysis. Due to the rebuttal's length constraints, we selected a representative subset of 3 Out-of-Distribution (OOD) categories (Window, chair, Globe) for this analysis. The results are showed below:

Thank you for your valuable feedback and for raising our score. Your insights significantly improved our paper's rigor and clarity, and we will incorporate all discussed changes into the revision.

We sincerely thank the reviewer for their positive assessment. We thanks for your recognition of our task's real-world significance, the clarity of our method and paper organization, and the efficacy of our results, especially the physical executability metric.

W1 & Q1: Novelty & Re-use of the [SEG] Token

Thanks for your valuable feedback.

Primary contribution

Our work's primary contribution is not a minor technical improvement, but rather the introduction of a fundamental Paradigm Shift to the field of articulated object reconstruction. Prior to our work, the prevailing paradigm in this field suffered from several inherent limitations. It typically relied on multi-stage, error-prone pipelines—for instance, using one model to segment parts and another to predict parameters. At the input stage, to simplify the problem, these methods either depended on representations with significant loss of geometric detail (e.g., OBBs) or were limited by 2D images, making it difficult to infer precise 3D structures. On the modeling front, they either employed specialized, small-scale models lacking world knowledge and generalization capabilities, or adopted methods requiring slow, per-instance online optimization (such as those based on Gaussian splatting), which are difficult to scale. Our work champions and implements a new paradigm of end-to-end joint reasoning. We are the first to demonstrate that a unified 3D Multimodal Large Language Model (3D MLLM) can directly take a complete 3D point cloud as input and, through joint optimization, produce a functional model that is both geometrically precise and kinematically consistent in a single, end-to-end pass. This new paradigm leverages the powerful prior knowledge and reasoning capabilities of large models to fundamentally address the problems of information loss, error accumulation, and poor generalization that plagued traditional methods. The most direct evidence of its effectiveness is that our model achieves a greater than 50% improvement in Physical Executability, directly validating the robustness and superiority of this new paradigm.

Innovative application of the [SEG] token

The innovation lies in being the first to fundamentally adapt and extend its application and mechanism for end-to-end articulated model reconstruction. This is primarily reflected in the following three core aspects, which also represent the essential differences from prior works like SegPoint and Reason3D:

• Operating Scale: SegPoint and Reason3D both operate at the scene level. In contrast, our URDF-Anything focuses on fine-grained part-level analysis.
• Function of Segmentation: In our framework, segmentation plays another critical role--A Booster: Through end-to-end joint training, the precise segmentation task in turn provides strong geometric constraints for the prediction of kinematic parameters, thereby improving their accuracy. As shown in our ablation study (Table 1 below), removing the segmentation task leads to a significant increase in kinematic prediction error, validating its "booster" effect.
• Implementation Mechanism: Both SegPoint and Reason3D directly use the hidden state of the [SEG] token ($h_{seg}$) to guide the segmentation decoder. We have made a key improvement upon this: we fuse the hidden state $h_{seg}$ with the hidden state of its immediately preceding "category token" (e.g., "drawer"), $h_{category}$, resulting in $h_{combined} = [h_{seg} ; h_{category}]$. This design injects strong, part-aware context into the segmentation process, which is crucial for distinguishing between multiple visually similar parts on the same object. We validated this through a dedicated ablation study (Table 2 below), which shows our context fusion design achieves superior segmentation accuracy.

To experimentally support the above claims, we conducted two key ablation studies:

Table 1: Validating the "Booster" Effect of Segmentation on Kinematic Prediction

Table 2: Validating the Superiority of the Context Fusion Mechanism

Finally, we will revise the end of our introduction to clearly summarize our core contributions, including: (1) Proposing the first end-to-end 3D MLLM framework for articulated object reconstruction, championing a new paradigm from complete 3D input to joint prediction output; (2) Achieving deep coupling and joint prediction of kinematic parameters and geometric segmentation through an innovative application of the [SEG] token; (3) Demonstrating the superiority of this new paradigm through extensive experiments. And we add citations to these relevant works [1, 2, 3] in our revised paper.

W2: Explanation on Input Modality

We thank the reviewer for pointing out that the explanations regarding the input modalities for evaluation could be clearer. We agree and appreciate the opportunity to clarify this here and in revision.

As the reviewer noted, we have provided more details in our answers to the specific Q 3, 4, 5, and 6.

Q2: Error Analysis on Physical Executability

Thanks for this question. As your suggestion, we have conducted a detailed analysis of the failure cases for our method. The results are as follows, break down the 22% overall failure rate observed in Table 3.

Table 3: Breakdown of failure cases for URDF-Anything on the physical executability task.

This analysis reveals that the vast majority of failures (21% out of 22%) are due to the prediction of incorrect or physically implausible joint parameters (e.g., origin, axis). In contrast, structural errors, such as generating a malformed JSON or an invalid URDF file, are extremely rare (1%).

This breakdown confirms that our MLLM-based framework is highly proficient at learning the correct syntax and structure for URDF generation. The primary remaining challenge lies in the precise quantitative prediction of kinematic parameters, not in generating a valid file format.

Due to space constraints, we did not include this analysis in the initial submission. We will add this table and the corresponding discussion to the experimental section in the revision to provide a more comprehensive evaluation.

Q3: "Count Accuracy" Metric

Thank you for pointing out this omission. You are correct, we should have defined this metric in the text. "Count Accuracy" measures the percentage of test samples for which the number of predicted articulated parts (i.e., links with non-fixed joints) exactly matches the ground-truth number of parts. We will add this definition to Section 4 (Evaluation Metrics) in the final version of the paper.

Q4: "Oracle" Term

Thank you for the question. The "Oracle" setting in Table 2, a protocol from the Articulate-Anything paper, provides baselines with ground-truth part segmentations. This is to fairly evaluate their kinematic prediction module in isolation, by removing errors from their preceding segmentation step.

Q5: Baseline Results Split

Thank you for pointing out this omission. To clarify our experimental setup, our data split is consistent across the baselines. Our In-Distribution (ID) split consists of the five categories shown in our supplementary materials: "StorageFurniture", "Laptop", "Box", "Table", and "Refrigerator". The remaining 41 object categories from the PartNet-Mobility dataset constitute our Out-of-Distribution (OOD) split.

Q6: Concern about Reliance on Prior Knowledge

Thank you for the question. The example you saw in the appendix is indeed one of the diverse templates we used during training, but it does not mean our method must rely on this prior knowledge at inference time. We apologize for any potential misunderstanding the appendix may have caused and wish to offer a detailed clarification here. Our approach is divided into two phases: training and inference.

During Training: We designed highly diverse prompt templates to enhance the model's robustness. A portion of these templates did include the "number of parts," primarily serving as a pedagogical tool or a strong supervisory signal to ease the fitting pressure and help the model learn the correlation between an object's visual form and its internal articulated structure more effectively. However, our training data also contained numerous generic templates that did not provide this information, which compelled the model to learn to infer the number of parts autonomously from visual and geometric features, rather than relying solely on the prompt.

During Inference (Testing): Providing the number of parts is entirely optional at inference time. It is crucial to emphasize that for all quantitative evaluations in the paper and for fair comparison with other models, we uniformly used a generic prompt that did not contain any information about the number of parts or joints (e.g., "Please segment the object and predict its joint parameters").

We will revise the appendix to feature a simpler object JSON as an example for better clarity.

We appreciate your feedback and are pleased that our clarifications on the terminology and the [SEG] token were helpful.

Regarding your question about the table data for Articulate-Anything, I would like to offer the following explanation:

We initially reported only a single "ALL Classes" score because Articulate-Anything is a training-free, retrieval-based method. Since it is not trained on our In-Distribution (ID) dataset, the ID vs. OOD distinction is not directly relevant to it in the same way it is for trained models. This reporting style is also consistent with the evaluation in the original Articulate-Anything paper.

To provide the most complete comparison and fully address your request for a split analysis, we have re-evaluated Articulate-Anything on our specific ID and OOD splits.The complete results are presented in the table below:

We will replace the original table with this updated version in the revision.

We sincerely thank the reviewer for their positive feedback. We are particularly encouraged by your recognition of the importance of the problem we are addressing, the integrated nature of our end-to-end framework, and our strong experimental results and clear reproducibility.

W1: Unclear Mechanism of the [SEG] Token for Kinematics

Thank you for this insightful and critical question. In our paper (lines 47-50), we described this as a "tight coupling" that "ensures full consistency between predicted kinematics and reconstructed geometry". We would like to take this opportunity to "demystify" this mechanism with both a theoretical explanation and new experimental validation.

1. Theoretical Explanation: Geometric Regularization via Joint Optimization Our model does not learn this association "magically"; the core principle is Geometric Regularization, an effect that emerges from our end-to-end joint optimization. Essentially, the segmentation task ($L_{seg}$) provides a powerful physical anchor for the kinematic prediction task ($L_{text}$). A model predicting only kinematics might hallucinate a joint that is textually plausible but physically baseless (e.g., a rotation axis outside the object). Our joint segmentation task forces the model to ground its abstract kinematic predictions (e.g., a rotation axis) in the concrete geometric entities of the point cloud. To accurately segment a door, the model must implicitly learn where its center of rotation and axis should be. This geometric grounding effectively constrains the parameter search space, leading to more physically consistent and accurate articulation structures.

This is enabled by the Shared Representation in our end-to-end model. Because both tasks share most network parameters and are optimized via a unified loss function ($L = λ_{text}*L_{text} + λ_{seg}*L_{seg}$), the model must learn an internal representation effective for both. This means the hidden states used to generate kinematic parameters must also contain information useful for the subsequent geometric segmentation, thus tightly coupling them in the latent space.

2. Experimental Validation: From Ablation to Visualization To empirically validate this theory, we will add two key experiments to our revised manuscript to demonstrate how the [SEG] token facilitates kinematic learning:

• Ablation Study: Quantifying the Contribution of Segmentation.
  • Design: We will train a baseline model where the [SEG] token and its associated segmentation loss ($L_{seg}$) are removed. This model will predict kinematic parameters using only the language modeling loss ($L_{text}$). We then compare its parameter accuracy against our full model.
  • Results: As shown in Table 1, without the geometric regularization from the segmentation task, this baseline has higher error in its kinematic predictions. This result quantitatively proves that the segmentation task is crucial for learning accurate kinematic structures.

Table 1: Quantifying the Contribution of Segmentation.

• Attention Visualization: Opening the "Black Box".
  • Design: To visually inspect how the model grounds its kinematic predictions, we visualized the self-attention distribution from a specific layer of the two model versions in our Ablation Study: our full model (URDF-Anything w/ seg) and the baseline without the segmentation task (URDF-Anything w/o seg). This allowed us to observe which parts of the input point cloud the models focus on when generating a kinematic parameter token (e.g., 'axis').
  • Results: The visualizations (which can't be shown here and we will add to the revision) show that when generating kinematic tokens, our full model's attention is highly concentrated on the physically relevant joint regions of the point cloud (e.g., hinges, sliders). In contrast, the attention of the model trained without the segmentation loss is far more diffuse and unfocused.
  • Significance: This visualization provides direct, intuitive evidence at a "neural level" of how our joint training forces the model to associate abstract kinematic concepts with concrete geometric features. It demystifies the process, showing that the connection is not "magic" but a learned behavior enforced by the geometric regularization of the segmentation task, directly addressing your concern about the "black-box" nature of the mechanism.

We will incorporate these new experiments, including the visualizations, into our revised manuscript. Thank you again for pushing us to clarify this crucial aspect of our work.

W2: Insufficient Ablation Studies

We sincerely thank you for your valuable suggestion regarding the ablation studies.

We would like to take this opportunity to clarify the core focus of our current work. Our primary objective is to be the first to demonstrate and validate the feasibility and superiority of a novel, unified 3D MLLM framework for the task of articulated object reconstruction. Through this, we aim to help guide the field's evolution from traditional multi-stage or optimization-based paradigms toward a more efficient and versatile end-to-end large model paradigm. Based on this core objective, our experimental design (including the ablation study in Section 4.4) prioritized validating the most fundamental hypothesis of this new paradigm: that multimodal inputs combining explicit 3D point clouds and text instructions are crucial compared to relying solely on 2D images or pure geometric information. We believe that in the initial stages of establishing a completely new framework, proving the correctness of its core design philosophy is more critical than optimizing one of its replaceable components.

However, we also agree that conducting more comprehensive diagnostic experiments would better demonstrate the superiority and robustness of our unified MLLM joint prediction architecture. Therefore, we have supplemented our work with a new preliminary experiment. Specifically, we replaced the original Uni3D backbone with a classic alternative, PointNet++, while keeping all other parts of the model (such as the 3D MLLM, the [SEG] token mechanism, and the training strategy) unchanged.

Table 2: Ablation Study on 3D Backbone

Our preliminary results show the following:

Validation of Our Choice: When using PointNet++ as the backbone, the model's overall performance sees a slight decrease compared to using Uni3D. This, to some extent, validates our initial choice of Uni3D as a powerful and suitable feature extractor.

Demonstration of Framework Robustness: Despite the slight performance drop, this new PointNet++-based variant still significantly outperforms all baseline methods compared in our paper across all key metrics.

This strongly demonstrates that the success of our framework is not merely dependent on a single specific SOTA component; its core advantage stems from the effectiveness of our proposed unified MLLM joint prediction architecture. This result also reaffirms that there is ample room for future research to conduct more detailed component-level comparisons and optimizations (e.g., exploring more efficient backbones) upon our framework to further enhance performance.

W3: Limited Novelty / Compositional Work

We appreciate the opportunity to clarify our perspective on the novelty of URDF-Anything.

We respectfully argue that the novelty of our work lies not in inventing a single, isolated algorithm, but in successfully conceiving, implementing, and validating a new end-to-end paradigm for articulated object reconstruction. Within this new paradigm, we performed crucial, non-trivial adaptations of existing technologies to solve challenges unique to this domain.

Previously, the field relied on fragmented, multi-stage pipelines with simplified inputs, which led to error accumulation and physically inconsistent models. Our unified paradigm solves this by processing complete 3D point clouds to jointly optimize geometry and kinematics in a single pass. The success of this fundamental shift is demonstrated by a >50% improvement in physical executability and superior accuracy over piecemeal approaches (Table 2 in paper), validating its superior robustness.

This paradigm is not a simple composition; it required key technical innovations and has pushed the boundaries of the subfield:

Technical Novelty: We didn't just reuse the [SEG] token; we fundamentally adapted it. Our context-fusion mechanism (merging category and [SEG] token information) provides the essential part-level semantic understanding required for this complex, single-object deconstruction task.

Pushing Boundaries: Our most significant contribution is a leap in generalization. By leveraging a 3D MLLM, we shift the field from dataset-specific models to those that can generalize to entirely new object categories. This is proven by our strong performance on out-of-distribution (OOD) objects (Tables 1-3 in paper), a crucial advancement for the field.

In essence, our work provides the first robust "existence proof" for applying 3D MLLMs to this domain. We believe establishing this powerful and effective new baseline is a significant contribution that opens new avenues for future research.

Dear Reviewer euKz,

We would like to extend our sincere thanks for your thorough review and valuable feedback on our paper.

To address the points raised in your review, we have performed additional experiments and revised the paper. Here is a summary of the main changes:

(1)On the "black-box" mechanism: We added a theoretical explanation and two new experiments (an ablation and a visualization) to demystify how our model learns kinematics. (2)On insufficient ablations: We conducted a new ablation study on the 3D backbone (using PointNet++) to justify our architectural choice and prove our framework's robustness. (3)On novelty: We clarified that our core contribution is establishing a new, end-to-end paradigm, whose superiority is proven by major gains in physical executability and generalization.

We have done our best to address all your suggestions and hope you find our revisions satisfactory. We are happy to engage in further discussion if needed. Should our response resolve your concerns, we would be deeply appreciative if you would consider raising your score.

Q2: Novelty Beyond Combination

Step 3: The Interaction Mechanism Dilemma – Why a Naive Static segmentation Fails

Within our joint framework, a crucial question remains: how should the LLM interact with the 3D features to produce a segmentation? This section explains why our dynamic [SEG] mechanism is superior to simpler, static alternatives.

• Naive Approach: A naive but plausible approach is to treat segmentation as a static, post-hoc classification task. This is represented by our Uni3D w/ text baseline, where a model tries to match pre-defined text labels (e.g., "drawer") to geometric regions in the point cloud, decoupled from the main kinematic reasoning process.

• Failure Mode: This approach fails because the segmentation signal (a static text label) is context-poor. It is ignorant of the object's overall kinematic structure (e.g., which drawer this is in a stack) and the specific motion parameters being predicted. This decoupling prevents the model from learning the crucial correlation between a part's geometry and its function.

• Our Non-trivial Solution & Conclusion: Our superiority stems from two fundamental differences. First, our [SEG] token is not a static label but a dynamic, context-rich signal. It is generated by the LLM after it has already processed both the complete point cloud geometry and the preceding kinematic parameters (e.g., parent-child links). Its embedding is therefore inherently informed by both geometric and structural context. Second, our framework learns a joint probability distribution over both structure and geometry. This transforms segmentation from a simple matching task into a context-aware grounding problem.

• Evidence: Our ablations prove these two points.

  • To prove the necessity of geometric context, the "Geometry-Blind" experiment denies the model access to the point cloud. The signal becomes ungrounded, and the model fails completely (mIoU ≈ 0.1).
  • To prove the necessity of structural (kinematic) context, the "Segmentation-Only" experiment removes the kinematic prediction task. This model (mIoU 0.61) is consistently outperformed by our full model (mIoU 0.63), showing that kinematic context refines segmentation.

Table 3: Ablation on Key Components for Segmentation (Adapted from our response to W3.3 of Reviewer qimd for convenience)

Step 4: The Core Mechanism Dilemma – Why a Generic Segmentation Signal Fails

Even with the right framework, the challenge of generating precise segmentation commands for multiple similar parts on a single object remains.

• Naive Approach: A naive approach would be to directly apply the generic [SEG] token mechanism, inspired by its success in 2D image segmentation. In that paradigm, the token typically acts as a universal pointer to an object or region.

• Failure Mode: This generic approach is insufficient for our task. A universal [SEG] token lacks the fine-grained context needed to distinguish "drawer 1" from "drawer 2" and will fail when faced with the ambiguity of single-object, multi-part reconstruction.

• Our Non-trivial Solution & Conclusion: We designed a task-specific "Context Fusion" mechanism that merges the [SEG] token's state with that of its preceding category token. This seemingly small modification is a key innovation for resolving part-level ambiguity. Experiments confirm that this targeted design significantly outperforms the generic application.

Table 4: Validating the Superiority of the Context Fusion Mechanism (Adapted from our response to W1 of Reviewer KtPh for convenience)

Thanks for the authors’ reply. I believe that steps 3 and 4 in A2 have largely addressed my concern regarding novelty. I would encourage adding a statement in the paper to explicitly note that this compositional work (i.e., integrating different modules) is technically non-trivial.

I have one additional short question: in the specific field of URDF generation, when leveraging an MLLM, are you the first to introduce the point cloud modality? If so, this could also be highlighted as a novelty point. If not, could you please provide relevant references and explain why your approach is superior?

We sincerely thank the reviewer for their positive and encouraging feedback. We are particularly grateful for your recognition of our method's technical strengths—including its neat integration of MLLMs and its ability to handle objects with an arbitrary number of parts—as well as our significant quantitative results and the clarity of the paper.

W1: Comparisons

Thank you for this insightful question, which allows us to clarify our work's positioning relative to other important lines of research. We did not focus our comparison on methods like CARTO or the PARIS line of work because they address a fundamentally different and more constrained problem setting, and possess significant limitations that make a direct comparison inequitable. To clarify:

• Different Technical Routes & Inherent Flaws The primary reason for the separate comparison is that these methods follow different technical routes with inherent flaws that our work is designed to overcome.

  • PARIS[2] represents a completely different approach. It relies on slow, online per-instance optimization, requiring over 3 minutes at test time. This is in stark contrast to our fast, feed-forward inference model (~13s). As our new experimental results show, this optimization-based method also suffers from high errors and very low physical executability. [3]was posted on arXiv after the NeurIPS submission deadline.
  • CARTO[1] is a specialized, small-scale feed-forward model with severe input and output restrictions. It is designed only for single-joint objects and cannot perform part segmentation or generate a complete, executable URDF with precise kinematic parameters (axis, origin). This inherent lack of capability and poor generalization to complex objects makes it unsuitable for the general-purpose reconstruction task our work addresses.

• Quantifying the Differences in Speed and Capability To illustrate these fundamental differences, we conducted new experiments. The tables below quantify the critical trade-offs and highlight the limitations of these alternative approaches.

Table 1: Comparison of Inference Speed and Methodology

Table 2: Comparison of Reconstruction Accuracy and Physical Executability

While our MLLM-based approach has a modest speed trade-off compared to a simple model like CARTO(as shown in Table 1), this is precisely what enables us to overcome these limitations, delivering vastly superior accuracy, full-featured URDF generation, and the ability to generalize to complex, novel objects(demonstrated in Table 2). We will add this detailed comparison and discussion to our final paper.

W2: The definition of "Out-of-Distribution" (OOD)

That is an excellent question for clarifying our experimental setup. In our study, an Out-of-Distribution (OOD) instance is defined as an object belonging to a category that was entirely unseen during the model's training phase. We determine the In-Distribution (ID) and OOD splits through a strict, category-based partitioning of the dataset to rigorously evaluate the model's true generalization capabilities. Specifically, to ensure a fair and consistent comparison, we adopted the experimental setup used by prior works such as Articulate-Anything and Real2Code. The In-Distribution (ID) set comprises five core object categories: 'Laptop', 'Box', 'Refrigerator', 'StorageFurniture', and 'Table'. Our model is trained and validated only on instances from these categories. The Out-of-Distribution (OOD) set consists of all instances from the remaining 41 distinct object categories in the dataset, which are completely held out from training. Therefore, the distinction between our ID and OOD sets is not merely a variation in object size. It is a fundamental difference in the object category itself, which encompasses distinct geometries, functionalities, and articulation structures. This setup is designed to test our model's ability to generalize to entirely novel classes.

W3: Generalization Scope

Our method is explicitly designed to generalize to novel classes, leveraging a 3D MLLM backbone with robust geometric priors to reason effectively about unseen objects. We systematically validate this capability in Section 4 (Experiments). As demonstrated in Tables 1, 2, and 3 in our paper, our method's superior performance on Out-of-Distribution (OOD) objects across all key metrics—including segmentation accuracy, kinematic parameter precision, and physical executability—confirms the powerful generalization capabilities of our framework.

W4: Lack of Real-World Evaluation

Thank you for this constructive suggestion regarding real-world evaluation. First, regarding the suggestion to use the dataset from SplArt [3], we would like to clarify that this work was posted on arXiv after the NeurIPS submission deadline, so we were unable to include it in our original comparisons. Regarding the PARIS [2] dataset, it consists of two parts: a real-world set and a simulated set. Their simulated data is also derived from PartNet-Mobility, which is the primary dataset used in our work. However, to directly address your valuable point about real-world evaluation, we have conducted new experiments on the real-world portion of the PARIS dataset during this rebuttal period. This dataset is limited to two main categories: Fridge and Storage. The quantitative results of our method, presented in Table 3, are as follows:

Table 3: Zero-Shot Sim-to-Real Performance on the PARIS Real-World Dataset

These results(Table 3) provide several key insights. First, our method achieves reasonable performance on geometric and continuous-valued tasks (segmentation, mesh reconstruction, and predicting joint axis/origin) in a zero-shot, sim-to-real setting. We attribute this to the challenging sim-to-real domain gap. This experiment provides a valuable and realistic baseline for zero-shot sim-to-real transfer on this complex task. It highlights that while our framework transfers well for geometry, robustly classifying kinematic types in the wild without any real-world fine-tuning remains a challenging open problem. We will include this full real-world evaluation and discussion in the revision of our paper.

W5: Quantitative Shape Reconstruction Metric

Thank you for this excellent suggestion. We agree that a quantitative metric for shape reconstruction quality is crucial for a complete evaluation. To address this, we have conducted a new experiment during the rebuttal period to measure the geometric fidelity of the final output meshes using the Chamfer Distance (CD), as you recommended. We compared our method against strong baselines that also produce explicit mesh outputs. The results, shown in Table 4, clearly demonstrate the superiority of our method in shape reconstruction quality:

Table 4: Comparison of Shape Reconstruction Quality (Chamfer Distance)

Minor 1: Extensibility to other objects

Yes, in principle, our framework can be extended to other categories of articulated objects, such as robot arms. The core of our method—the autoregressive generation of a flexible, structured JSON output—is general-purpose and capable of describing complex kinematic chains. However, our current model is trained on the PartNet-Mobility dataset, which primarily features furniture and household items. To achieve optimal performance on a specialized domain like robot arms, which have distinct visual features and structural priors, we would recommend fine-tuning the model on a domain-specific corpus to adapt to these new patterns.

Minor 2: Sim-to-real transfer

Yes, our method is designed to be trained solely in simulation and then deployed directly in the real world without fine-tuning. This is achieved through a decoupled architecture. Our pipeline uses a powerful, pre-trained perception front-end (e.g., DUSt3R, LGM) to convert any image (sim or real) into a unified point cloud representation. Our core 3D MLLM then operates exclusively on this geometric data. This design ensures that the sim-to-real domain gap (e.g., lighting, textures) is primarily handled by the perception module, while our core model is shielded from pixel-level discrepancies and focuses on the underlying geometry, which transfers well.

[1] CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects, CVPR 2023 [2] PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects, ICCV 2023 [3] SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splatting, ICCV 2025