Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting

Q8. Canonical Gaussian initialization.

For "w/o Cano. init", we randomly initialize canonical Gaussians.

• Using single-state Gaussians $\mathcal{G}_{single}^{0/1}$ to initialize the canonical Gaussians. As discussed in Sec.4, we set the canonical state as $t = 0.5$. We found in experiments that using single-state Gaussians to initialize the canonical Gaussians easily converges to local optimal. Assuming we use $\mathcal{G}_{single}^0$ as initialization, then the loss of state 0 will be very small, while the loss of state 1 will be very large. This asymmetry will cause the model to fail to correct the canonical Gaussian to the ideal t=0.5. In the end, most Gaussians remain static, leading to incorrect articulation parameters.
• If we set the canonical state as t=0/1, lacking the motion consistency that $T^{c\rightarrow0}=(T^{c\rightarrow1})^{-1}$, the model will easily collapse or degenerate, especially on complex objects. In this setting, a local optimal found by the model is that all Gaussians have low opacity, which prevents the loss of another state from being too large.  However, since the pruning strategy will eliminate the Gaussians with low opacity, all Gaussians are pruned and it results in training collapse.  We tried multiple sets of hyper-parameters, but the performance was extremely erratic.

Q9. Obvious improvements in articulation parameters estimation.

Our ArtGS demonstrates obvious improvements in articulation parameters estimation due to:

1. Explicit 3D representation of GS enables direct part assignment and motion modeling, achieving more precise articulation estimation.
2. Our center-based part assignment module utilizes spatial information, which brings an enhanced decomposition capability.
3. Our novel initialization strategies, including canonical Gaussian and part centers, greatly reduce the difficulty of training the model and helping it converge to a better solution.

Q10. Minor improvement in geometry reconstruction.

For geometry reconstruction, ArtGS demonstrates large improvement on complex or real-world objects but has minor improvement on simple synthetic objects. This results from two reasons:

1. TSDF limitation. Our method's performance on simple synthetic objects, particularly in terms of CD-w metric, is constrained by our use of TSDF for mesh extraction from Gaussian Splatting-rendered depths. To analyze this limitation, we compare against meshes reconstructed using ground-truth depth with TSDF.

We report the results here:

As shown in this table, even with ground-truth depth input, TSDF-based reconstruction cannot surpass algorithms using marching cubes with NeRF, primarily due to the fundamental differences between TSDF and marching cubes algorithms on simple geometries. However, for complex or real-world objects where articulation reconstruction becomes more critical, the advantages of our model become evident. Additionally, TSDF with ground truth depth on real-world objects may produce poor-quality meshes (e.g., real_storage) due to depth sensor noise, while ArtGS achieves high-quality reconstruction.

2. As discussed in Sec.C, our current implementation adopts the original Gaussian Splatting, which has limited quality for mesh reconstruction compared with NeRF-based methods. Future work could enhance mesh reconstruction quality by incorporating recent advances in Gaussian Splatting (e.g. 2D Gaussians).

Importantly, our primary objective is to create digital twins of real-world articulated objects, where ArtGS demonstrates significant performance improvements.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further questions!

Q4. More visualization results for each stage.

We will provide more visualizations of the initialized canonical Gaussians and optimized Gaussians in the appendix.

Q5. Perception-based metrics.

Following DTA, we focus on the quality of mesh reconstruction and articulation estimation, thus we do not report perception-based metrics. Since DTA doesn't provide code to measure perception-based metrics, we report the results of PARIS and ArtGS in Tab.7. As shown in Tab.7, ArtGS achieves comparable or superior performance relative to PARIS. We also show results here:

Q6. Applications of ArtGS.

The interactable meshes reconstructed by ArtGS can be put into the simulator to train embodied robots. This is an immediate application but not the problem our paper aims to address. We provide a video in the anonymous project website for simulation results in IsaacSim.

Q7. Failure cases and limitations

1. Failure cases. We will add a visualization of failure cases in the appendix soon and we describe them here.

• For real-world objects with multiple parts, some part centers derived from clustering may be inaccurate, and these incorrectly initialized centers may not be corrected during the optimization process, resulting in degraded performance for parts with misaligned centers. However, we found that if we manually correct the wrong part centers before training (i.e. modifying the position of the wrong part centers), it works well.
• Similar motion. If two parts move in the same way in two states, such as two drawers being pulled apart together, our method may fail.

2. Limitations. We've discussed some limitations of our method in Sec.C in the supplementary, and we briefly summarize it here. Please refer to Sec.C for detailed discussions.

• Stability of randomness. The stability issues often stem from the initialization of three key components: canonical Gaussians $\mathcal{G}^c$, part centers $C$, and joint articulation parameters $\Psi$. While our initialization strategies have greatly improved stability, severe initialization errors in $\mathcal{G}^c$ and $C$ may result in part mis-segmentation. Integrating prior models such as SAM will help to enhance the ability to correct center errors.
• Limited states. Our approach is limited to two states, which may not fully capture the complexity of real-world objects. Extending our method to more states or videos is worthwhile future work.
• Mesh reconstruction fidelity. We use the original Gaussian Splatting as implementation, which is limited in mesh reconstruction compared with NeRF-based methods. Integrating recent advancements in reconstruction with Gaussian Splatting may be helpful.

We gratefully appreciate the reviewer for acknowledging our contribution and the effectiveness of our method. Please refer to the general response for a summary of all changes we have made in this revision. We make further clarifications to address the reviewer's concerns as follows:

Q1. Extension for more states and the number of parts.

• Extension beyond two states. While our current implementation focuses on two states, our method can be extended to multiple states by (1) using the same canonical Gaussians as a bridge between all states, (2) extending articulation parameters $\Psi$ for multiple states, either optimize a state-depend function that $\Psi_t =f(t)$ or optimize a set of learnable parameters {$\Psi_t$}$_{t=0}^T$ . We leave this for future work.
• Number of parts. Previous works assumed the number of parts, and we followed their setting. In fact, the number of parts can be easily obtained with GPT-4o. Automating the entire pipeline is worthwhile future work.

Q2. Advantages of 3D Gaussians.

3D Gaussians offer several key advantages over NeRF and MVS point clouds for articulated object reconstruction.

• Compared with NeRF:
  • Explicit 3D representation enables direct part assignment and motion modeling, achieving more precise articulation estimation.
  • Significantly faster training (3-4x speedup over DTA) and rendering.
• Compared with point clouds
  • We can obtain high-quality images and meshes through 3D Gaussians.

Q3. Additional baselines for visualization comparison.

We thank the reviewer for suggesting Real2Code and Articulated Object Neural Radiance Field.

• We tried to compare our ArtGS with Real2Code but we found that it only releases the training code, lacking pre-trained checkpoints and training data. Given that Real2Code is also a submission paper for ICLR 2025, we may not be able to make a comparison before the deadline of rebuttal.
• Articulated Object Neural Radiance Field is the experimental implementation of a few papers including CLA-NeRF (ICRA 2022) and NASAM (CVPR 2022). We only found the data of laptop-11586 from their data link 2 and data link1 is expired. Thus we provide the results of ArtGS on the laptop data in the anonymous project website. The qualitative comparison shows that our method works better.

We added more visual comparisons with PARIS in the anonymous project website and our paper revision to address the reviewer's concern.

Q3. More qualitative results.

We have added comprehensive video results and interactable meshes in the anonymous project website. These videos clearly demonstrate our method's advantages in handling complex dynamic scenarios and its practical utility for creating interactive digital twins.

Q4. Failure cases and limitations.

1. Limitations. We've discussed some limitations of our method in Sec.C in the supplementary, and we briefly summarize it here. Please refer to Sec.C for detailed discussions.

• Stability of randomness. The stability issues often stem from the initialization of three key components: canonical Gaussians $\mathcal{G}^c$, part centers $C$, and joint articulation parameters $\Psi$. While our initialization strategies have greatly improved stability, severe initialization errors in $\mathcal{G}^c$ and $C$ may result in part mis-segmentation. Integrating prior models such as SAM will help to enhance the ability to correct center errors.
• Limited states. Our approach is limited to two states, which may not fully capture the complexity of real-world objects. We will extend our method to more states or videos in future work.
• Mesh reconstruction fidelity. We use the original Gaussian Splatting as implementation, which is limited in mesh reconstruction compared with NeRF-based methods. Integrating recent advancements in reconstruction with Gaussian Splatting may be helpful.
• Number of parts. Previous works assumed the number of parts, and we followed their setting. In fact, the number of parts can be easily obtained with GPT-4o. Automating the entire pipeline is worthwhile future work.
• Pre-aligned states. All we need is to make all cameras in both states to be aligned. Synthetic data has pre-aligned cameras. For real-world data, we can use either ICP alignment or continuous video capture to maintain consistent camera coordinates.
• Object state. More visibility might lead to better reconstruction. But we don't need parts to be opened to expose visibility as much as possible, we just need the parts to have different states in the two states, providing enough motion information to identify these movable parts.

2. Failure cases.

We will add a visualization of failure cases in the appendix soon and we describe them here.

• For real-world objects with multiple parts, some part centers derived from clustering may be inaccurate, and these incorrectly initialized centers may not be corrected during the optimization process, resulting in degraded performance for parts with misaligned centers. However, we found that if we manually correct the wrong part centers before training (i.e. modifying the position of the wrong part centers), it works well.
• Similar motion. If two parts move in the same way in two states, such as two drawers being pulled apart together, our method may fail.

Q5. Missed References.

We have added these references in our revision.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!

We gratefully appreciate the reviewer for acknowledging our contribution and the effectiveness of our method. We also thank the reviewer for constructive and detailed feedback. Please refer to the general response for a summary of all changes we have made in this revision. We make further clarifications to address the reviewer's concerns as follows:

Q1. Questions for the method section.

• Section 4.1
  • Label static/dynamic Gaussians. We distinguish static/dynamic Gaussians only for initializing canonical Gaussians and part centers. Once $\mathcal{G}_{init} ^c$ and part centers are initialized, we no longer distinguish static/dynamic Gaussians by motion prior. Instead, we optimize the part assignment module to obtain the part IDs of each Gaussian. We set part 0 as the static root part.
  • Sparsity of movable Gaussians. The obtained $\mathcal{G}_{init} ^c$ has dense static Gaussians and sparse dynamic Gaussians. We densify all Gaussians during the training process. In the end, we obtain dense static and dynamic Gaussians, achieving high-quality reconstruction.
  • The number of initialized Gaussians. About 10k-50k Gaussians will be initialized in  $\mathcal{G}_{init} ^c$ , depending on the specific object.
• Section 4.2
  • Intuition of learnable centers and $R_{k}$'s relation to articulation. Because we do not know the centers of each part, we model them as learnable parameters and optimize these learnable centers for better part assignments. $R_{k}$ is not related to the articulation.
  • Definition of $X_{i}^{k},\mu_{i}^{c},W_\Delta$. $X_{i}^{k}$ is the relative position of Gaussian $i$ in the coordinate of part $k$, $\mu_{i}^{c}$ is the position of Gaussian $i$ in the canonical space, and $W_\Delta=\mathrm{MLP}(\mu,X,D)$ is a residual term described at line 258-260.
• Sec 4.3
  • Definition of $M_{ik}$. $M$ has been defined in Eq.6, and $M_{ik}$ denotes the probability of that Gaussian $i$ is belong to part $k$ .
  • Definition of $r_{i}^c$. We use $r_{i}$ to denote the rotation of Gaussian $i$ (line 144-145), and $r_{i}^c$ means the rotation of Gaussian $i$ in the canonical space.

Q2. Experiments evaluation.

• Highlighting problems. Thanks a lot for such detailed feedback. We've corrected all highlighting misplacements in our revision.
• Axis Pos metric timed by 1000. Following DTA and PARIS, we multiply the 'Axis Pos' metric by 10 in Tab.1 and Tab.5. We report the 'Axis Pos' metric multiplied by 1000 in Tab.6 and ArtGS demonstrates superior performance compared to DTA. We also show the results here:

We thank the reviewer for the clear summary of our proposed method and for acknowledging its effectiveness. However, we respectfully disagree with the reviewer simply summarizing our paper as an "integration of explored components". Our contributions extend well beyond mere integration, as recognized by multiple reviewers, including significant improvements on an important research problem (JGzH), reasonable designs for part/articulation modeling and innovative initialization strategies (x9y2), and the effectiveness of our proposed method (JGzH, AX3x, x9y2). Please refer to the general response for a summary of all changes we have made in this revision. We make further clarifications to address the reviewer's concerns as follows:

Q1. Technical novelty and contributions.

The application of Gaussian Splatting to articulated object reconstruction is far from straightforward, as evidenced by our ablation studies. As shown in Tab.4 and Fig.4, without our proposed initialization strategies, simply integrating Gaussian Splatting and center-based part assinment module together gets very poor results. Our proposed novel techniques try to solve a fundamental challenge: the simultaneous optimization of canonical Gaussians, part assignment, and part articulation. This joint optimization is extremely challenging because errors in any one component can cascade and cause the entire system to fail. Our method systematically addresses this challenge through:

• Better Canonical Gaussian Design:
  • Set the canonical Gaussian state as t=0.5 for natural motion consistency (i.e. $T^{c\rightarrow0}=(T^{c\rightarrow1})^{-1}$), which is significant for preventing the collapse of optimization.
  • A novel matching-based initialization strategy that provides better starting points for optimization, and helps the model find better solutions.
  • Leverage motion information to identify static and dynamic parts.
• Spatial-aware Part Discovery with Motion Information.
  • Introduce a center-based part assignment module that leverages the spatial and dynamic information of 3D GS.
  • Utilize motion information and clustering for part center initialization.

These novel designs effectively solve the challenge and bring insight that providing good initialization of canonical Gaussians, part assignment, and part articulation is essential for articulated object reconstruction.

Q2. Visual renderings of more categories.

While our paper visualizes representative examples of multi-part objects, our method successfully handles diverse object categories. We have added comprehensive visual comparisons in the anonymous project website, showing successful reconstructions across all these categories. These results demonstrate our method's strong generalizability across different motion types and structural complexities.

Q3. More visualization results for video.

We have added comprehensive video results and interactable meshes in the anonymous project website. These videos clearly demonstrate our method's advantages in handling complex dynamic scenarios and its practical utility for creating interactive digital twins.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!

Dear reviewer,

We hope our response has addressed your questions and concerns. If so, could we kindly ask you to consider increasing the score? Thank you once again for your valuable feedback!

We gratefully thank the reviewer for acknowledging the effectiveness and efficiency of our proposed model. Please refer to the general response for a summary of all changes we have made in this revision. We make further clarifications to address the reviewer's concerns as follows:

Q1. Intermediate states visualization.

We agree that showing intermediate states would better demonstrate our method's capabilities. We have added Fig.6 in the supplementary, showing rendering results at t=0, 0.25, 0.5, 0.75, and 1 for various objects. We also render a continuous video in the anonymous project website. Our method produces smooth and physically plausible interpolations between states across all objects.

Q2. Present the results when different parts are in different states.

Our method can indeed handle different movable parts in different states. Actually, all movable parts are in two different states, which provide motion clues for decomposing them. This flexibility stems from our part-aware design where each part's transformation is independently optimized through its own dual-quaternion parameters, allowing for arbitrary combinations of part states.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!

In the latest revision, we add a visualization of failure cases (Fig. 5) and a visualization of canonical Gaussian evolution (Fig. 6) in the appendix.

Failure Cases.

Case.1 Incorrect Initialization of Part Centers. For real-world objects with multiple parts, clustering-derived part centers may be inaccurate (Fig. 5 (a)) due to sensor noise, occlusion, and varying illumination conditions. These incorrectly initialized centers often persist through optimization, degrading performance for parts with misaligned centers (Fig. 5 (c)). Manual correction of erroneous part centers prior to training (Fig. 5 (b)) yields improved results (Fig. 5 (d)). As discussed in Sec.C, incorporating prior models like SAM for automatic, accurate part center initialization remains a promising direction for future work.

Cases.2 Similar Motions. Our method exhibits limitations when handling parts with identical motion across states, as demonstrated in case 2 of Fig.5 where two drawers are pulled with the same distance. In such scenarios, the model tends to learn a single joint to fit both parts, failing to distinguish between the independently movable parts. As discussed in Sec.C, expanding ArtGS to incorporate additional states would provide richer motion information, potentially enabling better part separation.

Canonical Gaussian Evolution

We visualize the evolution of canonical Gaussians in Fig.6, showing both their part assignments and centers. Our initialization strategy begins with dense static Gaussians and sparse dynamic Gaussians. As training progresses, the Gaussians undergo densification while simultaneously refining their part centers and assignments. These visualization results demonstrate the effectiveness of ArtGS.

Note: Due to the addition of new visualizations, the interpolation results previously referenced as Fig. 6 in the latest revision are now shown in Fig. 8.

We welcome further discussion regarding any remaining concerns.