Template-free Articulated Gaussian Splatting for Real-time Reposable Dynamic View Synthesis

Thank you for the detailed comments.

• Limitations/W2: Limitations and broader impacts

  Answer: We will move the limitations and broader impacts to the main paper in the final version.

• Q1: More discussion about Eq. 13.

  Answer: Based $g_i$ calculated by Eq. 13, we can densify more superpoints to accommodate complex motions. Therefore, in theory, this formula will not split a large rigid part into multiple sub-parts. Even if more sub-parts are generated, since the motion of these sub-parts is the same, the merging process will merge the sub-parts into one again.

• Q2: Why are the skeletons so complex and noisy?

  Answer: To re-animate (/re-pose) the $k$-th joint of the object, we simply apply an additional rotation matrix $\Delta \mathbf{R}_k \in \mathrm{SO}(3)$ on the rotation $\hat{\mathbf{R}}_k$ predicted by deformable filed $\Psi$ (Eq. 11), i.e., $\hat{\mathbf{R}}_k' = \Delta \mathbf{R}_k \hat{\mathbf{R}}_k$. Compared with 6 DoF of superpoints in the Dynamic stage, each joint in the Kinematic Stage has only 3 DoF i.e., only can rotate but not translate. Therefore, the extra DoF of superpoints is the main reason are the main reason why the discovered skeletons are so complex and noisy. As shown in Fig. 1 in the attached PDF, there are some inconsistencies in the 3D model in canonical space of Dynamic stage. However, the rendered image in the given timestamp matches the ground truth. The price of achieving all this is that the skeleton is complex and noisy.

  Another reason is that reconstucted motion of superpoint is imperfect. The number of superpoints is therefore greater than the number of physically rigid parts. As a result, the skeleton discovered by the motion of superpoints becomes complex and noisy.

• Q3: Why are there different resolutions reported for each method?

  Answer: To solve this concern, we present the results of our method with the same resolutions in Table 1 in the attached PDF. The training and testing resolutions are the same. For example, when AP-NeRF is trained on $400\times 400$ images, it is also tested on $400 \times 400$ images.

• A4: Any failure case examples/images

  Answer: For scenes with complex motions (e.g., combining long chains of rotations with texture cue ambiguity), our method can achieve better performance than AP-NeRF. Figure 2 in the attached PDF compares our method with AP-NeRF for the failure case of AP-NeRF. As shown in the figure, our method may not be able to reconstruct the endpoints of objects well.

• W3: Statistical Significance

  Answer: The answer of Statistical Significance is No. And the Justification is: We do not report error bars or other information about statistical significance as this is not a common procedure in the field and does not contribute to understanding our evaluation.

• W4: Higher quality captured dataset.

  Answer: Thanks for your suggestion. We follow the experiment setting of AP-NeRF. We will evaluate our method on higher-quality captured datasets in the future.

We thank the reviewer for the useful suggestions.

• Q1: Higher resolution

  Answer: We provide the results at the same resolution as WIM and AP-NeRF in the attached PDF. We believe the improved performance mainly comes from the powerful representation ability of 3D-GS and the better capture of articulated structures.

• Q2: Evaluate skeleton model

  Answer: The biggest challenge in evaluating skeleton models is that the learned skeleton is uncertain in terms of the number of joints and the connections between joints.

  In our view, there are three imperfect methods that might be used to evaluate the quality of the learned skeleton structure.

  1. As shown in WIM, we can first learn a mapping between the learned skeleton model and a template (e.g., SMPL for humans), and then evaluate the mapped template. However, this way requires a template and the mapping process will introduce more errors.
  2. Given test images with camera poses, all parameters except the rotation of joints are fixed, and the learned model is then optimized to fit the test images. After test-time optimization, the difference between the rendered image and test images shows how good the learned skeleton structure is. However, test-time optimization may fail or be suboptimal.
  3. The repose process can be treated as an image generating process (e.g., GAN, Diffusion model). Therefore, we evaluate the learned skeleton structure by using the metrics (e.g., FID) that are common in the generative model. However, obtaining the rotation range of each joint and test camera pose is challenging.

  Overall, rigorously evaluating skeleton models is a challenging problem, and we will try to address in future work.

We sincerely thank you for your time and efforts.

• W1: Lacking in innovation

  Answer: While our work builds upon previous works, to the best of our knowledge, it is the first work to discover the skeleton of articulated objects represented by 3D Gaussian Splatting.

• W2: Distinguish this paper from AP-NERF by a thorough analysis

  Answer: Both our method and AP-NeRF are proposed for learning articulated representations of dynamic objects. However, there are two main differences between our method and AP-NeRF:

  1. While AP-NeRF is based on point-based NeRF representations, our method is based on 3D Gaussian Splatting, which is key to achieving real-time rendering.
  2. While AP-NeRF extracts the skeleton using the Medial Axis Transform (MAT), our method discovers the skeleton based on the motion of superpoints.

• W3: This method compares fewer baselines, and the quantitative results do not show significant improvements

  Answer:

  • Unlike other methods focused on reconstructing dynamic scenes (e.g., D-NeRF, HyperNeRF, Deformable-3D-GS, 4D-GS), our work, alongside with WIM and AP-NeRF, uniquely addresses explicit skeleton extraction from videos without any priors.
  • As evidenced in Tables 3, 4, and 5, our approach achieves real-time rendering (>100 FPS), surpassing WIM and AP-NeRF (<1 FPS). Furthermore, our method exhibits superior rendering quality on the D-NeRF and Robots datasets.

• Q1: Ablation study for the number of $M$

  Answer: $M$ is the number of superpoints, which is initialized as 512 as shown in L212. We conduct the ablation study for $M$. The results are shown in Table 2 in the attached PDF.

• Q2: Comparison with SC-GS

  Answer: Quantitative comparison between our method and SC-GS is provided in Table 1 in the attached PDF. Besides,

  • Difference: Our method extracts the explicit skeleton model and refines it during the Kinematic stage, while SC-GS does not.
  • Similarity: Both our method and SC-GS utilize the sparse points (i.e., superpoints in ours, control points in SC-GS) with LBS. Notably, the Dynamic stage in our approach can be replaced with SC-GS.
  • Difference: While SC-GS employs the Gaussian-kernel RBF to compute the weights of LBS, our method directly learns it (i.e., $\mathbf{W}$ in Eq 7).
  • Difference: SC-GS can edit motion by minimizing APAR energy. Compared with SC-GS, our method directly manipulates the skeleton, which is more efficient and simpler.

First of all, although SC-GS uses control points to implement motion editing, there are no constraints between the control points. Therefore, compared to re-pose with skeleton, the motion editing in SC-GS :

• 1. is easy to generate physically unrealistic motion, and thus render unrealistic images;
• 2. need an additional optimization process, which is more complex and time-consuming.

We believe that discovering the skeleton of an object will be helpful in certain fields, such as game production, video generation, physical simulation, etc.

Second, it must be pointed out that the design of LBS is not the contribution of our paper. Our main contribution is to discover the skeleton of an articulated object by utilizing the motion of superpoints.

Third, the main reasons for the performance gap are as follows:

• To simplify the skeleton, the proposed adaptive control strategy significantly reduces the number of superpoints. ($512\to \sim 100$). The number of superpoints has an impact on performance.
• According to the code of SC-GS, SC-GS uses more tricks than ours, such as longer training time (80k), an extra MLP for input time (called time-net), additional hyper-features to help k-nearest neighbor search, the initialization train for control points and so on.

We are incorporating SC-GS into our work to achieve even higher performance.

If you have any further questions or would like additional clarification, please do not hesitate to contact us. We would be more than happy to provide additional information or discuss any aspect of our work in greater detail. Your feedback is deeply appreciated, and we remain fully committed to addressing any concerns you may have.

Thanks for your careful and valuable comments. Below are our responses to the specific points raised.

• Q1/W2: Optimization time and required resources

  Answer: Similar to 3D-GS, the optimization time and required resources are dependent on the number of Gaussians. For the D-NeRF dataset, the average optimization time is 6.3 hours, and the average peak GPU memory is 4.08 GB. Please refer to Tables 1 and 3 in the attached PDF for details.

• Q2: Performance on monocular video with a moving camera

  Answer: The camera setup in the D-NeRF dataset can be considered as using a camera to capture 360-degree surround images centered on the target object. Therefore, using a monocular video with a moving camera as input, our method can achieve good performance. However, using forward-facing monocular videos as input may result in reconstruction failures or inaccurate skeletons.

• Q3: Enforce the skeleton to be located "inside" the geometry

  Answer: We acknowledge the suggestion to add an extra regularizer term to encourage the skeleton to be "inside" the geometry. We will explore this idea in our future work.

• W1: More qualitative examples

  Answer: We have provided additional qualitative examples in the supplementary video material.

• W3: Limitations in the supplementary material

  Answer: Thanks for your suggestion. We will move it into the main text in the final version.