Learning Implicit Representation for Reconstructing Articulated Objects

We are thankful to Wx9e for noting that (1) our paper addresses a challenging task, (2) the clarity of writing is reasonable, (3) most of the related works are discussed, (4) the limitations of our work are aptly discussed, and (5) our SIOS2 algorithm is intuitive, works and adds value to literature.

We greatly appreciate Wx9e for the weaknesses noted and the questions they have raised. We outline below our response and how we plan to use these comments to improve the quality of our paper.

1a. Complexity: As shown in Alg.1 from the Appendix, LIMR iteratively updates the reconstruction model (shape, time-varying parameters) and the skeleton, and the skeleton is updated according to the 2 intuitive physical constraints. We will try to make LIMR more easy to follow and not seem over-complicated to the readers.

1b. Extensibility: Regarding extensibility to new projects, we demonstrate LIMR's scalability to NMR-based methods (e.g.,[5]) and NeRF-based methods (e.g.,[3]), showing it can adapt to new tasks that use blend skinning for object pose manipulation with video input. Our method eliminates the need for ground truth skeletons required by existing approaches and does not require any additional annotations. Moreover, LIMR is able to obtain a skeleton comparable to that obtained by RigNet [8], which is trained with massive 3D data (mesh, skeleton, and skinning weights). We will include comparisons in the paper and examples are here https://drive.google.com/file/d/18evWJ-uBl7U7NoHH0PN1tKiz_7FAFX1U/view for immediate reference. Therefore, LIMR can be easily extended to methods like [2] that leverage RigNet [8] for the skeleton.

1c. Reproducibility: To ensure reproducibility, we have provided all implementation details of LIMR in both NMR-based and NeRF-based contexts in the Appendix (Section A.3). Additionally, we have described each loss function utilized in our study in detail (Section A.2). Following publication of our paper, we will immediately release the code for public access and reference.

2. Diversity of Shape Samples: The diversity of the PlanetZoo and DAVIS datasets is limited. The structural, appearance, and size differences captured are indicated by the differences among, e.g., horse, dinosaur, giraffe, elephant, and tiger. We will include additional results for more animals, e.g., bear, cow, giraffe, dinosaur and tiger in the paper.

3. Rendering Image Quality: We will surely provide higher-quality renderings. In the meantime, we believe that Figures 3 and 7 already illustrate the improvements made by LIMR, particularly in challenging image regions within the red boxes. We will show video results on the project pages; meanwhile, we have added some examples at https://drive.google.com/drive/folders/1IUrSi-1VcBtr4FbIRrRSt9PuVrKYFE14 for immediate reference.

4. More Discussion about Quantitative Performance: We agree that an additional, Sec 4.3-like discussion about quantitative comparisons will be good to include, but we had to limit these details in Section 4.2 due to space constraints. We will add this to the Appendix.

5. Additional Evaluation Metrics: We will add evaluation metrics of F-score and IoU. F-score clearly brings out our improvements over BANMo. Our IoU results are less pronounced due to the dominance of the silhouette loss component within reconstruction loss (In certain instances, LASR [5] may incorrectly deform the shape, but still yield a favorable IoU score. For example, rather than positioning the legs correctly, it erroneously stretches the area beneath the belly without correspondingly hurting the IoU score.

Responses to Questions:

1. Terminology: We agree and will replace our term 'implicit' with 'latent'.

2. Video Length: There's no maximum limit to video number and length. LIMR can produce satisfactory results with approximately 15 frames, but we can assure accurate reconstruction for only the viewed angles. We have not tested performance with fewer frames as they may not fully contain an action. Exploring the minimal frame limit is a potential avenue for future work.

3. Video Resolution: For NMR-based LIMR, the training image resolution is 1920x1080 same as in [5]. For NeRF-based LIMR, images are downsampled to 64x64, matching the settings used in [3]. We will add these details to the paper.

4. Grayscale Videos: Although not included in the paper, we had performed experiments showing that removing texture (color) related losses has a negligible effect on the resulting mesh quality (likely because texture information is already minimal for some animals which are of almost a single color, e.g., camels). This suggests that LIMR will be negligibly affected if we use gray-scale videos instead of color videos.

5. Bone Number Range: Initially, we worked with about 150-200 bones; later we could reduce this number to around 10 (e.g., for a tiger) while maintaining good performance.

6. Multiple Objects: We use GroundingDINO [6] and SAM [7] to obtain a mask for the target object. When multiple objects are present, GroundDINO and SAM can track and segment the target object using textual prompts. LIMR reconstructs the object corresponding to the mask. Thus, LIMR works for one object at a time, selected by the user, from among the multiple objects detected and tracked. We will add these details and provide code in the supplementary materials or on GitHub.

[1] Noguchi, Atsuhiro, et al. "Watch it move: Unsupervised discovery of 3D joints for re-posing of articulated objects." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Kuai, Tianshu, et al. "CAMM: Building Category-Agnostic and Animatable 3D Models from Monocular Videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Yang G, Vo M, Neverova N, et al. “Banmo: Building animatable 3d neural models from many casual videos” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[4] Wu, Shangzhe, et al. "Magicpony: Learning articulated 3d animals in the wild." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Yang, Gengshan, et al. "Lasr: Learning articulated shape reconstruction from a monocular video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[6] Liu, Shilong, et al. "Grounding dino: Marrying dino with grounded pre-training for open-set object detection." arXiv preprint arXiv:2303.05499 (2023).

[7] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023).

[8] Xu, Zhan, et al. "Rignet: Neural rigging for articulated characters." arXiv preprint arXiv:2005.00559 (2020).

Responses to Specific Questions:

(a) Fig 1 Caption: The rigidity coefficient is indeed derived from skinning weights; we will clarify this in the revised caption.

(b) Video Results: We plan to show the video results on the project pages; we have included some examples that can be found here:https://drive.google.com/drive/folders/1IUrSi-1VcBtr4FbIRrRSt9PuVrKYFE14?usp=sharing for immediate reference.

(c) BANMo's Results: As discussed in our section A.5 (NeRF-based methods VS NMR-based methods), the NeRF-based methods such as ([1], [2], [3], [4], [6]) require many videos with from diverse viewpoints as input to obtain reasonable results. Given a single monocular video, they are not able to provide good results, which is also noted in CASA [11], Fig.6. And we will provide more results from NeRF-based methods in the paper.

(d) We will provide the progression of the skeleton over iterations in the paper.

We thank 2K9U for these invaluable suggestions and contributions to enhancing the quality of our work. We welcome any further feedback. We will incorporate all of the above changes and any further suggestions made during our discussion to revise our paper.

[1] Noguchi, Atsuhiro, et al. "Watch it move: Unsupervised discovery of 3D joints for re-posing of articulated objects." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Kuai, Tianshu, et al. "CAMM: Building Category-Agnostic and Animatable 3D Models from Monocular Videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Yang G, Vo M, Neverova N, et al. “Banmo: Building animatable 3d neural models from many casual videos” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[4] Wu, Shangzhe, et al. "Magicpony: Learning articulated 3d animals in the wild." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Yang, Gengshan, et al. "Lasr: Learning articulated shape reconstruction from a monocular video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[6] Yang, Gengshan, et al. "Reconstructing animatable categories from videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[7] Yao, Chun-Han, et al. "Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[8] Xu, Zhan, et al. "Rignet: Neural rigging for articulated characters." arXiv preprint arXiv:2005.00559 (2020).

[9] Xu, Zhan, et al. "Morig: Motion-aware rigging of character meshes from point clouds." SIGGRAPH Asia 2022 Conference Papers. 2022.

[10] Yao, Chun-Han, et al. "Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery." Advances in Neural Information Processing Systems 35 (2022): 15296-15308.

[11] Wu, Yuefan, et al. "Casa: Category-agnostic skeletal animal reconstruction." Advances in Neural Information Processing Systems 35 (2022): 28559-28574.

We thank 2K9U for noting the (1) originality of and flexibility of our loss function DR, (2) its validation in the paper, and (3) our use of inverse blend skinning for skeleton discovery.

We greatly appreciate 2K9U for the weaknesses noted and the questions they have raised. We outline below our responses and how we plan to use these comments to improve the quality of our paper.

Presentation Improvements:

We thank 2K9U for valuable suggestions about improving the readability, and we will implement these suggestions as explained below.

1. We agree with the suggestions to rename some of our terms for better understanding, as follows.

(a) We will replace our term "semi-rigid part decomposition" with "skinning weights," which is a more conventional term within the graphics and vision literature.

(b) We will replace our term "implicit motion representation" with "latent representation," to avoid confusion with explicit transformations; we agree that "latent" better captures the intended underlying skeletal movements that are not directly observable.

(c) We will replace our term "inverse blend skinning" (IBS) with simply "blend skinning" to align with standard practices and better convey its true role; we will also provide a more detailed explanation of it in mapping vertex properties to bones with an existing set of skinning weights.

2. We will add pointer sentences to guide the reader through our methodology section. For instance, we will clearly state and explain the relevance, with reference to Section 3, of the rationale behind computing bone motion direction introduced in Eq (4).

3. Purpose of Section 3.2: We agree that the discussion in this section (about our method's ability to integrate with NeRF-based and NMR-based approaches) may distract the reader from the primary focus of our method, and will be better located as part of implementation details; accordingly, we will move this discussion to the implementation section.

Related Works and Discussion:

(a) We thank 2K9U for pointing us to additional related works. We find these useful, will refer to them, and contrast our approach with these methods, particularly highlighting the differences in requirements for prior information and applicability in in-the-wild scenarios.

(b) We agree that a comparison of our work with the approaches mentioned by 2K9U, particularly with Rignet [8] will strengthen the paper. We will include these in our discussion and also note the differences in that they require extensive ground truth 3D annotations such as high-quality mesh, skeleton, and skinning weights, which are expensive and challenging to obtain. An example of a comparison between RigNet and LIMR can be found at https://drive.google.com/file/d/18evWJ-uBl7U7NoHH0PN1tKiz_7FAFX1U/view?usp=sharing. These results show that our LIMR yields skeletons comparable to those given by RigNet, even though we do not require any 3D supervision.

(d) Regarding alternative bone merging strategies: We have explored the use of the similarity of SE(3) transformations for bone merging. However, we found this approach to be not ideal due to the non-sparsity of skinning weights, which makes the motion of vertices (parts) depend on multiple bones, especially in the early stages with around 200 bones. Therefore, representation of the motion of each part by the SE(3) of one bone will lead to unreliable motion similarity assessments. Upon further exploration, we discovered that applying optical flow warp techniques significantly enhances the accuracy and logical coherence of part motions. We will add this discussion in the updated paper.

(e) General Discussion on Our Method vs Related Works: Methods like [2], [6], and [10] require a ground truth skeleton or pre-trained model like Rignet to obtain the skeleton. [1] gives decent results given multiple videos captured from different views and ground truth camera poses. In contrast, as discussed in sections A.5 (NeRF-based methods VS NMR-based methods), such methods ([1], [2], [3], [4], [6]) struggle to obtain decent results given a short monocular video. Among current methods, Hi-Lassie [7] shows the best results in our setting (short monocular video without extra annotations and camera pose), and we will add a comparison with it in our paper; illustrative results we have obtained are at the link: https://drive.google.com/file/d/1VRdZeV0GK3pELwL8Y7u8OYHuXW9xWAoI/view

In contrast, we believe that our method's ability to learn a satisfactory skeleton structure from a short monocular video without any prior skeletal, shape information, or camera poses is a significant contribution. We will revise our paper to emphasize this distinction, include the limitations of other methods in in-the-wild scenarios, and provide a visual comparison.

We are thankful to dR3E for noting the interestingness of the ideas of learning skinning and differentiable skeleton our method uses, and that these may inspire other representations and general dynamic scene modeling and greatly appreciate dR3E for the weaknesses noted and the questions they have raised. We outline below our responses and how we plan to use these comments to improve the quality of our paper.

We disagree with the statement made by the reviewer that the model-based methods are the SOTA. Despite the use of predefined models in model-based methods that endow them with a wealth of prior knowledge, intuitively, one would expect their performance to surpass that of template-free methods. However, experimental results often contradict this assumption. For instance, references [2] (Fig 1) and [1] (Fig 6) illustrate that the outcomes of model-based approaches ([3],[4],[5],[6]) are suboptimal despite employing predefined templates. Several reasons account for this:

1. Model-based methods assume that the shape template is sufficiently accurate, significantly restrict the shape's deformability, and permit only simple transformations such as stretching the length of each bone. They do so because (1) when mesh deformation is unconstrained, the mesh often loses detail and transitions into a crude shape in the first several stages, resulting in the complete loss of the original prior information, and (2) even if the shape were allowed to undergo substantial changes, the overall scale or scale of each part might vary, potentially creating new parts, which prevent the mesh from aligning with the skeleton template. The adverse impact of this restriction is seen in recent works such as [12] and [11], where the results closely resemble the initial shape templates, focusing primarily on learning pose without reconstructing the unique detailed shapes of each instance. For example, in [12], since the video of a cow used a buffalo template, the resulting model still retained horns, and the A-CSM [5] reconstructed a camel with only one hump, despite the video featuring a two-humped camel (shown in [2] Fig 1)

2. Although there are templates for many animal species, individual variances, even within the same species, are vast. For example, the camel and horse templates used in A-CSM [5] and SMALify [13] differ greatly from actual camel instances, and often, these instance shapes do not exist within the deformation space allowed by these template-based methods. Hence, even with the correct pose, the rendered silhouette may significantly differ from the ground truth silhouette.

3. In real-world applications, out-of-distribution (OOD) objects are inevitable, and articulated objects are not limited to quadrupeds and humans. For instance, our tests (Table 1) included dinosaurs (Tyrannosaurus Rex) that cannot be accommodated by any existing quadruped/human templates. Such cases highlight the need for more flexible and adaptable reconstruction techniques.

2. Ignoring the articulation and directly learning general dynamic 4D functions often yields unsatisfactory results because such methods require a large number of videos, captured from multiple viewing angles, and ground truth camera poses. Also, it leads to a large number of parameters, entailing lengthy training periods (e.g., D-NeRF [7] needs over two days as it learns per-point motion without blend skinning), and making it difficult to converge (mentioned in [2]). Also, they rely on accurate camera pose (mentioned in [8], [9]) and have difficulties handling large motions (mentioned in [9]). They always leverage COLMAP to obtain the accurate camera pose, whose performance is known to be bad when the input video is short and the object has a large motion. These requirements are not met by our, more challenging scenario of a single monocular video without camera poses. Hence, directly getting the geometry and then long-term correspondence is not an option in our case.

3. Importances of Components: To the best of our understanding and experience, we feel that in a challenging task like the one we have attempted in this paper (single, monocular, short video without prior template), it is not easy to depend on one or a few sources of information to estimate the shape, joint structure, and articulated motion. Because of the highly compressed, convoluted, and time-varying interdependence of the sources, we think any estimation algorithm will have to make joint use of one or more of the information sources captured in our formulation and loss function, their relative contributions possibly varying from time to time. We therefore think that there is no single component in the framework that is consistently important to reach the end goal. This distributed nature of information is confirmed by the related previous papers [1-1], which also do not quantify the relative contributions of the different sources beyond presenting the ablations. In our paper Sec. 4.3, we have presented the following ablation studies to demonstrate the effectiveness and analyze the reasons for the effectiveness of the various modules: (a) Physical Skeleton (ours) vs Virtual Bones (b) Refined Skeleton (ours) vs Initial Skeleton (Mesh Contraction) (c) Different Thresholds for Skeleton Refinement (d) Impact of Video Content on Skeleton (e) Efficacy of Dynamic Rigidity and Part Refinement

Although Reviewers Z1Zw and gF7P have expressed satisfaction with the range of these ablations, we would be grateful if you (dR3E) could help us with which other ablations we could add; or how else could we specify module importances (given that there is no single consistently dominant module). We look forward to your suggestions about any additional experiments, analyses, etc. that we could include to improve our paper. In addition to a-e above, we agree with your suggestion and plan to provide ablation studies on different parameters of the mesh contraction module.

4. We would be interested to know in what ways we could improve the quality of reconstruction to make it more appealing. The results we have shown are analogous to those presented in SOTA papers; we have shown that our results are superior to those of the SOTA methods [2], under the same conditions, and even better than those of the methods that use additional information (e.g., Banmo [8], that uses DensePose to predict camera pose). Despite our challenging experimental setup (single short monocular video without any shape/skeleton prior and without ground truth camera), LIMR has shown impressive results (mesh and skeleton). We will provide analogous comparisons of our mesh results with a more recent SOTA method [10] at https://drive.google.com/file/d/1VRdZeV0GK3pELwL8Y7u8OYHuXW9xWAoI/view; we will add these to the paper.

[1] Tan, Jeff, Gengshan Yang, and Deva Ramanan. "Distilling Neural Fields for Real-Time Articulated Shape Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Yang, Gengshan, et al. "Lasr: Learning articulated shape reconstruction from a monocular video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[3] Rueegg, Nadine, et al. "Barc: Learning to regress 3d dog shape from images by exploiting breed information." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[4]Saito, Shunsuke, et al. "Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[5] Kulkarni, Nilesh, et al. "Articulation-aware canonical surface mapping." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[6] Kocabas, Muhammed, Nikos Athanasiou, and Michael J. Black. "Vibe: Video inference for human body pose and shape estimation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[7] Pumarola, Albert, et al. "D-nerf: Neural radiance fields for dynamic scenes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[8] Weng, Chung-Yi, et al. "Humannerf: Free-viewpoint rendering of moving people from monocular video." Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition. 2022.

[9] Yang G, Vo M, Neverova N, et al. “Banmo: Building animatable 3d neural models from many casual videos” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[10] Yao, Chun-Han, et al. "Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[11] Goel, Shubham, et al. "Humans in 4D: Reconstructing and Tracking Humans with Transformers." arXiv preprint arXiv:2305.20091 (2023).

[12] Wu, Yuefan, et al. "Casa: Category-agnostic skeletal animal reconstruction." Advances in Neural Information Processing Systems 35 (2022): 28559-28574.

[13] Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla. Creatures great and smal: Recovering the shape and motion of animals from video. In ACCV, pages 3–19. Springer, 2018.

I appreciate the authors' effort in responding to each reviewer's comments with such detail. After reading both the comments and the authors' rebuttal, I am partially convinced that their approach makes sense under the context and setting of this paper. However, I suggest that the authors take the time to carefully clarify the novelty and motivation of their revision. Additionally, as other reviewers have pointed out, improving the presentation would be helpful in making the paper clearer. I have slightly raised my rating but have also lowered my confidence in the paper.

[1] Noguchi, Atsuhiro, et al. "Watch it move: Unsupervised discovery of 3D joints for re-posing of articulated objects." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Kuai, Tianshu, et al. "CAMM: Building Category-Agnostic and Animatable 3D Models from Monocular Videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Yang G, Vo M, Neverova N, et al. “Banmo: Building animatable 3d neural models from many casual videos” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[4] Wu, Shangzhe, et al. "Magicpony: Learning articulated 3d animals in the wild." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Yang, Gengshan, et al. "Lasr: Learning articulated shape reconstruction from a monocular video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[6] Yang, Gengshan, et al. "Reconstructing animatable categories from videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[7] Yao, Chun-Han, et al. "Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[8] Wu, Yuefan, et al. "Casa: Category-agnostic skeletal animal reconstruction." Advances in Neural Information Processing Systems 35 (2022): 28559-28574.

[9] Xu, Zhan, et al. "Rignet: Neural rigging for articulated characters." arXiv preprint arXiv:2005.00559 (2020).

We are thankful to gF7P for noting (1) the novelty of our adaptive learning mechanism; (2) the advantage of not requiring category-specific training that our method offers, (3) the improvements it makes over prior works; and (4) the extensive ablation studies we present.

We greatly appreciate gF7P for the weaknesses noted and the questions they have raised. We outline below our response and how we plan to use these comments to improve the quality of our paper.

(1) Ablation of considering optical flow visibility (Eq. 4) and Laplacian contraction: Thanks for pointing these out. We agree that this ablation will be useful. Our experiments reveal that removing visibility leads to instability and inaccuracy in calculating 2D bone motion when mapping 2D flow to 3D space. This is because typically each ray intersects the mesh at two points, with the 2D flow corresponding to the visible point. The non-visible point, often from a different part with an entirely different motion, causes instability when visibility is disregarded, resulting in an unreasonable skeleton. We will add this ablation to the main paper. We will also supplement it with ablation studies of the hyper-parameters of the mesh contraction module, including visual comparisons.

(2) Effectiveness of Dynamic Rigidity (DR): The 2D key point transfer accuracy we measure is primarily from visible views, and this does not effectively assess the physical plausibility of the entire mesh. We agree that Figure 9 could be enhanced by including more views to show the mesh differences (e.g., the unreasonable depressions in the body in the 'without DR' experiment); accordingly, we will add more views and animal comparisons to better demonstrate the improvements brought by DR.

(3) Comparison with MagicPony [4], WIM [1], CASA [8], and other Related Works: CAMM [2], and RAC [6] are similar as they all require ground truth skeletons (or pre-trained RigNet [9] to obtain skeleton) as input and leverage implicit fields (NeRF-based). As discussed in our Section A.5 (NeRF-based methods vs. NMR-based methods), such NeRF-based methods ([1],[2],[3],[4],[6]) place significant requirements on input videos in terms of the numbers of viewing angles and frame counts (over 1000 frames), as well as on ground truth camera poses (or use of pre-trained PoseNet [3] to obtain camera pose). When the input is a short monocular video, these methods fail to achieve satisfactory results (as seen from our Fig. 3 (b)). We will show more results from NeRF-based methods in the Appendix. In addition, we have found another recent work, Hi-lassie [7], which is similar to Magic Pony [4] but is based on NMR and yields acceptable results on monocular videos; our method, however, outperforms it as shown in https://drive.google.com/file/d/1VRdZeV0GK3pELwL8Y7u8OYHuXW9xWAoI/view. Further, we attempted to run CASA [8] but did not succeed in doing so since the authors did not release the ground truth skeleton for each animal, which is required to run the code as they use both predefined templates and skeletons. This makes a direct comparison difficult. We reached out to the authors multiple times via email but received no response.

We will include a comparative discussion of all these methods in the paper.

(4) How should we interpret the improvement by Dynamic Rigidity (DR): As shown in Figure 9, using ARAP terms leads to unnatural and uneven depressions in the reconstruction. We believe this is because the ARAP term uniformly encourages distance preservation between all connected vertices in the mesh, which is unreasonable because certain areas (junctions of different parts) should have the freedom to deform, and only distances within semi-rigid parts should be restricted. Changing ARAP to DR allows deformations to occur more at the junctions of parts rather than within semi-rigid parts, thus maintaining the shape integrity of each part. In contrast, using ARAP, deformations might occur within a part, resulting in unreasonable depressions or uneven shapes. We will add this explanation to the paper.

I appreciate the authors for the updated manuscript with the additional visualizations and descriptions. The questions/weaknesses have been largely addressed except for the two items: ablation on flow visibility, and Laplacian contraction. I appreciate the authors’ effort in providing an additional explanation for the above items, however, it would be nicer to include the actual qualitative/quantitative results in the updated manuscript. As reviewer 2K9U also points out, it will make the paper stronger if a direct comparison of WIM on bone discovery is presented as well.

We are thankful to Z1Zw for noting that: (1) our method effectively models both canonical structure and deformation. We conduct (2) a fair comparison with SOTA, (3) extensive ablation studies, and (4) discuss implementation and limitations in detail.

We greatly appreciate Z1Zw for the weaknesses noted and the questions they have raised. We outline below our response and how we plan to use these comments to improve the quality of our paper.

1. Visual Comparisons: We will show videos of moving objects on the project page; we are including some examples here: https://drive.google.com/drive/folders/1IUrSi-1VcBtr4FbIRrRSt9PuVrKYFE14.

2. Hyperparameters and Robustness: The impact of video content and threshold hyperparameters related to skeleton updates on the reconstruction process is discussed in the experiments section. Further, we will: (a) elucidate the effects of skeleton initialization using different mesh construction parameters; (b) stability of results by comparing the results from multiple runs and the standard deviations; and (c) demonstrate the impact of input video contents by showing results for varying viewing directions.

3. Ordering of Figures and References: We acknowledge this weakness and will rectify it by renaming the figures.

4. Explaining Notation: We will add the clarification that $N_i$ denotes the set of neighbor vertices of vertex $i$, as well as look for any other such places for improvement.

5. Typographical Errors: Thank you for noting this typographical error. We will replace the incorrect reference to "quantitative" in the Appendix with "qualitative".