TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene

Time consumed per iteration: Thank you for bringing this to our attention. Our approach takes around 0.3 sec/frame with INN while 0.9 sec/frame with the Broyden approach used in TAVA. We will include this information in our final paper.

Information on how we obtain semantic masks: Please refer to the Author Rebuttal section for the response.

Gives impression of general dynamic scenes, but works on human subjects: Please refer to the Author Rebuttal section for the response.

Method does not work for scenes without humans or datasets lacking 3D skeleton information: We acknowledge that our method relies on 3D skeleton information to constrain the network and produce accurate shapes and geometries of the scene objects. However, this 3D skeleton information can be easily obtained for both rigid and non-rigid objects. For rigid objects, skeleton joints are derived from uniformly sampled 3D points along the x, y, and z axes, passing through the center of the 3D bounding box (obtained from multiview semantic masks) surrounding the object. For non-rigid objects, 3D skeletons can also be easily acquired using available tools (e.g. [3],[6], etc.).

Table 3, HyperNeRF is second best: Thank you for pointing out the error. We will rectify the error in the final submission.

[3] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Black Michael J., and Tao Mei. Monocular, One-stage, Regression of Multiple 3D People. In ICCV, 2021.
[6] Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie W. Mathis, and Matthias Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 2018.

Hard to follow introduction: Thank you for your feedback regarding the introduction. We apologize if the storyline was challenging to follow. We will carefully review and revise the introduction to ensure it is clearer and more cohesive. Your input is valuable, and we appreciate your efforts to help us improve our work.

Claim on multiple dynamic entities: Please refer to the Author Rebuttal section.

Claim on addressing occlusion challenge, but no experiment: Thank you for the insightful review. We have included an experiment (Figure 3 in the attached PDF in Author Rebuttal section).

Our goal is to generate a 3D reconstruction of the scene without using any template model. Unlike methods that use the T-pose or A-pose of SMPL models as canonical representations to handle self-occlusions, we select a keyframe from the sequence as the canonical pose. Consequently, if there is no frame in the sequence without occlusion, the SDF network struggles to capture the occluded geometry in the canonical space.

To address this occlusion challenge, we use a strategy that disentangles the motions of distinct entities. This involves semantic-aware sampling and distance-based encoding of 3D points on the rays. Additionally, we employ separate SDF prediction networks for each entity, which helps mitigate the issue. Our overall design helps mitigate the occlusion problem.

Contribution compared to SemanticFlow Thank you for mentioning this method. SemanticFlow primarily focuses on semantic rendering and scene editing. In contrast, our work is focused on semantic geometry reconstruction. Techniques that excel in rendering may not necessarily perform well in 3D geometry reconstruction ([4], [5], [7]). Therefore, our primary emphasis has been on methods that are strong in geometry reconstruction. However, we are considering evaluating this method to validate its effectiveness and to determine if it could be integrated into our final submission if appropriate.

Separates the points on the rays based on the semantic labels of the pixels We apologize for not being clear in explaining this approach. We utilize information from both 2D semantic maps and 3D skeletons. While the rays are sampled in image space within 2D bounding boxes around each entity, we also perform an encoding for every 3D point on the sampled rays based on their distance from the 3D skeletons of individual elements. As mentioned in lines 230-238 of the initial submission for details, to provide the SDF prediction network information about which object the point belongs to, we assign a semantic label to each 3D point. This is defined as a weightage, $\omega^j = \exp(-dist^2/\sigma^2)$ based on the distance of the point from the nearest 3D joints in the per entity canonical skeletons $\mathbf{J^j_{p0}}$.

Formatting issues Thank you for the detailed reviews and for pointing out these errors in the manuscript. We will rectify all these errors in the final paper.

[4] Huang B, Yu Z, Chen A, Geiger A, Gao S. 2d gaussian splatting for geometrically accurate radiance fields. InACM SIGGRAPH 2024 Conference Papers 2024 Jul 13 (pp. 1-11).
[5] Guédon A, Lepetit V. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 5354-5363).
[7] Martin-Brualla R, Radwan N, Sajjadi MS, Barron JT, Dosovitskiy A, Duckworth D. Nerf in the wild: Neural radiance fields for unconstrained photo collections. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2021 (pp. 7210-7219).

Dear Reviewer 3yt5,

As the discussion period is drawing to a close, we would greatly appreciate it if you could let us know if there are any further clarifications or additional information we can provide to assist with any remaining questions.

We have addressed the concerns you raised regarding unclear expressions for multiple dynamic entities, the lack of clarity on semantic-aware ray sampling, the perceived limited contribution compared to the Semantic Flow paper, and the missing experiments related to our claim on occlusion handling. Detailed responses to all these concerns are included in the Author Rebuttal and the review-specific rebuttal section. Additionally, we have added new experiments on occlusion handling in the attached PDF.

Thank you once again for your thorough review and valuable suggestions on paper writing. We will ensure these points are addressed in the final version of the paper.

Thank you for your feedback.

• We would like to emphasize the main contributions of our paper:

  • We present a time-efficient, template-free NeRF-based 3D reconstruction method for dynamic scenes involving two interacting entities.
  • We focus on the semantic reconstruction of dynamic scenes, emphasizing the detailed and explicit geometry of each entity within the scene. \

  While our system does address occlusion challenges that typically arise during entity interactions, it's important to note that handling occlusion is not the primary contribution of our work, and we did not claim it as such. Since this is not a main contribution, it should not be a reason for an unfavorable rating.

• We agree that "generic scene" or "multiple entities" terms can be misleading, and we are willing to revise these phrases. However, we believe these are minor adjustments.

• We agree that Lines 175-177 might be difficult to understand as they require forward reference for the details on label assignment. We believe this also needs only a minor revision, and we are happy to make these changes in the revised version.

• Regarding "$\downarrow$ Methods/Metric $\rightarrow$" in Tables 5 and 6: The $\rightarrow$ symbol indicates that the columns represent metrics. "$R^{1+256}$" defines that the SDF prediction network predicts both an SDF value and a global feature representation of the scene (as used in general NeRF). We will clarify these in the revised paper. These are straightforward fixes and are considered minor comments.

How is the semantic masks calculated: Please refer to the Author Rebuttal section.

Limiting novelty and distinctiveness w.r.t TAVA, INN: Our method focuses on the research gap of producing semantic 3D reconstruction of dynamic scenes under multiple object interactions. While our problem setup of reconstructing the scene guided by a 3D skeleton is similar to TAVA, extending this concept for various object interactions is not trivial. Moreover, TAVA relies on an iterative method for solving the forward LBS, which is not time-efficient. In our method, we utilize INN to make the solution more time efficient, which makes it even more challenging under multiple object interactions setup, causing occlusion and different motions.

Classifier for rigid, non-rigid objects: Thank you for your question. Based on the semantic masks, it is indeed possible to infer whether an object is rigid or non-rigid. We do not explicitly employ a separate classifier for this purpose. Instead, our approach leverages the information provided by the semantic masks to distinguish between rigid and non-rigid objects.

Number of rays sampled for rigid and non-rigid objects: The rays are sampled in a 60:40 ratio for the non-rigid and rigid objects, respectively.

Sorry if there is any confusion regarding the ground-truth segmentation masks.

• All the methods (SOTA and ours) in Tables 2,3, and 4 in the initial submission are trained with ground-truth segmentation masks. So, the values for our method given in Table 1 of the attached PDF in rebuttal are not directly comparable with SOTA metrics in Tables 2,3, and 4 in the initial submission.

• We acknowledge that we require 3D pose information which is not required by the SOTA methods. However, using a 3D skeleton helps in achieving much better reconstruction quality (as you will be able to find out in the qualitative results of Figure 5 in the main paper). Without using any constraints on the structure of the entities it is difficult to achieve a good reconstruction quality, especially when there are multiple entities present and entities are in large motions. Also, it should be noted that using the predicted poses does not affect the quality of the reconstruction (Figure 1 in rebuttal PDF) and the metric value decrease occurs due to predicted pose inaccuracies.

• We would also like to emphasize another point, that we do not use any off-the-shelf skinning model (like SMPL). Rather we predict the skinning weights (Figure 3 in the main paper) using an MLP and train an end-to-end network for the reconstruction.

Thank you for your question. Regarding Tables 2 and 3, a single NeRF model was trained for the entire scene, which includes both the human and the object.

Robustness to mask accuracy: Please refer to the Author Rebuttal section.

Robustness to pose accuracy: We agree that for the practicability of TFS-NeRF, it is important to evaluate it from this aspect, and apologize for not presenting the same in the initial submission. We have added quantitative results for the reconstruction quality of our method using predicted 3D poses (Table 1 in the attached pdf in Author Rebuttal section). We generate these 3D poses by using the state-of-the-art 3D pose estimation network [3]. As the results show, our method needs a good quality 3D pose to constrain the shape and motions of the individual elements. Even though the reconstruction quality does not get affected, the inaccuracies come from the predicted pose (Figure 1 in the attached pdf in Author Rebuttal section). We will include this evaluation in the final paper.

Comparison with current methods (Hexplane, Tensor4D, GS-based methods): Thank you for highlighting these methods. We primarily compared our approach with techniques that emphasize geometry reconstruction, such as NDR, ResField, D-NeRF, and HyperNeRF, which predict Signed Distance Functions (SDF) for underlying geometries. We agree that including Tensor4D in the comparison is valuable and have conducted additional experiments to assess its performance relative to our method (Figure 2 and Table 2 in attached pdf in Author Rebuttal section).

Our experiments show that our method provides superior reconstruction quality compared to Tensor4D. While Tensor4D's key innovation lies in its efficient 4D tensor decomposition, which represents dynamic scenes as a 4D spatiotemporal tensor, it does not account for the relative motions between scene elements. Instead, it captures the scene's overall motion. In contrast, our approach models the motion of individual elements and employs skeleton-guided reconstruction, leading to more accurate geometry.

Additionally, current GS-based methods for dynamic scenes primarily focus on rendering rather than surface reconstruction. Research indicates that methods emphasizing geometry reconstruction often fall short in rendering quality, and vice versa (refer to Table 3 in 2DGS [4], Table 1 in SuGaR [5]). Similarly, Hexplane prioritizes image rendering over geometry reconstruction which is a NeRF-based method. Similar arguments are applicable for the NeRf-based method [7]. Therefore, these methods may not be directly comparable. Hence, given the limited time frame for rebuttal, we are concentrating on the geometry-based method. However, we are open to including these methods in the final comparison.

INN vs Lossy auto-encoder: We appreciate your feedback and would like to clarify any confusion. We chose INN because we expect to maintain the invertible mapping between the 3D point space and the deformation space, which is the property that INN naturally has. A lossy auto-encoder could also become a choice only if it has the same property. We left the model structure exploration as future work to enhance the semantic reconstruction task further.

Computation of $B_i$ for arbitrary objects: We apologize for the lack of clarity in the submission. For arbitrary non-articulated objects, $B_i$​ is computed based on the 3D bounding boxes, which are derived from multiview semantic masks. Specifically, for non-articulated objects, $B_i$​ represents the rigid transformation between the bounding boxes of the canonical frame (keyframe) and each input frame. We will ensure that this explanation is included in the final version of the paper to provide clearer context.

Evaluation on more datasets (daily life objects with door close, opening): Thank you for highlighting this important aspect. Given the limited time frame for rebuttal, we are unable to evaluate additional datasets at this moment. Moreover, we have already evaluated several benchmark datasets (including datasets like BEHAVE containing daily life objects) to showcase the efficacy of our method. We appreciate your understanding and interest in expanding the scope of our evaluation.

Rendering quality: We appreciate your feedback regarding the importance of appearance quality. We have compared our rendering quality with Tensor4D and obtained the following results: PSNR of 30.09 and SSIM of 0.970, whereas Tensor4D achieved a PSNR of 35.78 and an SSIM of 0.985. Methods, that show very good geometric reconstruction quality do not necessarily perform well in rendering and vice versa ([4], [5], [7]). The improvements in geometric detail often reduce the rendering quality, as the optimization process becomes more complex and computationally demanding.​ In our work, we aim to do good reconstruction but we will not care about rendering quality.

Formatting issues: Thank you for the detailed reviews and feedback. We will rectify all these errors in the final paper

[3] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Black Michael J., and Tao Mei. Monocular, One-stage, Regression of Multiple 3D People. In ICCV, 2021.
[4] Huang B, Yu Z, Chen A, Geiger A, Gao S. 2d gaussian splatting for geometrically accurate radiance fields. InACM SIGGRAPH 2024 Conference Papers 2024 Jul 13 (pp. 1-11).
[5] Guédon A, Lepetit V. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 5354-5363).
[7] Martin-Brualla R, Radwan N, Sajjadi MS, Barron JT, Dosovitskiy A, Duckworth D. Nerf in the wild: Neural radiance fields for unconstrained photo collections. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2021 (pp. 7210-7219).

Dear Reviewer Rbt4, KrvP, 3yt5

Thank you once again for your thoughtful reviews. Your insights have significantly helped in improving the quality and clarity of our paper. We sincerely hope that our rebuttal has adequately addressed your questions and concerns. Please let us know if there are any further clarifications or additional information you require. We appreciate your valuable suggestions.

Thank you very much for your time.