MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Field

Thank you for your detailed review and constructive suggestions. We will address your concerns as much as we can. We are also preparing an additional webpage for presenting the video results and ablations you mentioned in the experiments section. We will send it out soon.

(A) Concept of Eulerian and Lagrangian.

It is a good suggestion to explicitly mention forward and backward deformation in the paper and we will add it in a revision. We also agree with the reviewer that both forward and backward methods have some notion of particle. In particular, in D-NeRF or Nerfies, with the backward deformation, their volume density and color (the radiance field properties) are defined in the canonical space and can be thought of being models on the object material points/particles. However, please note that we focus on motion modeling (when it comes to Eulerian and Lagrangian views) here, and the backward deformation function itself is defined in the world coordinate space at fixed locations and cannot be considered as Lagrangian flow. In contrast, our forward deformation is defined in the canonical object coordinate space (on material particles) and exactly corresponds to what the standard Lagrangian flow is. To be more specific, our forward deformation can directly be used to describe the trajectories or pathlines of any object particles through the temporal frames, which is a classical lagrangian description; however, this is what the backward deformation in D-NeRF or Nerfies cannot achieve at all.
Therefore, though the backward deformation is not identical to standard Eulerian flow (as the one the reviewer described in NSFF), we still consider the backward deformation as motions in an Eulerian description since they are defined at fixed world-space locations. We are happy to further clarify this in the paper.

(B) Intractability of tracking particles with backward deformation.

The reviewer is indeed correct that an invertible backward deformation model can achieve a similar goal because the forward deformation can be derived analytically. However, we think this actually aligns well with our argument that a forward deformation module is needed and an invertible function is an alternative way to achieve so. As pointed out by the reviewer, the invertible function also has other costs (lacking expressivity and computationally expensive) and our approach avoids these by modeling the two deformations separately with a cycle consistency loss. We will add the discussion with the invertible deformation in the paper.

(C, D, E, F) Discussions on DeVRF, Nerfies, Articulated NeRF, etc.

Thanks for pointing out the missing references. We will add and discuss them in a revision.

(G) Averaged group feature may be different across different views.

As you pointed out, the motion features utilized for grouping are sampled from points along rays in a single image, with 4096 rays sampled per batch. While there might be slight variations in the corresponding motion features across different views, we haven't observed any convergence issues.

(H) Gradient for Lagrangian module and TV loss.

The loss function for supervising the Lagrangian module includes the cycle consistency loss and total variation loss. We refer to this in the appendix.

(I) Group merging module.

Indeed, as you think, the group merge module is not engaged in training; it serves as a post-processing step to consolidate redundant groups. The APE cost is used as a quantitative measure of the disparities in motion patterns between groups, enabling the aggregation of similar groups.

(J) Piece-wise rigid motion.

Thanks for the suggestion. We primarily focus on general objects with piecewise rigid motion in this paper, and we will emphasize this in a revision.

(K, L, O, P) Video results, ablations and Watch-It-Move.

We are preparing an additional webpage for presenting the video results, ablations, etc. All these experiments' results will be shown in the next feedback soon.

(M) Tracking visualization.

Thank you for your suggestion. We will consider adding this kind of visualization to our paper.

(N) LPIPS.

Thanks for the comment, we will add the LPIPS score in a revision.

(Q) Part segmentation evaluation.

As you said, in section 5.3, we evaluated part segmentation on the 2D images. This is due to the fact that there is no guarantee that NeRF can model the geometry and motion of unobservable insides of objects.

Thanks again for your detailed review and valuable comments. We have revised and updated our draft based on the comments (highlighted in blue). In addition to this, we have created an anonymous website for providing relevant video visualizations, hope it can address your concerns. (https://anonymous418de.github.io/MovingParts/)

In alignment with the order of your questions, below we provide our updates and feedback:

(A, J) Piece-wise rigid assumption and forward/backward deformation.

We emphasize our piece-wise rigid motion assumption and add the description of forward/backward deformation to align with existing dynamic NeRF methods in the Introduction section.

(C) Discussion on cyclic consistency loss of DeVRF.

In Section 4.4, we have introduced a discussion on the cyclic consistency loss of DeVRF. Additionally, in Section A.1, we have included an ablation study to compare their implementation with ours.

(D, F) Discussions on Articulated NeRF and Human NeRF.

We added references and discussions of articulated NeRF and human NeRF in the related work section.

(E) Discussions on SE(3) parametrization of Nerfies.

We added a discussion on Nerfies when using SE(3) as a motion parameterization, which adopts a similar representation to us (In Section 4.2).

(K, L) Video results.

We have visually represented all experimental results discussed in the paper, including the generated Kubric-based dataset and real-world data, etc, in the URL provided above. We hope these videos can help to address your concerns.

(N) LPIPS score.

In our revised version, we have incorporated LPIPS scores for all tabular data and discussions.

(O) Ablation Study.

In Appendix A.1, we have included ablations on the Lagrangian module, cycle consistency loss, and more. To demonstrate the visual difference of these ablated models, we have provided the corresponding video results of these ablation experiments on the provided webpage. As you mentioned, the presence of the Lagrangian module causes a slight drop in the rendering quality, which we consider to be due to the constraints of the rigid body parts of the Lagrangian module. But the quality drop is very marginal (less than 0.2dB); in general, our Lagrangian module allows for part discovery without losing the high rendering quality. It would be interesting to explore if a post-processing step with our module could also work for part discovery. But we think jointly training the two modules at the same time could likely better regularize the motion features, making them more meaningful for parts and leading to better part discovery.

(P) Watch-It-Move on D-NeRF dataset.

On the provided webpage, we have included visualizations of the Watch-It-Move results on the D-NeRF dataset. Notably, the monocular setup of D-NeRF presents a substantial challenge for Watch-It-Move due to its reliance on multi-view information for scene construction. Although it can approximate the images within the training views, it encounters a failure when applied to the test view.

We are very grateful for your response and support of our work. We are happy to see that our revision and additional clarification have addressed most of your concerns. In this revision, we have corrected the typos, and also changed the part you mentioned to help make the paper clearer for readers. Here are our responses to each of your feedback:

Gradient for optimizing D_L.

Thank you for pointing it out. It's worth noting that the rigid transformation decoders, D_L and D_E, actually share parameters. As a result, when employing cycle consistency loss to ensure consistent motion features across both views, it leads to their decoded rigid transformation parameters being consistently aligned as well. Therefore, the parameters of D_L and D_E (which are, in fact, the same network) could be optimized by the gradient in the Eulerian branch. In this revision, we have modified the corresponding part in Appendix A.5 to emphasize this point.

Typos and other modifications in paper.

Thank you for your careful observation, we have corrected this typo in the revised version. Also, we have put "Ours" behind "A1" in Table 2 to highlight our full model in these ablation studies. In Section 4.5, we have emphasized that our group merge module is a post-processing for adjusting the number of parts, which does not affect training.

Articulated objects.

We'd like to point out that articulated objects generally consist of multiple rigid components that are connected by joints. Although these components deform in a piecewise rigid manner, their motion is constrained by the joints, which limits the freedom of possible rigid motions. Specifically, 'Watch-it-move' explicitly models the joints of an articulated object in their method and is limited to handling a single such object. In contrast, our approach does not impose additional joint/connectivity constraints and instead assumes general piece-wise rigid motion. As shown in many examples in our paper, our method effectively handles scenes with multiple disconnected objects moving independently in space, extending beyond the scope of articulated object motion.

We are grateful to you for your time and the review. We will clarify some of your questions or concerns in this feedback.

Non-rigid scenarios.

Please note that the main objective of our method is to discover rigid parts in high-quality dynamic NeRF reconstruction. The rigid motion patterns are used as the evidence of part discovery. Therefore, we make explicit the assumption of our input here, which is the general object with piece-wise rigid motion. We emphasize that the automatic discovery of 3D rigid parts (with rigid motion) in dynamic NeRF is already a highly challenging task and it remains relatively unexplored in prior research. Watch-it-move is the (only) one previous work that has a similar goal to ours, but they can only handle articulated objects (even more constrained than our rigid assumption) and does not work with monocular input. We not only achieve automatic part discovery from monocular input but also achieve high rendering quality on par with previous dynamic NeRF methods that primarily focus on rendering quality and do not address part discovery at all. Please let us know if there are any additional experiments you would like us to undertake under the rigid motion assumption.

Thank you once again for your insightful reviews.

In this revision, we emphasize our piece-wise rigid motion assumption in the introduction section of the updated paper (highlighted in blue). It is crucial to note that even under this assumption, the automatic discovery of rigid parts from a dynamic NeRF remains a highly challenging task. As Watch-it-move is the (only) previous work that had a similar goal to ours, we visualized its results on our used D-NeRF dataset. The monocular setup poses a significant challenge to their method and leads to severe artifacts, as their method relies on multi-view information.

To facilitate a clearer understanding, we have constructed an anonymous website showcasing the visualization results mentioned above and we hope you can find it useful.

https://anonymous418de.github.io/MovingParts/

Thank you once more for your valuable reviews. Please kindly let us know if our response and updated paper/website have addressed your concerns. We are happy to answer your remaining concerns and questions if you have any.

Thank you for the positive feedback, we address your problems below.

Traditional motion segmentation with long-term motion trajectories.

Thanks for pointing out the connection to motion trajectory and motion segmentation. We will cite the papers and add discussion in a revision. In general, different from the motion segmentation methods that mainly focus on recovering long-term point trajectories (spatio-temporal curves) in the 2D image plane from videos, our approach recovers 3D motion trajectories while ensuring multi-view consistency.

Part-wise NeRF with motion segmentation.

It is interesting to combine motion segmentation methods with dynamic NeRF approaches. This connection could enhance the establishment of long-term correspondences, thereby aiding in motion modeling and particle tracking for dynamic NeRF applications. However, it is challenging to generate consistent masks through the entire video, especially when there are strong occlusions, which might lead to blurry reconstruction or flickering rendering. Our approach instead naturally ensures consistency via joint optimization.

Ablations of the proposed modules.

Thank you for the suggestion, we are preparing an additional webpage for presenting more video results and ablations. It will be shown in the next response soon.

Thank you once again for your insightful suggestions and valuable comments. Taking your feedback into consideration, we have implemented the following changes in the paper, which are highlighted in blue for your convenience:

1. We have incorporated a discussion on the traditional motion segmentation you mentioned. Please refer to the paragraph on Part Discovery from Motion in the related work section.

2. We have included ablation studies on our key modules and loss functions, and for detailed information, kindly consult Appendix A.1 in the revised version. Additionally, we have developed an anonymous webpage featuring video visualizations of the ablations and other experimental results. We believe it will provide a more intuitive understanding of the paper. (https://anonymous418de.github.io/MovingParts/)

We sincerely thank you for your review and insightful thoughts. Below we would like to give detailed responses to your comments.

About the material derivative.

Our hybrid motion modeling is not directly connected to the material derivate. The material derivative describes the time rate of change of some physical property of a field, generally linking the Eulerian and Lagrangian descriptions of the same property's derivatives. While we adopt the Eulerian and Lagrangian perspectives from simulation, our hybrid model is designed to model the field motion/flow, instead of the time derivative of any additional field properties. We also do not define traditional field velocity in the model and hence there is no direct relationship with the material derivative. The goal of our designs is to build cycle consistency between the fixed (Eulerian) Cartesian grids and moving (Lagrangian) material points. It will be interesting to explore future extensions of our model to incorporate velocity and other material derivatives in dynamic fields.

The coordinates of the Eulerian and Lagrangian modules.

The Eulerian volume is defined with a fixed Cartesian grid in the world coordinate system and the Eulerian module maps each $x^w$ in the grid at any moment $t$ to the canonical space $x^c$ (i.e., the object coordinate system). In contrast, the Lagrangian volume is the same as the canonical space, aligned with the object at the specific moment $t_c$, and the Lagrangian module continuously tracks the trajectory of each object material particle $x^c$ from the canonical space to any moment $t$ according to the motion feature of each particle stored in the Lagrangian volume.

Motion models other than rigid ones.

In this paper, our goal is to achieve 3D rigid part discovery in dynamic NeRF reconstruction in an unsupervised manner. Please note that this is already a highly challenging task and relatively unexplored by prior arts. Watch-it-move is the (only) previous work that has a similar goal to ours, but they can only handle articulated objects (even more constrained than our rigid assumption) and does not work with monocular input. We not only achieve automatic part discovery from monocular input but also achieve high rendering quality on par with previous dynamic models that focus on rendering only. Potentially, our model can also be extended to handle more complex non-rigid motions in the future. However, the model might require other motion priors (e.g. motions of a specific kind of object), with ideally another low-dimensional motion parameter space, since we solve this problem in a per-scene optimization pipeline without additional part supervision.

Thanks again for the positive feedback. In response, we have created an anonymous webpage that offers an extensive collection of video results, Please check it out if you are interested in it.

https://anonymous418de.github.io/MovingParts/