Neural SDF Flow for 3D Reconstruction of Dynamic Scenes

• Scene Flow Evaluation

Render results. Thank you for the suggestion. We provide the meshes with texture and rendered videos in the updated supplementary material (named supp_vid_rebuttal.mp4 and textured_meshes/). The color of a surface point is obtained either from the current time step or from the corresponding point in future time steps (specified by the title of the file. For example, for the mesh file named geo_1_col_2.ply means that the geometry is from time step 1 and the color is from time step 3). The correspondences are defined by the scene flow computed from SDF flow.

Long-term scene flow evaluation. We now provide the full optical flow evaluation results on the CMU Panoptic dataset and the long-term scene flow evaluation results on the synthetic sequence (up to future 5 frames).

Since we cannot obtain the ground truth scene flow for real-world sequences of the CMU Panoptic dataset, we try to evaluate the scene flow with optical flow by projecting it onto the image plane. In the table below, we provide such evaluation results on all 5 sequences of the CMU Panoptic dataset. The optical flow computed from our scene flow is consistently better than that of NDR.

In addition, we follow the advice of Reviewer jfGg to directly evaluate the scene flow on the synthetic sequence shown in Figure 7 of the main script. The results are reported in the table below. We compare the results of three models: NDR (Cai et al., 2022), Ours, Ours with scene flow regularization which includes several regularization computed from SDF flow and scene flow (Please refer to the response to Reviewer nuCZ W-3 or Section A6 of the main script for more details). We also evaluate the long-term correspondences up to 5 frames. In particular, for every surface point in a frame, we compute the scene flow to the future first, second, and 5-th frames ("+1", "+2", and "+5"). We report the average end-point-error (EPE) in millimeter. Our model without any explicitly scene flow regularization consistently outperforms NDR (Cai et al., 2022). With further regularization, the scene flow can be drastically improved.

• Integral Evaluation

Yes, the temporal integration scales linearly with scene length. The second-order Runge-Kutta solver requires $23*2=46$ function evaluations for the 24-th frame.

• More Related Works

Thank you for the suggestion. We have updated the related work section to discuss those papers.

• Limitations

Thank you for the suggestion. We add the following discussion about the limitation of our work in the main script.

Similar to all other multi-view reconstruction works, the reconstruction quality highly rely on the number of views. Another limitation of our SDF flow is the time complexity. To solve for the integral (Equation 7) for any query point, the number of function evaluation grows with respect to time. We aim to address those limitations in the future work. For the first limitation, since we now have an explicit relation between the scene geometry and scene motion, we may be able to achieve better geometry with limited number of views by seeking extra regularization from motion such as optical flow. For the second limitation, a potential solution could be that instead of integrating from the initial time step, we can integrate within a fixed time window.

• Input Assumptions

We acknowledge that the SDF itself does not have any restrictions on its temporal evolution. The continuity and differentiability of the SDF as a function of time for a given 3D point come from the assumption that the scene geometry evolves in a continuous way such as cars running on the road, and all human activities. Although such assumption does not hold when an object appears out of nowhere, we would like to argue that such case is scarcely possible in our daily life.

• Wrong Argument

Thank you for pointing it out. We have updated that paragraph as follow to avoid confusion.

To extend NeRF to dynamic scenes, the most commonly adopted strategy is to jointly optimize an additional function that models the deformation in 3D space such function either maps all observation spaces to a canonical one such as (Park et al., 2021a) or models the temporal motion of the scene such as (Li et al., 2021). In the next section, we propose a drastically different representation, i.e., SDF flow, which can also model dynamic scenes.

• Labelling

Thank you for pointing it out. We have updated the Figure.

• Tangential Motion

Thank you for the suggestion. We have updated the exception analysis in Section A2 of the main script as follow.

Given the SDF flow of 6 points on a surface that moves smoothly and rigidly, one can obtain the scene flow $\begin{bmatrix}\\boldsymbol\\omega\\\\\\mathbf{v}\end{bmatrix}$ of the surface by solving a linear equation

$$\\mathbf{d} = -\\mathbf{A}\begin{bmatrix} \\boldsymbol\\omega\\\\ \\mathbf{v} \end{bmatrix},$$

where $\\mathbf{d}\\in\\mathbb{R}^6$ is the SDF flow of those 6 points. $\\mathbf{A}\\in\\mathbb{R}^{6\\times6}$ whose rows are computed from the normals and locations of those 6 points. To the best of our knowledge, the only exception when we cannot obtain the correct scene flow is that when the scene motion does not lead to any SDF changes to any 6 points on the surface, thus, $\\mathbf{d}=\\mathbf{0}$. In this case, there are many solutions to the linear equation. For example, a plane moving along its tangent direction or a ball rotating around its centre. For the former, any velocity on the tangent plane could be the solution. For the latter, any angular velocity by the ball centre will work. Note that, the linear relation between scene flow and SDF flow defined in Equation 16 still holds on these exceptions. In fact, as long as the surface and scene motion is smooth, Equation 16 holds.

We would also like to clarify that tangential motion does not necessarily lead to degenerate case when using the above linear equation to solve for scene flow. For example, given a ball translating in the space, there are always points on the ball where the velocity direction is along their tangent planes. However, we can always find another 2 points, together with any of those points, to solve for the velocity of the ball (we use 3 points because there is only translation.).

• Topology Change

There are a few topology change instances in the supplementary video: at 00:10 when the woman's hands leaving the table, at 00:19 when the two girls shaking hands, at 00:24, when the two hands of the guitarist in the middle are put apart.

We acknowledge that all works based on SDF, such as Shao et al. (2023), including ours can handle the topology change. For those works that map observation frames to canonical frame such as Cai et al. (2022); Park et al. (2021a), they require to bring the input space into a higher dimension space to handle the topology change as has done in HyperNeRF (Park et al., 2021b). For other works that estimate per-frame densities and use scene flow to enforce the temporal constraints of the scene such as Li et al. (2021); Song et al. (2022), they can also handle the topology change due to the per-frame density estimation.

At last we would like to emphasize that handling topology change is not the focus nor the contribution of our work. Our main contribution is the novel dynamic scene representation: SDF flow which can do whatever SDF-based representations can and we further derive the mathematical relation between SDF flow and scene flow which bridges the scene geometry with scene motion. We believe such relation will benefit the community of dynamic scene reconstruction. To avoid misunderstanding, we have revised paragraphs with topology changes

• Theoretical analysis about the assumption

For assumption 1, it assumes the scene to be reconstructed is piece-wise rigid which is a common assumption in modeling scene motion (Vogel et al., 2013).

For assumption 2, we provide a more detailed description as follow.

Let us denote a surface point as $\\mathbf{x}$, its corresponding point on the evolved surface as $\\mathbf{x}'$, the normal of the tangent plane at $\\mathbf{x}'$ as $\\mathbf{n}(\\mathbf{x}')$, and the closest point on the evolved surface to $\\mathbf{x}$ as $\\mathbf{x}''$. According to the definition of SDF, the absolute SDF change of $\\mathbf{x}$ i.e., $|\\Delta s|$ is equal to $||\\mathbf{x}-\\mathbf{x}''||_2$. With the assumption that the infinitely small surface region around $\\mathbf{x}$ as well as its deformation is smooth, as the time period $\\Delta t$ approaches zero, $\\mathbf{x}''$ becomes infinitely close to $\\mathbf{x}'$ which can be regarded as lying on the tangent plane to the evolved surface at $\\mathbf{x}'$. So that the absolute SDF change $|\\Delta s|$ of $\\mathbf{x}$ will equal to the distance from $\\mathbf{x}$ to such tangent plane i.e., $|(\\mathbf{x}'-\\mathbf{x})^T\\mathbf{n}(\\mathbf{x}')|$. The sign of $\\Delta s$ is determined by angle between the scene flow and the normal of the tangent plane.

We have updated the Figure 3 for better visual explanation of the assumption.

• Time consuming

Thank you for pointing it out. We acknowledge that the time complexity is one of our limitations and will try to improve it in our future research. We also add the following discussion about the limitation in the script.

Similar to all other multi-view reconstruction works, the reconstruction quality highly rely on the number of views. Another limitation of our SDF flow is the time complexity. To solve for the integral (Equation 7) for any query point, the number of function evaluation grows with respect to time. We aim to address those limitations in the future work. For the first limitation, since we now have an explicit relation between the scene geometry and scene motion, we may be able to achieve better geometry with extra regularization from motion such as optical flow. For the second limitation, a potential solution could be that instead of integrating from the initial time step, we can integrate within a fixed time window.

References

Christoph Vogel, Konrad Schindler, and Stefan Roth. Piecewise rigid scene flow. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1377–1384, 2013.

• Scene flow evaluation

Thank you for the suggestion. We now provide the full optical flow evaluation results on the CMU Panoptic dataset and the scene flow evaluation results on the synthetic sequence.

Since we cannot obtain the ground truth scene flow for real-world sequences of the CMU Panoptic dataset, we try to evaluate the scene flow with optical flow by projecting it onto the image plane. In the table below, we provide such evaluation results on all 5 sequences of the CMU Panoptic dataset. The optical flow computed from our scene flow is consistently better than that of NDR.

In addition, we directly evaluate the scene flow on the synthetic sequence shown in Figure 7 of the main script. The results are reported in the table below. We compare the results of three models: NDR (Cai et al., 2022), Ours, Ours with scene flow regularization includes several regularization computed from SDF flow and scene flow (Please refer to the response to Reviewer nuCZ W-3 or Section A6 of the main script for more details). We also evaluate the long-term correspondences up to 5 frames. In particular, for every surface point in a frame, we compute the scene flow to the future first, second, and 5-th frames ("+1", "+2", and "+5"). We report the average end-point-error (EPE) in millimeter. Our model without any explicitly scene flow regularization consistently outperforms NDR (Cai et al., 2022). With further regularization, the scene flow can be drastically improved.

• Details about the scene flow optimization

Thank you for the comments, we clarify the details as below and add them to the Section A2 of the main script.

Given a set of points $\\mathcal{P}$ sampled uniformly from a reconstructed mesh, for any point $\\mathbf{x}$ in this set, we first obtain its $K$ nearest neighbours $\\{\\mathbf{y}_1,\\mathbf{y}_2,\cdots,\\mathbf{y}_K\\}$, where $\\mathbf{y}_i\\in\\mathcal{P}$. For every neighbouring points $\\mathbf{y}_i$, we query its surface normal $\\mathbf{n}(\\mathbf{y}_i)$ and SDF flow $d(\\mathbf{y}_i)$ from the MLPs. According to Equation 17, we obtain a matrix $\\mathbf{A}\\in\\mathbb{R}^{K\\times 6}$ where the $i$-th row $\\mathbf{A}_i=[(\\mathbf{y}_i\\times\\mathbf{n}(\\mathbf{y}_i))^T,\\mathbf{n}(\\mathbf{y}_i)^T]$ and the SDF flow vector $\\mathbf{d}\\in\\mathbb{R}^K$ where $\\mathbf{d}_i=d(\\mathbf{y}_i)$. We can then solve such problem as

$$\begin{aligned} \begin{bmatrix}\\boldsymbol\\omega^*\\\\ \\mathbf{v}^*\end{bmatrix} &= \text{arg min}_{\\boldsymbol\\omega, \\mathbf{v}} || \\mathbf{d} + \\mathbf{A}\begin{bmatrix} \\boldsymbol\\omega\\\\ \\mathbf{v} \end{bmatrix} ||_2^2 \\\\ &= - (\\mathbf{A}^T\\mathbf{A})^{-1}\\mathbf{A}^T\\mathbf{d} \end{aligned}$$

where $\boldsymbol\omega^*, \mathbf{v}^*$ is the optimal angular velocity and velocity.

In practice, we uniformly sample around $2000$ points from a reconstructed mesh and choose around $200$ neighbouring points to compute the scene flow of one surface point. To further study the influence of the number of neighbouring point, we compare the scene flow error with respect to the number of neighbouring points in the table below.

• SDF flow v.s. SDF

Thank you for the suggestion. We compare the reconstruction results of our model directly output the SDF and SDF flow. During training we set aside several frames to evaluate the interpolation ability of both representations. For example, we train the model on $\\{1,4,7,\\cdots\\}$-th frames and the evaluation results on those frames are refer to as reconstruction and those on $\\{2,3,5,6,\\cdots\\}$-th frames are refer to as interpolation. Our model with SDF flow representation consistently outperforms that with SDF representation especially at the unseen frames (interpolation).

• W-1 Low fidelity results

The reason for low fidelity results is because of the challenging data. We try to reconstruct the geometry of a dynamic scene with only 10 sparse views. In Section A5, Figure 17, we show all the 10 views at particular time step. The view changes are very large making it extreme hard to reconstruct the geometry not only for NeRF-based methods but also for other multi-view stereo ones. In fact, even COLMAP (Sch ̈onberger & Frahm, 2016) fails on those images due to “no good initial image pair found”.

Scene flow evaluation. Since we cannot obtain the ground truth scene flow for real-world sequences of the CMU Panoptic dataset, we try to evaluate the scene flow with optical flow by projecting it onto the image plane. In the table below, we provide such evaluation results on all 5 sequences of the CMU Panoptic dataset. The optical flow computed from our scene flow is consistently better than that of NDR.

In addition, we follow the advice of Reviewer jfGg to directly evaluate the scene flow on the synthetic sequence shown in Figure 7 of the main script. The results are reported in the table below. We compare the results of three models: NDR (Cai et al., 2022), Ours, Ours with scene flow regularization which we will introduce in the response to W-3. We also evaluate the long-term correspondences up to 5 frames. In particular, for every surface point in a frame, we compute the scene flow to the future first, second, and 5-th frames ("+1", "+2", and "+5"). We report the average end-point-error (EPE) in millimeter. Our model without any explicitly scene flow regularization consistently outperforms NDR (Cai et al., 2022). With further regularization, the scene flow can be drastically improved.

• W-2. Validity on the assumption 2.

Let us denote a surface point as $\\mathbf{x}$, its corresponding point on the evolved surface as $\\mathbf{x}'$, the normal of the tangent plane at $\\mathbf{x}'$ as $\\mathbf{n}(\\mathbf{x}')$, and the closest point on the evolved surface to $\\mathbf{x}$ as $\\mathbf{x}''$. According to the definition of SDF, the absolute SDF change of $\\mathbf{x}$ i.e., $|\\Delta s|$ is equal to $||\\mathbf{x}-\\mathbf{x}''||_2$. With the assumption that the infinitely small surface region around $\\mathbf{x}$ as well as its deformation is smooth, as the time period $\\Delta t$ approaches zero, $\\mathbf{x}''$ becomes infinitely close to $\\mathbf{x}'$ which can be regarded as lying on the tangent plane to the evolved surface at $\\mathbf{x}'$. So that the absolute SDF change $|\\Delta s|$ of $\\mathbf{x}$ will equal to the distance from $\\mathbf{x}$ to such tangent plane i.e., $|(\\mathbf{x}'-\\mathbf{x})^T\\mathbf{n}(\\mathbf{x}')|$. The sign of $\\Delta s$ is determined by angle between the scene flow and the normal of the tangent plane.

• W-3. Necessity of Sec. 3.3.

We acknowledge that currently the training of our model does not involve the computation of scene flow. However, bridging the gap between SDF flow and scene flow could potentially be applied to train a better model. We evidence this by the toy example shown in Figure 7 of the main script. In particular, we train a new model (named ``Ours w/ reg'') that outputs not only the SDF flow but also the scene flow, we then apply several priors:

1. SDF flow and scene flow consistency prior from Equation 16.

2. Piece-wise rigid prior: we encourage the scene flow of close points to be similar.

3. Optical flow loss: we project the scene flow onto image plane to obtain the optical flow and compute optical flow loss with it.

More details about this experiment are added to Section A6 of the main script. The evaluation results are show in the table below. The scene flow regularization helps to improve the reconstruction quality. As also evidenced in the response to W-1, the estimated scene flow from SDF flow is also improved. We believe revealing the mathematical relation between the scene motion and scene geometry has more potential applications which we will explore in the future.

References

Johannes Lutz Sch ̈onberger and Jan-Michael Frahm. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

We thank the reviewer for all the valuable comments that help to improve the paper.

• The reason why SDF flow is better

We think it is because that SDF flow brings temporal regularization to the SDF. Given multi-view videos, one can directly predict the per-frame SDF. However, since each frame are predicted almost independently (using the same function with time as an extra input), the temporal information of the scene is not fully utilized. By contrast, SDF flow models the change of the scene geometry which captures the temporal dependencies across different frames via the integral defined in Equation 7. For example, it is hard for the pre-frame SDF to recover an occluded object while it is possible for SDF flow to reconstuct it if it is observed in future or/and past frames. Similarly, for fast moving surfaces that often lead to motion blur on the images, it is hard for SDF to recover the correct geometry given only such blurry images. However, the temporal information introduced by SDF flow could be very helpful in this case.