DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization

Thank you for your comments. Please find our responses to each of them below. If you have any further concerns or questions, please feel free to inform us.

Q1. More insights from ablation study.

We appreciate your suggestion and have incorporated more discussion regarding the ablation study results in Section 4.5 of the revised paper. Specifically, we delve into the impact of different stages during the training process:

• Warmup Stages: We conducted an additional ablation study performing 4D voxel optimization from scratch with randomly initialized $\mathcal{V}{t=1}$ and $f_\phi(\cdot)$. This study clearly illustrates that excluding the warmup stages (which includes both the "3D voxel warmup stage" and the "3D-to-4D voxel expansion stage") has a substantial impact on the final performance, particularly for the scene decomposition results.
• 4D Voxel Optimization Stage: We emphasize that the performance of DynaVol significantly degrades when excluding the 4D voxel optimization stage. These results demonstrate the importance of refining the object-centric voxel representation with a slot-based renderer.

Q2. The method proposed in the manuscript is pretty interesting, although it would be great to highlight the novelty and difference between the proposed method and previous literature.

We have included more discussion of the differences and relationships between our model and the suggested existing work [1,2] in Section 5 in the revision. We here provide a more detailed comparison of these methods on the dynamic modeling side:

• DynaVol: In addition to the rendering MLP, our model employs additional networks that learn $(\mathbf{x},t) \rightarrow \Delta \mathbf{x}\in \mathbb{R}^{3}$ to explicitly transport the density values in each voxel grid to the corresponding voxel grid at an arbitrary future time step.
• Method from [1]: This approach extends the original rendering MLP in NeRF to incorporate the dynamics information. Basically, it learns a mapping function of (RayPoint, ViewDirection, Time) $\rightarrow$ (Color, Density, Offset). Here, "offset" determines the 3D correspondence of the RayPoint at nearby time steps.
• Method from [2]: This approach has achieved significant improvements on dynamic scene benchmarks by representing motion trajectories by finding 3D correspondences for sampling points in nearby views.

Furthermore, while all of these methods leverage temporal-consistency constraints, unlike [1,2], which establish the constraint in pixel color space, we incorporate it in the canonical space to encourage coherent transportations of the density values across time.

Q7. Could not find the per-point color loss in the DVGO paper.

Yes, indeed, DVGO did not introduce the formulation of the per-point RGB loss in the main text, but examined its effectiveness in Table I6 in the supplementary material (please refer to its arxiv version).To improve clarity, we have presented the formulation of this loss term in Eq. (5) in our original manuscript.

Q8. What happens without the warmup phase?

To answer this question, we conducted an ablation study by training the entire model from scratch --- We directly performed "4D voxel optimization" with randomly initialized $\mathcal{V}\_{t=1}$ and $f\_\phi(\cdot)$. The obtained results in terms of PSNR and FG-ARI (higher values are favorable) are presented below. It is evident that removing the warmup stage significantly impacts the final performance, especially for the scene decomposition results. We also include these comparisons in Table 4 ("4D voxel optim. from scratch") in the revised paper.

A side note: We have reorganized the entire training scheme for better clarity. Hence, the "w/o Warmup" results in the above table are obtained by excluding the first two stages below:

• 3D voxel warmup stage: Corresponds to the training phase in the original warmup stage.
• 3D-to-4D voxel expansion stage: Corresponds to the post-processing part of the original warmup stage (i.e., the computation of connected components).
• 4D voxel optimization stage: Corresponds to the dynamic scene optimization stage in the original manuscript.

Q9. How are the occupancy grid and the deformation function represented? Just dense values on a grid? what resolution?

The resolution of the occupancy grids is $\mathcal{V}\_{density}^{t=1} \in \mathbb{R}^{N \times N\_x \times N\_y \times N\_z}$, where $N$ is the number of slots, and $N\_x$, $N\_y$, and $N\_z$ represent the spatial resolution along each axis. Specifically, we set the spatial resolution of the voxel grids to $110^3$.

As for the deformation function, we employ an MLP that takes $(\mathbf{x},t)$ with positional encoding as inputs to predict the position movement $\Delta \mathbf{x} \in \mathbb{R}^{3}$.

Q10. How is the feature graph computed after warmup?

We have included Algorithm 1 in Appendix B to provide a detailed explanation of the 3D-to-4D voxel expansion process, which consists of the following two steps:

• Feature Graph Generation: We consider the voxel set $\\{X_k\\}\_{k=1}^{N_x \times N_y \times N_z}$ from $\mathcal{F}\_{t=1}$. By filtering out invalid locations with density values below a pre-defined threshold (inspired by DVGO), we can get the node set $\\{X_k\\}\_{k=1}^{K}$ to build the feature graph $G$. For any pair of nodes $(u,v)$ in $G$, an edge is defined between them if the following three conditions are satisfied:
  • $u$ and $v$ correspond to spatially adjacent voxels.
  • $u$ and $v$ result in similar emitted color $(c\_u^r, c\_v^r)$ produced by $N_{\phi^\prime}$ along an arbitrary ray $r$, such that $\frac{1}{\mathcal{R}}\sum\_r^\mathcal{R} d_\text{rgb}(c\_u^r, c\_v^r)<\text{Threshold}\_\text{rgb}$, where $\mathcal{R}$ is the number of rays randomly sampled with different directions. We take the Euclidean distance to calculate $d\_\text{rgb}(c\_u^r, c\_v^r)$.
  • $u$ and $v$ lead to similar velocities across all time steps, which can be acquired through the forward deformation network $f^\prime\_\xi$, such that $\text{max}\_t(d_\text{vel}(u,v)) < \text{Threshold}\_\text{vel}$, where we take the maximum Euclidean distance $||(f^\prime\_\xi(u,t)-f^\prime\_\xi(u,t-1))-(f^\prime\_\xi(v,t)-f^\prime\_\xi(v,t-1))||_2$ over all time steps ($t \in [2,T]$).

• Connected Components Computation: In this step, we first feed the feature graph $G$ into the connected components algorithm to obtain $M$ clusters. We then sort the clusters by size and merge the smallest $M-N+1$ clusters. For each cluster $p$ ($1 \le p \le N$), we assign the density value of voxels in $\mathcal{F}\_{t=1}$ to the corresponding channel at the same localation in $\mathcal{V}\_{t=1}$.

We appreciate your great efforts in reviewing our paper and hope that the following responses can address most of your concerns.

Q1. The proposed model DynaVol is complex. The text and the architectural figures were not easy to follow.

We apologize for any difficulty you experienced in reading our paper. We have tried our best to organize the writing but found it challenging due to the complexity of the proposed method, which involves multiple training stages and many details. To improve the overall reading experience, we have made the following changes:

• We refined Section 3.1 and Figure 2 for a better overview of DynaVol.
• We revised Section 3.4 to reorganize the entire training pipeline and separate it into three stages, including a "3D voxel warmup stage", a "3D-to-4D voxel expansion stage", and a "4D voxel optimization stage". We further clarified the key insight of each stage and the connections between these stages.
• We introduced Algorithm 1 in Appendix.B to present the details of 3D-to-4D voxel expansion, outlining how to initialize the 4D voxel representation using the connected component algorithm.

Please refer to the updated paper for the details.

Q2. When comparing such an overfit model to a fully generalizing model like SAM caution has to be used to clarify very precisely.

Thanks for the suggestion! We have clarified the differences in the experimental setups of DynaVol and SAM in the caption of Table 3: "The compared models, including SAM, are fully generalizing models trained beyond the test scenes. In contrast, our model enables 3D scene decomposition by overfitting each individual single scene in an unsupervised manner." Despite the distinct training setups, the comparison in Table 3 aims to showcase DynaVol's ability for 3D scene decomposition by leveraging knowledge of the segmented scene.

Note: Both the quantitative results in Table 3 and the qualitative results in Figure 4 correspond to images of novel views outside the scope of DynaVol's training set.

Q3. DynaVol works well on easily segmented objects (from the simulation) but on real sequences, DynaVol does not much outperform other methods.

We have added new real-world experiments on the ENeRF-Outdoor dataset. Please refer to Figure 7 in Appendix.C for the qualitative results. Below, we present the quantitative comparisons in PSNR and SSIM:

Q4. Do I understand correctly that there is only one occupancy grid at t=0 and the other ts are reconstructed using the deformation field? A query at time t is warped back to t=0 and then interpolated into the feature volume?

Yes.

Q5. Where is the GRU in Fig 2?

We consider GRU as a component of slot attention, thereby placing it in $Z\_\omega$. We clarified this in the caption and the legend in Figure 2.

Q6. For the NERF rendering is it correct that the occupancy comes from the occupancy grid and the color comes from the slot features? Why this choice and not push occupancy into the slot features too?

Yes, as illustrated in the updated Figure 2, we determine the color $\bar{\mathbf{c}}$ at a ray sampling point through the following steps, all starting from the occupancy grids $\mathcal{V}\_t$:

• $\mathcal{V}\_t$ $\rightarrow$ [Volume Slot Attention] $\rightarrow$ Slot features $s_t^n$ $\rightarrow$ [MLP] $\rightarrow$ Per-slot color $c\_n$,
• $\mathcal{V}\_t$ $\rightarrow$ [Fetch and Trilinear Interpolation] $\rightarrow$ Per-slot occupancy $\sigma\_n$,
• $(c\_n, \sigma\_n)$ $\rightarrow$ [Accumulated as Eq. (3)] $\rightarrow \bar{\mathbf{c}}$.

Representing per-object volume density through explicit occupancy grids offers three advantages:

• It speeds up the training process.
• It improves the convenience of the downstream scene editing applications.
• It allows seamless integration with off-the-shelf scene decomposition algorithms such as the connected components.

Thank you for the clarifications. A lot of things are more clear now. I appreciate the additional ablations of no warmup. The real world experiment in the appendix is also nice and clean.

Re Q1: 3D voxel warmup is also training the deformation field though it seems? That means a optimization in 4D is executed. Based on my understanding of the algorithm maybe something more like:

• 4D voxel warmup
• Slot Instantiation
• 4D Slot optimization

Does that make sense?

Q7. It's unclear how exactly the connected components and the "feature graph" connect with the rest of the method.

To highlight the relationship of the computation of connected components with the rest of the method, we have reorganized the entire training scheme into three stages. As clarified in our response to Q6, the corresponding "3D-to-4D voxel expansion stage" acts as a bridge between the previous "3D voxel warmup stage" and the subsequent "4D voxel optimization stage". It extends the 3D density values to 4D object-centric representation:

The 3D-to-4D voxel expansion stage includes the following two steps:

• Feature Graph Generation: We consider the voxel set $\\{X_k\\}\_{k=1}^{N_x \times N_y \times N_z}$ from $\mathcal{F}\_{t=1}$. By filtering out invalid locations with density values below a pre-defined threshold (inspired by DVGO), we can get the node set $\\{X_k\\}\_{k=1}^{K}$ to build the feature graph $G$. For any pair of nodes $(u,v)$ in $G$, an edge is defined between them if the following three conditions are satisfied:
  • $u$ and $v$ correspond to spatially adjacent voxels.
  • $u$ and $v$ result in similar emitted color $(c_u^r, c_v^r)$ produced by $N_{\phi^\prime}$ along an arbitrary ray $r$, such that $\frac{1}{\mathcal{R}}\sum\_r^\mathcal{R} d\_\text{rgb}(c_u^r, c_v^r)<\text{Threshold}\_\text{rgb}$, where $\mathcal{R}$ is the number of rays randomly sampled with different directions. We take the Euclidean distance to calculate $d\_\text{rgb}(c_u^r, c_v^r)$.
  • $u$ and $v$ lead to similar velocities across all time steps, which can be acquired through the forward deformation network $f^\prime\_\xi$, such that $\text{max}\_t(d_\text{vel}(u,v)) < \text{Threshold}\_\text{vel}$, where we take the maximum Euclidean distance $||(f^\prime\_\xi(u,t)-f^\prime\_\xi(u,t-1))-(f^\prime\_\xi(v,t)-f^\prime\_\xi(v,t-1))||_2$ over all time steps ($t \in [2,T]$).

• Connected Components Computation: In this step, we first feed the feature graph $G$ into the connected components algorithm to obtain $M$ clusters. We then sort the clusters by size and merge the smallest $M-N+1$ clusters. For each cluster $p$ ($1 \le p \le N$), we assign the density value of voxels in $\mathcal{F}_{t=1}$ to the corresponding channel at the same localation in $\mathcal{V}\_{t=1}$.

We have included Algorithm 1 in Appendix B to provide a detailed explanation of the 3D-to-4D voxel expansion process.

Q8. $\mathcal{L}_\text{point}$ seems like it is never defined.

The formulation of $\mathcal{L}_\text{point}$ is provided in Eq. (5). Intuitively, this regularization term encourages similar color radiance values for different sampling points along the same viewing direction.

Additionally, we acknowledge that it would to more reasonable to allow various sampling points on the same ray to have slightly different colors. Therefore, we tune the loss weight $\alpha_p$ and discuss of the effect of $\mathcal{L}_\text{point}$ in the appendix. As demonstrated in Table 7:

• $\alpha_p=0.1$ (final DynaVol) outperforms $\alpha_p=0$. This suggests that $\mathcal{L}_\text{point}$ helps the training process by moderately penalizing the discrepancy among nearby sampling points on the same ray.
• $\alpha_p=0$ (w/o $\mathcal{L}\_\text{point}$) outperforms $\alpha\_p=1$. This indicates that a strong $\mathcal{L}\_\text{point}$ may introduce bias to neural rendering and affect the final results.

Q9. Section 3.3 seems to complain that prior compositional NeRFs used an MLP to learn a mapping from position+direction+feature to color+density, and then proposes to do the exact same thing.

Most prior compositional NeRFs learn a mapping function of (Position, Direction, Feature) $\rightarrow$ (Color, Density), where the object-centric density values are generated by the model.

In DynaVol, we slightly modify this approach by using the MLP to learn a mapping function like (Position, Direction, Feature) $\rightarrow$ Color, while the density values are directly retrieved from the 4D voxel grids of $\mathcal{V}^t_{\text{density}}$. The value in each grid cell indicates the occupancy probabilities of each object. This method allows for easy manipulation of objects for scene editing.

Q6. It is unclear what is happening in the warmup stage vs. the dynamics stage. The section also says "To obtain the dynamics information, we train an additional module $f^\prime_\xi$ -- but dynamics information is already obtained by $f_\psi$, so there is some mixup here, unless the authors do not see $f_\psi$ as obtaining dynamics information.

(1) Reorganizing the training stages:

Yes indeed, the warmup stage also exploits the dynamics information to initialize the voxel grid features $\mathcal{V}_{t=1}$. From this perspective, we acknowledge that the names of the training stages might be somewhat misleading. We have reorganized the entire training scheme and separated it into three stages for better clarity:

• 3D voxel warmup stage (the training part of the original warmup stage): This training stage does NOT incorporate object-centric features. Instead, we train the bi-directional deformation networks $(f^\prime_\xi, f_\psi)$ and the neural renderer $N_{\phi^\prime}$ based on 3D voxel densities $\mathcal{F}_t \in \mathbb R^{N_x \times N_y \times N_z}$.
• 3D-to-4D voxel expansion stage (the post-processing part of the original warmup stage): We extend the 3D voxel grids $\mathcal{F}\_{t=1}$ to 4D voxel grids $\mathcal{V}\_{t=1}$ using the connected components algorithm. The input features for the connected components algorithm involve the forward canonical-space transitions generated by $f^\prime_\xi$ and the emitted colors by $N_{\phi^\prime}$. We give more details in Appendix.B in the revision.
• 4D voxel optimization stage (the original dynamic scene optimization stage): In this stage, we train the entire model based on a set of 4D voxel features $\mathcal{V}_t \in \mathbb R^{N \times N_x \times N_y \times N_z}$. It's important to note that the additional dimension from 3D to 4D corresponds to the number of slots ($N$) rather than the time horizon ($T$).

We have made corresponding revisions in Section 3.1 (Overview of DynaVol), Section 3.4 (Descriptions of training stages), and Figure 2. Please refer to our revised paper!

(2) The effect of $f^\prime_\xi$:

As pointed out by the reviewer, both $f^\prime_\xi$ and $f_\psi$ are designed to capture the dynamics information, but in different time directions. We use $f^\prime_\xi$ for two purposes:

• It allows for the computation of a cycle-consistency loss, which can enhance the coherence of the learned canonical-space transitions.
• It provides useful input features to the subsequent connect components algorithm, representing the forward dynamics starting from the initial time step, which can improve the initialization of $\mathcal{V}_{t=1}$.

Thank you for your comments. We hope that our responses below can adequately address your concerns regarding this paper.

Q1. The method would be easier to understand if the stages were presented in a more central way, instead of at the very end.

Thank you for the suggestion. We have included more discussions on the warmup stage and dynamic scene optimization stage in Section 3.1. Please refer to the revised paper for further details.

Q2. On the meaning of an "episode".

Throughout the training stage, DynaVol iteratively processes each frame in the video in chronological order and repeats: $\\{I\_1, I\_2, \ldots, I\_T\\}\_\text{episode=1}$, $\\{I\_1, I\_2, \ldots, I\_T\\}\_\text{episode=2}$, $\ldots$, where we use the term "episode" to denote one round spanning from the first frame to the last.

Q3. Section 2 emphasizes that "we also consider scenarios where only one camera pose is available at the first timestamp". What does it mean for only one camera pose to be available?

The term "one camera pose" means we only have a single-view image captured by the monocular camera at $t=1$, as opposed to having multiple images from different views. We apologize for any confusion and have rephrased this statement in the revised paper.

Q4. Section 3.1 says the method will be trained "without any further supervision", without defining any initial supervision.

As shown in Eq. (5), we use the pixel values of the observed video frames as the training targets for DynaVol. We here emphasize "without any further supervision" because some existing object-centric rendering approaches, such as those proposed by Stelzner et al. (2021) and Driess et al. (2022), incorporate additional information like depth maps or binary masks for each object to supervise the model.

Thank you for your valuable comments. We provide the responses below to each of the specific concerns. Please do not hesitate to let us know of any additional comments on the paper.

Q1. My primary concern is the robustness of DynaVol when applied to intricate real-world scenes.

We have added new real-world experiments on the ENeRF-Outdoor dataset. Please refer to Figure 7 in Appendix.C for the qualitative results. Below, we present the quantitative comparisons in PSNR and SSIM:

Q2. (1) How important the background entropy loss is? (2) Will the slot decompose the background?

(1) On the background entropy loss:

The background entropy loss is borrowed from DVGO, which aims to encourage the renderer to concentrate on either the foreground or background information. We further analyzed the impact of the background entropy loss using different loss weights (metrics: PSNR/FG-ARI):

Findings: Incorporating the background entropy loss with an appropriate weight is beneficial to the scene decomposition results as it penalizes ambiguous geometric estimations. However, a higher $\alpha_e$ value can introduce instability to the training procedure.

(2) Will the slot decompose the background?

Yes, our proposed method is motion-aware and is capable of decomposing the background, as the background can be treated as a static component. Additionally, during the 3D-to-4D voxel expansion stage, voxel grids with significantly distinct velocities estimated by the dynamics modules tend to be initiated with different slot indices by the connected components algorithm (Please refer to Algorithm 1 in the appendix).

As illustrated in Figure 3(b), our model effectively decomposes backgrounds in real-world scenarios. In the synthetic scenes in Figure 4, our model successfully decomposes the background table with an individual slot. For additional visualization results of per-slot rendering, please refer to the newly added Figure 8 in Appendix.F.

Q3. Models utilizing 5 views don't consistently surpass the performance of those with just a single view.

Our motivation for using multiple views at $t=1$ stems from empirical observations in particular challenging scenarios. For instance, in 3ObjMetal, the color of a moving object with a metal surface may change rapidly across different time steps due to the drastic changes in specular reflection, which presents difficulties in learning the geometries. Incorporating additional views at the initial time step can enrich the geometric information, thereby improving the quality of the learned 3D voxel grids $\mathcal{F}\_{t=1}$ and corresponding 4D object-occupancy grids $\mathcal{V}\_{t=1}$.

I thank the authors for the time they have contributed to addressing my concerns. I have increased my rating accordingly.