Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynamic Gaussian Splatting

Thank you very much for the constructive feedback. We hope that our response below will address your concerns.

Q1: Citation of the two related works and writing problem.

A1: We will add the citation of the mentioned related works and change the presentation in the final version.

Q2: Should the metrics of the wrong scene 'Lego' be removed ?

A2: We will remove the metrics of the 'Lego' scene in the final version.

Q3: Use the NeRF-DS dataset with more accurate camera poses than the HyperNeRF dataset ?

A3: According to the original paper, we think that NeRF-DS dataset is designed for dynamic scenes with specular objects which is not suitable for general dynamic scene rendering. Therefore, to further demonstrate the improvements of Grid4D, we also choose the Neu3D dataset. Neu3D is captured by 21 cameras with fixed poses and has more accurate camera poses than HyperNeRF. Additionally, the scenes in the Neu3D dataset are larger and more challenging.

We report the comparison results with PSNR on the Neu3D[1] dataset in the following table, and our model has better performance than the state-of-the-art models. The qualitative results can be found in Figure 1 of the top comment PDF. We will add these experiments to our paper in the final version.

Q4: How does Grid4D perform when handling difficult motions.

A4: The 'Hand', 'Peel Banana', 'Chocolate', 'Broom', and 'Teapot' scenes in HyperNeRF dataset have large and complex motions. As listed in Table 5, Figure 9, and Figure 10 of our paper, our methods have a much better rendering quality than the state-of-the-art models due to the discriminability improvement of explicit features. However, Grid4D still has several artifacts in the 'Broom' and 'Teapot' scenes, which might need further research.

Q5: Comparison to SC-GS.

A1: We conduct the comparison with SC-GS on the D-NeRF dataset. The comparison results with PSNR can be found in the following table, and the qualitative results can be found in Figure 2 of the top comment PDF. Notably, the original SC-GS rendering resolution is $400\times 400$, lower than our model, so we changed the rendering resolution of SC-GS to $800\times 800$ for fairness. Additionally, we remove the 'Lego' scene because of the incorrect test ground truth. We will add these experiments in the final version.

According to the results, Grid4D has better performance than SC-GS on average. The improvement of Grid4D might be because of the performance gap between implicit deformation fields (which SC-GS is based on) and Grid4D. SC-GS uses a full MLP-based implicit model to predict deformations, which has the over-smooth inherent property. Our proposed 4D decomposed hash encoder generates explicit features with high discriminability to represent the deformations more accurately.

We consider that SC-GS is an excellent work mainly focused on dynamic scene editing and rendering. Therefore, combining the advantages of SC-GS and Grid4D or applying Grid4D to SC-GS are probably great future works that might get better rendering quality.

Q6: What hyperparameters are set in different scenes ?

A6: In our experiments, the hyperparameters that need to change mainly include the resolution of the time dimension, the weight of smooth regularization term $\lambda_r$, and the perturbation range $\epsilon=(\epsilon_x, \epsilon_y, \epsilon_z, \epsilon_t)$.

The resolutions of the time dimension are set according to the frame number, usually about a half and a quarter of the frame number according to the Nyquist-Shannon Sampling Theorem. The resolution can be set automatically by the algorithm.

The weight of smooth regularization term $\lambda_r=0.5$ for most scenes, $\lambda_r=1.0$ for the 'vrig' part of HyperNeRF dataset. In most cases, setting $\lambda_r=0.5$ could obtain reliable results. For better performance, increasing $\lambda_r$ can help Grid4D render better when the scene primarily consists of rigid objects and simple motions. Similarly, decreasing $\lambda_r$ can help Grid4D for modeling complex motions and non-rigid objects.

For D-NeRF dataset and the 'vrig' part of HyperNeRF dataset, we set the perturbation range into $\epsilon\in [-10^{-2}, 10^{-2}]$, For the 'interp' part of HyperNeRF dataset, we set the perturbation range into $\epsilon\in [-10^{-3}, 10^{-3}]$. In most cases, setting the range to $\epsilon\in [-10^{-3}, 10^{-3}]$ can obtain reliable results.

[1] Li, Tianye, et al. "Neural 3d video synthesis from multi-view video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Dear authors and reviewer T422, I would like to join the discussion and kindly defend the deformation-based GS:

Recent works, such as "Shape of Motion" (Wang et al., ArXiv24) and "MoSCA" (Lei et al., ArXiv24), seem to excel in monocular setups through explicit deformation models. I believe these methods could handle HyperNerf's setups. Therefore, I don't think 4DGS (Yang et al.) and Spacetime GS are similar methods. However, as the authors mentioned, the implicit global deformation network also has its advantages. Generalizability could be another considerable potential advantage.

Exploring implicit representations is an important research topic in 3D vision. While NeRF has its own advantages compared to 3D-GS, even though most researchers currently prefer the latter, we must remember that CNNs eventually replaced handcraft-designed convolutional kernels.

Thank you very much for the constructive feedback. We hope that our response below will address your concerns.

Q1: Comparison to SC-GS.

A1: We conduct the comparison with SC-GS on the D-NeRF dataset. The comparison results with PSNR can be found in the following table, and the qualitative results can be found in Figure 2 of the top comment PDF. Notably, the original SC-GS rendering resolution is $400\times 400$, lower than our model, so we changed the rendering resolution of SC-GS to $800\times 800$ for fairness. Additionally, we remove the 'Lego' scene because of the incorrect test ground truth. We will add these experiments in the final version.

According to the results, Grid4D has better performance than SC-GS on average. The improvement of Grid4D might be because of the performance gap between implicit deformation fields (which SC-GS is based on) and Grid4D. SC-GS uses a full MLP-based implicit model to predict deformations, which has the over-smooth inherent property. Our proposed 4D decomposed hash encoder generates explicit features with high discriminability to represent the deformations more accurately.

We think that SC-GS is an excellent work mainly focused on dynamic scene editing and rendering. Therefore, combining the advantages of SC-GS and Grid4D or applying Grid4D to SC-GS are probably great future works that might get better rendering quality.

Q2: Results on Neu3D dataset.

A2: We conduct the experiments on the Neu3D dataset and report PSNR in the following table. From this table, one can see that our model has better performance than the state-of-the-art model. The qualitative results can be found in Figure 1 of the top comment PDF. We will add these experiments to our paper in the final version.

Q3: Can the authors provide comparison videos ?

A3: To demonstrate the effectiveness of our model, we provide additional rendering results in Figure 12, Figure 11 and Figure 9 of our supplementary. To demonstrate the temporal coherence of our model, we provide some videos by displaying included images in Figure 3 of the top comment PDF. We uniformly interpolate the time and randomly select a camera pose to render the scene reconstructed by Grid4D, and select the images per 5 frames (full 150 frames) to display. We will provide more videos on our GitHub repository in the final version.

Q4: Experimental results on other datasets such as Neu3D since D-NeRF is a toy dataset ?

A4: To further evaluate our model, we also conduct experimental comparisons on the real-world HyperNeRF and Neu3D datasets. The results demonstrate the effectiveness of our model on the real-world datasets.

Q5: Use average results of the time and space consumption instead of a range.

A5: We provide the detailed average metrics in the following table. We will add the table to our paper in the final version.

Thank you very much for the constructive feedback. We hope that our response below will address your concerns.

Q1: How to demonstrate the temporal coherence of the rendering results ?

A1: To demonstrate the temporal coherence of our model, we provide some videos by displaying included images in Figure 3 of the top comment PDF. We uniformly interpolate the time and randomly select a camera pose to render the scene reconstructed by Grid4D, and select the images per 5 frames (full 150 frames) to display. We will provide more videos on our GitHub repository in the final version.

In the test stage of the dynamic scene rendering task, the camera poses and timestamps are usually not selected in a video sequence. To our knowledge, there has no existing metric to evaluate the temporal coherence of a video without the ground truth. To further demonstrate the temporal coherence of the rendering results, we attempt to design a simple metric to evaluate the coherence. We use the optical flow estimation model RAFT[1] to predict the forward and backward flow between the two neighbouring frames of the rendered video. Similar to Chamfer Distance, we calculate the mean of total pixel displacements, as the following formula,

$F^{forward} = \frac{1}{NHW}\sum_{i, h, w}||f_{ihw}^{forward}||_2$

$F^{backward} = \frac{1}{NHW}\sum_{i, h, w}||f_{ihw}^{backward}||_2$

$F = \frac{1}{2}(F^{forward}+F^{backward})$

where $N, H, W$ denote to the frames, height, width of the video, $f$ means the corresponding flow. The unit of $F$ is pixel. The lower value of $F$ denotes the better temporal coherence keep in the video.

We use the official pretrained RAFT[1] model of PyTorch to estimate the flow. We uniformly interpolate 150 timestamps along the timeline and randomly select a camera pose for our model to render the video. The results on the D-NeRF dataset are listed in the following table. For reference, we random select a video from the Kinetics-400[2] dataset and calculate the same metric $F$.

According to the results, the average pixel displacements of the rendered videos are all below 2.0pix and could demonstrate the temporal coherence of our rendering results. However, the metrics mentioned above are not well explored which might need further research.

Q2: What are the values of $\lambda_c$ and $\lambda_r$ ?

A2: In our experiments, $\lambda_c$ and $\lambda_r$ are set as follows: $\lambda_c=0.2$ for all scenes. $\lambda_r=0.5$ for most scenes, $\lambda_r=1.0$ for the 'vrig' part of HyperNeRF dataset.

In most cases, set $\lambda_r=0.5$ can obtain reliable results. For better performance, increasing $\lambda_r$ can help Grid4D render better when the scene primarily consists of rigid objects and simple motions. Similarly, decreasing $\lambda_r$ can help Grid4D for modeling complex motions and non-rigid objects.

Q3: Problems of writing.

A3: We will add the introduction of Multi-resolution Hash Encoding and the description of Figure 1 in the final version.

[1] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020:16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.

[2] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950,2017.

Thank you for considering the review and the efforts for answering it. I have no further questions. I will reflect a little bit about the presented data about temporal smoothness and coherence and will update my rating to reflect the conclusions.

Thank you very much for the constructive feedback. We hope that our response below will address your concerns.

Q1: Grid4D does not significantly improve the training speed in comparison to existing models.

A1: Although our model has no improvement in training speed, our model can obtain a good balance between time consumption and performance. Additionally, the GPU memory consumed with our model is comparable to DeformGS and SC-GS. We report the average time, GPU memory and PSNR on the D-NeRF dataset in the following table.

Q2: Will the complexity and specific tuning of Grid4D limit its generalizability ?

A2: Although our model has multiple hash encodings, it has fewer reconstruction failures than DeformGS on the real-world HyperNeRF dataset, as shown in 'Teapot' (last row of Figure 9 in our paper) and 'Hand' scenes (middle row of Figure 11 in our paper).

To demonstrate the versatility of our model, we also apply Grid4D to the Neu3D[1] dataset which has larger scenes and more complex motions. We report PSNR in the following table. The qualitative results can be found in Figure 1 of the top comment PDF. From this table, one can see that although our model has multiple hash encodings, our model can still yield better performance. We will add these experiments to our paper in the final version.

The main tuning of our model is the resolution of the temporal hash encoder because different scenes have different numbers of frames. Additionally, according to the Nyquist-Shannon Sampling Theorem, the temporal resolution can be set to half or a quarter of the frame number automatically by the algorithm.

Q3: How does Grid4D handle scenes with highly non-uniform motions ?

A3: Firstly, explicit representation methods are much more flexible than implicit ones, making Grid4D better suited for modeling highly non-uniform motions. Secondly, our hash encoder decomposes 4D coordinates into four 3D coordinates. When encoding heavily overlapping 4D inputs, the 3D decomposition results will have less same parts. Therefore, the encoded explicit features will have fewer overlapped parts, resulting in a more accurate representation of the corresponding non-uniform motions.

In our experiments, the 'Hand', 'Peel Banana', and 'Chocolate' scenes in HyperNeRF dataset have much more highly non-uniform motions. As listed in Table 5, Figure 9, and Figure 10 of our paper, our methods can significantly achieve better rendering quality than the state-of-the-art models.

Q4: Can the directional attention mechanism be adapted for other usage ?

A4: We consider that directional attention could also be used in 4D generation models for temporal feature selection. For example, we could generate the directional attention score from the features unrelated to the timestamp, and apply the score to the other features.

Q5: What are the specific challenges faced when applying Grid4D to real-world datasets in comparison to synthetic ones?

A5: The main challenge is that the real-world scenes usually have imprecise camera poses which might lead to the failure or degradation of rendering. Another challenge is that real-world scenes usually have more complex motions.

For the imprecise camera poses, the smooth regularization term helps model keep consistency in the neighboring region. Such smoothness makes the model tend to fit the average results captured by the imprecise camera poses, which improves the robustness of our model. We can increase the weights of the smooth regularization term to handle more imprecise camera poses in the real-world datasets.

To handle complex motions, our model improves the discriminability of the explicit features. Encoded by our 4D decomposed hash encoder, our explicit features have fewer overlapped parts, resulting in a more accurate representation of the corresponding non-uniform motions. We can increase the resolution of 4D decomposed hash encoder to model more complex motions in the real-world datasets.

Q6: How does the smooth regularization impact the real-time rendering capabilities of the model ?

A6: The smooth regularization does not affect real-time rendering capabilities. During the rendering period, the smooth regularization term and other losses are not calculated by the model. Smooth regularization is mainly used in the training process and improve the rendering quality.

[1] Li, Tianye, et al. "Neural 3d video synthesis from multi-view video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.