DN-4DGS: Denoised Deformable Network with Temporal-Spatial Aggregation for Dynamic Scene Rendering

1. Thank you for your suggestions. We will cite the relevant works as recommended.

2. We appreciate your understanding of HyperNeRF and the insights regarding both HyperNeRF and Deformable-GS. In response, we have conducted additional visual comparisons in Figure 2 of the attached PDF, including HyperNeRF and Deformable-GS. It is worth emphasizing that our method is also applicable to Deformable-GS. As shown in Figure 3 (b) of the attached PDF, while the canonical GS in Deformable-GS performs better than D-4DGS, it still contains noise. Consequently, we not only present results for D-4DGS + our method but also for Deformable-GS + our method. The visual results demonstrate that our approach provides notable improvements.

3. Thank you for your suggestions. In fact, our goal is to design our method as a plug-and-play solution. Therefore, we have conducted experiments using both D-4DGS and Deformable-GS as baselines. Specifically, Tables 1 and 2 in the main text present results with D-4DGS as the baseline, while Table 3 in the main text includes results with Deformable-GS as the baseline. To provide further insights, we have also shown results for Deformable-GS + our method on the HyperNeRF dataset in Figure 2 of the attached PDF. Additionally, we present results for SpaceTime-GS + our method on the PlenopticVideo dataset in the table below, highlighting the effectiveness of our two-stage deformation strategy (NSS) and spatial aggregation process (DSAM).

   Method	PSNR	LPIPS
   D-4DGS	31.15	0.049
   D-4DGS+ours	32.02	0.043
   spacetime-GS	32.05	0.026
   spacetime-GS+ours	32.42	0.023

   We appreciate your suggestions on exploring new directions and acknowledge the importance of camera pose accuracy for deformation-based Gaussian splatting. We also recognize the potential negative impacts of strong deformation regularization in D-4DGS. As a result, we are also actively investigating methods to optimize camera poses through network feedback and exploring new ways to better represent deformation fields.

4. Our work initially focused on improving dynamic reconstruction in real-world scenarios, which is why we did not extensively test on the synthetic D-NeRF dataset. However, we appreciate your reminder, as testing on D-NeRF is indeed valuable for demonstrating the generalization and completeness of our method. Consequently, we have conducted experiments on D-NeRF as well. As illustrated in the Figure 3 (a) of the attached PDF, we have visualized the canonical results for both D-4DGS and Deformable-GS. The results show that Deformable-GS has less noise compared to D-4DGS, but noise is still present in moving areas. The quantitative results on D-NeRF dataset, as shown in the table below, indicate that our method improves performance on both D-4DGS and Deformable-GS, with enhancements on D-4DGS surpassing those on Deformable-GS. | Method | PSNR | SSIM | | --- | --- | --- | | D-4DGS | 34.05 | 0.9787 | | D-4DGS+ours | 34.53 | 0.9811 | | Deformable-GS | 39.51 | 0.9902 | | Deformable-GS+ours | 39.87 | 0.9922 |

1. Novelty

The primary motivations of this paper are twofold:

1. Noise in Canonical + Deformable Design: In the canonical + deformable design, point-to-point relationships within the canonical Gaussians are chaotic and erroneous (noisy), which can be transferred into the deformation field. To address this issue, we propose the Noise Suppression Strategy (NSS). This strategy involves two stages of deformation to mitigate the impact of noise on the deformation network. During the first stage, we supervise only the initial deformation. Once the Gaussian coordinates from the first deformation are relatively accurate, we introduce the second deformation and shift the supervision to it. The rendering quality can be progressively improved through this approach.

2. Lack of Explicit Temporal-Spatial Feature Aggregation: Previous methods lacked explicit aggregation of temporal and spatial features. We address this by proposing the Decoupled Temporal-Spatial Aggregation Module (DTS), which includes:

   • Temporal Aggregation Module (TAM): Handles the temporal aggregation process.
   • Denoised Spatial Aggregation Module (DSAM): Manages spatial aggregation.

   Given that inaccuracies in coordinate relationships can be amplified during spatial aggregation (KNN), we decouple temporal and spatial aggregation, placing them in two distinct deformation stages. Since the Gaussian coordinates obtained after the first deformation are relatively accurate, we perform spatial aggregation in the second stage to avoid the amplification of noise due to inaccurate spatial relationships. We name this module Denoised Spatial Aggregation Module (DSAM) because the Gaussian coordinates input into DSAM are more accurate (denoised), as shown in the fourth column of Figure 8 in the main text. The term "Denoised" is used to highlight the improved accuracy of the inputs rather than implying that DSAM performs denoising.

2. Method

1. Concatenation in the Formula: In this formula, we concatenate $F_t(i)\in \mathbb{R}^{ 1\times C_1}$ with $F_{\text{max}}(i) \in \mathbb{R}^{ 1\times C_2}$ and $\mathcal{Y}_i \in \mathbb{R}^{ 1\times C_3}$, rather than $F(i)$. We concatenate along the second dimension to obtain $F_t(i)' \in \mathbb{R}^{ 1\times (C_1+C_2+C_3)}$.
2. Purpose of $\mathcal{Y}_i$: The primary purpose of $\mathcal{Y}_i$ is to add a coordinate-independent embedding to improve quality by accurately modeling different deformations of various Gaussians without adjacency interference. We have also observed a similar design in a recent work, Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting (E-D3DGS) [1], presented at ECCV 2024.
3. Clarification on DSAM: We apologize for any misunderstanding caused by our previous description. In fact, DSAM itself does not perform denoising but is focused on spatial feature aggregation. The name Denoised Spatial Aggregation Module (DSAM) was chosen because the Gaussian coordinates entering DSAM are more accurate (denoised) after the first deformation, as illustrated in the fourth column of Figure 8 in the main text. Hence, we prepend "Denoised" to emphasize the improved accuracy of the inputs rather than suggesting that DSAM performs denoising.

3. Related Work and Comparison

Thank you for your suggestions. We will incorporate references to and discussions of the following papers in the related work section. Additionally, we have compared these approaches using the PlenopticVideo dataset.

4. Ablation study

Here's a detailed explanation of the ablation study for Table 4:

• The second row refers to using two-stage deformation operations, but both stages of feature extraction networks still utilize the ordinary deformation network.
• The third row indicates that we do not perform two-stage deformation operations. Instead, we directly replace the ordinary deformation network from the first row (baseline) with our proposed TAM.
• The fourth row means that we replace the feature extraction network used in the first stage of the two-stage deformation operations (from the second row) with our proposed TAM.
• The fifth row specifies that we do not use two-stage training. Instead, we couple temporal and spatial aggregation into a single process, training TAM and DSAM simultaneously throughout the entire process.
• The last row shows that we replace the feature extraction network used in the second stage of the two-stage deformation operations (from the fourth row) with our proposed DSAM.

[1] Bae J, Kim S, Yun Y, et al. Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting[J]. arXiv preprint arXiv:2404.03613, 2024.

[2] Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642.

[3] Wu G, Yi T, Fang J, et al. 4d gaussian splatting for real-time dynamic scene rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20310-20320.

[4] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. 2023. Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis. arXiv preprint arXiv:2312.16812 (2023).

Dear reviewer 3vN1,

We received diverse reviews for this submission. Please review the rebuttal and the comments from other reviewers and join the discussion as soon as you can. Thank you!

Your AC

Weaknesses:

We apologize for any confusion caused by our expression. Below, we will clarify your questions.

1. "Noise": In the canonical + deformable design, we input the canonical Gaussian coordinates $x, y, z$ and time $t$ into the deformable network. The deformable network essentially performs basic operations (addition, subtraction, multiplication, division) on the coordinates $x, y, z$ and time $t$. Since the point-to-point relationships within the canonical Gaussians are chaotic and erroneous, as shown in Figures 2 and 5 of the main text, it is predictable that feeding these erroneous coordinates into the deformable network will transfer this error into the deformation field, introducing inaccuracies in the final deformations $\Delta x, \Delta y, \Delta z$.
2. NSS and DTS: NSS is a strategy that uses two stages of deformation to reduce the impact of noise on the deformable network. During this process, we have two training phases. In the early training phase, we only supervise the first deformation. Once the Gaussian coordinates obtained from the first deformation are relatively accurate, we add the second deformation and shift the supervision to it, as detailed in Figure 1 of the attached PDF. This design does have limitations, which can be seen in Section 5.4 of the main text. DTS is a specific feature aggregation method we propose. Unlike 4DGaussian, D-3DGS lacks explicit spatiotemporal aggregation, so we designed DTS to aggregate spatiotemporal information. Considering that inaccurate coordinate relationships would be amplified through spatial aggregation (KNN), we only perform spatial aggregation in the second stage of NSS, which is DSAM. We name the spatial aggregation module "Denoised Spatial Aggregation Module" (DSAM) because the Gaussian coordinates input into DSAM are more accurate (denoised), as shown in the fourth column of Figure 8 in the main text. Therefore, we prefix the Spatial Aggregation Module with "Denoised." DSAM itself does not have denoising capabilities; it solely performs spatial feature aggregation.
3. "The authors claim that after the first deformation, the noise is attenuated. Why is that?": We emphasize that there are two training phases in the process. In the first phase, we only train the first deformation operation (TAM). Once the Gaussian coordinates obtained from the first deformation are relatively accurate, we add the second deformation (DSAM) into the network for training. Because the Gaussian coordinates and their relationships are relatively accurate after the first phase of training, we claim that after the first deformation, the noise is attenuated.
4. Difference Between the Last Two Rows of Table 4. We apologize for the oversight in not clearly explaining the settings for the penultimate row. The penultimate row indicates that we did not employ two training phases; instead, we trained TAM and DSAM simultaneously throughout the entire process. The main issue with this training approach is that the Gaussian coordinates deformed by TAM are never explicitly supervised, and thus, they still contain noise. Consequently, the Gaussian coordinates input into the DSAM module remain noisy. This is reflected in the quantitative results, where the performance in the penultimate row is actually lower compared to the third row of Table 4.

Questions:

We reported the time in the "Time" columns of Table 1 and Table 2 in the main text. Indeed, this two-stage training and aggregation approach does result in some delay in training speed. However, the difference is not substantial, and the method remains competitive. The specific reason is that we reduced the number of MLP layers in the deformable network.

Dear reviewer 1vtH,

We received diverse reviews for this submission. Please review the rebuttal and the comments from other reviewers and join the discussion as soon as you can. Thank you!

Your AC

1. Thank you for pointing that out. It was indeed a typo, and we will correct "$G_t(i)$" to "$G(i)$".

2. Thank you for your reminder. We will correct the above formulas and add these two parts.

   Formula 3: $\Delta x, \Delta y, \Delta z = \Psi(x, y, z, T_n)$, where $T_n$ represents the set of $t$'s neighbors.

   Formula 5: $\Delta x', \Delta y', \Delta z' = \Psi'(P_k, t)$, where $P_k$ represents the set of $k$ neighbors of ($x′, y′, z′$).

3. Thank you for your suggestions. We conducted experiments where we used only DSAM without TAM, replacing TAM with an ordinary deformation network. The results can be seen in the table below. In the camera-ready version, we will incorporate this experiment into the original table to make it complete.

   Setting	PSNR	SSIM	LPIPS
   w TAM	32.02	0.944	0.043
   w/o TAM	31.72	0.943	0.045

4. Video results: According to NeurIPS 2024 policy, we can only provide the video link to the Area Chair (AC). Please contact the AC for the video link.

5. Similarities: Our method and 3DGStream both perform deformations on 3D Gaussians (3DGs) where absolute positions and relative positional relationships are more accurately maintained. For each timestep $i$, 3DGStream uses the 3DGs from the previous timestep $i-1$ as initialization. In contrast, our method uses canonical Gaussians (which are time-independent) as the initialization. To achieve accurate relative positional relationships and minimize noise interference from the canonical Gaussians, we employ a two-stage deformation strategy. The first stage obtains accurate 3DGs, and the second stage further deforms these 3DGs to achieve preciser rendering results.

   Advantages:

   1. Flexibility: Unlike 3DGStream, our method does not require the results from the previous timestep. We can render at any arbitrary time without needing the previous timestep's 3DGs. On the other hand, 3DGStream relies on the 3DGs from the previous timestep for rendering the next.
   2. Robustness: 3DGStream heavily depends on the reconstruction quality at timestep 0. If the initial reconstruction is poor, subsequent reconstructions will be negatively affected, and these errors can accumulate over time. Our method, however, starts with noisy canonical Gaussians and improves the reconstruction quality through a two-stage deformation process, resulting in progressively better reconstructions.

   Disadvantages:

   1. Training Efficiency: 3DGStream employs an online reconstruction approach, leading to shorter training time and faster rendering speed. In contrast, our method involves offline training, which results in relatively longer training time.

6. Thank you for your reminder. We will consider the application of our method in scenarios with significant movement.