1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering

We sincerely appreciate the thorough review and insightful feedback provided by each reviewer. The reviewers asked perceptive questions and comments, which are answered in detail in individual responses and have improved our submission.

>>> Q1: The discrepancy between FPS and raster FPS should be clarified.

>>> A1: We truly appreciate the reviewer for this excellent question, as it, in fact, touches upon a key aspect of our results.

The two metrics, "raster FPS" and "end-to-end FPS", are presented to serve complementary purposes.

• The "raster FPS" (1092 FPS) is reported because it most directly measures our contribution: pruning and filtering the number of Gaussians. By isolating the rasterization step, this metric clearly demonstrates the efficiency of our method.
• The "end-to-end FPS" (805 FPS) represents the practical rendering speed. It provides a holistic view by including all stages of the rendering module. Both are important, as we aim to demonstrate that we improve the rasterization speed, and consequently, the end-to-end (E2E) speed is also enhanced.

>>> Q2: Performance on mid-range hardware or edge devices.

>>> A2: Thanks for your interest. Our method maintains high rendering speed even on devices with limited computational resources. We have discussed this issue in the Appendix (Section A.2, Lines 778–780). We implement our model on the NVIDIA TITAN X GPU(released in 2015). It still achieves over 200 FPS on the N3V dataset, significantly outperforming the vanilla 4DGS (20 FPS). It is worth noting that this rendering speed is nearly on par with the performance of other baselines running on an RTX 3090.

>>> Q3: A theoretical justification for the aggregation method (e.g., why multiplicative instead of additive?).

>>> A3: Thanks for your insightful suggestions. Each 4D Gaussian can be decomposed into a conditional 3D spatial Gaussian and a marginal 1D temporal Gaussian, as demonstrated in 4DGS[1], where $p(x, y, z, t) = p(x, y, z|t)p(t)$. This perspective inspires us to model the spatial and temporal terms as a product. Furthermore, we also report the 'additive' results on the N3V dataset following the default experiment setting. As shown in this table, the product formulation yields higher PSNR (31.92 vs. 31.55) and SSIM (0.9462 vs. 0.9416), and achieves a lower LPIPS score (0.0513 vs. 0.0611). This experiment will be appended to the main paper to support our design of the score.

[1]Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

>>> Q4: A sensitivity analysis of pruning threshold.

>>> A4: The sensitivity analysis is provided in Fig.9 in the Appendices. We analyze the impact of varying pruning ratios on rendering quality. It demonstrates that our spatial-temporal variation score-based pruning can even improve scene rendering quality when the pruning ratio is relatively low. Moreover, it can maintain results comparable to the vanilla 4DGS at higher thresholds.

Thank you for your positive feedback and insightful comments! Below, we respond to address your concern. We welcome further discussion to enhance the clarity and effectiveness of our work.

>>> W1: Limitations on the application in much higher-performance models, such as Deformable-GS, SC-GS and Grid4D.

>>> A1: We acknowledge that our method was initially designed for 4DGS; however, we argue that our model still demonstrates strong generalizability. While our method is not directly applicable to deformation-based family, it is designed to be compatible with the entire 4DGS family. For example, our method is compatible with other works like Temporal Gaussian Hierarchy[1] and FreeTimeGS[2].

[1] Representing Long Volumetric Video with Temporal Gaussian Hierarchy.

[2] FreeTimeGS: Free Gaussian Primitives at Anytime and Anywhere for Dynamic Scene Reconstruction.

>>> W2: Rendering quality degradation on the D-NeRF dataset.

A2: There is no degradation in rendering quality on the D-NeRF datasets. While the results from the original 4DGS [1] are higher than ours, those were obtained under different devices and settings. For a fair comparison, we retrained the official 4DGS code on the D-NeRF dataset using the same protocol. As shown in Table 2, retrained 4DGS achieves 32.99 dB, while our method reaches 33.37 dB. Thus, there is no degradation.

>>> Q1: Baselines such as SC-GS and Grid4D on the D-NeRF dataset.

>>> A3: Thanks for mentioning these excellent works. We compare our method with these methods on the D-NeRF dataset. This table will be appended to Tab.2 in the main paper to provide a more comprehensive comparison. While our method exhibits a reduction in rendering quality compared with these models, it achieves significantly accelerated rendering performance and minimized storage requirements, underscoring its efficiency advantages.

>>> Q2: Lack of training speed evaluation. Will the pruning and filtering processes take a lot of time?

>>> A2: No. On the N3V dataset, pruning and filtering processes take nearly 10 minutes. Moreover, 4DGS-1K only takes approximately 30 minutes to fine-tune. This result is provided in Line 773 in the Appendices.

>>> Q3: Why not use the $\left|\Sigma_t\right|$ rather than the second derivative for the temporal score?

>>> A3: We appreciate your valuable insight. Directly using $\left|\Sigma_t\right|$ for pruning results in a reduction in rendering quality. This has been discussed in the Appendix(See Tab.4, panels (e)). Using $\left|\Sigma_t\right|$ yields a PSNR of 33.47 dB on the Sear Steak scene, compared to 33.60 dB for vanilla 4DGS. The PSNR achieved by our method is 33.76 dB. This is because $\left|\Sigma_t\right|$ follows a long-tailed distribution, with its values extending over the entire set of real numbers. As a result, this score disproportionately highlights a minority of large $\left|\Sigma_t\right|$, and diminishes the discriminative capacity of Gaussians with smaller $\left|\Sigma_t\right|$.

We would like to thank the reviewer for their insightful feedback and interesting observations. We address the reviewer’s comments below and will include all the feedback in the revised version of the manuscript.

>>> W1: Comparisons with more recent and competitive works such as 4K4D, Swift4D.

>>> A1: Thanks for mentioning these excellent works! As suggested, we compared our method with 4K4D[1], Swift4D[2], Grid4D[3] and DynMF[4] on the N3V dataset. As shown in the table below, the rendering quality achieved by our model exceeds that of the majority of models, except for Swift4D[2]. Furthermore, our model demonstrates substantially superior rendering speed and storage performance. This table will be appended to Tab.1 in the main paper, and will cite these impressive works.

[1] 4K4D: Real-Time 4D View Synthesis at 4K Resolution.

[2] Swift4D: Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene.

[3] Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynamic Gaussian Splatting.

[4] DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting.

>>> W2: Inconsistent performance between Sec.C and average performance. How do the proposed strategies directly improve performance via floaters pruning?

>>> A2:

• Sorry for any confusion! There is no inconsistency between the results. While the results from the original 4DGS [1] are higher than ours, those were obtained under different devices and settings. For a fair comparison, we retrained the official 4DGS code on the D-NeRF dataset using the same protocol. As shown in Table 2, retrained 4DGS achieves 32.99 dB, while our method reaches 33.37 dB. Thus, we claim the improvement.
• For floaters pruning, floaters are typically characterized by limited spatial extent and short lifespan. Therefore, floaters tend to receive lower scores relative to other Gaussians. They are pruned directly during this stage, which leads to enhanced rendering quality.

[1] Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

>>> W3: Why do most of the baselines not report values for raster FPS and the number of Gaussians?

>>> A3: We sincerely apologize for any confusion and wish to emphasize that these two evaluation metrics are meaningful within our context. However, they are not applicable when making comparisons with other baselines.

Most baselines represent the motion of Gaussians through deformation fields or trajectory functions. Before rasterization, they need to obtain the Gaussian attributes at the current moment through an MLP or trajectory function $f$. Therefore, for rendering speed, rasterization is not the only factor affecting FPS and cannot be considered in isolation. For the number of Gaussians, we wish to emphasize that each method defines Gaussian attributes differently, resulting in varying storage per Gaussian. Therefore, comparing the number of Gaussians across these baselines is both unfair and not applicable. The general evaluation metric should be the storage footprint of these models.

Why do we report these two metrics? These metrics serve primarily as indicators to demonstrate the improvements our method brings to 4DGS. Raster FPS is reported to avoid the influence of other operations within the rendering module, as discussed in Lines 864 to 868. The number of Gaussians is reported to demonstrate the high pruning rate of our method.

Overall, these two metrics are applied and reported solely to 4DGS and 4DGS-1K, and are therefore not reported for the other baselines.

>>> W4: A limited generalizability which could reduce the broader significance and impact of the work.

>>> A4: Thanks for the question! While our work builds upon 4DGS, we argue it is a significant leap forward, not incremental at all.

In terms of the task, our method introduces order-of-magnitude improvements over the existing 4DGS. Our method is the first to achieve over 1000 FPS for dynamic scenes. We deliver an $8×$ rendering speedup and a $41×$ storage reduction over the baseline.

Regarding generalization, dynamic Gaussian methods fall broadly into two families: deformation-based methods and direct 4D Gaussians. While our method is not directly applicable to deformation-based family, it is designed to be compatible with the entire 4DGS family. For example, our method is compatible with other works like Temporal Gaussian Hierarchy[1] and FreeTimeGS[2].

[1] Representing Long Volumetric Video with Temporal Gaussian Hierarchy.

[2] FreeTimeGS: Free Gaussian Primitives at Anytime and Anywhere for Dynamic Scene Reconstruction.

>>> Q1: How are inactive Gaussians and the visibility mask actually identified?

>>> A1: The visibility mask $m_{i,j}$ from $j^{th}$ training viewpoint at a given timestamp $t_i$ is naturally obtained through the rasterization, which has already been implemented in 3DGS[1]. Furthermore, we define inactive Gaussians as the complement of the visibility mask, denoted as $\neg~ m_{i,j}$. We apologize for any confusion and will include the relevant description in the main paper.

[1] 3D Gaussian Splatting for Real-Time Radiance Field Rendering.

>>> Q2: Reorganize Section 5.3 to improve readability.

>>> A2: We appreciate your suggestion. Sec.5.3 discusses the effects of pruning, filtering, and fine-tuning, which may appear somewhat disjointed. We plan to reorganize this section and revise the table layout to improve clarity and readability. Additionally, we would like to clarify that the results in elements (e) and (f) are referred to in Line 300 to demonstrate the effect of fine-tuning on filtering.

>>> Q3: Do the proposed strategies also improve training speed?

>>> A3: No. Our method does not demonstrate a significant improvement in training speed.

We thank the reviewer for the constructive comments. We provide our feedback as follows.

>>> Q1: If this method could be applied during training to mitigate quality loss?

>>> A1: Yes. This is feasible. For example, we can periodically prune and update the visibility mask of Gaussians and continue training. In our experiments, fine-tuning the remaining Gaussians can be considered as a part of this process, and it demonstrates a significant alleviation of the rendering quality loss.

>>> Q2: What does NHW mean in Equation 4?

>>> A2: $N$ denotes the number of training views, while $H$ and $W$ denote the height and width of the images, respectively. By $NHW$, we mean all pixels in the training views. In Equation 4, this indicates that aggregating the contributions of each Gaussian to all pixels across all training views. We appreciate your reminder and will revise this in the main paper.

>>> Q3: How are transient objects in the scene handled? Could these dynamic elements be mistakenly filtered out?

>>> A3: Thanks for your interest.

• We treat static and dynamic objects, including transient objects, within a unified framework. All Gaussians are evaluated together to estimate their scores and record their visibility masks. It is consistent with the insight on 4DGS for modeling dynamic scenes.
• In fact, the dynamic elements will not be filtered out. We provide a visualization of the pruned Gaussians in the Sear Steak Scene in the Appendix(See Sec.A.4 and Fig.7). Our method prunes only redundant parts in vanilla 4DGS, while the remaining Gaussians still accurately render dynamic content such as moving hands and pets. This is because dynamic objects are not represented by a single long-lifespan Gaussian but instead by an aggregation of numerous redundant short‑lifespan Gaussians, as we mentioned in Lines 127 to 134. Our pruning strategy only removes redundant and outlier Gaussians, while preserving legitimate dynamic elements like moving objects.

>>> Q4: What does "pp" stand for in Table 3? Provide an ablation study on the fine-tuning process.

>>> A4:

• The abbreviation “PP” refers to the Post-Processing techniques mentioned in Lines 259–261.
• We conducted an ablation study on fine-tuning provided in Tab.3. The fine-tuning process helps mitigate the degradation in rendering quality introduced by pruning and filtering. It achieves an improvement in PSNR from 31.63 dB to 31.88 dB. A comprehensive comparison can be found in Table 3, particularly elements (b) and (c), as well as elements (e) and (f).

>>> Q5: Discussion about implementing the methods on the recent SOTA 4D Gaussian Representations, e.g., Representing Long Volumetric Video with Temporal Gaussian Hierarchy.

>>> A5: Thanks for sharing this impressive work and offering this valuable suggestion. We will discuss two current SOTA 4D Gaussian representations, Temporal Gaussian Hierarchy[1] and FreeTimeGS[2]. Meanwhile, it helps demonstrate the generalizability of our method. It will be appended to our paper.

Temporal Gaussian Hierarchy[1] provides a hierarchical organization of 4DGS[3] to represent Gaussians with varying speeds. Therefore, for the pruning strategy, while considering the $\Sigma_t$ of each Gaussian, the hierarchical depth also needs to be taken into account. For filtering, this hierarchical organization provides additional information for this process to exploit. Different intervals can then be assigned to segments at different levels. For instance, smaller intervals can be assigned to the higher-level segments to capture some short‑lifespan Gaussians. This helps us record Gaussians at a finer granularity, thereby mitigating the loss in rendering quality caused by filtering.

For FreeTimeGS, it models the lifespan of Gaussians by defining a unimodal function with a scaling parameter.

$

\sigma(t)=exp(-\frac{1}{2}(\frac{t-\mu_t}{s})^2)

$

Therefore, for the pruning strategy, the temporal score can still be derived from the second derivative of $\sigma(t)$, whereas the spatial scores follow the original processing procedure. As for the filter, FreeTimeGS is likely to share similar properties that active Gaussians in adjacent frames overlap considerably. Therefore, our filtering strategy can still be applied to this method.

[1] Representing Long Volumetric Video with Temporal Gaussian Hierarchy.

[2] FreeTimeGS: Free Gaussian Primitives at Anytime and Anywhere for Dynamic Scene Reconstruction.

[3] Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting.