3D Gaussian Rendering Can Be Sparser: Efficient Rendering via Learned Fragment Pruning

Averaged on the Mip-NeRF 360 [6] dataset. FPS Measured on Orin NX; train time measured on A5000.

Thank you for the issues you raised! We address your concerns below:

W1: Limited novelty and scope.

First, we humbly clarify that this work does deliver new technical insight, which is “reducing fragment-level redundancy, in addition to commonly used primitive-level redundancy, enables a better trade-off between the rendering speed and quality.” In addition, compared to existing 3DGS pruning works, which prune entire Gaussian primitives, we are the first to explore redundancy at the fragment level. Hence, our approach is complementary to existing works and serves as an effective plug-in module of easy use. As such, we believe our insight and technique could extend beyond 3D Gaussian Splatting to other works that utilize a rasterization-based rendering pipeline, such as point clouds [a] and surface splatting [b,c].

Second, 3D Gaussian-based rendering techniques have not only achieved superior performance in scene reconstruction and rendering but have also been successfully applied to various other 3D vision-related applications, such as animatable human avatars [d], scene editing [e], open vocabulary querying of dynamic scenes [f], and even physical simulation [g]. Given their success in these downstream tasks, we believe that fully optimizing the efficiency of the 3D Gaussian-based rendering pipeline is a crucial and timely research question, which can largely facilitate the deployment of these 3D Gaussian-powered applications on edge devices like AR/VR headsets and smartphones.

Given the promise of 3DGS, our technique can serve as a plug-and-play add-on, allowing faster rendering speed without sacrificing rendering quality, thus promising to benefit the entire 3D community. We demonstrate this by extending our work to 4D Gaussian Splatting [29] (see Tab. 3 of our paper) and 3DGS-MCMC [h] (attached at the end of our response).

[a] TRIPS: Trilinear Point Splatting for Real-Time Radiance Field Rendering, Eurographics 2024

[b] Surface Splatting, SIGGRAPH 2001

[c] Differentiable surface splatting for point-based geometry processing, ToG 2019

[d] D3GA - Drivable 3D Gaussian Avatars, Arxiv 2023

[e] GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting, CVPR 2024

[f] LangSplat: 3D Language Gaussian Splatting, cvpr 2024

[g] PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics, CVPR 2024

[h] 3D Gaussian Splatting as Markov Chain Monte Carlo, Arxiv 2024

W2: Extra training cost not reflected in Tab. 1 and 2.

Thank you for the suggestion! For Tab. 1 in our paper, we would like to clarify that it presents the profiling experiments conducted on top of existing works, thus incurring no additional training cost. For Tab. 2, we have summarized the corresponding additional training cost (measured on an RTX A5000 GPU) of our method compared to the baselines, as shown in the table below.

We can see that the additional training cost adds less than 25% overhead compared to the baselines, as we optimize opacity and truncation thresholds for less than 30% of the iterations in the second training stage. Furthermore, if the goal is to achieve rendering quality on par with the baselines, we can further reduce the number of iterations in the second stage of our method (e.g., reducing from 5,000 to 1,000 iterations), achieving 1.7× higher FPS while incurring only 3.5% additional training cost and maintaining the same rendering quality (i.e., PSNR/SSIM) on the Mip-NeRF 360 dataset compared to native 3DGS. We will include this discussion in the final version of our paper.

W3: Why increased PSNR after pruning? Naive optimization: easily reduce the opacity to 0 instead of pruning it.

Good question! In a nutshell, the PSNR gain achieved by our work stems from the ability of a Gaussian function with a cutoff to better fit sharp signals in real-world scenes compared to the native Gaussian function, as shown in Fig. 1 of the global 1-page response. Specifically, we would like to clarify the following two points:

P1: Enhancing the Gaussian function with learnable thresholds increases its ability to fit signals. As illustrated in Fig. 1(a), enhancing each Gaussian function with a learnable threshold essentially gives each Gaussian a flexible cutoff range. In Fig. 1(c), we show that the truncated Gaussian function better fits various signals as compared to the native Gaussian functions, thanks to the enhanced capability of the learnable threshold/cutoff. Since sharp signals widely exist in real-world scenes and are the primary motivation behind existing anti-aliasing works [6,7], better fitting these signals using a Gaussian with a cutoff, as in our method, improves the PSNR by an average of 0.16 on the Tanks & Temples dataset, as shown in Tab. 2 of our paper.

P2: Mathematically, reducing the opacity of Gaussians is NOT equivalent to fragment pruning. As illustrated in Fig. 1(a) and Fig. 1(b), we demonstrate that fragment pruning (i.e., a learnable cutoff of the Gaussian function) is not mathematically equivalent to lowering the opacity (i.e., a global magnitude factor) of a Gaussian function. The former impacts only the Gaussian value around the cutoff threshold, while the latter reduces the magnitude of the Gaussian function globally. Therefore, the Gaussian with cutoffs adopted in our method can better fit sharp signals, such as square signals, compared to “easily reduce the opacity to 0”.

Thank you for raising questions regarding our baseline choices! We provide the following clarifications:

W1 & Q2: No comparison with other state-of-the-art Gaussian pruning pipelines on static scenes.

Following your suggestion, we have added the comparison with additional Gaussian primitive pruning methods on the Mip-NeRF-360 dataset in the table below.

We observe that (1) among all prior Gaussian primitive pruning methods, Mini-Splatting [19] achieved the best trade-off between rendering quality and speed, and (2) our method, which performs pruning at the fragment level, further improves both the rendering quality and speed of Mini-Splatting [19]: +0.04 PSNR and 1.54× faster rendering speeds.

[a] LP-3DGS: Learning to Prune 3D Gaussian Splatting, Arxiv 2024

W2 & Q2: Lack of comparison with other Gaussian primitive pruning works on dynamic scenes.

We would like to clarify that, to the best of our knowledge, existing Gaussian primitive pruning works focus exclusively on static 3D Gaussians. We are not aware of any work that performs pruning on dynamic 3D Gaussians. Our work is the first post-processing pruning framework designed to improve rendering speed on dynamic scenes. Consequently, our experiments on dynamic scenes were conducted on top of RT-4DGS [29], the state-of-the-art open-source 3DGS for dynamic scenes in terms of rendering quality and speed trade-off. As shown in Table 3 of our paper, our method achieved 1.57× faster rendering speeds with a 0.03 PSNR improvement compared to RT-4DGS [29].

Q1: Why did you primarily compare with neural radiance field methods instead of rasterization-based radiance field rendering as your baseline?

We did not primarily compare our work with rasterization-based methods due to their significantly lower rendering quality (e.g., more than 3 PSNR lower, as shown in the table below) compared to NeRF-based methods and native 3DGS. This baseline selection strategy aligns with that used in native 3DGS. Meanwhile, following your suggestion, we have included a comparison with additional rasterization-based methods in the table below.

We can see that despite the faster rendering speeds (> 60 FPS), the rasterization-based method fails to achieve the same rendering quality as the native 3DGS. In contrast, our method not only achieves approximately 60 FPS rendering speeds but also slightly improves the rendering quality (+0.04 PSNR).

[b] Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement, ICCV 2023

[c] Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes, 3DV 2024

Dear Reviewer TxBU,

Thank you for reviewing our rebuttal and for your prompt response! We are pleased to hear that our response has addressed some of your concerns.

We greatly appreciate your further clarification on Question 1 and your constructive suggestions. It is worth noting that during the rebuttal period, we successfully extended our framework to 3DGS-MCMC [d], a 3DGS variant with better rendering quality than the vanilla 3DGS [16]. The corresponding results on the Mip-NeRF 360 dataset [6] have been included in the last table of our response to Reviewer ytrw and are listed below.

We will continue to extend our pruning framework to additional 3DGS pipelines and provide quantitative evaluations as you suggested. We will keep you updated on our progress during this discussion period.

Thank you again for your valuable feedback.

[d] 3D Gaussian Splatting as Markov Chain Monte Carlo, ArXiv 2024

Based on the benchmark results summarized in the tables above, we can draw the following conclusions:

• Original Rendering Pipelines [e,d,f] vs. Ours: Our proposed fragment pruning method consistently enhances rendering speed without compromising rendering quality across various 3DGS-based rendering pipelines. Specifically, the proposed method (Original + Ours) increases rendering speed by 1.4x, 1.8x, and 1.9x on Mip-Splatting [e], GS-MCMC [d], and RAIN-GS [f], respectively, with a 0.03 to 0.05 higher PSNR.

• Baseline Pruning Techniques [35, a] vs. Ours: Our proposed fragment pruning not only achieves better accuracy vs. efficiency trade-offs than baseline primitive-level pruning techniques [35, a], e.g., 0.08 to 0.16 higher PSNR and 1.5x to 1.6x faster rendering speeds on GS-MCMC [d], but also can further improve these existing primitive-level pruning techniques when applied on top of them,e.g., 0.19 to 0.27 higher PSNR and 1.9x to 2.0x faster rendering speeds on GS-MCMC [d].

The conclusions above are consistent with our observations on the vanilla 3DGS as discussed in our submitted manuscript: our proposed method (1) provides better accuracy vs. efficiency trade-offs than baseline rendering pipelines and (2) is complementary to existing primitive-based pruning techniques. We greatly appreciate your constructive suggestions and believe adding these additional baselines and experiments will help strengthen our work and contributions.

[e] Mip-Splatting: Alias-free 3D Gaussian Splatting, CVPR 2024

[f] Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting, Arxiv 2024

[i] RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS, Arxiv 2024

[j] Gaussian Opacity Fields: Efficient High-quality Compact Surface Reconstruction in Unbounded Scenes, Arxiv 2024

Thank you once again for taking the time to read our rebuttal and further providing your thoughtful feedback! In response to your suggestion, we have extended our evaluation to include three rasterization-based (3DGS-based) rendering pipelines that enhance 3DGS rendering quality from different aspects:

• Mip-Splatting [e]: A 3DGS-based pipeline for anti-aliasing in extreme camera locations (e.g., very close to objects).

• GS-MCMC [d]: A 3DGS-based pipeline with a more principled approach to densifying Gaussian primitives, resulting in fewer artifacts (e.g., floaters) in the reconstructed scenes.

• RAIN-GS [f]: A 3DGS-based pipeline that relies less on accurate point cloud initialization, utilizing a dedicated point cloud optimization process.

Benchmark Results on Mip-Splatting [e] on the Mip-NeRF 360 dataset [6]:

*: Note that the official Mip-Splatting codebase has been enhanced with an improved densification process [j], resulting in higher rendering quality than what was originally reported in its paper. We used the single-scale training and single-scale testing setting of Mip-Splatting [e].

Benchmark Results on GS-MCMC [d] on the Mip-NeRF 360 dataset [6]:

*: We used the random point cloud initialization setting of GS-MCMC [d].

Benchmark Results on RAIN-GS [f] on the Mip-NeRF 360 dataset [6]:

*: We used the random point cloud initialization setting of RAIN-GS [f].

With the addition of the three extra rendering pipelines, we benchmarked our fragment pruning method against two state-of-the-art plug-and-play pruning methods: LightGaussian [35] and LP-3DGS [a], as shown in rows 3-4 in the above tables. In addition to directly comparing our method with these baseline pruning methods, we also applied our method on top of the baseline pruning method which provides better rendering quality (see row 6 of the above tables) to demonstrate its compatibility with existing primitive-based pruning techniques. We would like to note that Mini-Splatting [1] was not included in this set of additional experiments due to insufficient time during the discussion period to integrate its end-to-end, fully customized optimization pipeline (i.e., densification, reinitialization, and simplification techniques) with the customized optimization pipelines of those 3DGS variants [e, d, f]. We will continue this experiment and include it in our final version.

We appreciate your recognition of our work and your suggestions to further enhance our experimental results.

W1: I would love to see an experiment showing rendering speed and PSNR for different fixed values of threshold vs the learnt threshold. This would effectively show that having per Gaussian optimized threshold is truly needed.

We conducted a grid search on fixed threshold values and followed the same fine-tuning protocol as described in Section 6.2 of our paper. The results summarized in Fig. 3 (b) of the global 1-page response indicate that our method achieves a better accuracy vs. efficiency trade-off than tuning fixed threshold values, highlighting that learning a per-Gaussian threshold, as in our method, is truly necessary.

W2: Did you explore doing an alternating optimization between the usual 3DGs parameters and the thresholds during training? I am not sure if that would result in lower training time in total than the usual training + post-processing, but it is interesting to see.

Thank you for this insightful question! Following your suggestion, we conducted an alternating optimization experiment on the MipNeRF 360 dataset [6]. Specifically, we maintained the training process for the first 15,000 iterations as in the vanilla 3D Gaussian Splatting [16], then alternated between threshold learning and 3D Gaussian parameter learning every 5,000 iterations up to 40,000 iterations. We tested the PSNR every 5,000 iterations, and the results are shown in the table below:

We can observe that the PSNR quickly saturates at around 20,000 iterations and fails to match the PSNR of the vanilla 3D Gaussian and our method. We conjecture that the lower rendering quality in this setting, compared to our method, is due to reducing the covered pixels per Gaussian too early in the training stage, which limits the number of pixels from which it can receive gradients. In contrast, our method strikes a balance between ease of use and achieved performance compared to alternating optimization. However, we believe that the alternating optimization you proposed has the potential to achieve superior rendering quality with more thorough parameter tuning. We will continue to explore this approach and include a discussion on optimization methods in our final version.

W3: Additional point cloud renderings to assess the density of points/fragments on the qualitative figures of the supplementary material are helpful to have

We have added the per-pixel fragment density in Fig. 2 of the global 1-page response, demonstrating the effectiveness of our proposed method in reducing fragment density across different pixels (e.g., reducing the average fragments per pixel from 49.2 to 32.9 in the Bicycle scene [6]), thereby improving overall rendering speed. We will include this discussion and provide additional visualizations of the point cloud renderings in our final version.

W4: The results for overfitting with training on all the 3DGS parameters are not shown in the supplemental material.

Thank you for the suggestion! We conducted overfitting experiments by loading pre-trained checkpoints at 30,000 iterations from native 3DGS and continuing training for an additional 20,000 iterations following the native 3DGS settings. As shown in Fig. 3 (c) of the global 1-page response, while overfitting does improve the overall PSNR, it still results in a slightly lower PSNR (e.g., -0.03) compared to our method, even with 40% more iterations.

Dear Reviewer,

I wanted to gently remind you to please review the rebuttal provided by the authors. Please enter a discussion with the authors to address any outstanding questions.

Your feedback is invaluable to the decision-making process, and if you feel that the rebuttal addresses any of your concerns, please consider updating your score accordingly.

Thank you for your continued dedication to ensuring a fair and thorough review process!

Best, Your AC

I would like to thank the authors for their response to my concerns, I have no other major concerns. I will retain my original score, as I believe the paper is technically sound, gives a valuable analysis and will have a moderate-to-high impact on the 3D rendering area.

We greatly appreciate your positive feedback and constructive suggestions for our work. Below, we address each of your comments in detail:

W1: Title mismatch: The method changes the # of Gaussians per pixel ray.

Thank you for the suggestion! We will revise the title accordingly and clarify that the projection of the 3D Gaussian onto the 2D image to be rendered is sparser, not the 3D Gaussian itself.

W2 & W3: Device details, # of Gaussians, and losses.

Thanks for pointing these out! To address your suggestions, the following details will be added to our final version:

• Line 18: The speedup is achieved on an edge GPU device Jetson Orin NX [18].
• Line 35: The concrete number of Gaussians is 3,161,131 on average in the Mip-NeRF 360 dataset [6] when trained using the original 3D Gaussian implementation [16].
• Sec. 6.2: We used the same loss function as the original 3D Gaussian Splatting paper [16] for static scenes (i.e., both SSIM Loss and L1 Loss) and the same loss function as the RT-4DGS paper [29] for dynamic scenes.

W4: Connection with two staged frameworks.

The connection between the prior two-stage works and our work can be summarized as follows:

Similarity: Both prior works [14,a,b] and our work reduce computational workload by decreasing the computation per pixel ray.

Differences: Our technique serves as an add-on to one of the most advanced scene representations, 3DGS, which allows for faster rendering without sacrificing quality. Different from prior two-stage works, our method does not alter the scene representation and thus can better maintain the rendering quality. As illustrated in Fig.2 of the global 1-page response, our method reduced the average number of fragments/Gaussians per pixel without strictly enforcing it to be 1. As a result, our approach can boost rendering speed while maintaining the rendering quality, achieving a +0.04 PSNR improvement on [6] over the vanilla 3DGS [16]. In contrast, the prior two-stage frameworks [14,a,b] convert NeRFs into meshes to enforce a single sample per pixel ray. This significant reduction, along with the differences between the scene representation used in training and rendering, leads to noticeable quality loss. For instance, on the MipNeRF-360 dataset [6], the works [14,a,b] achieved PSNR scores of 21.95/22.74/23.59, which are lower than even the vanilla NeRF [4] (23.85 PSNR, as reported by [6]).

We will incorporate the discussion above into our final version.

[a] Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement, ICCV 2023

[b] Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes, 3DV 2024

W5: Why can it automatically discover the perfect Gaussian-wise threshold to reduce the number of Gaussians per pixel ray? It could theoretically learn thresholds that increase the number as well.

Your understanding is correct! Learning the threshold can either decrease or increase the number of Gaussians per pixel ray. The key insight behind this is that the decrease or increase is determined by the types of signals we want to represent using the Gaussian function. Specifically, as shown in Fig. 1(a) of our global 1-page response, enhancing each Gaussian function with a learnable threshold essentially gives each Gaussian a flexible cutoff range. Thus, when the signal we want to fit has sharp edges (e.g., the square signal in Fig. 1(d) of our global 1-page response), learning the Gaussian-specific threshold offers the highest fidelity in fitting the ground truth signal compared to the original Gaussian function. Since the real-world scenes captured in our experiments contain many such sharp edges, also the source of anti-aliasing artifacts that [6,7] target, the threshold is learned to make the Gaussian cutoff narrower. This results in a reduced number of Gaussians per pixel ray. Our visualization of the number of fragments per pixel, with and without our method, in Fig. 2 of the global 1-page response also indicates that the number of fragments decreases most significantly at the edges of scenes.

W6: Naive methods: smaller scales of Gaussians.

Following your suggestions, we conducted experiments to encourage smaller scales of Gaussians by introducing an additional loss on scales. As summarized in Fig.3(a) of the global 1-page response, we found that while this approach can also improve rendering speed, it does not recover rendering quality to the same extent as our method. Specifically, the lower quality of the native method is due to the fact that smaller scales of Gaussians do not have the same effect as Gaussians with a cutoff, as illustrated in Fig. 1(a) and (c). This results in a failure to fit sharp signals, such as the square signal shown in Fig. 1(d) of the global 1-page response, as effectively as our method.

W7: Edge-device demo.

We have demonstrated the real-time edge-device demo setup on a 15-watt edge GPU device, the Nvidia Jetson Orin NX 16GB, as shown in Fig. 4(a) of the global 1-page response. We will include this setup as part of our code release in the final version.

W8: Profiling with error bars.

Great suggestion! We have further enhanced our profiling in Fig. 4(b) of the global 1-page response by including the standard error.

W9: Validate our methods in other variants of 3DGS.

Thank you for the suggestion! Given the limited rebuttal time, we have extended our work to GS-MCMC [c]. As shown in the table below, our proposed method improves the rendering speed of GS-MCMC by 1.8× without compromising the rendering quality. We will conduct experiments on other 3D Gaussian variants, such as Mip3DGS, in the final version.

[c] 3D Gaussian Splatting as Markov Chain Monte Carlo, Arxiv 2024

Thank you once again for your constructive review and for providing your positive feedback on our initial rebuttal! We are writing to share our updated experimental results on your suggested additional 3DGS variants, beyond GS-MCMC [c] that was included in our previous response.

Specifically, we have extended our evaluation to include three 3DGS variants that enhance 3DGS from different aspects:

• Mip-Splatting [e]: A 3DGS variant for anti-aliasing in extreme camera locations (e.g., very close to objects).
• GS-MCMC [c]: A 3DGS variant with a more principled approach to densifying Gaussian primitives, resulting in fewer artifacts (e.g., floaters) in the reconstructed scenes.
• RAIN-GS [f]: A 3DGS variant that relies less on accurate point cloud initialization, utilizing a dedicated point cloud optimization process.

Benchmark Results on Mip-Splatting [e] on the Mip-NeRF 360 dataset [6]:

*: Note that the official Mip-Splatting codebase has been enhanced with an improved densification process [j], resulting in higher rendering quality than what was originally reported in its paper. We used the single-scale training and single-scale testing setting of Mip-Splatting [e].

Benchmark Results on GS-MCMC [c] on the Mip-NeRF 360 dataset [6]:

*: We used the random point cloud initialization setting of GS-MCMC [c].

Benchmark Results on RAIN-GS [f] on the Mip-NeRF 360 dataset [6]:

*: We used the random point cloud initialization setting of RAIN-GS [f].

With the addition of the three extra rendering pipelines, we benchmarked our fragment pruning method against two state-of-the-art plug-and-play pruning methods: LightGaussian [35] and LP-3DGS [d], as shown in rows 3-4 in the above tables. In addition to directly comparing our method with these baseline pruning methods, we also applied our method on top of the baseline pruning method which provides better rendering quality (see row 6 of the above tables) to demonstrate its compatibility with existing primitive-based pruning techniques. We would like to note that Mini-Splatting [1] was not included in this set of additional experiments due to insufficient time during the discussion period to integrate its end-to-end, fully customized optimization pipeline (i.e., densification, reinitialization, and simplification techniques) with the customized optimization pipelines of those 3DGS variants [e, c, f]. We will continue this experiment and include it in our final version.

Based on the benchmark results summarized in the tables above, we can draw the following conclusions:

• Original Rendering Pipelines [e,c,f] vs. Ours: Our proposed fragment pruning method consistently enhances rendering speed without compromising rendering quality across various 3DGS-based rendering pipelines. Specifically, the proposed method (Original + Ours) increases rendering speed by 1.4x, 1.8x, and 1.9x on Mip-Splatting [e], GS-MCMC [c], and RAIN-GS [f], respectively, with a 0.03 to 0.05 higher PSNR.

• Baseline Pruning Techniques [35, d] vs. Ours: Our proposed fragment pruning not only achieves better accuracy vs. efficiency trade-offs than baseline primitive-level pruning techniques [35, d], e.g., 0.08 to 0.16 higher PSNR and 1.5x to 1.6x faster rendering speeds on GS-MCMC [c], but also can further improve these existing primitive-level pruning techniques when applied on top of them, e.g., 0.19 to 0.27 higher PSNR and 1.9x to 2.0x faster rendering speeds on GS-MCMC [c].

The conclusions above are consistent with our observations on the vanilla 3DGS as discussed in our submitted manuscript: our proposed method (1) provides better accuracy vs. efficiency trade-offs than baseline rendering pipelines and (2) is complementary to existing primitive-based pruning techniques. We greatly appreciate your constructive suggestions and believe adding these additional baselines and experiments will help strengthen our work and contributions.

[d] LP-3DGS: Learning to Prune 3D Gaussian Splatting, Arxiv 2024

[e] Mip-Splatting: Alias-free 3D Gaussian Splatting, CVPR 2024

[f] Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting, Arxiv 2024

[i] RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS, Arxiv 2024

[j] Gaussian Opacity Fields: Efficient High-quality Compact Surface Reconstruction in Unbounded Scenes, Arxiv 2024

Dear Reviewer ytrw,

Thank you very much again for your constructive review! We believe that including your suggested experiments and clarification will further strengthen our work, which we will incorporate in the final version.

You mentioned that our rebuttal has addressed most of your questions and that you will adjust your evaluation, but we have not yet seen your updated evaluation on our end. As we are approaching the end of the author-reviewer discussion period, we are following up to check whether you need any further information or clarification from our side to assist in your adjustment.

Thank you again for your time and effort in reviewing our paper!