Lightweight Predictive 3D Gaussian Splats

We sincerely thank the reviewer for the positive comments. We address the suggested typos in the revised version. We will release our code and pre-trained models upon the paper acceptance.

Regarding the choice of K, we choose different K for different scenes. As our adaptive tree manipulation (ATM) can grow trees in both depth and breadth, we notice that setting K as 2 achieves the best results for most scenes. In Supplementary (Tab.4 and Tab.5), we show the value of K for different datasets.

We sincerely thank the reviewer for the positive feedback and suggestions. In the revised version, we have corrected the suggested typos and added an additional graph to explain our approach in Appendix A.1.

The explanation in the appendix improves the cleanness of the tree manipulation. The feedbacks to my questions seem reasonable.

I will keep my original rating.

Q1. Clarification on Novelty.

We would like to thank the reviewer for suggesting the concurrent work (e.g., HAC [1], Compressed3D [2], Reduced3D [3], SOG [4], and Compact3D [5]). They are indeed relevant and we will discuss them in the revised paper. In the following, we would like to kindly emphasize the major differences between our method and the concurrent and suggested works.

1. Ours v.s. Anchor-based Methods. As discussed in the main paper (Line 142-145 and Line 150-154), there are two major differences between anchor-based approaches (e.g., ScaffoldGS [10]) and our approach. First, the anchor structure in ScaffoldGS [10] is essentially a reparameterization of the 3DGS [9] representation, where splat positions are parameterized relative to anchors. Consequently, ScaffoldGS [10] must store both the anchors and the relative positions during inference, which results in a lower compression rate, as shown in Table 1 of its paper. In contrast, our hierarchical tree structure is predictive, leveraging spatial relationships via the tree combined with an attention mechanism to enforce similarity. This allows us to store only the parent splats, achieving significantly higher compression rates (as mentioned by Reviewers zk78, 9HaV). Second, the proposed hierarchical tree structure fundamentally differs from the anchor-based structure. Unlike anchors, which only capture a single level of connectivity between splats, our tree representation combined with the proposed ATM takes into account the significance of both parent and children splats, allowing for a strategy of sub-tree expansion.

   HAC [1] closely follows the anchor-based method ScaffoldGS [10]. The underlying representation (i.e., anchor) is essentially the same as ScaffoldGS [10] (the discussion above between anchors and hierarchical tree representation still apply). Moreover, our ATM is different from the learnable masks in HAC [1] in many aspects. ATM contains three operations: Promotion, Pruning and Cloning (Line 283-287), introducing an effective way to expand and delete the trees. The learnable mask in HAC [1] is similar to the Gaussian volume mask in CompactGS [8], which prunes the unimportant individual splats with binary masks without considering the anchors as a whole. Our ATM is the key to enabling the hierarchical tree in our framework.

2. Ours v.s. Hash-grid based Methods. Our method uses hash-grids (displacement and attributes) to query features for both positions and geometric attributes (Line 147-148, Figure 2) and predict the children's positions and attributes on-the-fly, whereas the CompactGS [8] only predicts the positions. To facilitate such workflow, we adopt a specific design of hash-grid. We assume the features of positions and attributes of splats should not be disentangled in the spatial grid. Thus, we propose a strategy to first query the positions and re-query the hash-grid with the predicted positions for their geometric attributes. Our use of a hash-grid combined with the proposed attention mechanism is to ensure the inherent feature sharings between the splats in their vicinity. We kindly ask the reviewer to check Section 4.1 - Predicting Position for more detailed explanation.

Thanks again for the suggestions. We add an explanation to the revised version in Appendix A.2.

Q2. About Post-processed Metrics.

We thank the reviewer for suggesting on reporting the post-processed metrics of CompactGS [8]. We include the table for the MipNeRF-360 [7] dataset below. To have the same setting between our work and CompactGS [8], we remove the Huffman encoding and pruning accordingly from the CompactGS [8]. As can be seen, our model achieves smaller storage and better reconstruction than CompactGS [8].

Scene	PSNR	SSIM	LPIPS	Storage (MB)		PSNR	SSIM	LPIPS	Storage (MB)
CompactGS					Ours-C2
bicycle	24.75	0.723	0.286	47.62		24.78	0.701	0.325	31.72
garden	26.77	0.831	0.161	48.46		27.38	0.842	0.156	33.01
stump	26.40	0.755	0.280	39.03		26.41	0.751	0.260	41.09
room	30.88	0.918	0.209	20.21		31.82	0.916	0.224	23.02
counter	28.68	0.901	0.205	20.43		28.75	0.894	0.218	24.04
kitchen	30.48	0.919	0.131	29.59		30.71	0.913	0.146	30.05
bonsai	32.03	0.938	0.193	21.60		32.14	0.935	0.192	23.61
Average	28.57	0.855	0.210	32.42		28.86	0.851	0.217	29.50

Q3. About Training Time.

We thank the reviewer for the suggestion and will incorporate a discussion on convergence speed in our revised paper. Leveraging the hash-grid structure increases the per-step training time compared to 3DGS [9]. For example, 3DGS [9] requires approximately 23 minutes to reach a PSNR of 30.63 at 30K steps on scene room from MipNeRF-360 [7] dataset, our method takes about 27.6 minutes, and HAC [1] requires a similar training time of 27.1 minutes. Nonetheless, our approach strikes a good balance between size, speed, and quality. Furthermore, thanks to our inference phase optimization strategy (Line 452 of main paper), the hash-grid structure has a minimal impact on inference time, allowing our approach to run in real-time on mobile devices.

Thanks again for the suggestions. In the revised version, we explain the training convergence in Appendix A.3.5.

Q4. More Explanation for Lightweight.

We thank the reviewer for the feedback. We have reported the number of parent and child splats in our Supplementary. For instance, our approach requires an average of 2.52M total splats on the MipNeRF-360 [7] dataset, compared to 3.07M splats required by 3DGS[9]. The inference overhead primarily arises from the first-frame feature queries and the MLP used for color computation during per-frame rendering (Lines 452–457). We appreciate the reviewer’s suggestion and will clarify the term "lightweight" more explicitly in our revised paper.

Q5. Comparison to Concurrent Works.

We sincerely thank the reviewer for the suggestion. While it is not necessary to compare our work with ECCV 2024 papers, we aim to address the reviewer’s concerns comprehensively. To this end, we provide quantitative comparisons between our approach and concurrent works on two unbounded scenes from the MipNeRF-360 [7] dataset. Please kindly note that the results for Compact3D [5] have already been discussed and are presented in Table 1 of our paper (Line 390), so they are not included in the comparisons below.

Metric	Compressed3D	HAC	Reduced3DGS	Ours-C3
PSNR	25.86	26.27	26.04	26.27
SSIM	0.797	0.796	0.807	0.782
LPIPS	0.192	0.199	0.170	0.225
Storage (MB)	46.85	40.14	47.50	38.60

Our approach achieves the smallest storage requirements while maintaining comparable or superior PSNR compared to various concurrent works. Notably, we require less storage than HAC [1], even though HAC [1] leverages BiRF [6] to achieve 1-bit quantization of the grid features, whereas our method uses 8-bit. These results demonstrate the efficacy of our tree representation compared to anchor-based methods. Since quantization techniques are orthogonal to our representation, we anticipate achieving an even higher compression ratio by adopting the same settings as HAC [1].

References

1. Chen et al., HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression, ECCV 2024
2. Niedermayr et al., Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis, CVPR 2024
3. Papantonakis et al., Reducing the Memory Footprint of 3D Gaussian Splatting, I3D 2024
4. Morgenstern et al., Compact 3D Scene Representation via Self-Organizing Gaussian Grids, ECCV 2024
5. Navaneet et al., Compact3D: Smaller and Faster Gaussian Splatting with Vector Quantization, ECCV 2024
6. Shin et al., Binary Radiance Field, NeurIPS 2024
7. Barron et al., Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields, CVPR 2022
8. Lee et al., Compact 3D Gaussian Representation for Radiance Field, CVPR 2024
9. Kerbl et al., 3D Gaussian Splattingfor Real-Time Radiance Field Rendering, SIGGRAPH 2023
10. Lu et al., Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering, CVPR 2024

Q5. We thank the reviewer’s kindness and willingness to raise the score. In addition to the metrics, we provide a concise discussion comparing the methodologies of existing works (e.g., Compressed3D and Reduced3DGS) and our approach.

1. Compressed3D employs a fundamentally different approach compared to existing methods in the field. Instead of integrating compression techniques during the training or reconstruction phase, it operates as a post-processing framework. This requires an already trained or reconstructed scene to begin with. Subsequently, a series of techniques, such as vector quantization (VQ) codebooks, fine-tuning, and dictionary-based compression like LZ77, are applied to achieve compression. The reliance on a fully reconstructed scene as input distinguishes it from methods that incorporate compression techniques directly into the scene's modeling process. Techniques like LZ77 are orthogonal and applicable to many representation-based approaches.
2. Reduced3DGS closely follows the methodology of 3DGS but introduces additional techniques to enhance efficiency and storage. Specifically, it leverages pruning strategies to remove unnecessary splats, thereby reducing redundancy in the representation. During optimization, it adaptively selects the appropriate bands of spherical harmonics to balance detail retention and computational cost. Following this optimization, codebook-based quantization is applied to further compress the scene representation

Our method is a representation-based approach that leverages a hierarchical representation of spats to model the spatial relationships which disjoint with post-processing based methods. More importantly, we demonstrate the real-world rendering applications on mobile devices while other methods are unclear (as not illustrated).