Locality-aware Gaussian Compression for Fast and High-quality Rendering

We sincerely appreciate your insightful comments. We have tried to respond to all these valuable concerns as below. Please note that we have provided additional figures, tables, and videos.

[W1, Q2] Complicated implementation (plan to publish code)

We do not agree that our approach is particularly more complex than previous Gaussian compression approaches. Achieving high compression performance requires considering multiple properties of the 3D Gaussian. Therefore, previous approaches such as CompGS, Compact3DGS, EAGLES, LightGaussian, SSGS and HAC also consist of several steps, including several components such as pruning, quantization, sorting, hash-grids, and arithmetic encoding.

Nevertheless, for better understanding, we provide pseudocode for the locality-aware Gaussian representation learning step of LocoGS, our main component, in Sec. A.4 in the revised appendix. We will also publicly release our source code as well as pre-trained checkpoints for benchmarking.

[W2, Q1] Potential trade-offs or limitations, (different or more challenging rendering environments. e.g. metal and lighting area)

As briefly discussed in the conclusion section in the manuscript, our method is not free from limitations. Beyond the training efficiency already mentioned in the submitted manuscript, our method has additional limitations. First, our Gaussian representation learning process requires more memory space than 3DGS due to the gradient computation of the hash grid and Gaussian attributes, which poses challenges for large-scale scenes. Further details on applying our method to large-scale scenes can be found in the global response.

Second, our method assumes a cube-shaped, object-centric scene where target objects are located around the center of the scene. Based on this assumption, we set the hyperparameters of the hash grid, and adopt the contraction strategy of Mip-NeRF 360, which may be less effective for complex structures in large-scale scenes.

Additionally, our method shares limitations with 3DGS, such as limited performance on specular surfaces, despite using a spherical harmonics-based color representation. Nevertheless, our method still performs similar to 3DGS even for such specular surfaces.

We will include a more detailed discussion in the revision.

[Q3] Performance on large-scale datasets

Please refer to the global response.

Dear Reviewer EsF7,

Thank you for your insightful feedback. We have tried to address your concerns during the discussion period. Please let us know if you have any additional questions or concerns that need further discussion. We will promptly respond to your feedback.

Best regards,
Authors

[C4] Encoding schemes

We would like to clarify that the encoding schemes in Sec. 5.2 are not our main contribution, and exploring all possible combinations of encoding schemes to find the optimal combination is beyond the scope of our paper. Instead, in our framework, we adopt commonly used compression techniques for each attribute following previous work. To compress positions, we apply octree encoding, which is the most common baseline compression method in point cloud compression approaches (e.g., D-PCC [3], SparsePCGC [4]). Quantization and entropy coding, employed for other attributes in our framework, are also one of the most basic compression techniques, which are also adopted in previous works (e.g., C3DGS, Compact-3DGS, and LightGaussian).

3. He et al., Density-preserving Deep Point Cloud Compression, CVPR 2022
4. Wang et al., Sparse Tensor-Based Multiscale Representation for Point Cloud Geometry Compression, TPAMI 2022

We leverage Morton sorting to maintain the correspondence between the position and other attributes of each Gaussian, while it also helps compression as it places nearby Gaussians close to each other in the sorted sequence. In fact, any other sorting methods that work in a similar manner can be adopted. We found negligible differences in compression performance across different sorting methods, while all of them enhance the compression performance compared to random sorting in our preliminary experiments.

We acknowledge that more optimal combinations of encoding schemes may exist. However, we stress that our main contribution lies in our locality-exploiting representation and the compression framework based on this representation. Seeking optimal combinations of encoding schemes is outside the scope of this paper.

Nevertheless, as suggested by the reviewer, we provide additional details and analyses of encoding schemes. For quantization, we use k-bit uniform quantization due to its simplicity and computational efficiency. For compressing all other data, including base color, base scale, hash grid parameters, and MLP parameters, we considered widely used general-purpose lossless compression techniques such as Zip, GZip, and NPZ as our candidates. We found no notable difference between these methods regarding compression performance, so we chose NPZ, an entropy coding-based scheme, due to its convenience.

Table R8 compares different compression schemes for different attributes. All the coding schemes in the table, except for G-PCC, are general-purpose lossless coding schemes, while G-PCC is an octree-based lossless coding scheme for point clouds. As the table shows, G-PCC outperforms all other general-purpose coding schemes for position encoding. For other data, all the general-purpose coding schemes perform similarly.

Table R8. Comparison of compression schemes. The numbers in the table are compressed sizes in MB.

Table R9 compares the performance of k-bit uniform quantization and k-means clustering, a vector quantization method. While k-means clustering is a more sophisticated algorithm that can potentially result in higher compression performance, its performance severely relies on target data distribution. The table shows that k-bit uniform quantization performs better than k-means clustering for all data despite its simplicity, potentially due to the distributions being challenging to cluster.

Table R9. Comparison between k-means clustering and k-bit uniform quantization.

We hope the details and analyses provided here help you understand our framework more clearly.

[W3] Longer training time is a limitation.

Our neural field design and dense initialization result in a longer training time than 3DGS. This is primarily because the hash-grid-based neural field incurs additional computational costs compared to the direct optimization of attributes in 3DGS. Additionally, dense initialization produces a larger number of Gaussians than COLMAP at the beginning, leading to further computational overhead. To reduce the longer training time, adopting an effective culling strategy to pre-filter invisible Gaussians or employing a more efficient neural field architecture could be viable solutions.

[W4] No demos

We prepare example rendering videos and share them in the supplementary material. We will also release the code with pre-trained checkpoints once the paper gets accepted.

[Q1] Why lower rendering quality than Scaffold-GS

Scaffold-GS extends the original 3DGS framework by adopting MLPs for higher rendering quality. Specifically, Scaffold-GS decodes view-adaptive Gaussians using the view direction as input for MLPs. Thanks to the expressive power of MLPs, Scaffold-GS outperforms 3DGS in rendering quality. However, its per-view decoding process yields a large computational overhead for rendering each view. In our work, we do not follow the extension of Scaffold-GS to use the efficient rendering process of 3DGS. This may result in a minor rendering performance gap between Scaffold-GS and ours. However, as shown in Tables 1 and 2 of the manuscript, our method achieves comparable rendering quality while offering significantly faster rendering speeds.

[Q2] Performance on large-scale datasets

Please refer to the global response.

[W2] Comparison of dense initialization methods (Nerfacto and 3DGS)

We use Nerfacto because Nerfacto requires a shorter training time than 3DGS for the same number of iterations and provides more accurate geometry information. Regarding the training time, Nerfacto reports that it takes 2 minutes for 5K iterations and 30 minutes for 70K iterations on a single A5000 GPU. On the other hand, Compact-3DGS reports that 3DGS takes 12 to 28 minutes for 30K iterations on a single A6000 GPU.

For a clear comparison, we evaluated both methods in the same environment (RTX 3090). Table R6 reports the training times of Nerfacto and 3DGS on different scenes from the Mip-NeRF 360, Tanks and Temples, and Deep Blending datasets. We used 30K iterations for both methods, utilizing the publicly released codes with default settings. Note that 3DGS requires its own initialization, so we initialized it using COLMAP. On the other hand, Nerfacto does not require such an additional initialization and starts with random initialization. The table compares only the training times without initialization, and as shown, Nerfacto takes around half the training time of 3DGS.

Table R6. Evaluation of training time for 30K iterations.

Additionally, we found that, while Nerfacto typically produces results with lower rendering quality than 3DGS, it generates more accurate geometry information thanks to the regularization terms in its training loss function, which helps our training process achieve more accurate Gaussian representation learning. Table R7 compares the final rendering qualities of LocoGS initialized with 3DGS and Nerfacto. As shown in the table, LocoGS with Nerfacto achieves higher rendering quality, faster rendering speeds, and fewer Gaussians, thanks to the better initial solutions provided by Nerfacto.

Table R7. Performance of LocoGS according to the initialization method.

We sincerely appreciate your insightful comments. We have tried to respond to all these valuable concerns as below. Please note that we have provided additional figures, tables, and videos.

[W1] Dense initialization is not related to compression. It would be unfair...

Although dense initialization is not directly related to compression, it is especially effective for compression when combined with compression techniques such as pruning and quantization. Thus, we proposed the dense initialization scheme as one component of our framework.

Fig. R2 and Table R3 present comparisons against previous methods with dense initialization. Simply applying dense initialization to previous methods increases the number of Gaussians, leading to higher rendering quality, slower rendering speed, and larger storage size, as shown in Fig. R2(b) and Table R3. Fortunately, some compression techniques provide parameters for compression, such as pruning strength and quantization strength. By tuning such parameters, we can maintain the storage size even when using dense initialization. Fig. R2(c) and Table R3 show that dense initialization with compression parameter tuning can enhance the rendering quality while maintaining the rendering speed and storage size for most compression methods.

More importantly, Fig. R2 and Table R3 show that our method still has distinctive advantages over existing methods with dense initialization and compression parameter tuning. Compared to them, our approach produces smaller-sized results while providing higher rendering speed and comparable rendering quality, thanks to our locality-aware approach. Please refer to the response for W2 of reviewer 6k9A for more details. We will include the comparisons in the revision.

[Link] Fig. R2. Evaluation of previous approaches applying dense initialization.

[Q2] Design of explicit attributes

The explicit attributes are selected based on the purpose of optimization as described in L229-L233, not based on their local coherence. The optimization process of 3DGS, which involves explicit initialization of the Gaussian attributes and the adaptive growing strategy that directly adjusts the scale and base color of each Gaussian primitive, necessitates some of the Gaussian attributes to be stored in an explicit form for each Gaussian. Therefore, we define the explicit attributes as the minimal set of attributes that need to be stored in an explicit form (position, base scale, and base color) while defining the implicit attributes as the remaining attributes that can be stored in a compact neural field.

To validate the optimality of our current selection of explicit attributes, we compared the performance of LocoGS when the base color and base scale are treated as either explicit or implicit attributes. Table R5 summarizes the comparison results. As the table shows, treating either the base scale or base color as implicit attributes degrades performance, as they cannot be explicitly initialized or manipulated during the training process. Specifically, treating the base scale as an implicit attribute makes the training process highly unstable, resulting in a severely small number of Gaussians and poor rendering quality. Treating the base color as an implicit attribute also degrades the rendering quality while increasing the number of Gaussians. Thus, we selected them as explicit attributes along with positions.

Table R4. Ablation study on the attribute design.

We sincerely appreciate your insightful comments. We have tried to respond to all these valuable concerns as below. Please note that we have provided additional figures, tables, and videos.

[Q1] Cosine similarity of opacities

As the opacity of each Gaussian is a scalar value, it is unsuitable for computing cosine similarity. Therefore, to evaluate the local coherence of the opacity values, we constructed vectors from the opacity values of nearby Gaussians and computed the cosine similarities between them. The other Gaussian attributes are vectors, so we directly computed their cosine similarities. We acknowledge that this approach may seem unintuitive and that cosine similarity may not be suitable for all Gaussian attributes. Consequently, we plan to replace cosine similarity with Euclidean distance in the revision. Fig. R3 presents analyses using the Euclidean distance. The top row in Fig. R3 shows histograms of the Euclidean distances between two Gaussians for each attribute. Following our submitted manuscript, the pink histograms represent spatially adjacent Gaussian pairs, while the yellow histograms represent spatially distant Gaussian pairs. Regardless of the spatial distances, all histograms have high peaks at small Euclidean distances, while their shapes become flatter as the spatial distances between Gaussians increase. The bottom row in Fig. R3 shows the average distances between two Gaussians for each attribute. Similarly, the pink bars represent spatially adjacent Gaussians, and the yellow bars represent spatially distant Gaussians. The graphs demonstrate that spatially adjacent Gaussians have smaller Euclidean distances between their attributes. These analyses clearly indicate the local coherence of the Gaussian attributes.

[Link] Fig. R3. Evaluation of the local coherence of Gaussian attributes. We measure the Euclidean distance of neighboring Gaussian attributes.

We appreciate the reviewer’s feedback. To highlight the novelty and effectiveness of our locality-aware strategy, we compare our method with previous approaches, emphasizing the differences in locality utilization below. We will revise the paper to make these differences more apparent.

We may categorize existing 3D Gaussian compression approaches into global-statistic-based approaches, and locality-exploiting approaches. Global-statistic-based approaches include quantization-based approaches (CompGS, LightGaussian, and EAGLES), and an image-codec-based approach (SSGS). These approaches do not exploit the local coherence of Gaussian attributes, but they rely on global statistics of Gaussian attributes for quantization, and for image-codec-based compression.

Locality-exploiting approaches include C3DGS, Compact-3DGS, and anchor-based approaches. However, these approaches restrictively exploit locality and achieve limited performance compared to ours.

Specifically, C3DGS sorts Gaussian attributes according to their positions along a space-filling curve. This sorting process places spatially close Gaussians in the 3D space at nearby positions in a sorted sequence, enabling subsequent compression steps to exploit the local coherence of Gaussians. However, projecting 3D Gaussians to a 1D sequence along a space-filling curve cannot fully leverage the local coherence of Gaussian attributes. In contrast, our method directly encodes local information in an effective structure, a multi-resolution hash grid, during optimization. Therefore, our framework can more effectively exploit local coherence in the 3D space.

Compact-3DGS leverages a hash grid to encode view-dependent colors. However, for geometric attributes including rotation and scale, it uses vector quantization, which exploits the global statistics instead of local coherence. As a result, while Compact-3DGS achieves noticeable compression performance with an 81.8x compression ratio for color, it achieves only a 5.5x compression ratio for vector-quantized attributes. In contrast, we reveal that geometric attributes also exhibit local coherence in our analysis in Sec. 4.1. Based on this analysis, we propose a locality-aware framework that exploits local coherence not only for view-dependent color but also for geometric attributes, achieving superior compression performance.

Anchor-based methods like Scaffold-GS and HAC also exploit local coherence for Gaussian attributes. Scaffold-GS uses an anchor-point-based representation for high-quality rendering, where each anchor point encodes multiple locally-adjacent Gaussians with attributes such as shapes and colors changing based on the viewing direction. However, Scaffold-GS focuses on rendering quality rather than compression and requires computationally expensive per-view processing. HAC builds on Scaffold-GS, incorporating a hash grid for compression. However, unlike our method, HAC encodes only the context of anchor point features using a hash grid. Also, despite its high rendering quality and compression ratio, HAC requires slow per-view rendering due to its anchor-point-based representation.

[Q1] Ablation on the number of Gaussians

Thank you for your thoughtful feedback. We have added the number of Gaussians in Table 3 in the revised manuscript.

[Q2] Decoding time

Table R4 compares the decoding times of different compression methods. LocoGS shows comparable decoding times despite its highly compact storage sizes compared to the other approaches, requiring less than 9 seconds for a single scene.

Table R4. Evaluation of the decoding time (sec).

[W2] Dense initialization for previous approaches

Simply applying dense initialization to previous methods results in an increase in the number of Gaussians, leading to higher rendering quality. However, this also causes an increase in storage size and a decrease in rendering speed. To avoid the increase in the number of Gaussians, we can adjust the compression parameters of the existing compression methods, such as pruning strength and quantization strength. When properly combined with such parameter tuning, dense initialization can still enhance the performance of these existing compression methods regarding rendering speed, storage requirement, and rendering quality since dense initialization provides a better initial solution than COLMAP-based initialization.

To validate this, we conducted an additional evaluation comparing the performance of existing approaches without dense initialization, with dense initialization but no compression parameter tuning, and with dense initialization and compression parameter tuning on the MipNeRF-360 dataset. Fig. R2 and Table R3 summarize the evaluation results. In this evaluation, we also included two non-compression methods, 3DGS and Scaffold-GS, which have no compression parameters, so we did not perform compression parameter tuning for them. As shown in the figure and table, dense initialization without compression parameter tuning increases the number of Gaussians and storage size, and it decreases the rendering speed for all methods. On the other hand, dense initialization with compression parameter tuning enhances rendering speed and storage requirements while achieving comparable or higher rendering quality for most existing approaches. It is worth mentioning that dense initialization with compression parameter tuning does not perform well with HAC, as shown in Table R3, because its anchor-based approach does not directly encode individual Gaussians, thus lacks a pruning mechanism to reduce the number of Gaussians.

Finally, Fig. R2(c) shows that our approach still has distinctive advantages over existing compression methods with dense initialization and compression parameter tuning. Compared to them, our approach produces smaller-sized results while providing higher rendering speed and comparable rendering quality. This result again proves the effectiveness of our locality-aware approach.

[Link] Fig. R2. Evaluation of previous approaches applying dense initialization.

Table R3. Ablation study on dense initialization. * denotes applying dense initialization and † denotes applying dense initialization with stronger compression parameters.

We sincerely appreciate your insightful comments. We have tried to respond to all these valuable concerns as below. Please note that we have provided additional figures, tables, and videos.

[W1] Number of Gaussians

We report the number of Gaussians of each variant in Table R2. Compared to 'Ours' and 'Ours-Sparse (Ours w/o dense initialization)', 'Ours-Small' has fewer Gaussians, resulting in higher rendering speed and slight degradation in rendering quality. In our framework, the rendering speed is inversely proportional to the number of Gaussians. While the rendering quality is influenced by the number of Gaussians, as shown in the table, it is also affected by other factors, such as dense initialization.

Table R2. Evaluation of rendeing quality, storage size, rendering speed, and number of Gaussians.

We also provide the numbers of Gaussians along the training iterations in Fig. R1. In the figure, 'Ours' and 'Ours-small' begin with dense initialization. Thus, at the beginning, they have more Gaussians than 3DGS and 'Ours (w/o dense init.)'. We prune Gaussians for the entire optimization process (30K), whereas 3DGS preserves a large number of Gaussians after densification (15K). Therefore, we can achieve a small Gaussian quantity after optimization. 'Ours-Small' uses a larger weight for the mask loss, so it has fewer Gaussians than 'Ours' after training.

[Link] Fig. R1. Number of Gaussians along iterations.

Dear Reviewer 6k9A,

Thank you for your insightful feedback. We have tried to address your concerns during the discussion period. Please let us know if you have any additional questions or concerns that need further discussion. We will promptly respond to your feedback.

Best regards,
Authors