Fast Feedforward 3D Gaussian Splatting Compression

W1: Figure for performance comparison.

Thanks for pointing this out. Using curves to compare compression performance (i.e., RD performance) is a widely adopted practice in the compression field [1][2]. Curves provide an intuitive visualization of the relative performance variances among different compression approaches, where a curve closer to the upper-left indicates better performance. Meanwhile, Table C to Table E in Appendix Section F serves as a necessary supplement, presenting precise numerical data for a detailed comparison. In future versions, we will strive to incorporate more quantitative data into the main paper without significantly changing the current content.

W2: Running time of 3DGS.

Thanks for your comments. We have now added a parenthesis indicating the training time of 3DGS in front of the time consumption of finetuning-based approaches and our approach in Tables C to Table E.

W3: Comparison with SoTA methods.

Since we configure FCGS without per-scene optimization, this setup naturally puts it at a disadvantage when compared with optimization-based methods, and indeed, there is a performance gap compared to SoTA methods like HAC [3] and ContextGS [4]. However, these two approaches employ anchor-based structures [5], whose representations diverge from the standard 3DGS structure and are inherently about 5X smaller than the vanilla 3DGS. In this paper, we target FCGS at the standard 3DGS structure, same as the comparative methods in our experiment. It is also promising to extend FCGS to anchor-based 3DGS variants [5] to achieve better RD performance, which we leave for future work. We have added this discussion to Appendix Section D and pointed out the SoTA of per-scene optimization-based 3DGS compression methods.

Q1: Regarding $MLP_m$ and Figure 2.

Thanks for pointing this out.

• We have now re-formulated Equation (3) to clarify the process of generating $m$ using $MLP_m$ and updated it in the paper. The updated formulation is as follows: $m_i = \text{Sig}(MLP_m(f^\text{gau}_i)) > \epsilon_m,$where $\epsilon_m$ is a hyperparameter that defines the threshold for binarizing the mask, and gradients of the binarized mask $m$ are backpropagated using STE [6][7].
• We have also improved Figure 2 to show the process of generating $m$ by $MLP_m$, and also make the pathway of the mask $m$ easier to understand.

Please find them in our revised paper.

If you have any further questions or concerns, please feel free to reach out to us for discussion.

Reference:

[1] Ballé J, Minnen D, Singh S, et al. Variational image compression with a scale hyperprior [C]. ICLR 2018.

[2] He D, Zheng Y, Sun B, et al. Checkerboard context model for efficient learned image compression [C]. CVPR 2021.

[3] Chen Y, Wu Q, Lin W, et al. HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression [C]. ECCV 2024.

[4] Wang Y, Li Z, Guo L, et al. ContextGS: Compact 3D Gaussian Splatting with Anchor Level Context Model [C]. NIPS 2024.

[5] Lu T, Yu M, Xu L, et al. Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering [C]. CVPR 2024.

[6] Bengio Y, Léonard N, Courville A. Estimating or propagating gradients through stochastic neurons for conditional computation[J]. arXiv preprint, 2013.

[7] Lee J C, Rho D, Sun X, et al. Compact 3d gaussian representation for radiance field[C]. CVPR 2024.

Thanks for your response. This has addressed most of my concerns. I will maintain my positive rating.

Q3: Number of scenes to acquire a feasible compression performance.

We design FCGS to train on massive 3DGS representations to acquire sufficient prior information for compression, enabling it to conduct a feedforward-based compression using the learned prior information. Although preparing this training data is time-intensive, it is a one-time investment—once trained, FCGS can be directly applied for compression without requiring further optimization, making the effort highly valuable. To evaluate FCGS’s performance when trained on significantly less data, we conducted experiments using only $100$ scenes, as shown in the table below. Even with this reduced training set, FCGS still delivers strong results, achieving comparable fidelity metrics with only $12.9$% increase in size. This robustness is due to FCGS’s compact and efficient architecture, with a total model size of less than $10$ MB. Discussions of this evaluation has been added to Appendix Section L.

Table 8: Results of training with less data. Experiments are conducted on the DL3DV-GS dataset with $\lambda=1e-4$.

Q4: Regarding ContextGS.

Thanks for your advice. ContextGS proposes structuring anchors into multiple levels to model entropy. However, its anchor location quantization process introduces deviations, which impacts fidelity and requires finetuning. Differently, our inter-Gaussian context model overcomes this limitation by creating grids in a feed-forward manner for context modeling, without modifying Gaussian locations, eliminating the need of finetuning. We have now added this discussion to Line 262-266.

Q5: Open-source of the DL3DV-GS dataset.

The DL3DV-GS dataset will be open-source. However, the DL3DV-GS dataset is derived from DL3DV, which requires access permissions. To facilitate open-source access to DL3DV-GS, we will coordinate with the DL3DV authors to establish a legal and compliant method for release.

If you have any further questions or concerns, please feel free to reach out to us for discussion.

Q2: storage size of each attribute.

Thanks for your suggestion. We now provide a detailed analysis of the storage size for each attribute in Appendix Section G. This includes two tables: one showing the total storage size of each attribute and the other detailing the per-parameter bits, offering a comprehensive breakdown. These two tables also allow for the calculation of the compression ratio for each component before and after compression. With increasing $\lambda$, the sizes of both color and geometry attributes decrease. Within the color attributes, the conditions for paths $m=1$ and $m=0$ differ due to the rising mask rate as $\lambda$ increases. A more comprehensive analysis is available in Appendix Section G.

Table 6: Storage size of different components on the DL3DV-GS dataset. All sizes are measured in MB. The mask rate indicates the proportion of color attributes $f^{\text{col}}$ assigned to the $m=1$ path (i.e., compressed via the autoencoder).

Table 7: Per-parameter bits of different components on the DL3DV-GS dataset. All sizes are measured in bit. The mask rate indicates the proportion of color attributes $f^{\text{col}}$ assigned to the $m=1$ path (i.e., compressed via the autoencoder).

W3&Q1: Regarding the coding efficiency of arithmetic coding.

Thanks for your suggestion. We have implemented a CUDA-based arithmetic codec which is fast. The total time for encoding is presented in Figure 4. In Appendix Section H, we now provide a comprehensive analysis of both the encoding and decoding time of each single component. Here are the two tables in Appendix Section H, which indicate Encoding time and Decoding time, respectively. During encoding, the GPCC takes most of the time to compress Gaussian locations due to its complex RD search process. During decoding, the model is much faster. Please refer to Appendix Section H for more details.

Table 4: Encoding time of different components on the DL3DV-GS dataset,which has over $1.57$ million Gaussians on average in the testing set. All times are measured in seconds, with each component’s percentage share of the total encoding time provided in parentheses. The mask rate indicates the proportion of color attributes $f^{\text{col}}$ assigned to the $m=1$ path (i.e., compressed via the autoencoder). Results are obtained using a single GPU.

Table 5: Decoding time of different components on the DL3DV-GS dataset, which has over $1.57$ million Gaussians on average in the testing set. All times are measured in seconds, with each component’s percentage share of the total decoding time provided in parentheses. The mask rate indicates the proportion of color attributes $f^{\text{col}}$ assigned to the $m=1$ path (i.e., compressed via the autoencoder). Results are obtained using a single GPU.

W1: Show relations of Gaussians.

Thanks for your suggestion. To show relations of Gaussians, we visualize bit allocation conditions of both inter- and intra-Gaussian context models. Please refer to Appendix Section J for a comprehensive analysis. Below, we provide an example table of the result, which shows per-parameter bit allocation conditions of color attributes ($m=1$) over different parts using context models. This table highlights the effectiveness of the context models.

• For the inter-Gaussian context model, from a statistical perspective, each Gaussian in different batches is expected to hold similar amounts of information on average due to the random splitting strategy, which ensures an even distribution of Gaussians across the 3D space. Thus, as batches progress deeper (i.e., B1 $\rightarrow$ B4), the bits per parameter decrease. The most significant reduction occurs between B1 and B2, since B1 lacks inter-Gaussian context, leading to less accurate probability estimates. For subsequent batches, the reduction is less pronounced, as B1 already includes $1/6$ of the total Gaussians, providing sufficient context for accurate prediction. Adding more Gaussians from later batches yields diminishing benefits. This finding validates our approach of splitting batches with varying proportions of Gaussians, where the first two batches contain fewer Gaussians.
• For the intra-Gaussian context model, different from that in the inter-Gaussian context model, the information distribution among different chunks is not guaranteed to be equal, as the latent space is derived from a learnable MLP. As a result, C1 receives the least information because it lacks intra-Gaussian context, leading to less accurate probability predictions. Therefore, allocating more information to C1 would make entropy reduction challenging. To counter this, the neural network compensates by assigning minimal information to C1, thereby saving bits. Conversely, C2 receives the most information as it plays a pivotal role in predicting subsequent chunks (i.e., C3 and C4), while its own entropy can be reduced by leveraging C1. As a result, the bits per parameter decrease consistently from C2 to C4 due to the increasingly rich context.

A full comprehensive analysis of more tables and discussions covering all attributes (color and geometry) can be found in Appendix Section J.

Table 1: Per-parameter bit consumption of color attributes ($m$=1) over different parts using context models. Values are scaled by 100x for readability. B1 $\rightarrow$ B4 and C1 $\rightarrow$ C4 indicate batches and chunks of inter- and intra-Gaussian context models, respectively.

W2: Ablation over the strategy for dividing chunks.

Thank you for your suggestion. We have conducted ablation studies on the split-dividing strategies for both inter- and intra-Gaussian context models and included a detailed analysis in Appendix Section I. Increasing the number of splits does not consistently lead to improved performance and may increase the model’s complexity. Our experiments confirm that the default splitting strategy strikes a good balance between performance and complexity. For a more comprehensive discussion, please refer to Appendix Section I. Below are the tables of the ablation studies for both inter- and intra-Gaussian context models.

Table 2: Ablation study on the number of splits in the inter-Gaussian context model on the DL3DV-GS dataset with $\lambda=1e-4$.

Table 3: Ablation study on the number of splits in the intra-Gaussian context model on the DL3DV-GS dataset with $\lambda=1e-4$.

Q1: Visualization of inter- and intra-Gaussian context models.

For visualization of inter- and intra-Gaussian context models, please refer to Appendix Section J for a comprehensive analysis. Below, we provide an example table of the result, which shows per-parameter bit allocation conditions of color attributes ($m=1$) over different parts using context models. This table highlights the effectiveness of the context models.

• For the inter-Gaussian context model, from a statistical perspective, each Gaussian in different batches is expected to hold similar amounts of information on average due to the random splitting strategy, which ensures an even distribution of Gaussians across the 3D space. Thus, as batches progress deeper (i.e., B1 $\rightarrow$ B4), the bits per parameter decrease. The most significant reduction occurs between B1 and B2, since B1 lacks inter-Gaussian context, leading to less accurate probability estimates. For subsequent batches, the reduction is less pronounced, as B1 already includes $1/6$ of the total Gaussians, providing sufficient context for accurate prediction. Adding more Gaussians from later batches yields diminishing benefits. This finding validates our approach of splitting batches with varying proportions of Gaussians, where the first two batches contain fewer Gaussians.
• For the intra-Gaussian context model, different from that in the inter-Gaussian context model, the information distribution among different chunks is not guaranteed to be equal, as the latent space is derived from a learnable MLP. As a result, C1 receives the least information because it lacks intra-Gaussian context, leading to less accurate probability predictions. Therefore, allocating more information to C1 would make entropy reduction challenging. To counter this, the neural network compensates by assigning minimal information to C1, thereby saving bits. Conversely, C2 receives the most information as it plays a pivotal role in predicting subsequent chunks (i.e., C3 and C4), while its own entropy can be reduced by leveraging C1. As a result, the bits per parameter decrease consistently from C2 to C4 due to the increasingly rich context.

A full comprehensive analysis of more tables and discussions covering all attributes (color and geometry) can be found in Appendix Section J.

Table 2: Per-parameter bit consumption of color attributes ($m$=1) over different parts using context models. Values are scaled by 100x for readability. B1 $\rightarrow$ B4 and C1 $\rightarrow$ C4 indicate batches and chunks of inter- and intra-Gaussian context models, respectively.

Q2: Open-source of the code and dataset.

Both the code and the DL3DV-GS dataset will be open-source. The DL3DV-GS dataset is derived from DL3DV, which requires access permissions. To facilitate open-source access to DL3DV-GS, we will coordinate with the DL3DV authors to establish a legal and compliant method for release.

If you have any further questions or concerns, please feel free to reach out to us for discussion.

Reference:

[1] Chen Y, Xu H, Zheng C, et al. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images[C] ECCV 2024.

W1: Amount of the Training data.

We design FCGS to train on massive 3DGS representations to acquire sufficient prior information for compression, enabling it to conduct a feedforward-based compression using the learned prior information. Although preparing this training data is time-intensive, it is a one-time investment—once trained, FCGS can be directly applied for compression without requiring further optimization, making the effort highly valuable. To evaluate FCGS’s performance when trained on significantly less data, we conducted experiments using only $100$ scenes, as shown in the table below. Even with this reduced training set, FCGS still delivers strong results, achieving comparable fidelity metrics with only $12.9$% increase in size. This robustness is due to FCGS’s compact and efficient architecture, with a total model size of less than $10$ MB. Discussions of this evaluation has been added to Appendix Section L.

Table 1: Results of training with less data. Experiments are conducted on the DL3DV-GS dataset with $\lambda=1e-4$.

W2: Analysis of failure cases.

Thanks for pointing this out. Failure cases and limitations of FCGS had been discussed in Appendix Section B. We now add indicators in the main paper to this content and also polish it for clearer expression. Failures mainly arise from incorrect selections of $m$ by MEM due to the domain gap: a Gaussian that should have been assigned to the $m=0$ path for fidelity preservation may be mistakenly assigned to $m=1$, and the autoencoder in this path fails to recover these Gaussian values properly from their latent space, leading to significant fidelity degradation. This issue becomes noticeable for 3DGS from feed-forward models such as MVSplat [1], due to a distribution gap between our training data (3DGS from optimization) and those from feed-forward models like MVSplat [1]. We mitigate this problem by enforcing all Gaussians to the $m=0$ path for compressing 3DGS from feed-forward models, thereby prioritizing fidelity. For instance, to compress 3DGS from MVSplat, $m$ deduced from MEM may not be correct, which would result in a PSNR drop of approximately $0.7$dB at a slightly smaller compression size compared to setting all $m$ as 0s. Overall, this is our major limitation, which points out an interesting problem for future work: how to design optimization-free compression models that can be well generalized to 3DGS with different value characteristics. Please refer to Appendix Section B for a more detailed analysis.

W1: Complexity of the method.

Our method is well structured for compression, which consists of both MEM and context models to achieve efficient compression. MEM selectively projects Gaussians to a latent space to balance the fidelity and size; While we adapt the common idea of context models in traditional data compression to 3DGS compression, which uses decoded parameters as context to predict remaining parameters. Both MEM and context models are critical and serve for different purposes to achieve a good compression performance, as stated by Reviewer 1 (xSne): “It introduces a Multi-path Entropy Module (MEM) to adaptive control the compression size and quality. It also designs inter- and intra-Gaussian context models to enhance compression efficiency.” Moreover, the batch dividing strategy has been stated in Line 370 in the implementation details, and additional ablation studies on various splitting strategies for our context models are presented in Appendix Section C and Appendix Section I. Overall, rather than adding complexity, these two components are methodically organized to achieve effective compression performance.

W2: Compression speed versus PSNR/size.

As discussed in Line 82-88, the two pipelines of per-scene optimization-based compression (ours) and generalizable optimization-free compression (others) cater to distinct use cases. Selecting the appropriate compression pipeline depends on specific application requirements. For scenarios where compression speed is less critical (i.e., your example), such as offline processing, optimization-based compression can be advantageous. However, FCGS supports the compression of 3DGS from both optimization and feed-forward models, as demonstrated in Subsection 4.2. Moreover, generalizable optimization-free compression offers many other advantages:

• Independence from ground-truth supervision: Unlike optimization-based approaches that rely on costly, image-based guidance, optimization-free methods operate directly on 3DGS representations without the need for such supervision.
• Speed sensitivity: This is crucial for 3DGS generated by feed-forward models. Such models produce 3DGS rapidly, and applying per-scene optimization compression in these 3DGSes incurs significant computational and time overhead.
• Suitability for lightweight devices: Devices like mobile phones lack the computational power for resource-intensive per-scene optimization.

Overall, our FCGS can operate in all situations and its advantages become more pronounced when compressing 3DGS generated by feed-forward models.

W3: More qualitative evaluation compression to other baselines.

Thank you for your suggestion. We have incorporated additional baseline methods for qualitative comparison:

• Figure 5 in the main paper now includes qualitative results of the baselines.
• Figure E in Appendix Section N further extends these comparisons.

These updates can be found in the revised version of the paper. Additionally, we have added a note in the caption of Figure 5 to guide readers to Figure E in Appendix Section N for a more detailed qualitative comparison.

Q1: Analysis transform and synthesis transform.

They are implemented as simple MLPs: $g_a$ and $g_s$ consist of 4 layers each, while $h_a$ and $h_s$ have 3 layers each. This description has now been added to the caption of Figure 2.

Q2: Potential nonlinearity of MLPs.

The forward pass of MLPs is inherently non-invertible due to non-linear activation functions, which means that projecting an input (i.e., $x$) into latent space (i.e., $y$) via an MLP does not allow for exact recovery of the original input from the latent. As a result, MLP-decoded attributes (i.e., $\hat{x}$) cannot always align precisely with the original attributes (i.e., $x$) for each Gaussian. This misalignment would be amplified in rendering, leading to a significant fidelity drop in the rendered images. We have clarified and polished the writing in Line 221-226.

Q3: Split of Gaussians.

The split criterion is in the implementation details (Line 370), where we utilize a random split with 4 steps. Note that a consistent randomness is applied between the encoder and decoder to ensure accurate decoding. Importantly, this randomness minimally impacts performance—experiments with different seeds yielded similar results, as discussed in Appendix Section C. We also tested Farthest Point Sampling (FPS) for splitting but found that it underperformed our random splitting, likely because FPS does not distribute samples evenly. Furthermore, as discussed in Appendix Section I, we explored various numbers of splits; While more steps bring slight benefits, 4 steps already achieve strong performance.

If you have any further questions or concerns, please feel free to reach out to us for discussion.