We sincerely thank you for your thorough review and valuable suggestions

"The primary contribution to compression is adopted from Compact-3DGS"

Our method is significantly different from Compact-3DGS. We only adopt the masking loss from Compact-3DGS. We also have significantly better performance than Compact-3DGS, i.e., 3.5dB improvements using only 16.9% bitstream measured on Bungeenerf.

While Compact-3DGS used entropy coding, they only used entropy coding as a post-processing step. Specifically, they use Huffman coding to encode the index after vector quantization (VQ) when completing training. As a comparison, we optimize the estimated entropy of the 3D scenes by learning the distribution of anchors during training.

To demonstrate the difference between Compact-3DGS and the effectiveness of the proposed method, we set the weight of the masking loss used in Compact-3DGS in Eq. 10 to $0$, and the results are as follows:

Since the entropy loss is trained in an end-to-end manner, we can achieve a similar or even better rate-distortion trade-off compared with using the masking loss proposed in Compact-3DGS in our framework. This strongly highlights the difference between our end-to-end entropy framework and the Huffman coding as a post-process in Compact-3DGS.

4MB Improvement

The improvement from 18.67MB to 14MB is indeed a significant improvement. As illustrated in the highlight of the summary of the rebuttal, the relative improvements depend on the selection of the baseline. It is worth noting that while HAC and the proposed method both use Scaffold-GS as the backbone model, the baseline model we used for the ablation study is much stronger than that used by HAC, as shown in the following table. Our well-designed baseline model can achieve much better rendering quality with almost the same size as HAC. On top of the SOTA method/such a strong baseline, a 25% performance improvement is already very significant (as mentioned previously, there is only ~11% accumulated improvement over 3 years for image compression).

"Specifically customized to Scaffold-GS"

The core idea of the proposed context model is not limited to ScaffoldGS, as it does not involve new data formats and does not make any assumptions about the basic elements representing the 3D scene, such as anchors or 3D Gaussians. The context model aims to explore the relationship among existing elements by predicting the probability distribution of the current context based on given decoded contexts. In principle, it could also be applied to vanilla 3DGS or new backbones in the future, which may be left for future exploration.

Teaser figure

Yes, figures (b, c, d) in the teaser figure are all from the proposed ContextGS.

Our method does not aim to reduce the similarity between different levels, i.e., whether using the proposed context model does not significantly affect the similarity among different levels of anchors.

In fact, we need to use the high similarity to better model the relationship among two levels, i.e., predicting the probability distribution of the current context based on given decoded contexts. Usually, the higher similarity between the current context and decoded contexts leads to more significant improvements from the context model as follows.

• Higher similarity $\rightarrow$ More accurate distribution modeling $\rightarrow$ Reduced entropy (bitstream)

Fig. (c) aims to visualize the similarity to illustrate the feasibility of implementing the context model in 3DGS. It shows that many anchors in level 0 have highly similar or almost the same features as their corresponding anchors in a coarser level.

The detailed explanations are as follows:

• The relationship between similarity and redundancy. Accurately speaking, high similarity is a prerequisite but not a sufficient condition for context modeling. Fig. (c) shows that although the representation of a 3D scene is sparse, similar to natural images where flat areas (areas with similar values) exist that make the context modeling work, 3D scenes also have flat areas that make the context modeling possible.

• Visualization of the bit-saving map. The bit saving is calculated by estimating the number of bits to encode each anchor with and without using the information of already coded anchors from coarser levels (similar to previous works that calculate the bit-saving maps, e.g., Fig. 4 in [R1]).

• The anchors of which storage costs are reduced are those who have high feature similarities with their corresponding anchors in the coarser levels and are difficult to directly represent using existing hyperprior features. This is the reason why the bit-saving map in Fig. (d) and the similarity map in Fig. (c) do not necessarily match exactly.

We have enlarged the point size of anchors in coarser levels to make them more visible since they only occupy a small portion. However, it is still challenging to avoid the anchors being small and chaotic due to the large number of anchors representing a scene.

[R1] Checkerboard Context Model for Efficient Learned Image Compression, CVPR 2020

Typos and Neural Gaussian

We use "neural Gaussian" to convey their differentiable nature. We follow the symbols and definitions in ScaffoldGS, which mixes the use of "3D Gaussian" and "Neural Gaussian." To avoid confusion, we will unify it to "3D Gaussian" in the revised paper

Dear Reviewer 8ybR,

Thank you for your valuable feedback, which has significantly improved our submission.

Regarding your concerns about customizing to Scaffold-GS, we are conducting experiments on various backbones, including vanilla 3DGS. Preliminary results, attached below, show huge improvements over the most recent SOTA methods on different backbones.

We will provide more detailed results soon.

Best regards,
Authors

Table R1: Performance of the proposed method on the vanilla 3DGS backbone.

Table R2: Performance of the proposed method on the Compact-3DGS backbone (CVPR'24 Oral). Unlike the vanilla 3DGS, Compact-3DGS uses a small MLP to predict the colors of 3D Gaussians.

Dear Reviewer 8ybR,

Attached are the updated full results on the BungeeNeRF dataset, demonstrating the generalizability of our method on different backbones. On different backbones, we all achieve huge improvements.

Table R1: The performance of the proposed method on the vanilla 3DGS backbone.

Table R2: The performance of the proposed method on the backbone used by Compact3DGS.

We sincerely thank you for your review and valuable suggestions on our paper.

Typos

Thank you for pointing out the typo; we have corrected it.

Ablation of Different Number of Levels

Thank you for your suggestions. We conducted an ablation study on the Rome-Bungeenerf dataset without using the learnable hyperprior. The results are shown in the following table:

We find that decreasing the level number to 2 leads to an obvious degradation in performance. However, increasing the level number slightly increases the storage cost. We observe that increasing the level number does not significantly enhance the compression rate of anchors while it increases the size of MLPs by approximately 0.1MB for the new level.

Notations

Thank you for pointing out the duplicate use of $\hat{\mathbf{V}}$. We will use $\tilde{\mathbf{V}}$ instead to avoid confusion.

Without Encoding Anchors in Table 5 (Appendix)

"Coding" in Table 5 refers to using entropy coding techniques to encode the anchor positions, i.e., the detailed results of (w/ APC) in Table 4. We do not encode the anchor positions in the main paper because, as shown in Table 4, it significantly slows down the coding speed.

Specifically, we find that the anchor position is very important and difficult to compress. Through training with adaptive quantization width, we find the anchor position requires high numerical precision for storage. This results in a limited compression ratio and a slow coding speed due to the large number of symbols required for Arithmetic (entropy) Coding.

Since anchor positions only occupy approximately 15% of the storage space and are crucial for rendering, compressing them sometimes does not contribute to significant improvements in performance.

Dear Reviewer gvyt,

We sincerely thank you for your support and recognition of our work.

Regarding the concerns raised by other reviewers about the dependency on Scaffold-GS, we are pleased to share our new results demonstrating the effectiveness of our method on other backbones, such as vanilla 3DGS. By utilizing our proposed end-to-end entropy optimization and context model, we achieved impressive performance on these backbones, as shown in the attached tables. We will release the models based on both Scaffold-GS and vanilla 3DGS upon acceptance.

Thank you again for your time and valuable insights.

Sincerely,

Authors

The results of the following tables are measured on BungeeNeRF dataset.

Table R1: The performance of the proposed method on the vanilla 3DGS backbone.

Table R2: The performance of the proposed method on the backbone used by Compact3DGS.

We deeply appreciate your thorough review and valuable feedback on our submission. Here are our detailed responses to your comments and suggestions:

Performance Improvement

As illustrated in the summary of the rebuttal, we argue that the proposed main components indeed bring significant improvements.

Any method of compression with SOTA performance relies on entropy coding, and most of them contribute to the improvement of entropy coding as well. Different papers use various entropy-based backbones as the baseline. Our entropy baseline (i.e., Ours w/o HP w/o CM) is well-designed with very strong performance. The baseline method we used in the ablation study even outperforms the concurrent work HAC (ECCV'24).

A 25% improvement on such a strong baseline is already significant. If we use a plain baseline entropy-based method like the one used in HAC (removing the additionally used anchor position as the prior), the size is ~30 MB, with around a 50% performance improvement. Besides, as mentioned previously, there is only ~11% accumulated improvement over 3 years for image compression.

Additional Cost from Hyperprior

While introducing the hyperprior requires additional bitstream and storage costs, it is also compressed and optimized in an end-to-end manner. The allocated bit number to them is jointly optimized. As shown in Table 2, using the hyperprior model further contributes to size reduction. Besides, almost all image compression works utilize a hyperprior model to improve their performance since its proposition [a] .

[a] Variational image compression with a scale hyperprior, ICLR 2018

Anchor Division

Yes, the anchors that have the same position after quantization are selected for a higher level. The reason that "v_3^{k-1} in Fig. 2(b), which may be far from the quantization center, is selected as the anchor for the upper level" is as follows:

• The quantization center of the voxel does not necessarily have a corresponding anchor.
• If we create an anchor for it, it goes against the core idea of our work, which is using the context model to improve coding efficiency (since the context model does not involve new storage requirements).
• Currently, we select the anchor in the voxel that has the minimum index as elaborated in Eq. 5. This is due to the concern of high efficient implementation and this strategy already demonstrates significant improvements.

Besides, we modified Fig. 2(b) to make it clearer (by swapping the positions of $v_0^{k-1}$ and $v_1^{k-1}$ and adding text description below "Anchor forward") as in the attached PDF. The reason for these modifications is to highlight that the index of anchors in the same voxel is unsorted since all the anchors are stored discretely and unordered, e.g., $\{v_0^{k-1}, v_1^{k-1}, v_2^{k-1}\}$ in the figure. We select the anchor with the minimal index, i.e., $v_0^{k-1}$, which may be either on the border or center of the voxel.

Hope the modified figure is clearer.

The Effects of the Learnable Hyperprior

We included the ablation study of the learnable hyperprior in Table 2. Removing the learnable hyperprior increases the size from 14.00 MB (Ours) to 15.41 MB (Ours w/o HP), demonstrating its effectiveness and complementarity with the context model. The baseline model (Ours w/o CM w/o HP) plus learnable hyperprior (Ours w/o CM) leads to a size reduction from 18.67 MB to 15.03 MB (~20% reduction).

It is worth noting that, different from HAC, the baseline model in our paper (Ours w/o CM w/o HP) already uses the anchor position as a kind of hyperprior. This is one of the reasons why the proposed baseline method (Ours w/o CM w/o HP) can already achieve comparable or even slightly better performance than HAC.

Limitations

We discussed limitations in L391-396 in the submitted paper.

Dear Reviewer otkA,

Thank you very much for your insightful review and valuable suggestions. We have carefully considered your feedback and addressed your comments thoroughly in our rebuttal. We believe the clarifications and improvements we've made effectively address the concerns you raised.

If you find that our responses have satisfactorily resolved the issues, could you please consider adjusting your rating accordingly?

If you still have concerns, we are more than happy to provide any additional information or discuss further if needed.

Additional Highlight on Performance if you still have concerns

We want to highlight that a strong end-to-end entropy baseline is also our contribution since there is no standard implementation and different papers have different entropy models, e.g., Huffman coding in Compact3DGS, and entropy and run-length encoding in Compressed3D. Our proposed entropy framework is much stronger and can be applied to different 3DGS backbones, not only limited to ScaffoldGS.

More results on different 3DGS backbones

We further conducted experiments on the vanilla 3DGS, and the modified 3DGS backbone used in Compact3DGS (CVPR'24). The results on the challenging benchmark BungeeNeRF are as follows

Table R1: The performance of the proposed method on the vanilla 3DGS backbone.

As shown in Table R1, under the same backbone, we achieve 0.9dB PSNR improvements with a size reduction of ~3x compared with the most recent SOTA on the same backbone. We use less than 1% bitrate and achieve better PSNR than the vanilla 3DGS.

Table R2: The performance of the proposed method on the backbone used by Compact3DGS.

Compact3DGS uses a slightly modified backbone from the vanilla 3DGS, incorporating a small MLP for 3D Gaussian color prediction. As shown in Table R2, our method significantly outperforms the latest SOTA method. Compared with Compact3DGS (CVPR'24 Oral), we achieves 2.5dB improvements with a ~5x size reduction.

Because half of the discussion period has passed, please feel free to raise any concerns so that we can better address any potential misunderstandings.

Thank you again for your time and thoughtful consideration.

Best regards,

Authors

Thanks for the detail response and clarification, part of the concerns have been addressed, I have increased the score.

Thanks for your valuable comments!

Novelty

We want to highlight that our method has significant and essential differences from both HAC and ScaffoldGS. The context model, in the generally accepted sense, does not require additional storage. We use already coded anchors (which are part of existing anchors instead of newly created ones) to model uncoded ones in our work. HAC did not use the context model in this definition since they need additional storage for hash features; actually, the termed "context" in HAC is a kind of hyperprior model [1].

The differences in "multi-level" between different papers are summarized as follows:

Taking decompression as an example, HAC decompresses the hash grid feature first and then uses it to help decode the anchor features, i.e., all anchors are decoded at the same time. (As mentioned above, this acts as the role of hyperprior [1].) In our work, we decode some anchors first and use the already decoded anchors to decode the undecoded ones like in an autoregressive manner. (This is the commonly termed context model and no additional bitstream cost is needed for the "extra levels".)

Encoding and Decoding Time

The encoding and decoding time are required by all entropy-coding-based 3DGS compression methods, e.g., HAC (ECCV'24 Oral) and Compact3DGS(CVPR'24 Oral).

For example, compared with Compact3DGS, our decoding speed is much faster as shown in the following table. Besides, the decoding time (17.85s) is worth it compared to the reduced model size (187MB->14MB).

"Especially, the encoding/decoding time of 'w/ APC requires more than 40 sec for a single scene"

This might be a misunderstanding. As indicated in L267-268, the results in "w/ APC" only aim to explain why we do not encode the position, even though it leads to some improvements in rate-distortion performance. Thanks for your comments, but actually we did not use "APC" in all the experiments as indicated in L267-268.

ScaffoldGS vs. 3DGS

While Scaffold-GS needs a per-view decoding process for view-adaptive rendering, it can achieve comparable (or even faster speed in some cases) by limiting the prediction of neural Gaussians to anchors within the view frustum. Scaffold-GS also has better rendering quality than vanilla 3D-GS. It is worth noting that we do not involve encoding or decoding for view-adaptive rendering.

Mask in Scaffold-GS

Thanks for pointing out the typo. The "mask" of Scaffold-GS originally aims to represent the property "opacity (float32, dim=1)" in the saved checkpoint. Since it is not used in the rendering, we will revise it to "N/A" in the revised paper. Other "mask"s in Table 4 refer to the encoded 3D Gaussian-level mask using Lee et al.'s pruning strategy.

Fewer Anchors and Faster Speed

Lee et al. [14] demonstrate that using their proposed masking loss can significantly reduce the number of Gaussians and increase the rendering speed. In our work, any anchor for which all its 3D Gaussians are masked will also be removed. As a result, as shown in the "Number of anchors" in Table 4, compared with ScaffoldGS, which utilizes 61.9K anchors, the proposed method uses only 52.5K anchors, approximately 15% less, and achieves better PSNR.

Our rendering is exactly the same as ScaffoldGS so fewer anchors lead to faster rendering speed when other hyperparameters are the same. We re-trained a model with similar rendering quality to ScaffoldGS, and the results are as follows. We achieve slightly faster rendering speed due to a smaller number of anchors and Gaussian points.

Training Speed

While entropy coding is relatively slow in the inference due to its serial properties, estimating the entropy loss in the training phase is parallel and fast.

As mentioned in our limitations section, estimating the entropy during training will indeed slightly increase the training time. Nevertheless, the additional minor training cost is worthwhile for the reduced size. For example, many existing concurrent works also explicitly estimate the entropy during training, e.g.,

• End-to-End Rate-Distortion Optimized 3D Gaussian Representation, ECCV'24 (The training time is not reported, and the code is not released yet)
• Hash-grid Assisted Context for 3D Gaussian Splatting, ECCV'24

Compared with other models for compression, we have a similar training speed.

"Does not fit standard 3DGS"

The core idea of our context model is not limited to ScaffoldGS, as it does not involve new data formats and does not make any assumptions about the basic elements of the 3D scene. Applying to vanilla 3DGS may be left for future exploration.

Does the encoding anchor in Table 5 denote that APC in Table 4?

Yes, the encoding anchor is the “w/ APC” (w/ Anchor Point Coding) in Table 4. "w/ APC" in Table 5 is only for reference, and we do not use it in the main paper.

Dear Reviewer fG6F,

Thank you very much for your insightful review and valuable suggestions. We have carefully considered your feedback and addressed your comments thoroughly in our rebuttal. We believe the clarifications and improvements we've made effectively address the concerns you raised.

If you find that our responses have satisfactorily resolved the issues, could you please consider adjusting your rating accordingly?

If you still have concerns, we are more than happy to provide any additional information or discuss further if needed.

To further alleviate your concerns in the limitation part, we evaluated the proposed method on both the Compact-3DGS and vanilla 3DGS backbones. The results are presented in the following tables:

Table R2: Performance of the proposed method on the Compact-3DGS backbone (CVPR'24 Oral). Unlike the vanilla 3DGS, Compact-3DGS uses a small MLP to predict the colors of 3D Gaussians.

The results show significant improvements over the most recent SOTA methods on different backbones, strongly demonstrating the significance of the proposed method as a general framework.

Thank you again for your time and thoughtful consideration.

Best regards,
Authors

Dear Reviewer fG6F,

Thanks for your response and we are happy to provide additional information for your concerns.

"The key idea to compress Gaussians (entropy minimization w/ hyper-prior for anchor-based Gaussians) has been explored in the previous approaches."

1. As shown in the title of our paper, the main contribution we claimed is that we are the first to apply the context model to the 3DGS.

2. As shown in our previous comment, our method is not limited to anchor-based Gaussians. On vanilla 3DGS, we can achieve 0.93dB improvements with a ~3x size reduction compared with the CVPR'24 Oral work.

3. We have shown significant improvements of the proposed context model. On the proposed strong entropy baseline (even stronger than the SOTA of ECCV'24), a ~25% improvement is huge. (For your reference, the accumulated improvements on recent entropy-based compression is only ~11% over 3 years.)

4. For the two papers you mentioned in the initial reviewer: The ScaffoldGS does not introduce hyperprior since it does not use entropy coding. HAC is a concurrent work and has significant differences in the hyper-prior design.

• Specifically, HAC was submitted to Arxiv on March 21st (visible even later) and was accepted on July 2nd. Our work submitted the abstract to NeurIPS on May 15th. This is a concurrent work according to the NeurIPS guidelines.

• "Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions. Authors are not expected to compare to work that appeared only a month or two before the deadline." from the NeurIPS 2024 guideline.

• Even though HAC is a concurrent work and is only an Arxiv paper at our submission, we still included it for comparison for completeness. We have significant differences in both motivation and model design, leading to a significantly better performance.

5. Could you please provide references for the papers that share the same core idea besides HAC? As far as we know, there is no previous work that uses a similar idea with us.

6. Our novelty is acknowledged by all other reviewers, e.g., "for the first time, introduce autoregressive entropy models for spatial prediction of anchors." by Reviewer otkA, "The main idea is novel within the 3DGS framework." by Reviewer gvyt, and "The method is novel, integrating the concept of context modeling from the image compression domain." by Reviewer 8ybR.

Additional computational costs

1. With significant performance improvement, slightly increased training overhead shall not be a reason to reject a paper. Many papers even do not report their training time, e.g., [a] accepted by ECCV'24.

[a] End-to-End Rate-Distortion Optimized 3D Gaussian Representation, ECCV'24

2. Our training speed is in fact similar to Compact3DGS (CVPR'24), i.e., around ~1 hour for a city-level scene. However, as shown in the comment, we can achieve 0.46dB improvements with a ~3x size reduction compared with it.

Table R1: The performance of the proposed method on the same backbone with Compact-3DGS.

"Encoding time has not been addressed"

Encoding time is included in the training time since our model directly outputs the bitstreams. If take it out from the pipeline, our encoding time is ~20s for a large-scale city-level dataset. If needed, we will report the detailed encoding time later for comparisons.

Decoding time

As far as we know, almost all the SOTA works for 3DGS compression dependent on coding techniques. Could you provide reference papers that achieve SOTA performance w/o coding techniques?

Besides, the results in our rebuttal demonstrate that our decoding time is faster than other SOTA works.

If you find that our new responses have resolved your concerns, could you please consider adjusting your rating accordingly? Thank you again for your time and please do not hesitate to reply to us if you still have any concerns.

Best regards,

Authors

While we still hold the opinion that slightly increased training overhead shall not be a reason to reject a paper, to further alleviate your concern regarding the training time, we do some preliminary exploration of reducing the number of total training iterations. The results are shown in Table R1. We can achieve much better performance than HAC using less training time. Even using a similar time, we can achieve similar performance with the ScaffoldGS

Table R1: The performance, size, and training time of different methods on Rome/BungeeNerf.

The encoding time (already included in our training time) is ~15s, as a comparison Compact3DGS uses ~50s and HAC uses ~32s.

We believe the clarifications we have made can effectively address all of the concerns you raised. If our responses have resolved your concerns, could you please consider adjusting your rating accordingly? If not, as the discussion period is coming to an end, could you let us know of any remaining concerns?

Thank you for your time.

Best regards,

Authors