CAT-3DGS: A Context-Adaptive Triplane Approach to Rate-Distortion-Optimized 3DGS Compression

[W4] Figure 2 may not be entirely accurate; to my knowledge, CompGS does not utilize a context model.

We clarify that CompGS has its own 3DGS representation (which is different from ScaffoldGS). It includes the anchor and coupled primitives. It does utilize a context model although not autoregressive-based. As stated in Section 3.3 of their paper [R1], "Furthermore, $\tilde{f}{\omega}$ is used as contexts to model the probability distributions of $\tilde{\Sigma}{\omega}$ and $\tilde{g}_{k}$." In their work, the decoded reference embeddings of the anchor primitives serve as the contextual information for coding both the covariance of the anchor primitives and the residual embeddings of the coupled primitives.

We thank the reviewer for their insightful and constructive comments. We have carefully addressed all the concerns and comments and hope the reviewer find our responses satisfactory. If so, we would greatly appreciated if the reviewer could consider reflecting this in the rating for this early attempt at efficient coding of 3DGS representations, which we believe would invite more contributions from the community to advance this largely unexplored yet important area.

References

[R1] X. Liu et al., "Compgs: Efficient 3d scene representation via compressed gaussian splatting," arXiv, 2024.

[W1] The ablation study of PCA guidance need to be included since there are already serval works that utilizes triplanes to compress the 3DGS/Nerf.

First, to our best knowledge, most of the prior works that utilize triplanes for 3DGS/NeRF scene representations do NOT consider the need to encode these representations into compressed bistreams for transmission, which requires entropy coding and rate-distortion optimized training. Our CAT-3DGS distinguishes from these works by focusing specifically on how to entropy encode these scene representations in a rate-distortion optimized fashion. Notably, it involves triplanes as the hyperprior.

Second, as per the request, we follow the ablation setup in the paper to report the BD-rate comparison between CAT-3DGS with and without PCA on Mip-NeRF 360 dataset (9 scenes). The variant without PCA serves as the anchor for BD-rate evaluation. The negative BD-rate number of the variant with PCA shows that PCA brings a marginal rate saving (2.2%). In the paper, we consider PCA to be a pre-processing step, not our major contribution.

[W2] The efficiency of the model. The model seems a bit complicated. The coding time greatly increased due to the proposed context model. Besides, how about the training time?

Thank you for noting this. In reporting the decoding runtime of our CAT-3DGS in the initial submission, the runtime spent on the rate estimation for coding triplanes was accidentally included, leading to our prolonged decoding time. We have updated our decoding time in Table 2 of the revised manuscript and provide the following table, which includes HAC, as well as the initial and revised versions of CAT-3DGS.

As per the request, we provide the training time comparison as follows. The training time of CAT-3DGS is on par with or higher than that of HAC. The longer training time may be attributed to the differences in the training recipe and the use of SARM in estimating the triplane's bit rate. Additionally, for the Amsterdam scene, CAT-3DGS requires approximately twice the training time of HAC. This is due primarily to the larger number of anchor points (1599K anchors), which significantly increases the triplane query time. In contrast, the room scene has relatively fewer anchor points (109K anchors), resulting in a less noticeable increase in training time. Nonetheless, we stress that the training time is highly variable and dependent on the training recipe. It is among our future work to speed up the training process.

[W3] Missing quantitative comparison with ContextGS.

We did not compare our scheme with ContextGS for a number of reasons. First, the code at their GitHub page (https://github.com/wyf0912/ContextGS) is regrettably NOT available yet. Second, the authors reported only two rate-distortion points in their paper, hindering a comprehensive rate-distortion comparison with our and the other existing approaches. Lastly, we notice the ICLR FAQ for Reviewers (https://iclr.cc/Conferences/2025/ReviewerGuide) that "if a paper was published (i.e., at a peer-reviewed venue) on or after July 1, 2024, authors are not required to compare their own work to that paper." At the time of submission, ContextGS was still an arXiv preprint (dated October 1, 2024).

Nonetheless, as per the request, we provide additional rate-distortion plots, including ContextGS, in Figure 4 of our revised manuscript. Note that the results of ContextGS are from their paper. Roughly speaking, our CAT-3DGS performs better or comparably to ContextGS.

Dear Reviewer, we thank you very much for the effort you put into reviewing our paper. We have addressed your comments to the extent possible. As the interactive discussion phase for the rebuttal will end on Nov. 27 AOE, we look very much forward to your further comments (if any). If you find our responses satisfactory, we would ask your kind consideration for increasing your rating on this early and novel attempt at addressing 3DGS coding.

[W1] Autoregressive Model Use: While the channel-wise and pixel-wise autoregressive models are novel in this context, they are a commonly applied strategy in image compression tasks. However, this paper does make a unique contribution by applying it to 3DGS.

We thank the reviewer for recognizing our autoregressive models as being novel in the present context. We stress that this work marks the first attempt to introduce autoregressive models to the triplane-based hyperprior for efficient coding of the 3DGS representation. These techniques along with the view-frequency aware masking achieve the state-of-the-art coding performance. We believe this work will inspire more contributions from the community to advance this largely unexplored yet fast progressing area.

[W2] Lack of comparative analysis with ContextGS.

We did not compare our scheme with ContextGS for a number of reasons. First, the code at their GitHub page (https://github.com/wyf0912/ContextGS) is regrettably NOT available yet. Second, the authors reported only two rate-distortion points in their paper, hindering a comprehensive rate-distortion comparison with our and the other existing approaches. Lastly, we notice the ICLR FAQ for Reviewers (https://iclr.cc/Conferences/2025/ReviewerGuide) that "if a paper was published (i.e., at a peer-reviewed venue) on or after July 1, 2024, authors are not required to compare their own work to that paper." At the time of submission, ContextGS was still an arXiv preprint (dated October 1, 2024).

Nonetheless, as per the request, we provide additional rate-distortion plots, including ContextGS, in Figure 4 of our revised manuscript. Note that the results of ContextGS are from their paper. Roughly speaking, our CAT-3DGS performs better or comparably to ContextGS.

We thank the reviewer for their insightful and constructive comments. We have carefully addressed all the concerns and comments and hope the reviewer find our responses satisfactory. If so, we would greatly appreciated if the reviewer could consider reflecting this in the rating for this early attempt at efficient coding of 3DGS representations, which we believe would invite more contributions from the community to advance this largely unexplored yet important area.

Dear Reviewer, we thank you very much for the effort you put into reviewing our paper. We have addressed your comments to the extent possible. As the interactive discussion phase for the rebuttal will end on Nov. 27 AOE, we look very much forward to your further comments (if any). If you find our responses satisfactory, we would ask your kind consideration for increasing your rating on this early and novel attempt at addressing 3DGS coding.

[W1] The effectiveness of view frequency-aware masking is somewhat ambiguous. It’s unclear whether the anchor configuration (labeled as “w/o VFM” in Fig. 10) excludes masking altogether, or retains the masking operation but removes the weights $p_{n,k}$ only.

Thank you for pointing out this ambiguity. The curve labeled as 'w/o VFM' in Figure 10 still retains the masking operation but removes the weights $p_{n,k}$ only. In the revised manuscript, we have updated the text in lines 514-515 and 743-744 accordingly. Please see the text in red color.

[W2] Spatial redundancies within attributes are not fully exploited. The proposed SARM only addresses inter-redundancies within triplanes (hyperpriors) rather than anchors.

This is an interesting point! In Figure 4 of our revised manuscript, we provide additional rate-distortion plots, including ContextGS, which directly leverages the decoded anchors' attributes as context for decoding the remaining anchors' attributes. We see that our CAT-3DGS outperforms ContextGS on most of the datasets. This suggests that our triplane-based hyperprior and SARM are effective in capturing the inter-redundancies among the anchors' attributes. We remark that ContextGS [R1] and our method could be combined to further enhance the coding efficiency.

We thank the reviewer for their insightful and constructive comments. We have carefully addressed all the concerns and comments and hope the reviewer find our responses satisfactory. If so, we would greatly appreciated if the reviewer could consider reflecting this in the rating for this early attempt at efficient coding of 3DGS representations, which we believe would invite more contributions from the community to advance this largely unexplored yet important area.

References

[R1] Y. Wang et al., "Contextgs: Compact 3d gaussian splatting with anchor level context model," arXiv, 2024.

Dear Reviewer, we thank you very much for the effort you put into reviewing our paper. We have addressed your comments to the extent possible. As the interactive discussion phase for the rebuttal will end on Nov. 27 AOE, we look very much forward to your further comments (if any). If you find our responses satisfactory, we would ask your kind consideration for increasing your rating on this early and novel attempt at addressing 3DGS coding.

[W1] Ablation experiment for CARM is not sufficient, the proposed method is only evaluated on one scenario, e.g., bicycle scene.

Table 1 of our initial submission presents the ablation results on CARM based on all the scenes (9 in total) in the Mip-NeRF 360 dataset. Our unevenly partitioned CARM achieves an 11.9% BD-rate saving compared to the variant without CARM. We shall make this clear in the final paper. Due to the page limit, Figure 7 provides a breakdown analysis for the bicycle scene only. As per the request, we provide additional breakdown results for flower scene and bonsai scene in Figure 11 in Appendix F of our revised manuscript. Consistent with the results presented in our paper, adopting CARM efficiently reduces the compressed size of the latent features.

[W2] The paper is based on the work of HAC. However, the rate parameter setting used by HAC in the comparison experiment is not the same as that used by the proposed method. I doubt the fairness of this setting.

First, we clarify that we adopt the same ScaffoldGS representation as HAC; however, our CAT-3DGS is NOT based on HAC. In fact, our CAT-3DGS differs from HAC in several significant ways: (1) it utilizes triplanes as the hyperprior instead of the hash grids in HAC, (2) it introduces SARM and CARM for contextual entropy coding, which is NOT seen in HAC, and (3) it incorporates a view frequency-aware masking mechanism to filter out unimportant anchors. These novel elements are first proposed in this work.

Second, since our CAT-3DGS is fundamentally different from HAC, it is natural that our rate parameters differ from those of HAC. To ensure a fair comparison and align with the common practice adopted by the learned image/video coding community, we choose the rate parameters for both HAC and our CAT-3DGS in such a way that their reconstructed images span a wide (and approximately the same) range of quality to facilitate BD-rate evaluation. Following this common practice approach, which is widely adopted by the image/video coding community, our CAT-3DGS consistently achieves superior BD-rate results across various datasets, as shown in the following table.

[Note: The BD-rate is a common rate-distortion metric that quantifies the average bitrate saving achieved by a tested codec (the red curve at https://imgur.com/a/eRv2aWN) when compared with an anchor codec (the blue curve) at the same PSNR levels. A negative BD-rate value in percentage terms (e.g. -50%) indicates that when compared with the anchor codec, the tested codec on average achieves a 50% rate saving at the same quality level.]

[Q1] The FARM proposed by the author is interesting. According to the manuscript, the anchor feature is a latent representation of the anchor point, which does not have a clear mathematical correlation like the spherical correlation coefficient of 3DGS. What is the motivation of the proposed channel-wise autoregressive processing of anchor features?

The anchor features are learnable latent parameters and encode collectively an anchor's attributes. During training, there is no constraint imposed to ensure that the anchor features are de-correlated. Our CARM is thus proposed to leverage the potential correlation among the anchor features for better coding efficiency. Our experimental results confirm the effectiveness of CARM, corroborating the intra-correlation among the anchor features.

[Q2] It is noted that the rate parameters setting used by the authors on different datasets are not consistent. Why?

Following the common practice adopted by the learned image/video coding community, we aim to characterize better the rate-distortion performance of the competing methods over a wide range of quality levels and bitrates. We thus extend the rate parameters of HAC reported in their paper to include more rate-distortion points. We then repeat the same process for our CAT-3DGS. Efforts are made to align its quality levels with those of HAC (the major anchor). In doing so, it happens that the same rate parameters are chosen for CAT-3DGS on Mip-NeRF 360, Deep Blending, and Tank & Temple datasets, while slightly different rate parameters are used for BungeeNeRF dataset. We stress that this does not induce any bias towards their rate-distortion performance.

We thank the reviewer for their insightful and constructive comments. We have carefully addressed all the concerns and comments and hope the reviewer find our responses satisfactory. If so, we would greatly appreciated if the reviewer could consider reflecting this in the rating for this early attempt at efficient coding of 3DGS representations, which we believe would invite more contributions from the community to advance this largely unexplored yet important area.

Dear Reviewer, we thank you very much for the effort you put into reviewing our paper. We have addressed your comments to the extent possible. As the interactive discussion phase for the rebuttal will end on Nov. 27 AOE, we look very much forward to your further comments (if any). If you find our responses satisfactory, we would ask your kind consideration for increasing your rating on this early and novel attempt at addressing 3DGS coding.

[W1] Entropy coding is a variable-length coding technique, typically decoded sequentially. However, 3DGC, used in rendering, is a highly parallelized method. The paper does not clearly explain how the proposed compression method can manage this parallelization without impacting rendering speed.

As indicated in lines 529–532 of our paper, the entropy decoding of the Gaussian primitives and the rendering of images are two distinct and decoupled processes. The entropy decoding is an one-off operation that decodes the offsets $\\{\hat{O} _ i\\}^K _ {i=1}$, scaling $\hat{l}$, and feature $\hat{f}$ from the compressed bitstream. Only after these parameters have been entropy decoded can the rendering of images be started, as with the existing works [R1, R2, R3]. The rendering of images can be performed multiple times and with any parallelization techniques applicable to ScaffoldGS.

[W2] The paper does not address decoding complexity. While CHARM enhances compression performance, it could significantly impact decoding speed, which in turn may affect rendering speed.

In Table 2 of our initial submission, we have followed the existing works [R1, R2, R3] to report the decoding time as a rough indicator of the decoding complexity. Also, as noted in our previous response [W1], the entropy decoding of Gaussian primitives and the rendering of images are two decoupled processes. CARM is part of the entropy decoding process. Thus, it does NOT really impact the rendering throughput.

As per the request, we provide additional results comparing between our schemes with and without CARM (the channel-wise AR model). The results indicate that CARM has a negligible impact on decoding time.

[Note: In reporting the decoding runtime of our CAT-3DGS in the initial submission, the runtime spent on the rate estimation for coding triplanes was accidentally included, leading to our prolonged decoding time. We have updated our decoding time in Table 2 and provided this ablation experiment in Appendix G of our revised manuscript. Please see the text in red.]

[W3] The paper does not clarify whether real entropy coding was used during rendering or if the bit-rate was calculated based on theoretical entropy coding.

Yes, at inference time, the real entropy encoding/decoding is performed to generate compressed bitstreams and report our experimental results. In the revised manuscript, we have updated the text in lines 448-450 accordingly. Please see the text in red color.

[Q1] Are there scenarios where CAT-3DGS underperforms? Which modules in CAT-3DGS may contribute to a performance drop?

A very good question! As shown in Figure 4 of our initial submission, our CAT-3DGS achieves the state-of-the-art compression results across all the test datasets. However, its coding gain over HAC is relatively smaller on the large-scale BungeeNeRF dataset. Currently, the three triplanes are set to have the same spatial dimensions. We conjecture that this setting is sub-optimal for BungeeNeRF, where most of the Gaussian primitives are densely distributed along one major plane.

We thank the reviewer for their insightful and constructive comments. We have carefully addressed all the concerns and comments and hope the reviewer find our responses satisfactory. If so, we would greatly appreciated if the reviewer could consider reflecting this in the rating for this early attempt at efficient coding of 3DGS representations, which we believe would invite more contributions from the community to advance this largely unexplored yet important area.

Reference
[R1] X. Liu et al.,"Compgs: Efficient 3d scene representation via compressed gaussian splatting," arXiv, 2024.
[R2] Y. Chen et al., "Hac: Hash-grid assisted context for 3d gaussian splatting compression," arXiv, 2024.
[R3] Y. Wang et al., "Contextgs: Compact 3d gaussian splatting with anchor level context model," arXiv, 2024.

Dear Reviewer, we thank you very much for the effort you put into reviewing our paper. We have addressed your comments to the extent possible. As the interactive discussion phase for the rebuttal will end on Nov. 27 AOE, we look very much forward to your further comments (if any). If you find our responses satisfactory, we would ask your kind consideration for increasing your rating on this early and novel attempt at addressing 3DGS coding.